Introduction

Often when I work with LLMs it is with text inputs, but I also use image inputs too sometimes. I continue to spend time learning the internals of the transformer architecture used in decoder style LLMs. But most of my studying has been with text inputs. I was always a little curious, and confused, on how images were passed into LLMs. I just never took the time to dig into it more. Now I am finally getting around to it. This blog post is some notes I'm taking as I learn more about this topic. I write about things so I can better understand them, and my future self is always grateful.

The main motivation for this post was Sebastian Raschka's recent blog post Understanding Multimodal LLMs. I highly recommend reading it. I'm going to focus on learning a subset of the topics that his blog post discusses. I am starting with focusing on what he refers to as Method A: Unified Embedding Decoder Architecture. I may go deeper on other topics in other blog posts, but for now this is a good start for me.

High Level Overview

Large Language Models (LLMs) have revolutionized the way we interact with text, enabling capabilities like natural language generation, reasoning, and conversation. However, their utility isn’t limited to text alone. Modern multimodal models can also understand and process images. How exactly are images passed into these models? That’s what this post aims to clarify.

This post starts with a recap of how transformer-based LLMs process text inputs. We then transition to how images can be converted into sequences of embeddings (just like tokens in text) and fed into LLMs. We’ll look at Vision Transformers (ViT), show how they encode images, and finally explain how these embeddings can be integrated into decoder-style LLMs for multimodal tasks such as image captioning and visual question answering.

Key Takeaways:

• Transformers fundamentally operate on sequences of embeddings—be it text tokens or image patches.
• Images are typically processed by a specialized transformer-based image encoder (like ViT) into a sequence of patch embeddings.
• These image patch embeddings can then be projected into the decoder LLM’s embedding space, allowing the LLM to accept and reason over both text and images.
• Text → tokens → embeddings → transformer
• Images → patches → embeddings → transformer

Recap of Transformer Architecture for Text Inputs

Decoder Style LLMs

We first need to have an understanding of the transformer architecture used in decoder style LLMs. Earlier this year I wrote my first blog post with some notes on the transformer architecture. To get the most out of this post, it would be good to have some familiarity with the transformer architecture. We will give a quick reminder of some basic concepts.

Most LLMs you interact with (like GPT-style models) are decoder-only transformers. In a decoder transformer:

• Input text is first tokenized into discrete tokens.
• Each token is mapped to an embedding vector from a learned embedding lookup table.
• A sequence of token embeddings is passed through the transformer layers (which use self-attention and feed-forward layers).
• The model outputs a hidden state for each token, which is then passed through a classification head to predict the probability distribution over the next token.

We will load one of the SmolLM2 LLM models created by the Hugging Face team. This is not the instruction fine tuned model, but rather the base pre-trained model. This model may not be as well known as some of the other models, but it is a good model to start with since it is really small and easy to run locally.

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M")
model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M")
model

The input to the transformer layers is a sequence of embeddings. In the case of text inputs, the input first gets converted into a sequence of tokens. Then each token is converted into an embedding vector.

Here is the conversion of the input text to tokens ids.

inputs = tokenizer(["The dog jumped over the"], return_tensors="pt")
input_ids = inputs.input_ids
print(inputs)
print(input_ids.shape)
print(input_ids)

Each token id has an associated embedding vector. In the case of this SmolLM2 model, the embedding dimension is 576 and there are 49152 tokens in the vocabulary.

embedding_lkp = model.model.embed_tokens
print(embedding_lkp.weight.shape)

We can get the token embeddings by passing the token ids to the embedding lookup table. Each row of the returned tensor, ignoring the batch dimension, is a vector representation of a token.

embedding_vectors = embedding_lkp(input_ids)
print(embedding_vectors.shape)
print(embedding_vectors)

It is this sequence of embedding vectors that flows through the transformer layers. The input shape to the transformer layers is (batch_size, sequence_length, embedding_dim) and the output shape is (batch_size, sequence_length, hidden_size). You can get the last hidden state by passing the inputs to the model, excluding the final classification head.

last_hidden_state = model.model(**inputs).last_hidden_state
print(last_hidden_state.shape)
last_hidden_state

Then this final transformer output is passed to the classification head. The classification head is a single linear layer that maps the hidden state to the logits for the next token. The output shape of the classification head is (batch_size, sequence_length, vocab_size).

logits = model.lm_head(last_hidden_state)
assert torch.allclose(logits, model(**inputs).logits)
logits.shape

Next we convert the logits to probabilities using the softmax function. While this is useful for visualization and inference, during training we typically use the raw logits directly with CrossEntropyLoss for better numerical stability. Note that we get logits (and after softmax, probabilities) for the next token at each position in the sequence. During inference, we typically only care about the last position's values since that's where we'll generate the next token.

probs = F.softmax(logits, dim=-1)
probs.shape

This next code block shows that at inference time we get the probabilities for the next token at each position in the sequence. It prints the top 5 predictions for each token in the sequence.

K = 5  # Number of top predictions to show
top_probs, top_indices = torch.topk(probs[0], k=K, dim=-1)  # Remove batch dim and get top K

# Convert token indices to actual tokens and print predictions for each position
input_text = tokenizer.decode(input_ids[0])  # Original text
print(f"Original text: {input_text}\n")

for pos in range(len(input_ids[0])):
    token = tokenizer.decode(input_ids[0][pos])
    print(f"After token: '{token}'")
    print(f"Top {K} predicted next tokens:")
    for prob, idx in zip(top_probs[pos], top_indices[pos]):
        predicted_token = tokenizer.decode(idx)
        print(f"  {predicted_token}: {prob:.3f}")
    print()

In summary, the input to the transformer layers is a sequence of embeddings, of shape (batch_size, sequence_length, embedding_dim). The transformer layers process this sequence and return a new sequence of hidden states, of shape (batch_size, sequence_length, hidden_size). It is often the case that the hidden size is the same as the embedding dimension, but this is not a requirement. Even if you forget the details of the inner workings of the transformer layers (self attention, etc.), this is a useful mental model to keep in mind. The final classifier layer returns a probability distribution over the next token for each position in the sequence, of shape (batch_size, sequence_length, vocab_size).

Encoder Models

In contrast, encoder-only models like BERT process the entire input sequence at once without causal masking. They often use a special [CLS] token at the start of the sequence, whose final embedding serves as a global representation of the entire input for tasks like classification. Here are some of the key differences between decoder and encoder models:

• Attention Masking:

  • Decoder: Uses causal (or triangular) masked attention to ensure that each position can only attend to previous positions, enforcing an autoregressive quality. This prevents the model from "seeing the future," which is essential for tasks like text generation.
  • Encoder: Doesn't require causal masking because it processes the entire input sequence at once. Each token can attend to every other token in the sequence, providing a comprehensive context.

• Purpose and Data Flow:

  • Decoder: Designed for autoregressive tasks, where each output token is generated one by one, conditioning on previously generated tokens. This step-by-step generation is central to tasks like text generation, where each token "builds" upon the preceding tokens.
  • Encoder: Designed to encode the entire input sequence into a contextualized representation in one shot, capturing relationships across the whole sequence. It’s typically used in understanding or embedding tasks and classification tasks, where the model needs a holistic view of the input.

Let's load a simple encoder model to illustrate some points.

from transformers import AutoModel

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModel.from_pretrained("distilbert-base-uncased")
model

inputs = tokenizer(["The dog jumped over the"], return_tensors="pt")
input_ids = inputs.input_ids
print(input_ids)
print(tokenizer.decode(input_ids[0]))
last_hidden_state = model(**inputs).last_hidden_state
print(last_hidden_state.shape)

In the case of encoder models, the output shape is (batch_size, sequence_length, hidden_size). In the case of Bert, the hidden size is 768 and a 768 dimensional vector is returned for each token in the input sequence.

When using the encoder output for other tasks, such as classification, we typically take the [CLS] token embedding, which is the embedding for the first token.

last_hidden_state[:, 0, :].shape  # `[CLS]` token embedding,

I think it's worth elaborating on the importance of the [CLS] token. Why Use the [CLS] token embedding as the final representation?

When processing text with transformer-based models, each input sequence usually begins with a special token, in this case the [CLS] token. This token doesn’t represent a word or phrase from the input but acts as a placeholder for capturing information about the entire sequence. During training, the [CLS] token is specifically optimized for sequence-level tasks like classification. For example, in sentiment analysis, the model learns to encode the overall sentiment of the input sequence into the [CLS] token embedding. As a result, the [CLS] token becomes a rich summary representation of the entire input sequence. Self-attention mechanisms allow the [CLS] token to attend to all other tokens in the sequence. This means it “sees” the entire context of the input. Recall that we typically don't use the masked attention in the encoder model. Through this process:

• The [CLS] token learns to aggregate information from all other tokens.
• It serves as a global representation, capturing both local token-level features and high-level semantic patterns.
• Adapts dynamically to the task during fine-tuning.

Using the [CLS] token provides a single, fixed-size vector (e.g., 768 dimensions for BERT) that can directly feed into a classifier or other downstream layers. These concepts are useful to keep in mind when we discuss image encoders later on.

Introducing Images into Transformers and LLMs

Now that we've revisited how transformers handle text inputs, let's explore how images can be incorporated into transformers. Our eventual goal is to understand how to pass images into decoder style LLMs, along side text, to generate text outputs. If you remember that the input to the transformer layers is a sequence of embeddings, then passing in images is no different. We just need to convert the images into a sequence of embeddings suitable for the transformer layers. The fundamental idea: Transformers work on sequences of embeddings. Text tokens are straightforward; they map directly from discrete tokens to embeddings via lookup tables. Images, on the other hand, must be transformed into a sequence of patch embeddings.

Key Insight:

• Text → tokens → embeddings → transformer
• Image → patches → embeddings → transformer

By treating images as a sequence of flattened patches, we can feed them into a transformer architecture—just like we feed tokens into a text transformer.

We’ll first look at how images are handled by Vision Transformers (ViT). Then we’ll explore models like CLIP, which bridge text and image embeddings, and finally see how these image embeddings are integrated into LLMs for multimodal tasks.

Vision Transformers (ViT)

The first architecture we will focus on is transformer-based image encoders. Specifically, we will examine the Vision Transformer (ViT), a model that adapts the transformer architecture from natural language processing to computer vision tasks. The ViT processes images by dividing them into fixed-size patches, embedding these patches as input tokens, and applying a transformer encoder to learn meaningful representations of the input image. Just like the transformer layers process a sequence of token embeddings, the ViT processes a sequence of image patch embeddings, and returns a sequence of hidden states.

Key Steps for ViT:

• Patch Extraction: Divide the image into non-overlapping patches (e.g., 16x16).
• Flatten + Project: Flatten each patch and apply a linear projection to get a 1D embedding vector.
• Positional Embeddings: Add positional embeddings so the model knows each patch’s location.
• [CLS] Token: Prepend a learnable [CLS] token to represent the entire image.
• Transformer Encoder: Pass this sequence (patch embeddings + [CLS]) through the encoder layers.
• Global Representation: The final hidden state corresponding to the [CLS] token serves as a global image representation.

The First Vision Transformer

The first Vision Transformer was introduced in the paper AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE.

• Split the image into equal sized patches of size 16x16x3 pixels.
• Flatten each patch into a 1D vector of size 16x16x3 = 768.
• Put each flattened patch representation through a linear projection layer to embed each patch into a vector of size 768.
  • This is the patch embedding.
  • Add a learnable positional embedding to each patch embedding. The positional embeddings help the model understand the spatial relationships between patches
  • Also add a [CLS] token embedding to start of the sequence.
• Pass the sequence of patch embeddings and [CLS] token embedding, which is a sequence length of 197, through the transformer layers (encoder) to produce a new sequence of hidden states.
• The transformed [CLS] token representation (the first position in the final hidden states) serves as a representation of the entire image and can be used as input to a classifier for downstream tasks.

We can load such a pre-trained ViT model from Hugging Face.

from PIL import Image

image = Image.open("imgs/underwater.jpg")
image

from transformers import ViTImageProcessor, ViTModel

processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
model = ViTModel.from_pretrained("google/vit-base-patch16-224-in21k")
model

inputs = processor(images=image, return_tensors="pt")
inputs.keys()

inputs.pixel_values.shape  # (batch_size, num_channels, height, width)

# Create patch embeddings from image
patch_embeddings = model.embeddings.patch_embeddings(inputs.pixel_values)
patch_embeddings.shape  # [1, 196, 768]

# Get complete input embeddings and pass through encoder manually.
# These embeddings are the patch embeddings plus the positional embeddings.
# As well as the `[CLS]` token embedding.
full_input_embeddings = model.embeddings(inputs.pixel_values)
encoder_outputs = model.encoder(full_input_embeddings)
manual_output = model.layernorm(encoder_outputs.last_hidden_state)

# Get output using full model forward pass
with torch.no_grad():
    model_outputs = model(**inputs)
    full_model_output = model_outputs.last_hidden_state

# Verify shapes match
assert manual_output.shape == full_model_output.shape

# Verify outputs are identical
assert torch.allclose(manual_output, full_model_output, atol=1e-6)

model_outputs.keys()

model_outputs.last_hidden_state.shape  # (batch_size, sequence_length, hidden_size)

model_outputs.last_hidden_state[:, 0, :].shape  # `[CLS]` token embedding

In the case of the ViT encoder model, followed by a classification task/head, it is this [CLS] token embedding that we will use as the image representation for downstream tasks. Just like in text encoders, the [CLS] token in ViT learns to aggregate information from all the image patches through self-attention. During training, this token's representation is optimized to capture the global features needed for image classification. However, it's important to note that this [CLS] token approach is specific to encoder-based vision transformers used for classification tasks. When we later discuss feeding images into decoder style LLMs for tasks like image captioning or visual question-answering, we'll see a different approach where the sequence of patch embeddings themselves are used directly, without needing a [CLS] token.

We've now seen how Vision Transformers process images in a way that's analogous to how text transformers process words. The image is divided into patches (like words in a sentence), each patch is flattened from a 16x16x3 grid of pixels into a 768-dimensional vector, then transformed through a learned linear projection layer to create patch embeddings (like word embeddings). Positional embeddings are added to maintain spatial information (like position encodings in text). The key insight is that both text and image transformers fundamentally operate on sequences of embeddings - the main difference is just in how we create these embeddings from the raw input. For ViT, it's through patch extraction, flattening, and linear projection; for text, it's through token lookup tables.

CLIP

CLIP (Contrastive Language-Image Pre-training) represents a significant milestone in connecting visual and textual understanding. Unlike the original ViT which focused solely on image classification, CLIP learns to understand the relationship between images and their natural language descriptions. CLIP was created by OpenAI.

CLIP's architecture consists of two encoders working in parallel:

A text encoder (transformer) that:

• Processes sequences of text tokens
• Produces a final representation using the [CLS] token
• Projects this representation into a normalized embedding space

An image encoder (can be ViT or other CNN architecture but let's focus on ViT) that:

• Processes images as sequences of patch embeddings
• Transforms these through transformer layers
• Uses the [CLS] token for final representation
• Projects into the same normalized embedding space (Note: While CLIP can also use ResNet CNN architectures that don't use patches, we're focusing on the ViT version)

The key innovation is the contrastive learning process:

• Pairs of images and text descriptions are encoded into the same embedding space
• The model learns to maximize similarity between matching pairs while minimizing similarity for non-matching pairs
• This creates a shared semantic space where similar concepts in either modality (image or text) end up close together

This aligned semantic space enables powerful capabilities:

• Comparing images and text directly
• Finding semantic similarities across modalities

from PIL import Image
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

image = Image.open("imgs/tropical_island.jpg")
image