Architecture of Transformers in Gen AI

The Transformer Architecture in Generative AI

Large Language Models (LLMs) built upon the Transformer architecture have revolutionized the field of Natural Language Processing (NLP) and Generative AI. They have demonstrated superior performance compared to earlier recurrent architectures like Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks in a wide array of tasks, including machine translation, text summarization, question answering, and complex text generation.

The remarkable capabilities of Transformers stem from their unique architecture, first introduced in the seminal paper “Attention Is All You Need” by Vaswani et al. (2017). This chapter will explain the core concepts of the original Transformer model in an accessible manner, focusing on its key components: the encoder, the decoder, and the innovative self-attention mechanism that is central to its operation.

Conceptual Overview: How Transformers Process Information

Let’s consider how a Transformer conceptually processes a sentence for a task like translation or text generation:

• Simultaneous Word Consideration with Attention: Unlike RNNs that process words sequentially, Transformers can consider all words in an input sentence simultaneously. The ‘attention mechanism’ allows the model to weigh the importance of different words relative to each other when processing any given word. For example, when processing the word ‘it’ in “The animal didn’t cross the street because it was too tired,” attention helps identify that ‘it’ refers to ‘animal’.
• Identifying Relationships: By processing words in context with each other, the Transformer learns complex dependencies and relationships between words, even if they are far apart in the sentence. This helps in understanding nuanced meanings.
• Building Rich Representations: The input sentence is transformed into a rich, contextualized representation that captures not just individual word meanings but also how they interact to form the overall meaning of the sentence.
• Generating Output: For generative tasks, the Transformer uses these rich representations and the attention mechanism to generate an output sequence (e.g., a translation or a continuation of text) one word (or token) at a time, always considering the context of what has been generated so far and the input representation.

Core Transformer Architecture: Encoder and Decoder

The original Transformer model comprises two main parts: an Encoder stack and a Decoder stack. Below is a textual description of its high-level architecture:

Imagine a diagram with two main blocks. On the left is the Encoder stack, which receives the input sequence (e.g., a sentence in English). The input sequence first passes through an ‘Input Embedding’ layer and a ‘Positional Encoding’ layer. Then, it goes through a series of N identical encoder layers. Each encoder layer has two main sub-layers: a ‘Multi-Head Self-Attention’ mechanism and a ‘Position-wise Feed-Forward Network’. There are residual connections around each of these two sub-layers, followed by layer normalization.

On the right is the Decoder stack, which generates the output sequence (e.g., the translated sentence in French). The decoder also takes the previously generated output sequence as input (shifted right, and also passed through ‘Output Embedding’ and ‘Positional Encoding’). It too consists of N identical decoder layers. Each decoder layer has three main sub-layers: a ‘Masked Multi-Head Self-Attention’ mechanism, a ‘Multi-Head Cross-Attention’ mechanism (which also receives the output from the Encoder stack), and a ‘Position-wise Feed-Forward Network’. Similar to the encoder, residual connections and layer normalization are applied around each sub-layer. Finally, the output of the decoder stack passes through a Linear layer and a Softmax layer to produce probability distributions over the target vocabulary for the next word/token.

Notably, the Transformer architecture discards recurrence (found in RNNs/LSTMs) entirely and relies solely on attention mechanisms to draw global dependencies between input and output.

Let’s discuss the key components and sub-layers in more detail:

Input/Output Processing

• Embeddings: Words (or sub-word tokens) from the input and output sequences are first converted into dense vector representations called embeddings. This allows the model to work with numerical data.
• Positional Encoding: Since Transformers process all words in a sequence simultaneously and lack inherent sequential awareness like RNNs, positional encodings are added to the embeddings. These encodings provide information about the position of each word in the sequence.

The Encoder Stack: Processing the Input

The encoder’s role is to process the entire input sequence and transform it into a sequence of continuous representations (contextual embeddings) that capture the meaning and context of each word in relation to the whole sequence. The encoder stack in the original paper consists of N=6 identical layers (though this number varies in modern LLMs).

A single encoder layer can be visualized as follows: The input (embedding + positional encoding for the first layer, or output from previous layer) first enters a ‘Multi-Head Self-Attention’ sub-layer. The output of this attention sub-layer is then added to its input (a residual connection) and passed through ‘Layer Normalization’. This normalized output then goes into a ‘Position-wise Feed-Forward Network’ sub-layer. Again, its output is added to its input (residual connection) and passed through ‘Layer Normalization’. This final normalized output is the output of the encoder layer, which is then fed to the next encoder layer or to the decoder stack.

Each encoder layer contains two main sub-layers:

• Multi-Head Self-Attention Mechanism: This allows each word in the input sequence to ‘attend’ to all other words in the sequence (including itself) to understand how they relate to each other. ‘Multi-head’ means it performs several attention calculations in parallel, each focusing on different aspects of the relationships, and then combines their outputs.
• Position-wise Feed-Forward Network (FFN): This is a simple fully connected feed-forward network applied independently to each position (i.e., to each word’s representation from the attention layer). It consists of two linear transformations with a ReLU activation in between, helping to further process and transform the representations.

Residual Connections and Layer Normalization: Each sub-layer (attention and FFN) in an encoder layer has a residual connection around it, followed by layer normalization. The output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself (e.g., self-attention). Residual connections help with gradient flow in deep networks, preventing vanishing gradients, while layer normalization stabilizes the activations.

The Decoder Stack: Generating the Output

The decoder’s role is to generate the output sequence (e.g., the translated sentence) one token at a time. It takes the encoder’s output (the contextual representations of the input sequence) and the previously generated tokens of the output sequence as inputs. Like the encoder, the decoder stack in the original paper also consists of N=6 identical layers (this also varies in modern LLMs).

A single decoder layer has a slightly more complex structure: The input (embedding of previously generated tokens + positional encoding for the first layer, or output from previous decoder layer) first enters a ‘Masked Multi-Head Self-Attention’ sub-layer. Its output is added to its input and normalized. This then goes to a ‘Multi-Head Cross-Attention’ sub-layer, which also takes the output of the entire encoder stack as an additional input. The output of this cross-attention is added to its input and normalized. Finally, this result passes through a ‘Position-wise Feed-Forward Network’, followed by another residual connection and layer normalization. This is the output of one decoder layer.

Each decoder layer contains three main sub-layers:

• Masked Multi-Head Self-Attention Mechanism: This is similar to the self-attention in the encoder, but with a crucial modification: it incorporates a mask. This mask ensures that when predicting the current output token, the decoder can only attend to previously generated tokens in the output sequence and not to future tokens. This maintains the autoregressive property (generating one token at a time based on past tokens).
• Multi-Head Cross-Attention Mechanism (Encoder-Decoder Attention): This sub-layer allows each position in the decoder to attend to all positions in the input sequence (via the encoder’s output). This is where the decoder links the input context to the output being generated. For example, in translation, it helps the decoder decide which input words are most relevant for predicting the current output word.
• Position-wise Feed-Forward Network (FFN): This is identical in structure to the FFN in the encoder layer.

Similar to the encoder, residual connections and layer normalization are applied around each of these three sub-layers.

After the final decoder layer, the output representations are passed through a final linear layer (to project to the vocabulary size) and a softmax function. The softmax converts these scores into probabilities, indicating the likelihood of each word in the vocabulary being the next word in the output sequence. The word with the highest probability is typically chosen (though other sampling strategies exist).

Conclusion

This chapter has detailed the fundamental architecture of Transformers in Generative AI, focusing on its two primary components: the encoder and the decoder, and the central role of attention mechanisms. The original Transformer proposed N=6 layers for both encoder and decoder, but modern implementations, especially in Large Language Models, often use significantly more layers to achieve greater capacity and performance.

The encoder’s role is to build a rich, contextual understanding of the input sequence using self-attention and feed-forward layers. The decoder then utilizes this understanding, along with masked self-attention on its own outputs and cross-attention to the encoder’s output, to generate the target sequence step-by-step.

The Transformer architecture, by replacing recurrence with attention, has enabled parallel processing of sequences, leading to more efficient training on large datasets and the ability to capture long-range dependencies effectively. This design has fundamentally transformed the landscape of natural language processing and is the backbone of most state-of-the-art generative AI models for text, and increasingly, for other modalities as well.