Positional Encoding

Understanding Positional Encoding in Transformer Models

Transformer models process input tokens (words, sub-words, or characters) by first converting them into vector representations called input embeddings. However, these embeddings themselves do not inherently contain information about the order or position of tokens within a sequence. Since word order is crucial for meaning in language (e.g., “dog bites man” vs. “man bites dog”), Transformers need a mechanism to incorporate this positional information. This is achieved through Positional Encoding.

Positional encodings are added to the input embeddings at the bottom of the encoder and decoder stacks. This chapter explains what positional encoding is, why it’s necessary, how a common method works, and provides a conceptual Python implementation.

What is Positional Encoding?

Positional Encoding is a technique used in Transformer models to inject information about the relative or absolute position of tokens in a sequence. Since the self-attention mechanism in Transformers processes all tokens in parallel, it doesn’t inherently consider their order. Positional encodings provide this missing sequential context.

In the original Transformer architecture, the positional encoding vectors have the same dimension as the input embeddings (d_model), allowing them to be directly added together. This combined representation (embedding + positional encoding) then serves as the input to the subsequent layers of the Transformer.

The positional encoding component is typically located immediately after the input embedding sub-layer in both the encoder and decoder parts of the Transformer. The input embeddings provide semantic meaning of the tokens, while positional encodings provide their context within the sequence.

Why is Positional Encoding Necessary in Transformers?

Unlike Recurrent Neural Networks (RNNs) or Long Short-Term Memory networks (LSTMs) which process sequences token by token and thus inherently incorporate order, Transformers process all tokens in a sequence simultaneously. The self-attention mechanism calculates attention scores between all pairs of tokens, regardless of their distance. Without positional information, the Transformer would treat a sentence like a ‘bag of words,’ losing crucial sequential meaning.

For example, the sentences “The cat chased the dog” and “The dog chased the cat” use the same words but have different meanings due to word order. Positional encodings ensure that the model can distinguish between such cases by making the representation of each token unique based on its position in the sequence.

How Does Sinusoidal Positional Encoding Work?

While positional information could be learned (e.g., as learnable embedding vectors for each position), the original Transformer paper by Vaswani et al. (2017) proposed using fixed sinusoidal functions. This method has several advantages: it can generalize to sequences longer than those seen during training, and it allows the model to easily learn to attend by relative positions, since PE(pos+k) can be represented as a linear function of PE(pos).

The positional encoding (PE) for a token at position ‘pos’ in the sequence and dimension ‘i’ of the encoding vector is defined using sine and cosine functions of different frequencies:

$$\mathrm{PE}(\text{pos}, 2i) = \sin\left(\frac{\text{pos}}{10000^{\frac{2i}{d_{\text{model}}}}}\right)$$

$$\mathrm{PE}(\text{pos}, 2i+1) = \cos\left(\frac{\text{pos}}{10000^{\frac{2i}{d_{\text{model}}}}}\right)$$

Where:

• pos is the position of the token in the sequence (0, 1, 2,…).
• i is the dimension index within the positional encoding vector (0, 1, …, d_model/2 - 1).
• d_model is the dimensionality of the embeddings (and thus, of the positional encoding vector, e.g., 512).
• The term $10000^{\frac{2i}{d_{\text{model}}}}$ creates wavelengths that range from $2\pi$ to $10000 \cdot 2\pi$. Each dimension of the positional encoding corresponds to a sinusoid. For even dimensions (2i), a sine function is used, and for odd dimensions (2i+1), a cosine function is used.

This formulation ensures that each position has a unique encoding. Moreover, the choice of sinusoidal functions allows the model to potentially extrapolate to sequence lengths longer than encountered during training.

Conceptual Python Implementation of Sinusoidal Positional Encoding

Below is a Python script using NumPy to generate sinusoidal positional encodings and add them to example input embeddings. This demonstrates the core concept.

import numpy as np

# --- Step 1: Example Text, Tokenization, and Input Embeddings ---
# Example sentence and basic tokenization
text = "Transformers use positional encoding"
tokens = text.lower().split()

# Create a simple vocabulary
vocab = {word: idx for idx, word in enumerate(tokens)}

# Example input: sequence of token indices
input_indices = np.array([vocab[word] for word in tokens])

print(f"Vocabulary: {vocab}")
print(f"Input Indices: {input_indices}")

# Parameters for embeddings
vocab_size = len(vocab)
embed_dim = 6 # Using a small dimension for easier visualization (d_model)

# Initialize a dummy embedding matrix (e.g., learned during training)
# In a real scenario, these would be meaningful learned vectors.
np.random.seed(42) # for reproducibility
embedding_matrix = np.random.rand(vocab_size, embed_dim)

# Get the embeddings for the input indices
input_embeddings = embedding_matrix[input_indices]

print(f"\nEmbedding Matrix (shape: {embedding_matrix.shape}):\n{embedding_matrix}")
print(f"Input Embeddings (shape: {input_embeddings.shape}):\n{input_embeddings}")

# --- Step 2: Positional Encoding Function ---
def get_positional_encoding(max_seq_len, d_model):
    """Generates sinusoidal positional encodings.

    Args:
        max_seq_len: Maximum sequence length.
        d_model: Dimensionality of the model (embedding dimension).

    Returns:
        A numpy array of shape (max_seq_len, d_model) with positional encodings.
    """
    pe = np.zeros((max_seq_len, d_model))
    position = np.arange(0, max_seq_len).reshape(-1, 1) # Column vector for positions

    # Term for calculating frequencies/wavelengths
    # div_term for even dimensions (0, 2, 4, ...)
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))

    pe[:, 0::2] = np.sin(position * div_term) # Apply sin to even indices in pe
    pe[:, 1::2] = np.cos(position * div_term) # Apply cos to odd indices in pe

    return pe

# --- Step 3: Generate and Apply Positional Encodings ---
max_len_sequence = len(tokens) # Max length for this specific sequence

# Generate positional encodings
pos_encodings = get_positional_encoding(max_len_sequence, embed_dim)

print(f"\nPositional Encodings (shape: {pos_encodings.shape}):\n{pos_encodings}")

# Add positional encodings to input embeddings
# The `pos_encodings` matrix might be pre-calculated for a max possible length.
# Here, we take only up to the current sequence length.
# (In this example, max_len_sequence is already len(tokens), so pos_encodings[:len(tokens)] is just pos_encodings)
final_embeddings = input_embeddings + pos_encodings

print(f"\nInput Embeddings with Positional Encoding (shape: {final_embeddings.shape}):\n{final_embeddings}")

Expected Output (Illustrative)

Running the script will produce output showing the vocabulary, input indices, the randomly initialized embedding matrix, the input embeddings derived from it, the generated positional encodings, and finally, the input embeddings combined with positional encodings. The shapes of the matrices will also be printed. (Note: exact numerical values for random embeddings will vary unless a seed is fixed as in the example).

Vocabulary: {'transformers': 0, 'use': 1, 'positional': 2, 'encoding': 3}
Input Indices: [0 1 2 3]

Embedding Matrix (shape: (4, 6)):
[[0.37454012 0.95071431 0.73199394 0.59865848 0.15601864 0.15599452]
 [0.05808361 0.86617615 0.60111501 0.70807258 0.02058449 0.96990985]
 [0.83244264 0.21233911 0.18182497 0.18340451 0.30424224 0.52475643]
 [0.43194502 0.29122914 0.61185289 0.13949386 0.29214465 0.36636184]]
Input Embeddings (shape: (4, 6)):
[[0.37454012 0.95071431 0.73199394 0.59865848 0.15601864 0.15599452]
 [0.05808361 0.86617615 0.60111501 0.70807258 0.02058449 0.96990985]
 [0.83244264 0.21233911 0.18182497 0.18340451 0.30424224 0.52475643]
 [0.43194502 0.29122914 0.61185289 0.13949386 0.29214465 0.36636184]]

Positional Encodings (shape: (4, 6)):
[[ 0.          1.          0.          1.          0.          1.        ]
 [ 0.84147098  0.54030231  0.2171651   0.97613297  0.04642049  0.99892228]
 [ 0.90929743 -0.41614684  0.42481076  0.9052784   0.09259899  0.99569976]
 [ 0.14112001 -0.9899925   0.6137753   0.789471    0.13830076  0.99039312]]

Input Embeddings with Positional Encoding (shape: (4, 6)):
[[ 0.37454012  1.95071431  0.73199394  1.59865848  0.15601864  1.15599452]
 [ 0.89955459  1.40647845  0.81828011  1.68420555  0.06699498  1.96883213]
 [ 1.74174007 -0.20380773  0.60663573  1.08868291  0.39684123  1.52045619]
 [ 0.57306503 -0.69876336  1.22562819  0.92896486  0.43044541  1.35675496]]

Alternative: Some Transformer implementations use learned positional embeddings instead of fixed sinusoidal ones. These are parameters that are learned during the training process, similar to token embeddings. Both approaches have their pros and cons.

Conclusion

Positional encoding is a vital component of the Transformer architecture, enabling it to process sequences by providing information about the order of tokens. The sinusoidal method is a common and effective way to achieve this, offering unique encodings for each position that can generalize to different sequence lengths.

Understanding positional encoding is crucial for grasping how Transformers effectively model sequential data for complex NLP tasks like translation, summarization, and generation. It’s a clever solution to integrate sequential awareness into a parallel processing architecture.