Sequence Processing with Convents

PyTorch - Sequence Processing with 1D Convolutional Neural Networks (ConvNets)

Convolutional Neural Networks (CNNs or ConvNets), while famously successful in computer vision, are also highly effective for sequence processing tasks. Instead of 2D convolutions (for images), 1D convolutions are applied to sequence data like text or time series. This chapter introduces how to build and use 1D ConvNets in PyTorch for sequence modeling.

1D ConvNets apply filters across sequences, capturing local patterns or n-grams (in text). Key advantages include parameter sharing (reducing model size) and hierarchical feature learning. They can be very efficient and performant for tasks like text classification.

Step 1: Core PyTorch Imports

We’ll need standard PyTorch modules for building our network.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

Step 2: Defining a Basic 1D ConvNet for Sequences

Let’s define a simple 1D ConvNet suitable for a task like text classification. The typical architecture involves:

An embedding layer to convert input token IDs into dense vectors.
One or more 1D convolutional layers to extract features.
An activation function (e.g., ReLU) after convolutions.
A pooling layer (e.g., Max Pooling) to reduce dimensionality and capture the most important features.
A fully connected layer to produce the final output (e.g., class probabilities).

class BasicSequenceCNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_size, output_dim, pad_idx):
        super().__init__()

        # 1. Embedding Layer
        # Maps token indices to dense vectors. padding_idx ensures padded tokens are zero vectors and don't contribute to gradients.
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)

        # 2. 1D Convolutional Layer
        # in_channels = embedding_dim (features per token)
        # out_channels = n_filters (number of filters/feature maps to produce)
        # kernel_size = filter_size (like n-gram size, e.g., 3 for trigrams)
        self.conv = nn.Conv1d(in_channels=embedding_dim,
                               out_channels=n_filters,
                               kernel_size=filter_size)

        # 3. Fully Connected Layer
        # Takes pooled features and maps to output_dim (e.g., number of classes)
        self.fc = nn.Linear(n_filters, output_dim)

    def forward(self, text_indices):
        # text_indices shape: (batch_size, seq_len)

        # Pass through embedding layer
        # embedded shape: (batch_size, seq_len, embedding_dim)
        embedded = self.embedding(text_indices)

        # Conv1d expects input shape: (batch_size, in_channels/embedding_dim, seq_len)
        # So, we permute dimensions 1 and 2.
        embedded = embedded.permute(0, 2, 1)
        # permuted embedded shape: (batch_size, embedding_dim, seq_len)

        # Pass through convolutional layer
        # conved shape: (batch_size, n_filters, seq_len - filter_size + 1)
        conved = self.conv(embedded)

        # Apply ReLU activation
        conved = F.relu(conved)

        # Perform Global Max Pooling over time (sequence length dimension)
        # This takes the maximum value from each filter map.
        # conved.shape[2] is the length of the convolved sequence.
        # pooled shape before squeeze: (batch_size, n_filters, 1)
        pooled = F.max_pool1d(conved, kernel_size=conved.shape[2])
        # pooled shape after squeeze: (batch_size, n_filters)
        pooled = pooled.squeeze(2)

        # Pass through fully connected layer
        # output shape: (batch_size, output_dim)
        output = self.fc(pooled)
        return output

Step 3: Data Preparation (Conceptual)

Using this model for text involves several preprocessing steps:

Tokenization: Splitting text into individual words or sub-words.
Vocabulary Building: Creating a mapping from tokens to unique integer IDs.
Numericalization: Converting token sequences into sequences of integer IDs.
Padding/Truncation: Ensuring all sequences in a batch have the same length by padding shorter sequences or truncating longer ones. The pad_idx in nn.Embedding is important here.
DataLoader: Using PyTorch’s Dataset and DataLoader to efficiently load and batch data.

For example, ‘hello world’ might become [10, 15] after tokenization and numericalization. If the required sequence length is 5 and pad token ID is 0, it becomes [10, 15, 0, 0, 0].

Step 4: Training the Model (Conceptual Overview)

A typical training loop involves:

# --- Hyperparameters & Instantiation (Example) ---
# VOCAB_SIZE = 5000       # Example: size of your vocabulary
# EMBEDDING_DIM = 100   # Dimension of word embeddings
# N_FILTERS = 64        # Number of filters in the conv layer
# FILTER_SIZE = 3       # Kernel size of the conv layer (e.g., for trigrams)
# OUTPUT_DIM = 2        # Example: for binary classification (e.g., sentiment positive/negative)
# PAD_IDX = 1           # Index of the padding token in your vocabulary
# LEARNING_RATE = 0.001

# device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
# model = BasicSequenceCNN(VOCAB_SIZE, EMBEDDING_DIM, N_FILTERS, FILTER_SIZE, OUTPUT_DIM, PAD_IDX).to(device)

# --- Loss Function and Optimizer ---
# criterion = nn.CrossEntropyLoss() # If OUTPUT_DIM > 1 and using raw scores
# # If OUTPUT_DIM = 1 for binary, use nn.BCEWithLogitsLoss()
# optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)

# --- Dummy Input for demonstration ---
# BATCH_SIZE = 32
# SEQ_LEN = 50
# dummy_input_indices = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LEN), dtype=torch.long).to(device)
# # Ensure PAD_IDX is used appropriately if sequences have variable lengths in real data
# # dummy_labels = torch.randint(0, OUTPUT_DIM, (BATCH_SIZE,), dtype=torch.long).to(device)

# --- Training Epoch (Conceptual) ---
# model.train() # Set model to training mode
# optimizer.zero_grad()
# predictions = model(dummy_input_indices)
# # loss = criterion(predictions, dummy_labels)
# # loss.backward()
# # optimizer.step()
# # print(f'Loss: {loss.item()}')

# --- Evaluation (Conceptual) ---
# model.eval() # Set model to evaluation mode
# with torch.no_grad():
#   pass # Perform evaluation on a validation set
print("BasicSequenceCNN defined. Implement data loading and training loop to use.")

Applications of 1D ConvNets for Sequences

1D ConvNets are versatile for various sequence-based tasks:

Text Classification: Sentiment analysis, topic categorization, spam detection.
Named Entity Recognition (NER): Identifying entities like names, locations, organizations in text (often combined with other layers).
Time Series Analysis: Forecasting, anomaly detection in sensor data or financial data.
DNA/Protein Sequence Analysis: Finding patterns in biological sequences.

While more complex architectures like LSTMs, GRUs, and Transformers exist, 1D ConvNets offer a strong and computationally efficient baseline for many sequence processing problems. You can explore variations like using multiple convolutional layers with different filter sizes and concatenating their outputs for richer feature extraction (similar to the Inception idea in vision).