Keras - Time Series Prediction using LSTM RNN

Keras - Time Series Prediction using LSTM

Recurrent Neural Networks (RNNs), particularly Long Short-Term Memory (LSTM) networks, are well-suited for tasks involving sequential data, where the order of information matters. Time series prediction (forecasting future values based on past values) and natural language processing (understanding text) are common applications.

In this chapter, we’ll build a simple LSTM-based model using tf.keras to perform sentiment analysis on IMDB movie reviews. Sentiment analysis is a classification task: determining whether a piece of text expresses a positive or negative opinion. While text isn’t strictly a ‘time’ series, it’s a sequence where the order of words is crucial for meaning.

Why LSTM for Sequences? LSTMs have internal mechanisms (gates) that allow them to selectively remember or forget information over long sequences, overcoming limitations of simple RNNs in capturing long-range dependencies.

Conceptual Model for Sequence Classification:

Input Sequence (e.g., words in a review) -> Embedding Layer (map words to dense vectors) -> LSTM Layer (process sequence, capture context) -> Dense Output Layer (classify sentiment).

Model Specifications:

• Dataset: IMDB Movie Reviews (provided by tf.keras.datasets).
• Preprocessing: Limit vocabulary size (e.g., top 20,000 words), pad sequences to a fixed length (e.g., 80 words).
• Input Layer: Embedding layer to convert integer word indices into dense vectors (e.g., 128 dimensions).
• Recurrent Layer: LSTM layer (e.g., 128 units) to process the sequence of embeddings. Include dropout for regularization.
• Output Layer: Dense layer with 1 unit and ‘sigmoid’ activation for binary classification (positive/negative).
• Compilation: binary_crossentropy loss, Adam optimizer, accuracy metric.
• Training: Use appropriate batch size and number of epochs.

Step 1: Import Necessary Modules

import tensorflow as tf
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding, LSTM # Changed import
from tensorflow.keras.datasets import imdb
import numpy as np

Step 2: Define Parameters and Load Data

max_features = 20000  # Vocabulary size (consider only top N words)
maxlen = 80         # Max sequence length (cut texts after this number of words)
batch_size = 32
embedding_dims = 128
lstm_units = 128

print('Loading data...')
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
print(len(x_train), 'train sequences')
print(len(x_test), 'test sequences')

The imdb.load_data function returns sequences of word indices (integers).

Step 3: Preprocess the Data (Padding)

RNNs require input sequences of uniform length. We use pad_sequences to either truncate longer reviews or pad shorter reviews with a special value (usually 0) to reach the maxlen.

print('Pad sequences (samples x time)')
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)

The shape will now be (num_samples, maxlen).

Step 4: Build the LSTM Model

print('Build model...')
model = Sequential()

# Input Embedding Layer
model.add(Embedding(max_features, embedding_dims, input_length=maxlen))

# LSTM Layer
# Note: 'dropout' applies to input, 'recurrent_dropout' to recurrent state (can be computationally expensive on GPU)
model.add(LSTM(lstm_units, dropout=0.2, recurrent_dropout=0.2))

# Output Classification Layer
model.add(Dense(1, activation='sigmoid'))

model.summary() # Print model architecture

The Embedding layer maps each word index to a dense vector. The LSTM layer processes these vectors sequentially. The final Dense layer outputs a single probability.

Step 5: Compile the Model

Configure the learning process.

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

Step 6: Train the Model

Fit the model to the training data. We’ll use the test set as validation data to monitor performance during training.

print('Train...')
epochs = 5 # Reduced epochs for faster demonstration

history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    validation_data=(x_test, y_test))

Training output will show loss and accuracy for each epoch on both training and validation sets. Monitor val_accuracy and val_loss for signs of overfitting (where training performance improves but validation performance stagnates or degrades).

0.8528

# Example Output Snippet (actual values will vary):
# Epoch 1/5
# Epoch 2/5
# 782/782 [==============================] - 85s 109ms/step - loss: 0.2218 - accuracy: 0.9134 - val_loss: 0.3636 - val_accuracy: 0.8430
# ... (subsequent epochs)

Step 7: Evaluate the Model

Assess the final performance on the test set.

score, acc = model.evaluate(x_test, y_test,
                            batch_size=batch_size)

print('Test score:', score)
print('Test accuracy:', acc)

This gives the final evaluation metrics on unseen data.

Further Considerations

• Hyperparameter Tuning: Experiment with embedding_dims, lstm_units, dropout rates, maxlen, optimizer learning rate, etc.
• Bidirectional LSTMs: Process sequences in both forward and backward directions (tf.keras.layers.Bidirectional(LSTM(...)))), often improving performance by capturing context from both past and future.
• Stacked LSTMs: Use multiple LSTM layers (return_sequences=True on all but the last LSTM).
• Other Architectures: Explore GRUs (tf.keras.layers.GRU) or Transformer-based models for sequence tasks.