Recurrent Neural Networks

TensorFlow - Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are a class of neural networks well-suited for processing sequential data, such as time series, text, or speech. Unlike feedforward neural networks where inputs are assumed to be independent, RNNs have connections that form directed cycles, allowing them to maintain an internal state or ‘memory’ of past information. This memory enables them to capture temporal dependencies and patterns in sequential data.

The core idea of an RNN is that it performs the same computation for every element of a sequence, and the output at each step depends on previous computations. They are called ‘recurrent’ because they perform the same task for every element of a sequence, with the output being dependent on the previous computations.

Training an RNN typically involves these conceptual steps:

Step 1: Provide a sequence (or a batch of sequences) from the dataset as input.

Step 2: The network processes the sequence one element at a time, updating its internal state and producing an output at each step (or at the end of the sequence). Initial states and weights are often randomly initialized.

Step 3: A predicted result is computed based on the network’s output(s).

Step 4: The prediction is compared to the actual target value(s) using a loss function, generating an error signal.

Step 5: The error is propagated backward through time (an algorithm called Backpropagation Through Time, or BPTT) and through the network layers to adjust the model’s weights.

Step 6: Steps 1-5 are repeated for many epochs and batches until the model’s performance on a validation set is satisfactory.

Step 7: The trained model can then be used to make predictions on new, unseen sequences.

A simple RNN can be visualized as a loop. At each time step ‘t’, the RNN takes an input x(t) and its hidden state h(t-1) from the previous time step to compute the new hidden state h(t) and an output y(t). The same set of weights (U, W, V) are used across all time steps.

Recurrent Neural Network Implementation with TensorFlow Keras

We’ll demonstrate how to implement a simple RNN using tf.keras to classify MNIST handwritten digits. While CNNs are typically better for image data, this example illustrates how an RNN can process an image by treating its rows (or columns) as a sequence.

Step 1: Import necessary modules and load data.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Preprocess data: Normalize pixel values
x_train = x_train.astype("float32") / 255.0
x_test = x_test.astype("float32") / 255.0

# Convert labels to one-hot encoded vectors
num_classes = 10
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

print(f"x_train shape: {x_train.shape}") # (60000, 28, 28)
print(f"y_train shape: {y_train.shape}") # (60000, 10)

For an RNN, we can treat each image (28x28 pixels) as a sequence of 28 time steps, where each time step has 28 features (the pixel values of a row).

Step 2: Define RNN model parameters and build the model.

# RNN parameters
# Input image is 28x28. We treat it as 28 timesteps, each with 28 features.
n_steps = 28  # Number of timesteps (image height)
n_input = 28  # Number of features per timestep (image width)
n_hidden = 128 # Number of units in the LSTM layer

model = keras.Sequential([
    # LSTM layer: Processes the sequence of rows
    # input_shape will be (timesteps, features_per_timestep)
    layers.LSTM(n_hidden, input_shape=(n_steps, n_input)),
    # You could add Dropout here for regularization if needed
    # layers.Dropout(0.5),
    # Dense output layer for classification
    layers.Dense(num_classes, activation='softmax')
])

model.summary()

We use an LSTM (Long Short-Term Memory) layer, which is a type of RNN cell particularly good at capturing long-range dependencies and mitigating the vanishing gradient problem that can affect simple RNNs. The input_shape for the LSTM layer is (n_steps, n_input).

Step 3: Compile the model.

learning_rate = 0.001
optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)

model.compile(optimizer=optimizer,
              loss='categorical_crossentropy',
              metrics=['accuracy'])

We compile the model with the Adam optimizer, categorical cross-entropy loss (suitable for multi-class classification), and track accuracy.

Step 4: Train the model.

batch_size = 128
training_epochs = 5 # Reduced for quick demonstration

print("\nTraining the model...")
history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=training_epochs,
                    validation_data=(x_test, y_test),
                    verbose=1)

print("\nOptimization Finished!")

# Evaluate the model on the test set
print("\nEvaluating model on test data...")
loss, acc = model.evaluate(x_test, y_test, verbose=0)
print(f"Test Loss: {loss:.6f}")
print(f"Test Accuracy: {acc:.5f}")

# Example prediction
if len(x_test) > 0:
    sample_idx = 0
    sample_image = x_test[sample_idx]
    prediction = model.predict(np.expand_dims(sample_image, axis=0))
    predicted_class = np.argmax(prediction[0])
    actual_class = np.argmax(y_test[sample_idx])
    print(f"Sample {sample_idx} - Predicted class: {predicted_class}, Actual class: {actual_class}")

The training progress will be displayed, followed by the final test accuracy. While LSTMs can process images this way, CNNs are generally more effective for image tasks due to their ability to capture local spatial patterns efficiently.

Expected output (accuracy may vary, especially with few epochs):

x_train shape: (60000, 28, 28)
y_train shape: (60000, 10)
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 lstm (LSTM)                 (None, 128)               80384

 dense (Dense)               (None, 10)                1290

=================================================================
Total params: 81,674
Trainable params: 81,674
Non-trainable params: 0
_________________________________________________________________

Training the model...
Epoch 1/5
469/469 [==============================] - 7s 11ms/step - loss: 0.7749 - accuracy: 0.7544 - val_loss: 0.2873 - val_accuracy: 0.9136
Epoch 2/5
469/469 [==============================] - 5s 10ms/step - loss: 0.2228 - accuracy: 0.9335 - val_loss: 0.1629 - val_accuracy: 0.9509
Epoch 3/5
469/469 [==============================] - 5s 10ms/step - loss: 0.1466 - accuracy: 0.9564 - val_loss: 0.1250 - val_accuracy: 0.9610
Epoch 4/5
469/469 [==============================] - 5s 10ms/step - loss: 0.1125 - accuracy: 0.9660 - val_loss: 0.1048 - val_accuracy: 0.9667
Epoch 5/5
469/469 [==============================] - 5s 10ms/step - loss: 0.0914 - accuracy: 0.9726 - val_loss: 0.0938 - val_accuracy: 0.9709

Optimization Finished!

Evaluating model on test data...
Test Loss: 0.093812
Test Accuracy: 0.97090
Sample 0 - Predicted class: 7, Actual class: 7