Single Layer Perceptron

TensorFlow - Single Layer Perceptron

To understand the single-layer perceptron, it’s helpful to first grasp the basics of Artificial Neural Networks (ANN). ANNs are information processing systems inspired by the functionality of biological neural circuits. An ANN consists of many interconnected processing units, often called neurons. These networks typically have an input layer, one or more hidden layers, and an output layer.

In a basic feedforward network, information flows from the input layer, through any hidden layers, to the output layer. The hidden units process information from the input layer and pass it towards the output. The input and output units usually interact with the external environment or other parts of a larger system, while hidden layers perform intermediate computations.

The architecture of a neural network is defined by the pattern of connections between nodes, the total number of layers, the arrangement of nodes within layers, and the number of neurons per layer.

Neural network architectures can be broadly categorized based on their structure and data flow. Two fundamental types include:

Single Layer Perceptron
Multi-Layer Perceptron

Single Layer Perceptron

The single-layer perceptron (SLP) is one of the earliest and simplest types of neural network models. It consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. The computation involves calculating a weighted sum of the inputs. This sum is then passed through an activation function, which produces the output. An SLP has no hidden layers.

Let’s implement a single-layer perceptron for an image classification problem using TensorFlow. A classic example that can be modeled as an SLP is Logistic Regression, especially for multi-class classification using a softmax activation function.

The basic steps for training a model like logistic regression (which acts as a single-layer perceptron here) are as follows:

Initialize weights: The model’s weights (and biases) are typically initialized with small random values or zeros at the start of training.
Iterate and update: For each batch of training data, the model makes predictions. The error (difference between desired output and actual output) is calculated using a loss function (e.g., cross-entropy). This error is then used to adjust the weights through an optimization algorithm like gradient descent.
Repeat: This process is repeated for a set number of epochs (passes through the entire training dataset) or until the error on a validation set stops improving or reaches a desired threshold.

The following code demonstrates logistic regression for MNIST digit classification using TensorFlow 2.x and Keras, which serves as an example of a single-layer perceptron:

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np

# Load MNIST data using tf.keras.datasets
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# Preprocess the data
# Normalize pixel values to be between 0 and 1
x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0

# Flatten the images (28x28) into 784-dimensional vectors
x_train = x_train.reshape((-1, 784))
x_test = x_test.reshape((-1, 784))

# Convert labels to one-hot encoded vectors (e.g., 2 -> [0,0,1,0,0,0,0,0,0,0])
y_train = tf.keras.utils.to_categorical(y_train, num_classes=10)
y_test = tf.keras.utils.to_categorical(y_test, num_classes=10)

# Parameters
learning_rate = 0.01
training_epochs = 25
batch_size = 100
display_step = 1

# Define the model using Keras Sequential API (a single dense layer is an SLP)
model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='softmax', input_shape=(784,))
    # This Dense layer implements: output = softmax(dot(input, kernel) + bias)
    # 'kernel' are the weights (W), 'bias' is the bias (b)
])

# Compile the model
# Optimizer: Stochastic Gradient Descent (SGD)
# Loss function: Categorical Crossentropy for multi-class classification
optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate)
model.compile(optimizer=optimizer,
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=training_epochs,
                    verbose=0, # Set to 1 or 2 for more verbose output during training
                    validation_data=(x_test, y_test)) # Optionally track validation loss/accuracy

# Display training progress (loss)
print("Training phase finished.")
for epoch in range(training_epochs):
    if epoch % display_step == 0:
        # Access loss from history object (Keras model.fit returns a History object)
        # Note: history.history['loss'] is a list of losses per epoch
        print(f"Epoch: {epoch+1:04d} cost= {history.history['loss'][epoch]:.9f}")

# Plot training cost
plt.plot(np.arange(1, training_epochs + 1), history.history['loss'], 'o-', label='Logistic Regression Training cost')
plt.ylabel('Cost (Categorical Crossentropy)')
plt.xlabel('Epoch')
plt.title('Training Cost Over Epochs')
plt.legend()
plt.show()

# Evaluate the model on the test set
loss, accuracy = model.evaluate(x_test, y_test, verbose=0)
print(f"Model accuracy on test set: {accuracy:.4f}")

# For further exploration:
# - `tf.data` API for efficient input pipelines: https://www.tensorflow.org/guide/data
# - Keras overview: https://www.tensorflow.org/guide/keras/overview

Output

Running the code will first download the MNIST dataset if not already present. Then, it will train the logistic regression model. During training, it will print the cost (loss) at specified epoch intervals. After training, a plot will be displayed showing the training cost decreasing over epochs. Finally, it will print the model’s accuracy on the test dataset. The accuracy should be reasonably high (typically >90% for this simple model on MNIST), demonstrating that the single-layer perceptron has learned to classify the handwritten digits.

Logistic regression, as implemented here, is a linear model used for classification. It describes the relationship between input features and the probability of belonging to a particular class. Despite its simplicity, it’s a foundational algorithm in machine learning.