Supervised Learning: Regression

AI with Python – Supervised Learning: Regression

Regression analysis is a cornerstone of statistical modeling and machine learning. It’s often one of the first techniques learned. Regression models predict a continuous output value (dependent variable) based on one or more input features (independent variables). The core idea is to learn the relationship between these variables from data.

Applications of regression are widespread, including predicting stock prices, house values, temperature, sales figures, or any quantity that can be represented by a real number.

Building Regressors in Python with Scikit-learn

Python, with its powerful libraries like Scikit-learn, NumPy, and Matplotlib, provides an excellent environment for building and evaluating regression models.

Linear Regressor (Single Variable)

Linear regression assumes a linear relationship between the input feature(s) and the output. For a single variable, this relationship is y = mx + c, where y is the output, x is the input, m is the slope, and c is the intercept.

Let’s start by importing necessary packages:

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics
import matplotlib.pyplot as plt

Next, we’ll create some sample data. In real-world scenarios, this data would come from files or databases.

# Sample data: X (feature), y (target)
# Let's create data that roughly follows y = 2x + noise
np.random.seed(0) # for reproducibility
X = np.sort(5 * np.random.rand(80, 1), axis=0)
y = (2 * X).ravel() + np.random.randn(80) * 2

Split the data into training and testing sets to evaluate model performance on unseen data:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

Create and train the linear regression model:

linear_regressor = LinearRegression()
linear_regressor.fit(X_train, y_train)

Make predictions on the test data:

y_pred = linear_regressor.predict(X_test)

Visualize the results:

plt.figure(figsize=(8, 6))
plt.scatter(X_test, y_test, color='red', label='Actual values')
plt.plot(X_test, y_pred, color='black', linewidth=2, label='Predicted regression line')
plt.xlabel('X (Feature)')
plt.ylabel('y (Target)')
plt.title('Linear Regression - Single Variable')
plt.legend()
plt.grid(True)
plt.show()

[Image description: A scatter plot showing data points in red and a black line representing the linear regression model’s fit. The x-axis is ‘X (Feature)’ and the y-axis is ‘y (Target)’. The line attempts to capture the general trend of the scattered points.]

Evaluate the model’s performance:

print('Linear Regressor Performance:')
print('Mean Absolute Error (MAE):', round(metrics.mean_absolute_error(y_test, y_pred), 2))
print('Mean Squared Error (MSE):', round(metrics.mean_squared_error(y_test, y_pred), 2))
print('Root Mean Squared Error (RMSE):', round(np.sqrt(metrics.mean_squared_error(y_test, y_pred)), 2))
print('R-squared (R2 Score):', round(metrics.r2_score(y_test, y_pred), 2))

Performance metrics explained:

• Mean Absolute Error (MAE): Average absolute difference between actual and predicted values.
• Mean Squared Error (MSE): Average of the squared differences. Penalizes larger errors more.
• Root Mean Squared Error (RMSE): Square root of MSE, in the same units as the target variable.
• R-squared (R2 Score): Proportion of the variance in the dependent variable predictable from the independent variable(s). Ranges from 0 to 1 (or lower for poor models); higher is better.

The output values for these metrics will depend on the random data generation but will provide an assessment of the model’s fit.

Multivariable Regressor (Polynomial Regression Example)

When the relationship between features and target is not linear, polynomial regression can be used. It transforms input features into polynomial features (e.g., x, x^2, x^3) and then applies linear regression.

First, import PolynomialFeatures:

from sklearn.preprocessing import PolynomialFeatures

Let’s generate some non-linear sample data, e.g., y = 0.5x^2 - 2x + 1 + noise:

np.random.seed(0)
X_poly_data = np.sort(6 * np.random.rand(100, 1) - 3, axis=0)
y_poly_data = (0.5 * X_poly_data**2 - 2 * X_poly_data + 1).ravel() + np.random.randn(100) * 1.5

# Reshape X for sklearn
X_poly_train, X_poly_test, y_poly_train, y_poly_test = train_test_split(X_poly_data, y_poly_data, test_size=0.3, random_state=42)

Create polynomial features (e.g., degree 2):

poly_features = PolynomialFeatures(degree=2, include_bias=False)
X_train_transformed = poly_features.fit_transform(X_poly_train)
X_test_transformed = poly_features.transform(X_poly_test)

Train a linear regression model on these transformed features:

poly_regressor = LinearRegression()
poly_regressor.fit(X_train_transformed, y_poly_train)

Make predictions:

y_poly_pred = poly_regressor.predict(X_test_transformed)

Visualize the polynomial regression fit:

# For plotting the curve, we need to sort X_poly_test
sort_axis = np.argsort(X_poly_test[:,0])
X_poly_test_sorted = X_poly_test[sort_axis]
y_poly_pred_sorted = y_poly_pred[sort_axis]

plt.figure(figsize=(8, 6))
plt.scatter(X_poly_test, y_poly_test, color='blue', label='Actual values')
plt.plot(X_poly_test_sorted, y_poly_pred_sorted, color='red', linewidth=2, label='Polynomial regression fit (degree 2)')
plt.xlabel('X (Feature)')
plt.ylabel('y (Target)')
plt.title('Polynomial Regression')
plt.legend()
plt.grid(True)
plt.show()

[Image description: A scatter plot with blue data points exhibiting a curve. A red line, representing the polynomial regression fit, follows the curve of the data points more closely than a straight line would. The x-axis is ‘X (Feature)’ and the y-axis is ‘y (Target)’.]

Evaluate the polynomial regression model:

print('\nPolynomial Regressor Performance (degree 2):')
print('Mean Absolute Error (MAE):', round(metrics.mean_absolute_error(y_poly_test, y_poly_pred), 2))
print('Mean Squared Error (MSE):', round(metrics.mean_squared_error(y_poly_test, y_poly_pred), 2))
print('R-squared (R2 Score):', round(metrics.r2_score(y_poly_test, y_poly_pred), 2))

Choosing the right degree for polynomial regression is crucial. Too low a degree might underfit, while too high a degree can lead to overfitting. Cross-validation is often used to select the optimal degree. For datasets with multiple input features (multivariable regression), linear regression extends naturally: y = b0 + b1*x1 + b2*x2 + ... + bn*xn. Scikit-learn’s LinearRegression handles this directly if X has multiple columns.