Stochastic Gradient Descent

Scikit-Learn - Stochastic Gradient Descent (SGD)

This chapter covers Stochastic Gradient Descent (SGD), a powerful and widely used optimization algorithm in Scikit-Learn for fitting linear models.

Stochastic Gradient Descent (SGD) is an efficient optimization algorithm used to find the parameters (coefficients) of a function that minimize a cost function. It is particularly well-suited for large-scale datasets because it processes one training instance (or a small batch) at a time to update the model’s coefficients, rather than processing the entire dataset for each update. This makes it scalable and suitable for online learning. SGD is used for discriminative learning of linear classifiers and regressors under various (often convex) loss functions, such as Support Vector Machines (SVM) and Logistic Regression.

SGDClassifier

The SGDClassifier in Scikit-Learn implements a plain SGD learning routine. It supports various loss functions and penalties, enabling it to fit different linear classification models.

Key Parameters

Important parameters for sklearn.linear_model.SGDClassifier:

Parameter	Description
loss	str, default=‘hinge’. The loss function to be used. Common options: ‘hinge’ (linear SVM), ‘log_loss’ (logistic regression), ‘modified_huber’ (smooth loss, tolerant to outliers, produces probability estimates), ‘squared_hinge’ (quadratically penalized hinge), ‘perceptron’. Other specialized losses exist.
penalty	{‘l2’, ‘l1’, ‘elasticnet’, None}, default=‘l2’. The regularization (penalty) term. ‘l2’ is standard Ridge regularization, ‘l1’ is Lasso (can lead to sparse solutions), ‘elasticnet’ combines L1 and L2. None means no regularization.
alpha	float, default=0.0001. Constant that multiplies the regularization term. Higher values mean stronger regularization.
l1_ratio	float, default=0.15. The Elastic Net mixing parameter, with 0 <= l1_ratio <= 1. l1_ratio=0 corresponds to L2 penalty, l1_ratio=1 to L1. Only used if penalty='elasticnet'.
fit_intercept	bool, default=True. Whether the intercept should be estimated or not. If False, the data is assumed to be already centered.
max_iter	int, default=1000. The maximum number of passes over the training data (epochs).
tol	float or None, default=1e-3. The stopping criterion. If not None, training will stop when (loss > best_loss - tol) for n_iter_no_change consecutive epochs.
shuffle	bool, default=True. Whether or not the training data should be shuffled after each epoch.
epsilon	float, default=0.1. Epsilon in the epsilon-insensitive loss functions (e.g., ‘huber’, ‘epsilon_insensitive’ - though these are more for regressors). For ‘modified_huber’, it’s the point where the loss becomes linear.
n_jobs	int, default=None. Number of CPUs to use for One-Vs-Rest (‘ovr’) computation. None means 1, -1 means all CPUs. This is only used for multi-class classification.
random_state	int, RandomState instance or None, default=None. Used for shuffling the data.
learning_rate	str, default=‘optimal’. The learning rate schedule: ‘constant’: eta = eta0; ‘optimal’: eta = 1.0 / (alpha * (t + t0)) (t0 is heuristic); ‘invscaling’: eta = eta0 / pow(t, power_t); ‘adaptive’: eta = eta0, as long as training loss keeps decreasing. eta0 is reduced by a factor of 5 each time n_iter_no_change consecutive epochs fail to decrease training loss or fail to increase validation score if early_stopping=True.
eta0	double, default=0.0. The initial learning rate for ‘constant’, ‘invscaling’, ‘adaptive’. Default is 0.01 for ‘adaptive’, 0.0 otherwise.
power_t	double, default=0.5. The exponent for inverse scaling learning rate ('invscaling').
early_stopping	bool, default=False. Whether to use early stopping to terminate training when validation score is not improving. If True, it automatically sets aside a fraction of training data as validation.
validation_fraction	float, default=0.1. The proportion of training data to set aside as validation set for early stopping. Only used if early_stopping is True.
n_iter_no_change	int, default=5. Number of iterations with no improvement to wait before stopping fitting. Converges when loss or score is not improving by at least tol for n_iter_no_change consecutive iterations.
class_weight	dict, {class_label: weight}, ‘balanced’ or None, default=None. Weights associated with classes. If ‘balanced’, weights are inversely proportional to class frequencies.
warm_start	bool, default=False. When set to True, reuse the solution of the previous call to fit as initialization, otherwise, just erase the previous solution.
average	bool or int, default=False. When set to True, computes the averaged SGD weights and stores the result in the coef_ and intercept_ attributes. If set to an int > 1, averaging will begin once the total number of samples seen reaches average.

Key Attributes

After fitting SGDClassifier:

Attribute	Description
coef_	array, shape (1, n_features) if n_classes==2 else (n_classes, n_features). Weights assigned to the features.
intercept_	array, shape (1,) if n_classes==2 else (n_classes,). Independent term (bias) in decision function.
n_iter_	int. The actual number of iterations before reaching the stopping criterion.
loss_function_	The concrete LossFunction object used by the estimator.

Implementation Example:

SGDClassifier needs to be fitted with two arrays: X (training samples, shape [n_samples, n_features]) and Y (target values/class labels, shape [n_samples]).

import numpy as np
from sklearn.linear_model import SGDClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# Sample data
X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1], [-1, 2], [2, -2]])
Y = np.array([1, 1, 2, 2, 1, 2]) # Two classes: 1 and 2

# It's highly recommended to scale data before using SGD
# Create a pipeline to scale data and then fit SGDClassifier
sgd_clf = make_pipeline(StandardScaler(),
                        SGDClassifier(loss='hinge', penalty='l2',
                                      max_iter=1000, tol=1e-3, random_state=42))
sgd_clf.fit(X, Y)
print("Pipeline with SGDClassifier fitted.")
# To access the classifier itself from the pipeline:
# final_estimator = sgd_clf.named_steps['sgdclassifier']
# print(f"Coefficients: {final_estimator.coef_}")
# print(f"Intercept: {final_estimator.intercept_}")

Output of print statement:

Pipeline with SGDClassifier fitted.

Predicting new values:

new_data_point = np.array([[2., 2.]])
prediction = sgd_clf.predict(new_data_point)
print(f"Prediction for {new_data_point}: {prediction}")

Output:

Prediction for [[2. 2.]]: [2]

Accessing coefficients and intercept (from the classifier within the pipeline):

final_estimator = sgd_clf.named_steps['sgdclassifier']
print(f"Coefficients (weights): {final_estimator.coef_}")
print(f"Intercept: {final_estimator.intercept_}")

Output (example values, will vary based on random_state and exact scaling):

Coefficients (weights): [[ 8.60414862 -0.97295213]]
Intercept: [1.72006884]

Signed distance to the hyperplane using decision_function:

decision_value = sgd_clf.decision_function(new_data_point)
print(f"Decision function value for {new_data_point}: {decision_value}")

Output (example value):

Decision function value for [[2. 2.]]: [12.8626418]

SGDRegressor

SGDRegressor implements SGD for linear regression models, supporting various loss functions and penalties.

Key Parameters

Parameters are largely similar to SGDClassifier. Key differences:

• loss: str, default=‘squared_error’. Options include: ‘squared_error’ (ordinary least squares), ‘huber’ (less sensitive to outliers than squared_error), ‘epsilon_insensitive’ (ignores errors within a certain epsilon tube, used in SVR), ‘squared_epsilon_insensitive’.
• epsilon: float, default=0.1. Epsilon for ‘huber’, ‘epsilon_insensitive’, and ‘squared_epsilon_insensitive’ loss functions. Defines the threshold at which errors start to be penalized.
• power_t: double, default=0.25 (different from SGDClassifier’s 0.5). Exponent for ‘invscaling’ learning rate.
• No class_weight parameter as it’s for regression.

Key Attributes

Similar to SGDClassifier. If average=True or an int, coef_ and intercept_ will store the averaged values. t_ is an additional attribute:

• t_: int. Number of weight updates performed during training. This is n_iter_ * n_samples_ if all samples are used in each iteration.

Implementation Example:

import numpy as np
from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.datasets import make_regression # For generating regression data

# Generate sample regression data
X_reg, y_reg = make_regression(n_samples=100, n_features=5, random_state=0, noise=0.1)

# Create a pipeline with StandardScaler and SGDRegressor
sgd_reg = make_pipeline(StandardScaler(),
                        SGDRegressor(loss='squared_error', penalty='elasticnet', l1_ratio=0.2,
                                     max_iter=1000, tol=1e-3, random_state=0, average=False))
sgd_reg.fit(X_reg, y_reg)
print("Pipeline with SGDRegressor fitted.")

# Access the regressor from the pipeline
final_regressor = sgd_reg.named_steps['sgdregressor']
print(f"Coefficients: {final_regressor.coef_}")
print(f"Intercept: {final_regressor.intercept_}")
print(f"Number of iterations: {final_regressor.n_iter_}")
# print(f"Total weight updates (t_): {final_regressor.t_}") # t_ might not be directly exposed in all versions or under all conditions in pipeline

Output (example values):

Pipeline with SGDRegressor fitted.
Coefficients: [32.2253343  16.72874275 41.86994217 60.73392231 28.50582354]
Intercept: [0.02206273]
Number of iterations: 11

Predicting with the fitted regressor:

new_reg_data_point = X_reg[0].reshape(1, -1) # Take the first sample for prediction
reg_prediction = sgd_reg.predict(new_reg_data_point)
print(f"Prediction for first sample: {reg_prediction}, Actual value: {y_reg[0]}")

Output (example value):

Prediction for first sample: [-7.8509133], Actual value: -7.915208493840426

Advantages and Disadvantages of SGD

Pros:

• Efficiency: SGD is computationally efficient, especially for large datasets.
• Scalability: Handles large-scale problems (many samples and/or features) effectively.
• Online learning: Can be updated as new data arrives using partial_fit.

Cons:

• Sensitivity to feature scaling: SGD is very sensitive to feature scaling. It’s crucial to scale features (e.g., using StandardScaler) before training.
• Hyperparameter tuning: Requires careful tuning of several hyperparameters, such as the learning rate and regularization parameter.
• Convergence: The path to the minimum can be noisy due to the stochastic nature of updates.

For more details and advanced usage, refer to the Scikit-Learn SGD documentation.