Scikit Learn - Support Vector Machines

Scikit-Learn - Support Vector Machines (SVMs)

This chapter delves into Support Vector Machines (SVMs), a set of powerful supervised learning models used for classification, regression, and outlier detection.

Introduction to SVMs

Support Vector Machines are versatile and effective machine learning algorithms. They are particularly well-suited for classification of complex but small- or medium-sized datasets. SVMs are memory efficient because they use a subset of training points (the support vectors) in the decision function.

The core idea of SVM for classification is to find a hyperplane in an N-dimensional space (where N is the number of features) that distinctly classifies the data points. For a linearly separable dataset, SVM aims to find the hyperplane that maximizes the margin, i.e., the distance between the hyperplane and the nearest data points from any class. This is known as the maximum marginal hyperplane (MMH).

Key concepts in SVM:

• Support Vectors: These are the data points closest to the hyperplane. They are critical in defining the position and orientation of the hyperplane. If these points were moved, the hyperplane would also move.
• Hyperplane: In a 2D space, this is a line; in 3D, a plane; and in higher dimensions, a hyperplane. It’s the decision boundary that separates classes.
• Margin: The distance between the hyperplane and the closest support vectors from either class. SVMs try to maximize this margin.

Imagine a 2D scatter plot with two classes of points. The SVM would try to find a line that separates these two classes with the largest possible ‘street’ or ‘gap’ between them. The edges of this street are defined by the support vectors.

Scikit-Learn’s SVM implementations support both dense (NumPy arrays) and sparse (SciPy sparse matrices) input data.

SVM for Classification

Scikit-Learn provides three main classes for SVM classification: SVC, NuSVC, and LinearSVC.

SVC (sklearn.svm.SVC)

SVC (C-Support Vector Classification) is based on the libsvm library. It can perform binary and multi-class classification. For multi-class problems, it uses a one-vs-one scheme by default (decision_function_shape='ovo'), though decision_function_shape='ovr' (one-vs-rest) is also available and often preferred for consistency with other classifiers.

Key Parameters for SVC

Parameter	Description
C	float, default=1.0. Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive. Penalty parameter C of the error term.
kernel	{‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’}, default=‘rbf’. Specifies the kernel type to be used in the algorithm. ‘rbf’ (Radial Basis Function) is a common choice.
degree	int, default=3. Degree of the polynomial kernel function (‘poly’). Ignored by all other kernels.
gamma	{‘scale’, ‘auto’} or float, default=‘scale’. Kernel coefficient for ‘rbf’, ‘poly’ and ‘sigmoid’. If gamma='scale' (default), it uses 1 / (n_features * X.var()). If 'auto', it uses 1 / n_features.
coef0	float, default=0.0. Independent term in kernel function. Only significant in ‘poly’ and ‘sigmoid’.
shrinking	bool, default=True. Whether to use the shrinking heuristic, which can speed up training.
probability	bool, default=False. Whether to enable probability estimates. Must be enabled before calling fit, and it will slow down training.
tol	float, default=1e-3. Tolerance for stopping criterion.
cache_size	float, default=200.0. Specify the size of the kernel cache (in MB).
class_weight	dict or ‘balanced’, default=None. Set the parameter C of class i to class_weight[i]*C. If ‘balanced’, weights are inversely proportional to class frequencies.
verbose	bool, default=False. Enable verbose output.
max_iter	int, default=-1. Hard limit on iterations within solver, or -1 for no limit.
decision_function_shape	{‘ovo’, ‘ovr’}, default=‘ovr’. Whether to return a one-vs-rest (‘ovr’) decision function of shape (n_samples, n_classes) as all other classifiers, or the original one-vs-one (‘ovo’) decision function of shape (n_samples, n_classes * (n_classes - 1) / 2).
break_ties	bool, default=False. If True, predict will break ties according to the confidence values of decision_function; otherwise the first class among the tied classes is returned.
random_state	int, RandomState instance or None, default=None. Controls the pseudo-randomness for shuffling data for probability estimates when probability=True.

Key Attributes for SVC

Attribute	Description
support_	array-like of shape (n_SV,). Indices of support vectors.
support_vectors_	array-like of shape (n_SV, n_features). Support vectors.
n_support_	array-like of shape (n_classes,), dtype=int32. Number of support vectors for each class.
dual_coef_	array, shape (n_classes-1, n_SV). Coefficients of the support vectors in the decision function (for binary and multi-class ovo).
coef_	array, shape (n_classes * (n_classes-1) / 2, n_features) or (1, n_features) for binary. Weights assigned to the features (coefficients in the primal problem). Only available when kernel='linear'.
intercept_	array, shape (n_classes * (n_classes-1) / 2,) or (1,) for binary. Constants in decision function.
classes_	array of shape (n_classes,). The unique class labels.

Implementation Example (SVC)

import numpy as np
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# Sample data (linearly separable for simplicity with linear kernel)
X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1], [-1, 2], [2, -2]])
y = np.array([1, 1, 2, 2, 1, 2]) # Two classes

# It's good practice to scale data for SVMs
svc_pipeline = make_pipeline(StandardScaler(),
                            SVC(kernel='linear', C=1.0, random_state=42))
svc_pipeline.fit(X, y)

# Making a prediction
new_point = np.array([[-0.8, -0.8]])
prediction = svc_pipeline.predict(new_point)
print(f"Prediction for {new_point}: {prediction}")

# Accessing attributes from the SVC model within the pipeline
svc_model = svc_pipeline.named_steps['svc']
print(f"Support vectors:\n{svc_model.support_vectors_}")
if svc_model.kernel == 'linear':
    print(f"Coefficients (w): {svc_model.coef_}")
print(f"Intercept (b): {svc_model.intercept_}")

Output Example (SVC):

Prediction for [[-0.8 -0.8]]: [1]
Support vectors:
[[-1. -1.]
 [-1.  2.]
 [ 1.  1.]]
Coefficients (w): [[ 0.72239822 -0.18059955]]
Intercept (b): [0.03611991]

NuSVC (sklearn.svm.NuSVC)

NuSVC (Nu-Support Vector Classification) is similar to SVC but uses a parameter nu to control the number of support vectors and training errors. nu is an upper bound on the fraction of training errors and a lower bound of the fraction of support vectors. It should be in the interval (0, 1].

Key Parameter nu for NuSVC

nu: float, default=0.5. Replaces the C parameter of SVC for controlling model complexity.

Implementation Example (NuSVC)

from sklearn.svm import NuSVC
# (Using X, y, StandardScaler, make_pipeline from SVC example)

nusvc_pipeline = make_pipeline(StandardScaler(),
                               NuSVC(kernel='rbf', nu=0.5, gamma='scale', random_state=42))
nusvc_pipeline.fit(X, y)

prediction_nusvc = nusvc_pipeline.predict(new_point) # Using new_point from SVC example
print(f"NuSVC Prediction for {new_point}: {prediction_nusvc}")

nusvc_model = nusvc_pipeline.named_steps['nusvc']
print(f"Number of support vectors for each class in NuSVC: {nusvc_model.n_support_}")

Output Example (NuSVC):

NuSVC Prediction for [[-0.8 -0.8]]: [1]
Number of support vectors for each class in NuSVC: [3 3]

LinearSVC (sklearn.svm.LinearSVC)

LinearSVC (Linear Support Vector Classification) is an implementation of SVM specifically for linear kernels. It is based on liblinear rather than libsvm (which SVC and NuSVC use). LinearSVC is generally faster for large datasets and offers more flexibility in the choice of penalties (L1/L2) and loss functions (‘hinge’, ‘squared_hinge’). It does not have a kernel parameter (it’s always linear).

Key Parameters for LinearSVC

Differences from SVC:

• penalty: {‘l1’, ‘l2’}, default=‘l2’. Specifies the norm used in the penalization.
• loss: {‘hinge’, ‘squared_hinge’}, default=‘squared_hinge’. Specifies the loss function. ‘hinge’ is the standard SVM loss.
• dual: bool or ‘auto’, default=‘auto’. Select algorithm to either solve the dual or primal optimization problem. Prefer dual=False when n_samples > n_features. ‘auto’ makes this choice.
• No kernel, gamma, degree, coef0, shrinking, probability parameters.
• Lacks support_, support_vectors_, n_support_, dual_coef_ attributes.

Implementation Example (LinearSVC)

from sklearn.svm import LinearSVC
from sklearn.datasets import make_classification
# (StandardScaler, make_pipeline already imported)

# Generate a larger dataset for LinearSVC demonstration
X_lin, y_lin = make_classification(n_samples=200, n_features=4, random_state=0)

linearsvc_pipeline = make_pipeline(StandardScaler(),
                                   LinearSVC(penalty='l2', loss='squared_hinge', dual='auto',
                                             random_state=0, tol=1e-4, C=1.0))
linearsvc_pipeline.fit(X_lin, y_lin)

# Making a prediction on a sample point
sample_lin_point = X_lin[0].reshape(1, -1)
prediction_linearsvc = linearsvc_pipeline.predict(sample_lin_point)
print(f"LinearSVC Prediction for {sample_lin_point[0,:2]}...: {prediction_linearsvc}")

linearsvc_model = linearsvc_pipeline.named_steps['linearsvc']
print(f"LinearSVC Coefficients: {linearsvc_model.coef_}")
print(f"LinearSVC Intercept: {linearsvc_model.intercept_}")

Output Example (LinearSVC):

LinearSVC Prediction for [0.16762098 0.20222924]...: [1]
LinearSVC Coefficients: [[ 0.15689569 -0.24839626  0.75454966  0.41789389]]
LinearSVC Intercept: [0.16693748]

SVM for Regression (SVR)

The concepts of SVMs can be extended to solve regression problems. This is known as Support Vector Regression (SVR). In SVR, the goal is to find a function (hyperplane) that deviates from y by a value no greater than a margin epsilon for as many training points as possible, while also being as flat as possible. Points within the epsilon-tube around the prediction do not contribute to the loss.

Scikit-Learn provides SVR, NuSVR, and LinearSVR.

SVR (sklearn.svm.SVR)

Epsilon-Support Vector Regression, based on libsvm. Key parameters are C (regularization) and epsilon (defines the margin of tolerance where no penalty is given to errors).

Key Parameter epsilon for SVR

epsilon: float, default=0.1. Specifies the epsilon-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual value.

Implementation Example (SVR)

from sklearn.svm import SVR
from sklearn.datasets import make_regression
# (StandardScaler, make_pipeline, np already imported)

X_reg, y_reg = make_regression(n_samples=100, n_features=1, random_state=42, noise=5)

svr_pipeline = make_pipeline(StandardScaler(),
                           SVR(kernel='rbf', C=100, gamma=0.1, epsilon=0.1))
svr_pipeline.fit(X_reg, y_reg)

sample_reg_point = np.array([[1.5]]) # Example new data point
prediction_svr = svr_pipeline.predict(sample_reg_point)
print(f"SVR Prediction for {sample_reg_point}: {prediction_svr}")
# SVR attributes like support_vectors_ are also available

Output Example (SVR):

SVR Prediction for [[1.5]]: [66.9114178]

NuSVR (sklearn.svm.NuSVR)

Nu-Support Vector Regression. Similar to SVR, but uses nu to control the number of support vectors, instead of epsilon to control the insensitivity region.

Implementation Example (NuSVR)

from sklearn.svm import NuSVR
# (Using X_reg, y_reg, StandardScaler, make_pipeline, np from SVR example)

nusvr_pipeline = make_pipeline(StandardScaler(),
                               NuSVR(kernel='rbf', C=100, nu=0.1, gamma='scale'))
nusvr_pipeline.fit(X_reg, y_reg)

prediction_nusvr = nusvr_pipeline.predict(sample_reg_point)
print(f"NuSVR Prediction for {sample_reg_point}: {prediction_nusvr}")

Output Example (NuSVR):

NuSVR Prediction for [[1.5]]: [66.6761768]

LinearSVR (sklearn.svm.LinearSVR)

Linear Support Vector Regression, based on liblinear. Faster for linear SVR on large datasets. Key parameter is epsilon. loss can be 'epsilon_insensitive' or 'squared_epsilon_insensitive'.

Implementation Example (LinearSVR)

from sklearn.svm import LinearSVR
# (Using X_reg, y_reg, StandardScaler, make_pipeline, np from SVR example)

linearsvr_pipeline = make_pipeline(StandardScaler(),
                                   LinearSVR(epsilon=0.0, C=1.0, loss='squared_epsilon_insensitive',
                                             dual='auto', random_state=0, tol=1e-4))
linearsvr_pipeline.fit(X_reg, y_reg)

prediction_linearsvr = linearsvr_pipeline.predict(sample_reg_point)
print(f"LinearSVR Prediction for {sample_reg_point}: {prediction_linearsvr}")

linearsvr_model = linearsvr_pipeline.named_steps['linearsvr']
print(f"LinearSVR Coefficients: {linearsvr_model.coef_}")
print(f"LinearSVR Intercept: {linearsvr_model.intercept_}")

Output Example (LinearSVR):

LinearSVR Prediction for [[1.5]]: [67.53298058]
LinearSVR Coefficients: [45.66113378]
LinearSVR Intercept: [-0.95663479]

SVMs are powerful models. Choosing the right kernel and parameters (like C, gamma, nu, epsilon) is crucial and often requires experimentation (e.g., using GridSearchCV). Feature scaling is highly recommended. For more information, visit the Scikit-Learn SVM documentation.