Machine Learning - Supervised

Machine Learning - Supervised Learning

Supervised learning is a fundamental paradigm in Machine Learning where the model learns from labeled data. This means that each training example is paired with an output label or target value. This chapter explores common algorithms used in supervised learning.

Key Algorithms for Supervised Learning

Several algorithms are available for supervised learning tasks (classification and regression). Some of the most widely used include:

• k-Nearest Neighbours (kNN)
• Decision Trees
• Random Forests
• Gradient Boosting Machines (e.g., XGBoost, LightGBM)
• Naive Bayes
• Logistic Regression
• Support Vector Machines (SVMs)

Let’s discuss each of these in more detail.

k-Nearest Neighbours (kNN)

The k-Nearest Neighbours (kNN) algorithm is a non-parametric method used for both classification and regression. For classification, imagine a 2D plot with data points belonging to different classes, say red, blue, and green. Each class might form a loose cluster of points. When a new, unlabeled data point appears on this plot, kNN classifies it by identifying its ‘k’ closest neighbors (based on a distance metric like Euclidean distance). The new point is then assigned to the class that is most common among its k neighbors. For example, if k=5 and three of the five nearest neighbors are blue, the new point would be classified as blue. The decision boundaries formed by kNN can be complex and non-linear.

For regression, kNN predicts the value of the new point by averaging the values of its k nearest neighbors. kNN is simple to understand but can be computationally expensive for large datasets as it requires calculating distances to all training points for each new prediction.

Decision Trees

A Decision Tree is a model that uses a tree-like graph of decisions and their possible consequences. It’s like a flowchart where each internal node represents a test on an attribute (e.g., ‘Is email marked as urgent?’), each branch represents an outcome of the test (e.g., ‘Yes’ or ‘No’), and each leaf node represents a class label (e.g., ‘Read immediately’) or a continuous value (for regression trees). To classify a new instance, you start at the root and traverse down the tree according to the outcomes of the tests until you reach a leaf node. Decision Trees are intuitive and easy to interpret, but a single tree can be prone to overfitting the training data.

Random Forests

Random Forests are an ensemble learning method that builds multiple decision trees during training and outputs the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. By introducing randomness in tree construction (e.g., using random subsets of features and data samples), Random Forests generally achieve higher accuracy and are more robust against overfitting than individual decision trees.

Gradient Boosting Machines (GBMs)

Gradient Boosting is another powerful ensemble technique that builds trees sequentially, where each new tree corrects the errors of the previous ones. Algorithms like XGBoost, LightGBM, and CatBoost are highly efficient and effective implementations of gradient boosting, often providing state-of-the-art results on structured or tabular data. They are known for their high predictive accuracy but can require careful tuning of hyperparameters.

Naive Bayes

Naive Bayes classifiers are a family of simple probabilistic classifiers based on applying Bayes’ theorem with strong (naive) independence assumptions between the features. For example, to classify fruits from a basket based on features like color, size, and shape, Naive Bayes would assume that the color of a fruit is independent of its size, given its class (e.g., ‘Apple’). Despite this often unrealistic independence assumption, Naive Bayes classifiers perform surprisingly well in many real-world situations, especially in document classification and spam filtering, and require relatively small amounts of training data.

Logistic Regression

Despite its name, Logistic Regression is a widely used algorithm for binary classification problems (where the output is one of two classes). Imagine a 2D plot with data points belonging to two distinct groups, say red dots and green dots. Logistic Regression aims to find a line (or, more generally, a hyperplane in higher dimensions) that best separates these two groups. The algorithm models the probability that a given input point belongs to a particular class using a logistic (or sigmoid) function. New data points are then classified based on which side of the decision boundary they fall, or more precisely, based on their predicted probability.

Support Vector Machines (SVMs)

Support Vector Machines (SVMs) are powerful classifiers that aim to find the optimal hyperplane that maximally separates classes in the feature space. Consider a scenario where data points of different classes cannot be separated by a single straight line; for example, one class might form a ring around another. SVMs can handle such non-linear separation by using a ‘kernel trick,’ which implicitly maps the data into a higher-dimensional space where a linear separation might be possible. The boundary found by SVMs is determined by ‘support vectors,’ which are the data points closest to the decision boundary. SVMs can be effective in high-dimensional spaces and are versatile due to the choice of different kernel functions.