1

Supervised Learning Algorithms

Orina Dennis

24/09072

DSA 6201: Machine Learning

6/10/2025
 2

Supervised Learning Algorithms

Supervised learning stands as a pillar of modern artificial intelligence. By training models

on labeled datasets, it allows systems to make accurate predictions or classifications. Whether the

goal is to forecast house prices or distinguish spam emails from genuine ones, supervised

learning algorithms are the key tools. Below, we creatively unpack five of the most widely used

supervised learning algorithms, blending technical clarity with real-world analogy, and explain

their theoretical principles and whether they are best suited for classification or regression.

1. Linear Regression – The Straight-Line Thinker (Regression)

Imagine trying to guess someone's salary based on their years of experience. You might

assume there's a line that generally increases: the more experience, the higher the salary. This is

the world of linear regression, where the goal is to find a straight-line relationship between input

features (independent variables) and an output (dependent variable).

The theoretical backbone of linear regression is the ordinary least squares (OLS) method,

which minimizes the squared differences between observed values and predictions.

Mathematically, the model fits the line:

Y = β 0+ β 1 X+ ϵ

Where Y is the predicted output, X the input, β0 and β1 are coefficients, and ϵ is the error term

(Shetty et al., 2022).

Linear regression is the go-to algorithm for regression tasks—like predicting house

prices, CO₂ levels, or sales forecasts (Supervised Learning in Machine Learning, 2025).
 3

2. Logistic Regression – The Probability Bouncer (Classification)

Now shift gears: imagine you're building a system to detect whether an email is spam or

not. The outcome is binary, spam or not-spam, and this is where logistic regression shines.

Despite its name, it’s a classification algorithm.

Instead of drawing a straight line, logistic regression fits an S-shaped curve (sigmoid

function), transforming any real-valued number into a value between 0 and 1, interpreted as a

probability:

1
P( y =1)= −(β 0 + β 1 X )
1+e

This curve acts like a gatekeeper, if the probability exceeds a threshold (say 0.5), the

input is classified as one class; otherwise, it falls into another (Shetty et al., 2022).

It’s ideal for binary classification tasks, such as fraud detection, spam filtering, and

medical diagnosis.

3. Decision Trees – The Flowchart Oracle (Classification & Regression)

If algorithms were people, decision trees would be the meticulous planner, drawing out

every possible outcome based on questions. For instance, to determine if someone qualifies for a

loan, a decision tree might ask: “Is their credit score above 700?”, “Do they have a steady

income?”, and so on.

Theoretically, decision trees use Information Gain and Entropy for classification,

choosing features that best split the dataset. For regression tasks, they rely on Standard Deviation

Reduction (Gianey & Choudhary, 2017).

Each node in the tree represents a feature test, each branch an outcome, and each leaf a final

decision. Decision trees can be used for both classification and regression, depending on the

nature of the target variable.

They are particularly valued for their interpretability, making them favorites in domains

like healthcare and finance where transparency is vital.

4. Support Vector Machines (SVM) – The Boundary Setter (Classification & Regression)

Think of SVMs as perfectionists with a ruler, carefully drawing a line (or hyperplane)

that separates two groups with the widest possible margin. If you were separating apples and

oranges on a table, SVM would find the most balanced divider.

The algorithm works by finding a hyperplane that best separates data into classes. For

non-linear separable data, kernel functions (like polynomial or radial basis functions) project

data into higher dimensions for linear separation (Shetty et al., 2022).

For regression, SVM aims to fit the best possible function that deviates no more than ε

from the actual target, known as Support Vector Regression (SVR) (Gianey & Choudhary, 2017).

SVMs are robust, particularly in high-dimensional spaces, and are used in applications

ranging from facial recognition to text categorization.

5. K-Nearest Neighbors (KNN) – The Crowd-Sourcer (Classification)

KNN is the democratic algorithm. It doesn’t try to learn a model during training. Instead,

when faced with a new data point, it simply asks: “Who are my nearest neighbors?” Then, it lets

them vote.
 5

This instance-based learning method classifies a new point based on the majority label of

its k closest data points, using a distance metric like Euclidean or Manhattan distance (Gianey &

Choudhary, 2017).

KNN is purely used for classification, although variants can be adapted for regression. It's

particularly useful in recommendation systems and medical diagnosis—fields where proximity in

data often correlates with similar outcomes.

Conclusion

From the linear simplicity of regression to the complex boundaries drawn by SVMs,

supervised learning offers a rich toolkit tailored to diverse problems. The best algorithm isn’t

always the most sophisticated, it depends on the data, the problem, and the trade-off between

interpretability and accuracy. As Gianey and Choudhary (2017) conclude, no single algorithm

outperforms others universally, each has its moment to shine depending on the context.
 6

References

Gianey, H., & Choudhary, R. (2017). Comprehensive Review on Supervised Machine

Learning Algorithms. IEEE International Conference on Machine Learning and Data

Science. pp. 37–42. doi: 10.1109/MLDS.2017.11.

Shetty, S. H., Shetty, S., Singh, C., & Rao, A. (2022). Supervised Machine Learning:

Algorithms and Applications. In P. Singh (Ed.), Fundamentals and Methods of

Machine and Deep Learning (pp. 1–16). Scrivener Publishing LLC.

Supervised Learning in Machine Learning lecture notes (2025). Part 2, pp. 5–6.