Machine Learning Algorithms

Notes, Intuition, Implementation & Examples

Study Notes • Python & scikit-learn

Overview
This document walks through nine of the most important machine learning algorithms. For each one you'll find: a
plain-English explanation, a step-by-step summary of how it works, a diagram to build visual intuition, ready-to-run
Python code using scikit-learn, and a concrete real-world example of where it's used.
Algorithms are grouped into three families:
● Supervised Learning – Regression: Linear Regression
● Supervised Learning – Classification: Logistic Regression, KNN, Decision Tree, Random Forest, SVM,
Naive Bayes
● Unsupervised Learning: K-Means Clustering, PCA
1. Linear Regression
Supervised Learning — Regression
Linear Regression predicts a continuous numeric value by fitting a straight line (or hyperplane) that best describes
the relationship between input features (X) and a target variable (y). It finds the line y = mx + b that minimizes the
sum of squared errors between predicted and actual values.

How It Works
● Assumes a linear relationship between inputs and output.
● Learns coefficients (weights) that minimize the Mean Squared Error (MSE).
● Uses the Ordinary Least Squares method or Gradient Descent to fit the line.
● Works best when the relationship between variables is approximately linear.

Visual Intuition

Implementation (Python / scikit-learn)

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from [Link] import mean_squared_error, r2_score

# X: house size (sq. ft. / 100), y: price ($1000s)

X = [Link]([[5], [6], [7], [8], [9], [10], [12], [14]])
y = [Link]([25, 28, 33, 36, 40, 44, 50, 58])

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.2, random_state=42)

model = LinearRegression()
[Link](X_train, y_train)

y_pred = [Link](X_test)
print("Slope:", model.coef_[0])
print("Intercept:", model.intercept_)
print("R2 Score:", r2_score(y_test, y_pred))

# Predict price for a 1100 sq ft house

print([Link]([[11]]))

Real-World Example
Example: Predicting house prices from square footage, predicting a student's exam score from hours studied, or
forecasting monthly sales revenue based on advertising spend.

Key Points
● Fast, simple, and highly interpretable (each coefficient shows feature impact).
● Sensitive to outliers and assumes linearity — poor fit for curved relationships.
● Evaluate with R² score and Mean Squared Error (MSE).
2. Logistic Regression
Supervised Learning — Classification
Despite the name, Logistic Regression is a classification algorithm. It uses the sigmoid function to squeeze a linear
combination of inputs into a probability between 0 and 1, then applies a threshold (usually 0.5) to assign a class
label.

How It Works
● Computes a weighted sum of the input features, like linear regression.
● Passes that sum through the sigmoid function: σ(z) = 1 / (1 + e^-z).
● Output is interpreted as the probability of belonging to the positive class.
● Trained by minimizing log-loss (cross-entropy) instead of squared error.

Visual Intuition

Implementation (Python / scikit-learn)

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score, confusion_matrix
from [Link] import make_classification

# Study hours & attendance -> Pass (1) / Fail (0)

X, y = make_classification(n_samples=200, n_features=2,
n_redundant=0, random_state=7)

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.25, random_state=42)

model = LogisticRegression()
[Link](X_train, y_train)

y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))

# Probability of passing
print(model.predict_proba(X_test[:1]))

Real-World Example
Example: Email spam detection (spam / not spam), predicting whether a customer will churn, medical diagnosis
(disease present / absent), or credit approval (approve / reject).

Key Points
● Outputs probabilities, not just labels — useful for ranking or risk scoring.
● Works best when classes are linearly separable.
● Extendable to multi-class problems (softmax / one-vs-rest).
3. K-Nearest Neighbors (KNN)
Supervised Learning — Classification / Regression
KNN is a simple, instance-based ('lazy') algorithm. To classify a new point, it looks at the 'k' closest labeled points
in the training data (using a distance metric like Euclidean distance) and assigns the majority class among them.

How It Works
● Choose a value of k (number of neighbors to consider).
● Compute the distance from the new point to every training point.
● Select the k nearest points.
● Assign the class that appears most frequently among those neighbors (majority vote).

Visual Intuition

Implementation (Python / scikit-learn)

from [Link] import KNeighborsClassifier
from [Link] import StandardScaler
from sklearn.model_selection import train_test_split
from [Link] import make_classification
from [Link] import accuracy_score

X, y = make_classification(n_samples=200, n_features=2,
n_redundant=0, random_state=3)

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.25, random_state=42)

# KNN is distance-based, so features MUST be scaled

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = [Link](X_test)

model = KNeighborsClassifier(n_neighbors=5)
[Link](X_train, y_train)

y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))

Real-World Example
Example: Recommender systems ('customers who bought this also liked...'), handwriting/digit recognition, and
recommending similar houses based on price, size, and location.

Key Points
● No training phase — all computation happens at prediction time.
● Choice of k matters: small k = noisy/overfit, large k = smoother/underfit.
● Feature scaling is essential since it relies on distance calculations.
4. Decision Tree
Supervised Learning — Classification / Regression
A Decision Tree splits the data repeatedly using if-else questions on features (e.g., 'Is age > 30?'), forming a tree of
decisions that leads to a final prediction at the leaf nodes. Splits are chosen to maximize information gain (or
minimize Gini impurity).

How It Works
● Start with the entire dataset at the root node.
● Find the feature and threshold that best splits the data into purer groups (using Gini impurity or entropy).
● Repeat recursively on each branch until a stopping condition is met (e.g., max depth).
● Leaf nodes hold the final predicted class or value.

Visual Intuition
Implementation (Python / scikit-learn)
from [Link] import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split
from [Link] import make_classification
from [Link] import accuracy_score

X, y = make_classification(n_samples=200, n_features=2,
n_redundant=0, random_state=5)

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.25, random_state=42)

model = DecisionTreeClassifier(max_depth=3, criterion="gini")

[Link](X_train, y_train)

y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
print(export_text(model, feature_names=["f1", "f2"]))

Real-World Example
Example: Loan approval decisions ('income > $50k?' -> 'credit score > 700?'), medical triage flowcharts, and
customer churn prediction where each decision node is human-readable.

Key Points
● Highly interpretable — you can literally read the decision path.
● Prone to overfitting if grown too deep (use max_depth or pruning).
● Forms the building block for ensemble methods like Random Forest.
5. Random Forest
Supervised Learning — Ensemble (Classification / Regression)
Random Forest builds many Decision Trees, each trained on a random subset of the data and features (bagging),
then combines their predictions by majority vote (classification) or averaging (regression). This reduces overfitting
and improves accuracy.

How It Works
● Create many bootstrap samples (random subsets with replacement) of the training data.
● Train an independent decision tree on each sample, using a random subset of features at each split.
● For prediction, every tree 'votes' and the majority class wins (or values are averaged for regression).

Visual Intuition

Implementation (Python / scikit-learn)

from [Link] import RandomForestClassifier
from sklearn.model_selection import train_test_split
from [Link] import make_classification
from [Link] import accuracy_score

X, y = make_classification(n_samples=300, n_features=2,
n_redundant=0, flip_y=0.05, random_state=9)

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.25, random_state=42)

model = RandomForestClassifier(
n_estimators=100, max_depth=4, random_state=0)
[Link](X_train, y_train)
y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Feature importance:", model.feature_importances_)

Real-World Example
Example: Fraud detection in banking, predicting equipment failure in manufacturing (predictive maintenance), and
Kaggle-style tabular data competitions where it's a strong baseline.

Key Points
● Much more robust and accurate than a single decision tree.
● Provides feature importance scores — useful for understanding drivers.
● Slower to train and less interpretable than a single tree ('black box' ensemble).
6. Support Vector Machine (SVM)
Supervised Learning — Classification / Regression
SVM finds the hyperplane that best separates classes by maximizing the margin — the distance between the
hyperplane and the nearest data points from each class (support vectors). For non-linear data, it uses the 'kernel trick'
to project data into higher dimensions where it becomes separable.

How It Works
● Find the decision boundary (hyperplane) that maximizes the margin between classes.
● Only the closest points to the boundary (support vectors) matter for defining it.
● For non-linear data, apply a kernel function (e.g., RBF, polynomial) to transform the feature space.
● The parameter C controls the trade-off between a wide margin and misclassification.

Visual Intuition

Implementation (Python / scikit-learn)

from [Link] import SVC
from [Link] import StandardScaler
from sklearn.model_selection import train_test_split
from [Link] import make_classification
from [Link] import accuracy_score

X, y = make_classification(n_samples=200, n_features=2,
n_redundant=0, random_state=2)

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.25, random_state=42)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = [Link](X_test)

model = SVC(kernel="rbf", C=1.0, gamma="scale")

[Link](X_train, y_train)

y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))

Real-World Example
Example: Image classification (cat vs. dog), face detection, text/document categorization, and bioinformatics
(classifying proteins/genes) where data is high-dimensional.

Key Points
● Effective in high-dimensional spaces, even when features > samples.
● Kernel trick lets it model complex, non-linear boundaries.
● Requires feature scaling and can be slow to train on very large datasets.
7. Naive Bayes
Supervised Learning — Classification
Naive Bayes applies Bayes' Theorem to compute the probability of each class given the input features, assuming
(naively) that all features are independent of each other. Despite this simplifying assumption, it performs remarkably
well on text and categorical data.

How It Works
● Uses Bayes' Theorem: P(class | features) ∝ P(features | class) × P(class).
● Assumes features are conditionally independent given the class ('naive' assumption).
● Estimates P(feature | class) from the training data (Gaussian, Multinomial, or Bernoulli distributions
depending on data type).
● Predicts the class with the highest posterior probability.

Visual Intuition

Implementation (Python / scikit-learn)

from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score

# Example: spam detection with text

texts = ["win free money now", "meeting at noon tomorrow",
"claim your free prize", "let's grab lunch"]
labels = [1, 0, 1, 0] # 1 = spam, 0 = not spam

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
model = MultinomialNB()
[Link](X, labels)

new_email = [Link](["free prize waiting for you"])

print("Spam?" , [Link](new_email))
print("Probabilities:", model.predict_proba(new_email))

Real-World Example
Example: Email spam filtering, sentiment analysis (positive/negative reviews), and news article categorization —
all classic text-classification use cases.

Key Points
● Extremely fast to train, even on large datasets, and needs little data.
● The independence assumption is rarely true in practice, yet it still works well.
● Great baseline model for text classification problems.
8. K-Means Clustering
Unsupervised Learning — Clustering
K-Means groups unlabeled data into k clusters by iteratively assigning each point to its nearest centroid and then
updating each centroid to be the mean of its assigned points, repeating until the centroids stop moving.

How It Works
● Choose the number of clusters, k.
● Randomly initialize k centroids.
● Assign each data point to its nearest centroid.
● Recalculate each centroid as the mean of its assigned points.
● Repeat steps 3-4 until centroids stabilize (convergence).

Visual Intuition

Implementation (Python / scikit-learn)

import numpy as np
from [Link] import KMeans
from [Link] import StandardScaler

# Customer data: [annual income, spending score]

X = [Link]([[15, 39], [16, 81], [17, 6], [18, 77],
[19, 40], [65, 55], [70, 15], [72, 90],
[80, 20], [85, 60]])

X_scaled = StandardScaler().fit_transform(X)

model = KMeans(n_clusters=4, n_init=10, random_state=0)

[Link](X_scaled)
print("Cluster labels:", model.labels_)
print("Centroids:", model.cluster_centers_)

# Use the Elbow Method to choose k:

inertias = []
for k in range(1, 8):
km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(X_scaled)
[Link](km.inertia_)

Real-World Example
Example: Customer segmentation for targeted marketing, grouping similar news articles or documents, image
compression (color quantization), and identifying regions of similar land-use in satellite imagery.

Key Points
● Requires you to pick k in advance — use the Elbow Method or Silhouette Score to choose it.
● Sensitive to feature scale and outliers — always standardize features first.
● Assumes roughly spherical, similarly-sized clusters; struggles with irregular shapes.
9. Principal Component Analysis (PCA)
Unsupervised Learning — Dimensionality Reduction
PCA is not a predictive algorithm but a preprocessing technique that compresses many correlated features into a
smaller number of uncorrelated 'principal components' while retaining as much variance (information) as possible.
It's often used before other algorithms to speed up training and reduce noise.

How It Works
● Standardize the data so all features are on the same scale.
● Compute the covariance matrix of the features.
● Find the eigenvectors (principal components) and eigenvalues (variance explained) of that matrix.
● Project the data onto the top k components that explain the most variance.

Visual Intuition

Implementation (Python / scikit-learn)

from [Link] import PCA
from [Link] import StandardScaler
from [Link] import load_iris

iris = load_iris()
X = StandardScaler().fit_transform([Link]) # 4 features

pca = PCA(n_components=2) # compress to 2 dimensions

X_reduced = pca.fit_transform(X)

print("Explained variance ratio:", pca.explained_variance_ratio_)

print("New shape:", X_reduced.shape) # (150, 2) instead of (150, 4)
Real-World Example
Example: Compressing high-resolution images while preserving key features, visualizing high-dimensional gene-
expression or customer data in 2D/3D, and speeding up training for algorithms like KNN or SVM on datasets with
hundreds of features.

Key Points
● Components are ranked by how much variance (information) they explain.
● Trades some interpretability for reduced dimensionality and less noise.
● Always standardize features first — PCA is sensitive to feature scale.
Quick Comparison
Algorithm Type Best For Watch Out For
Linear Regression Regression Predicting continuous numbers Assumes linear relationships
Logistic Regression Classification Binary outcomes, probabilities Struggles with non-linear boundaries
KNN Both Simple pattern matching Slow on large datasets, needs scaling
Decision Tree Both Interpretable rules Overfits if too deep
Random Forest Both High accuracy, tabular data Less interpretable, slower
SVM Both High-dimensional data Slow on large datasets
Naive Bayes Classification Text/spam classification Independence assumption is unrealistic
K-Means Clustering Grouping unlabeled data Must choose k in advance
PCA Dim. Reduction Compressing features Reduces interpretability