PAGE 1 – Introduction to Supervised Learning Algorithms

Definition:
Supervised Learning involves training a model on labeled data, where the input features are
mapped to known output labels.

Goal: Predict the label for new, unseen data.

Common Supervised Algorithms:

1. Linear Regression

2. Logistic Regression

3. K-Nearest Neighbors (KNN)

4. Decision Trees

5. Naive Bayes

PAGE 2 – Linear Regression Overview

Definition: Predicts a continuous numeric value based on input features.

Example: Predict house prices based on size, location, and age.

Equation (Simple Linear Regression):

y = b0 + b1*x

 y: predicted output

 x: input feature

 b0: intercept

 b1: slope

PAGE 3 – Linear Regression Example

Size ([Link]) Price (₹)
1000 50,000
1200 60,000
1500 75,000

Goal: Fit a line that minimizes error between predicted and actual prices.

Technique: Ordinary Least Squares (OLS) – minimizes Mean Squared Error (MSE).

PAGE 4 – Multiple Linear Regression

When multiple features exist:

y = b0 + b1*x1 + b2*x2 + b3*x3 + ...

Example: Predict house price using:

 x1 = Size

 x2 = Number of rooms

 x3 = Age of house

PAGE 5 – Evaluation Metrics (Regression)

 Mean Absolute Error (MAE)

 Mean Squared Error (MSE)

 Root Mean Squared Error (RMSE)

 R² Score

Goal: Smaller error = better model

PAGE 6 – Logistic Regression Overview

Definition: Predicts categorical outcomes (binary or multi-class).
Example: Email spam detection – Spam (1) / Not Spam (0)

Equation (Sigmoid Function):

p = 1 / (1 + e^-(b0 + b1*x))

 p = probability of class 1

PAGE 7 – Logistic Regression Example

Hours Studied Passed (1)/Failed (0)
2 0
5 1
3 0

The model predicts probability of passing based on study hours.

PAGE 8 – Evaluation Metrics (Classification)

 Accuracy

 Precision

 Recall

 F1 Score

 Confusion Matrix

PAGE 9 – K-Nearest Neighbors (KNN) Overview

Definition: A simple algorithm that classifies new data points based on similarity to neighbors.

Idea: “Birds of a feather flock together”

Steps:

1. Choose k (number of neighbors)

 2. Compute distance to all training points

3. Select k nearest neighbors

4. Take majority class → predict label

PAGE 10 – KNN Example

Classify whether a fruit is apple or orange based on:

Weight (g) Color Score Label

150 3 Apple
170 2 Orange

New fruit: Weight = 160, Color Score = 2 → KNN predicts Orange (based on nearest
neighbors).

PAGE 11 – Choosing k in KNN

 Small k → sensitive to noise

 Large k → smooth predictions, may ignore small patterns

Tip: Use cross-validation to choose optimal k.

PAGE 12 – Distance Metrics

Common metrics:

 Euclidean Distance:

d = sqrt((x1-y1)^2 + (x2-y2)^2 + ...)

 Manhattan Distance: sum of absolute differences

 Minkowski Distance: generalization of both

PAGE 13 – Decision Trees Overview
Definition: Supervised algorithm that splits data into branches based on feature values.

Analogy: A flowchart that asks yes/no questions.

Advantages:

 Easy to understand

 Handles numerical & categorical data

PAGE 14 – Decision Tree Example

Predict whether a student passes based on:

 Hours Studied

 Attendance

Tree Structure:

 Hours > 4 → Pass

 Hours ≤ 4 → Check Attendance

o Attendance > 70% → Pass

o Attendance ≤ 70% → Fail

PAGE 15 – Splitting Criteria

 Gini Impurity: Measures misclassification probability

 Entropy / Information Gain: Measures reduction in uncertainty

 Chi-Square: Statistical significance of split

PAGE 16 – Advantages of Decision Trees
 Simple visualization

 Non-linear relationships handled

 Can handle mixed data types

PAGE 17 – Limitations of Decision Trees

 Prone to overfitting

 Sensitive to small changes in data

 Can be biased toward features with more levels

PAGE 18 – Random Forest (Brief Mention)

 Ensemble of multiple decision trees

 Reduces overfitting

 Improves accuracy

 Voting or averaging for final prediction

PAGE 19 – Naive Bayes Overview

Definition: Probabilistic classifier based on Bayes’ Theorem assuming features are
independent.

Bayes’ Theorem:

P(C|X) = (P(X|C) * P(C)) / P(X)

 C = class

 X = features
PAGE 20 – Naive Bayes Example
Classify email as Spam or Not Spam:

Feature Spam Not Spam

Contains “win” 0.8 0.1
Sender unknown 0.7 0.3

Combine probabilities → classify email.

PAGE 21 – Types of Naive Bayes

 Gaussian: Continuous data

 Multinomial: Count/frequency data (text)

 Bernoulli: Binary data (yes/no features)

PAGE 22 – Advantages of Naive Bayes

 Simple & fast

 Works well with small datasets

 Good for text classification (spam, sentiment)

PAGE 23 – Limitations of Naive Bayes

 Assumes feature independence

 May perform poorly if independence assumption violated

PAGE 24 – Comparison of Supervised Algorithms

 Algorithm Type Output Notes
Linear Regression Regression Numeric Sensitive to outliers
Logistic
Classification Binary/Prob Good for probabilities
Regression
KNN Classification Discrete Simple, needs distance metric
Decision Tree Classification/Regression Discrete/Continuous Easy to interpret
Probabilistic, assumes
Naive Bayes Classification Discrete
independence

PAGE 25 – Choosing the Right Algorithm

 Numeric prediction → Linear Regression

 Binary classification → Logistic Regression or Naive Bayes

 Non-linear data → Decision Tree / Random Forest

 Small dataset, simple → KNN

PAGE 26 – Model Training Steps (Recap)

1. Split data → Training & Testing

2. Choose algorithm

3. Train model on training data

4. Test on unseen data

5. Evaluate using metrics

PAGE 27 – Overfitting in Supervised Learning

 Model performs well on training data but poorly on test data

 Solutions:

o Cross-validation
 o Regularization (L1, L2)

o Ensemble methods

PAGE 28 – Cross-Validation
 Split data into k folds

 Train on k-1 folds, test on remaining fold

 Repeat k times → Average performance

PAGE 29 – Hyperparameter Tuning

 Algorithm parameters not learned from data

 Examples:

o K in KNN

o Max depth in Decision Tree

o Regularization in Linear/Logistic Regression

PAGE 30 – Scaling and Normalization

 Many algorithms (KNN, Logistic Regression) are sensitive to scale

 Techniques:

o Min-Max Scaling

o Standardization

PAGE 31 – Handling Categorical Features

  Label Encoding → Convert categories to numbers

 One-Hot Encoding → Create binary columns for each category

PAGE 32 – Example: Predict Student Result

Dataset Features: Hours studied, Attendance, Sleep hours
Label: Pass / Fail

Steps:

1. Split dataset

2. Train KNN, Decision Tree, and Logistic Regression

3. Compare accuracy, precision, recall

4. Select best model

PAGE 33 – Practice Problem 1

Problem: Predict whether a patient has diabetes using features: Age, BMI, Blood Pressure.

Questions:

1. Which algorithm would you choose?

2. How would you split the dataset?

3. Which metrics would you use?

PAGE 34 – Practice Problem 2

Problem: Classify emails as spam or not spam using Naive Bayes.

Questions:

1. Identify features from emails

 2. Explain how probabilities are calculated

3. Test model on new emails

PAGE 35 – Advantages of Supervised Learning

 Accurate predictions

 Easy to evaluate performance

 Works well when labeled data is available

PAGE 36 – Limitations
 Requires labeled data

 Sensitive to noisy or missing data

 May overfit if model is too complex

PAGE 37 – Summary
 Supervised Learning → Uses labeled data

 Linear Regression → Continuous numeric prediction

 Logistic Regression → Binary classification

 KNN → Simple distance-based method

 Decision Tree → Flowchart-like model

 Naive Bayes → Probabilistic classifier

PAGE 38 – Key Terms Recap

  Label / Target

 Feature / Input

 Training / Testing Data

 Overfitting / Underfitting

 Cross-Validation

 Hyperparameter

PAGE 39 – Quick Tips

 Preprocess your data before training

 Always split dataset for evaluation

 Choose algorithms based on problem type

 Check for overfitting and tune parameters

PAGE 40 – Practice Questions

1. Explain the difference between Linear and Logistic Regression

2. How does KNN classify new data points?

3. Draw a simple Decision Tree for passing/failing students

4. What is Naive Bayes and why is it “naive”?

5. How do you prevent overfitting in supervised learning?