Module 6: Introduction to ML (08) CO6

1. Introduction to Machine Learning,

2. Types of Machine Learning:
3. Supervised
Logistic Regression,
Decision Tree,
Support Vector Machine
4. Unsupervised
K Means Clustering,
Hierarchical Clustering,
Association Rules
5. Issues in Machine learning,
6. Application of Machine Learning
7. Steps in developing a Machine Learning Application.

Notes By Archana S (VPPCOE& VA)

Confusion Matrix Evaluation

Actual \ Predicted Class 1 Class 2

Class 1 True Positive False Negative
Class 2 False Positive True Negative

•True Positive (TP): Correctly predicted Class 1

•False Positive (FP): Incorrectly predicted Class 1
•True Negative (TN): Correctly predicted Class 2
•False Negative (FN): Incorrectly predicted Class 2

Metrics Derived
𝑇𝑃+𝑇𝑁
•Accuracy = 𝑇𝑜𝑡𝑎𝑙
𝑇𝑃
•Precision = 𝑇𝑃+𝐹𝑃
𝑇𝑃
•Recall = 𝑇𝑃+𝐹𝑁
•F1 Score = Harmonic mean of precision and recall
Confusion Matrix Evaluation
Example: Diabetes Prediction
Problem
Classify patients as:
Diabetic (Class 1)
Non-Diabetic (Class 2)

Features
Age.
BMI (Body Mass Index)
Blood pressure
Glucose level
Insulin level
Skin thickness
Decision Boundaries
A classifier (e.g., Logistic Regression or Random Forest) learns to separate diabetic and non-diabetic patients based on patterns in these features.
Note: A decision boundary divides the feature space — patients with high glucose and BMI may fall into the diabetic region.
Prediction Logic
A new patient’s data:
Age: 45
BMI: 32
Glucose: 160
Blood Pressure: 85
The classifier evaluates this input and places the patient in the diabetic class → Predicted: Diabetic
Actual \ Predicted Diabetic Non-Diabetic
Diabetic TP FN
Non-Diabetic FP TN
Metrics
•Precision: How many predicted diabetics were correct?
•Recall: How many actual diabetics were detected?
•F1 Score: Balances precision and recall
1. Introduction to Machine Learning
Definition:
Machine Learning (ML) is a subset of Artificial Intelligence that enables systems to learn from data and improve performance without
explicit programming.

Instead of writing rules manually, the system:

❑ Observes data
❑ Identifies patterns
❑ Makes predictions or decisions

Traditional Programming vs ML

Traditional Programming Machine Learning

Rules + Data → Output Data + Output → Model
Manual logic Automatic learning

Example
•Traditional: If CGPA < 2.0 → “At Risk”
•ML: Model learns risk level from past student performance data

Why Machine Learning?

•Handles large data
•Adapts to change
•Improves accuracy over time
Machine learning works by feeding data to algorithms, allowing them to learn patterns, and then making
predictions on new data.

The accuracy of predicted output depends upon the amount of data, as the huge amount of data helps to build
a better model which predicts the output more accurately

Suppose we have a complex problem, where we need to perform some predictions, so instead of writing a
code for it, we just need to feed the data to generic algorithms, and with the help of these algorithms, machine
builds the logic as per the data and predict the output. Machine learning has changed our way of thinking
about the problem. The below block diagram explains the working of Machine Learning algorithm:
2. Types of Machine Learning
Main Types
[Link] Learning
[Link] Learning
[Link] Learning (overview only if needed)

2.1 Supervised Learning

•Uses labeled data
•Output variable is known
Example:
Predict Pass/Fail using previous exam records

2.2 Unsupervised Learning

•Uses unlabeled data
•Discovers hidden patterns
Example:
Group customers based on shopping behavior
Machine Learning: Types & Subtypes
1. Supervised Learning
•Definition: Learns from labeled data (input-output pairs).
•Goal: Predict outcomes for new, unseen data.
•Subtypes:
• Regression: Predict continuous values (e.g., house prices).
• Classification: Predict discrete categories (e.g., spam vs. not spam).
•Algorithms:
• Linear Regression
• Logistic Regression
• Decision Trees
• Random Forest
• Support Vector Machines (SVM)
• k-Nearest Neighbors (k-NN)
• Neural Networks

2. Unsupervised Learning
Definition: Works with unlabeled data, finds hidden patterns.
Goal: Discover structure in data.
Subtypes:
Clustering: Group similar data points.
Dimensionality Reduction: Simplify data while retaining structure.
Algorithms:
k-Means Clustering
Hierarchical Clustering
DBSCAN
Principal Component Analysis (PCA)
t-SNE
3. Reinforcement Learning
•Definition: Agent learns by interacting with environment, receiving rewards/penalties.
•Goal: Maximize cumulative reward.
•Subtypes:
• Model-Free RL: Learns directly from experience (Q-learning, SARSA).
• Model-Based RL: Builds a model of environment to plan actions.
Algorithms:
Q-Learning
Deep Q-Networks (DQN)
Policy Gradient Methods
Actor-Critic Models

4. Semi-Supervised Learning
Definition: Uses a mix of labeled and unlabeled data.
Goal: Improve learning efficiency when labels are scarce.
Algorithms:
Self-training
Co-training
Semi-supervised SVM

5. Self-Supervised Learning
Definition: Learns by generating labels from raw data itself.
Goal: Pre-train models for downstream tasks.
Algorithms:
Contrastive Learning (SimCLR, MoCo)
Masked Language Models (BERT)
Supervised learning Definition:
Supervised learning is a machine learning approach where the model is trained on labeled data — each input has a corresponding correct
output. The model learns to map inputs to outputs and generalizes this mapping to make predictions on new data.
Supervised learning is a type of machine learning method in which we provide sample labeled data to the machine learning
system in order to train it, and on that basis, it predicts the output.

Key Characteristics
•Requires labeled training data
•Learns a function 𝑓 𝑥 = 𝑦
•Evaluated using metrics like accuracy, precision, recall, RMSE

Subtypes of Supervised Learning

1. Regression
Goal: Predict continuous numeric values.
Examples:
Predicting house prices
Estimating temperature

2. Classification
Goal: Predict discrete class labels.
Examples:
Spam detection (spam vs. not spam)
Disease diagnosis (positive vs. negative)

Subtype Output Type Example Use Case Algorithms

Regression Continuous Predicting house prices Linear Regression, SVR, RF
Classification Categorical Email spam detection SVM, k-NN, Decision Tree, NN
Examples of Supervised learning
1. Student Result Prediction 6. Loan Approval System
•Input: Attendance, Study Hours, Internal Marks •Input: Income, credit score
•Output: Pass / Fail •Output: Approve / Reject
•Used in academic analytics •Used in finance sector

2. Email Spam Detection 7. Weather Forecasting

•Input: Email content, sender details •Input: Temperature, humidity
•Output: Spam / Not Spam •Output: Rain / No Rain
•Common classification problem •Prediction based on historical data

3. Disease Diagnosis 8. Image Classification

•Input: Blood test values, symptoms •Input: Image pixels
•Output: Disease present / absent •Output: Cat / Dog
•Used in healthcare systems •Used in computer vision

4. Credit Card Fraud Detection 9. Employee Attrition Prediction

•Input: Transaction amount, location •Input: Experience, salary, work hours
•Output: Fraud / Genuine •Output: Will leave / Will stay
•Used by banks •Used in HR analytics

5. House Price Prediction 10. Crop Yield Prediction

•Input: Area, location, number of rooms •Input: Rainfall, soil type, fertilizer
•Output: House price •Output: Expected yield
•Example of regression •Used in smart agriculture
 Types of Supervised Learning
Supervised learning is broadly classified into two types:
[Link]
[Link]
Examples of Classification Examples of Regression
Problem Output Problem Output

Email Spam Detection Spam / Not Spam House Price Prediction ₹ Price

Student Result Prediction Pass / Fail Salary Prediction Salary amount

Temperature Prediction °C
Disease Diagnosis Positive / Negative
Loan Approval Approve / Reject Sales Forecasting Units sold

Image Classification Cat / Dog Crop Yield Prediction Tons/hectare

Student Result Prediction

House Price Prediction
•Input: Attendance, Study Hours
•Input: Area, location, number of rooms
•Output: Pass or Fail
•Output: Price (₹)
Since the output is categorical → ? Since output is numerical
Regression (e.g., Linear Regression) Classification (e.g., SVM)
[Link] Data: Numeric features and target values. [Link] Data: Features with class labels.
[Link] Algorithm: Fits a line (or curve) to minimize error. [Link] Algorithm: Learns decision boundaries.
[Link] Plot: Shows data points and fitted regression line. [Link] Plot: Shows class separation (e.g., circles vs. crosses).
[Link]: Outputs continuous values (e.g., price, temperature). [Link] Label: Predicts discrete categories (e.g., spam or not).
Regression?
Regression is a supervised learning technique used to predict continuous numeric values based on input features. It models the relationship
between independent variables (features) and a dependent variable (target).
Objective
To learn a function 𝑓 𝑥 that maps input features 𝑥 to a continuous output 𝑦.

How Regression Works

1. Input Data
Features: 𝑥1 , 𝑥2 , . . . , 𝑥𝑛
Target: 𝑦
2. Model Training
Fit a function 𝑓 𝑥 to minimize error between predicted and actual values.
Common loss function: Mean Squared Error (MSE)
𝑛
1
MSE = ෍( 𝑦𝑖 − 𝑦ො𝑖 )2
𝑛
𝑖=1
3. Prediction
Use the trained model to predict 𝑦for
ො new inputs.
4. Evaluation
Metrics: RMSE, MAE, R² Score
Example: Predicting House Prices
Problem Statement
You want to build a model that predicts the price of a house based on:
•Square footage
•Number of bedrooms
•Location rating (e.g., proximity to city center)

Step-by-Step Workflow
1. Input Data
Square Footage Bedrooms Location Rating Price (Target)
1200 3 8 ₹60 lakh
1500 4 9 ₹75 lakh
900 2 6 ₹45 lakh
... ... ... ...

2. Model Selection
Use Linear Regression:
Price = 𝑤1 ⋅ Size + 𝑤2 ⋅ Bedrooms + 𝑤3 ⋅ Location + 𝑏
Where:
•𝑤1 , 𝑤2 , 𝑤3 are weights learned during training
•𝑏is the bias term
3. Training the Model
•Fit the model to minimize Mean Squared Error (MSE):
𝑛
1
MSE = ෍( 𝑦𝑖 − 𝑦ො𝑖 )2
𝑛
𝑖=1
•The model learns the best weights to predict price from features.

4. Prediction
Given a new house:
Size: 1400 sq ft
Bedrooms: 3
Location Rating: 7
The model predicts:
𝑦ො = 1400 ⋅ 𝑤1 + 3 ⋅ 𝑤2 + 7 ⋅ 𝑤3 + 𝑏 = ₹68𝑙𝑎𝑘ℎ

5. Evaluation
Use metrics like:
RMSE: Root Mean Squared Error
R² Score: Measures how well the model explains variance in data

Scatter Plot: Each point represents a house (square footage vs. price).
Regression Line: The straight line is the model’s best fit, minimizing the error between
predicted and actual prices.
Interpretation: As square footage increases, the predicted price also increases (positive
slope).
•X-axis: Square Footage
•Y-axis: Price
•Regression Line: Learned from training data
•New Input: A house with 1400 sq ft
•Prediction: Project 1400 onto the regression line → get predicted price (e.g., ₹68 lakh)
This shows how the model uses the learned relationship to estimate outcomes for unseen data.
•Actual Value: The true price of a house (e.g., ₹72 lakh).
•Predicted Value: What the model estimates (e.g., ₹68 lakh).
•Error: The vertical distance between the actual and predicted values — this is what the
model tries to minimize during training.

Common Error Metrics

•Mean Squared Error (MSE): Average of squared errors.
•Root Mean Squared Error (RMSE): Square root of MSE.
•Mean Absolute Error (MAE): Average of absolute errors.
•R² Score: Proportion of variance explained by the model.
Classification?
Classification is a supervised learning technique where the goal is to assign input data into discrete categories (classes) based on labeled
training examples.
•Input: Features (e.g., email text, patient health metrics)
•Output: Class label (e.g., spam/not spam, diabetic/non-diabetic)
•Objective: Learn decision boundaries that separate classes in the feature space.

How Classification Works

Input Data
Training dataset with features and class labels.
Example: Emails labeled as spam or not spam.
Model Training
Algorithm learns patterns in the data.
Builds a decision boundary (line, curve, or hyperplane) that separates classes.
Prediction
For a new input, the model checks which side of the boundary the data falls on.
Assigns the corresponding class label.
Evaluation
Performance measured using metrics like accuracy, precision, recall, F1-score.
Confusion matrix is used to visualize correct vs. incorrect predictions.
1. Input Data
•Collect features and labels. Example: Patient health metrics (age, BMI, glucose level) → Label: Diabetic / Non-Diabetic.
2. Training
•Feed labeled data into a classification algorithm (e.g., Logistic Regression, SVM, Decision Tree).
•The algorithm learns patterns and builds a decision boundary.
3. Decision Boundary
•Separates different classes in the feature space.
•Example: Patients with glucose > 140 and BMI > 30 → Diabetic region.
4. Prediction
•For new input data, the model checks which side of the boundary the point falls on.
•Assigns the corresponding class label.
5. Evaluation
•Compare predictions with actual labels using a confusion matrix.
•Metrics: Accuracy, Precision, Recall, F1-score.

[ Input Data ]
↓
[ Classification Algorithm Training ]
↓
[ Decision Boundary Learned ]
↓
[ Prediction for New Input ]
↓
[ Evaluation (Confusion Matrix, Metrics) ]
Explanation of Each Stage
[Link] Data
1. Features + Labels (e.g., patient health metrics → diabetic/non-
diabetic).
2. This is the raw dataset used for training.
[Link] Training
1. Classification algorithm (Logistic Regression, SVM, Decision
Tree, etc.) learns patterns.
2. Builds a mathematical model to separate classes.
[Link] Boundary
1. The model defines a boundary in feature space.
2. Example: Glucose > 140 → Diabetic region.
[Link]
1. New input data is passed through the model.
2. The model assigns a class label based on which side of the
boundary the input falls.
[Link]
1. Compare predictions with actual labels using a confusion
matrix.
2. Metrics: Accuracy, Precision, Recall, F1-score.
Common Classification Algorithms

Algorithm How It Works Example Use Case

Logistic Regression Uses probability thresholds Disease diagnosis
Decision Trees Splits data by feature thresholds Loan approval
Ensemble of trees for robust
Random Forest Fraud detection
classification
k-Nearest Neighbors (k-NN) Classifies based on closest neighbors Image recognition
Support Vector Machines Finds optimal hyperplane Spam detection
Probabilistic model using Bayes’
Naive Bayes Text classification
theorem
Learns complex non-linear
Neural Networks Speech recognition
boundaries
Linear Regression
Concept
Linear regression models the relationship between a dependent variable 𝑦and one or more independent variables 𝑥by fitting a straight line.
For simple linear regression (one feature):
𝑦 = 𝑤1 𝑥 + 𝑏
For multiple linear regression (many features):
𝑦 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑤𝑛 𝑥𝑛 + 𝑏
Here:
•𝑤𝑖 =weight (coefficient) for each feature
•𝑏= bias (intercept)
•𝑦= predicted output
Working (Step-by-Step)
Collect Data Example: House prices dataset with features like square footage, number of bedrooms, and location rating.
Fit the Model
The algorithm finds the best line (or hyperplane) that minimizes the difference between predicted and actual values.
Uses Ordinary Least Squares (OLS) method.
Loss Function
Commonly Mean Squared Error (MSE):
𝑛
1
MSE = ෍( 𝑦𝑖 − 𝑦ො𝑖 )2
𝑛
𝑖=1
The model adjusts weights 𝑤𝑖 to minimize this error.
Prediction
For new input data, plug values into the equation to get predicted output.
Evaluation
Metrics: RMSE, MAE, R² Score (explains variance captured by the model).
Example
Predicting House Prices
•Dataset: Square footage (X) vs. Price (Y)
•Model learns:
Price = 5000 ⋅ Size + 100000
•For a 1200 sq ft house:
𝑦ො = 5000 ⋅ 1200 + 100000 = ₹70,00,000

Strengths
Easy to understand and interpret.
Fast to train.
Works well when relationship is approximately linear.
Limitations
Assumes linearity (not suitable for curved
relationships).
Sensitive to outliers.
Struggles with multicollinearity (correlated features).
Polynomial Regression
Concept
Polynomial Regression is an extension of Linear Regression that models the relationship between the dependent variable 𝑦and the
independent variable 𝑥as an 𝑛-degree polynomial:
𝑦 = 𝑤0 + 𝑤1 𝑥 + 𝑤2 𝑥 2 + 𝑤3 𝑥 3 + ⋯ + 𝑤𝑛 𝑥 𝑛
•It captures non-linear relationships between variables.
•Still considered a linear model in terms of coefficients (linear in parameters).
Step-by-Step Working
1. Input Data
Collect data with a clear non-linear trend.
Example: Crop height over time.

Day Height (cm)

1 5
2 15
3 35
4 65
5 100

2. Feature Transformation
•Convert input 𝑥into polynomial features:
• 𝑥, 𝑥 2 ,𝑥 3 … ,up to degree 𝑛
•For degree 2:
𝑋 = 𝑥 𝑥2
3. Model Training
•Fit the polynomial equation to the data using Least Squares.
•Learn coefficients 𝑤0 , 𝑤1 , 𝑤2 , …that minimize:
1
MSE = ෍( 𝑦𝑖 − 𝑦ො𝑖 )2
𝑛
4. Prediction
•For a new input 𝑥 = 6, compute:
𝑦 = 𝑤0 + 𝑤1 ⋅ 6 + 𝑤2 ⋅ 62

5. Evaluation
Use metrics like:
RMSE (Root Mean Squared Error)
R² Score (explained variance)

Example
Crop Growth Modeling
Fit a 2nd-degree polynomial:
𝑦 = 2𝑥 2 + 3𝑥 + 5
For Day 6:
𝑦 = 2 ⋅ 36 + 3 ⋅ 6 + 5 = 72 + 18 + 5 = 95 cm

Strengths
Captures complex, curved relationships.
Easy to implement using PolynomialFeatures in scikit-learn.
Limitations
Risk of overfitting with high-degree polynomials.
Sensitive to outliers.
Poor extrapolation outside training range.
Linear Regression assumes a constant rate of change — it’s too rigid for curved data.
Polynomial Regression introduces higher-degree terms (like 𝑥2,𝑥3) to model non-linear patterns.

Example
If you're modeling crop height over time, and the growth accelerates:
•Linear regression might say: “Height increases by 10 cm/day.”
•Polynomial regression might say: “Height increases slowly at first, then rapidly — like a curve.”

Linear Regression (Dashed Line)

•Fits a straight line through the data.
•Assumes a constant rate of change.
•Limitation: Too rigid for curved data → leads to underfitting.

Polynomial Regression (Curved Line)

•Fits a smooth curve by adding polynomial terms (𝑥 2 , 𝑥 3 , ….(
•Captures non-linear patterns in the data.
•Advantage: Models curved growth or acceleration more accurately.
•Risk: High-degree polynomials can cause overfitting.
Example
•Linear Fit: Predicts crop height increases steadily (e.g., +10 cm/day).
•Polynomial Fit: Captures reality — slow growth at first, then rapid acceleration.
Ridge Regression (L2 Regularization)
Ridge Regression is a regularized version of linear regression
𝜆controls regularization strength:
that adds a penalty term to the loss function to prevent
Small 𝜆: behaves like linear regression
overfitting, especially when features are highly correlated or
Large 𝜆: shrinks weights aggressively
numerous.
4. Training
•It minimizes:
𝑛 Uses gradient descent or closed-form solution to minimize the
Loss = MSE + 𝜆 ෍ 𝑤𝑖2 regularized loss.
Coefficients are adjusted to balance fit and simplicity.
𝑖=1
•The penalty term 𝜆 ෌ 𝑤𝑖2 is called L2 regularization. 5. Prediction
For new input 𝑥, compute:
•It shrinks coefficients but does not eliminate them.
𝑦ො = 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑤𝑛 𝑥𝑛 + 𝑏
.6Evaluation
Step-by-Step Working
Metrics: RMSE, MAE, R² Score
1. Input Data
Compare performance across different 𝜆values using cross-
Dataset with multiple features, possibly correlated.
validation.
Example: Predicting student exam scores using:
Example
Study hours
Predicting Exam Scores
Sleep hours
Features:
Stress level
Study hours = 5
Attendance rate
Sleep hours = 6
2. Model Equation
Stress level = 3
Similar to linear regression:
Attendance = 90%
𝑦 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑤𝑛 𝑥𝑛 + 𝑏
Ridge model (with 𝜆 = 0.5) learns:
.3Loss Function
Score
Adds L2 penalty to discourage large weights:
1 = 8 ⋅ Study + 2 ⋅ Sleep − 1.5 ⋅ Stress + 0.3 ⋅ Attendance + 20
Loss = ෍( 𝑦𝑖 − 𝑦ො𝑖 )2 + 𝜆 ෍ 𝑤𝑖2 Prediction:
𝑛 𝑦ො = 8 ⋅ 5 + 2 ⋅ 6 − 1.5 ⋅ 3 + 0.3 ⋅ 90 + 20 = 40 + 12 − 4.5 + 27 + 20
= 94.5
Strengths
•Handles multicollinearity well.
•Reduces model complexity.
•Improves generalization on unseen data.
Limitations
•Doesn’t perform feature selection (unlike Lasso).
•Requires tuning of 𝜆.

Explanation
Axes
•X-axis: Features (e.g., Study Hours, Sleep, Stress)
•Y-axis: Coefficient Magnitude (how strongly each feature influences prediction)
Linear Regression (Dashed Line)
•Coefficients can be large, especially when features are correlated.
•Risk: Overfitting — model fits noise in training data.
Ridge Regression (Solid Curve)
•Coefficients are shrunk toward zero but not eliminated.
•Benefit: Improves generalization and reduces overfitting.

Imagine you're predicting exam scores:

Linear Regression might say: “Study hours = 10x more important than sleep.”
Ridge Regression adjusts to: “Study hours = 6x, sleep = 3x — more balanced.”
Lasso Regression (L1 Regularization)
Concept 4. Training
Lasso Regression is a regularized version of linear regression that adds an •Uses optimization techniques (e.g., coordinate descent) to
L1 penalty to the loss function. This penalty encourages sparsity — minimize the loss.
meaning it can shrink some coefficients to zero, effectively performing •Automatically eliminates irrelevant features by setting their
feature selection. weights to zero.
•Loss function: 5. Prediction
𝑛
•For new input 𝑥, use only the selected features:
Loss = MSE + 𝜆 ෍ ∣ 𝑤𝑖 ∣ 𝑦ො = 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑏
𝑖=1 .6Evaluation
•𝜆controls the strength of regularization. •Metrics: RMSE, MAE, R² Score
•Use cross-validation to find optimal 𝜆
•Step-by-Step Working Example
Predicting Crop Yield
1. Input Data Features:
Dataset with many features, some of which may be irrelevant. Nitrogen level
Example: Predicting crop yield using 50 soil and climate features. Phosphorus
2. Model Equation Potassium
Similar to linear regression: Soil pH
𝑦 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑤𝑛 𝑥𝑛 + 𝑏 Rainfall
3. Loss Function Temperature
Adds L1 penalty to encourage sparsity: 44 other features
1
Loss = ෍( 𝑦𝑖 − 𝑦ො𝑖 )2 + 𝜆 ෍ ∣ 𝑤𝑖 ∣ Lasso model (with 𝜆 = 0.1) learns:
𝑛 Only 6 features have non-zero coefficients.
As 𝜆increases, more coefficients shrink to zero. Remaining 44 features are eliminated.
Final model:
Yield = 0.8 ⋅ Nitrogen + 0.5 ⋅ Rainfall + ⋯ + 𝑏
Strengths Axes
•Performs automatic feature selection. •X-axis: Features (e.g., Feature 1, Feature 2, Feature 3, …)
•Reduces model complexity. •Y-axis: Coefficient Magnitude (how strongly each feature influences prediction)
•Useful for high-dimensional datasets.
Limitations Linear Regression (Dashed Line)
•Can be unstable when features are highly correlated. Retains all features regardless of relevance.
•May discard useful features if 𝜆is too large. Coefficients can be large → risk of overfitting.
Ridge Regression (Curved Line)
Shrinks coefficients but keeps all features.
Useful when features are correlated.
Lasso Regression (Flat Line)
Shrinks some coefficients to exactly zero.
Effectively eliminates irrelevant features → built-in feature selection.

Imagine you're predicting crop yield using 50 soil features:

Linear Regression: Uses all 50, even noisy ones.
Ridge Regression: Uses all 50, but dampens their influence.
Lasso Regression: Uses only the 6 most relevant features — the rest are dropped.
Elastic Net Regression
Elastic Net Regression blends L1 (Lasso) and L2 (Ridge) regularization to 5. Prediction
balance feature selection and coefficient shrinkage. •Uses selected features with adjusted weights:
•Loss function: 𝑦ො = 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑏
.6Evaluation
Loss = MSE + 𝜆1 ෍ ∣ 𝑤𝑖 ∣ +𝜆2 ෍ 𝑤𝑖2 •Metrics: RMSE, MAE, R² Score
•Use cross-validation to tune 𝜆1 and 𝜆2
•𝜆1 :controls Lasso (sparsity)
Example
•𝜆2 :controls Ridge (shrinkage)
Predicting Crop Yield
Step-by-Step Working
Features:
1. Input Data
Nitrogen
Dataset with many features, some correlated and some irrelevant.
Rainfall
Example: Predicting crop yield using 50 soil, climate, and satellite features.
Soil pH
2. Model Equation
Temperature
Same as linear regression:
46 other features
𝑦 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑤𝑛 𝑥𝑛 + 𝑏
Elastic Net learns:
.3Loss Function
Selects 12 features (like Lasso)
Combines L1 and L2 penalties:
Shrinks correlated ones (like Ridge)
1 2
Loss = ෍( 𝑦𝑖 − 𝑦ො𝑖 ) + 𝜆1 ෍ ∣ 𝑤𝑖 ∣ +𝜆2 ෍ 𝑤𝑖 2 Final model:
𝑛 Yield = 0.7 ⋅ Nitrogen + 0.4 ⋅ Rainfall + ⋯ + 𝑏
Encourages both sparsity and stability. Strengths
4. Training Handles correlated features better than Lasso.
Uses optimization techniques (e.g., coordinate descent). Performs feature selection unlike Ridge.
Selects important features and shrinks correlated ones. More robust than using L1 or L2 alone.
Limitations
Requires tuning of two hyperparameters.
Slightly more complex to implement.
Linear Regression: Uses all features with full coefficients.
Ridge Regression: Shrinks all coefficients but keeps every feature.
Lasso Regression: Eliminates irrelevant features (coefficients = 0).
Elastic Net: Combines both — shrinks some, eliminates others.
Classification Algorithms
1️⃣ Logistic Regression
Concept
•Despite its name, Logistic Regression is used for classification, not regression.
•It models the probability that an instance belongs to a particular class using the sigmoid function:
1
𝑃 𝑦=1∣𝑥 =
1 + 𝑒 − 𝑤1 𝑥1+𝑤2 𝑥2+⋯+𝑏
Step-by-Step Working
[Link] Data: Features (e.g., hours studied, attendance) → Output (Pass/Fail).
[Link] Combination: Compute weighted sum of inputs.
[Link] Function: Convert sum into probability between 0 and 1.
[Link] Boundary: If probability ≥ 0.5 → Class 1, else Class 0.
[Link]: Optimize weights using Maximum Likelihood Estimation.
[Link]: Accuracy, Precision, Recall, F1-score, ROC-AUC.

Example
Predicting whether a student passes an exam:
Input: Study hours = 5, Attendance = 80%
Model outputs probability = 0.78
Since 0.78 > 0.5 → Predict Pass.
Strengths
Simple, interpretable.
Works well for binary classification.
Outputs probabilities.
Limitations
Assumes linear decision boundary.
Struggles with complex non-linear data.
K-Nearest Neighbors (KNN)
Concept
•KNN is a lazy learning algorithm (no explicit training phase).
•It classifies a new data point based on the majority class of its nearest neighbors.
•Distance metrics (e.g., Euclidean, Manhattan) are used to measure closeness.
Step-by-Step Working
1. Input Data
•Dataset with labeled examples.
•Example: Classifying fruits based on weight and sweetness (Apple vs Orange).
2. Choose K
•Select the number of neighbors (K).
•Common choices: 3, 5, 7.
•Small K → sensitive to noise; large K → smoother decision boundary.
3. Distance Calculation
•Compute distance between the new point and all training points.
Euclidean Distance (most common):

𝑑 𝑝𝑞 = ෍( 𝑝𝑖 − 𝑞𝑖 )2

.4Find Nearest Neighbors

Sort distances and pick the K closest points.
5. Majority Voting
Assign the class that appears most frequently among the K neighbors.

6. Prediction
Output the predicted class label.
7. Evaluation
Metrics: Accuracy, Precision, Recall, F1-score.
Cross-validation used to choose optimal K.
Example
Fruit Classification
•Features: Weight (grams), Sweetness (scale 1–10).
•New fruit: Weight = 150 g, Sweetness = 7.
•K = 3 → Nearest neighbors: Apple, Apple, Orange.
•Majority class = Apple → Predict Apple.

Strengths
Simple, intuitive, non-parametric.
Works well with small datasets.
Can capture complex decision boundaries.
Limitations
Computationally expensive for large datasets (needs distance calculation
for all points).
Sensitive to irrelevant features and scaling.
Choice of K and distance metric greatly affects performance.
Decision Tree Classifier 4. Leaf Nodes
Concept •Each leaf node assigns a class label based on majority
•A tree-structured model that splits data into subsets based on feature voting.
values. 5. Prediction
•Each internal node represents a decision rule (e.g., “Is rainfall > 500 •For a new sample, traverse the tree from root to leaf
mm?”). following decision rules.
•Each leaf node represents a class label (e.g., “High Yield” or “Low Yield”). •Output the class label at the leaf.
•The algorithm recursively partitions the dataset until stopping criteria are 6. Evaluation
met. •Metrics: Accuracy, Precision, Recall, F1-score.
Step-by-Step Working •Use cross-validation to avoid overfitting.
1. Input Data
Dataset with features and labels. Example
Example: Predicting whether a crop yield is High or Low based on rainfall Crop Yield Classification
and temperature. Root Node: “Is rainfall > 500 mm?”
2. Splitting Criteria Yes → Next split: “Is temperature > 25°C?”
At each node, choose the best feature to split the data. If Yes → High Yield
Common criteria: If No → Moderate Yield
Gini Index No → Low Yield
𝐺𝑖𝑛𝑖 = 1 − ෍ 𝑝𝑖2
Strengths
Easy to interpret and visualize.
Entropy (Information Gain)
Handles both numerical and categorical data.
𝐸𝑛𝑡𝑟𝑜𝑝𝑦 = − ෍ 𝑝𝑖 log 2
𝑝𝑖 Captures non-linear relationships.
Limitations
3. Recursive Partitioning Prone to overfitting if tree grows too deep.
Split data into subsets based on chosen feature. Sensitive to small changes in data (unstable).
Continue splitting until: Less accurate compared to ensemble methods.
All samples in a node belong to the same class, or
Maximum depth / minimum samples per node is reached.
Random Forest Classifier
•Random Forest is an ensemble learning algorithm that builds multiple 6. Evaluation
decision trees and combines their outputs. Metrics: Accuracy, Precision, Recall, F1-score, ROC-AUC.
•Each tree is trained on a random subset of data and features (bagging + Cross-validation used to tune hyperparameters (e.g., number of
feature randomness). trees, max depth).
•Final prediction is made by majority voting (for classification).
Example
Step-by-Step Working Diabetes Classification
1. Input Data •Features: Glucose level, BMI, Age, Blood pressure.
Dataset with features and labels. •Random Forest builds 100 trees.
Example: Predicting whether a patient has diabetes based on medical test •For a new patient:
results. • 70 trees predict “Diabetic”
2. Bootstrap Sampling (Bagging) • 30 trees predict “Non-Diabetic”
Randomly select subsets of training data (with replacement). •Majority vote → Predict Diabetic.
Each subset is used to train one decision tree.
3. Random Feature Selection
At each split in a tree, only a random subset of features is considered. Strengths
This reduces correlation between trees and improves diversity. High accuracy and robustness.
4. Train Multiple Trees Handles large datasets and many features.
Build many decision trees (e.g., 100, 500). Reduces overfitting compared to a single decision tree.
Each tree learns slightly different patterns due to randomness. Works well with both categorical and numerical data.
5. Prediction Limitations
For a new sample: Less interpretable than a single decision tree.
Computationally expensive with many trees.
Each tree predicts a class label. Slower predictions compared to simpler models.
The forest aggregates predictions via majority voting.
Final output = most common class.
•Random Forest is widely used in practice because it balances
accuracy, robustness, and generalization.
Support Vector Machine (SVM)
Concept 6. Prediction
•SVM is a supervised classification algorithm that finds the optimal •For a new data point, SVM checks which side of the
hyperplane separating classes. hyperplane it falls on.
•The goal is to maximize the margin — the distance between the •Assigns the corresponding class label.
hyperplane and the nearest data points (called support vectors). 7. Evaluation
•Works well in both linear and non-linear classification using kernel •Metrics: Accuracy, Precision, Recall, F1-score, ROC-AUC.
functions. •Cross-validation used to tune hyperparameters (C, kernel
Step-by-Step Working type, gamma).
1. Input Data
Dataset with features and labels. Example
Example: Classifying emails as Spam or Not Spam. Spam Classification
2. Linear Separation Features: Word frequency, presence of links, sender
If data is linearly separable, SVM finds the hyperplane: reputation.
𝑤⋅𝑥+𝑏 =0 SVM builds a hyperplane separating spam vs non-spam
This hyperplane maximizes the margin between classes. emails.
New email: Falls on the “Spam” side of the hyperplane →
3. Support Vectors Predict Spam.
Data points closest to the hyperplane are called support vectors. Strengths
They determine the position and orientation of the hyperplane. Works well in high-dimensional spaces.
4. Margin Maximization Effective for both linear and non-linear data.
SVM chooses the hyperplane with the largest margin to improve Robust to overfitting when margin is maximized.
generalization. Limitations
5. Non-linear Data (Kernel Trick) Computationally expensive for very large datasets.
If data is not linearly separable, SVM uses kernels to project data into Choice of kernel and parameters is critical.
higher [Link] kernels: Less interpretable compared to simpler models.
Linear
Polynomial
Radial Basis Function (RBF)
Sigmoid
Unsupervised Learning: Unsupervised learning works on unlabeled data and finds hidden patterns.
⮚ Unsupervised learning is a learning method in which a machine learns without any supervision.
⮚ The training is provided to the machine with the set of data that has not been labeled, classified, or categorized, and the
algorithm needs to act on that data without any supervision.
⮚ The goal of unsupervised learning is to restructure the input data into new features or a group of objects with similar patterns.
⮚ In unsupervised learning, we don't have a predetermined result. The machine tries to find useful insights from the huge
amount of data. It can be further classifieds into two categories of algorithms:
• Clustering
• Association
Types of Unsupervised Learning Anomaly Detection
Clustering Algorithms •Goal: Identify unusual or rare patterns in data.
•Goal: Group similar data points together. •Examples:
•Examples: • Isolation Forest → isolates anomalies by random
• K-Means Clustering → partitions data into K clusters. partitioning.
• Hierarchical Clustering → builds a dendrogram of nested • One-Class SVM → learns boundary around normal data.
clusters. • Local Outlier Factor (LOF) → detects points with low
• DBSCAN (Density-Based Spatial Clustering of Applications local density.
with Noise) → finds clusters of arbitrary shape and identifies Neural Network–Based Unsupervised Methods
noise/outliers. Goal: Learn representations without labels.
• Gaussian Mixture Models (GMM) → probabilistic clustering Examples:
using mixture of Gaussian distributions. Self-Organizing Maps (SOMs) → map high-dimensional data to 2D
Dimensionality Reduction Algorithms grids.
Goal: Reduce the number of features while preserving important Restricted Boltzmann Machines (RBMs) → probabilistic models
information. for feature learning.
Examples: Generative Models (GANs, VAEs) → learn data distributions and
Principal Component Analysis (PCA) → projects data into fewer generate synthetic samples.
dimensions using variance.
t-SNE (t-distributed Stochastic Neighbor Embedding) → visualizes
high-dimensional data in 2D/3D. Clustering → Grouping (K-Means, DBSCAN, Hierarchical).
UMAP (Uniform Manifold Approximation and Projection) → preserves
both local and global structure. Dimensionality Reduction → Simplification (PCA, t-SNE,
Autoencoders (Neural Networks) → compress and reconstruct data. UMAP).

Association Rule Learning Association Rules → Relationships (Apriori, FP-Growth).

Goal: Discover relationships between items in large datasets. Anomaly Detection → Outliers (Isolation Forest, One-Class
Examples:
Apriori Algorithm → generates frequent itemsets and rules. SVM).Neural Methods → Representation learning (SOMs,
FP-Growth Algorithm → faster frequent pattern mining. Autoencoders, GANs).
Common application: Market Basket Analysis (e.g., “If bread is bought,
butter is likely bought too”).
K-Means Clustering
•K-Means is a partitioning algorithm that groups data into K clusters Example
based on similarity. •Customer Segmentation:
•Each cluster is represented by its centroid (mean of points in that cluster). • Cluster 1 → High spenders, frequent buyers
•Goal: Minimize the distance between points and their cluster centroid. • Cluster 2 → Moderate spenders
• Cluster 3 → Low spenders, occasional buyers
Step-by-Step Working Strengths
Input Data •Simple, fast, scalable.
Dataset without labels. •Works well when clusters are spherical and evenly sized.
Example: Customer purchase behavior (spending amount, frequency). Limitations
Choose K (number of clusters) •Requires predefining K.
Decide how many groups you want (e.g., 3 clusters). •Sensitive to outliers and initialization.
Often chosen using the Elbow Method. •Struggles with irregular cluster shapes.
Initialize Centroids
Randomly select K points as initial centroids.
Assign Points to Nearest Centroid •K-Means is widely used in customer segmentation, image
Compute distance (usually Euclidean). compression, and pattern discovery.
Assign each point to the closest centroid.
Update Centroids
Recalculate centroids as the mean of points in each cluster.
Repeat
Reassign points and update centroids until convergence (no major
changes).
Output
Final clusters with grouped data points.
Hierarchical Clustering
Concept
•Hierarchical clustering builds a tree-like structure (dendrogram) to Example
represent nested groupings of data. Plant Species Grouping
•Unlike K-Means, it doesn’t require predefining the number of clusters. •Step 1: Each plant = its own cluster.
•Two main approaches: •Step 2: Merge two most similar plants.
• Agglomerative (bottom-up): Start with each point as its own •Step 3: Merge clusters iteratively.
cluster, then merge. •Final dendrogram shows nested groupings → cut at height to form 3
• Divisive (top-down): Start with one big cluster, then split. clusters (e.g., Small-leaf, Medium-leaf, Large-leaf species).

Step-by-Step Working (Agglomerative Approach) Strengths

1. Input Data Doesn’t require predefining number of clusters.
Dataset without labels. Produces dendrogram → easy to interpret.
Example: Grouping plant species based on leaf size and shape. Works well for hierarchical/nested data.
2. Initialization Limitations
Treat each data point as its own cluster. Computationally expensive for large datasets.
3. Distance Calculation Sensitive to noise and outliers.
Compute pairwise distances between clusters. Choice of linkage method affects results.
Common metrics: Euclidean, Manhattan.
Linkage criteria decide how to measure distance between clusters: •Hierarchical clustering is widely used in taxonomy (biology),
Single linkage: Minimum distance between points. document clustering, and gene expression analysis.
Complete linkage: Maximum distance between points.
Average linkage: Mean distance between points.
4. Merge Clusters
Merge the two closest clusters into one.
Update the distance matrix.
5. Repeat
Continue merging until all points are in a single cluster.
6. Dendrogram
Visualize the hierarchy of merges.
Cut the dendrogram at a chosen level to form final clusters.
Association Rule Learning Step-by-Step Working
Concept 1. Input Data
•Association rules discover relationships between items in large •Transaction dataset (e.g., supermarket purchases).
datasets. 2. Frequent Itemset Generation
•Commonly used in market basket analysis (e.g., “If a customer buys •Use algorithms like Apriori or FP-Growth to find itemsets
bread, they are likely to buy butter”). with support above a threshold.
•Rules are expressed as: 3. Rule Generation
𝑋⇒𝑌 •From frequent itemsets, generate rules 𝑋 ⇒ 𝑌.
meaning “If X occurs, then Y is likely to occur.” 4. Evaluate Rules
•Calculate support, confidence, lift.
Key Metrics •Keep rules that meet minimum thresholds.
Support 5. Output
Frequency of itemset in the dataset. •A set of strong association rules.
Transactions containing X Example
𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑋 =
Total transactions Supermarket Transactions
Confidence Rule: {Bread} ⇒ {Butter}
Probability that Y occurs given X. Support = 20% (20% of transactions contain both bread and
𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑋 ∪ 𝑌 butter)
𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒 𝑋 ⇒ 𝑌 =
𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑋 Confidence = 70% (70% of bread buyers also buy butter)
Lift Lift = 2.0 (Bread buyers are twice as likely to buy butter
Strength of the rule compared to random chance. compared to random chance)
𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒 𝑋 ⇒ 𝑌
𝐿𝑖𝑓𝑡 𝑋 ⇒ 𝑌 =
𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑌
Strengths
•Reveals hidden patterns in data.
•Useful for recommendation systems.
•Easy to interpret.
Limitations
•Can generate too many rules (requires pruning).
•Computationally expensive for large datasets.
•Doesn’t capture sequential patterns (needs sequential rule mining).

•Association rules are widely used in retail, e-commerce, healthcare

(symptom–disease associations), and web usage mining.
Overfitting in Machine Learning
Concept
•Overfitting happens when a model learns the training data too well, including noise and random fluctuations.
•The model performs very well on training data but poorly on unseen test data.
•It fails to generalize.

Example: Exam Score Prediction

Scenario
Suppose we want to predict a student’s exam score based on hours studied.
Case 1: Simple Linear Model
Model: Score = 5 × Hours + 20
Fits the general trend: more study hours → higher score.
Works well on both training and test data.
Case 2: Overfitted Model
Model: A high-degree polynomial (e.g., 10th degree).
Fits every single training point perfectly.
Training accuracy = 100%.
But on new students, predictions are wildly inaccurate because the model captured noise (like one student who studied 2 hours but still scored
high due to luck).

•Underfitting: Model too simple → misses [Link] Fit: Model captures the trend → generalizes [Link]: Model too complex → zig-
zags through every training point.
Underfitting in Machine Learning
•Underfitting happens when a model is too simple to capture the underlying patterns in the data.
•It performs poorly on both training and test data because it fails to learn enough from the dataset.
•Typically caused by using an overly simplistic model or insufficient training.

Example: Exam Score Prediction

Scenario
We want to predict a student’s exam score based on hours studied.
Case: Underfitted Model
Model: Score = 50 (constant prediction).
Regardless of study hours, it always predicts 50.
Training accuracy is low because it doesn’t capture the upward trend (more study hours → higher score).
Test accuracy is also low because the model is too simplistic.

Real-World Analogy
Imagine a teacher who says:
“Every student will score 50 marks, no matter how much they study.”
This ignores the obvious relationship between study hours and performance.
The model is biased and fails to represent reality.

Causes of Underfitting
Model too simple (e.g., linear model for highly non-linear data).
Too few features used.
Insufficient training (not enough epochs in neural networks).

How to Fix Underfitting

Use a more complex model (e.g., polynomial regression instead of linear).
Add more relevant features.
Train longer or tune hyperparameters.
Reduce regularization strength (since too much regularization can oversimplify the model).
•Underfitting → High bias, low variance. Model is too rigid.
•Good Fit → Balanced bias and variance. Best generalization.
•Overfitting → Low bias, high variance. Model memorizes noise.
Real-World Analogy
•Underfitting: A student barely studies → poor performance everywhere.
•Good Fit: A student learns concepts → performs well on both practice and real exams.
•Overfitting: A student memorizes past exam answers → aces practice but fails real exam.

Case Description Visual Intuition Example

Model is too simple, fails to Straight line through Predicting exam scores
Underfitting capture the underlying scattered curved data with a constant average
trend. points. score for all students.
Predicting exam scores
Model captures the true
Smooth curve following the with a linear model: more
Good Fit pattern without memorizing
general trend of data. study hours → higher
noise.
score.
Using a 10th-degree
Model is too complex, polynomial to fit exam
Wiggly curve passing
Overfitting memorizes training data scores, perfectly matching
through almost every point.
including noise. training data but failing on
new students.
Noise in Machine Learning
In machine learning, noise refers to random, irrelevant, or erroneous data that interferes with learning. It represents deviations from the true
underlying signal in a dataset. Noise can mask meaningful patterns, reduce accuracy, and cause models to misinterpret relationships.

Sources of Noise
Data collection errors → malfunctioning sensors, human mistakes in data entry.
Measurement inaccuracies → imprecise instruments or environmental interference.
Irrelevant features → variables that don’t contribute to prediction but add confusion.
Random fluctuations → natural variability in real-world processes.

Example
Imagine predicting crop yield based on rainfall and soil nutrients:
If a sensor records rainfall incorrectly (e.g., 200 mm instead of 20 mm), that’s noise.
The model may wrongly learn that extreme rainfall leads to high yield, even though it’s just a measurement error.
Impact on Models
Models may overfit noise, treating random errors as patterns.
Accuracy drops on test data because the model fails to generalize.
Analysis pipelines can be distorted if noise isn’t handled.

Handling Noise
Data cleaning → remove or correct erroneous entries.
Feature selection → drop irrelevant variables.
Robust algorithms → use models less sensitive to noise (e.g., ensemble methods).
Regularization → prevent overfitting noisy data.
Signal-to-noise ratio analysis → quantify noise levels in datasets.
Normalization: Rescales values into a fixed range. Standardization: Centers values around mean 0 with unit variance.
Definition: Rescaling data to fit within a specific range, usually [0, 1] or •Definition: Rescaling data so that it has mean = 0 and standard
[-1, 1]. deviation = 1.
Formula (Min-Max Scaling): •Formula (Z-score Scaling):
𝑋 − 𝑋𝑚𝑖𝑛 𝑋−𝜇
𝑋′ = 𝑋′ =
𝑋𝑚𝑎𝑥 − 𝑋𝑚𝑖𝑛 𝜎
Purpose: Ensures all features contribute equally by removing scale where 𝜇= mean, 𝜎= standard deviation.
differences. •Purpose: Centers data and makes variance uniform across
Use Case: Algorithms that rely on distance metrics (e.g., KNN, K- features.
Means, Neural Networks). •Use Case: Algorithms that assume Gaussian distribution or rely
Example: on linear models (e.g., Logistic Regression, SVM, PCA).
Feature: Student study hours = [2, 4, 6, 8, 10] Example:
Normalized to [0, 1] → [0, 0.25, 0.5, 0.75, 1] •Feature: Exam scores = [50, 60, 70, 80, 90]
•Mean = 70, Std Dev = 15
•Standardized → [-1.33, -0.67, 0, 0.67, 1.33]

Aspect Normalization (Min-Max) Standardization (Z-score)

Range [0, 1] or [-1, 1] Mean = 0, Std Dev = 1
Preserves Shape Yes Yes
Sensitive to Outliers Yes (min/max affected) Less sensitive
Distance-based methods (KNN, K-
Best For Linear models, PCA, SVM
Means, NN)
Goal Equal scale Centered distribution
Bias in Machine Learning Impact
Definition •High bias → Underfitting (model misses important
•Bias refers to the error introduced when a model makes overly simplistic patterns).
assumptions about data. •Low bias but high variance → Overfitting (model memorizes
•It’s part of the bias–variance tradeoff: noise).
• High bias → Model is too simple, underfits data. •Goal: Find the right balance for generalization.
• Low bias → Model is flexible enough to capture patterns. How to Reduce Bias
Types of Bias •Use more complex models (e.g., decision trees, ensembles).
Algorithmic Bias •Include relevant features.
Comes from the model’s assumptions (e.g., linear regression assumes •Collect diverse, representative datasets.
linear relationships). •Apply fairness-aware algorithms to reduce social bias.
Example: Using a linear model for highly non-linear data → underfitting.
Data Bias
Arises from skewed or incomplete datasets. •Bias = systematic error due to wrong assumptions or
Example: Training a face recognition system mostly on lighter-skinned skewed data. It’s not always bad — some bias is necessary
faces → poor accuracy for darker-skinned faces. to simplify models, but too much leads to poor generalization.
Human/Selection Bias
Introduced during data collection or labeling.
Example: Survey only includes urban participants → biased against rural
populations.

Example: House Price Prediction

Suppose we want to predict house prices based on size and location.
If we use a linear regression model assuming only size matters, ignoring
location:
Predictions will be systematically wrong for houses in premium
neighborhoods.
This is bias caused by oversimplification.
 Variance in Machine Learning
•Variance measures how much a model’s predictions change when trained How to Reduce Variance
on different subsets of the data. •Use simpler models (reduce complexity).
•High variance means the model is too sensitive to small fluctuations in •Apply regularization (Ridge, Lasso).
the training set. •Use ensemble methods (Random Forest, Bagging).
•It’s the counterpart to bias in the bias–variance tradeoff. •Increase training data size.
•Apply cross-validation to check stability.
•High Variance → Model memorizes training data (overfitting).Low
Variance → Model predictions are stable across [Link]: Achieve a Variance = instability of predictions across
balance where the model generalizes well. different datasets.
High variance → overfitting;
Example: House Price Prediction low variance → stable generalization.
Suppose we train a model to predict house prices:
High Variance Case:
Model fits training data perfectly, capturing noise (e.g., one house sold
unusually high due to a celebrity neighbor).
On new data, predictions fluctuate wildly.
Low Variance Case:
Model ignores noise, predictions remain consistent across different
samples.

Bias–Variance Tradeoff
Bias → Error due to overly simplistic assumptions.
Variance → Error due to sensitivity to training data.
Total error = Bias² + Variance + Irreducible error.
Good models balance bias and variance.
Category Term Meaning / Use
Data Basics Dataset Collection of data points used for training/testing
Feature (Variable) Input attribute used for prediction
Label (Target) Output variable the model predicts
Training/Test/Validation Set Splits of data for model building and evaluation

Model Concepts Bias Error due to oversimplification (underfitting)

Variance Error due to sensitivity to fluctuations (overfitting)

Hyperparameters Settings chosen before training (e.g., learning rate, depth)

Feature Engineering Creating/modifying features to improve performance

Performance Metrics Accuracy % of correct predictions
Precision Correct positives out of predicted positives

Recall (Sensitivity) Correct positives out of actual positives

F1-Score Harmonic mean of precision & recall
ROC Curve / AUC Graphical evaluation of classification performance
Data Handling Normalization/Standardization Scaling features for consistency
Handling incomplete data
Missing Values
(imputation/removal)
Outliers Extreme values that distort models
Noise Random errors or irrelevant data
Advanced Topics Overfitting Model too complex, memorizes noise
Underfitting Model too simple, misses patterns
Penalizes large coefficients to prevent
Regularization (L1/L2)
overfitting
Robust evaluation by splitting data
Cross-Validation
multiple times
Simplifying features while preserving
Dimensionality Reduction
info
In supervised machine learning, algorithms learn from labeled data.
After understanding the data, the algorithm determines which label should be given to new data by associating patterns to the unlabeled new data.
Supervised learning can be divided into two categories: classification and regression.

What Is Classification?
Classification predicts the category the data belongs to. Some examples of classification include spam detection, churn prediction, sentiment
analysis, dog breed detection and so on.

5 Types of Classification Algorithms for Machine Learning: Classification is a technique for determining which class the dependent
belongs to based on one or more independent variables.
What Is a Classifier?
A classifier is a type of machine learning algorithm that assigns a label to a data input.
Classifier algorithms use labeled data and statistical methods to produce predictions about data input classifications.
• Classification is used for predicting discrete responses.
 1. Logistic Regression
Logistic regression is kind of like linear regression, but is used when the dependent variable is not a number but something else (e.g., a
“yes/no” response).
It’s called regression but performs classification based on the regression and it classifies the dependent variable into either of the classes.

Firstly, linear regression is performed on the relationship between variables to get the model. The threshold for the classification line is assumed to
be at 0.5.
Logistic Sigmoid Function
Logistic regression is used for prediction of output which is binary, as stated above. For example, if a credit card company builds a model to
decide whether or not to issue a credit card to a customer, it will model for whether the customer is going to “default” or “not default” on
their card.

Firstly, linear regression is performed on the relationship between variables to get the model. The threshold for the classification line is assumed
to be at 0.5.

Logistic function is applied to the regression to get the probabilities of it belonging in either class.
It gives the log of the probability of the event occurring to the log of the probability of it not occurring. In the end, it classifies the variable based
on the higher probability of either class.

Logistic Regression works by calculating probability using sigmoid function and converting it into class
labels. Work flow:
Input Features
↓
Linear Combination
↓
Sigmoid Function
↓
Probability
↓
Threshold
↓
Class Output
Working of Logistic Regression:
Step 1: Input Data
Logistic Regression takes independent variables (features) as input.
These features are numerical.
Example:
Student dataset
1. Study Hours
2. Attendance
Step 2: Linear Combination of Inputs
Each input feature is multiplied by a weight.
All are added with a bias term.

Meaning:
•𝑥→ input values
•𝑤→ learned weights
•𝑏→ bias
Step 3: Apply Sigmoid Function
The value 𝑧is passed through the sigmoid (logistic) function.

Why Sigmoid?
•Converts any value into range 0 to 1
•Represents probability
Step 4: Probability Estimation
•Output of sigmoid is the probability of belonging to class 1.
Example:
•Probability = 0.82 → 82% chance of passing
Step 5: Apply Decision Threshold
A threshold (usually 0.5) is applied.

Probability Output Class

≥ 0.5 Class 1
< 0.5 Class 0

Example:
0.82 → Pass
0.35 → Fail
Step 6: Model Training
•Model adjusts weights using gradient descent
•Objective: minimize loss (log loss)

Step 7: Final Prediction

•After training, the model predicts classes for new unseen data.

Example:
New student → Predict Pass / Fail
Logistic Regression – Stepwise Solved Real-World Medical Problem

Problem Statement
A hospital wants to predict whether a patient has Diabetes based on:
[Link] Sugar Level (mg/dL)
[Link] Mass Index (BMI)
Output:
•1 → Diabetic
•0 → Non-Diabetic
This is a binary classification problem.

Step 1: Identify the Type of Problem

Output has two classes → Yes / No
Data is labeled
Logistic Regression is suitable.
Step 2: Define Variables
Symbol Description Step 3: Training Dataset (Sample)

(x_1) Blood Sugar Level Blood Sugar

Patient BMI ((x_2)) Diabetes ((y))
((x_1))
(x_2) BMI
1 110 22 0
(y) Diabetes Status (1/0)
2 130 24 0
3 160 28 1
4 180 30 1
5 200 32 1
Step 4: Logistic Regression Model
𝑧 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + 𝑏
Assume the trained model parameters are:
𝑤1 = 0.04
𝑤2 = 0.6
𝑏 = −15
Step 5: New Patient Data (Prediction)
Blood Sugar = 150 mg/dL
BMI = 26
Step 6: Calculate Linear Combination (z)
𝑧 = 0.04 × 150 + 0.6 × 26 − 15
Step 7: Apply Sigmoid Function

Step 8: Probability Interpretation

•Probability of Diabetes = 99.86%
Step 9: Apply Decision Threshold
Threshold = 0.5

Probability Prediction
≥ 0.5 Diabetic
< 0.5 Non-Diabetic

Since 0.9986 ≥ 0.5 The logistic regression model predicts that the patient is diabetic with very high probability
Predicted Result = DIABETIC (99.86%).
A Decision Tree is a supervised machine learning algorithm used for classification that makes decisions using a tree-like structure
of rules.
• Data is split into branches using if–else conditions
• Each internal node represents a decision
• Each leaf node represents a class label
Why It Is Supervised
Uses labeled training data
Learns rules from input–output pairs

2. Structure of a Decision Tree

Component Meaning
Root Node First split of data
Decision Node Condition-based split
Branch Outcome of decision
Leaf Node Final class (output)

Simple Tree Example

Attendance ≥ 75%?
├── Yes → PASS
└── No → FAIL
Decision tree builds classification or regression models in the form of a tree structure. It breaks down a dataset into smaller and smaller
subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision
nodes and leaf nodes. It follows Iterative Dichotomiser 3 (ID3) algorithm structure for determining the split.

Entropy and information gain are used to construct a decision tree.

Entropy: Entropy is the degree or amount of uncertainty in the randomness of elements. In other words, it is a measure of
impurity.

Intuitively, it tells us about the predictability of a certain event. Entropy calculates the homogeneity of a sample. If the sample is
completely homogeneous the entropy is zero, and if the sample is equally divided it has an entropy of one.
Information Gain
Information gain measures the relative change in entropy with respect to the independent attribute. It tries to estimate the
information contained by each attribute. Constructing a decision tree is all about finding the attribute that returns the highest
information gain (i.e., the most homogeneous branches).

Where Gain(T, X) is the information gain by applying feature X. Entropy(T) is the entropy of the entire set, while the second
term calculates the entropy after applying the feature X.
Information gain ranks attributes for filtering at a given node in the tree. The ranking is based on the highest information gain
entropy in each split.
3. Working of Decision Tree (Step-by-Step)

Step 1: Input Labeled Dataset

•Dataset contains:
• Features (inputs)
• Class labels (outputs)
Example Features
•Study Hours
•Attendance
Output
•Pass / Fail

Step 2: Select Best Feature for Splitting

Algorithm selects the feature that best separates classes
Uses measures such as:
Information Gain
Gini Index
Example
Attendance gives better separation than Study Hours
→ Chosen as root node
Step 3: Split the Dataset
•Dataset is divided into subsets based on selected feature
•Attendance ≥ 75% → Group 1
•Attendance < 75% → Group 2

Step 4: Create Decision Nodes

Each subset becomes a new node
Process repeats recursively for remaining features

Step 5: Stop Splitting

Splitting stops when:
All records belong to the same class
No features are left
Tree reaches maximum depth

Step 6: Assign Class to Leaf Nodes

Leaf node stores the final decision
•Attendance ≥ 75% AND Study Hours ≥ 4 → PASS
4. Classification Example (Real-World)
Student Result Prediction

Study Hours Attendance Result

High High Pass
Low Low Fail

Decision Tree
Attendance ≥ 75%?
├── Yes → Study Hours ≥ 4?
│ ├── Yes → PASS
│ └── No → FAIL
└── No → FAIL

7. Applications of Decision Tree (Classification)

5. Advantages of Decision Tree
•Medical diagnosis (Disease / No disease)
•Easy to understand and interpret
•Loan approval systems
•Works with numerical & categorical data
•Student performance prediction
•No need for data normalization
•Fraud detection
6. Limitations of Decision Tree
•Can overfit if tree is deep
•Sensitive to small data changes

Decision Tree is a supervised classification algorithm that predicts output by learning decision rules in a tree structure.
Decision Tree – Step-by-Step Solved Agriculture Classification Example
Problem Statement
An agriculture department wants to decide whether a farmer should APPLY FERTILIZER or NOT, based on:
Soil Moisture (High / Low)
Rainfall (Yes / No)
Crop Health (Good / Poor)
Output (Class Label):
Yes → Apply Fertilizer
No → Do Not Apply
This is a supervised classification problem.
Step 1: Identify Type of Learning Step 2: Training Dataset
Output is categorical (Yes / No) Record Soil Moisture Rainfall Crop Health Fertilizer

Data is labeled 1 High Yes Good No

Decision Tree (Classification) is suitable. 2 High No Good No

3 Low Yes Poor Yes
4 Low No Poor Yes
5 Low Yes Good Yes
6 High No Poor No
Step 3: Select Best Attribute for Root Node
The Decision Tree algorithm chooses the attribute that best separates the data using:
•Information Gain or
•Gini Index
•E,g When Soil Moisture = Low, fertilizer is mostly Yes
When Soil Moisture = High, fertilizer is mostly No
Soil Moisture gives the best split
→ Selected as Root Node

Step 4: First Split (Root Node)

Soil Moisture?
├── High
└── Low
Step 5: Create Branches
Case 1: Soil Moisture = High

Rainfall Crop Health Fertilizer

Yes Good No
No Good No
No Poor No

All outputs = No Soil Moisture = High → NO Fertilizer

Case 2: Soil Moisture = Low

Rainfall Crop Health Fertilizer

Yes Poor Yes
No Poor Yes
Yes Good Yes

All outputs = Yes

Create leaf node
Soil Moisture = Low → YES Fertilizer

Step 6: Final Decision Tree

Soil Moisture?
├── High → DO NOT APPLY FERTILIZER
└── Low → APPLY FERTILIZER

Step 7: Prediction for New Case

New Field Condition
•Soil Moisture = Low
•Rainfall = No
•Crop Health = Good
Decision Path
Soil Moisture = Low → APPLY FERTILIZER
The decision tree predicts that fertilizer should be applied when soil moisture is low.
Underfitting: Model is too simple to learn patterns from data.

Overfitting: Model learns training data too well but fails on new data.

1. Underfitting
Concept
Underfitting occurs when a model is too simple to capture the underlying pattern in the data.

Key Idea
• Model does not learn enough
• Performs poorly on both training and testing data

Causes of Underfitting
• Very simple model
• Too few features
• Insufficient training

Example (Student Performance)

Problem: Predict marks based on study hours
If we use a straight line for a clearly non-linear relationship, the model cannot fit the data well.
• Actual pattern: curved
• Model used: straight line
• → Poor prediction
Actual pattern: curved
Model used: straight line
→ Poor prediction

Result
•Training accuracy → Low
•Testing accuracy → Low
Characteristics of Underfitting:
•The model performs poorly on both
training and test datasets.

•It fails to capture the complexity of the

dataset.

•Adding more training data does not

improve performance significantly.

•It occurs when models are too simple

(e.g., using linear regression for a non-
linear dataset).
Example of Underfitting:

Consider a dataset where you want to predict house prices based on square

footage. If you apply a simple linear regression model to this data when the

relationship is actually quadratic or exponential, the model will struggle to provide

accurate predictions. The result is underfitting, where the model is too simple to

grasp the underlying trend.

2. Overfitting
Concept
Overfitting occurs when a model learns the training data too well, including noise and outliers.

Key Idea
•Model becomes too complex
•Performs very well on training data but poorly on new data

Causes of Overfitting
•Very complex model
•Too many features
•Small dataset

Example (Agriculture – Crop Yield)

Problem: Predict crop yield based on rainfall
If the model fits every single fluctuation in training data, it memorizes instead of learning.

•Training data: perfect fit

•New data: poor prediction

Result
Training accuracy → Very High
Testing accuracy → Low
Overfitting occurs when a model learns the training data too well, including its noise and irrelevant details. Instead of generalizing
patterns, the model memorizes the data, leading to poor performance on unseen test data. This happens due to high variance in the
model.
Characteristics of Overfitting:
•The model performs exceptionally well on training
data but poorly on test data.

•It captures noise and anomalies instead of general

patterns.

•The model has too many parameters and is overly

complex.

•It fails to generalize well to new datasets.

Example of Overfitting:

Imagine training a deep neural network on a dataset of handwritten digits. If the model is

excessively complex, it might memorize the exact pixel patterns of each image instead

of learning general features like stroke patterns and curves. As a result, when tested on

new handwritten digits, it may struggle to recognize them correctly.

The Bias-Variance Tradeoff
The bias-variance tradeoff explains why underfitting and overfitting occur. It
represents the relationship between a model’s complexity and its ability to
generalize well.
•High Bias (Underfitting): The model makes strong assumptions about the data
and fails to capture patterns.
•High Variance (Overfitting): The model is too sensitive to small fluctuations in
the data, leading to poor generalization.
A well-balanced model should achieve an optimal balance between bias and
variance, ensuring it captures the necessary patterns without memorizing noise.

How to Identify Overfitting and Underfitting

The best way to detect whether a model is underfitting or overfitting is by
evaluating its performance on training and test datasets.
Underfitting: Low accuracy on both training and test sets.
Overfitting: High accuracy on the training set but significantly lower accuracy on the
test set.

Graphical Representation
•Underfitting (High Bias): The model’s prediction line does not cover all data points.
•Overfitting (High Variance): The model’s prediction line fits the training data perfectly, including noise and outliers.
•Good Fit: The model’s prediction line captures the trend while avoiding excessive complexity.
 Techniques to Overcome Underfitting and Overfitting

Now that we understand the causes of underfitting and overfitting, let’s explore techniques to mitigate them.

[Link] Model Complexity – Use more complex

models, such as polynomial regression instead of
simple linear regression.

[Link] More Features – Include additional relevant input

features to improve learning.

[Link] Regularization – If regularization is too strong

(e.g., high L1/L2 penalties), it may overly simplify the
model.

[Link] for a Longer Time – Some models require more

training epochs to learn patterns effectively.
How to Overcome Overfitting

[Link] Techniques:
1. L1 Regularization (Lasso Regression): Helps reduce
complexity by setting some feature weights to zero.
2. L2 Regularization (Ridge Regression): Shrinks feature
weights without setting them to zero.
3. Elastic Net: A combination of L1 and L2 regularization.
[Link]-Validation:
1. Use k-fold cross-validation to ensure that the model
generalizes well across different data subsets.
[Link] Decision Trees:
1. In decision trees, reduce the depth to prevent over-
complexity.
[Link] in Neural Networks:
1. Randomly drop some neurons during training to prevent
over-reliance on specific patterns.
[Link] Training Data:
1. More data helps the model generalize better and avoid
learning noise.
[Link] Stopping:
1. Stop training when validation loss starts increasing to
prevent memorization.
Real-World Applications
Overfitting and underfitting are crucial concerns across various domains:

•Healthcare: Predicting diseases based on medical data requires generalizable models.

•Finance: Stock market predictions should avoid memorizing past fluctuations.
•Autonomous Vehicles: Object detection systems must generalize across diverse environments.
•Natural Language Processing: Sentiment analysis models should not memorize specific training phrases.
Achieving a balance between underfitting and overfitting is key to building robust machine

learning models. While underfitting leads to poor learning due to excessive

simplification, overfitting results in poor generalization due to unnecessary complexity.

By understanding the bias-variance tradeoff and applying techniques like regularization, cross-

validation, dropout, and pruning, we can develop models that perform well on both training and

unseen test data.

Machine learning practitioners must always strive to optimize their models to achieve a good fit,

ensuring high accuracy and reliability in real-world applications.

Support Vector Machine (SVM)

1. Definition
Support Vector Machine (SVM) is a supervised machine learning algorithm used mainly for classification, which separates data points using
an optimal hyperplane.
• It Find a line (2D) or plane (3D+) that best separates classes
• It Choose the hyperplane with maximum margin
• Here Only critical points (support vectors) decide the boundary

Term Meaning
Hyperplane Decision boundary between classes
Margin Distance between hyperplane and nearest data points
Support Vectors Data points closest to the hyperplane
Kernel Function to handle non-linear data
Working of SVM:
Step 1: Input Labeled Data
Dataset with features and class labels
Example:
Email → Spam / Not Spam
Step 2: Plot Data in Feature Space
Each data point represents a class
Classes should be separable
Step 3: Find Possible Hyperplanes
Multiple separating lines are possible
Step 4: Select Optimal Hyperplane
SVM selects the hyperplane with maximum margin
Class A |---- margin ----| Class B
Step 5: Identify Support Vectors
Closest points to hyperplane
Determine the position of the boundary
Step 6: Classification of New Data
New data point is classified based on which side of the hyperplane it falls
5. Simple Example (Real-World)
Medical Diagnosis
•Input: Blood pressure, cholesterol
•Output: Disease / No Disease
SVM finds the best boundary separating patients into two classes.

6. Handling Non-Linear Data (Kernel Trick)

If data is not linearly separable:
SVM uses kernel functions
Transforms data into higher dimension

Common Kernels
Linear
Polynomial
Radial Basis Function (RBF)
Support Vector Machine (SVM) – Numerical Example (2D Points)

Problem Statement
We want to classify data points into Class +1 and Class –1 using SVM.

Point x₁ x₂ Class
A 1 1 +1
B 2 2 +1
C 2 0 –1
D 0 0 –1

Step 1: Plot the Points (Conceptual)

•Class +1: (1,1), (2,2)
•Class –1: (2,0), (0,0)

x₂
↑
2| B(+)
1 | A(+)
0 | D(-) C(-)
+----------------→ x₁
0 1 2
Step 2: Check Linearly Separable
Yes
A straight line can separate the two classes.

Step 3: Identify Candidate Hyperplanes

Possible separating lines:
x₁ + x₂ = 1
x₁ + x₂ = 2
x₁ − x₂ = 0.5
SVM chooses the hyperplane with maximum margin.

Step 4: Identify Support Vectors

Support vectors are closest points to the boundary:
A (1,1) → Class +1
C (2,0) → Class –1
These points control the position of the hyperplane.
Step 5: Optimal Hyperplane
Assume optimal decision boundary:
𝑥1 + 𝑥2 − 2 = 0

Step 6: Margin Calculation (Conceptual)

Distance from support vectors to hyperplane is maximized
Margin = 2 / ||w||
Here:
w=[1,1]
∣∣ 𝑤 ∣∣= 12 + 12 = 2
Step 7: Classification Rule

Condition Decision
f(x) > 0 Class +1
f(x) < 0 Class –1

Step 8: Test Data Point

New Point: (1.5, 0.8)
𝑓 𝑥 = 1.5 + 0.8 − 2 = 0.3
Classified as +1

Step 9: Final Result

•Optimal hyperplane separates classes
•Support vectors define the boundary
•New points classified using sign of decision function

SVM classifies data by constructing a hyperplane that maximizes the margin using support vectors.
Type
Type Full Form Purpose / Use Real-World Example
Category

Learning Task SVC Support Vector Classification Classifies data into categories Spam vs Non-Spam emails

SVR Support Vector Regression Predicts continuous values House price prediction

Margin Type Hard Margin SVM Hard Margin Support Vector Machine Perfect separation, no error allowed Lab experiment dataset

Soft Margin SVM Soft Margin Support Vector Machine Allows some misclassification Medical diagnosis data

Kernel Type Linear SVM Linear Support Vector Machine Linearly separable data Text classification

Polynomial SVM Polynomial Kernel Support Vector Machine Non-linear curved boundary Image pattern recognition

RBF SVM Radial Basis Function Support Vector Machine Complex non-linear data Face recognition

Sigmoid SVM Sigmoid Kernel Support Vector Machine Neural-network-like behavior Bioinformatics

Class Type Binary SVM Binary Support Vector Machine Two-class classification Pass / Fail

Multi-Class SVM Multi-Class Support Vector Machine More than two classes Handwritten digits (0–9)

SVM = Task (SVC/SVR) + Margin (Hard/Soft) + Kernel (Linear/RBF) + Classes (Binary/Multi)

SVC: Used when output is categorical, SVR: Used when output is continuous,
RBF: Most popular kernel for real-world data, Soft Margin: Best choice for noisy datasets
Support Vector Machine (SVM) Using Kernel Trick
(Non-Linear Classification)

1. Why Kernel is Required?

Sometimes data cannot be separated by a straight line in 2D space.
Example
•Disease vs No Disease
•XOR data
•Image / pattern recognition
Linear SVM fails → Kernel SVM required

2. Concept of Kernel Trick

Kernel transforms data into a higher-dimensional space where it becomes linearly separable.

3. Common Kernel Functions

Kernel Formula Use

Linear K(x,y)=x·y Linearly separable
Polynomial (x·y + c)ᵈ Curved boundary
RBF (Gaussian) exp(−γ
Sigmoid tanh(x·y + c) Neural-like
 4. Numerical Example (2D → 3D Mapping)
Given XOR Dataset

Point x₁ x₂ Class
A 0 0 –1
B 1 1 –1
C 1 0 +1
D 0 1 +1

It is Not linearly separable in 2D

5. Apply Polynomial Kernel (Degree = 2)

We introduce a new feature:

6. Transformed Dataset (3D)

x₁ x₂ z=x₁x₂ Class
0 0 0 –1
1 1 1 –1
1 0 0 +1
0 1 0 +1

Now separable by a plane

7. New Hyperplane in 3D 11. Advantages of Kernel SVM
Assume hyperplane: •Handles non-linear data
•High accuracy
Condition Class
•Flexible decision boundary
z > 0.5 –1
z < 0.5 +1
12. Limitations
•Computationally expensive
8. Test New Data Point •Kernel selection is difficult
Input: (1, 1) •Hard to interpret
𝑧 =1×1=1
f(x)=1−0.5=0.5
13. Real-World Applications
Classified as –1 •Face recognition
•Medical image classification
Here •Fraud detection
• No need to compute features explicitly •Bioinformatics
• Kernel computes dot product in higher dimension
• Faster and efficient

RBF Kernel – Intuition (No Math)

•Creates circular boundaries
•Measures distance between points
•Handles complex patterns

Kernel SVM maps data into higher-dimensional space to achieve linear separability.
Regression:
Regression is a supervised machine learning technique used to predict a continuous numerical value based on one or more input
features.
Some examples of regression include house price prediction, stock price prediction, height-weight prediction and so on.

Aspect Description
Input Labeled data
Output Continuous value
Find relationship
Goal
between variables
Price, marks,
Examples
temperature
General Regression Model 3 Polynomial Regression
𝑦 =𝑓 𝑥 +𝜀 𝑦 = 𝑎𝑥 2 + 𝑏𝑥 + 𝑐
Example:
Where: Growth rate vs time (curved relationship)
y → dependent (target) variable
x → independent (input) variable 4 Ridge Regression
ε → error term Uses L2 regularization
Reduces overfitting
How Regression Works (Step-by-Step) Example:
Step 1: Collect labeled data High-dimensional financial data
Step 2: Select regression model
Step 3: Learn best-fit function 5 Lasso Regression
Step 4: Minimize error (loss function) Uses L1 regularization
Step 5: Predict output for new input Feature selection
Example:
Medical datasets with many features
Types of Regression (Supervised ML)
1 Linear Regression 6 Elastic Net Regression
Equation: Combination of Ridge + Lasso
𝑦 = 𝑚𝑥 + 𝑐 Example:
Example: Genomics and bioinformatics
Predict salary based on years of experience

2 Multiple Linear Regression

𝑦 = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥2
Example:
House price based on area & location
Different Loss Functions Used:

Loss Use
Mean Squared Error (MSE) Penalizes large errors
Mean Absolute Error (MAE) Robust to outliers
RMSE Root of MSE
Advantages of Regression
Regression vs Classification •Easy to interpret
•Strong mathematical foundation
Feature Regression Classification •Works well with small datasets

Output Continuous Categorical

Example Price prediction Spam detection 10. Limitations
•Sensitive to outliers
Algorithms Linear, Ridge Logistic, SVM
•Assumes linearity (for linear models)
•Overfitting if poorly regularized
Real-World Regression Examples
[Link] price prediction
[Link] price forecasting
[Link] yield estimation
[Link] temperature prediction
[Link] consumption forecasting
 Nature of
Regression Type Output Type Key Idea / Feature Handles Overfitting? Typical Example
Relationship
Simple Linear
Continuous Linear One input variable No Salary vs Experience
Regression
Multiple Linear
Continuous Linear Multiple inputs No House price prediction
Regression
Polynomial Crop growth vs
Continuous Non-linear Uses polynomial terms No
Regression fertilizer

Ridge Regression Continuous Linear / Non-linear L2 regularization Yes Financial prediction

L1 regularization
Lasso Regression Continuous Linear / Non-linear Yes Medical data analysis
(feature selection)

Elastic Net Regression Continuous Linear / Non-linear Ridge + Lasso Yes Genomics data

Logistic Regression Categorical Non-linear (Sigmoid) Probability-based Yes Disease detection

Support Vector Energy load

Continuous Non-linear ε-insensitive margin Yes
Regression (SVR) forecasting

Decision Tree
Continuous Non-linear Rule-based splitting Partial Rainfall prediction
Regression
Random Forest
Continuous Non-linear Ensemble of trees Yes Stock price prediction
Regression
Uses probability
Bayesian Regression Continuous Probabilistic Yes Risk analysis
distributions
 Simple Linear Regression – Numerical Problem
Problem Statement
Predict salary (Y) based on experience (X).

X (Years) Y (Salary in LPA)

1 2
2 3
3 5
4 7
5 8

Step 1: Formula

Step 2: Compute Values

෍ 𝑋 = 15, ෍ 𝑌 = 25
Step 3: Calculate Slope (m)

Step 4: Calculate Intercept (c)

Final Regression Equation

Prediction

For X = 6 years:
𝑦 = 1.9 6 − 0.7 = 10.7 LPA

Support Vector Regression (SVR) – Conceptual Numerical

Idea
•Predicts value within ε-margin
•Errors inside margin → ignored
Given:
•ε = 0.5
•Predicted = 10
•Actual = 10.3
Error ignored (inside margin)
Multiple Linear Regression – Numerical Concept Ridge Regression – Numerical Concept
Model Equation Loss Function
𝑦 = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥2
𝐿𝑜𝑠𝑠 = 𝑀𝑆𝐸 + 𝜆 ෍ 𝑤 2
Example
Predict house price using:
Given
•x₁ = Area
•MSE = 20
•x₂ = Number of rooms
•λ = 0.5
Given:
•Weights = [2, 1]
𝑦 = 2 + 0.5𝑥1 + 1.2𝑥2
𝑃𝑒𝑛𝑎𝑙𝑡𝑦 = 0.5 22 + 12 = 2.5
For:
𝑇𝑜𝑡𝑎𝑙𝐿𝑜𝑠𝑠 = 20 + 2.5 = 22.5
•x₁ = 1000 [Link]
Reduces overfitting
•x₂ = 3 rooms
𝑦 = 2 + 0.5 1000 + 1.2 3 = 505.6 (₹ Lakhs)

Lasso Regression – Numerical Concept

Polynomial Regression – Numerical Example
Loss Function
Model
𝑦 = 𝑎𝑥 2 + 𝑏𝑥 + 𝑐 𝐿𝑜𝑠𝑠 = 𝑀𝑆𝐸 + 𝜆 ෍ ∣ 𝑤 ∣
Given:
𝑦 = 𝑥 2 + 2𝑥 + 1 Given:
For x = 3: •Weights = [2, 0, 1]
𝑦 = 9 + 6 + 1 = 16 𝑃𝑒𝑛𝑎𝑙𝑡𝑦 =∣ 2 ∣+∣ 0 ∣+∣ 1 ∣= 3
Used when data shows curved relationship Feature selection (zero coefficient)
Aspect Classification Regression

Output Categorical Continuous

Example Pass/Fail Marks

Algorithms Logistic, SVM Linear, SVR

Use case Decision making Prediction

Classification → Class labels

Regression → Real values

Unsupervised Learning: Unsupervised learning works on unlabeled data and finds hidden patterns.
⮚ Unsupervised learning is a learning method in which a machine learns without any supervision.
⮚ The training is provided to the machine with the set of data that has not been labeled, classified, or categorized, and the
algorithm needs to act on that data without any supervision.
⮚ The goal of unsupervised learning is to restructure the input data into new features or a group of objects with similar patterns.
⮚ In unsupervised learning, we don't have a predetermined result. The machine tries to find useful insights from the huge
amount of data. It can be further classifieds into two categories of algorithms:
• Clustering
• Association
Unsupervised Learning: Unsupervised learning works on unlabeled data and finds hidden patterns.
Unsupervised Learning Examples:

1. Customer Segmentation 6. Gene Expression Analysis

•Group customers based on buying behavior •Group genes with similar expression patterns
•Used in marketing strategies •Used in bioinformatics

2. Market Basket Analysis 7. Image Segmentation

•Find items frequently bought together •Divide an image into meaningful regions
•Example: Bread → Butter •Used in medical imaging

3. Student Performance Grouping 8. Fraud Pattern Detection

Group students as slow, average, fast Identify unusual transaction behavior
learners No labeled fraud data required
No predefined labels

9. Customer Churn Pattern Discovery

4. Social Media User Clustering Discover hidden reasons for customer loss
Group users based on interests and activity Used in telecom sector
Used in content recommendation

10. Website User Behavior Analysis

5. Document Clustering Group users based on navigation patterns
Group news articles by topic Used for UI/UX improvement
Used in search engines
Unsupervised Learning is a type of machine learning where the model learns patterns from unlabeled data without any predefined output.

Feature Description
Data Type Unlabeled
Output Groups / patterns / associations
Human intervention Minimal
Goal Discover hidden structure

How Unsupervised Learning Works Types of Unsupervised Learning

A. Clustering
Step 1: Input unlabeled dataset
Groups similar data points together.
Step 2: Identify patterns or similarity B. Association Rule Learning
Finds relationships between variables.
Step 3: Group or associate data
C. Dimensionality Reduction
Step 4: Interpret results
Reduces number of features.
Unsupervised Learning – Types, Uses & Examples

Type Algorithm / Technique What it Does (Use) Real-World Example

Groups similar data points into
K-Means Clustering Customer segmentation
K clusters
Clustering Creates tree-like clusters
Hierarchical Clustering Gene classification
(dendrogram)
DBSCAN Finds dense regions and outliers Fraud detection
Finds frequent itemsets and
Apriori Algorithm Market basket analysis
Association Rule rules
Learning E-commerce
FP-Growth Faster frequent pattern mining
recommendations
PCA (Principal Component
Reduces number of features Image compression
Dimensionality Analysis)
Reduction LDA (Linear Discriminant
Maximizes class separability Face recognition
Analysis)*
Isolation Forest Detects rare and unusual data Network intrusion
Anomaly Detection Manufacturing defect
One-Class SVM Learns normal pattern only
detection
Latent Dirichlet Allocation
Topic Modeling Extracts topics from text News article categorization
(LDA)
K-Means Clustering (Unsupervised Learning)
1. Definition
K-Means Clustering is an unsupervised machine learning algorithm that groups data into K clusters, where each data point belongs
to the cluster with the nearest mean (centroid).

2. Objective of K-Means
Minimize within-cluster variance

Where:
•𝐶𝑖 =ith cluster
•𝜇𝑖 =centroid of ith cluster

Feature Description
Learning Type Unsupervised
Input Data Unlabeled
Distance Measure Euclidean (commonly)
Output K clusters
How K-Means Works (Algorithm Steps)

Step 1: Choose number of clusters K Flowchart

Start → Choose K → Initialize Centroids
Step 2: Initialize K centroids randomly
↓
Step 3: Assign each data point to the nearest centroid Assign Points → Update Centroids
↓
Centroids Change?
↓Yes ↓No
Step 4: Recalculate centroids of clusters Repeat Stop

Step 5: Repeat steps 3 & 4 until centroids do not change

Numerical Example
Dataset (2D points)Point

Point X Y
P1 1 1
P2 1.5 2
P3 5 8
P4 6 9
P5 1 0.5

Step 1: Choose K = 2
Step 2: Initialize Centroids
•C1 = P1 (1,1)
•C2 = P3 (5,8)

•Step 3: Distance Calculation

Point Dist to C1 Dist to C2 Assigned Cluster
P1 0 8.06 C1
P2 1.12 7.21 C1
P3 8.06 0 C2
P4 9.43 1.41 C2
P5 0.5 8.51 C1
Step 4: Recalculate Centroids Advantages of K-Means
•Simple and fast
•Easy to implement
•Scales well for large datasets

Limitations
•Must predefine K
•Sensitive to initial centroids
•Struggles with non-spherical clusters
•Sensitive to outliers
Step 5: Repeat Assignment
No change in cluster → Stop
Applications of K-Means
Final clusters formed
[Link] segmentation
[Link] compression
Choosing Optimal K (Elbow Method)
[Link] clustering
•Plot K vs WCSS
[Link] analysis
•Point where curve bends → optimal K
[Link] data grouping

K-Means is distance-based—bad centroids give bad clusters.

Hierarchical Clustering (Unsupervised Learning)
1. Definition
Hierarchical Clustering is an unsupervised learning technique that builds a hierarchy of clusters either by merging smaller clusters or
splitting larger clusters, represented using a dendrogram.

Feature Description
Data Type Unlabeled
Cluster Structure Tree-like (Hierarchy)
Output Dendrogram
K Required No (optional later)

3. Types of Hierarchical Clustering

3.1 Agglomerative Clustering (Bottom-Up)

•Each data point starts as a single cluster
•Closest clusters are merged step by step
•Most commonly used

3.2 Divisive Clustering (Top-Down)

•All data points start in one cluster
•Cluster is split recursively
•Computationally expensive

4. Distance Measures Used

•Euclidean distance
•Manhattan distance
•Cosine similarity
5. Linkage Methods

Linkage Distance Between Clusters

Single Linkage Minimum distance
Complete Linkage Maximum distance
Average Linkage Average distance
Ward’s Method Minimizes variance

Working of Agglomerative Hierarchical Clustering

Step-by-Step Algorithm

[Link] each data point as a separate cluster

[Link] distance matrix

[Link] two closest clusters

[Link] distance matrix

[Link] until one cluster remains

Numerical Example Step 4: Next Closest
•Merge C & AB → Cluster ABC
Point Value
A 2
Step 5: Final Merge
B 4 •Merge ABC & D
C 6 Hierarchy completed

D 10

Step 1: Distance Matrix 8. Dendrogram Explanation

•X-axis → Data points
A B C D •Y-axis → Distance
A 0 2 4 8 •Cutting dendrogram at a level gives clusters
B 2 0 2 6
C 4 2 0 4 9. Choosing Number of Clusters
D 8 6 4 0 •Cut dendrogram horizontally
•Number of vertical lines cut = number of clusters

Step 2: Merge Closest Points

•Merge A & B → Cluster AB

Step 3: Update Clusters

Clusters: AB, C, D
 Advantages
No need to specify K initially
Produces meaningful hierarchy
Easy to visualize

11. Limitations
Computationally expensive
Not suitable for large datasets
Once merged/split, cannot undo

12. Applications
Gene expression analysis
Document clustering
Customer segmentation
Image analysis
Social network analysi

Hierarchical clustering builds a tree-like structure of clusters using agglomerative or divisive methods.
 Association Rules (Unsupervised Learning)

1. Definition
Association Rule Mining is an unsupervised learning technique used to discover interesting relationships (rules) between variables in
large datasets.

2. Typical Form of Association Rule

IF 𝑋 ⇒ THEN 𝑌
Example:
IF customer buys Bread → THEN buys Butter

Term Meaning
Item Individual product
Itemset Collection of items
Transaction Set of items purchased
Relationship between
Rule
itemsets

4. Measures of Association Rules (VERY IMPORTANT)

4.1 Support
4.2 Confidence

4.3 Lift

If Lift value 1 | Positive association |

=1 | Independent |
<1 | Negative association |

5. Popular Algorithms

Algorithm Description Example dataset

In a supermarket dataset, Apriori first

Uses candidate generation and pruning based on
finds frequent 1-itemsets (Bread, Milk),
Apriori minimum support Small datasets
then 2-itemsets (Bread–Milk), and
: Generates and tests itemset candidates level by level
generates rules like Bread → Milk

Uses FP-Tree structure, avoids candidate generation In online shopping data, FP-Growth
(faster) builds an FP-Tree and directly finds
FP-Growth Large datasets
:Compresses data using FP-Tree and mines patterns frequent patterns like Laptop → Mouse →
efficiently Keyboard

In a transaction log, items are stored with

Uses vertical data format (TID sets)
transaction IDs, e.g., Milk = {T1, T3, T4},
Eclat : Finds frequent itemsets using transaction ID Sparse datasets
and frequent itemsets are found by
intersections
intersecting TID sets

Apriori thinks a lot, FP-Growth remembers patterns, Eclat intersects IDs

Numerical Example
Transaction Dataset
Transaction Items
T1 Bread, Milk
T2 Bread, Diaper, Beer
T3 Milk, Diaper, Beer
T4 Bread, Milk, Diaper, Beer
T5 Bread, Milk

Total transactions = 5
Step 1: Calculate Support

Step 2: Generate Association Rule

𝐵𝑟𝑒𝑎𝑑 ⇒ 𝑀𝑖𝑙𝑘

Step 3: Calculate Confidence

Step 4: Calculate Lift

Slight negative association

Interpretation of Rule
•75% of customers who buy Bread also buy Milk
•Lift < 1 → Weak association

Association rule mining discovers relationships among items using support, confidence, and lift.

Advantages
•Simple and intuitive
•Useful for large transactional data
•Helps in decision making

Limitations
•Generates too many rules
•Computationally expensive
•Needs threshold tuning

Applications
[Link] basket analysis
[Link] systems
[Link] usage mining
[Link] diagnosis
[Link] detection
Supervised Learning Regression
Reinforcement Learning

⮚ Reinforcement learning is a feedback-based learning method, in which a learning agent gets a reward for each right action
and gets a penalty for each wrong action.
⮚ The agent learns automatically with these feedbacks and improves its performance. In reinforcement learning, the agent
interacts with the environment and explores it.
⮚ The goal of an agent is to get the most reward points, and hence, it improves its performance.
⮚ The robotic dog, which automatically learns the movement of his arms, is an example of Reinforcement learning.
Notes By Archana S (VPPCOE& VA)
 Machine Machine Learning at present:
Learning at Now machine learning has got a great advancement in its research, and it is
present: present everywhere around us, such as self-driving cars, Amazon
Alexa, Catboats, recommender system, and many more.
It includes Supervised, unsupervised, and reinforcement learning with
clustering, classification, decision tree, SVM algorithms, etc.
Modern machine learning models can be used for making various predictions,
including weather prediction, disease prediction, stock market analysis, etc.

Prerequisites
Before learning machine learning, you must have the basic
knowledge of followings so that you can easily understand the
concepts of machine learning:

⮚ Fundamental knowledge of probability and linear algebra.

⮚ The ability to code in any computer language, especially in
Python language.
⮚ Knowledge of Calculus, especially derivatives of single variable
and multivariate functions.
Notes By Archana S (VPPCOE& VA)
 ⮚Application of Machine Learning

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

1. Image Recognition:
Image recognition is one of the most common applications of
Application of machine learning. It is used to identify objects, persons,
Machine places, digital images, etc. The popular use case of image
Learning
recognition and face detection is, Automatic friend tagging
suggestion:
Facebook provides us a feature of auto friend tagging
suggestion. Whenever we upload a photo with our Facebook
friends, then we automatically get a tagging suggestion with
name, and the technology behind this is machine
learning's face detection and recognition algorithm.
It is based on the Facebook project named "Deep Face," which
is responsible for face recognition and person identification in
the picture.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

2. Speech Recognition
While using Google, we get an option of "Search by voice," it
Application of
Machine comes under speech recognition, and it's a popular application
Learning of machine learning.
Speech recognition is a process of converting voice
instructions into text, and it is also known as "Speech to text",
or "Computer speech recognition." At present, machine
learning algorithms are widely used by various applications of
speech recognition. Google assistant, Siri, Cortana,
and Alexa are using speech recognition technology to follow
the voice instructions.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

3. Traffic prediction:
If we want to visit a new place, we take help of Google Maps,
Application of which shows us the correct path with the shortest route and
Machine
predicts the traffic conditions.
Learning
It predicts the traffic conditions such as whether traffic is
cleared, slow-moving, or heavily congested with the help of two
ways:
• Real Time location of the vehicle form Google Map app and
sensors
• Average time has taken on past days at the same time.
Everyone who is using Google Map is helping this app to make it
better. It takes information from the user and sends back to its
database to improve the performance.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

4. Product recommendations:
Machine learning is widely used by various e-commerce and
Application of entertainment companies such as Amazon, Netflix, etc., for
Machine product recommendation to the user. Whenever we search for
Learning some product on Amazon, then we started getting an
advertisement for the same product while internet surfing on the
same browser and this is because of machine learning.
Google understands the user interest using various machine
learning algorithms and suggests the product as per customer
interest.
As similar, when we use Netflix, we find some recommendations
for entertainment series, movies, etc., and this is also done with
the help of machine learning.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

5. Self-driving cars:
Application of One of the most exciting applications of machine
Machine
Learning
learning is self-driving cars. Machine learning plays
a significant role in self-driving cars. Tesla, the
most popular car manufacturing company is
working on self-driving car. It is using unsupervised
learning method to train the car models to detect
people and objects while driving.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

6. Email Spam and Malware Filtering:

Whenever we receive a new email, it is filtered automatically as
Application of important, normal, and spam. We always receive an important mail in our
Machine inbox with the important symbol and spam emails in our spam box, and
Learning the technology behind this is Machine learning. Below are some spam
filters used by Gmail:
• Content Filter
• Header filter
• General blacklists filter
• Rules-based filters
• Permission filters
Some machine learning algorithms such as Multi-Layer
Perceptron, Decision tree, and Naïve Bayes classifier are used for
email spam filtering and malware detection.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

7. Virtual Personal Assistant:

We have various virtual personal assistants such as Google
Application of assistant, Alexa, Cortana, Siri. As the name suggests, they
Machine
help us in finding the information using our voice instruction.
Learning
These assistants can help us in various ways just by our voice
instructions such as Play music, call someone, Open an email,
Scheduling an appointment, etc.
These virtual assistants use machine learning algorithms as an
important part.
These assistant record our voice instructions, send it over the
server on a cloud, and decode it using ML algorithms and act
accordingly.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

8. Online Fraud Detection:

Machine learning is making our online transaction safe and
Application of secure by detecting fraud transaction. Whenever we perform
Machine some online transaction, there may be various ways that a
Learning fraudulent transaction can take place such as fake
accounts, fake ids, and steal money in the middle of a
transaction. So to detect this, Feed Forward Neural
network helps us by checking whether it is a genuine
transaction or a fraud transaction.
For each genuine transaction, the output is converted into
some hash values, and these values become the input for the
next round. For each genuine transaction, there is a specific
pattern which gets change for the fraud transaction hence, it
detects it and makes our online transactions more secure.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

9. Stock Market trading:

Machine learning is widely used in stock market trading. In
Application of the stock market, there is always a risk of up and downs in
Machine shares, so for this machine learning's long short term
Learning
memory neural network is used for the prediction of stock
market trends.
10. Medical Diagnosis:
In medical science, machine learning is used for diseases
diagnoses. With this, medical technology is growing very fast
and able to build 3D models that can predict the exact
position of lesions in the brain.
It helps in finding brain tumors and other brain-related
diseases easily.

Notes By Archana S (VPPCOE& VA)

 ⮚ Application of Machine Learning

11. Automatic Language Translation:

Nowadays, if we visit a new place and we are not aware of
Application of the language then it is not a problem at all, as for this also
Machine machine learning helps us by converting the text into our
Learning
known languages. Google's GNMT (Google Neural Machine
Translation) provide this feature, which is a Neural Machine
Learning that translates the text into our familiar language,
and it called as automatic translation.
The technology behind the automatic translation is a
sequence to sequence learning algorithm, which is used with
image recognition and translates the text from one language
to another language.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning
1. Inadequate Training Data
The major issue that comes while using machine learning algorithms is the lack of
quality as well as quantity of data. Although data plays a vital role in the processing of
issues in machine learning algorithms, many data scientists claim that inadequate data, noisy
Machine data, and unclean data are extremely exhausting the machine learning algorithms. For
Learning example, a simple task requires thousands of sample data, and an advanced task
such as speech or image recognition needs millions of sample data examples. Further,
data quality is also important for the algorithms to work ideally, but the absence of
data quality is also found in Machine Learning applications. Data quality can be
affected by some factors as follows:
• Noisy Data- It is responsible for an inaccurate prediction that affects the decision as
well as accuracy in classification tasks.
• Incorrect data- It is also responsible for faulty programming and results obtained in
machine learning models. Hence, incorrect data may affect the accuracy of the
results also.
• Generalizing of output data- Sometimes, it is also found that generalizing output
data becomes complex, which results in comparatively poor future actions.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

2. Poor quality of data

issues in As we have discussed above, data plays a significant role in machine
Machine learning, and it must be of good quality as well. Noisy data, incomplete data,
Learning inaccurate data, and unclean data lead to less accuracy in classification and
low-quality results. Hence, data quality can also be considered as a major
common problem while processing machine learning algorithms.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

3. Non-representative training data

issues in To make sure our training model is generalized well or not, we have to ensure
that sample training data must be representative of new cases that we need
Machine
to generalize. The training data must cover all cases that are already
Learning occurred as well as occurring.
Further, if we are using non-representative training data in the model, it
results in less accurate predictions. A machine learning model is said to be
ideal if it predicts well for generalized cases and provides accurate decisions.
If there is less training data, then there will be a sampling noise in the model,
called the non-representative training set. It won't be accurate in predictions.
To overcome this, it will be biased against one class or a group.
Hence, we should use representative data in training to protect against being
biased and make accurate predictions without any drift.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

4. Overfitting and Underfitting

issues in Overfitting:
Machine Overfitting is one of the most common issues faced by Machine Learning
Learning engineers and data scientists. Whenever a machine learning model is trained
with a huge amount of data, it starts capturing noise and inaccurate data into
the training data set. It negatively affects the performance of the model.
Let's understand with a simple example where we have a few training data
sets such as 1000 mangoes, 1000 apples, 1000 bananas, and 5000 papayas.
Then there is a considerable probability of identification of an apple as papaya
because we have a massive amount of biased data in the training data set;
hence prediction got negatively affected. The main reason behind overfitting
is using non-linear methods used in machine learning algorithms as they build
non-realistic data models. We can overcome overfitting by using linear and
parametric algorithms in the machine learning models.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

4. Overfitting and Underfitting

issues in Methods to reduce overfitting:
Machine • Increase training data in a dataset.
Learning • Reduce model complexity by simplifying the model by selecting
one with fewer parameters
• Ridge Regularization and Lasso Regularization
• Early stopping during the training phase
• Reduce the noise
• Reduce the number of attributes in training data.
• Constraining the model.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

Underfitting:
issues in Underfitting is just the opposite of overfitting. Whenever a machine
Machine learning model is trained with fewer amounts of data, and as a result, it
provides incomplete and inaccurate data and destroys the accuracy of the
Learning machine learning model.
Underfitting occurs when our model is too simple to understand the base
structure of the data, just like an undersized pant. This generally happens
when we have limited data into the data set, and we try to build a linear
model with non-linear data. In such scenarios, the complexity of the model
destroys, and rules of the machine learning model become too easy to be
applied on this data set, and the model starts doing wrong predictions as
well.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

Methods to reduce Underfitting:

issues in •Increase model complexity
Machine
•Remove noise from the data
•Trained on increased and better features
Learning •Reduce the constraints
•Increase the number of epochs to get better results.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

5. Monitoring and maintenance

issues in As we know that generalized output data is mandatory for any machine
Machine learning model; hence, regular monitoring and maintenance become
compulsory for the same. Different results for different actions require
Learning
data change; hence editing of codes as well as resources for monitoring
them also become necessary.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

6. Getting bad recommendations

issues in A machine learning model operates under a specific context which results
in bad recommendations and concept drift in the model. Let's understand
Machine
with an example where at a specific time customer is looking for some
Learning gadgets, but now customer requirement changed over time but still
machine learning model showing same recommendations to the customer
while customer expectation has been changed. This incident is called a
Data Drift. It generally occurs when new data is introduced or
interpretation of data changes. However, we can overcome this by
regularly updating and monitoring data according to the expectations.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

7. Lack of skilled resources

issues in Although Machine Learning and Artificial Intelligence are
Machine continuously growing in the market, still these industries are
Learning fresher in comparison to others. The absence of skilled
resources in the form of manpower is also an issue. Hence,
we need manpower having in-depth knowledge of
mathematics, science, and technologies for developing and
managing scientific substances for machine learning.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

issues in 8. Customer Segmentation

Machine Customer segmentation is also an important issue while
Learning developing a machine learning algorithm. To identify the
customers who paid for the recommendations shown by
the model and who don't even check them. Hence, an
algorithm is necessary to recognize the customer behavior
and trigger a relevant recommendation for the user based
on past experience.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

issues in 9. Process Complexity of Machine Learning

Machine The machine learning process is very complex, which is
Learning also another major issue faced by machine learning
engineers and data scientists. However, Machine Learning
and Artificial Intelligence are very new technologies but are
still in an experimental phase and continuously being
changing over time. There is the majority of hits and trial
experiments; hence the probability of error is higher than
expected. Further, it also includes analyzing the data,
removing data bias, training data, applying complex
mathematical calculations, etc., making the procedure
more complicated and quite tedious.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

issues in 10. Data Bias

Machine Data Biasing is also found a big challenge in
Learning Machine Learning. These errors exist when certain
elements of the dataset are heavily weighted or
need more importance than others. Biased data
leads to inaccurate results, skewed outcomes, and
other analytical errors. However, we can resolve
this error by determining where data is actually
biased in the dataset. Further, take necessary
steps to reduce it.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

issues in Methods to remove Data Bias:

Machine • Research more for customer segmentation.
Learning • Be aware of your general use cases and potential outliers.
• Combine inputs from multiple sources to ensure data
diversity.
• Include bias testing in the development process.
• Analyze data regularly and keep tracking errors to resolve
them easily.
• Review the collected and annotated data.
• Use multi-pass annotation such as sentiment analysis,
content moderation, and intent recognition.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

issues in
11. Lack of Explainability
Machine This basically means the outputs cannot be easily
Learning comprehended as it is programmed in specific ways
to deliver for certain conditions. Hence, a lack of
explainability is also found in machine learning
algorithms which reduce the credibility of the
algorithms.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

issues in
12. Slow implementations and results
Machine This issue is also very commonly seen in machine
Learning learning models. However, machine learning models
are highly efficient in producing accurate results but
are time-consuming. Slow programming, excessive
requirements' and overloaded data take more time to
provide accurate results than expected. This needs
continuous maintenance and monitoring of the model
for delivering accurate results.

Notes By Archana S (VPPCOE& VA)

 issues in Machine Learning

issues in
13. Irrelevant features
Machine Although machine learning models are intended to
Learning give the best possible outcome, if we feed garbage
data as input, then the result will also be garbage.
Hence, we should use relevant features in our training
sample. A machine learning model is said to be good
if training data has a good set of features or less to no
irrelevant features.

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.

Steps in developing a 1. Understand the problem

Machine Learning
Application 2. Collect and Process the data
3. Split the data
4. Choose appropriate model
5. Train the model
6. Evaluate the model
7. Hyperparameter Tuning
8. Prediction

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.

1 . Understand the problem

Steps in Before starting to build any machine learning model, first try to analyze and understand the
developing a purpose for which you are building the model. It would help to choose the appropriate algorithm
for our model and also gives better results. If you understand the problem clearly, you can able to
Machine list some potential solutions to test in order to generate the best model. Understand that you have
Learning to try out a few solutions before you land on a good working model.
To understand the steps more clearly, let us consider the example of identifying fruits by their
Application. color, shape, and size. This basic example is to understand the process in a simple way. For our
model, we have different parameters to classify a fruit. We can add more features to get better
results. For the sake of simplicity, we have taken three different parameters to identify the fruit.
The first feature is the color of the fruit, the second one is the shape of the fruit and the last one is
the size of the fruit. Using these features our model will identify the name of the fruit.
It is good to have basic knowledge of the field in which you are developing the model. For
example, if you are developing a model for credit card fraud detection, you should learn to
understand how the industry operates and analyze the problem completely to build a better model.

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.

2. Data Collection and Data Preprocessing

Steps in The collection of data is the foundation of the Machine Learning process. The better the
developing a collection of data, the better will be the model. Choosing incorrect features or Choosing
limited features for the dataset may reduce the efficiency of the model. So it is very
Machine important to concentrate on the dataset before starting to build the model. If you want to
Learning build a model with a sample dataset, there are a lot of datasets already available
Application. at [Link] So, you can use the dataset required for your model from
there.

The dataset used in the Machine Learning model is simply an M x N matrix, where M is the columns (features) and
N is the rows (samples). It can be further broken into X (independent variables) and Y (dependent variables).

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.
2. Data Collection and Data Preprocessing
Exploratory Data Analysis
Steps in Exploratory Data Analysis (EDA) is done in order to gain an understanding of the dataset.
developing a Some of the common approaches for EDA include:
Machine • Data visualization – Scatter plot, heat map, box plot, etc.
• Descriptive statistics – Mean, median, Standard deviation, etc.
Learning Once we have analyzed data with suitable features, the next step is to preprocess the
Application. data for further steps.
The quality of data will have a huge impact on the quality of the model. Data preprocessing is
a technique to convert the data collected into a clean dataset. It is one of the vital and
important steps in building a machine-learning model. It is also known as data wrangling or
data cleaning.
Some datasets contain missing values, duplicate values, or incorrect data. In those cases,
remove a specific row that has a null value for a feature or a particular column where the
values are missing. Or calculate the mean of a particular row that contains a missing value and
replace the result for the missing value. Most datasets have a large number of features. Which
increases planes, it is difficult to model and visualize. The volume of data is reduced by
methods like Principal Component Analysis (PCA) and SVD.
It is mandatory to detect and remove the above issues because it can negatively affect
the quality of the outcome.

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.
2. Data Collection and Data Preprocessing
Feature Scaling
Steps in It is the final step in the data preprocessing phase. The learning algorithms perform much
developing a better when the features are on the same scale. A well-prepared data always gives better
Machine results. So It is important to fine-tune the data at each and every step of building the
model. The most common techniques used in feature scaling are:
Learning • Normalization – It is a method to rescale the features of the data within a specific range
Application. mostly [0,1]. To normalize the data min-max scaling method is applied to each feature
column.

X changed = ( X – Xmin )/ ( Xmax – Xmin )

• Standardization – It centers the feature columns at mean 0 and standard deviation 1 so that
the feature columns have the same scale. It keeps the information about outliers and makes
the model less sensitive to them.

Z = ( X-μ )
where, μ – Mean, σ – Standard deviation

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.

Steps in [Link] the data

The next important step is to explore the dataset and divide the dataset into training and
developing a testing data. The dataset for the Machine Learning model must be split into two separate
Machine sets – training and test set.
Learning The training data denotes the subset of a dataset used for training the machine learning
model. Here we already know the output. The testing data is the subset of the dataset,
Application. used for testing the machine learning model.
The Machine Learning model predicts the outcomes of the test dataset. The breaking of
data should be 80:20 or 70:30 ratio approximately. The larger part is for training
purposes and the smaller part is for testing purposes.
This is more important because using the same data for training and testing would not
produce good results.
One more common approach is to split the data into 3 portions training data,
validation data, and testing data.
As explained before, the training set is used to train the model,
the validation set is used for evaluation where model tuning (like hyperparameter tuning) is
done.
The testing set can be used to further test the model.

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.

[Link] the data

Steps in Validation In this method, we perform training on the 50% of the given data-set and
developing a rest 50% is used for the testing purpose. The major drawback of this method is that
we perform training on the 50% of the dataset, it may possible that the remaining
Machine 50% of the data contains some important information which we are leaving while
Learning training our model i.e higher bias. LOOCV (Leave One Out Cross Validation) In
Application. this method, we perform training on the whole data-set but leaves only one data-
point of the available data-set and then iterates for each data-point.
It has some advantages as well as disadvantages also. An advantage of using this
method is that we make use of all data points and hence it is low bias. The major
drawback of this method is that it leads to higher variation in the testing model as we
are testing against one data point. If the data point is an outlier it can lead to higher
variation. Another drawback is it takes a lot of execution time as it iterates over ‘the
number of data points’ times.

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.

[Link] the data

Steps in Cross-Validation
developing a In order to use the data effectively, K Fold Cross Validation Or N-fold cross-
validation is used in splitting the data. Here data is split into N folds.
Machine For example in a 10-fold CV, 2 folds are used for training and the remaining 8 folds
Learning are used for the training phase. iteratively all the folds get interchanged such that
Application. every fold gets a chance to be testing data. So a total of 10 models can be built using
this can performance metrics values will be calculated for those models. And the final
decision will be made after analyzing the performance metrics.

Example The diagram shows an example of the training subsets and evaluation
subsets generated in k-fold cross-validation. Here, we have total 25 instances. In first
iteration we use the first 20 percent of data for evaluation, and the remaining 80
percent for training([1-5] testing and [6-25] training) while in the second iteration we
use the second subset of 20 percent for evaluation, and the remaining three subsets
of the data for training([6-10] testing and [1-5 and 11-25]

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.

4 Choose appropriate model

Steps in After segregating the data, our next work is to find a good algorithm suited for our
developing a model. This is one of the most important steps in machine learning. There are
various existing models and algorithms are there to use. Our job is to find an
Machine appropriate algorithm from the variety of options over there.
Learning There are three types of Machine Learning Models
Application. – Supervised Learning, Unsupervised Learning, and Reinforcement Learning.

Supervised Learning – deals with training the model with labeled data. The
outcome is known, so the model is continuously refined to get the accuracy.
Our sample data with fruits fall under this category since we already know the
outcome.
Linear Regression Algorithm can be used to build a model with our data. Some of
the common algorithms in Supervised Learning are Linear regression, Logistic
regression, Polynomial regression, Random forest, Decision tree, K-nearest
neighbors, and Naive Bayes. We have the data with predefined output to train the
model in the case of Supervised Learning.

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.
4 Choose appropriate model

Steps in Unsupervised Learning – If the label data is not available and the outcome is
unknown it falls under the category of Unsupervised Learning. This algorithms
developing a
clusters and groups the data into categories. Some of the algorithms used in
Machine Unsupervised Learning are Principal component analysis, K-means clustering,
Learning Apriori Algorithm, Partial least squares, Fizzy means, Hidden Markov models,
Application. and Hierarchical clustering.

Reinforcement Learning– learns and makes decisions based on trial and error
method. A common example of the Reinforcement Learning algorithm is
Markov’s decision process.
There are various algorithms available for various purposes of the model. Some
algorithms are suited for dealing with text, some for images, and much more.
Choose an appropriate algorithm to build your model after analyzing the dataset
and aim to build the model. We can implement our model to distinguish fruits
with Linear Regression since we are dealing with 3 independent variables to
predict the outcome.

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.

5 Train the model

Steps in
The main process of building our model starts with the training phase.
developing a Here we use the split part of the data set allocated for training to make our model learn.
Machine Our model learns to identify fruits by analyzing their characteristics.
Learning Our 3 features have a coefficient called the weight of features.
And the constant is known as the bias of the model.
Application. First, we pick random values and compare them with actual output, and then the
difference can be reduced by trying different biases and values.
Repeat the iteration until the model reaches a decent amount of accuracy.

You should spend a quality amount of time during the training phase.
The more you tune and prepare the model during training, the better will be the results.
This phase requires a lot of patience and experimentation.
If you succeed in training your model well, then you can expect good results from
your model.

Notes By Archana S (VPPCOE& VA)

 ⮚ Steps in developing a Machine Learning Application.
5 Train the model
Steps in The most important problems considered during the training of models are
developing a optimization and generalization.
Machine • Optimization – is defined as the process of adjusting the model to get the best
Learning performance possible on training data i.e. the learning process.
Application. • Generalization – is said to be how well the model performs on unseen data.
The main goal is to get the best generalization possible.
Let’s understand two important terms before moving further steps in the
Machine Learning model.
Bias – They are the assumptions made by the model to make a function easy to
learn
Variance – After training data and obtaining low error. Upon changing the data,
then training the same model and experiencing a high error, is known as a
variance.

Notes By Archana S (VPPCOE& VA)

 5 Train the model
Steps in Overfitting and Underfitting
developing a A model is said to be under-fitted when it cannot capture the
Machine underlying data. It denotes that our algorithm or model does not fit
Learning the data well. It happens when we have less data to build the
Application. model. Overfitting simply means high bias and low variance.
Underfitting reduces the accuracy of the model.

Underfitting can be reduced by

⮚Increasing the model complexity
⮚Increasing the number of features
⮚Removing the noise from the data
⮚Increasing the duration of training

Notes By Archana S (VPPCOE& VA)

 5 Train the model
Steps in
developing a A model is said to be over-fitted when we train with a lot
Machine of data. This happens when the model gets trained with
Learning so many inaccurate data entries and noise. Overfitting
Application. simply means high variance and low bias.
Overfitting can be reduced by
⮚Reducing model complexity
⮚Increasing the training data
⮚Ridge Regularization and Lasso Regularization
⮚Using dropout for neural networks to tackle
overfitting.

Notes By Archana S (VPPCOE& VA)

 6 Evaluate the model
Steps in After training the model with trained data, the model has to be tested.
developing a The purpose of testing is to evaluate how the model will work in real-world
Machine scenarios.
We can evaluate the accuracy of the model during this phase.
Learning
In our case, the model tries to identify the type of fruit with the learning done in
Application. the previous phase.
The evaluation phase is very important and we can check whether the model
achieves the goal we planned.
If the model does not perform well up to mark during the testing phase then
the previous steps have to be re-iterated until we attain the required accuracy.
As mentioned earlier we should not use the same data used during the
training phase. The separate data splitter from our dataset should be
used for evaluation.

Notes By Archana S (VPPCOE& VA)

 6 Evaluate the model

Steps in
Regression metrics
developing a regression models deal with a continuous range of values
Machine instead of classes. Mean squared error in the model is
Learning calculated by taking the average of squared differences
Application. between the predicted output and the actual output.
Learning curves are plotted with training data and
validation data.
If our model gives high bias, we can come to the
conclusion of having errors in the validation and training
datasets. If the model suffers from high bias, training is to
be done more to improve the model.

Notes By Archana S (VPPCOE& VA)

 6 Evaluate the model

Classification metrics
Steps in Classification models are concerned only with whether the outcome
developing a is correct or not.
Machine When performing classification predictions like our model, there are
Learning four possible outcomes that could be expected. They are true
Application. positives, true negatives, false positives, and false negatives.
These four outcomes are plotted on a confusion matrix. You can
generate the matrix after predictions on the test data and categorize
each prediction as one of the possible outcomes.
The accuracy of the model is the percentage of correct predictions
made by the test data. The accuracy of the model can be evaluated
by dividing the number of correct predictions by the number of total
predictions. There are other metrics like precision and recall that are
also used to evaluate Classification models.

Notes By Archana S (VPPCOE& VA)

 7 Hyperparameter tuning
Steps in Once the evaluation is successful, proceed to the next phase Parameter tuning.
developing a This step in the machine learning model improves the results gained during the
evaluation step.
Machine A Hyperparameter of the model is a configuration that is external to the model
Learning and whose value cannot be estimated from the data provided.
Application. They are used in processes to help estimate model parameters, they are tuned
for a given predictive modeling problem and they can often be set using
heuristics.

Some of the examples in model hyperparameters include the C and sigma

hyperparameters for support vector machines, K in K-nearest neighbors,
learning rate for training a neural network, and the penalty in Logistic
Regression Classifier.
The model parameters are estimated from data automatically and model
hyperparameters are set manually and are used in processes to help estimate
model parameters.

Notes By Archana S (VPPCOE& VA)

 7 Hyperparameter tuning
Steps in Hyperparameter tuning is choosing a set of optimal
developing a hyperparameters for a learning algorithm.
Machine Model hyperparameters are often referred to as parameters
Learning because they are the parts of machine learning that must be set
Application. manually and tuned.
In our case, we can make our model recognize fruits better by
hyperparameter tuning. There are multiple ways we can improve
and tune the model. You can revisit the training phase and use
multiple sweeps of data to train the model. This can lead to
better accuracy and also the long duration of training
provides better accuracy and results. You can also refine the
initial values given to the model. There are many other parameters
we can tune and achieve desired results.

Notes By Archana S (VPPCOE& VA)

 8 Prediction
Steps in The final step in machine learning model building is prediction.
developing a This is the phase where our model can be considered to be
Machine ready for applications.
Learning Our fruit model should be able to identify the name of the fruit.
Application. The model is given independence from human interference
and it exposes its conclusion on the basis of the datasets and
the training given. We succeed in the Machine Learning model
If it can able to perform well in different scenarios.
This step is executed by the end-users when they use the
particular model in the respective domain. Machine Learning
models can process large amounts of data and make
decisions. The model we build to find fruits is very simple yet
the steps can be easily understood with the help of that.

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised 2. Classification
(Logistic Classification algorithms are used when the output
Regression,
variable is categorical, which means there are two
Decision
Tree, classes such as Yes-No, Male-Female, True-false, etc.
Support Spam Filtering,
Vector • Random Forest
Machine) • Decision Trees
• Logistic Regression
• Support vector Machines

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)
Logistic Regression Algorithm
Supervised Logistic regression may be a supervised learning classification algorithm wont
(Logistic to predict the probability of a target variable.
Regression,) We use logistic regression for the binary classification of data-points. We perform
categorical classification such that an output belongs to either of the two classes
(1 or 0).

For example – we can predict whether it will rain today or not, based on
the current weather conditions.
Two of the important parts of logistic regression are Hypothesis and Sigmoid
Curve. With the help of this hypothesis, we can derive the likelihood of the event.

The data generated from this hypothesis can fit into the log function that creates
an S-shaped curve known as “sigmoid”. Using this log function, we can further
predict the category of class.

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)
Logistic Regression Algorithm
Supervised We can represent the sigmoid as follows:
(Logistic
Regression,)
The produced graph is through this logistic function:
1 / (1 + e^-x)
The ‘e’ in the above equation represents the S-shaped
curve that has values between 0 and 1.
We write the equation for logistic regression as follows:
y = e^(b0 + b1*x) / (1 + e^(b0 + b1*x))
In the above equation, b0 and b1 are the two coefficients of
the input x.
We estimate these two coefficients using “maximum
likelihood estimation”.

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)
Logistic Regression Algorithm
Supervised Logistic regression transforms the output using a sigmoid function and returns a probability
value, which can then be mapped to two or more discrete classes.
(Logistic
If the estimated probability is greater than 50% or has the largest value, then the model
Regression,) predicts that the instance belongs to that class and if it does not or has a small value, the
model predicts that it is not in that class. For example, 60% is class 1 and 40% is class 0.

• Logistic regression can be used in several

classification cases, such as binary classification
(pass or fail), multiclass classification (cat, dog, or
pig), and ordinal classification (high, medium,
low).
• To evaluate the logistic regression model, we can
use the confusion metric, ROC Curve, AIC
(Akaike Information Criteria), Null Deviance, and
Residual Deviance.
Notes By Archana S (VPPCOE& VA)
 Logistic Regression Algorithm
Simple implementation of logistic regression using Scikit-
learn:
from [Link] import make_classification
Supervised from sklearn.linear_model import LogisticRegression
(Logistic
# generate dummy datasets
Regression,)
features, target = make_classification(n_samples = 100,
n_features = 3,
n_informative = 3,
n_redundant = 0,
n_classes = 2,
random_state = 1)

X_train, X_test, y_train, y_test = train_test_split(features,target,test_size=0.2)

# Logistic regression
logit = LogisticRegression()
[Link](X_train, y_train)
[Link](X_test)

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised Decision Tree

(Decision Decision Tree algorithms are used for both predictions as well
Tree) as classification in machine learning.
Using the decision tree with a given set of inputs, one
can map the various outcomes that are a result of
the consequences or decisions.
We can understand decision trees with the following example:
Let us assume that you have to go to the market to buy some
products. At first, you will assess if you really need the product.
Suppose, you will only buy shampoo if you run out of it. If you do
not have the shampoo, you will evaluate the weather outside and
see if it is raining or not. If it is not raining, you will go and
otherwise, you will not.

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised Decision Tree

(Decision We can visualize this in the form of a decision tree as follows:

Tree)

This decision tree is a result of various hierarchical steps that will help you to reach certain decisions. In order to
build this tree, there are two steps – Induction and Pruning. In induction, we build a tree whereas, in pruning,
we remove the several complexities of the tree.

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised Decision Tree

(Decision
Tree)

A Decision Tree can also estimate the probability that an instance belongs to a particular class. In practice, the
decision tree model is very susceptible to overfitting. To solve this problem, You can stop the creation of the tree
early or build the tree but then remove or collapse nodes that contain little information. To stop tree creation early,
you must set the maximum depth of the tree and the maximum number of leaves.

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised Decision Tree

from [Link] import load_iris
(Decision from [Link] import DecisionTreeClassifier, DecisionTreeRegressor
Tree) from sklearn.model_selection import train_test_split
# load datasets iris
iris = load_iris()
features = [Link]
target = [Link]
X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.2)

# Decision tree for regression

tree_reg = DecisionTreeRegressor()
tree_reg.fit(X_train, y_train)
tree_reg.predict(X_test)

# Decision tree for classification

tree_classifier = DecisionTreeClassifier()
tree_classifier.fit(X_train, y_train)
tree_classifier.predict(X_test)

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised Support Vector Machine Algorithm

(Support Support Vector Machines are a type of supervised machine learning
Vector algorithm that provides analysis of
Machine) data for classification and regression analysis.
While they can be used for regression, SVM is mostly used
for classification.
We carry out plotting in the n-dimensional space. The value of each
feature is also the value of the specified coordinate.
Then, we find the ideal hyperplane that differentiates between the
two classes.
These support vectors are the coordinate
representations of individual observation. It is a frontier
method for segregating the two classes.

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised Support Vector Machine Algorithm

(Support SVM can work on both classification tasks and regression tasks. In
Vector the classification task, SVM is trying to fit the largest possible street
Machine) between two classes while limiting margin violations. Whereas in the
regression task, SVM tries to fit as many instances as possible on the
street while limiting margin violations and the width of the street is
controlled by a hyperparameter.
On classification tasks, SVM works well on datasets that can be
separated linearly or cannot be separated linearly. For datasets that
can be separated, we can use Soft Margin Classification, while for
datasets that cannot be separated, we can use Polynomial Kernel,
Similarity Features, and Gaussian RBF Kernel.

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised Support Vector Machine Algorithm

(Support
Vector
Machine)

Notes By Archana S (VPPCOE& VA)

 Supervised (Logistic Regression, Decision Tree, Support Vector Machine)

Supervised Support Vector Machine Algorithm

(Support
Vector
Machine)

Notes By Archana S (VPPCOE& VA)

 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)

Unsupervised Unsupervised Machine Learning:

(K Means Unsupervised learning is another machine learning method in
Clustering, which patterns inferred from the unlabeled input data.
Hierarchical The goal of unsupervised learning is to find the structure and
Clustering, patterns from the input data.
Association Unsupervised learning does not need any supervision. Instead,
Rules) it finds patterns from the data by its own.
Unsupervised learning can be used for two types of
problems: Clustering and Association.

Example: To understand the unsupervised learning, we will use the example given above. So unlike
supervised learning, here we will not provide any supervision to the model. We will just provide the input
dataset to the model and allow the model to find the patterns from the data. With the help of a suitable
algorithm, the model will train itself and divide the fruits into different groups according to the most
similar features between them.

Notes By Archana S (VPPCOE& VA)

 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)

K Means Clustering
Unsupervised (K K-means clustering is an unsupervised machine learning algorithm
Means Clustering,) and the most commonly used clustering algorithm. Clustering is the
task of partitioning the dataset into groups, called clusters. If we wish
to group data on unlabeled data, the K-means algorithm is the way to
go. K-means clustering tries to find the center of the cluster that
represents a certain region of the data and assigns each data point
to the closest cluster center.

The most important thing here is that you have to determine the
number of k clusters that the algorithm has to find. When there are
too few or too many clusters, the model will perform poorly.
Furthermore, choosing the correct Centroid initialization strategy has
a significant impact on the model’s performance during training

Notes By Archana S (VPPCOE& VA)

 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)

K Means Clustering
Unsupervised (K
Means Clustering,)

Notes By Archana S (VPPCOE& VA)

 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)

K Means Clustering
Unsupervised (K
Means Clustering,) from [Link] import make_blobs
from [Link] import KMeans

# generate dummy datasets

features, _ = make_blobs(n_samples = 500,
n_features = 2,
centers = 5,
cluster_std = 1,
shuffle = True,
random_state = 1)

# Kmeans clustering model

kmeans_model = KMeans(n_clusters=5, random_state=1)
kmeans_model.fit(features)

Notes By Archana S (VPPCOE& VA)

 Hierarchical Clustering
Hierarchical clustering is a popular method for grouping objects. It creates groups so
that objects within a group are similar to each other and different from objects in other
Unsupervised
groups. Clusters are visually represented in a hierarchical tree called a dendrogram.
(Hierarchical
Clustering) There are two main types of hierarchical clustering:
[Link]: Initially, each object is considered to be its own cluster. According
to a particular procedure, the clusters are then merged step by step until a single
cluster remains. At the end of the cluster merging process, a cluster containing all the
elements will be formed.

[Link]: The Divisive method is the opposite of the Agglomerative method. Initially,
all objects are considered in a single cluster. Then the division process is performed
step by step until each object forms a different cluster. The cluster division or splitting
procedure is carried out according to some principles that maximum distance
between neighboring objects in the cluster.

Between Agglomerative and Divisive clustering, Agglomerative clustering is generally

the preferred method. The below example will focus on Agglomerative clustering
algorithms because they are the most popular and easiest to implement.
Notes By Archana S (VPPCOE& VA)
 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)
Hierarchical Clustering
Hierarchical clustering has a couple of key benefits:
Unsupervised [Link] is no need to pre-specify the number of clusters. Instead, the dendrogram
(Hierarchical can be cut at the appropriate level to obtain the desired number of clusters.
Clustering)
[Link] is easily summarized/organized into a hierarchy using dendrograms.
Dendrograms make it easy to examine and interpret clusters.

Applications
There are many real-life applications of Hierarchical clustering. They include:
• Bioinformatics: grouping animals according to their biological features to
reconstruct phylogeny trees
• Business: dividing customers into segments or forming a hierarchy of employees
based on salary.
• Image processing: grouping handwritten characters in text recognition based on the
similarity of the character shapes.
• Information Retrieval: categorizing search results based on the query.

Notes By Archana S (VPPCOE& VA)

 Hierarchical Clustering

Hierarchical clustering steps

Unsupervised
(Hierarchical
Hierarchical clustering employs a measure of distance/similarity to
Clustering) create new clusters. Steps for Agglomerative clustering can be
summarized as follows:
• Step 1: Compute the proximity matrix using a particular distance
metric
• Step 2: Each data point is assigned to a cluster
• Step 3: Merge the clusters based on a metric for the similarity
between clusters
• Step 4: Update the distance matrix
• Step 5: Repeat Step 3 and Step 4 until only a single cluster remains

Notes By Archana S (VPPCOE& VA)

 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)
Hierarchical Clustering
Hierarchical Clustering using Scipy
Unsupervised The Scipy library has the linkage function for hierarchical (agglomerative) clustering.
(Hierarchical The linkage function has several methods available for calculating the distance between
Clustering) clusters: single, average, weighted, centroid, median, and ward. We will compare these methods
below. For more details on the linkage function, see the docs.
To draw the dendrogram, we'll use the dendrogram function. Again, for more details of
the dendrogram function.
First, we will import the required functions, and then we can form linkages with the various methods:
from [Link] import dendrogram, linkage
Z1 = linkage(X1, method='single', metric='euclidean')
Z2 = linkage(X1, method='complete', metric='euclidean')
Z3 = linkage(X1, method='average', metric='euclidean')
Z4 = linkage(X1, method='ward', metric='euclidean')
Now, by passing the dendrogram function to matplotlib, we can view a plot of these linkages:
[Link](figsize=(15, 10))
[Link](2,2,1), dendrogram(Z1), [Link]('Single')
[Link](2,2,2), dendrogram(Z2), [Link]('Complete')
[Link](2,2,3), dendrogram(Z3), [Link]('Average')
[Link](2,2,4), dendrogram(Z4), [Link]('Ward')
[Link]()

Notes By Archana S (VPPCOE& VA)

 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)
Hierarchical Clustering

Unsupervised
(Hierarchical
Clustering)

Notice that each distance method produces different

Notes By Archana S (VPPCOE& VA) linkages for the same data.
 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)
Hierarchical Clustering
Finally, let's use the fcluster function to find the clusters for the Ward linkage:
Unsupervised from [Link] import fcluster
(Hierarchical f1 = fcluster(Z4, 2, criterion='maxclust')
Clustering) print(f"Clusters: {f1}")

output
Clusters: [2 2 1 2 1 1 2 1 2]

[Link]
clustering/

Notes By Archana S (VPPCOE& VA)

 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)
Association Rule Learning
Association rule learning is a type of unsupervised learning
Unsupervised technique that checks for the dependency of one data item on
(Association Rules)
another data item and maps accordingly so that it can be more
profitable.
It tries to find some interesting relations or associations among
the variables of dataset.
It is based on different rules to discover the interesting relations
between variables in the database.
The association rule learning is one of the very important
concepts of machine learning, and it is employed in Market
Basket analysis, Web usage mining, continuous production,
etc.
Here market basket analysis is a technique used by the various
big retailer to discover the associations between items.
We can understand it by taking an example of a supermarket,
Notes By Archana S (VPPCOE&
as inVA)a supermarket, all products that are purchased together
 Unsupervised (K Means Clustering, Hierarchical Clustering, Association Rules)
Association Rule Learning
For example, if a customer buys bread, he most likely can also
Unsupervised buy butter, eggs, or milk, so these products are stored within a
(Association Rules)
shelf or mostly nearby. Consider the below diagram:

Notes By Archana S (VPPCOE& VA)

 Association Rule Learning
Association rule learning can be divided into three types of algorithms:
[Link]
Unsupervised [Link]
(Association Rules) 3.F-P Growth Algorithm

We will understand these algorithms in later chapters.

How does Association Rule Learning work?
Association rule learning works on the concept of If and Else Statement, such
as if A then B.

Here the If element is called antecedent, and then statement is called

as Consequent. These types of relationships where we can find out some
association or relation between two items is known as single cardinality. It is all
about creating rules, and if the number of items increases, then cardinality also
increases accordingly. So, to measure the associations between thousands of
data items, there are several metrics. These metrics are given below:
• Support
• Confidence
• Lift
Notes By Archana S (VPPCOE& VA)
 Association Rule Learning
Support
Support is the frequency of A or how frequently an item appears in the dataset. It is
Unsupervised defined as the fraction of the transaction T that contains the itemset X. If there are X
(Association Rules) datasets, then for transactions T, it can be written as:

Confidence
Confidence indicates how often the rule has been found to be true. Or how often the
items X and Y occur together in the dataset when the occurrence of X is already
given. It is the ratio of the transaction that contains X and Y to the number of records
that contain X.

Lift
It is the strength of any rule, which can be defined as below
formula:

Notes By Archana S (VPPCOE& VA)

 Association Rule Learning

It is the ratio of the observed support measure and expected

Unsupervised support if X and Y are independent of each other. It has three
(Association Rules) possible values:

• If Lift= 1: The probability of occurrence of antecedent and

consequent is independent of each other.
• Lift>1: It determines the degree to which the two itemsets are
dependent to each other.
• Lift<1: It tells us that one item is a substitute for other items,
which means one item has a negative effect on another.

Notes By Archana S (VPPCOE& VA)

 Association Rule Learning
Apriori Algorithm
This algorithm uses frequent datasets to generate association rules. It is
Unsupervised designed to work on the databases that contain transactions. This algorithm
(Association Rules) uses a breadth-first search and Hash Tree to calculate the itemset efficiently.
It is mainly used for market basket analysis and helps to understand the
products that can be bought together. It can also be used in the healthcare
field to find drug reactions for patients.
Eclat Algorithm
Eclat algorithm stands for Equivalence Class Transformation. This algorithm
uses a depth-first search technique to find frequent itemsets in a transaction
database. It performs faster execution than Apriori Algorithm.
F-P Growth Algorithm
The F-P growth algorithm stands for Frequent Pattern, and it is the improved
version of the Apriori Algorithm. It represents the database in the form of a
tree structure that is known as a frequent pattern or tree. The purpose of this
frequent tree is to extract the most frequent patterns.

Notes By Archana S (VPPCOE& VA)

 Association Rule Learning
Applications of Association Rule Learning
Unsupervised It has various applications in machine learning and data
(Association Rules) mining. Below are some popular applications of association
rule learning:
• Market Basket Analysis: It is one of the popular
examples and applications of association rule mining. This
technique is commonly used by big retailers to determine
the association between items.
• Medical Diagnosis: With the help of association rules,
patients can be cured easily, as it helps in identifying the
probability of illness for a particular disease.
• Protein Sequence: The association rules help in
determining the synthesis of artificial Proteins.
• It is also used for the Catalog Design and Loss-leader
Analysis and many more other applications.
Notes By Archana S (VPPCOE& VA)
 Supervised Learning Unsupervised Learning
Supervised learning algorithms are trained using labeled data. Unsupervised learning algorithms are trained using unlabeled data.

Supervised learning model takes direct feedback to check if it is Unsupervised learning model does not take any feedback.
predicting correct output or not.
Supervised learning model predicts the output. Unsupervised learning model finds the hidden patterns in data.

In supervised learning, input data is provided to the model along In unsupervised learning, only input data is provided to the model.
with the output.
The goal of supervised learning is to train the model so that it can The goal of unsupervised learning is to find the hidden patterns and
predict the output when it is given new data. useful insights from the unknown dataset.
Supervised learning needs supervision to train the model. Unsupervised learning does not need any supervision to train the
model.
Supervised learning can be categorized Unsupervised Learning can be classified
in Classification and Regression problems. in Clustering and Associations problems.
Supervised learning can be used for those cases where we know the Unsupervised learning can be used for those cases where we have
input as well as corresponding outputs. only input data and no corresponding output data.
Supervised learning model produces an accurate result. Unsupervised learning model may give less accurate result as
compared to supervised learning.
Supervised learning is not close to true Artificial intelligence as in Unsupervised learning is more close to the true Artificial
this, we first train the model for each data, and then only it can Intelligence as it learns similarly as a child learns daily routine
predict the correct output. things by his experiences.
It includes various algorithms such as Linear Regression, Logistic It includes various algorithms such as Clustering, KNN, and Apriori
Regression, Support Vector Machine, Multi-class Classification, algorithm.
Decision tree, Bayesian Logic, etc.

Notes By Archana S (VPPCOE& VA)

 Confusion Matrix in Machine Learning
The confusion matrix is a matrix used to determine the performance of the
classification models for a given set of test data. It can only be determined if the true
values for test data are known. The matrix itself can be easily understood, but the
related terminologies may be confusing. Since it shows the errors in the model
performance in the form of a matrix, hence also known as an error matrix. Some
features of Confusion matrix are given below:
• For the 2 prediction classes of classifiers, the matrix is of 2*2 table, for 3 classes, it
is 3*3 table, and so on.
• The matrix is divided into two dimensions, that are predicted values and actual
values along with the total number of predictions.
• Predicted values are those values, which are predicted by the model, and actual
values are the true values for the given observations.
• It looks like the below table:

Notes By Archana S (VPPCOE& VA)

 The above table has the following cases:
• True Negative: Model has given prediction No, and the real or actual
value was also No.
• True Positive: The model has predicted yes, and the actual value was
also true.
• False Negative: The model has predicted no, but the actual value was
Yes, it is also called as Type-II error.
• False Positive: The model has predicted Yes, but the actual value was
No. It is also called a Type-I error.
Need for Confusion Matrix in Machine learning
• It evaluates the performance of the classification models, when they
make predictions on test data, and tells how good our classification
model is.
• It not only tells the error made by the classifiers but also the type of
errors such as it is either type-I or type-II error.
• With the help of the confusion matrix, we can calculate the different
parameters for the model, such as accuracy, precision, etc.

Notes By Archana S (VPPCOE& VA)

 From the above example, we can conclude that:
• The table is given for the two-class classifier, which has two
predictions "Yes" and "NO." Here, Yes defines that patient has
the disease, and No defines that patient does not has that
disease.
• The classifier has made a total of 100 predictions. Out of
100 predictions, 89 are true predictions, and 11 are incorrect
predictions.
• The model has given prediction "yes" for 32 times, and "No"
for 68 times. Whereas the actual "Yes" was 27, and actual
Notes By Archana S (VPPCOE& VA)
"No" was 73 times.