MACHINE LEARNING.

UNIT-1
• Introduction to Machine Learning: Machine Learning(ML), Need for
Machine Learning, ML from Knowledge-driven to Data Driven,
Applications of Machine Learning, Problems suitable for Machine
Learning, Advantages, Disadvantages and Challenges of Machine
Learning, Challenges of ML. General architecture of ML systems,
• Underlying Concepts in Machine Learning: Inductive Learning,
Generlization, Bias and Variance, Overfitting and Underfitting, Parametic
and Non Parametric algorithms
• Types of Machine Learning: Supervised and Unsupervised Learning,
Workflow, Semisupervised Learning, Reinforced Learning,
 Introduction to Machine Learning:
• Machine Learning (ML) is a branch of Artificial Intelligence (AI) that
enables computers or systems to learn from data and improve their
performance automatically without being explicitly programmed.
• In simple terms, machine learning algorithms find patterns and
relationships in data and use them to make predictions or decisions.
• Machine Learning (ML) is a part of Artificial Intelligence (AI).
It helps computers or machines learn from data.
It means the system can improve its work or performance on its
own.
It does not need to be programmed step-by-step for every task.
• Example:
• Email Spam Detection:
The system learns from thousands of emails labeled as spam or not
spam and then predicts whether a new email is spam based on
learned patterns.
• Movie Recommendation (Netflix, YouTube):
The system learns your viewing habits and suggests similar movies or
videos automatically.
• Machine learning mainly focuses on the design and development of
computer programs that can teach themselves to grow and change
when exposed to new data.
• Using machine learning, we can collect information from a dataset by
asking the computer to make some sense from data. Machine
learning is turning data into information.

DATA
COMPUTER PROGRAM
OUTPUT
• ML system takes the training data and background knowledge as input.
• Background knowledge and data help the Learner program to provide a
solution for a particular task or problem.
• Performance corresponding to the solution can also be measured.
• ML system comprises mainly two components — a Learner and a Reasoner.
• The Learner uses the training data and background knowledge to build the
model, and this can be used by the Reasoner to provide the solution for a
task.
• Machine learning can be applied to many applications such as politics to
geosciences.
• It is a tool that can be applied to many problems.
• Any application which needs to extract some information from data and
also takes some action on data can benefit from machine learning
methods.
• Some of the applications are spam filtering in email, face recognition, product
recommendations from [Link] and handwriting digit recognition.
• In detecting spam email, if you check for the occurrence of a single word it will
not be very helpful.
• But checking the occurrences of certain words used together and combining this
with the length of the email and other parameters, you could get a much clearer
idea of whether the email is spam or not.
• Machine learning is used by most of the companies to increase productivity,
forecast weather, to improve business decisions, detect disease and do many
more things.
• Machine learning uses statistics. There are many problems where the solution is
not deterministic.
• There are certain problems for which we don’t have that much information and
also don’t have that much computing power to properly model the problem.
• For these problems we need statistics; example of such type of problem is
prediction of motivation and behavior of humans.
• The behavior and motivation of humans is a problem that is currently very
difficult to model.
 Need for Machine Learning
1. Handling large amounts of data:
• In today’s world, data is being generated at an enormous rate. It is not
possible for humans to analyze such large amounts of data manually.
Machine Learning (ML) helps in analyzing, processing, and making sense of
this data automatically.
2. Improved decision-making:
• Machine Learning models can identify patterns and insights from data that
are not easily visible to humans. These patterns help in making better
business and scientific decisions.
3. Automation of tasks:
• ML automates repetitive and time-consuming tasks — for example, email
filtering, chatbots, and recommendation systems — which reduces human
effort and saves time.
4. Adaptability:
• Unlike traditional programming, ML models can adapt and improve
over time with new data without the need for reprogramming.
5. Complex problem solving:
• Many real-world problems, like image recognition, speech translation,
and disease prediction, are too complex to be solved by simple
algorithms. ML can handle such complex, nonlinear problems
effectively.
6. Personalization:
• ML helps provide personalized experiences such as customized
product recommendations on Amazon or Netflix based on users’ past
behaviors and preferences.
 ML from Knowledge-driven to Data Driven
1. Knowledge-Driven Approach (Before ML Era)
• Based on Human Expertise:
Early AI systems relied on rules and logic created by experts (like doctors,
engineers, etc.).
• Example:
In a medical diagnosis system, experts would write “If-Then” rules such as:
• If temperature > 102°F and cough is severe → possible flu.
• Characteristics:
• Manual feature selection.
• Heavy dependence on domain experts.
• Poor scalability — hard to handle large or complex data.
• Main Limitation:
Could not adapt automatically — required explicit programming for every
new situation.
2. Data-Driven Approach (Machine Learning Era)
• Based on Data and Patterns:
Instead of using human-written rules, ML systems learn patterns
directly from data.
• Example:
A medical ML model is trained on thousands of patient records and
learns automatically which features (like temperature, cough, etc.)
indicate flu.
• Characteristics:
• Learns from examples, not rules.
• Improves as more data is provided.
• Automatically discovers patterns and relationships.
• More accurate and flexible.
 Aspect Knowledge-Driven (Old AI) Data-Driven (Machine Learning)

Source of Knowledge Human experts Data and algorithms

Learning Method Manually programmed rules Automated learning from examples

Adaptability Low High

Performance with Big Data Poor Excellent

Neural Networks, Decision Trees,

Example Expert Systems
etc.
 Applications of Machine Learning
1. Image Recognition
• Used to identify objects, people, or scenes in images.
• Example: Face detection in smartphones, handwriting recognition, object
detection in self-driving cars.

2. Speech Recognition
• Converts spoken language into text or commands.
• Example: Google Assistant, Siri, Alexa.

3. Medical Diagnosis
• Helps doctors detect diseases early by analyzing medical data (like X-rays, MRIs,
or patient history).
• Example: Predicting diabetes or cancer using ML models.
4. Financial Services
• Used in fraud detection, credit scoring, and stock market prediction.
• Example: Banks use ML to detect unusual transactions and prevent
fraud.

5. Recommendation Systems
• Suggests products, videos, or music based on user preferences.
• Example: Netflix movie suggestions, Amazon product
recommendations, YouTube’s “Up Next”.

6. Autonomous Vehicles (Self-Driving Cars)

• ML helps vehicles detect traffic signals, pedestrians, and road lanes.
• Example: Tesla’s self-driving technology uses ML and computer vision.
7. Natural Language Processing (NLP)
• ML allows computers to understand and generate human language.
• Example: Chatbots, translation apps, grammar checkers.
8. Agriculture
• Predicting crop yield, detecting plant diseases, and optimizing irrigation
using data.
• Example: Smart farming with drone image analysis.
9. Cybersecurity
• Detects unusual patterns in network traffic to prevent attacks or malware.
• Example: Intrusion detection systems using ML.
10. E-commerce
• Personalized shopping experience, customer segmentation, and demand
forecasting.
• Example: Dynamic pricing on Amazon or Flipkart.
 Problems Suitable for Machine Learning
1. Problems with too many rules or fine-tuning
• Some problems require writing hundreds or thousands of rules if done
manually.
• Maintaining and updating these rules is difficult and error-prone.
• ML can automatically learn these patterns from data instead of relying on
fixed rules.
• Example:
• Spam Email Detection:
Instead of manually listing spam words (like “win,” “offer,” “free”), ML
models learn from large email datasets and automatically identify spam
patterns.
• Credit Card Fraud Detection:
ML models learn normal and abnormal transaction patterns instead of
using manually coded thresholds.
2. Complex problems with no clear algorithmic solution
• Some problems are too complex to express mathematically or
program using traditional algorithms.
• ML can model these complex relationships by learning from examples
(training data).
• Example:
• Image Recognition:
ML models (like CNNs) learn to identify objects such as cats, cars, or
people without explicitly being told what features to look for.
• Speech Recognition:
Converts spoken words into text by learning from large datasets of
voice samples.
3. Problems in changing or dynamic environments
• In many real-world situations, data patterns change over time (non-
stationary data).
• Traditional models fail because they need manual updates.
• ML models can continuously learn and adapt to new trends.
• Example:
• Stock Market Prediction:
ML models adjust to new market data and patterns.
• Weather Forecasting:
ML models improve predictions as more recent data is provided.
• Online Recommendations:
ML systems (like Netflix or YouTube) update suggestions based on
new user behavior.
4. Problems involving large and complex data (Big Data)
• When data is huge and contains hidden patterns, ML helps uncover
insights that humans cannot easily find.
• These insights are useful for decision-making and automation.
• Example:
• Customer Behavior Analysis:
Companies like Amazon use ML to predict what products users are
likely to buy.
• Healthcare Analytics:
ML identifies patterns in medical records to predict diseases early.
• Social Media Analysis:
Detecting trending topics or sentiment from large volumes of posts.
 Advantages of Machine Learning
1. Automation of Processes
• Machine Learning automates repetitive and time-consuming tasks.
• Once trained, ML models can make decisions or predictions automatically
without human involvement.
• Example:
• Email spam filters automatically separate spam from important emails.
• Banking systems detect fraudulent transactions automatically.

2. Continuous Improvement
• ML algorithms learn and improve from new data over time.
• The more data provided, the better the model performs.
• Example:
• Voice assistants like Alexa, Google Assistant, and Siri become smarter as they interact
with users.
• Recommendation systems improve as they gather more user feedback.
3. Efficient Handling of Complex and Large Data
• ML can analyze large and complex datasets that are impossible for humans to
process manually.
• It helps in discovering hidden patterns and relationships in the data.
• Example:
• In healthcare, ML analyzes medical images to detect diseases early.
• In finance, it identifies investment opportunities by studying stock data.

4. Accurate Predictions and Trend Detection

• ML models can predict future outcomes based on past data.
• It helps organizations make informed decisions.
• Example:
• E-commerce sites predict customer buying behavior.
• Weather forecasting models predict rainfall or storms using ML.
• Personalization
• ML helps in providing a personalized experience for users.
• It tailors recommendations or services according to user behavior and
preferences.
• Example:
• Netflix suggests shows based on viewing history.
• Amazon recommends products similar to previous purchases.
 Disadvantages of Machine Learning
1. Data Dependency
• ML requires a large amount of quality data to train models effectively.
• Poor or incomplete data leads to inaccurate results.
• Example:
• In medical diagnosis, if training data lacks diversity, the model might misdiagnose
certain cases.
2. Time and Resource Intensive
• Training ML models takes a lot of time and computational power.
• Requires high-end hardware (like GPUs) and skilled professionals.
• Example:
• Deep learning models such as image recognition systems take hours or days to
train
3. Lack of Transparency (Black Box Problem)
• Some ML algorithms are complex and difficult to interpret.
• It’s hard to understand why a model made a certain decision.
• Example:
• In credit scoring, a bank might not know the exact reason an ML model denied a loan.

4. Possibility of Bias and Errors

• ML models learn from data; if data is biased, predictions will also be biased.
• This can lead to unfair or inaccurate outcomes.
• Example:
• Facial recognition systems showing racial or gender bias due to unbalanced training data.
5. High Cost of Development
• Developing ML-based systems requires expert knowledge and infrastructure.
• Small organizations may find it costly to implement.
• Example:
• Building a large-scale recommendation engine like Netflix’s requires huge investment in
data and computation.
 Challenges of Machine Learning
1. Lack of Quality Data
• ML models require large, accurate, and diverse datasets to learn effectively.
• If the data is incomplete, noisy, or inconsistent, the model produces poor
results.
• Example:
• A hospital’s dataset missing patient records can lead to wrong disease predictions.
• An e-commerce system may give poor recommendations if product data is outdated.
2. Data Privacy and Security
• ML models often use sensitive personal data such as financial or medical
information.
• Storing and processing this data can raise privacy and ethical issues.
• Example:
• Health apps using patient data must protect it from leaks or misuse.
• Social media platforms collecting user data can cause privacy concerns
3. Choosing the Right Algorithm
• Selecting the correct ML algorithm for a specific problem is challenging.
• The wrong algorithm may lead to inaccurate or inefficient results.
• Example:
• Using a linear model for non-linear data like image recognition won’t give good accuracy.
• For small datasets, simpler models work better than deep learning models.

4. Overfitting and Underfitting

• Overfitting: The model learns training data too well, including noise — it
performs well on training data but poorly on new data.
• Underfitting: The model is too simple to capture data patterns.
• Example:
• A spam filter that memorizes specific email words (overfitting) fails to detect new spam
patterns.
• A model with too few features (underfitting) gives low accuracy.
5. Bias and Fairness Issues
• If the training data is biased, the ML model will also be biased.
• This leads to unfair or discriminatory outcomes.
• Example:
• A hiring system trained on biased data may prefer one gender over another.
• Facial recognition may work better for certain ethnicities due to biased data.

6. Lack of Skilled Professionals

• Designing and maintaining ML systems require skilled data scientists and
engineers.
• There is a shortage of professionals with expertise in ML.
• Example:
• Small businesses struggle to adopt ML because they lack technical knowledge.
General Architecture of Machine Learning Systems
1. Data Collection
• This is the first and most important step.
• In this stage, raw data is gathered from various sources such as
databases, sensors, files, APIs, or the internet.
• The quality and quantity of data directly affect the performance of the
ML model.
• Example:
• Collecting sales data from an e-commerce website.
• Gathering medical images for disease detection.
• Recording traffic data for self-driving cars.
2. Data Preprocessing
• Raw data is often noisy, incomplete, or inconsistent.
• In this step, data is cleaned, formatted, and transformed into a
suitable form for model training.
• Common preprocessing tasks include:
• Handling missing values
• Removing duplicates
• Normalizing or scaling data
• Encoding categorical variables
• Example:
• Filling missing patient age in a hospital dataset.
• Converting “Yes/No” answers into 1 and 0 for model input.
3. Feature Extraction / Feature Engineering
• Features are the important attributes or inputs used by the model for
learning.
• This step selects or creates the most relevant features from raw data
to improve model accuracy and efficiency.
• Example:
• In image recognition: extracting edges, color, and shapes as features.
• In email spam detection: using the frequency of specific words as
features.
• In house price prediction: selecting features like area, location, and
number of rooms.
4. Model Selection / Model Training
• In this step, a suitable machine learning algorithm is chosen and
trained using the prepared data.
• The model learns the relationship between input features (X) and
output labels (Y).
• Different algorithms are used based on the problem type —
classification, regression, or clustering.
• Example:
• Decision Tree or KNN for classification.
• Linear Regression for predicting house prices.
• K-Means for grouping customers based on purchase behavior
5. Model Evaluation
• After training, the model is tested using test data (unseen data) to
measure its performance.
• Common evaluation metrics include:
• Accuracy
• Precision and Recall
• F1-score
• Mean Squared Error (for regression tasks)
• Example:
• Checking if a spam classifier correctly predicts spam emails.
• Measuring the accuracy of a handwriting recognition system on new
samples.
6. Prediction / Deployment
• Once a model performs well, it is deployed into real-world
applications.
• The model now predicts outputs for new, unseen data.
• It can be integrated into apps, websites, or embedded systems.
• Example:
• A movie recommendation model used by Netflix.
• A credit scoring model used by banks for loan approval.
• A chatbot system predicting user intent.
7. Model Maintenance and Updating
• Over time, data patterns change (called data drift), so the model
must be retrained with new data.
• Regular monitoring ensures that the model continues to perform
accurately.
• Example:
• Re-training a sales prediction model every month with the latest data.
• Updating a spam filter as new spam patterns emerge.
 Data Collection
↓
Data Preprocessing
↓
Feature Extraction / Engineering
↓
Model Training
↓
Model Evaluation
↓
Deployment (Prediction)
↓
Monitoring & Updating
 Underlying Concepts in Machine Learning
1. Inductive Learning in Machine Learning
• Definition
• Inductive Learning is a type of learning where a machine learns
general rules or patterns from a set of specific examples (training
data) and then uses these rules to make predictions on new, unseen
data.
• In simple words:
• Inductive Learning means learning from examples.
• The system is not explicitly programmed with rules; instead, it infers
(induces) them from data.
• How It Works
• Input: A set of training examples (data + their correct outputs).
• Process: The learning algorithm finds patterns or relationships in the data.
• Output: A general hypothesis or model that can predict new outcomes.
• Example 2 – Weather Prediction
• Training Data:
A dataset with weather details such as temperature, humidity, and wind
speed labeled as Rain or No Rain.
• Learning Process:
The model identifies patterns — for example, “If humidity is high and
temperature is low → Rain.”
• Result:
When new weather data is given, the model predicts whether it will rain or
not.
• Example 3 – Student Performance Prediction
• Training Data:
Information like hours studied, attendance, and past scores labeled as
Pass or Fail.
• Learning Process:
The model learns the relationship between study hours and success.
• Result:
It can predict whether a new student is likely to pass or fail.
Advantages:-
• Can learn from real-world examples automatically.
• Flexible — can handle different types of data.
• Improves prediction accuracy with more data.
Disadvantages:-
• Requires labeled training data.
• May produce incorrect generalizations if data is noisy or biased.
• Performance depends on the quality and size of data.
Generalization:-
• Definition
• Generalization refers to a machine learning model’s ability to perform well
on new, unseen data — not just the data it was trained on.
• In simple words:
A model is said to generalize well if it can make accurate predictions on
data it has never seen before.
• Why Generalization is Important
• When we train a machine learning model, it learns from training data.
But in real-world applications, the model faces new data — so it must
generalize the patterns it learned.
• If a model only memorizes the training data and fails on new data, it is said
to have poor generalization.
• Example – Handwritten Digit Recognition (MNIST Dataset)

• Training Data: Thousands of images of handwritten digits (0–9).

• Test Data: New digit images written by different people.
• Good Generalization:
The model recognizes digits even if the handwriting style is different.
• Poor Generalization:
The model only works well on training digits but fails when
handwriting varies.
Bias and Variance:-
• Definition
• Bias is the error that occurs when a model is too simple and cannot capture
the underlying patterns in the data.
It leads to underfitting.
• Key Points
• Model makes strong assumptions about data
• Cannot learn complex relationships
• High bias → model predictions are far from the actual values
• Example
• Example 1: Straight Line for Non-linear Data
Imagine you want to fit a straight line to data that actually forms a curve.
Because the model is too simple, it underfits, leading to wrong predictions.
• Example 2: Student Using Only Simple Rules
A student who always uses one simple method for all math problems will make
mistakes because not all problems follow the same pattern → high bias.
 Variance
• Definition
• Variance is the error that occurs when a model is too complex and learns not only the
pattern but also the noise in the training data.
It leads to overfitting.
• Key Points
• Model becomes too sensitive to small fluctuations
• Memorizes the training data
• High variance → performs well on training but poorly on new data
• Example
• Example 1: Complex Curve for Simple Data
A model draws a very wiggly curve to perfectly pass through every training point.
It fits training data well but performs poorly on new data → overfitting.
• Example 2: Student Memorizing Answers
A student memorizes answers to sample questions.
During the exam, if questions change slightly, the student performs poorly → high
variance.
Overfitting vs Underfitting
 Overfitting
• Underfitting happens when a model is too simple to capture the
pattern in the data.
It doesn’t learn enough from the training data, so it performs poorly.
• You can think of it as:
Model is “under-trained” or “not capable enough.”
• Why does Underfitting happen?
• Too few features used
• Model is overly simple (e.g., linear model for complex data)
• Not trained long enough
• Too much regularization (we limited the model too much)
• Example of Underfitting
• Imagine you want to predict a student’s marks based on:
• Hours of study
• Lecture attendance
• Homework completion
• Sleep quality
• Notes quality
• But your model uses only 1 feature: hours studied.
• This is too simple!
It cannot capture the real pattern → Underfitting.
• Signs of Underfitting
• Model accuracy is low on training data
• Model accuracy is low on test data
• Model is too simple
• Predictions are too generalized
• High bias (model assumptions are too strong)
 OVERFITTING
• What is Overfitting?
• Overfitting happens when the model is too complex and learns everything,
including noise, mistakes, and random patterns.
• You can think of it as:
Model “over-learned” the training data and memorized it.

• Why does Overfitting happen?

• Too many features
• Model too complex (deep tree, big neural network)
• Too little data
• Training too long
• No regularization to control complexity
• Example of Overfitting
• You memorize answers for an exam word-by-word instead of
understanding the concepts.
• Practice questions → You get full marks (training accuracy high)
• Actual exam → You score low because questions are slightly different
(testing accuracy low)
• This is the perfect example of overfitting.
• Signs of Overfitting
• Training accuracy is extremely high
• Test accuracy is low
• Model is too complex
• Small change in input → big change in output
• High variance
 Parametric Algorithms
• Parametric algorithms assume a fixed number of parameters (fixed structure)
before training begins.
The model does not grow in size even if more training data is added.
• They make strong assumptions about the data’s pattern.
• How They Work
• Decide a model form (linear, logistic, etc.)
• Learn the best parameters (weights)
• Use these parameters to make predictions
• Parametric Example (Real Life)
• You assume everyone spends money based only on salary.
This is a strong assumption, so the model has fixed parameters.
• But people also spend based on age, needs, family size → model becomes too
simple → risk of underfitting
• Parametric Example
• 📌 Linear Regression
• Equation: y = mx + c
• Only 2 parameters (m, c)
• No matter how much data increases → parameters remain 2
 Non-Parametric Algorithms
• Non-parametric algorithms do not assume a fixed structure.
The number of parameters grows with data.
• They make fewer assumptions and learn the shape of data from scratch.

Key Characteristics:-
• No fixed parameters
• Model size increases with data
• Can learn complex patterns
• More flexible → low bias
• Higher risk of overfitting
• Usually need more data
• Slower prediction
• Non-Parametric Example (Real Life)
• To understand how students score, you do not assume a formula.
Instead, you look at all past students and compare the new student
to similar ones.
• More data = better accuracy.
This is exactly how KNN works.
• Non-Parametric Example
• KNN
• Stores all training data
• More data → more neighbors
• More data → bigger model
 Parametric vs Non-Parametric
Feature Parametric Non-Parametric

Parameters Fixed Grow with data

Assumptions Strong Few

Training speed Fast Slower

Flexibility Low High

Data requirement Less More

Risk Underfitting Overfitting

Linear Regression, Logistic KNN, Decision Trees, Random

Examples
Regression, Naive Bayes Forest, SVM (RBF)
 Types of Machine Learning:
1) Supervised Learning
• Supervised learning is a type of machine learning method in which we provide
sample labeled data to the machine learning system in order to train it, and on
that basis, it predicts the output.
• The system creates a model using labeled data to understand the datasets and
learn about each data, once the training and processing are done then we test
the model by providing a sample data to check whether it is predicting the exact
output or not.
• The goal of supervised learning is to map input data with the output data. The
supervised learning is based on supervision, and it is the same as when a student
learns things in the supervision of the teacher. The example of supervised
learning is spam filtering.
• Supervised learning can be grouped further in two categories of algorithms:
• Classification
• Regression
Supervised Learning Examples
• 1. Spam Email Detection
• The model is trained using labeled emails:
• Spam
• Not Spam
Then it predicts if a new email is spam.
• 2. Student Marks Prediction
• Using previous students’ data (hours studied → marks obtained), the
model predicts marks for a new student.
• 3. Weather Prediction
• Using past weather data (temperature, humidity, rainfall), it predicts
tomorrow’s weather.
2) Unsupervised Learning
• Unsupervised learning is a learning method in which a machine learns
without any supervision.
• The training is provided to the machine with the set of data that has
not been labeled, classified, or categorized, and the algorithm needs
to act on that data without any supervision. The goal of unsupervised
learning is to restructure the input data into new features or a group
of objects with similar patterns.
• In unsupervised learning, we don't have a predetermined result. The
machine tries to find useful insights from the huge amount of data. It
can be further classified into two categories of algorithms:
• Clustering
• Association
1. Customer Segmentation
• E-commerce companies group customers based on:
• Buying behavior
• Spending patterns
• Browsing history
No labels are used — the algorithm finds natural groups.
2. Market Basket Analysis
• Used in supermarkets to find items frequently bought together.
Example: People buying bread also often buy butter.
3. Anomaly Detection
• Detects unusual patterns in data.
Used in:
• Identifying unusual credit card transactions
• Network intrusion detection
3) Semi-Supervised Learning
• Semi-supervised learning is a type of machine learning method that uses both
labeled and unlabeled data for training.
Usually, a small amount of labeled data is combined with a large amount of
unlabeled data.
• This method is useful because labeling data is expensive or time-consuming, but
unlabeled data is easily available.
• How it works:
• A small labeled dataset helps the model understand the basic structure.
• A large unlabeled dataset helps the model learn deeper patterns.
• The model combines both to improve learning and accuracy.
• Goal of Semi-Supervised Learning:
• To improve learning accuracy using minimal labeled data and abundant unlabeled
data.
• Example:
• Identifying whether an email is spam when only a few emails are labeled as
spam/not spam, and the rest are unlabeled.
1. Google Search / Web Page Classification
• Only some web pages are labeled (e.g., “Sports”, “News”,
“Education”),
but millions are unlabeled.
The model uses a few labeled pages + many unlabeled pages to
classify new pages.
2. Speech Recognition
• Only a small amount of speech data is labeled with text transcripts,
but a huge amount of unlabeled audio exists.
Semi-supervised learning helps improve accuracy using both types of
data.
4) Reinforcement Learning
• Reinforcement learning is a feedback-based learning method, in
which a learning agent gets a reward for each right action and gets a
penalty for each wrong action. The agent learns automatically with
these feedbacks and improves its performance. In reinforcement
learning, the agent interacts with the environment and explores it.
The goal of an agent is to get the most reward points, and hence, it
improves its performance.
• The robotic dog, which automatically learns the movement of his
arms, is an example of Reinforcement learning.
• Self-Driving Cars
• The car (agent) learns by interacting with roads:
• Correct lane → reward
• Crash or wrong turn → penalty
It keeps improving driving decisions.
• Robotics (Robot Arm / Robot Dog)
• A robot learns how to walk, move hands, or pick objects:
• Successful movement → reward
• Falling or dropping objects → penalty
 Difference Between Supervised and Unsupervised Learning:-

Point of Difference Supervised Learning Unsupervised Learning

Data Type Uses labeled data Uses unlabeled data

Input Input (X) + Output/Label (Y) Only Input (X)

Goal Predict output for new data Find hidden patterns or groups

Examples of Tasks Classification, Regression Clustering, Association

Output Predicted value or class Clusters or pattern groups

Accuracy Usually higher (guided by labels) Lower (no labels for guidance)

Dependency Depends on labeled dataset No labeled dataset needed

Spam detection, price prediction, disease Customer segmentation, market

Real-Life Examples
diagnosis basket analysis, anomaly detection

Complexity More complex due to labeling Simpler as no labeling needed

UNIT-2
 Syllabus:-
• Introduction to Classification Algorithms: Concept of Classification,
Binary classification, Multi-Class Classification,
MultLabelClassification.
• K-Nearest Neighbor Method: need and Working of KNN, Computing
Distance, Pros and Cons of KNN
• Decision Tree based Algorithm: Terminologies assumptions,Working
of Decsion Trees, ID3 Alsogithm, Attribute selection Methods(Entropy,
Gini Impurity, Information Gain)
 Concept of Classification:-
• What is Classification?
• Classification is a Supervised Machine Learning technique where the
model learns from labelled data and predicts a category/class for
new data.
• Output is always a category, not a number.
Examples: Yes/No, Spam/Not Spam, Pass/Fail, Disease/No Disease,
Fraud/Not Fraud.
• Classification follows three main steps:
1. Training the model using labelled data
2. Learning patterns that separate classes
3. Predicting the class of new unseen data
Step 3: Prediction (Classifying New Data)
• What happens?
• Now the model uses the learned patterns to classify new unseen data.
• Example:
• A new student has:
• Study_Hours = 7
• Attendance = 85%
• The model checks the learned rule:
• Study_Hours > 4 and Attendance > 60% → Pass
• Predicted Result = Pass
• Another new student:
• Study_Hours = 1
• Attendance = 45%
• The model predicts:
Fail
Step 1: Training the Model (Using Labelled Data)
• What happens?
• You provide the model a dataset where:
• Features (inputs) are given
• Labels (output classes) are already known
• This helps the algorithm learn the relationship between inputs and
outputs.
Step 2: Learning Patterns (Model Finds Separating Boundaries)
• What happens?
• The algorithm analyses the data and finds patterns or decision rules
that separate classes.
• It tries to answer:
• “What conditions lead to Pass and what conditions lead to Fail?”
Example 1: Email Spam Classification
• We give the model many emails with labels:
• “Spam”
• “Not Spam”
• The model learns patterns like:
• Words used
• Sender
• Frequency of links
And then predicts whether a new email is spam or not.

Example 2: Loan Approval Classification

• Bank has previous records:
• Income
• Age
• Credit Score
• Previous Loans
• Approved or Not Approved (Label)
• The model learns the pattern and predicts:
Will the new applicant get loan approval? (Yes/No)
 Types of Classification
1. Binary Classification
• Only two possible classes.
Examples:
• Fraud vs Not Fraud
• Churn vs Not Churn
• Pass vs Fail
• Real-life Examples
• Spam Detection
• Email → Spam / Not Spam
• Medical Diagnosis
• Patient → Disease / No Disease
• Loan Approval
• Application → Approved / Not Approved
• Fraud Detection
• Transaction → Fraud / Not Fraud
2. Multi-Class Classification
• Definition
• Multi-class classification is when the output can belong to more than two
categories, but only one class at a time.
• Example: Classifying a fruit as Apple OR Banana OR Mango — not more than
one.
Real-life Examples
• Digit Recognition (MNIST)
Predict numbers 0–9 → 10 classes.
• Iris Flower Classification
Classes: Setosa, Versicolor, Virginica.
• Movie Genre Prediction
Predict: Action, Comedy, Drama, Romance (one at a time).
• Weather Classification
Sunny, Rainy, Cloudy, Snowy
3. Multi-Label Classification
• Definition
• In multi-label classification, a single data point can belong to multiple classes at the same time.
• The output is multiple labels instead of one.
Real-life Examples
• YouTube Video Tagging
One video may be tagged as:
• Education
• Technology
• AI
• Movie Genre Classification
One movie may be:
• Action
• Sci-Fi
• Thriller
• Image Tagging
A picture may contain:
• Dog
• Car
• Person
K-Nearest Neighbor (KNN) Method: Need and Working
• The K-Nearest Neighbor (KNN) algorithm is a simple, non-parametric,
supervised machine learning method used for both classification and
regression tasks.
• KNN is a supervised learning algorithm used for both classification
and regression.
Its main idea is:
“New data is classified based on how close it is to existing data."
• In simple terms:
Similar things stay near each other
Eg:-
Fruit Weight (g) Texture
Apple 150 Smooth
Apple 170 Smooth
Orange 140 Rough
Orange 130 Rough

You get a new fruit:

Weight: 160g, Texture: Smooth
Task: Is it an Apple or Orange?
 Need for KNN (Why Do We Use KNN?)

• KNN is needed because it is a simple, intuitive, and effective machine

learning method that works based on similarity.
It is especially useful when you need a model that:
• learns patterns from example data
• does NOT require complicated training
• works well when categories are separated by distance
1. Simple and Easy to Use
• KNN does not need any mathematical equation, training model, or
parameters.
It just compares distances.
• Example
• Imagine you want to identify whether a person is a kid, adult, or
senior, based only on age.
• To classify a new person (age 22), the model simply looks at:
• ages of existing people
• which ages are closest
• No math formulas, no training — just comparison.
This makes KNN very easy to understand and implement.
 2. Works Well with Real-Life Data
• The logic of KNN is human-like:
• If two things look similar, they belong to the same group.
• Example
• You are in a supermarket choosing fruits.
You pick a fruit that:
• looks round
• red
• weighs similar to an apple
• Your brain compares it with known apples → so you classify it as an
apple.
This is exactly how KNN works, using numerical similarity.
 3. Needs No Training
• Most algorithms spend time “training”.
KNN does NOT.
• It simply stores the data, and at the time of prediction:
• calculates distance
• finds nearest neighbors
• decides the class
• Example
• Suppose you build a system to classify flowers as
• Rose
• Sunflower
• Lily
• Algorithms like Decision Tree or SVM require a long training phase.
But KNN:
• instantly stores the data
• predicts when needed
• This makes KNN very helpful for quick prototyping.
 4. Works Well for Small, Clean Datasets
Study Hours Sleep (hrs) Result

5 8 Pass

2 6 Fail

6 7 Pass

When dataset is small, simple, and not noisy → KNN

performs extremely well.
✔ Example
Predicting whether a student passes or fails based on:
study hours
sleep hours
For small data, sophisticated models are unnecessary.
KNN gives excellent accuracy.
5. KNN Works for Both Classification & Regression

• KNN can predict:

• Categories → e.g., red/blue, pass/fail (classification)
• Numbers → e.g., house price, marks prediction (regression)
• Example
• Classification example:
Predict email → Spam or Not Spam
• Regression example:
Predict car price using similarity with recently sold cars.
Working of KNN
• KNN predicts the output using distance + nearest neighbors.
• KNN is a lazy learning algorithm, meaning it doesn't build a model
during a training phase; all computation is deferred until a prediction
is requested. It works on the core assumption that similar things exist
in close proximity to each other in the feature space ("birds of a
feather flock together")
• Step-by-step Working:
The working process for a new data point is typically as follows:-
1. Choose the value of K: Select the number of nearest neighbors (K) to consider. This is a crucial
hyperparameter (usually a small, odd integer like 3 or 5 to avoid ties in classification).
2. Calculate Distance: Calculate the distance (similarity) between the new data point and every point in the
training dataset. Common distance metrics include

1. Find K Nearest Neighbors: Sort the calculated distances and select the top K data points that have the
smallest distances to the new point.
2. Make a Prediction:
• Classification: The new data point is assigned to the class that is most frequent among its K nearest
neighbors (a "majority vote").
• Regression: The predicted value for the new data point is the average (or median) of the target values of its K
nearest neighbors
Pros of KNN
Advantage Description

Simple and Intuitive Easy to understand, interpret, and implement. It has few
hyperparameters to tune (only $K$ and the distance metric).

No Training Phase KNN is a lazy learner. The computational work is deferred

until classification, making the training process (data storage)
very fast.

Non-Parametric It makes no assumptions about the data distribution (e.g.,

that data is normally distributed), which makes it effective on
highly non-linear data.

Versatile Can be used effectively for both Classification (majority

vote) and Regression (average of neighbors).
Cons of KNN
Disadvantage Description
Computationally Expensive The prediction phase is slow because it requires calculating the
distance between the new data point and every single point in the
entire training dataset. The complexity is roughly $O(N \cdot D)$,
where $N$ is the number of data points and $D$ is the number of
features.

Sensitive to Scale/Outliers Features with larger scales will dominate the distance calculation,
regardless of their actual importance. Feature Scaling
(Normalization/Standardization) is mandatory. Outliers can also
disproportionately influence the result, especially with small values of
$K$.
Curse of Dimensionality As the number of features (dimensions) increases, the distance
between data points tends to become similar, making it harder to find
true nearest neighbors. This reduces the predictive power of the
model.
Storage Requirement Being a lazy learner, the entire training dataset must be stored in
memory for prediction.
 Decision Tree–Based Algorithms

• A Decision Tree is a supervised machine learning algorithm used for

classification and regression. It breaks the dataset into smaller
subsets based on feature values, forming a tree-like structure.
• It works by creating a model that predicts the value of a target
variable by learning simple decision rules inferred from the data
features, resulting in a hierarchical, flow-chart-like structure.
 Terminologies assumptions:-

1. Root Node: This is the starting node that represents the entire
dataset. It is subsequently split into two or more
homogeneous subsets.
2. Internal Node (or Decision Node): This node represents a
test on a specific feature (attribute) of the data. Branches
extending from this node represent the possible outcomes of
the test.
3. Branch: These are the connecting lines that show the flow from
the root node to the decision nodes, and finally to the leaf nodes.
They represent the rules derived from the test condition.

4. Leaf Node (or Terminal Node): This node represents the final
outcome or prediction (either a class label in classification or a
continuous value in regression). No further splitting is done at a
leaf node.
 Working of Decision Trees
• The process of building a Decision Tree is often called Recursive
Partitioning and follows a greedy, top-down approach:
1. Starting at the Root:-The entire training dataset is placed at the root
node. The algorithm employs a "divide and conquer" strategy to split
the data.
2. Finding the Best Split :- At each node, the core task is to select the
best feature and the best split-point for that feature that will divide the
data into the purest possible subsets. A subset is considered "pure" (or
homogeneous) if it contains data points belonging predominantly to a
single class (for classification) or has a very small variance in the target
value (for regression).This selection is based on different splitting
criteria depending on the algorithm (like ID3, C4.5, or CART):
 Gini Impurity
• The Gini Impurity is a measure used in Decision Trees to determine
how pure or mixed a set of data is. In the context of classification,
"pure" means all the data points in that subset belong to the same
class. The goal of a Decision Tree is to choose a split that results in the
lowest possible Gini Impurity (i.e., the purest child nodes)
Classification Trees: These typically use measures like:
• Information Gain (based on Entropy): Entropy measures the
randomness or impurity in the data. Information Gain is the decrease
in entropy after a split. The algorithm selects the split that yields the
highest Information Gain (maximum reduction in impurity).
• Gini Impurity: This measures the probability of incorrectly classifying
a randomly chosen element in the dataset if it were randomly labeled
according to the distribution of classes in the subset. The goal is to
choose the split that results in the lowest Gini Impurity.
Regression Trees: These typically use
• Variance Reduction (or reduction in Mean Squared Error) to find the
best split, aiming to minimize the variance of the target variable
within the resulting subsets.
• Recursive Splitting:- The splitting process is then repeated
recursively on each new child node (the resulting subsets) until a
stopping criterion is met.
4. Stopping Criteria:-
• The process stops when one or more of the following
conditions are met:
• All data points in a node belong to the same class (pure
node).A pre-defined maximum depth for the tree is reached.
The number of data points in a node falls below a minimum
sample size required for a split. The improvement in purity
(e.g., Information Gain) is less than a specified threshold.
• Once a stopping criterion is met, the node becomes a leaf
node, and its final classification is determined by the majority
class of the data points it contains (or the average target value
for regression).
Example: Calculating Gini Impurity
Let's use the Play Tennis example again. Imagine a node that contains 10 data points:
Class Count Proportion (Pi​)
Play Tennis = 7 $P_{Yes} = 7/10 = 0.7$
Yes
Play Tennis = 3 $P_{No} = 3/10 = 0.3$
No
Total 10
 ID3 (Iterative Dichotomiser 3) algorithm
• The ID3 (Iterative Dichotomiser 3) algorithm is a foundational, non-
parametric, supervised learning method used to construct a Decision
Tree for classification tasks. Developed by Ross Quinlan in 1986,
• It uses a top-down, greedy approach to build the tree, meaning it starts
at the root and, at each step, makes the locally optimal choice without
looking ahead. The core mechanism it uses to choose the best attribute
for splitting the data is Information Gain, which is calculated using the
concept of Entropy.
• The ID3 algorithm relies on two primary concepts from Information
Theory to select the most effective splitting attribute:
1. Entropy
2. Information Gain (IG)
1. Entropy (Measure of Impurity)
• Entropy quantifies the amount of uncertainty or disorder in a set of
data.
• Low Entropy (closer to 0): The dataset is mostly pure (contains
instances of only one class). This is the desired state for a leaf node.
• High Entropy (closer to 1): The dataset is highly mixed (classes are
equally distributed). This represents high uncertainty.
 Entropy eg:-
Suppose you have a small dataset of 10 students and whether
they Passed or Failed a test: Outcome Count
Pass 7
Fail 3

Final Answer:
The entropy of this dataset is approximately:
0.88 bits
Suppose we have a dataset of 20 fruits labeled as:
Fruit Type Count
Apple 10
Banana 6
Orange 4

Final Answer:
Entropy ≈ 1.49 bits
1. Entropy: The Measure of Impurity
In the context of machine learning, Entropy quantifies how mixed the class labels are
within a specific node of the tree.
High Entropy (Maximum Uncertainty): Occurs when the classes are evenly
distributed (e.g., in a binary classification, a node with 50% "Yes" and 50% "No"
samples has maximum entropy of 1.0). This represents the highest level of disorder or
uncertainty.
Low Entropy (Maximum Purity): Occurs when the node contains samples
belonging predominantly to a single class (e.g., a node with 99% "Yes" and 1%
"No"). This represents high homogeneity and low uncertainty.
Zero Entropy (Perfect Purity): Occurs when all samples in the node belong to a
single class (100% "Yes"). This node is a final decision or a leaf node.
 2. Information Gain:

The Splitting Metric Entropy itself is only the starting point. The
attribute selection is driven by Information Gain (IG), which uses
Entropy to determine the best feature for splitting the data at any given
node.
Information Gain measures the expected reduction in Entropy
achieved by splitting the current node based on a specific attribute.
The algorithm chooses the attribute that yields the maximum
Information Gain
 The Attribute Selection Process:-
The decision tree algorithm employs a greedy, top-down
strategy:
IMP QUESTION BANK:-
UNIT-3
 SYLLABUS:-
• Support Vector Machines: Working of SVM, SVM Concepts Support
Vectors, Hard Margin, soft Margin, Kernels, Advantages and
Disadvantages of SVM
• Probablistic Learning: Introduction to Bayes Learning, Interpretation
of Bayes Rule, Benefits and shortfalls of Bayesian Learning, Naïve
Bayes Classifier, Characteristics of Naïve Bayes
• Regression Methods: Linear Regression Models, Logistic Regression
 Support Vector Machines:-
• A Support Vector Machine (SVM) is a powerful and versatile
supervised machine learning algorithm primarily used for
classification and regression tasks. Its core idea is to find an optimal
decision boundary, called a hyperplane, that maximally separates
data points belonging to different classes in an N-dimensional space
(where N is the number of features).
• Its main objective is: To find the best separating boundary
(hyperplane) that maximizes the margin between different classes.
Example: Classifying Emails – Spam vs Not Spam
• Step 1: Features
• Each email is represented by features like:
• Number of links
• Number of capital words
• Presence of “offer”, “win”, “free”, etc.
• Let’s say the data looks like two clusters on graph:
• Spam emails: ●●●●●
• Not Spam emails: ○ ○ ○ ○ ○
• Step 2: Try to draw separating lines
• Many lines can separate the classes.
• Step 3: Choose the best line
• SVM chooses the line with maximum gap (margin) between closest ●
and ○.
• Step 4: Support vectors
• The emails that lie closest to the boundary are support vectors.
• Step 5: Prediction
• For a new email, SVM sees:
• On which side of the boundary it lies
→ Predicts spam or not spam
Concepts of SVM
Concepts of SVM
3. Support Vectors: These are the data points that lie closest to the
hyperplane (on the margin). They are the critical elements of the
training set because they directly influence the position and orientation
of the optimal hyperplane. If you move or remove any other data point,
the hyperplane won't change; only the support vectors matter
 Concept of Hard margin and Soft margin
• In the context of Support Vector Machines (SVM), the "margin" is the
empty space between the classes. Think of it as a "no-man's land"
that the algorithm tries to make as wide as possible to ensure the two
groups are clearly separated.
• There are two main ways to handle this margin: Hard Margin and Soft
Margin
• 1. Hard Margin SVM: The Perfectionist
• The Hard Margin approach assumes that your data is perfectly
separable. It is like a strict referee who refuses to allow even a single
mistake
• Rule: No data point is allowed to cross into the margin or be on the
wrong side of the boundary.
• The Goal: Find the widest possible "street" where every single "House
A" is on one side and every "House B" is on the other.
• The Problem: It is extremely sensitive to outliers. If you have just one
"red" dot accidentally sitting inside a cluster of "blue" dots, a Hard
Margin SVM will either fail completely (it can't find a solution) or
create a very tiny, messy boundary that won't work well for new data
(overfitting).
Advantages of Support Vector Machines
1. Effective in High-Dimensional Spaces
• SVM works extremely well when the number of features is very large
(e.g., thousands of words in a text document).
• Why?
Because the algorithm focuses only on the support vectors, not the
entire dataset.
Even if the data has many dimensions, SVM tries to find a hyperplane
that separates the classes with the maximum margin, which reduces
overfitting.
• Example: Email spam classification involves thousands of word
features — SVM performs exceptionally well in this scenario.
2. Works Well with a Clear Margin of Separation
• If the classes can be separated clearly with a straight line or curve, SVM
finds the best possible boundary.
• Why?
It maximizes the margin — the distance between the separating line and
the nearest points.
Better generalization → Better accuracy on unseen data.

3. Effective with Non-linear Data Using Kernel Trick

• Many real-world datasets are not linearly separable (cannot be separated
by a straight line).
• SVM uses kernels (like RBF, Polynomial) to convert the data into higher
dimensions where it becomes linearly separable.
• Example:
You cannot separate two circular clusters in 2D, but when SVM applies RBF
kernel, it maps data to a higher dimension where separation becomes
possible.
4. Robust Against Overfitting
• SVM is designed to maximize margin and depends only on support
vectors (very few points).
• This prevents the model from memorizing noise or irrelevant patterns
— very helpful when data is small.

5. Works Well with Small to Medium-Sized Datasets

• Unlike neural networks, SVM does not require very large datasets.
• Even if you have 500–2000 rows, SVM can still produce great
accuracy.
Disadvantages of Support Vector Machines
1. Not Suitable for Large Datasets
• If the dataset has lakhs (hundreds of thousands) of rows, SVM becomes
slow. Because training involves solving a complex quadratic optimization
problem that becomes computationally expensive.
2. Choosing the Right Kernel is Difficult
• Performance of SVM heavily depends on:
• Which kernel you choose (linear, polynomial, RBF)
• Kernel hyperparameters
• C value
• Gamma value
• Selecting wrong combination → low accuracy.
• Students often say:
“Changing gamma slightly changed accuracy from 90% to 40%.”
• This is normal for SVM
3. Less Effective When Data Has Too Much Noise
• If the dataset contains many outliers or overlapping classes, SVM
struggles.
Because the margin can get disturbed — SVM is sensitive to those
“support vectors”.
• Example:
If spam emails and non-spam emails look very similar, SVM may
misclassify.
4. Memory Intensive for Large Datasets
• Needs to store support vectors and compute kernel functions → high
memory usage.
• In comparison, models like Logistic Regression are much lighter
5. Hard to Interpret
• SVM boundary is based on support vectors in high-dimensional space.
• This makes SVM:
• Hard to visualize
• Hard to explain to non-technical users
• Not interpretable like Decision Trees
• You cannot easily explain why SVM predicted something.
 Working of SVM:-
• Step 1: Input the Training Data
• SVM starts with a labeled training dataset
• Each data point consists of:
• Features (x)
• Class label (y)

• Step 2: Check Data Linearity

• SVM checks whether data is:
• Linearly separable, or
• Non-linearly separable
• If linearly separable → Linear SVM
If not → Apply Kernel Trick
• Step 3: Select Kernel Function
• If data is non-linear, SVM maps data to a higher dimension.
• Common kernels:
• Linear
• Polynomial
• Radial Basis Function (RBF)
• Sigmoid
• Purpose:
• Convert complex data into linearly separable form
• Step 4: Define the Hyperplane
• SVM defines a decision boundary:
• 𝑤⋅𝑥+𝑏 =0
• Where:
• 𝑤= weight vector
• 𝑏= bias
• This hyperplane separates different classes.
Step 5: Identify Support Vectors
• SVM finds data points closest to the hyperplane
• These points are called support vectors
• They play a key role in determining the margin
• Removing other points does not affect the hyperplane.
Step 6: Maximize the Margin
• SVM computes two parallel margin lines:
• 𝑤 ⋅ 𝑥 + 𝑏 = +1
• 𝑤 ⋅ 𝑥 + 𝑏 = −1
• Distance between them is the margin
• SVM chooses the hyperplane with maximum margin
• Step 7: Handle Misclassification (Soft Margin)
• If data contains noise:
• Introduce slack variables (ξ)
• Use C parameter to control error
• 𝑦𝑖 𝑤 ⋅ 𝑥𝑖 + 𝑏 ≥ 1 − 𝜉𝑖

• Step 8: Solve Optimization Problem

• SVM solves a constrained optimization problem:
1
• Minimize ∣∣ 𝑤 ∣∣2 +𝐶 σ 𝜉𝑖
2
• This ensures:
• Large margin
• Minimal classification error
• Step 9: Construct the Final Model
• Hyperplane is finalized
• Support vectors are stored
• Kernel parameters are fixed

• Step 10: Classify New Data

• For a new data point 𝑥:
• 𝑓 𝑥 = sign 𝑤 ⋅ 𝑥 + 𝑏
• If output is +1 → Class 1
• If output is −1 → Class 2
 Probablistic Learning:
• Introduction to Bayes Learning:
• Bayesian Learning is a probabilistic approach to machine learning
where we update our belief about a model or hypothesis as new data
arrives, using Bayes’ Theorem.
• Instead of learning a single fixed model, Bayesian learning maintains
probabilities over possible hypotheses.
• Learning = updating beliefs using probability
• We start with an initial belief (prior), observe data, and then update
that belief to get a posterior probability.
• Example: Spam Email Classification
• Problem
• An email arrives. Is it Spam (S) or Not Spam (¬S)?
 Interpretation of Bayes’ Rule (Bayesian Learning)
• Bayes’ Rule is not just a formula — it tells us how to logically update our
belief when new evidence is observed.

• Interpretation of Each Term

• 1. Prior Probability – 𝑃 𝐻
• Represents initial belief about a hypothesis before seeing data
• Based on past experience or domain knowledge
• Interpretation:
• “How likely did I think this hypothesis was before observing evidence?”
• 2. Likelihood – 𝑃 𝐷 ∣ 𝐻
• Probability of observing the data assuming the hypothesis is true
• Interpretation:
• “If this hypothesis were true, how likely is the observed data?”

• 3. Evidence – 𝑃 𝐷
• Probability of observing the data under all possible hypotheses
• Acts as a normalizing constant
• Interpretation:
• “How common is this data overall?
• Posterior Probability – 𝑃 𝐻 ∣ 𝐷
• Updated belief after seeing the data
• Interpretation:
• “Given the evidence, how likely is this hypothesis now?”
 Benefits (Advantages) of Bayesian Learning
1. Handles Uncertainty Explicitly
• Represents uncertainty using probability distributions instead of single point estimates.
• Useful in real-world problems where data is noisy or incomplete.
• Example: Medical diagnosis, risk analysis.

2. Incorporates Prior Knowledge

• Can include expert knowledge through prior probabilities.
• Very useful when data is scarce.
• Example: Fraud detection using historical trends.

3. Works Well with Small Datasets

• Prior information compensates for lack of large training data.
• Produces reasonable predictions even with limited samples.
4. Continuous Learning
• Model can be updated incrementally as new data arrives.
• No need to retrain from scratch.
• Example: Online recommendation systems.

5. Avoids Overfitting
• Bayesian averaging considers all hypotheses, not just one best model.
• Regularization happens naturally through priors.

6. Probabilistic Predictions
• Outputs confidence levels along with predictions.
• Helps in decision-making under risk.
 Shortfalls (Limitations) of Bayesian Learning
• 1. High Computational Complexity
• Explanation
• Bayesian learning requires computing the posterior distribution:
• 𝑃 𝐻∣𝐷 ∝𝑃 𝐷∣𝐻 𝑃 𝐻
• This often involves complex integrations or summations over a large
hypothesis space.
• Exact inference is usually intractable for real-world problems.
• Example
• In Bayesian Neural Networks, computing posterior weights requires
integrating over millions of parameters → practically impossible without
approximation.
• Impact: Slower training, high CPU/GPU cost.
2. Difficulty in Choosing Prior Probabilities
• Priors represent initial beliefs.
• Poorly chosen priors can dominate the posterior, especially with small datasets.
• Priors can be subjective, reducing objectivity.
• Example
• In medical diagnosis, if a doctor assumes a disease is very rare (low prior), the model
may underestimate risk even after strong test results.
Impact: Biased predictions.

3. Scalability Issues with Large Datasets

• Bayesian inference grows expensive as:
• Number of samples increases
• Dimensionality increases
• Memory and computation scale poorly.
• Example
• In real-time ad-click prediction with millions of users, full Bayesian inference becomes
impractical.
• Impact: Not suitable for big data without simplification.
4. Need for Approximation Method
• Exact solutions rarely exist.
• Requires methods like:
• Markov Chain Monte Carlo (MCMC)
• Variational Inference
• These methods introduce approximation errors.
• Example
• Using Variational Inference in topic modeling may converge faster but produce inaccurate posterior
distributions.
• Impact: Reduced reliability.
5. Complex Mathematical Framework
• Requires deep understanding of:
• Probability theory
• Statistics
• Integration
• Harder to implement and debug.
• Example
• Students often struggle to implement Bayesian networks compared to decision trees or k-NN.
Impact: Steep learning curve.
6. Sensitivity to Model Assumptions
• Bayesian models assume:
• Correct likelihood function
• Correct independence assumptions
• Wrong assumptions → misleading results.
• Example
• Naive Bayes assumes feature independence. In text classification, correlated words violate this
assumption, reducing accuracy.
• Impact: Model mismatch.

7. Interpretability Issues in Complex Models

• Explanation
• Simple Bayesian models are interpretable.
• Complex ones (Bayesian deep learning) become black boxes.
• Example
• Posterior distributions of deep Bayesian models are difficult to explain to non-technical
stakeholders.
• Impact: Low transparency.
 Naive Bayes Classifier
• Naïve – Meaning “simple assumption”
• Naïve Bayes Classifier is a supervised machine learning algorithm
used for classification.
It is based on probability and Bayes’ Theorem.
• It predicts which class a data point belongs to by calculating
probabilities.
• The algorithm is based on Bayes' Theorem, a mathematical formula
used to determine the probability of an event based on prior
knowledge.
• Definition:- Naïve Bayes Classifier is a probabilistic supervised
learning algorithm based on Bayes’ theorem that assumes
independence among features and predicts the class with the
highest posterior probability
 How It Works (Step-by-Step)
• Let’s say we want to classify whether a message is Spam or Ham (Normal).
• Calculate Prior Probability: How many messages in our dataset are Spam
vs. Ham? (e.g., 40% are Spam).
• Calculate Likelihood: Look at a specific word, like "Winner." How often
does "Winner" appear in Spam emails versus Ham emails?
• Multiply it All: For a new email, the algorithm multiplies the probabilities
of all the words it contains.
• Pick the Winner: It compares the final score for "Spam" and "Ham."
Whichever score is higher is the category the message gets assigned to.
 How Naïve Bayes Works (Step-by-Step)
Step 1: Calculate Prior Probability
• Probability of each class in the dataset.
• Example:
• 60 spam emails out of 100
→ P(Spam) = 0.6
• 40 not spam emails
→ P(NotSpam) = 0.4
• Step 2: Calculate Likelihood
• Probability of features given the class.
• Example:
• P(word = “free” | Spam)
• P(word = “free” | NotSpam)
Step 5: Choose the Class with Highest Probability
That class becomes the prediction .
 Simple Numerical Example
Classify an email containing “free” and “offer”
Word Spam (50) Not Spam (50)
free 20 5
offer 15 3
Advantages of Naïve Bayes Classifier

• 1. Simple and Fast

• Explanation:
Naïve Bayes uses basic probability calculations.
It does not need complex computations or iterations
• Example:
In email spam filtering, thousands of emails can be classified in
seconds using Naïve Bayes, making it suitable for real-time systems.
• Compared to algorithms like SVM or Neural Networks, Naïve Bayes is
much faster
2. Works Well with Large Datasets
• Explanation:
Naïve Bayes scales very well as the dataset size increases.
• More data → better probability estimates
• Memory usage remains low
• Example:
Google and Gmail process millions of emails daily.
Naïve Bayes can efficiently handle such huge datasets without
performance issues.
• Large data improves accuracy instead of slowing the model.
3. Requires Less Training Data
• Explanation:
Unlike deep learning models, Naïve Bayes can work well even with
small training datasets.
• Reason:
• Uses probability
• Does not need to learn complex patterns
• Example:
In medical diagnosis, if only limited patient data is available, Naïve
Bayes can still predict diseases using prior probabilities.
• Useful when collecting data is expensive or difficult.
 Limitations of Naïve Bayes Classifier
1. Assumes Feature Independence
• Explanation:
Naïve Bayes assumes all features are independent, which is often not
true in real-world data.
• Example:
In spam emails:
• Words “free” and “offer” often appear together
• Naïve Bayes treats them as independent
• This can reduce accuracy when features are highly correlated.
3. Less Accurate for Complex Data
• Explanation:
Naïve Bayes cannot capture complex relationships between features.
• Example:
In image recognition:
• Pixel relationships matter
• Naïve Bayes fails to capture spatial patterns
• Algorithms like Neural Networks or Decision Trees perform better
for such tasks.
 Characteristics of Naïve Bayes
1. Probabilistic Classifier:
• Naïve Bayes doesn't just assign a category; it calculates the probability of each
possible outcome. For any given input, it determines the probability that it
belongs to class A, class B, etc., and selects the most likely one
2. Assumption of Conditional Independence (The "Naïve" Part):
• The core characteristic is its strong, and often unrealistic, assumption that the
presence of one feature does not affect the presence of any other feature,
given the class
• For example, if you are classifying an email as spam, the algorithm assumes
that the presence of the word “coin" is independent of the presence of the
word "money," given the email is spam. While in reality these words often
appear together, the algorithm treats them separately, which simplifies
computation significantly
3. Fast and Efficient:
Due to the independence assumption, the calculations required are
straightforward and efficient, making it very fast to train and make predictions
compared to more complex algorithms .
4. Requires Less Training Data:
Because it only needs to learn the probabilities of individual features within
each class, it often performs well even with relatively small training datasets [
5. Handles High-Dimensional Data Well:
It is particularly effective with datasets where the number of features
(dimensions) is very high, such as text classification, where every word can be a
feature
6. Strong Performance on Text Classification:It is a widely used and highly
effective benchmark algorithm for tasks like spam detection, sentiment
analysis, and categorizing documents
6. Can Handle Missing Data
Naïve Bayes can ignore missing values during probability calculation.
• Example:
In student performance prediction:
• Attendance missing
• Marks available
• Naïve Bayes still classifies correctly using available features.
Regression Methods:-
[Link] Regression Models
Definition:-
Linear Regression is a supervised machine learning algorithm used to
predict a continuous value (number) by finding a linear relationship
between input variables (features) and output variable (target).
It helps us predict one value using another value with the help of a straight
line.
• Examples of predictions:
• Predict house price based on area
• Predict salary based on years of experience
• Predict sales based on advertising cost
• Predict marks based on study hours
Example:- Study Hours (X)
1 hour
Marks (Y)
35
2 hours 45
• Study Hours vs Marks (Linear Regression)
3 hours 55
4 hours 65

Step 1: Understand the Relationship

• When a student studies more hours, the marks increase
• Every extra 1 hour of study increases marks by 10
• This increase is constant, so the relationship is a straight line
• this is why linear regression can be used
Step 4: Find the Value of m (Slope)
Look at the data:
From 1 hour → 2 hours
Marks increase from 35 → 45 (+10)
So,
m=10
For every 1 extra hour studied, marks increase by 10.
Predicted Marks = 55
• Why is it called “Regression”?
• Regression means estimating or predicting values based on
relationships in data.
• Even though the line may go upward, the name comes from
statistical regression analysis, not decreasing values.
• Types of Linear Regression:-
 Logistic Regression
• Think of Logistic Regression as the "Yes or No" algorithm. Despite the
name, it isn’t used to predict a continuous number (like a house price); it’s
used to classify things into categories.
• Most often, it helps us decide if something belongs to a group (1) or
doesn't (0).
• Imagine you are trying to predict if a student will Pass or Fail an exam
based on how many hours they studied.
• Linear Regression would try to predict a specific score (e.g., "You will get
an 82%").
• Logistic Regression predicts the probability of an outcome (e.g., "There is
an 85% chance you will pass").
• If the probability is greater than 50%, the model classifies the student as
"Pass." If it’s lower, "Fail."
Examples:

• Will the student pass or fail?

• Is the email spam or not spam?
• Will the customer buy or not buy?
• Is the tumor benign or malignant?
• So the output is usually:
• 0 or 1
• No or Yes
• False or True
How it Works (The "S" Curve)
• In standard math, a line can go to infinity. But probability must stay
between 0 and 1. To fix this, Logistic Regression uses a special
mathematical function called the Sigmoid Function.
• It takes any number and squashes it into an "S" shaped curve.
• At the top of the S: The probability is close to 1 (True/Yes).
• At the bottom of the S: The probability is close to 0 (False/No).
• In the middle: This is the "decision boundary" (usually 0.5).
How Logistic Regression Works (Step by Step)
QUESTION BANK
UNIT-4
SYLLABUS:-
• Performance Evaluation: Classification Metrics-Accuracy, Sensitivity,
Precision, F1 Score, ROC/AUC Curve, Cross Validation .
• Unsupervised Learning: Concept of unsupervised Learning,
Importance and Challenges of unsupervised Learning, Clustering and
its applications
• Hierarchical Clustering: Introduction, Types of Hierarchical Clustering,
Issues with Hierarchical Clustering,
Performance Evaluation
• In Machine Learning, after a model is trained, it is necessary to evaluate its
performance to understand how well it predicts unseen data.
• This process is called Performance Evaluation.
• For classification problems, performance is measured using metrics such
as:
• Accuracy
• Precision
• Recall
• F1-Score
• Confusion Matrix
• Among these, Accuracy is the most basic and commonly used metric.
Accuracy:-
• Accuracy measures the overall correctness of a classification model.
• Definition of Accuracy:-
• Accuracy is defined as the ratio of correctly classified instances to the
total number of instances in the dataset.
• Formula of Accuracy
Number of Correct Predictions
• Accuracy =
Total Number of Predictions
• In machine learning, Accuracy is the most intuitive performance
metric. It represents the proportion of total predictions that were
correct (both positive and negative)
• Think of it as the "grade" on a test: if there are 100 questions and you
get 90 right, your accuracy is 90%
• The Formula
• To calculate accuracy, we look at the results of a Confusion Matrix,
which tracks True Positives (TP), True Negatives (TN), False Positives
(FP), and False Negatives (FN)
• Simple Real-Life Example (Student Pass/Fail)
• Suppose a model predicts whether students Pass or Fail.
Student Actual Result Model Prediction
1 Pass Pass
2 Pass Pass
3 Fail Fail
4 Pass Fail
5 Fail Pass

Model Accuracy = 60%

 Sensitivity (Recall / True Positive Rate)
• Definition:
• Sensitivity measures the ability of the model to correctly identify actual
positive cases.
• It answers the question:
• Out of all actual positive instances, how many did the model correctly
identify?

• Formula:
𝑇𝑃
• Sensitivity (Recall) = 𝑇𝑃+𝐹𝑁
• Interpretation:
• High sensitivity → Model misses fewer positive cases
• Low sensitivity → Model fails to detect positives
• When Sensitivity is Important:
• Disease detection
• Fraud detection
• Spam filtering
• Any case where missing a positive is costly
 Precision
• Precision measures how accurate the positive predictions are.
• It answers the question:
• Out of all instances predicted as positive, how many are actually positive?

• Formula:
𝑇𝑃
• Precision =
𝑇𝑃+𝐹𝑃

• Interpretation:
• High precision → Few false alarms
• Low precision → Many false positives

• Example (Spam Email Detection)

• Using the same confusion matrix:
40 40
• Precision = = = 88.89%
40+5 45

When the model predicts an email as spam, it is correct ~89% of the time.
• When Precision is Important:
• Spam detection
• Search engines
• Recommendation systems
• Situations where false positives are costly
F1 Score
• The F1 Score is the "Goldilocks" metric. It is the harmonic mean of
Precision and Recall. It gives you a single score that balances both.
• If you have a model with 99% Precision but only 1% Recall, your F1
Score will be very low. It only stays high if both metrics are strong. It’s
particularly useful for imbalanced datasets.
• Formula:
Precision×Sensitivity
• F1−Score = 2 ×
Precision+Sensitivity
Summary of all
• Example
• Imagine a Security Camera looking for burglars:
• High Sensitivity: The camera triggers for anything that moves (cats,
shadows, burglars). You never miss a burglar (low FN), but you get
many false alarms (low Precision).
• High Precision: The camera only triggers when it is 100% sure it sees
a human face. You get zero false alarms (high Precision), but you
might miss a burglar wearing a mask (low Sensitivity).
• High F1 Score: The camera is tuned to catch most burglars while
keeping the false alarms to a minimum.