MODULE-3

Techniques in AI
• Artificial Intelligence combines massive data, iterative processing, and intelligent algorithms to
enable software to deduce patterns in data and perform tasks like humans.
• AI is an umbrella term that encompasses several specialized fields, each targeting different aspects
of simulating human intelligence. These sub-domains are often interconnected but have specific
goals and methodologies. Think of AI like a large tree, and the sub-domains are its branches, each
branching out to tackle a different problem or concept.
Machine Learning (ML)

• Machine learning is a sub-domain of AI focused on developing algorithms that enable

computers to learn from data and make decisions without human intervention.
• Example: Email spam filters use machine learning to identify and filter out spam
messages based on patterns in previous emails.
Deep Learning (DL)

• Deep learning is a subset of machine learning where algorithms are designed to learn
from large amounts of data using neural networks with many layers (hence the term
"deep").
• Example: Self-driving cars use deep learning to process real-time data from cameras and
sensors to make driving decisions.
 Natural Language Processing (NLP)
• NLP is a branch of AI that helps machines understand and process human language, enabling
communication between humans and computers.
• The goal is to allow machines to understand, interpret, and generate human language, making interactions
feel more natural.
• In this process, machines "learn" how words and sentences are structured by looking at lots of examples.
• For example, in sentiment analysis (a common NLP task), a machine learns from many reviews and can
predict if a new review is positive or negative based on the words used.
• In short, NLP in ML lets machines read, understand, and respond to human language without needing
specific instructions for each task.

• Applications:
• Chatbots: AI-powered chatbots like customer service bots on websites use NLP to understand user queries
and respond in natural language.
• Voice Assistants: Siri, Alexa, and Google Assistant are examples of NLP-based systems that process and
respond to voice commands.
Cognitive Computing

• Cognitive computing aims to simulate human cognitive processes such as learning,

reasoning, and perception, allowing for human-like interaction with machines.
• To create systems that can interpret data in a human-like way and provide intelligent
responses.

• Applications:
• Voice-Activated Assistants: Like Amazon Alexa, these systems can interact naturally
with users through voice commands.
• AI-Based Customer Support: Cognitive systems that understand customer queries and
provide solutions without human intervention.
 Computer Vision

• Computer Vision is a field of AI that enables machines to see, interpret, and understand the world in
images or videos, just like humans do. It's about teaching computers to analyze visual data (like
photos, videos, and live camera feeds) to make decisions or perform tasks based on that information.

• Real-World Applications of Computer Vision:

• Facial Recognition: Used in smartphones and security systems to recognize and verify faces.
• Autonomous Vehicles: Self-driving cars use computer vision to recognize road signs, pedestrians,
and other vehicles.
• Medical Imaging: Detecting diseases in X-rays, CT scans, and MRIs (e.g., identifying tumors).
• Retail: In stores, computer vision can track inventory, monitor customer behavior, or even assist in
checkout (e.g., Amazon Go stores).
• Agriculture: Drones and cameras can monitor crop health by analyzing images of fields.
 Neural Networks
• Neural Networks are a foundational concept in Artificial Intelligence (AI) and Machine Learning (ML). They are
inspired by how the human brain processes information and consist of a network of nodes, also known as neurons,
that work together to learn from data and make predictions or decisions.
• Neural networks are designed to recognize patterns, and they are used in various tasks such as image recognition,
speech processing, and even playing games.
• A neural network consists of layers of interconnected neurons, and each layer processes input data in a unique way.
These neurons are similar to the neurons in our brain that help process and pass information.

• Key Components(Structure):
• Neurons (Nodes): These are the basic units in a neural network, where the actual computation happens.
• Layers: A neural network consists of multiple layers of neurons. Each layer performs a certain function on the data.
• Input Layer: This is where the data enters the network. It consists of neurons that represent the features or attributes
of the input data.
• Hidden Layers: These are the layers between the input and output. They process the data received from the input
layer. Complex neural networks can have many hidden layers (called Deep Neural Networks).
• Output Layer: This layer gives the final output, such as a classification result or a prediction.
 How It Works:
Step 1: Forward Pass
• The input data is fed into the input layer.
• This data is then passed through the hidden layers. At each hidden layer, each neuron looks at the input, applies some math
(multiplying by weights), and sends the result to the next layer.
Step 2: Activation Function
• After the data goes through the neurons, an activation function decides if that neuron should "fire" (pass data) or not.
• Think of it like a gate: if the input is strong enough, the neuron lets the data pass. If not, it stops the data.
Step 3: Loss Calculation
• After the data reaches the output layer, the network makes a prediction. But the prediction might not be perfect, so we
compare the result to the correct answer (the real label).
• The difference between the predicted output and the real answer is called the loss (error).
Step 4: Backpropagation
• Now, the network learns from its mistakes. It sends the error back through the network (backpropagation), adjusting the
weights in each neuron to make the network better next time.
Step 5: Iteration (Training)
• This process (forward pass + backpropagation) happens many times, with the network constantly improving to minimize the
error. Over time, it becomes good at making predictions.
Applications of Neural Networks

• Image Recognition:
Neural networks are widely used in computer vision tasks, such as identifying objects or
faces in images.
• Speech Recognition:
Used in virtual assistants (like Siri, Alexa) to convert speech into text and understand
commands.
• Natural Language Processing (NLP):
Neural networks power language models for tasks like machine translation and sentiment
analysis.
• Medical Diagnosis:
Applied in healthcare for detecting diseases (e.g., cancer detection from medical images).
 Machine Learning Model

• A Machine Learning Model helps a machine learn from experience in order to solve real-world
problems.

• Task (T): The Problem to Be Solved

• The task (T) in machine learning refers to the real-world problem that the model is trying to solve.
This could be anything from predicting future values, classifying data, recognizing patterns, etc.

• Examples of ML Tasks:
• Predicting sales: For a business, we may use machine learning to predict how many units of a
product will be sold next month based on past data.
• Classifying emails: A common machine learning task is to classify whether an email is spam or not.
• Experience (E): Learning from Data
In machine learning, experience (E) is gained from the data provided to the model. This
data is the knowledge that the model uses to improve its performance. Just like how humans
learn from their experiences, machines also "learn" from data, meaning they adapt and
improve over time by identifying patterns, relationships, or structures in the data.

• Performance (P): Measuring Success

Once the model has learned from the experience, it needs to be evaluated to see how well it
performs on the given task. This evaluation is done using a performance measure (P), which
indicates how accurately the model is solving the task. The performance is analyzed based
on specific metrics.
• The task (T) defines what we want the machine to do — for example, classifying emails
as spam or not.
• The experience (E) comes from the data that is provided to the model, like past emails
labeled as spam or not spam.
• The performance (P) measures how well the model accomplishes the task, using metrics
like accuracy, precision, and recall.

• Email Spam Classification

• Let’s apply the concepts to a real-world example of email spam classification:
• Task (T): Classifying emails as spam or not spam.
• Experience (E): The model learns from a dataset of emails that are already labeled as
spam or not spam. The features might include words used in the email, the sender's
address, and the subject line.
• Performance (P): After training, the model is tested on a new set of emails. If it classifies
90% of the emails correctly as spam or not spam, we say the model’s accuracy is 90%.
Types of Machine Learning Algorithms

• Machine learning algorithms can be broadly categorized based on the way they learn from
data and the kind of tasks they perform.
• Machine Learning (ML) can be broadly divided into four main categories based on the
way the model learns from data:
• Supervised Learning
• Unsupervised Learning
• Semi-Supervised Learning
• Reinforcement Learning
• Each of these types uses different methods and algorithms to make predictions or
decisions based on data. Let’s explore each one in detail.
Supervised Learning
• In Supervised Learning, the model is trained on labeled data, meaning the data includes
both input features and the correct output (target). The goal is for the model to learn the
relationship between the input and the output, so it can make accurate predictions on new,
unseen data.
• The model learns from input-output pairs.
• The algorithm tries to map input data to the correct output using labeled training data.

• Example:
• Task: Classifying emails as spam or not spam.
• Data: A set of emails with labels (spam or not spam) and features like the subject line,
sender, and content.
• Goal: The model learns from these labeled examples and can later classify new emails as
spam or not spam.
• Common Algorithms:
• Linear Regression: Used for predicting continuous values (e.g., predicting house prices
based on features like area, number of rooms, etc.).
• Logistic Regression: Used for binary classification (e.g., predicting whether an email is
spam or not).
• Decision Trees: Used for classification and regression, where a tree-like model makes
decisions based on input features.
• Random Forest: An ensemble method that combines multiple decision trees to improve
accuracy and avoid overfitting.
• Support Vector Machines (SVM): Used for classification tasks, it finds the best boundary
(hyperplane) that separates data into different classes.
• K-Nearest Neighbors (KNN): A simple algorithm where the model classifies new data
based on the majority class of its nearest neighbors.
• Naive Bayes: A probabilistic classifier based on Bayes’ theorem, often used for text
classification.
 Unsupervised Learning
• In Unsupervised Learning, the model is trained on unlabeled data, meaning the dataset only includes
input features without any output labels. The goal is to find hidden patterns or structures in the data.
• Key Characteristics:
• The model is not provided with any output labels.
• The algorithm tries to find patterns or groupings in the data by identifying similarities and
relationships.

• Example:
• Task: Grouping customers based on their buying behavior.
• Data: A dataset of customer purchases, with no labels provided (i.e., we don’t know which customer
belongs to which group).
• Goal: The model clusters customers into different groups based on their purchasing habits (e.g.,
high-spenders, frequent shoppers, etc.).
 Semi-Supervised Learning
• Semi-Supervised Learning lies between Supervised and Unsupervised Learning. It uses a small
amount of labeled data combined with a large amount of unlabeled data. The labeled data helps the
model understand the basic relationships, while the unlabeled data allows the model to learn from a
larger pool of data, making it more robust.
• Key Characteristics:
• Labeled Data: A small portion of the dataset is labeled (like in supervised learning).
• Unlabeled Data: A larger portion of the dataset is unlabeled (like in unsupervised learning).
• The model can learn from both labeled and unlabeled data to improve its accuracy.
• Example:
• Task: Image classification where most images are unlabeled.
• Data: A small set of labeled images (e.g., 100 images labeled as either "cat" or "dog") and a large
set of unlabeled images.
• Goal: The model can initially learn from the labeled data, then use the unlabeled images to further
refine its understanding and improve classification accuracy.
 Reinforcement Learning (RL)
• In Reinforcement Learning, the model (also known as an agent) learns by interacting with an
environment and receiving feedback in the form of rewards or penalties. The goal is to maximize
cumulative reward over time.
• Key Characteristics:
• The agent learns by trial and error, exploring the environment and receiving feedback.
• Actions: The model takes actions based on its current state.
• Rewards/Penalties: Feedback is provided in the form of rewards (positive) or penalties (negative)
.
• Example:
• Task: Teaching an agent to play a game (e.g., Chess or Go).
• Environment: The game’s board and rules.
• Goal: The agent learns by playing many games, gradually improving its strategy by receiving
rewards when it wins or penalties when it loses.
How Reinforcement Learning Works

• Process:
• Agent starts in an initial state.
• Actions are taken by the agent to interact with the environment.
• Based on actions, the agent receives rewards or penalties.
• The agent uses these rewards/penalties to learn and decide the best actions for future
steps.
• Key Concept: RL uses a system of reward and punishment to guide the agent towards the
optimal behavior.
 Types of Reinforcement

• Positive Reinforcement:
• Positive reinforcement is when an agent’s behavior is encouraged by providing a reward.
The agent learns that performing a certain action leads to a positive outcome, so it is more
likely to repeat that action in the future.
• Example: A robot receives a reward every time it reaches a target.

• Negative Reinforcement:
• Negative reinforcement is when an agent’s behavior is strengthened by removing or
avoiding a negative condition. The agent learns to perform an action to avoid something
bad.
• Example: A robot avoids a fire hazard (negative condition) to not lose points.
 Applications of Reinforcement Learning
• Robotics & Industrial Automation:
• RL helps robots learn to perform tasks like assembly, packing, or welding in factories by interacting with the
environment and improving their performance over time.
• 2. Game Playing (e.g., Chess, Go):
• RL has been used in creating intelligent game-playing agents that can learn strategies by playing against
themselves or others.
• 3. Self-Driving Cars:
• RL is used to help self-driving cars learn to make decisions, such as steering, braking, and accelerating, based on
real-world driving conditions.
• 4. Personalized Education Systems:
• RL can create training systems that adapt to the student’s learning pace, providing customized lessons and
feedback.
• 5. Data Processing:
• RL algorithms help in optimizing data processing tasks like improving database queries or managing cloud
resources.
Aspect Supervised Learning Unsupervised Learning

Definition A type of machine learning where the model is A type of machine learning where the
trained on labeled data (input-output pairs). model is trained on unlabeled data (only
inputs).
Data Uses labeled data (data with known outcomes). Uses unlabeled data (data with no
predefined outcomes).
Output Predicts a specific output (e.g., classification or Identifies patterns or groupings in the
regression). data (e.g., clustering or dimensionality
reduction).
Goal To map input data to a correct output based on To discover underlying patterns or
labeled examples. structure in the data.
Training Process Requires a training dataset that includes both input Only requires input data; no labels or
features and the corresponding output labels. predefined outcomes.

Complexity Typically more controlled as the learning process More exploratory as the model must find
is guided by labeled data. patterns without supervision.
 Purpose of Validation Set, Training Set, and Test Set
in Machine Learning
• In machine learning, dividing the dataset into three main subsets—Training Set, Validation Set,
and Test Set—is essential for building robust models.
• Training Set
• The Training Set is used to train the model. The model learns patterns, relationships, and structures
from this data to make predictions or classifications.
• The training set provides both the input features (e.g., images, text, numbers) and the target output
(the correct label or value) to the algorithm. The model uses this data to adjust its internal
parameters (e.g., weights in a neural network).
• Example: If you're building a model to predict house prices, the training set would consist of past
data of houses with features like size, location, and price. The model learns the relationship
between the features and the price.
• Size: The training set is typically the largest portion of the dataset, as it is the main data used to
teach the model.
• Validation Set
• The Validation Set is used to tune the model's parameters and help select the best model
during training. It allows you to assess the model's performance during training and make
adjustments to avoid overfitting.
• The validation set helps in hyperparameter tuning, which involves adjusting settings like
learning rate, regularization strength, and the number of layers in a neural network. After
training the model on the training set, you evaluate its performance on the validation set
to see how well it generalizes to unseen data.
• The validation set gives an indication of how the model is performing on data that it
hasn’t seen before but is still part of the overall training process. If the model performs
poorly on the validation set, adjustments are made to improve it.
• Example: Continuing with the house price prediction model, you might use the validation
set to fine-tune the model’s parameters, such as the learning rate or the complexity of the
model, to get the best possible prediction results.
• Size: Typically, the validation set is around 10-20% of the total dataset.
• Test Set
• The Test Set is used to evaluate the final model’s performance on completely unseen data.
It provides an unbiased estimate of the model’s accuracy and generalizability after all
training and tuning are done.
• The test set is only used after the model has been fully trained and validated. It helps
assess the final model’s generalization ability—i.e., how well it will perform on new,
unseen data in the real world. The test set ensures that the model is not overfitting to the
training data and is capable of handling new situations.
• Example: After training and tuning the model for predicting house prices using the
training and validation sets, the test set is used to check how well the model can predict
prices for new houses it has never seen.
• Size: The test set typically makes up about 10-20% of the dataset and is kept completely
separate from the training and validation sets.
Splitting data into training and test sets is crucial for ensuring the generalization and
performance of a machine learning model. Here’s why this split is necessary:
• Preventing Overfitting(Overfitting occurs when a model learns the training data too
well—including its noise and irrelevant details—so it performs very well on the training
data but poorly on new, unseen data. This happens because the model is memorizing the
data instead of learning patterns that generalize.)
• Evaluating Model Performance on Unseen Data
• Ensuring Model Generalization(Generalization means that the model performs well on
both the training data and new, unseen data.

• Remember “Increasing the complexity of a machine learning model, such as using deep
neural networks, does not always improve performance. While complex models can fit
training data well, they may suffer from overfitting and poor generalization on unseen
data, leading to worse performance.”
Outliers in Machine Learning

• Outliers are data points that differ significantly from the rest of the data in a dataset. They
are unusually high or low compared to other values, and they don't follow the same
pattern or trend as the majority of the data.

• Examples:
• In a test score dataset: If most students scored between 50 and 80, but one student scored
10, that score is an outlier.
• In a list of people’s ages: If most people are aged between 20 and 40, but one person is 95
years old, that age is an outlier.
Regression
• Regression is a statistical method used to understand the relationship between a dependent variable (the outcome we
are trying to predict) and one or more independent variables (the factors that influence the outcome). The goal of
regression is to model this relationship so that we can use the independent variables to predict or estimate the
dependent variable.

• Why Use Regression?

• Predicting Future Values:
Once we have the regression model (the equation), we can use it to predict prices for new houses with different
sizes, even if we don’t have the exact price for those houses.
• Understanding Relationships:
Regression helps us understand how different factors (like house size) influence outcomes (like price). For instance,
if we know the slope is 100, we understand that every square foot of house size increases the price by $100.
• Making Data-Driven Decisions:
In business, regression can help make decisions, such as how much to invest in advertising to increase sales or how
to price products based on various factors.
Types of Regression

• Linear Regression: For continuous data with a linear relationship.

• Logistic Regression: For binary categorical data (0 or 1, True or False).
• Suppose you work for a real estate company, and you want to predict the price of a house
based on its size in square feet. The price of a house is the dependent variable (the one we
want to predict), and the size of the house (measured in square feet) is the independent
variable (the factor that we think affects the price).
• Now, you have data for several houses in the area, like:

House Size (sq ft) Price (in $)

1000 200,000

1500 250,000

2000 300,000

2500 350,000
Linear regression

• Linear regression is used when there is a linear relationship between the dependent
variable (target) and the independent variables (predictors).
• The equation for linear regression is:
Y=bX+C

Where:
• Y is the dependent variable (the value you want to predict)
• X is the independent variable (the predictor)
• b is the slope of the regression line
• C is the intercept (the value of Y when X=0)
• Use Cases:
• Predicting continuous values like sales, temperature, or price.
• Example: Predicting a car's price based on its age, mileage, and condition.
Pros:
• Simple and easy to interpret.
• Fast to implement.
• Good for small data sets where the relationship is roughly linear.

Cons:
• Overfitting: When the model is too simple and does not capture the complexities of the
data.
• May not work well with non-linear relationships.
• Example: Taxi Fare Calculation
• In a city, the fare charged by a taxi is determined by two factors: A fixed charge of Rs. 40, which is
charged regardless of the distance. A variable charge of Rs. 12 per kilometre travelled.
You need to:
[Link] a relationship between the total fare and the distance travelled.
[Link] the fare for a passenger who travels 15 km.

• Step-by-Step Solution:
1. Deriving the Relationship Between Cost and Variables:
• Fixed charge = Rs. 40
• Variable charge = Rs. 12 per kilometre
• So, the total fare F is calculated by adding the fixed charge to the variable charge for the distance
travelled:
• Fare=Fixed charge+(Variable charge per km×Distance travelled)
• Substituting the given values:
• Fare=40+12x
2. Substituting the Values:
• Fare=40+12×15
• Fare = 40 + 180
• Fare=220
Logistic Regression

• Logistic Regression is a statistical method used for binary classification problems, where
the goal is to predict a binary outcome (i.e., a response variable that has only two possible
values, such as 0 or 1, True or False, Yes or No). It is a type of regression but instead of
predicting continuous values (like in linear regression), it predicts the probability of a
binary outcome.
• Logistic Regression is a type of binary classification algorithm that is used when we want to
predict an outcome that can be one of two classes. The output is a probability that an event belongs
to a certain class, which can then be used to decide which class to assign the observation to.
• Example: Jacket or No Jacket?
• Let’s take your jacket example to explain logistic regression:
• You want to predict whether you should wear a jacket based on the temperature (e.g., if the
temperature is cold, you would wear a jacket, and if it’s warm, you wouldn't).
• Here, your dependent variable (the one you're trying to predict) is whether or not to wear a jacket.
This is a binary outcome: either "Yes" (wear a jacket) or "No" (don't wear a jacket). In logistic
regression, this is represented as 1 (Yes) and 0 (No).
• The independent variable is the temperature, which influences whether you will wear a jacket or
not.
• First, logistic regression tries to model a linear relationship between the input features
(like temperature) and the outcome (whether or not you wear a jacket).
• The equation for this relationship looks like this:
• z=b0+b1×X
• This gives us a value z, but z can be any number — it could be positive, negative, or zero.
• Now, we apply the sigmoid function to the value of z to get the probability (between 0
and 1).
• The sigmoid function looks like this:
• Sigmoid(z)=1 \1+e−z
• This maps the value of z into a probability between 0 and 1.
• Finally, based on the probability:
• If the probability is greater than or equal to 0.5, we predict "Yes"
• If the probability is less than 0.5, we predict "No"
Classification

• Classification is a type of supervised learning algorithm where the goal is to predict the
categorical label of new data based on previously seen labeled data. It’s used when the
output variable is discrete (i.e., the output belongs to a certain class or category).
• In classification, the algorithm learns from labeled training data and then predicts the
class or category for new, unseen data.
• The output is typically a label or category, such as:
Spam or Not Spam (email classification)
Healthy or Sick (medical diagnosis)
Red, Blue, or Green (object classification)
Aspect Classification Regression

Output Type Discrete categories or Continuous values (e.g.,

classes (e.g., Spam/Not Salary, Temperature)
Spam, Healthy/Sick)
Task Goal Assigning data to Predicting a continuous
predefined categories value
Example Predicting whether an Predicting the price of a
email is spam or not house based on features
K-Nearest Neighbor (KNN)

• K-Nearest Neighbor (KNN) is a supervised learning algorithm used for both classification
and regression tasks.
• KNN works by identifying the k closest data points (neighbors) to the point being
classified or predicted, based on a distance metric (usually Euclidean distance).
• Major Assumption: Similar data points are likely to belong to the same class or have
similar values.
• In classification, it classifies data into categories (classes) based on the most common
class among its nearest neighbors.
• In regression, it predicts the continuous output by averaging the output of the k-nearest
neighbors.
• How Does KNN Work?
• Input Data:
The KNN algorithm is a supervised learning algorithm that uses labeled data. The training data
is pre-labeled, meaning each data point in the training set has a known output or class.
• Select 'K' (Number of Neighbors):
K represents the number of nearest neighbors you want to consider when classifying a new data
point.
The choice of K is crucial and can affect the model's performance. A small value of K makes the
model sensitive to noise, while a large value smooths the decision boundary but may miss subtle
patterns.
• Calculate Distance:
For a new, unclassified data point, we calculate its distance from all points in the training set.
The most commonly used distance metric is Euclidean distance, but other metrics like
Manhattan or Minkowski can also be used.
The Euclidean distance between two points x1 and x2 is calculated as:
• Identify Neighbors:
After calculating the distance between the new data point and all points in the training set,
we sort the points in ascending order of distance.
Select the K closest neighbors.
• Classify or Predict:
For Classification: Assign the class (label) of the new data point based on the majority class
of the K nearest neighbors.
Example: If K=3, and the 3 nearest neighbors belong to classes A, A, B, the new data point
is classified as Class A.
For Regression: Calculate the average of the output values of the K nearest neighbors and
assign that value as the prediction.
• Advantages:
• Simple to implement and understand.
• Works well for both classification and regression tasks.

• Challenges of k-Nearest Neighbors (kNN) Algorithm

• Computational Complexity:
The kNN algorithm requires calculating the distance from the new data point to every data point in the training
set, which can be computationally expensive. As the dataset grows, the prediction time increases, making it slow
for large datasets.
• Choice of K (Hyperparameter):
The accuracy of kNN is highly dependent on the choice of the parameter K, i.e., the number of nearest neighbors
considered.
Small K: A small value of K leads to high variance and overfitting. The model may perform well on the
training set but fail to generalize to unseen data.
Large K: A large value of K leads to high bias and underfitting. The model becomes too generalized
and may not capture the subtleties of the data.
• Sensitivity to Noisy Data:
kNN is sensitive to noisy data and outliers in the training set. Noisy data can significantly
affect the accuracy of predictions since outliers can influence the nearest neighbors and thus
skew the classification or regression results.
• No Model Training:
Unlike other machine learning algorithms (e.g., decision trees, SVM), kNN doesn't have a
training phase where the model learns patterns. It simply stores the [Link] absence of a
model-building phase means that kNN doesn't generalize the data, leading to inefficiencies
when dealing with complex datasets.
• Memory Intensive:
kNN is a memory-based algorithm, meaning it stores the entire training dataset. The model
doesn't learn a compact representation of the data; instead, it keeps all the data in memory
and compares each new data point with all the training points during prediction.
 Decision Tree
• Imagine you're playing a "Guess What" game, where you're trying to guess something based on
certain clues. For example:
Is it an animal?
Does it live in the forest?
Is it herbivorous?
• Each of these clues helps you narrow down the possibilities until you finally guess the answer
(e.g., "Rabbit"). A Decision Tree works in exactly the same way.
• In a Decision Tree, the goal is to make decisions based on certain features or attributes of data.
• Decision Tree is a flowchart-like structure where:
• Each node represents a question or decision based on a feature (like asking “Is it an animal?”).
• Each branch represents the outcome of that decision (e.g., Yes or No).
• The leaf nodes at the end give the final decision or classification (e.g., Rabbit, Dog, or Cat).
• Decision trees can be used to divide a set of items into n predetermined classes based on a specified
criterion.
Structure of a Decision Tree:
• Root Node: The very first question or decision point. For example, "Is it a mammal?"
• Internal Nodes: After the root, further questions are asked to narrow down the options. For
example, "Does it live in water?"
• Leaf Nodes: The final result of all the decisions. This is the class or category you are predicting,
such as "Fish" or "Dog".

• While decision trees are sensitive to noisy data, they generally require less data cleaning compared
to some other machine learning algorithms. Decision trees can handle missing values and outliers
better than many other models like linear regression or SVM.
 How Does a Decision Tree Work?
• Start with the Entire Dataset:
You have a collection of data, like information about animals. For example: Animal: Dog, Cat,
Rabbit, etc. Features: Mammal, Herbivore, Lives in water, etc.
• Ask the First Question:
You start at the root node and ask a question that helps you separate the data as much as possible. For
example: "Is it a Mammal?"
• Split the Data:
Based on the answer, you split your data. For example:
If "Yes", go down the branch labeled Mammal.
If "No", go down the branch labeled Non-Mammal.
• Keep Asking Questions:
At each new node, you keep asking questions (based on the features) to keep splitting your data. For
example, ask, “Does it live in water?”
• Reach the Final Decision:
Eventually, you will reach a leaf node, which gives you the final prediction. For example, after
splitting on features like "Mammal" and "Lives in Water," you end up with a leaf node saying “Fish”.
 Principle of Decision Tree
• Selecting the Best Question (Root Node):
The first step in creating a decision tree is choosing a question that divides the data into the most distinct categories. This
question is the root node of the [Link]: If you're predicting whether a customer will buy a product, your first question
could be, "Did the customer visit the website?" This will split your customers into two groups: those who visited and those who
didn’t.
• Splitting Data (Creating Nodes):
Once the first question (or root node) is selected, we split the data into two or more branches (or child nodes) based on the
answers to the [Link] example, if the answer to "Did the customer visit the website?" is Yes, you can split that group
further based on another question like "Is their income above $50,000?"
• Repeating the Process:
After the first split, you repeat the process for each new group created. Choose the best question to split the group further,
creating more nodes.
• You keep doing this until:
There are no more questions to [Link] group is homogeneous (all members belong to the same category).You reach a pre-set
limit on the number of questions to ask (tree depth).
• Leaf Nodes (Final Decisions):
The leaf nodes are the final points where the decision tree ends. They give you the final prediction or decision. For classification
trees, leaf nodes will give you a class label (e.g., "Will Buy" or "Won't Buy"). For regression trees, leaf nodes will give a
numerical value (e.g., the price of a product).
 Key Terminologies in Decision Trees
• Root Node:
The root node is the topmost node in a decision tree where the first decision or split is made. It represents
the entire dataset before any division takes place.
The root node is crucial as it sets the foundation for the entire tree by determining the first split that is most
effective in dividing the data based on the target feature.
• Decision Node:
A decision node is a point where the data is split based on the value of a feature. It represents a decision-
making step, leading to further branches or sub-nodes. . Decision nodes are used to check the values of
specific features to guide the flow of data down the tree, and they form the core of the tree's decision-
making process.
• Leaf Node (Terminal Node):
A leaf node is the final node of the tree that provides the output or prediction. It contains the class label in
classification or a continuous value in regression.
Leaf nodes are the endpoints of the decision-making process where the model makes its final prediction or
decision, based on the conditions encountered along the path from the root.
• Branches:
Branches are the connections between nodes that represent the outcomes of decisions. They
guide the flow of data from one node to the next.
Each branch corresponds to a specific decision rule or condition applied at the parent node and
leads to a child node or a leaf node. The structure of branches helps define the hierarchy of the
tree.
• Splitting:
Splitting refers to dividing a node into sub-nodes based on the value of a feature, to reduce the
data set's heterogeneity and improve classification or regression accuracy.
Effective splitting ensures that the tree remains efficient and accurate.
• Pruning:
Pruning is the process of removing branches or nodes that do not contribute significantly to the
accuracy of the decision tree, helping to prevent overfitting and simplify the model.
It is typically done after the tree has been fully grown to ensure that the model generalizes well
to unseen data and doesn't become too complex, which can lead to overfitting.
• Advantages of Decision Trees
• Easy to Understand and Interpret: Decision trees are intuitive and easy to interpret. They
do not require statistical knowledge to understand and visualize.
• Handles Both Numerical and Categorical Data: Decision trees can work with both types
of input variables, making them versatile.
• Visual Representation: Decision trees provide a clear, graphical representation of
decisions and their possible consequences, making them useful for decision-makin

• Limitations of Decision Trees

• Overfitting: Decision trees can easily overfit the data, especially if they are deep and
complex. This happens when the tree models noise or irrelevant details, leading to poor
generalization.
• Sensitive to Small Changes: Small changes in the data can lead to a very different tree
structure, making decision trees unstable.
Random Forests

Random forests are an ensemble learning technique used for both classification and
regression tasks. They work by creating multiple decision trees during training and
combining their outputs for a final prediction. Random forests outperform individual
decision trees due to their ability to reduce overfitting and increase model accuracy.
• Step 1: Random Sampling (Bootstrapping)
• In this initial step, a random sample of data points is selected from the original training
dataset. This is done with replacement, meaning that some data points might appear
multiple times in the sample, while others may not be selected at all.
• For instance, if you have a dataset with 1000 data points, you might randomly pick 1000
data points to create a new training subset, but the same data point might be picked
multiple times.
• This creates K different training subsets (each with the same number of data points as the
original dataset) to train each decision tree. This process is known as bootstrapping.
• Why is this important?: By training multiple models on slightly different versions of the
dataset, the Random Forest reduces variance and is less prone to overfitting compared to a
single decision tree.
• Step 2: Building Multiple Decision Trees
• Once the random training samples are created, a decision tree is built for each sample.
The construction of each decision tree proceeds in a similar manner:
• Splitting Nodes: At each internal node, the algorithm looks at all possible splits and
chooses the best split based on some criterion, such as Gini impurity or Information Gain
(for classification) or Mean Squared Error (MSE) (for regression).
• Maximum Depth: Each tree is allowed to grow to its maximum depth without pruning.
This means the trees can be highly detailed, capturing all the nuances of the data in each
tree. This is done to avoid underfitting the model.
• Why is this important?: Growing each tree fully increases the diversity of trees in the
forest, which improves the overall model's ability to generalize on unseen data.
• Step 3: Random Feature Selection at Each Split
• Unlike traditional decision trees where all features (predictors) are considered at each
node split, in Random Forest, a random subset of features is considered at each split.
• Let’s say there are p features in your dataset. For each node in the decision tree, only a
random subset of m features (where m < p) is selected to find the best split.
• Why is this important?: This ensures that the trees are decorrelated because each tree will
likely use different features for splitting. By decorrelating the trees, random forests can
reduce overfitting and increase accuracy. If all trees were built using the same features,
they would likely make similar predictions, which would defeat the purpose of the
ensemble approach.
• Step 4: Voting or Averaging for Prediction
• Once all decision trees are trained, the final prediction is made by aggregating the results of each individual
tree:
• For Classification: Each tree makes a class prediction (e.g., Class A, Class B, etc.). The majority vote from all
trees determines the final class. This means the class that is predicted by the most trees is chosen as the final
output.
• Example: If there are 100 trees in the forest, and 60 trees predict Class A while 40 trees predict Class B, the
final prediction will be Class A.
• For Regression: Each tree predicts a continuous value. The mean (average) of the predicted values from all
trees is taken as the final prediction.
• Example: If there are 100 trees and they predict values like 50, 52, 54, etc., the final prediction will be the
average of all these values.
• Why Aggregation Works:
• The strength of Random Forest lies in this aggregation process. By combining the outputs of multiple decision
trees, the model can smooth out errors, reduce overfitting, and produce a robust, more accurate final
prediction.
• For classification tasks, this helps overcome the potential biases of individual trees, and for regression tasks, it
reduces the variance of the predicted values.
• Step 5: Final Output
• After all trees have made their predictions and the results are aggregated (via voting or
averaging), the Random Forest outputs the final prediction:
• Classification Problem: A class label (e.g., 'Spam' or 'Not Spam')
• Regression Problem: A continuous value (e.g., house price, stock value)
What is Clustering?
• Clustering is a machine learning technique used to group a set of objects (data points) into
clusters. These clusters contain objects that are similar to each other in some way, and
dissimilar to objects in other clusters. Clustering does not rely on any predefined labels (it
is an unsupervised learning technique), making it useful in situations where the goal is to
explore the data and find inherent groupings without knowing the output beforehand.
• Key Characteristics of Clustering:
• Similar Objects in a Cluster:
• Objects within the same cluster share some common characteristics or features, meaning
they are more similar to each other than to objects in other clusters.
• Dissimilar Objects Between Clusters:
• Objects in different clusters are dissimilar. The objective of clustering is to maximize the
distance between different clusters while minimizing the distance within the same cluster.
Criteria Classification Clustering
Definition A supervised learning technique An unsupervised learning
that assigns labels to data points technique that groups data
based on labeled training data. points into clusters based on
similarity, without predefined
labels.

Output Class labels (e.g., "Yes", "No", Groupings or clusters of similar

"Cat", "Dog"). data points (e.g., customer
segments, patient groups).

Data Requirement Requires labeled data for Does not require labeled data;
training. works with unlabeled data.
Examples Spam email detection, disease Customer segmentation, image
diagnosis, sentiment analysis. grouping, anomaly detection.
 Different clustering technique
• Partitional Clustering
Partitional clustering techniques divide data objects into non-overlapping groups such that each object
belongs to one and only one cluster. A key characteristic of these algorithms is that the number of clusters
(denoted by k) is specified by the user.
• Common Algorithms: K-means, K-medoids
• Process: These algorithms iteratively assign data points into k clusters. They are non-deterministic, meaning
the results can vary between different runs, even with the same input data.
• Advantages:
• Works well with spherical-shaped clusters: Partitional clustering is effective when the clusters have relatively
simple shapes and are approximately spherical.
• Scalability: These algorithms tend to scale well with large datasets, making them suitable for many practical
applications.
• Limitations:
• Poor Performance on Complex Shapes: It does not work well with clusters that have irregular or complex
shapes.
• Challenges with Different Densities: It struggles with clusters of varying densities, as partitional clustering
assumes clusters to be uniform in density.
• Hierarchical Clustering
Hierarchical clustering creates a hierarchy of clusters using either a bottom-up (agglomerative) or top-down
(divisive) approach. The hierarchy can be represented as a dendrogram, a tree-like structure that visually depicts
the merging or splitting of clusters.
• Agglomerative Clustering (Bottom-up): Starts with individual data points, and progressively merges the most
similar pairs of clusters until all points are in a single cluster.
• Divisive Clustering (Top-down): Starts with one large cluster and recursively splits the least similar data points
until each data point is in its own cluster.
• Advantages:
• Detailed Relationships: Hierarchical clustering provides insights into the finer relationships between the data
objects, which is useful for interpreting data.
• Easily Interpretable: The results can be visualized using dendrograms, making it easy to understand the
structure of the data and how clusters are formed.
• Limitations:
• Computationally Expensive: The algorithm can be slow, especially with large datasets, as it requires
calculating and comparing all pairwise distances.
• Sensitive to Noise and Outliers: Hierarchical methods can be adversely affected by noise or outliers, as they
can distort the clustering process.
• Density-Based Clustering
Density-based clustering techniques form clusters based on the density of data points in a region. It identifies
regions of high data density separated by regions of low density.
• Popular Algorithms: DBSCAN (Density-Based Spatial Clustering of Applications with Noise), OPTICS
(Ordering Points To Identify Clustering Structure)
• Process: This method does not require the user to specify the number of clusters. Instead, a distance-based
threshold is used to determine which points belong to the same cluster. It is particularly useful for
identifying clusters of arbitrary shapes.
• Advantages:
• Handles Non-spherical Clusters: Density-based clustering works well when clusters have irregular shapes
and sizes, which partitional methods cannot handle.
• Robust to Noise and Outliers: It performs well even when data contains noise or outliers, as noise points
are typically classified as "outliers" and excluded from the clusters.
• Limitations:
• Not Ideal for High-Dimensional Data: Density-based clustering struggles in high-dimensional spaces due to
the "curse of dimensionality," where the concept of density becomes less meaningful.
• Difficulty with Varying Densities: The algorithm may fail to correctly identify clusters of varying densities
because it assumes a uniform density for each cluster.
K-Means Algorithm

• The K-Means algorithm is one of the most widely used clustering techniques, which
groups data into a predefined number of clusters (k). It is an unsupervised learning
algorithm that works by minimizing the variance within each cluster. It is efficient,
simple, and scalable for large datasets. The algorithm iteratively refines the assignment of
data points to clusters and the positions of the centroids.
• Step 1: Initialize Centroids
• Specify the number of clusters (k):
• The user must specify the number of clusters (denoted by k) they want to divide the
dataset into. The value of k is pre-defined before running the algorithm. Choosing the
right value of k can be a challenge and is often done by techniques like the elbow method
or silhouette score.
• Randomly select k data points as initial centroids:
• Initially, the algorithm selects k random data points from the dataset. These data points
are treated as the initial centroids (or center points) of the clusters. A centroid is
essentially the "mean" point of a cluster, representing its center.
• These centroids are the starting positions for each of the k clusters.
• Step 2: Cluster Assignment
• Assign data points to the nearest centroid:
• For every data point in the dataset, calculate its Euclidean distance from each of the k
centroids.
• Assign each point to the nearest centroid.
• Each data point is assigned to the cluster whose centroid is closest (the centroid with the
minimum Euclidean distance). This forms the initial clusters.
• This step is also known as the cluster assignment step.
• After this step, each data point is assigned to a specific cluster, and we now have k groups
of points, each associated with one centroid.
• Step 2:Centroid Update
• Recalculate the centroids:
• After assigning all data points to their respective clusters, the next step is to update the
centroids.
• For each of the k clusters, calculate the mean of all the data points within that cluster. This
mean is the new centroid of that cluster.
• The new centroid for each cluster is calculated as:

New Centroid=1/N ∑Xi

• Where:
• N is the number of data points in the cluster.
• Xi is the value of the data point i.
• Update the centroids to new positions:
• Each centroid is now updated to the average position of all points in that cluster. The
centroids will "move" from their initial positions based on the recalculation.
• Repeat Steps 2 and 3
• Reassign points to new centroids:
• After updating the centroids, the next step is to reassign each data point to the closest
centroid again, based on the new centroid positions.
• Check for changes in assignments:
• After the reassignment, check if any data points have changed clusters. If data points have
changed clusters, the centroids will need to be recalculated again in the next step.
• Repeat this process of cluster assignment and centroid update until the assignments of
data points to clusters no longer change. This iterative process continues until the clusters
stabilize.
• Convergence
• Convergence criteria:
• The algorithm stops when the cluster assignments do not change after an iteration, or
when the algorithm reaches a predefined number of iterations.
• This means the algorithm has converged, and the centroids no longer shift significantly.
• Final output:
• The algorithm produces k final clusters, each represented by a centroid. These clusters are
the result of the K-Means algorithm, and each data point belongs to one of these clusters
based on the minimum distance to the centroids.
• The process of assigning data points to clusters and updating the centroids stops, and the
final clusters are output.
• Pros of K-Means Algorithm
• Simplicity and Speed:
K-Means is one of the simplest clustering algorithms. It is easy to understand and
implement, making it a popular choice for many applications.
It is computationally efficient, especially when dealing with large datasets, and it converges
quickly, especially when the number of clusters is small.
• Scalability:
The algorithm can handle large datasets effectively. It works well for both small and large-
scale problems, and the time complexity generally increases linearly with the number of
data points.
• Good for Spherical Clusters:
K-Means performs well when the data is well-separated and the clusters are spherical in
shape, which is a common assumption in many machine learning tasks.
• Cons of K-Means Algorithm
• Pre-defined Number of Clusters:
One of the biggest drawbacks of K-Means is that the number of clusters (k) must be
specified in advance. This is a limitation because in real-world applications, the correct
value of k is not always known and can be difficult to determine.
• Sensitivity to Outliers:
K-Means is sensitive to outliers. Outliers can skew the centroid calculation, resulting in
incorrect cluster assignments. Since centroids are calculated using the mean, extreme values
can distort the final clusters.
• Random Initialization of Centroids:
The performance of K-Means heavily depends on the initial selection of centroids. If the
centroids are initialized poorly, the algorithm might converge to a local minimum rather
than the optimal solution. This can lead to inconsistent results on different runs of the
algorithm.
EAMPLE OF CLUSTERING

• Market segmentation is the process of dividing a broad consumer or business market,

normally consisting of existing and potential customers, into sub-groups of consumers
based on some type of shared characteristics.
• Data Collection
• Data Preprocessing
• Choose a Clustering Algorithm
• Determine the Optimal Number of Clusters (k)
• Apply the Clustering Algorithm
• Interpret the Results
• Marketing Strategy Based on Segmentation
Support Vector Machine (SVM)

• A Support Vector Machine (SVM) is a supervised machine learning algorithm primarily

used for classification tasks, although it can also be used for regression. The main goal of
an SVM is to find the best possible hyperplane that separates data points of different
classes in an optimal way.
• Core Concepts of SVM:
• Hyperplane:
A hyperplane is a decision boundary that separates data into different classes. In 2D, a hyperplane is simply a
line; in 3D, it's a plane; and in higher dimensions, it becomes a hyperplane.
SVM aims to find the hyperplane that maximizes the margin (distance) between data points of different
classes.
• Support Vectors:
Support vectors are the data points that are closest to the hyperplane. These points directly affect the position
and orientation of the hyperplane.
They are critical in defining the margin and determining the optimal hyperplane for classification, as they
directly influence the model's performance.
• Margin:
The margin refers to the distance between the hyperplane and the support vectors. SVM tries to find the
hyperplane with the maximum margin because a larger margin generally leads to better generalization and a
lower risk of overfitting.
• Kernel Trick:
The kernel trick allows SVM to work with non-linearly separable data. It transforms data
from a lower-dimensional space to a higher-dimensional space where the data can be
separated by a linear hyperplane.
This transformation enables SVM to find a linear separation even when the original data
cannot be separated linearly in its original space.
• Linear vs Non-linear SVM:
Linear SVM: Used when the data is linearly separable, meaning a straight line (or
hyperplane) can separate the classes.
Non-linear SVM: Used for data that cannot be separated by a straight line. In such cases,
SVM uses the kernel trick to map data to a higher-dimensional space where it becomes
linearly separable.
• Working Principle of Support Vector Machine (SVM):
• Input Data Representation:
SVM is a supervised machine learning algorithm that works by mapping the input data into a multi-
dimensional feature [Link] data point is represented as a point in this feature space.
• Hyperplane Selection:
The goal of SVM is to find the optimal hyperplane (decision boundary) that best separates the classes in
the feature space.
The optimal hyperplane is the one that maximizes the margin (distance) between the closest points of
each class. These closest points are called support vectors.
• Margin and Support Vectors:
The margin is the distance between the hyperplane and the closest data points from both classes. SVM
aims to maximize this margin to ensure better generalization.
Support vectors are the points on the edges of the margin that influence the position and orientation of
the hyperplane.
• Linear and Non-linear Classification:
In cases where the data is linearly separable (i.e., it can be separated by a straight line or
hyperplane), SVM constructs a linear decision boundary.
For non-linearly separable data, SVM uses the kernel trick to map the data into a higher-
dimensional space where it becomes linearly separable.
• Kernel Trick:
When data is not linearly separable in the original space, a kernel function (e.g., RBF,
polynomial) transforms the data into a higher-dimensional space where it can be separated
by a linear hyperplane.
This allows SVM to handle complex, non-linear decision boundaries.
• Classification:
Once the hyperplane is determined, new data points can be classified by determining on
which side of the hyperplane they lie. The SVM assigns a class label based on this decision.
Aspect SVM Other Classification
Algorithms
Separation Strategy Maximizes the margin between Typically uses decision
classes using a hyperplane boundaries but doesn’t focus on
maximizing the margin

Handling Non-linearity Uses the kernel trick to handle non- May require feature
linear separations transformation (e.g.,
polynomial features in Logistic
Regression)

Overfitting Less prone to overfitting by Prone to overfitting, especially

maximizing margin and using support in models like Decision Trees
vectors

Scalability Computationally expensive for large Often faster, especially for

datasets smaller datasets (e.g., k-NN,
Logistic Regression)
Naive Bayes

• The Naive Bayes classifier is a simple, probabilistic machine learning algorithm used for
classification tasks. It is based on Bayes' Theorem, which helps to predict the class or
category of a given item based on its features.
• The term "naive" comes from the assumption the algorithm makes: it assumes that all the
features (attributes) used to classify the data are independent of each other. In reality, this
assumption is almost never true, but despite this, the Naive Bayes algorithm works
surprisingly well in practice.
• Bayes' Theorem
• Bayes’ Theorem is a formula that helps us calculate the probability of an event given some
evidence .

• P(C∣X) is the probability of class c (Spam or Not Spam) given the features x (the words in the
email).
• P(X∣C) is the likelihood, i.e., the probability of seeing the features (like "Free", "Offer") in a
particular class (Spam or Not Spam).
• P(C) is the prior probability of the class. For example, what is the probability of an email being
Spam before we know any features?
• P(X ) is the evidence or total probability of the features. This is usually constant and not calculated
when comparing classes.
How Does Naive Bayes Classifier Work? (Step by Step)

• Step 1: Calculate Prior Probabilities (P(C)

• Before looking at any features, Naive Bayes calculates the probability of each class. This
is called the prior probability. It represents how often each class occurs in the dataset.
• For example, in our Spam classification problem:
• What is the probability of an email being Spam? P(Spam)
• What is the probability of an email being Not Spam? P(NotSpam)
• If we have a dataset with 100 emails, and 40 are Spam and 60 are Not Spam, the prior
probabilities would be:
• P(Spam)=40/100=0.4
• P(NotSpam)=60/100=0.6
• Step 2: Calculate Likelihood (P(X|C)
• Next, Naive Bayes calculates the likelihood for each feature given the class. This is how likely
it is to see each feature (like the words "Free" or "Offer") in each class (Spam or Not Spam).
• For example:
• What is the probability of the word "Free" appearing in a Spam email? P(Free∣Spam)
• What is the probability of the word "Free" appearing in a Not Spam email? P(Free∣NotSpam)
• We do this for each feature (in this case, each word) in the email.

• Step 3: Multiply Likelihoods for All Features

• Since Naive Bayes assumes that all features are independent, we multiply the individual
likelihoods for each feature to calculate the combined likelihood for all features given a class.
• For example, for the features "Free", "Offer", and "Unknown Sender", we calculate:
• P(Free,Offer,Unknown∣Spam)=P(Free∣Spam)⋅P(Offer∣Spam)⋅P(Unknown∣Spam)
• Step 4: Apply Bayes' Theorem to Calculate Posterior Probability (P(C|X))
• Now that we have the prior probability and the likelihoods, we can apply Bayes' Theorem
to calculate the posterior probability for each class (Spam or Not Spam).
• Step 5: Classify the Data
• Finally, Naive Bayes compares the posterior probabilities for all classes (Spam and Not
Spam) and assigns the class with the highest probability to the data point.
• For example, if:
• P(Spam∣X)=0.8
• P(NotSpam∣X)=0.2
• Then, the classifier would classify the email as Spam because 0.8 is the higher
probability.
THANK YOU