UNIT I

Introduction to Machine Learning

1. Introduction

1.1 What Is Machine Learning?

Machine learning is programming computers to optimize a performance criterion using example
data or past experience. We have a model defined up to some parameters, and learning is the
execution of a computer program to optimize the parameters of the model using the training data or
past experience. The model may be predictive to make predictions in the future, or descriptive to gain
knowledge from data, or both.
Arthur Samuel, an early American leader in the field of computer gaming and artificial intelligence,
coined the term “Machine Learning” in 1959 while at IBM. He defined machine learning as “the field of
study that gives computers the ability to learn without being explicitly programmed.” However, there is
no universally accepted definition for machine learning. Different authors define the term differently.

Definition of learning
Definition
A computer program is said to learn from experience E with respect to some class of tasks T and
performance measure P, if its performance at tasks T, as measured by P, improves with experience E.

Examples
i) Handwriting recognition learning problem
• Task T: Recognising and classifying handwritten words within images
• Performance P: Percent of words correctly classified
• Training experience E: A dataset of handwritten words with given classifications
ii) A robot driving learning problem
• Task T: Driving on highways using vision sensors
• Performance measure P: Average distance traveled before an error
• training experience: A sequence of images and steering commands recorded while
observing a human driver
iii) A chess learning problem
• Task T: Playing chess
• Performance measure P: Percent of games won against opponents
• Training experience E: Playing practice games against itself
Definition
A computer program which learns from experience is called a machine learning program or
simply a learning program. Such a program is sometimes also referred to as a learner.

1.2 Components of Learning

Basic components of learning process

The learning process, whether by a human or a machine, can be divided into four components,
namely, data storage, abstraction, generalization and evaluation. Figure 1.1 illustrates the
variouscomponents and the steps involved in the learning process.

1
1. Data storage
Facilities for storing and retrieving huge amounts of data are an important component of the
learning process. Humans and computers alike utilize data storage as a foundation for advanced
reasoning.
• In a human being, the data is stored in the brain and data is retrieved using electrochemical signals.
• Computers use hard disk drives, flash memory, random access memory and similar devices to store
data and use cables and other technology to retrieve data.

2. Abstraction
The second component of the learning process is known as abstraction.
Abstraction is the process of extracting knowledge about stored data. This involves creating general
concepts about the data as a whole. The creation of knowledge involves application of known models
and creation of new models.
The process of fitting a model to a dataset is known as training. When the model has been trained, the
data is transformed into an abstract form that summarizes the original information.

3. Generalization
The third component of the learning process is known as generalisation.
The term generalization describes the process of turning the knowledge about stored data into a form
that can be utilized for future action. These actions are to be carried out on tasks that are similar, but
not identical, to those what have been seen before. In generalization, the goal is to discover those
properties of the data that will be most relevant to future tasks.

4. Evaluation
Evaluation is the last component of the learning process.
It is the process of giving feedback to the user to measure the utility of the learned knowledge. This
feedback is then utilised to effect improvements in the whole learning process

Applications of machine learning

Application of machine learning methods to large databases is called data mining. In data
mining, a large volume of data is processed to construct a simple model with valuable use, for example,
having
high predictive accuracy.

The following is a list of some of the typical applications of machine learning.

1. In retail business, machine learning is used to study consumer behaviour.
2. In finance, banks analyze their past data to build models to use in credit applications, fraud
detection, and the stock market.
3. In manufacturing, learning models are used for optimization, control, and troubleshooting.
2
4. In manufacturing, learning models are used for optimization, control, and troubleshooting.
5. In medicine, learning programs are used for medical diagnosis.
6. In telecommunications, call patterns are analyzed for network optimization and maximizing
the quality of service.
7. In science, large amounts of data in physics, astronomy, and biology can only be analyzed
fast enough by computers. The World Wide Web is huge; it is constantly growing and
searching for relevant information cannot be done manually.
8. In artificial intelligence, it is used to teach a system to learn and adapt to changes so that the
system designer need not foresee and provide solutions for all possible situations.
9. It is used to find solutions to many problems in vision, speech recognition, and robotics.
10. Machine learning methods are applied in the design of computer-controlled vehicles to steer
correctly when driving on a variety of roads.
11. Machine learning methods have been used to develop programmes for playing games such
as chess, backgammon and Go.
Types of Learning :
A machine is said to be learning from past Experiences(data feed-in) with respect to some
class of tasks if its Performance in a given Task improves with the Experience.

For example, assume that a machine has to predict whether a customer will buy a specific
product let’s say “Antivirus” this year or not. The machine will do it by looking at the
previous knowledge/past experiences i.e the data of products that the customer had bought
every year and if he buys Antivirus every year, then there is a high probability that the
customer is going to buy an antivirus this year as well. This is how machine learning works at
the basic conceptual level.

Types of Learning

SUPERVISED LEARNING:
Supervised is when the model is getting trained on a labelled dataset. A labelled dataset is one
that has both input and output parameters. In this type of learning both training and validation,
datasets are labelled as shown in the figures below.
Both the above figures have labelled data set as follows:
• Figure A: It is a dataset of a shopping store that is useful in predicting whether a
customer will purchase a particular product under consideration or not based on his/ her
gender, age, and salary.
Input: Gender, Age, Salary
Output: Purchased i.e. 0 or 1; 1 means yes the customer will purchase and 0 means
that the customer won’t purchase it.
• Figure B: It is a Meteorological dataset that serves the purpose of predicting wind
speed based on different parameters.
Input: Dew Point, Temperature, Pressure, Relative Humidity, Wind Direction
Output: Wind Speed
Training the system: While training the model, data is usually split in the ratio of 80:20 i.e.
80% as training data and the rest as testing data. In training data, we feed input as well as
output for 80% of data. The model learns from training data only. We use different machine
learning algorithms(which we will discuss in detail in the next articles) to build our model.
Learning means that the model will build some logic of its own.
Once the model is ready then it is good to be tested. At the time of testing, the input is fed from
the remaining 20% of data that the model has never seen before, the model will predict some
value and we will compare it with the actual output and calculate the accuracy.
Types of Supervised Learning:

Supervised learning is a machine learning technique that is widely used in various fields such
as finance, healthcare, marketing, and more. It is a form of machine learning in which the
algorithm is trained on labeled data to make predictions or decisions based on the data inputs.
In supervised learning, the algorithm learns a mapping between the input and output data. This
mapping is learned from a labeled dataset, which consists of pairs of input and output data.
The algorithm tries to learn the relationship between the input and output data so that it can
make accurate predictions on new, unseen data.
The labeled dataset used in supervised learning consists of input features and corresponding
output labels. The input features are the attributes or characteristics of the data that are used to
make predictions, while the output labels are the desired outcomes or targets that the algorithm
tries to predict.
Supervised learning is typically divided into two main categories: regression and classification.
In regression, the algorithm learns to predict a continuous output value, such as the price of a
house or the temperature of a city. In classification, the algorithm learns to predict a
categorical output variable or class label, such as whether a customer is likely to purchase a
product or not.
One of the primary advantages of supervised learning is that it allows for the creation of
complex models that can make accurate predictions on new data. However, supervised
learning requires large amounts of labeled training data to be effective. Additionally, the
quality and representativeness of the training data can have a significant impact on the
accuracy of the model.
Supervised learning can be further classified into two categories:
Regression: In regression, the target variable is a continuous value. The goal of regression is
to predict the value of the target variable based on the input variables. Linear regression,
polynomial regression, and decision trees are some of the examples of regression algorithms.
Classification: In classification, the target variable is a categorical value. The goal of
classification is to predict the class or category of the target variable based on the input
variables. Some examples of classification algorithms include logistic regression, decision
trees, support vector machines, and neural networks.
Supervised learning can be further divided into several different types, each with its own
unique characteristics and applications. Here are some of the most common types of
supervised learning:(Algorithms used in Supervised Learning)
Linear Regression
Linear regression is a type of regression algorithm that is used to predict a continuous output
value. It is one of the simplest and most widely used algorithms in supervised learning. In
linear regression, the algorithm tries to find a linear relationship between the input features and
the output value. The output value is predicted based on the weighted sum of the input
features.
Logistic Regression
Logistic regression is a type of classification algorithm that is used to predict a binary output
variable. It is commonly used in machine learning applications where the output variable is
either true or false, such as in fraud detection or spam filtering. In logistic regression, the
algorithm tries to find a linear relationship between the input features and the output variable.
The output variable is then transformed using a logistic function to produce a probability value
between 0 and 1.
Decision Trees
A decision tree is a type of algorithm that is used for both classification and regression tasks. It
is a tree-like structure that is used to model decisions and their possible consequences. Each
internal node in the tree represents a decision, while each leaf node represents a possible
outcome. Decision trees can be used to model complex relationships between input features
and output variables.
Random Forests
Random forests are an ensemble learning technique that is used for both classification and
regression tasks. They are made up of multiple decision trees that work together to make
predictions. Each tree in the forest is trained on a different subset of the input features and
data. The final prediction is made by aggregating the predictions of all the trees in the forest
• hine (SVM)
• Random Forest

UNSUPERVISED LEARNING:
Unsupervised learning is the training of a machine using information that is neither classified
nor labeled and allowing the algorithm to act on that information without guidance. Here the
task of the machine is to group unsorted information according to similarities, patterns, and
differences without any prior training of data.
Unlike supervised learning, no teacher is provided that means no training will be given to the
machine. Therefore the machine is restricted to find the hidden structure in unlabeled data by
itself.
For instance, suppose it is given an image having both dogs and cats which it has never seen.

Thus the machine has no idea about the features of dogs and cats so we can’t categorize it as
‘dogs and cats ‘. But it can categorize them according to their similarities, patterns, and
differences, i.e., we can easily categorize the above picture into two parts. The first may
contain all pics having dogs in them and the second part may contain all pics having cats in
them. Here you didn’t learn anything before, which means no training data or examples.
It allows the model to work on its own to discover patterns and information that was
previously undetected. It mainly deals with unlabelled data.
Unsupervised learning is classified into two categories of algorithms:
• Clustering: A clustering problem is where you want to discover the inherent groupings
in the data, such as grouping customers by purchasing behavior.
• Association: An association rule learning problem is where you want to discover rules
that describe large portions of your data, such as people that buy X also tend to buy Y.
Types of Unsupervised Learning:-
Clustering
1. Exclusive (partitioning)
2. Agglomerative
3. Overlapping
4. Probabilistic
Clustering Types:-
1. Hierarchical clustering
2. K-means clustering
3. Principal Component Analysis
4. Singular Value Decomposition
 5. Independent Component Analysis
Supervised vs. Unsupervised Machine Learning:

Parameters Supervised machine learning Unsupervised machine learning

Algorithms are trained using labeled Algorithms are used against data
Input Data
data. that is not labeled
Computational
Simpler method Computationally complex
Complexity
Accuracy Highly accurate Less accurate
No. of classes No. of classes is known No. of classes is not known
Data Analysis Uses offline analysis Uses real-time analysis of data
Linear and Logistics regression,
K-Means clustering, Hierarchical
Random forest,
clustering,
Algorithms used
Support Vector Machine, Neural
Apriori algorithm, etc.
Network, etc.

Output Desired output is given. Desired output is not given.

Training data Use training data to infer model. No training data is used.
It is not possible to learn larger and It is possible to learn larger and
Complex model more complex models than with more complex models with
supervised learning. unsupervised learning.
Model We can test our model. We can not test our model.
Supervised learning is also called Unsupervised learning is also
Called as
classification. called clustering.
Example: Optical character
Example Example: Find a face in an image.
recognition.

Advantages of unsupervised learning:

• It does not require training data to be labeled.
• Dimensionality reduction can be easily accomplished using unsupervised learning.
• Capable of finding previously unknown patterns in data.
• Flexibility: Unsupervised learning is flexible in that it can be applied to a wide variety
of problems, including clustering, anomaly detection, and association rule mining.
• Exploration: Unsupervised learning allows for the exploration of data and the
discovery of novel and potentially useful patterns that may not be apparent from the
outset.
• Low cost: Unsupervised learning is often less expensive than supervised learning
because it doesn’t require labeled data, which can be time-consuming and costly to
obtain.

Disadvantages of unsupervised learning :

• Difficult to measure accuracy or effectiveness due to lack of predefined answers during
training.
 • The results often have lesser accuracy.
• The user needs to spend time interpreting and label the classes which follow that
classification.
• Lack of guidance: Unsupervised learning lacks the guidance and feedback provided by
labeled data, which can make it difficult to know whether the discovered patterns are
relevant or useful.
• Sensitivity to data quality: Unsupervised learning can be sensitive to data quality,
including missing values, outliers, and noisy data.
• Scalability: Unsupervised learning can be computationally expensive, particularly for
large datasets or complex algorithms, which can limit its scalability.

REINFORCEMENT LEARNING:
Reinforcement learning is an area of Machine Learning. It is about taking suitable action to
maximize reward in a particular situation. It is employed by various software and machines to
find the best possible behavior or path it should take in a specific situation. Reinforcement
learning differs from supervised learning in a way that in supervised learning the training data
has the answer key with it so the model is trained with the correct answer itself whereas in
reinforcement learning, there is no answer but the reinforcement agent decides what to do to
perform the given task. In the absence of a training dataset, it is bound to learn from its
experience.
Reinforcement Learning (RL) is the science of decision making. It is about learning the
optimal behavior in an environment to obtain maximum reward. In RL, the data is
accumulated from machine learning systems that use a trial-and-error method. Data is not part
of the input that we would find in supervised or unsupervised machine learning.
Reinforcement learning uses algorithms that learn from outcomes and decide which action to
take next. After each action, the algorithm receives feedback that helps it determine whether
the choice it made was correct, neutral or incorrect. It is a good technique to use for automated
systems that have to make a lot of small decisions without human guidance.
Reinforcement learning is an autonomous, self-teaching system that essentially learns by trial
and error. It performs actions with the aim of maximizing rewards, or in other words, it is
learning by doing in order to achieve the best outcomes.
Example:
The problem is as follows: We have an agent and a reward, with many hurdles in between. The
agent is supposed to find the best possible path to reach the reward. The following problem
explains the problem more easily.
The above image shows the robot, diamond, and fire. The goal of the robot is to get the reward
that is the diamond and avoid the hurdles that are fired. The robot learns by trying all the
possible paths and then choosing the path which gives him the reward with the least hurdles.
Each right step will give the robot a reward and each wrong step will subtract the reward of the
robot. The total reward will be calculated when it reaches the final reward that is the diamond.

Main points in Reinforcement learning –

• Input: The input should be an initial state from which the model will start
• Output: There are many possible outputs as there are a variety of solutions to a
particular problem
• Training: The training is based upon the input, The model will return a state and the
user will decide to reward or punish the model based on its output.
• The model keeps continues to learn.
• The best solution is decided based on the maximum reward.
Difference between Reinforcement learning and Supervised learning:

Reinforcement learning Supervised learning

Reinforcement learning is all about making decisions In Supervised learning, the
sequentially. In simple words, we can say that the output decision is made on the initial
depends on the state of the current input and the next input input or the input given at the
depends on the output of the previous input start
In supervised learning the
In Reinforcement learning decision is dependent, So we decisions are independent of
give labels to sequences of dependent decisions each other so labels are given to
each decision.
Example: Object
Example: Chess game,text summarization
recognition,spam detetction
Types of Reinforcement:
There are two types of Reinforcement:
 1. Positive: Positive Reinforcement is defined as when an event, occurs due to a
particular behavior, increases the strength and the frequency of the behavior. In other
words, it has a positive effect on behavior.
Advantages of reinforcement learning are:
• Maximizes Performance
• Sustain Change for a long period of time
• Too much Reinforcement can lead to an overload of states which can diminish
the results
2. Negative: Negative Reinforcement is defined as strengthening of behavior because a
negative condition is stopped or avoided.
Advantages of reinforcement learning:
• Increases Behavior
• Provide defiance to a minimum standard of performance
• It Only provides enough to meet up the minimum behavior

Elements of Reinforcement Learning

Reinforcement learning elements are as follows:
1. Policy
2. Reward function
3. Value function
4. Model of the environment
Policy: Policy defines the learning agent behavior for given time period. It is a mapping from
perceived states of the environment to actions to be taken when in those states.
Reward function: Reward function is used to define a goal in a reinforcement learning
problem.A reward function is a function that provides a numerical score based on the state of
the environment
Value function: Value functions specify what is good in the long run. The value of a state is
the total amount of reward an agent can expect to accumulate over the future, starting from
that state.
Model of the environment: Models are used for planning.
Credit assignment problem: Reinforcement learning algorithms learn to generate an internal
value for the intermediate states as to how good they are in leading to the goal. The learning
decision maker is called the agent. The agent interacts with the environment that includes
everything outside the agent.
The agent has sensors to decide on its state in the environment and takes action that modifies
its state.
The reinforcement learning problem model is an agent continuously interacting with an
environment. The agent and the environment interact in a sequence of time steps. At each time
step t, the agent receives the state of the environment and a scalar numerical reward for the
previous action, and then the agent then selects an action.
Reinforcement learning is a technique for solving Markov decision problems.
Reinforcement learning uses a formal framework defining the interaction between a learning
agent and its environment in terms of states, actions, and rewards. This framework is intended
to be a simple way of representing essential features of the artificial intelligence problem.
Various Practical Applications of Reinforcement Learning –
• RL can be used in robotics for industrial automation.
• RL can be used in machine learning and data processing
• RL can be used to create training systems that provide custom instruction and materials
according to the requirement of students.

Application of Reinforcement Learnings

1. Robotics: Robots with pre-programmed behavior are useful in structured environments,
such as the assembly line of an automobile manufacturing plant, where the task is repetitive in
nature.
2. A master chess player makes a move. The choice is informed both by planning, anticipating
possible replies and counter replies.
3. An adaptive controller adjusts parameters of a petroleum refinery’s operation in real time.

RL can be used in large environments in the following situations:

1. A model of the environment is known, but an analytic solution is not available;
2. Only a simulation model of the environment is given (the subject of simulation-based
optimization)
3. The only way to collect information about the environment is to interact with it.

Advantages and Disadvantages of Reinforcement Learning

Advantages of Reinforcement learning
1. Reinforcement learning can be used to solve very complex problems that cannot be solved
by conventional techniques.
2. The model can correct the errors that occurred during the training process.
3. In RL, training data is obtained via the direct interaction of the agent with the environment
4. Reinforcement learning can handle environments that are non-deterministic, meaning that
the outcomes of actions are not always predictable. This is useful in real-world applications
where the environment may change over time or is uncertain.
5. Reinforcement learning can be used to solve a wide range of problems, including those that
involve decision making, control, and optimization.
6. Reinforcement learning is a flexible approach that can be combined with other machine
learning techniques, such as deep learning, to improve performance.
Disadvantages of Reinforcement learning
1. Reinforcement learning is not preferable to use for solving simple problems.
2. Reinforcement learning needs a lot of data and a lot of computation
3. Reinforcement learning is highly dependent on the quality of the reward function. If the
reward function is poorly designed, the agent may not learn the desired behavior.
4. Reinforcement learning can be difficult to debug and interpret. It is not always clear why the
agent is behaving in a certain way, which can make it difficult to diagnose and fix problems.
Hypothesis in Machine Learning
The hypothesis is a common term in Machine Learning and data science projects. As we
know, machine learning is one of the most powerful technologies across the world, which
helps us to predict results based on past experiences. Moreover, data scientists and ML
professionals conduct experiments that aim to solve a problem. These ML professionals and
data scientists make an initial assumption for the solution of the problem.
This assumption in Machine learning is known as Hypothesis. In Machine Learning, at
various times, Hypothesis and Model are used interchangeably. However, a Hypothesis is an
assumption made by scientists, whereas a model is a mathematical representation that is used
to test the hypothesis. In this topic, "Hypothesis in Machine Learning," we will discuss a few
important concepts related to a hypothesis in machine learning and their importance. So, let's
start with a quick introduction to Hypothesis.

What is Hypothesis?
The hypothesis is defined as the supposition or proposed explanation based on insufficient
evidence or assumptions. It is just a guess based on some known facts but has not yet been
proven. A good hypothesis is testable, which results in either true or false.
Example: Let's understand the hypothesis with a common example. Some scientist claims that
ultraviolet (UV) light can damage the eyes then it may also cause blindness.
In this example, a scientist just claims that UV rays are harmful to the eyes, but we assume
they may cause blindness. However, it may or may not be possible. Hence, these types of
assumptions are called a hypothesis.

Hypothesis in Machine Learning (ML)

The hypothesis is one of the commonly used concepts of statistics in Machine Learning. It is
specifically used in Supervised Machine learning, where an ML model learns a function that
best maps the input to corresponding outputs with the help of an available dataset.
In supervised learning techniques, the main aim is to determine the possible hypothesis out of
hypothesis space that best maps input to the corresponding or correct outputs.
There are some common methods given to find out the possible hypothesis from the
Hypothesis space, where hypothesis space is represented by uppercase-h (H) and hypothesis
by lowercase-h (h). These are defined as follows:

Hypothesis space (H):

Hypothesis space is defined as a set of all possible legal hypotheses; hence it is also known
as a hypothesis set. It is used by supervised machine learning algorithms to determine the best
possible hypothesis to describe the target function or best maps input to output.
It is often constrained by choice of the framing of the problem, the choice of model, and the
choice of model configuration.

Hypothesis (h):
It is defined as the approximate function that best describes the target in supervised machine
learning algorithms. It is primarily based on data as well as bias and restrictions applied to
data.
Hence hypothesis (h) can be concluded as a single hypothesis that maps input to proper output
and can be evaluated as well as used to make predictions.
The hypothesis (h) can be formulated in machine learning as follows:
y= mx + b
Where,
Y: Range
m: Slope of the line which divided test data or changes in y divided by change in x.
x: domain
c: intercept (constant)
Example: Let's understand the hypothesis (h) and hypothesis space (H) with a two-
dimensional coordinate plane showing the distribution of data as follows:
Now, assume we have some test data by which ML algorithms predict the outputs for input as
follows:

If we divide this coordinate plane in such as way that it can help you to predict output or result
as follows:
Based on the given test data, the output result will be as follows:

However, based on data, algorithm, and constraints, this coordinate plane can also be divided
in the following ways as follows:
With the above example, we can conclude that;
Hypothesis space (H) is the composition of all legal best possible ways to divide the
coordinate plane so that it best maps input to proper output.
Further, each individual best possible way is called a hypothesis (h). Hence, the hypothesis
and hypothesis space would be like this:
Performance measures for Regression:
Regression analysis is a subfield of supervised machine learning. It aims to model the
relationship between a certain number of features and a continuous target variable.
Following are the performance metrics used for evaluating a regression

1) Mean Absolute Error(MAE)

MAE is a very simple metric which calculates the absolute difference between actual
and predicted values.

To better understand, let’s take an example you have input data and output data and use
Linear Regression, which draws a best-fit line.

Now you have to find the MAE of your model which is basically a mistake made by
the model known as an error. Now find the difference between the actual value and
predicted value that is an absolute error but we have to find the mean absolute of the
complete dataset.

so, sum all the errors and divide them by a total number of observations And this is
MAE. And we aim to get a minimum MAE because this is a loss.

Advantages of MAE
• The MAE you get is in the same unit as the output variable.
• It is most Robust to outliers.

Disadvantages of MAE

• The graph of MAE is not differentiable so we have to apply various

optimizers like Gradient descent which can be differentiable.
Now to overcome the disadvantage of MAE next metric came as MSE.
2) Mean Squared Error(MSE)
MSE is a most used and very simple metric with a little bit of change in mean
absolute error. Mean squared error states that finding the squared difference between
actual and predicted value.

So, above we are finding the absolute difference and here we are finding the
squared difference.

What actually the MSE represents? It represents the squared distance between actual
and predicted values. we perform squared to avoid the cancellation of negative terms
and it is the benefit of MSE.

Advantages of MSE
The graph of MSE is differentiable, so you can easily use it as a loss function.
Disadvantages of MSE

• The value you get after calculating MSE is a squared unit of output. for example, the
output variable is in meter(m) then after calculating MSE the output we get is in
meter squared.
• If you have outliers in the dataset then it penalizes the outliers most and the
calculated MSE is bigger. So, in short, It is not Robust to outliers which were an
advantage in MAE.
3) Root Mean Squared Error(RMSE)
As RMSE is clear by the name itself, that it is a simple square root of mean squared error.

Advantages of RMSE
• The output value you get is in the same unit as the required output variable
which makes interpretation of loss easy.

Disadvantages of RMSE
• It is not that robust to outliers as compared to MAE.
for performing RMSE we have to NumPy square root function over MSE.

Most of the time people use RMSE as an evaluation metric and mostly when you are working
with deep learning techniques the most preferred metric is RMSE.

4) R Squared (R2)

R2 score is a metric that tells the performance of your model, not the loss in an
absolute sense that how many wells did your model perform.
In contrast, MAE and MSE depend on the context as we have seen whereas the
R2 score is independent of context.

So, with help of R squared we have a baseline model to compare a model which none
of the other metrics provides. The same we have in classification problems which we
call a threshold which is fixed at 0.5. So basically R2 squared calculates how must
regression line is better than a mean line.

Hence, R2 squared is also known as Coefficient of Determination or sometimes also

known as Goodness of fit
Now, how will you interpret the R2 score? suppose If the R2 score is zero then the above
regression line by mean line is equal means 1 so 1-1 is zero. So, in this case, both lines are
overlapping means model performance is worst, It is not capable to take advantage of the
output column.

Now the second case is when the R2 score is 1, it means when the division term is zero and
it will happen when the regression line does not make any mistake, it is perfect. In the real
world, it is not possible.

So we can conclude that as our regression line moves towards perfection, R2 score move
towards one. And the model performance improves.

The normal case is when the R2 score is between zero and one like 0.8 which means your
model is capable to explain 80 per cent of the variance of data.

5) Adjusted R Squared
The disadvantage of the R2 score is while adding new features in data the R2 score
starts increasing or remains constant but it never decreases because It assumes that
while adding more data variance of data increases.

But the problem is when we add an irrelevant feature in the dataset then at that time R2
sometimes starts increasing which is incorrect.
Hence, To control this situation Adjusted R Squared came into existence.
Now as K increases by adding some features so the denominator will decrease, n-1 will remain
constant. R2 score will remain constant or will increase slightly so the complete answer will
increase and when we subtract this from one then the resultant score will decrease. so this is the
case when we add an irrelevant feature in the dataset.

And if we add a relevant feature then the R2 score will increase and 1-R2 will decrease heavily
and the denominator will also decrease so the complete term decreases, and on subtracting
from one the score increases.

Hence, this metric becomes one of the most important metrics to use during the evaluation
of the model.
 Performance measures for Classification

Classification is about predicting the class labels given input data. In binary classification, there
are only two possible output classes(i.e., Dichotomy). In multiclass classification, more than two
possible classes can be present. I’ll focus only on binary classification.

A very common example of binary classification is spam detection, where the input data could
include the email text and metadata (sender, sending time), and the output label is either “spam”
or “not spam.” (See Figure) Sometimes, people use some other names also for the two classes:
“positive” and “negative,” or “class 1” and “class 0.”

Figure — Email spam detection is a binary classification problem

Confusion Matrix
Confusion Matrix is a performance measurement for the machine learning classification
problems where the output can be two or more classes. It is a table with combinations
of predicted and actual values.

A confusion matrix is defined as the table that is often used to describe the
performance of a classification model on a set of the test data for which the true
values are known.
It is extremely useful for measuring the Recall, Precision, Accuracy, and AUC-ROC curves.
Let’s try to understand TP, FP, FN, TN with an example of pregnancy analogy.

True Positive: We predicted positive and it’s true. In the image, we predicted that a woman is
pregnant and she actually is.

True Negative: We predicted negative and it’s true. In the image, we predicted that a man
is not pregnant and he actually is not.

False Positive (Type 1 Error)- We predicted positive and it’s false. In the image, we
predicted that a man is pregnant but he actually is not.

False Negative (Type 2 Error)- We predicted negative and it’s false. In the image,
we predicted that a woman is not pregnant but she actually is.

There are many ways for measuring classification performance. Accuracy, confusion
matrix, log-loss, and AUC-ROC are some of the most popular metrics. Precision-recall is
a widely used metrics for classification problems.

1. Accuracy
Accuracy simply measures how often the classifier correctly predicts. We can define accuracy as the ratio
of the number of correct predictions and the total number of predictions.

2. Precision —Precision explains how many of the correctly predicted cases actually turned out to be
positive. Precision is useful in the cases where False Positive is a higher concern than False Negatives. The
importance of Precision is in music or video recommendation systems, e-
commerce websites, etc. where wrong results could lead to customer churn and this could
be harmful to the business.

Precision for a label is defined as the number of true positives divided by

the number of predicted positives

3. Recall /Sensitivity/True Positive Rate — Recall explains how many of the actual
positive cases we were able to predict correctly with our model. It is a useful metric in cases
where False Negative is of higher concern than False Positive. It is important in medical
cases where it doesn’t matter whether we raise a false alarm but the actual positive cases
should not go undetected!

Recall for a label is defined as the number of true positives divided by

the total number of actual positives.

4. False Negative Rate

False Negative Rate (FNR) tells us what proportion of the positive class got incorrectly
classified by the classifier.

A higher TPR and a lower FNR is desirable since we want to correctly classify the
positive class.

5. Specificity / True Negative Rate

Specificity tells us what proportion of the negative class got correctly classified.

Taking the same example as in Sensitivity, Specificity would mean determining the
proportion of healthy people who were correctly identified by the model.

6. False Positive Rate

FPR tells us what proportion of the negative class got incorrectly classified by the classifier.
7. F1 Score — It gives a combined idea about Precision and Recall metrics. It is maximum
when Precision is equal to Recall.
F1 Score is the harmonic mean of precision and recall.

The F1 score punishes extreme values more. F1 Score could be an effective evaluation
metric in the following cases:
• When FP and FN are equally costly.
• Adding more data doesn’t effectively change the outcome
• True Negative is high

8. AUC-ROC — The Receiver Operator Characteristic (ROC) is a probability curve

that plots the TPR(True Positive Rate) against the FPR(False Positive Rate) at various
threshold values and separates the ‘signal’ from the ‘noise’.

The Area Under the Curve (AUC) is the measure of the ability of a classifier to distinguish
between classes. From the graph, we simply say the area of the curve ABDE and the X and Y-
axis.

From the graph shown below, the greater the AUC, the better is the performance of the model at
different threshold points between positive and negative classes. This simply means that When
AUC is equal to 1, the classifier is able to perfectly distinguish between all Positive and Negative
class points. When AUC is equal to 0, the classifier would be predicting all Negatives as Positives
and vice versa. When AUC is 0.5, the classifier is not able to distinguish between the Positive and
Negative classes.

Working of AUC —In a ROC curve, the X-axis value shows False Positive Rate (FPR), and
Y-axis shows True Positive Rate (TPR). Higher the value of X means higher the number of
False Positives(FP) than True Negatives(TN), while a higher Y-axis value indicates a higher
number of TP than FN. So, the choice of the threshold depends on the ability to balance
between FP and FN.
Cross Validation
Cross validation is a model evaluation method that is better than residuals. The problem with
residual evaluations is that they do not give an indication of how well the learner will do when
it is asked to make new predictions for data it has not already seen. One way to overcome this
problem is to not use the entire data set when training a learner. Some of the data is removed
before training begins. Then when training is done, the data that was removed can be used to
test the performance of the learned model on ``new'' data. This is the basic idea for a whole
class of model evaluation methods called cross validation.

Cross Validation refers to a technique in which we reserve a particular portion of a dataset on

which we do not train the model. After the training is over, we test the resulting model on this
portion of the data before finalizing it.

The steps involved in cross validation are as follows -

1. Reserve a small portion of data set called validation data.
2. Train the model using the remaining dataset
3. Test the model on the validation data set and check its accuracy.

Cross validation helps us in gauging the effectiveness of our model’s performance. If the
model delivers high accuracy on validation data, we can go ahead and use the model for
solving problems on real world data.

One popular way of doing cross validation is to use the k-fold cross- validation technique. In
this technique, we split the data into k different but similar folds or sets.

We then perform k iterations, and, in each iteration, we choose one-fold as the validation set or
test set and the rest as training sets. This helps us train the model better and avoid any bias as
we are using the entire data for training and testing.

To determine the number of iterations `k` for cross-validation, you should consider a value
such that each of the data samples is large enough to be statistically representative of the
broader dataset.

If you are unsure as to which value of k should be chosen, then you can take k=10 as a thumb
rule since it is common in the field of applied machine learning.
In the diagram given below, we perform a 5-fold validation.

Cross-validation might also be used to determine the value of K while using K-

NN. In most cases, the accuracy is highest for K=7 to K=13 and falls as the
value of K increases.

Types of cross validation methods are

1. holdout,
2. k-fold
3. leave-one-out.
1. Holdout Method
The holdout method is the simplest kind of cross validation. The data set is separated into two
sets, called the training set and the testing set. The function approximator fits a function using
the training set only.

2. K-Fold Cross-Validation
K-fold cross-validation approach divides the input dataset into K groups of samples of equal
sizes. These samples are called folds. For each learning set, the prediction function uses k-1
folds, and the rest of the folds are used for the test set. This approach is a very popular CV
approach because it is easy to understand, and the output is less biased than other methods.
The steps for k-fold cross-validation are:
• Split the input dataset into K groups
• For each group:
• Take one group as the reserve or test data set.
• Use remaining groups as the training dataset
• Fit the model on the training set and evaluate the performance of the model
using the test set.

Let's take an example of 5-folds cross-validation. So, the dataset is grouped into 5 folds. On 1st
iteration, the first fold is reserved for test the model, and rest are used to train the model. On
2nd iteration, the second fold is used to test the model, and rest are used to train the model.
This process will continue until each fold is not used for the test fold.
Consider the below diagram:

3. Leave One Out Cross Validation (LOOCV)

This variation on cross-validation leaves one data point out of the training data. For instance, if
there are n data points in the original data sample, then the pieces used to train the model are
n-1, and p points will be used as the validation set.
This cycle is repeated in all of the combinations where the original sample can be separated in
such a way. After this, the mean of the error is taken for all trials to give overall effectiveness.
We consider that the number of possible combinations is equal to the number of data points in
the original sample represented by n.
Introduction to Linear Regression:

 Linear regression a way to explain the relationship between a variable y given the
values of some other variable x. The target variable, y, is generally called the
"dependent variable". The other variable x is called the "independent variable".

 Regression, in general, is the problem of predicting the value of the dependent

variable. Linear regression is known as "linear" because the relation of the dependent
to the independent variables is a linear function of some parameters. Regression
methods that do not have a linear function are called nonlinear regression models.

 The term independent variable means that its value can be chosen at will, and the
dependent variable will adjust based on the value of the independent variable. Linear
regression helps us predict how much that adjustment is going to be. Depending on the
kind of relationship between the dependant and independent variable, the adjustment
can be positive or negative or zero.

There are several real-life applications of linear regression. We can classify most applications
fall into one of the following two broad categories. We usually use linear regression when we
want to know:
 The nature and strength relationship between two variables (e.g. is there a positive
relationship between rainfall and crop growth).
 The predicted value of the dependent variable for a given value of the independent
variable (e.g. the amount of crop growth for a certain level of rainfall).

 The diagram above shows a scatter plot of two variables x and y. The red line is known
as the “line of best fit”. Linear regression helps us to find the line of best fit.

 Once the line of best fit has been determined, we can easily say that the variables x and
y have a positive relationship, that is y increases as x increases. We can also find the
value of y at any value of x from the equation of the line of best fit.

 The equation for a simple linear regression is of the form

Y=m*X+b
where
Y is Dependent Variable,
X is an Independent Variable,
b is intercept and
m is slope.

Mean Absolute Error:

 The basic objective of linear regression is to try to reduce the vertical distance between
the line and the data points to make it minimum. This process is called "fitting the line
to the data." To do so we can use the Mean Absolute Error (MAE).

 While figuring out the line of best fit, we want to minimize the deviation of the line
from the actual data points. We do this by finding the mean absolute error and
minimizing it.

In the diagram above, the red dots are observed values, the blue line is the line of best fit and
the green lines represent the errors or residuals.

Mean Absolute Error measures the average magnitude of the errors in a set of predictions,
without considering their direction.

Root Mean Square Deviation:

 The Root Mean Square Deviation is used to determine how close the observed points
are to the model’s predicted values. Mathematically, the Root Mean Square Deviation
is the square root of the variance of the residuals.

 In real-life scenarios, it is best if the RSME value is small. A small RSME value means
that the model is a better bit to the data and thus more accurate. A large RSME value
shows that the model is not a good fit and might need to be retrained.
 The actual value of the RSME depends on the data and degree of accuracy required.
For example, a RSME of 1 cm might not be significant for designing a building but
will be very significant for designing a precision tool.
Multiple Linear Regression:

Multiple Linear Regression uses multiple independent variables to predict the outcome of a
dependent variable. For example, effects of age, weight and height on cholesterol levels of
an individual. Here, age, weight and height are independent variables and cholesterol level is
dependent variable because it is dependent on the factors age, height and weight.

A simple regression equation has an intercept on the right-hand side and an explanatory
variable with a coefficient. A multiple regression has multiple variables on the right-hand side,
each with its slope coefficient.

The basic model of multiple linear regression is

where i = 1, 2, 3, ..., n for each observation.

In the above formula, we consider n number of observations of one dependent variable and p
number of independent variables.

Therefore, Yi is the ith observation of the jth independent variable where j = 1, 2, 3, ..., p.
The values βj represent the features to be estimated and εi is the ith independent identically
distributed normal error. In more general multivariate linear regression, the above
observations can be defined into one equation.

Yij = β0j + β1jXi1 + β2jXi2 + β3jXi3 + ...+ βpjXip + εi

for all observations indexed as i = 1, 2, .., n and for all dependent variables indexed as j =
1,2, ..., m.
Non-linear Regression:

Non-linear regression is more flexible than linear regression. The formula for non-linear
regression is y ~ f (x, β) where x is a vector of independent variables and y is the dependent
variable. These functions are called non-linear functions.

Examples of non-linear functions include exponential functions, logarithmic functions,

trigonometric functions, power functions, etc.

The graph of linear regression follows the equation of line and the graph of non-linear
regression follows equation of a curve. In the formula, f (x, β) varies depending on the type of
curve.

For example, Kristy started a business 5 months back. He observed that his income increases
exponentially every month. We will try to predict what would be his income next month using
non- linear regression.

From the graph, we can see that here f(x, β) = 5x. So, the equation for this non- linear graph in
this case will be y = 5x. Therefore, we can predict that the income in the 6th month would be
= 15,625.
What is a Decision Tree ? How does it work ?
 Decision tree is a type of supervised learning algorithm (having a pre-defined target
variable) that is mostly used in classification problems.
 It works for both categorical and continuous input and output variables.
 In this technique, we split the population or sample into two or more homogeneous
sets (or sub-populations) based on most significant splitter / differentiator in input
variables.

Example:
 Let’s say we have a sample of 30 students with three variables Gender (Boy/ Girl),
Class( IX/ X) and Height (5 to 6 ft).
 15 out of these 30 play cricket in leisure time.
 Now, I want to create a model to predict who will play cricket during leisure period?
 In this problem, we need to segregate students who play cricket in their leisure time
based on highly significant input variable among all three.
 This is where decision tree helps, it will segregate the students based on all values of
three variable and identify the variable, which creates the best homogeneous sets of
students (which are heterogeneous to each other).
 In the snapshot below, you can see that variable Gender is able to identify best
homogeneous sets compared to the other two variables.

 As mentioned above, decision tree identifies the most significant variable and it‟s
value that gives best homogeneous sets of population.
 Now the question which arises is, how does it identify the variable and the split?
 To do this, decision tree uses various algorithms, which we will shall discuss in the
following section.

Types of Decision Trees:

Types of decision tree is based on the type of target variable we have. It can be of two types:

1. Categorical Variable Decision Tree: Decision Tree which has categorical target
variable then it called as categorical variable decision tree.
Example:- In above scenario of student problem, where the target variable was “Student will
play cricket or not” i.e. YES or NO.

2. Continuous Variable Decision Tree: Decision Tree has continuous target variable
then it is called as Continuous Variable Decision Tree.
Example:-
 Let’s say we have a problem to predict whether a customer will pay his renewal
premium with an insurance company (yes/ no).
 Here we know that income of customer is a significant variable but insurance company
does not have income details for all customers.
 Now, as we know this is an important variable, then we can build a decision tree to
predict customer income based on occupation, product and various other variables.
 In this case, we are predicting values for continuous variable.

Important Terminology related to Decision Trees:

Let‟s look at the basic terminology used with Decision trees:

1. Root Node: It represents entire population or sample and this further gets divided into
two or more homogeneous sets.

2. Splitting: It is a process of dividing a node into two or more sub-nodes.

3. Decision Node: When a sub-node splits into further sub-nodes, then it is called decision
node.

4. Leaf/ Terminal Node: Nodes do not split is called Leaf or Terminal node.

Pruning: When we remove sub-nodes of a decision node, this process is called pruning. You
can say opposite process of splitting.

5. Branch / Sub-Tree: A sub section of entire tree is called branch or sub-tree.

6. Parent and Child Node: A node, which is divided into sub-nodes is called parent node
of sub-nodes where as sub-nodes are the child of parent node.

These are the terms commonly used for decision trees. As we know that every algorithm has
advantages and disadvantages, below are the important factors which one should know.

Advantages:

1. Easy to Understand: Decision tree output is very easy to understand even for people from
non-analytical background. It does not require any statistical knowledge to read and interpret
them. Its graphical representation is very intuitive and users can easily relate their hypothesis.

2. Useful in Data exploration: Decision tree is one of the fastest way to identify most
significant variables and relation between two or more variables. With the help of decision
trees, we can create new variables / features that has better power to predict target variable.
You can refer article (Trick to enhance power of regression model) for one such trick. It can
also be used in data exploration stage. For example, we are working on a problem where we
have information available in hundreds of variables, there decision tree will help to identify
most significant variable.

3. Less data cleaning required: It requires less data cleaning compared to some other
modeling techniques. It is not influenced by outliers and missing values to a fair degree.

4. Data type is not a constraint: It can handle both numerical and categorical variables.

5. Non Parametric Method: Decision tree is considered to be a non-parametric method. This

means that decision trees have no assumptions about the space distribution and the classifier
structure.

Disadvantages:

1. Over fitting: Over fitting is one of the most practical difficulty for decision tree models.
This problem gets solved by setting constraints on model parameters and pruning .

2. Not fit for continuous variables: While working with continuous numerical variables,
decision tree looses information when it categorizes variables in different categories.
Regression Trees vs Classification Trees:
We all know that the terminal nodes (or leaves) lies at the bottom of the decision tree.
This means that decision trees are typically drawn upside down such that leaves are the
bottom & roots are the tops (shown below).

Both the trees work almost similar to each other, let’s look at the primary differences &
similarity between classification and regression trees:

1. Regression trees are used when dependent variable is continuous.

Classification trees are used when dependent variable is categorical.

2. In case of regression tree, the value obtained by terminal nodes in the training data is the
mean response of observation falling in that region. Thus, if an unseen data observation falls
in that region, we’ll make its prediction with mean value.

3. In case of classification tree, the value (class) obtained by terminal node in the training
data is the mode of observations falling in that region. Thus, if an unseen data observation
falls in that region, we‟ll make its prediction with mode value.

4. Both the trees divide the predictor space (independent variables) into distinct and non-
overlapping regions. For the sake of simplicity, you can think of these regions as high
dimensional boxes or boxes.

5. Both the trees follow a top-down greedy approach known as recursive binary splitting. We
call it as “top-down” because it begins from the top of tree when all the observations are
available in a single region and successively splits the predictor space into two new branches
down the tree. It is known as “greedy” because, the algorithm cares (looks for best variable
available) about only the current split, and not about future splits which will lead to a better
tree.

6. This splitting process is continued until a user defined stopping criteria is reached. For
example: we can tell the the algorithm to stop once the number of observations per node
becomes less than 50.

7. In both the cases, the splitting process results in fully grown trees until the stopping criteria
is reached. But, the fully grown tree is likely to overfit data, leading to poor accuracy on
unseen data. This bring “pruning”. Pruning is one of the technique used tackle overfitting.
How does a tree decide where to split?
The decision of making strategic splits heavily affects a tree’s accuracy. The decision criteria
is different for classification and regression trees.

Decision trees use multiple algorithms to decide to split a node in two or more sub-nodes. The
creation of sub-nodes increases the homogeneity of resultant sub-nodes. In other words, we
can say that purity of the node increases with respect to the target variable. Decision tree splits
the nodes on all available variables and then selects the split which results in most
homogeneous sub-nodes.

The algorithm selection is also based on type of target variables. Let’s look at the four most
commonly used algorithms in decision tree:

Gini Index:
Gini index says, if we select two items from a population at random then they must be of same
class and probability for this is 1 if population is pure.

1. It works with categorical target variable “Success” or “Failure”.

2. It performs only Binary splits
3. Higher the value of Gini higher the homogeneity.
4. CART (Classification and Regression Tree) uses Gini method to create binary splits.

Steps to Calculate Gini for a split:

1. Calculate Gini for sub-nodes, using formula sum of square of probability for success and
failure (p^2+q^2).
2. Calculate Gini for split using weighted Gini score of each node of that split

Example: – Here we want to segregate the students based on target variable ( playing cricket
or not ).
In the snapshot below, we split the population using two input variables Gender and Class.
Now, I want to identify which split is producing more homogeneous sub-nodes using Gini
index.

Split on Gender:
1. Calculate, Gini for sub-node Female = (0.2)*(0.2)+(0.8)*(0.8)=0.68
2. Gini for sub-node Male = (0.65)*(0.65)+(0.35)*(0.35)=0.55
3. Calculate weighted Gini for Split Gender = (10/30)*0.68+(20/30)*0.55 = 0.59

Similar for Split on Class:

1. Gini for sub-node Class IX = (0.43)*(0.43)+(0.57)*(0.57)=0.51
2. Gini for sub-node Class X = (0.56)*(0.56)+(0.44)*(0.44)=0.51
3. Calculate weighted Gini for Split Class = (14/30)*0.51+(16/30)*0.51 = 0.51
Above, you can see that Gini score for Split on Gender is higher than Split on Class, hence,
the node split will take place on Gender.

Chi-Square
It is an algorithm to find out the statistical significance between the differences between sub-
nodes and parent node. We measure it by sum of squares of standardized differences between
observed and expected frequencies of target variable.

1. It works with categorical target variable “Success” or “Failure”.

2. It can perform two or more splits.
3. Higher the value of Chi-Square higher the statistical significance of differences between
sub-node and Parent node.
4. Chi-Square of each node is calculated using formula,
5. Chi-square = ((Actual – Expected)^2 / Expected)^1/2
6. It generates tree called CHAID (Chi-square Automatic Interaction Detector)

Steps to Calculate Chi-square for a split:

1. Calculate Chi-square for individual node by calculating the deviation for Success and
Failure both
2. Calculated Chi-square of Split using Sum of all Chi-square of success and Failure of each
node of the split

Example: Let‟s work with above example that we have used to calculate Gini.

Split on Gender:
1. First we are populating for node Female, Populate the actual value for “Play Cricket” and
“Not Play Cricket”, here these are 2 and 8 respectively.
2. Calculate expected value for “Play Cricket” and “Not Play Cricket”, here it would be 5 for
both because parent node has probability of 50% and we have applied same probability
on Female count(10).
3. Calculate deviations by using formula, Actual – Expected.
It is for “Play Cricket” (2 – 5 = -3)
and for “Not play cricket” ( 8 – 5 = 3).
4. Calculate Chi-square of node for “Play Cricket” and “Not Play Cricket” using formula with
formula, = ((Actual – Expected)^2 / Expected)^1/2.
You can refer below table for calculation.
5. Follow similar steps for calculating Chi-square value for Male node.
6. Now add all Chi-square values to calculate Chi-square for split Gender.
Split on Class:
Perform similar steps of calculation for split on Class and you will come up with below table.

Above, you can see that Chi-square also identify the Gender split is more significant compare
to Class.
Information Gain:
 Look at the image below and think which node can be described easily.
 I am sure, your answer is C because it requires less information as all values are
similar.
 On the other hand, B requires more information to describe it and A requires the
maximum information.
 In other words, we can say that C is a Pure node, B is less Impure and A is more
impure.

 Now, we can build a conclusion that less impure node requires less information to
describe it.
 And, more impure node requires more information.
 Information theory is a measure to define this degree of disorganization in a system
known as Entropy.
 If the sample is completely homogeneous, then the entropy is zero and if the sample is
an equally divided (50% – 50%), it has entropy of one.

 Entropy can be calculated using formula:-

 Here p and q is probability of success and failure respectively in that node.

 Entropy is also used with categorical target variable.
 It chooses the split which has lowest entropy compared to parent node and other splits.
 The lesser the entropy, the better it is.

Steps to calculate entropy for a split:

1. Calculate entropy of parent node
2. Calculate entropy of each individual node of split and calculate weighted average of all
sub-nodes available in split.
Example: Let‟s use this method to identify best split for student example.

1. Entropy for parent node = -(15/30) log2 (15/30) – (15/30) log2 (15/30) = 1.
Here 1 shows that it is a impure node.

2. Entropy for Female node = -(2/10) log2 (2/10) – (8/10) log2 (8/10) = 0.72 and for male
node, -(13/20) log2 (13/20) – (7/20) log2 (7/20) = 0.93

3. Entropy for split Gender = Weighted entropy of sub-nodes = (10/30)*0.72 + (20/30)*0.93=

0.86

4. Entropy for Class IX node, -(6/14) log2 (6/14) – (8/14) log2 (8/14) = 0.99 and for Class X
node,
-(9/16) log2 (9/16) – (7/16) log2 (7/16) = 0.99.

5. Entropy for split Class = (14/30)*0.99 + (16/30)*0.99 = 0.99

Above, you can see that entropy for Split on Gender is the lowest among all, so the tree will
split on Gender. We can derive information gain from entropy as 1- Entropy.

Reduction in Variance
 Till now, we have discussed the algorithms for categorical target variable.
 Reduction in variance is an algorithm used for continuous target variables (regression
problems).
 This algorithm uses the standard formula of variance to choose the best split.
 The split with lower variance is selected as the criteria to split the population:

Above X-bar is mean of the values, X is actual and n is number of values.

Steps to calculate Variance:
1. Calculate variance for each node.
2. Calculate variance for each split as weighted average of each node variance.

Example:- Let‟s assign numerical value 1 for play cricket and 0 for not playing cricket.
Now follow the steps to identify the right split:
1. Variance for Root node, here mean value is (15*1 + 15*0)/30 = 0.5 and we have 15 one
and 15 zero.
Now variance would be ((1-0.5)^2+(1-0.5)^2+....15 times+(0-0.5)^2+(0-
0.5)^2+...15 times) / 30, this can be written as (15*(1-0.5)^2+15*(0-0.5)^2) / 30 = 0.25

2. Mean of Female node = (2*1+8*0)/10=0.2 and

Variance = (2*(1-0.2)^2+8*(0-0.2)^2) / 10 =0.16

3. Mean of Male Node = (13*1+7*0)/20=0.65 and

Variance = (13*(1-0.65)^2+7*(0-0.65)^2) /20 = 0.23

4. Variance for Split Gender = Weighted Variance of Sub-nodes = (10/30)*0.16 + (20/30)

*0.23= 0.21

5. Mean of Class IX node = (6*1+8*0)/14=0.43 and

Variance = (6*(1-0.43)^2+8*(0-0.43)^2) /14= 0.24

6. Mean of Class X node = (9*1+7*0)/16=0.56 and

Variance = (9*(1-0.56)^2+7*(0-0.56)^2) / 16= 0.25

7. Variance for Split Gender = (14/30)*0.24 + (16/30) *0.25 = 0.25

Above, you can see that Gender split has lower variance compare to parent node, so the split
would take place on Gender variable.

Until here, we learnt about the basics of decision trees and the decision making process
involved to choose the best splits in building a tree model.

As I said, decision tree can be applied both on regression and classification problems.
 UNDERFITTING & OVERFITTING
AGENDA:
1. Errors in machine Learning
2. Bias and Variance
3. Bias-Variance trade-off
4. Underfitting and Overfitting
Bias and Variance in Machine Learning:
 Machine learning is a branch of Artificial Intelligence, which allows machines to
perform data analysis and make predictions.
 However, if the machine learning model is not accurate, it can make predictions errors,
and these prediction errors are usually known as Bias and Variance.
 In machine learning, these errors will always be present as there is always a slight
difference between the model predictions and actual predictions.
 The main aim of ML/data science analysts is to reduce these errors in order to get
more accurate results.
 In this topic, we are going to discuss bias and variance, Bias-variance trade-off,
Underfitting and Overfitting.
 But before starting, let's first understand what errors in Machine learning are?

Errors in Machine Learning:

 In machine learning, an error is a measure of how accurately an algorithm can make
predictions for the previously unknown dataset.
 On the basis of these errors, the machine learning model is selected that can perform
best on the particular dataset.
 There are mainly two types of errors in machine learning, which are:
 • Reducible errors: These errors can be reduced to improve the model accuracy. Such
errors can further be classified into bias and Variance.

• Irreducible errors: These errors will always be present in the model regardless of
which algorithm has been used. The cause of these errors is unknown variables whose
value can't be reduced.

What is Bias?
 In general, a machine learning model analyses the data, find patterns in it and make
predictions.
 While training, the model learns these patterns in the dataset and applies them to test
data for prediction.
 While making predictions, a difference occurs between prediction values made by
the model and actual values/expected values, and this difference is known as bias
errors or Errors due to bias.
 It can be defined as an inability of machine learning algorithms such as Linear
Regression to capture the true relationship between the data points.
 Each algorithm begins with some amount of bias because bias occurs from
assumptions in the model, which makes the target function simple to learn. A model
has either:
• Low Bias: A low bias model will make fewer assumptions about the form of the target
function.
• High Bias: A model with a high bias makes more assumptions, and the model
becomes unable to capture the important features of our dataset. A high bias model
also cannot perform well on new data.
Generally, a linear algorithm has a high bias, as it makes them learn fast. The simpler the
algorithm, the higher the bias it has likely to be introduced. Whereas a nonlinear algorithm
often has low bias.
Some examples of machine learning algorithms with low bias are Decision Trees, k-Nearest
Neighbours and Support Vector Machines.
At the same time, an algorithm with high bias is Linear Regression, Linear Discriminant
Analysis and Logistic Regression.

Ways to reduce High Bias:

High bias mainly occurs due to a much simple model. Below are some ways to reduce the
high bias:
• Increase the input features as the model is underfitted.
• Decrease the regularization term.
• Use more complex models, such as including some polynomial features.

What is a Variance Error?

 The variance would specify the amount of variation in the prediction if the different
training data was used. In simple words, variance tells that how much a random
variable is different from its expected value.
 Ideally, a model should not vary too much from one training dataset to another, which
means the algorithm should be good in understanding the hidden mapping between
inputs and output variables.
 Variance errors are either of low variance or high variance.
 Low variance means there is a small variation in the prediction of the target function
with changes in the training data set.
 At the same time, High variance shows a large variation in the prediction of the target
function with changes in the training dataset.
 A model that shows high variance learns a lot and perform well with the training
dataset, and does not generalize well with the unseen dataset.
 As a result, such a model gives good results with the training dataset but shows high
error rates on the test dataset.
 Since, with high variance, the model learns too much from the dataset, it leads to
overfitting of the model.
 A model with high variance has the below problems:
• A high variance model leads to overfitting.
• Increase model complexities.
  Usually, nonlinear algorithms have a lot of flexibility to fit the model, have high
variance.

Some examples of machine learning algorithms with low variance are, Linear Regression,
Logistic Regression, and Linear discriminant analysis. At the same time, algorithms with
high variance are decision tree, Support Vector Machine, and K-nearest neighbours.

Ways to Reduce High Variance:

• Reduce the input features or number of parameters as a model is overfitted.
• Do not use a much complex model.
• Increase the training data.
• Increase the Regularization term.

Different Combinations of Bias-Variance

There are four possible combinations of bias and variances, which are represented by the
below diagram:
 1. Low-Bias, Low-Variance:
The combination of low bias and low variance shows an ideal machine learning model.
However, it is not possible practically.
2. Low-Bias, High-Variance: With low bias and high variance, model predictions are
inconsistent and accurate on average. This case occurs when the model learns with a
large number of parameters and hence leads to an overfitting
3. High-Bias, Low-Variance: With High bias and low variance, predictions are
consistent but inaccurate on average. This case occurs when a model does not learn
well with the training dataset or uses few numbers of the parameter. It leads to
underfitting problems in the model.
4. High-Bias, High-Variance:
With high bias and high variance, predictions are inconsistent and also inaccurate on
average.

How to identify High variance or High Bias?

High variance can be identified if the model has:

• Low training error and high test error.

High Bias can be identified if the model has:
• High training error and the test error is almost similar to training error.

Bias-Variance Trade-Off
 While building the machine learning model, it is really important to take care of bias
and variance in order to avoid overfitting and underfitting in the model.
 If the model is very simple with fewer parameters, it may have low variance and high
bias.
 Whereas, if the model has a large number of parameters, it will have high variance and
low bias.
  So, it is required to make a balance between bias and variance errors, and this balance
between the bias error and variance error is known as the Bias-Variance trade-off.

For an accurate prediction of the model, algorithms need a low variance and low bias. But this
is not possible because bias and variance are related to each other:
• If we decrease the variance, it will increase the bias.
• If we decrease the bias, it will increase the variance.
 Bias-Variance trade-off is a central issue in supervised learning.
 Ideally, we need a model that accurately captures the regularities in training data and
simultaneously generalizes well with the unseen dataset.
 Unfortunately, doing this is not possible simultaneously.
 Because a high variance algorithm may perform well with training data, but it may
lead to overfitting to noisy data.
 Whereas, high bias algorithm generates a much simple model that may not even
capture important regularities in the data.
 So, we need to find a sweet spot between bias and variance to make an optimal model.
Hence, the Bias-Variance trade-off is about finding the sweet spot to make a balance
between bias and variance errors.
Underfitting in Machine Learning
 A statistical model or a machine learning algorithm is said to have underfitting when it
cannot capture the underlying trend of the data, i.e., it only performs well on training
data but performs poorly on testing data. (It’s just like trying to fit undersized pants!)
Underfitting destroys the accuracy of our machine-learning model.
 Its occurrence simply means that our model or the algorithm does not fit the data well
enough.
 It usually happens when we have less data to build an accurate model and also when
we try to build a linear model with fewer non-linear data.
 In such cases, the rules of the machine learning model are too easy and flexible to be
applied to such minimal data, and therefore the model will probably make a lot of
wrong predictions.
 Underfitting can be avoided by using more data and also reducing the features by
feature selection.
In a nutshell, Underfitting refers to a model that can neither performs well on the training data
nor generalize to new data.

Reasons for Underfitting

1. High bias and low variance.
2. The size of the training dataset used is not enough.
3. The model is too simple.
4. Training data is not cleaned and also contains noise in it.

Techniques to Reduce Underfitting

1. Increase model complexity.
2. Increase the number of features, performing feature engineering.
3. Remove noise from the data.
4. Increase the number of epochs or increase the duration of training to get better results.

Overfitting in Machine Learning

 A statistical model is said to be overfitted when the model does not make accurate
predictions on testing data.
 When a model gets trained with so much data, it starts learning from the noise and
inaccurate data entries in our data set.
 And when testing with test data results in High variance.
 Then the model does not categorize the data correctly, because of too many details and
noise.
  The causes of overfitting are the non-parametric and non-linear methods because these
types of machine learning algorithms have more freedom in building the model based
on the dataset and therefore they can really build unrealistic models.
 A solution to avoid overfitting is using a linear algorithm if we have linear data or
using the parameters like the maximal depth if we are using decision trees.
In a nutshell, Overfitting is a problem where the evaluation of machine learning algorithms on
training data is different from unseen data.

Reasons for Overfitting:

1. High variance and low bias.
2. The model is too complex.
3. The size of the training data.

Techniques to Reduce Overfitting

1. Increase training data.
2. Reduce model complexity.
3. Early stopping during the training phase (have an eye over the loss over the training
period as soon as loss begins to increase stop training).
4. Ridge Regularization and Lasso Regularization.
5. Use dropout for neural networks to tackle overfitting.