Model Assesment

There are various ways to evaluate a machine-learning model. We can use MSE for
Regression; Precision, Recall, and ROC (Receiver operating characteristics) for a
Classification Problem along with Absolute Error. In a similar way, Bias and
Variance help us in parameter tuning and deciding better-fitted models among
several built.
Bias is one type of error that occurs due to wrong assumptions about data such as
assuming data is linear when in reality, data follows a complex function. On the
other hand, variance gets introduced with high sensitivity to variations in training
data. This also is one type of error since we want to make our model robust against
noise. There are two types of error in machine learning. Reducible error and
Irreducible error. Bias and Variance come under reducible error.

Bias :

Bias is simply defined as the inability of the model because of that there is some
difference or error occurring between the model's predicted value and the actual
value. These differences between actual or expected values and the predicted
values are known as error or bias error or error due to bias. Bias is a systematic
error that occurs due to wrong assumptions in the machine learning process.

• Low Bias: Low bias value means fewer assumptions are taken to build the target
function. In this case, the model will closely match the training dataset.
• High Bias: High bias value means more assumptions are taken to build the
target function. In this case, the model will not match the training dataset
closely.
The high-bias model will not be able to capture the dataset trend. It is considered
as the underfitting model which has a high error rate. It is due to a very simplified
algorithm.
Variance :

Variance is the measure of spread in data from its mean position. In machine
learning variance is the amount by which the performance of a predictive model
changes when it is trained on different subsets of the training data. More
specifically, variance is the variability of the model that how much it is sensitive to
another subset of the training dataset. i.e. how much it can adjust on the new
subset of the training dataset.
Let Y be the actual values of the target variable, and Y^ be the predicted values
of the target variable. Then the variance of a model can be measured as the
expected value of the square of the difference between predicted values and the
expected value of the predicted values.

where E[Y^]is the expected value of the predicted values. Here expected value is
averaged over all the training data.

Variance errors are either low or high-variance errors.

• Low variance: Low variance means that the model is less sensitive to changes in
the training data and can produce consistent estimates of the target function
with different subsets of data from the same distribution. However, low
variance can also indicate underfitting if the model is too simple and fails to
capture the underlying patterns in the data. This is when the model performs
poorly on both the training data and testing data.
• High variance: High variance means that the model is very sensitive to changes
in the training data and can result in significant changes in the estimate of the
target function when trained on different subsets of data from the same
distribution. This is the case of overfitting when the model performs well on the
training data but poorly on new, unseen test data. It fits the training data too
closely that it fails on the new training dataset.

There can be four combinations between bias and variance.

• High Bias, Low Variance: A model with high bias and low variance is said to be
underfitting.
• High Variance, Low Bias: A model with high variance and low bias is said to be
overfitting.
• High-Bias, High-Variance: A model has both high bias and high variance, which
means that the model is not able to capture the underlying patterns in the data
(high bias) and is also too sensitive to changes in the training data (high
variance). As a result, the model will produce inconsistent and inaccurate
predictions on average.
• Low Bias, Low Variance: A model that has low bias and low variance means that
the model is able to capture the underlying patterns in the data (low bias) and is
not too sensitive to changes in the training data (low variance). This is the ideal
scenario for a machine learning model, as it is able to generalize well to new,
unseen data and produce consistent and accurate predictions. But in practice,
it's not possible.

Now we know that the ideal case will be Low Bias and Low variance, but in
practice, it is not possible. So, we trade off between Bias and variance to achieve a
balanced bias and variance. A model with balanced bias and variance is said to
have optimal generalization performance. This means that the model is able to
capture the underlying patterns in the data without overfitting or underfitting.
Ensemble Learning
Ensemble learning is a method where we use many small models instead of just
one. Each of these models may not be very strong on its own, but when we put
their results together, we get a better and more accurate answer. It's like asking a
group of people for advice instead of just one person—each one might be a little
wrong, but together, they usually give a better answer.

Ensemble learning is a machine learning technique that aggregates two or more

learners (e.g. regression models, neural networks) in order to produce better
predictions. In other words, an ensemble model combines several individual models
to produce more accurate predictions than a single model alone. At times, sources
may refer to this technique as committee-based learning. Ensemble learning rests on
the principle that a collectivity of learners yields greater overall accuracy than an
individual learner.

Types :

Bagging :

Bagging (bootstrap aggregating) is an ensemble method that involves training

multiple models independently on random subsets of the data, and aggregating their
predictions through voting or averaging.

In detail, each model is trained on a random subset of the data sampled with
replacement, meaning that the individual data points can be chosen more than once.
This random subset is known as a bootstrap sample. By training models on different
bootstraps, bagging reduces the variance of the individual models. It also avoids
overfitting by exposing the constituent models to different parts of the dataset.
 The predictions from all the sampled models are then combined through a simple
averaging to make the overall prediction. This way, the aggregated model
incorporates the strengths of the individual ones and cancels out their errors.
Bagging is particularly effective in reducing variance and overfitting, making the
model more robust and accurate, especially in cases where the individual models are
prone to high variability.

Example :
Random Forest is a machine learning algorithm that uses many decision trees to
make better predictions. Each tree looks at different random parts of the data and
their results are combined by voting for classification or averaging for regression
which makes it as ensemble learning technique. This helps in improving accuracy
and reducing errors.
Working of Random Forest Algorithm
• Create Many Decision Trees: The algorithm makes many decision trees each
using a random part of the data. So every tree is a bit different.
• Pick Random Features: When building each tree it doesn’t look at all the
features (columns) at once. It picks a few at random to decide how to split the
data. This helps the trees stay different from each other.
• Each Tree Makes a Prediction: Every tree gives its own answer or prediction
based on what it learned from its part of the data.
• Combine the Predictions: For classification we choose a category as the final
answer is the one that most trees agree on i.e majority voting and
for regression we predict a number as the final answer is the average of all the
trees predictions.

Boosting :
Boosting is a powerful ensemble learning method in machine learning, specifically
designed to improve the accuracy of predictive models by combining multiple weak
learners—models that perform only slightly better than random guessing—into a
single, strong learner.
The essence of boosting lies in the iterative process where each weak learner is
trained to correct the errors of its predecessor, gradually enhancing the overall
model's performance. By focusing on the mistakes made by earlier models, boosting
turns a collection of weak learners into a more accurate model.
Boosting transforms weak learners into one unified, strong learner through a
systematic process that focuses on reducing errors in sequential model training. The
steps involved include:
1. Select Initial Weights: Assign initial weights to all data points to indicate their
importance in the learning process.
2. Train Sequentially: Train the first weak learner on the data. After evaluating its
performance, increase the weights of misclassified instances. This makes the next
weak learner focus more on the harder cases.
3. Iterate the Process: Repeat the process of adjusting weights and training subsequent
learners. Each new model focuses on the weaknesses of the ensemble thus far.
4. Combine the Results: Aggregate the predictions of all weak learners to form the final
output. The aggregation is typically weighted, where more accurate learners have
more influence.

Types of Boosting Algorithms

Let’s take a look at some of the most well-known boosting algorithms.

AdaBoost (Adaptive Boosting)

AdaBoost means Adaptive Boosting which is a ensemble learning technique that

combines multiple weak classifiers to create a strong classifier. It works by
sequentially adding classifiers to correct the errors made by previous models giving
more weight to the misclassified data points

Gradient Boosting

Gradient Boosting is a boosting algorithm and here each new model is trained to
minimize the loss function such as mean squared error or cross-entropy of the
previous model using gradient descent. In each iteration the algorithm computes
the gradient of the loss function with respect to predictions and then trains a new
weak model to minimize this gradient. Predictions of the new model are then
added to the ensemble (all models prediction) and the process is repeated until a
stopping criterion is met.

--------------------------------------------------------------------------------------------------------------

Maximum Likelihood Estimation

Maximum Likelihood Estimation (MLE) is a key method in statistical modeling, used
to estimate parameters by finding the best fit to the observed data.
Likelihood Estimation is a statistical method used to estimate the parameters of a
probability distribution or a statistical model based on observed data. Unlike
traditional estimation methods that focus on finding the "best-fitting" parameters,
likelihood estimation frames the problem in terms of the likelihood function.
The goal of likelihood estimation is to find the parameter values that maximize the
likelihood function. These parameter values are called Maximum Likelihood
Estimates (MLEs). Once the MLEs are obtained, they provide estimates of the
parameters that best explain the observed data. These estimates can be used for
inference, prediction, or further analysis depending on the context of the problem.
Bayesian Theory in Machine Learning

Bayesian theory forms one of the most powerful and conceptually elegant
foundations of modern machine learning. It provides a mathematical framework for
dealing with uncertainty in data and model predictions by applying the principles of
probability theory and Bayes’ theorem. Unlike traditional deterministic approaches,
Bayesian learning focuses not only on finding a single best estimate of a parameter
or outcome but on representing and updating degrees of belief about possible
outcomes as new information becomes available.

At the heart of Bayesian theory lies Bayes’ Theorem, formulated by Reverend

Thomas Bayes in the 18th century. It provides a mechanism to update our existing
beliefs (called prior beliefs) in light of new evidence (called data). Mathematically,
Bayes’ theorem is expressed as: