ADVANCED MACHINE LEARNING AND DEEP LEARNING

Module-1: Supervised Learning

Supervised learning is a machine learning approach where the model learns from labeled data —
meaning each input (X) has a known output (Y).
The goal is to learn a mapping f: X → Y to make predictions on unseen data.

In supervised learning, the main objective is to build a model that not only performs well on the training
data but also makes accurate predictions on new, unseen data. This ability of a model to adapt to new
inputs that share similar patterns with the training data is known as generalization. A model that
generalizes well can capture the essential structure of the data without being overly sensitive to noise
or irrelevant details. However, if the model is too closely tuned to the training data, it may overfit,
meaning it learns the random fluctuations and noise present in the training examples. Conversely, when
a model is too simplistic and fails to capture the underlying structure of the data, it is said to underfit.
The key challenge in supervised learning is to balance these two extremes to achieve the best
generalization performance.

Overfitting

• Definition:
When a model learns the training data too well, including noise and random fluctuations,
causing poor performance on new data.
• Symptoms:
o High accuracy on training data.
o Low accuracy on test data.
• Cause:
o Model is too complex for the available data.
o Too many parameters relative to data quantity.
• Example (Boat Buying Rule):
o Rule: “If customer is older than 45 and has <3 children or is not divorced → buys a
boat.”
o Works perfectly on training data but fails on new data — too specific.
• Analogy:
Like memorizing answers instead of learning concepts.

Underfitting

• Definition:
When a model is too simple to capture underlying data patterns.
• Symptoms:
o Poor accuracy on both training and test data.
• Cause:
o Model has too few parameters or uses an overly simplistic assumption.
• Example:
o Rule: “Everyone who owns a house buys a boat.”
→ Fails to represent data complexity.

Example

Consider a data scientist trying to predict whether a customer will buy a boat based on features like age,
marital status, or the number of children. If the scientist creates a very specific rule that fits every detail
of the training data—say, “People older than 45 and not divorced with fewer than three children buy
boats”—the rule may perfectly explain the given dataset. However, this overfitted model will likely fail
when new customer data is introduced because it captured coincidences specific to the training
examples rather than general trends. On the other hand, if the scientist makes a very broad rule, such as

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

“All homeowners buy boats,” it would be underfitted, as it ignores important variations in the data. The
best model lies somewhere between these two extremes, where it captures meaningful relationships
without memorizing the data.

The Trade-Off

• Model Complexity vs. Accuracy:

o As model complexity increases:
▪ Training accuracy increases.
▪ Test accuracy increases initially, then decreases after a point.

Relation of Model Complexity to Dataset Size

The complexity of a model should correspond to the amount and variety of data available in the training
set. A larger and more diverse dataset allows for building more complex models without the risk of
overfitting, since the model can learn from broader patterns. However, simply increasing the number of
similar or duplicate data points does not add value — the data must be varied to truly improve
generalization. In the boat-buying example, if the same rule holds true across many new and diverse
customer records, it becomes more reliable and representative, indicating that the model’s complexity
is appropriate for the dataset size.

1. k-Nearest Neighbors (k-NN)

k-NN is one of the simplest and most intuitive supervised machine learning algorithms — used for both
classification and regression tasks.
It does not build a mathematical model or equation during training.
Instead, it stores all the training data and makes predictions by comparing distances between data points.

When we want to predict the class or value for a new data point:

1. Compute the distance (usually Euclidean distance) between this new point and every point in
the training dataset.
2. Pick the k nearest (closest) points.
3. For classification, take a majority vote among those k neighbors (most common class wins).
For regression, take the average of their output values.

Visualizing 1-Nearest Neighbor (k=1)

[Link].plot_knn_classification(n_neighbors=1)

This adds three new (test) data points (marked as stars ★) to the forge dataset.

• The algorithm finds one nearest neighbor (the closest training point) for each test point.
• The predicted class is the label of that one nearest training point.
• The decision boundary (the line dividing class 0 vs class 1) is jagged because the model fits
every point perfectly.

🔍 Intuition:

• The model is very complex (it memorizes data).

• Perfect on training data, but poor on unseen data.
• This is called overfitting.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Visualizing k=3 (Three Neighbors)

[Link].plot_knn_classification(n_neighbors=3)

Now, instead of one neighbor, we look at the three nearest training points for each test point.

• The model takes a majority vote among those three points.

• The decision boundary becomes smoother.
• This means the model is less sensitive to noise or outliers.

Intuition:

• Slightly simpler model.

• Better generalization than k=1.
• Still flexible enough to capture data patterns.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Analyzing K-NeighborsClassifier

1. Visualizing Decision Boundaries

• For 2D datasets, the predictions of the k-Nearest Neighbors (k-NN) algorithm can be visualized
on an x-y plane.
• Each region in the plane is colored according to the class label assigned by the model.
• The line that separates these regions is called the decision boundary — it shows where the
model switches from predicting one class to another.

2. Effect of Changing k (Number of Neighbors)

• When k = 1, the model uses only the closest point to make predictions.
→ This leads to a highly complex model that fits the training data perfectly but overfits (poor
generalization).
• As k increases (e.g., 3 or 9), the decision boundary becomes smoother.
→ The model becomes simpler and less sensitive to noise.
• When k is very large (equal to total training points), the model predicts the majority class for
all inputs.
→ This leads to underfitting.

Summary:

• Small k → Complex model → High variance → Overfitting

• Large k → Simple model → High bias → Underfitting

3. Evaluating Model Performance

• To study how model complexity (via k) affects performance, the dataset (e.g., Breast Cancer
dataset) is split into training and test sets.
• The accuracy of the model is measured for different k values using the following steps:

Code: training_accuracy = []
test_accuracy = []

4. Observations from the Accuracy Plot

• k = 1:
o Training accuracy = 100% (model memorizes the data).
o Test accuracy = lower → model overfits.
• k between 5–7:
o Best balance between bias and variance.
o Model generalizes well → optimal k region.
• k ≥ 10:
o Both training and test accuracy drop → underfitting.
• The plot typically shows:
o Decreasing training accuracy as k increases.
o Test accuracy peaking at an intermediate k, then declining.

5. Key Takeaways

• The number of neighbors (k) controls the model complexity in k-NN.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

• Low k → high variance, high k → high bias.

• Best results occur at a moderate value of k, balancing generalization and accuracy.
• k-NN works well for small, well-scaled datasets, but can be computationally expensive for large
datasets.

K-Neighbors Regression

k-Nearest Neighbors Regression (k-NN Regression) is the regression counterpart of k-NN

classification. Instead of predicting a class label, it predicts a continuous numerical value.
The prediction is based on the average (mean) of the target values of the k nearest neighbors.

Working Principle

• For a new input point:

o Identify its k nearest neighbors (based on distance, usually Euclidean).
o Predict the average target value of these neighbors.

Example:

• When k = 1, the prediction is simply the target value of the nearest training sample.
• When k > 1, the prediction is the mean of those k neighbors.

Visual Understanding

• For 1-neighbor regression (k=1):

o The model exactly passes through all training points.
o The prediction line is very jagged → Overfitting.
• For 3 or 9 neighbors:
o Predictions become smoother and more generalized.
o Increasing k reduces sensitivity to noise → Underfitting may occur if k is too
large.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Linear Models

Linear Models are among the oldest and most widely used models in machine learning. They make
predictions using a linear combination of input features. Despite their simplicity, they are powerful
and interpretable, especially for high-dimensional data.

Linear regression assumes that the target variable (y) is a linear combination of the input features.
Geometrically:

o 1 feature → Line
o 2 features → Plane
o More features → Hyperplane

It’s a strong assumption, but in many real-world problems, this simple model performs surprisingly
well.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

When to Use Linear Models

When interpretability is important.

When you suspect linear relationships between features and target.
For datasets with few features and little noise.

Concept Meaning
Weights (w) Represent influence of each feature
Intercept (b) Base value of prediction when all features = 0
R² Score Goodness of fit measure (1 = perfect, 0 = mean predictor)
Overfitting High training score, low test score
Underfitting Both training and test scores are low

To handle overfitting, we use Regularized Linear Models such as:

• Ridge Regression → L2 Regularization

• Lasso Regression → L1 Regularization

Regularization is a technique used in machine learning to prevent overfitting by controlling model

complexity. When a model learns the training data too perfectly (including noise), it performs poorly
on new data — this is overfitting.
Regularization helps by penalizing overly complex models, forcing them to stay simpler and more
general.

Ridge Regression: Ridge Regression adds a penalty on the squared values of the coefficients
(weights).

In ridge regression, though, the coefficients (w) are chosen not only so that they predict well on the
train ing data, but also to fit an additional constraint. We also want the magnitude of coef ficients to be
as small as possible; in other words, all entries of w should be close to zero. Intuitively, this means each

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

feature should have as little effect on the outcome as possible (which translates to having a small slope),
while still predicting well. This constraint is an example of what is called regularization.

Lasso Regression: Lasso adds a penalty on the absolute values of the coefficients.
As with ridge regression, using the lasso also restricts coefficients to be close to zero, but in a slightly
different way, called L1 regularization.8 The consequence of L1 regularization is that when using the
lasso, some coefficients are exactly zero. This means some fea tures are entirely ignored by the model.
This can be seen as a form of automatic fea ture selection. Having some coefficients be exactly zero
often makes a model easier to interpret, and can reveal the most important features of your model.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Naive Bayes Classifiers

Naive Bayes (NB) is a probabilistic classification algorithm based on Bayes’ Theorem with a
strong (naive) assumption that all input features are independent of each other given the class label.

Example: Imagine you want to predict whether an email is spam or not spam.

You look at features like:

• whether it contains the word “free”

• whether it contains “win”
• how many links it has
• etc.

Even though these features are not truly independent (they often occur together), Naive Bayes
assumes they are — which simplifies computation a lot.

Then it uses probabilities from training data to estimate: How likely is an email to be spam,
given these words?

It belongs to the family of generative models, meaning it models the joint probability of
inputs and outputs, P(X,Y)P(X, Y)P(X,Y).

Despite its simplicity, Naive Bayes is fast, efficient, and effective for many real-world
problems — especially text classification, spam detection, and document categorization.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Types of Naive Bayes Classifiers (in scikit-learn)

Type Suitable For Description

Gaussian NB Continuous data Assumes features follow a normal (Gaussian)
distribution.
Multinomial NB Count data Used for document classification (e.g., word
frequencies).
Bernoulli NB Binary data Used when features are 0/1 (e.g., presence or
absence of a word).

Example: Spam Filter

Let’s say you want to classify an email as spam or not.

Features:

• Word “free” appears → yes

• Word “win” appears → yes
• Word “offer” appears → no

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Advantages

• Simple and fast

• Works well for high-dimensional data (like text)
• Needs small training data
• Robust to irrelevant features

Limitations

• Independence assumption is rarely true

• Performs poorly if features are highly correlated
• Doesn’t estimate probabilities well if data is sparse or continuous without smoothing

Decision Trees: From Data to Decisions:

A Decision Tree is one of the most intuitive and widely used algorithms in machine learning and data
analytics. It represents decisions and their possible consequences in a tree-like structure.
Each internal node represents a decision on a feature, each branch represents an outcome, and each
leaf node represents a final decision or class label.

Decision Trees are used for both:

• Classification (predicting categorical outcomes)

• Regression (predicting continuous values)

Decision Trees mimic human decision-making. For example, if we are predicting whether a
customer will buy a product:

1. We might first check their income level.

2. If high, we check age.
3. If young and high income → likely to buy.
This series of “if–then” decisions can be visualized as a tree structure.

Structure of a Decision Tree

• Root Node → Represents the entire dataset and the first feature chosen to split.
• Branches → Represent outcomes of a decision or test.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

• Internal Nodes → Represent further decisions based on features.

• Leaf Nodes → Represent final outcomes or predictions.

Key Concepts

Building Decision Trees

A Decision Tree is built through a process of recursive partitioning — where the dataset is
repeatedly split into subsets based on feature values that provide the most informative divisions.

Step-by-Step Process

1. Start with the entire dataset

The root node represents all the training samples.
2. Select the best attribute to split the data
The algorithm chooses the feature that provides the highest information gain or lowest
impurity.
o For classification: use Gini index or Entropy (Information Gain)
o For regression: use Variance Reduction or Mean Squared Error (MSE)
3. Create branches for each possible value (or range) of that attribute.
Each branch represents a subset of the data.
4. Repeat recursively
The process continues for each subset until a stopping criterion is reached.
5. Stopping conditions
o All data points in a node belong to the same class.
o No remaining attributes to split.
o Maximum tree depth reached.
o Number of samples in a node below a threshold (min_samples_split).

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

a. Splitting Criteria

The algorithm decides how to split data at each node using measures of impurity or
information gain.

b. Stopping Criteria

A Decision Tree keeps splitting until:

• All records in a node belong to the same class.

• No attribute remains for further splitting.
• The maximum tree depth is reached.

c. Controlling the Complexity of Decision Trees

While Decision Trees can fit data perfectly, they are prone to overfitting — especially when
the tree grows too deep and captures noise instead of general patterns.

Techniques to Control Complexity

A. Pre-pruning (Early Stopping)

Stop tree growth early by setting constraints:

• max_depth: limits the maximum depth of the tree.

• min_samples_split: minimum number of samples required to split a node.
• min_samples_leaf: minimum samples per leaf node.
• max_leaf_nodes: limits total number of leaf nodes.
• max_features: limits the number of features to consider at each split.

These parameters ensure the tree remains general and interpretable.

B. Post-pruning

Build a full tree first, then prune back unnecessary branches using validation data.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Advantages

Easy to understand and visualize

Requires little data preprocessing (no scaling or normalization)
Can handle both numerical and categorical data
Useful for feature selection — identifies important attributes
Non-parametric (no assumption about data distribution)

Limitations

Overfitting: Trees can become too complex and capture noise

Unstable: Small data changes can drastically alter the tree
Biased towards dominant classes or features
Less accurate alone — hence used in ensembles like Random Forests or Gradient Boosted
Trees

Real-World Applications

• Finance: Credit scoring and loan approval

• Marketing: Customer segmentation and product recommendations
• Healthcare: Disease diagnosis and risk prediction
• Human Resources: Employee attrition prediction
• Operations: Decision support and process optimization

Ensembles of Decision Trees

Ensembles are methods that combine multiple machine learning models to create more powerful
models.

A major limitation of Decision Trees is that they tend to overfit the training data. Overfitting
means the model learns the noise and details of the training set so well that it performs poorly
on unseen data.

To overcome this problem, an advanced ensemble technique called the Random Forest is
used. It combines the predictions of many decision trees to create a more robust, accurate,
and generalizable model.

Random Forest

A Random Forest is an ensemble (collection) of multiple decision trees.

Each tree in the forest is trained slightly differently from the others, and the final prediction is
made by aggregating (averaging or voting) their individual results.

• For classification tasks → The final class is decided by majority voting.

• For regression tasks → The final output is the average of all tree predictions.

The underlying idea:

“Many weak learners (decision trees) combined together form a strong learner.”

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

However, when many trees are trained on different subsets of data and features, each one
makes different errors.
By combining their predictions:

• Random errors and overfitting tend to cancel out.

• The final model generalizes better on unseen data.

This process retains the predictive power of individual trees while reducing overfitting.

The Randomness in Random Forests

Randomness is introduced in two main ways to ensure that each tree is different and
uncorrelated with others.

A. Random Sampling of Data Points (Bagging)

Each decision tree is trained on a bootstrap sample — that is, a random subset of the training
data selected with replacement.

• On average, each tree uses about 63% of the training samples.

• The remaining 37% (called Out-of-Bag samples) can be used to validate the model’s
performance internally.

This technique is known as Bootstrap Aggregation or Bagging.

Effect: Reduces variance and prevents any single tree from dominating the model.

B. Random Selection of Features

When a tree splits a node, it doesn’t consider all features — only a random subset of features
is used at each split.

• This ensures that no single strong feature is used in every tree.

• Different trees explore different patterns in the data.

Effect: Increases diversity among trees → reduces correlation → improves generalization.

The Random Forest Algorithm (Step-by-Step)

1. Input: Training dataset with N samples and M features.

2. For each tree (t = 1 to T):
a. Randomly select N samples with replacement (bagging).
b. At each split, randomly choose m features (m < M) and select the best among them
to split.
c. Grow the tree fully (without pruning).
3. Combine predictions from all trees:
o Classification → Majority vote
o Regression → Average of outputs

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Advantages of Random Forests

• Reduced Overfitting — Averaging multiple trees minimizes variance.

• High Accuracy — Performs well on both classification and regression tasks.
• Robustness — Works well even with missing data or noisy features.
• Feature Importance — Can measure which features contribute most to the
prediction.
• Internal Validation — Out-of-Bag samples can estimate model performance without
a separate validation set.

Limitations

• Less Interpretability: Harder to visualize and explain compared to a single decision

tree.
• Computational Cost: Training many trees takes more time and memory.
• Bias in Imbalanced Data: Like other models, it can be biased toward majority
classes unless handled carefully.

Out-of-Bag (OOB) Error Estimation

Since each tree is trained on a bootstrap sample, about one-third of the data is left out during
training.
These Out-of-Bag samples can be used to test the model’s performance — providing an
unbiased internal validation.

• No need for a separate validation dataset.

• OOB error is a good estimate of model generalization error.

Gradient boosted regression trees (gradient boosting machines)

The gradient boosted regression tree is another ensemble method that combines mul tiple
decision trees to create a more powerful model. Despite the “regression” in the name, these
models can be used for regression and classification. In contrast to the random forest
approach, gradient boosting works by building trees in a serial man ner, where each tree tries
to correct the mistakes of the previous one. By default, there is no randomization in gradient
boosted regression trees; instead, strong pre-pruning is used. Gradient boosted trees often use
very shallow trees, of depth one to five, which makes the model smaller in terms of memory
and makes predictions faster.

Key Working Principle

1. Start with an initial model (like a single tree).

2. Compute residuals (errors) between predictions and actual values.
3. Train the next tree to predict these residuals.
4. Combine the new tree’s output with the previous model (using a learning rate).
5. Repeat until reaching the desired number of trees (n_estimators).

Each tree is a weak learner — usually a shallow tree (depth 1–5) — that focuses on
improving overall performance step-by-step.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Main Parameters
1. n_estimators – Number of trees in the ensemble.
o More trees → higher complexity → possible overfitting.
2. learning_rate – How much each tree contributes to the final prediction.
o Lower learning rate → slower learning but better generalization (requires more trees).
3. max_depth / max_leaf_nodes – Controls tree size.
o Usually small (1–5) to keep trees weak and prevent overfitting.
4. subsample (optional) – Fraction of data used per tree for stochastic gradient boosting.

Strengths

• High predictive accuracy (often best-performing model in competitions).

• Works well with mixed feature types (binary + continuous).
• No need for feature scaling.
• Small memory footprint and fast inference due to shallow trees.

Weaknesses

• Sensitive to parameter tuning (especially learning_rate and n_estimators).

• Training time can be long.
• Not ideal for high-dimensional sparse data.
• Risk of overfitting if trees are too deep or learning rate too high.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Kernelized Support Vector Machines

Kernelized SVMs extend linear SVMs to handle nonlinear decision boundaries. Unlike
linear models that separate data using straight lines or hyperplanes, kernelized SVMs can
create complex, curved boundaries. The SVC (Support Vector Classifier) in scikit-learn is used
for classification. The concept is also applied to regression tasks as SVR (Support Vector
Regression).

Linear SVM Limitation: Linear models can only separate data points using a straight line (in 2D) or a
hyperplane (in higher dimensions).If classes are not linearly separable, linear SVMs perform poorly.

Feature Expansion Idea: To overcome this, we can add nonlinear features like squares or
interactions of existing features.

Example:
Instead of using only (x₁, x₂), we can include (x₁², x₂², x₁x₂, …) — mapping data to a higher-
dimensional feature space where a linear separator might exist.

The Kernel Trick

Computing all possible nonlinear features explicitly can be computationally expensive. The
kernel trick allows SVMs to operate in a high-dimensional space without ever explicitly
computing the transformed features. It works by directly computing the dot product
(similarity) between data points in that higher-dimensional space.

Support Vectors

During training, only a subset of data points defines the decision boundary — these are called
support vectors. They lie close to the edge between classes and determine the shape of the
separating surface. Their importance (weight) is stored in the attribute dual_coef_.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Neural Networks (Deep Learning)

A neural network is a method in artificial intelligence (AI) that teaches computers to process
data in a way that is inspired by the human brain. It is a type of machine learning (ML) process,
called deep learning, that uses interconnected nodes or neurons in a layered structure that
resembles the human brain. It creates an adaptive system that computers use to learn from their
mistakes and improve continuously. Thus, artificial neural networks attempt to solve
complicated problems, like summarizing documents or recognizing faces, with greater Simple
neural network architecture.

A basic neural network has interconnected artificial neurons in three layers:

Input Layer

Information from the outside world enters the artificial neural network from the input layer.
Input nodes process the data, analyze or categorize it, and pass it on to the next layer.

Hidden Layer

Hidden layers take their input from the input layer or other hidden layers. Artificial neural
networks can have a large number of hidden layers. Each hidden layer analyzes the output from
the previous layer, processes it further, and passes it on to the next layer.

Output Layer

The output layer gives the final result of all the data processing by the artificial neural network.
It can have single or multiple nodes. For instance, if we have a binary (yes/no) classification
problem, the output layer will have one output node, which will give the result as 1 or 0.
However, if we have a multi-class classification problem, the output layer might consist of
more than one output node.

Deep neural network architecture

Deep neural networks, or deep learning networks, have several hidden layers with millions of
artificial neurons linked together. A number, called weight, represents the connections between
one node and another. The weight is a positive number if one node excites another, or negative
if one node suppresses the other. Nodes with higher weight values have more influence on the
other-nodes.
Theoretically, deep neural networks can map any input type to any output type. However, they
also need much more training as compared to other machine learning methods. They need
millions of examples of training data rather than perhaps the hundreds or thousands that a
simpler network might need.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Artificial neural networks can be categorized by how the data flows from the input node to the
output node. Below are some examples:

Feedforward neural networks

Feedforward neural networks process data in one direction, from the input node to the output
node. Every node in one layer is connected to every node in the next layer. A feedforward
network uses a feedback process to improve predictions over time.

Backpropagation algorithm

Artificial neural networks learn continuously by using corrective feedback loops to improve
their predictive analytics. In simple terms, you can think of the data flowing from the input
node to the output node through many different paths in the neural network. Only one path is
the correct one that maps the input node to the correct output node. To find this path, the neural
network uses a feedback loop, which works as follows:

1. Each node makes a guess about the next node in the path.
2. It checks if the guess was correct. Nodes assign higher weight values to paths that lead
to more correct guesses and lower weight values to node paths that lead to incorrect
guesses.
3. For the next data point, the nodes make a new prediction using the higher weight paths
and then repeat Step 1.

Convolutional neural networks

The hidden layers in convolutional neural networks perform specific mathematical functions,
like summarizing or filtering, called convolutions. They are very useful for image classification
because they can extract relevant features from images that are useful for image recognition
and classification. The new form is easier to process without losing features that are critical for
making a good prediction. Each hidden layer extracts and processes different image features,
like edges, color, and depth.

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

Uncertainty Estimates from Classifiers

🔹 Concept

• Classifiers not only predict which class a sample belongs to, but can also indicate how
certain they are about that prediction.
• Useful in sensitive applications (e.g., medical diagnosis, fraud detection) where false
positives and false negatives have different consequences.
• In scikit-learn, two main methods give uncertainty information:
1. decision_function()
2. predict_proba()

1. Decision Function

Definition:
Returns a score that represents how strongly the model believes a sample belongs to the
positive class.

Key points:

• For binary classification, shape = (n_samples,)

• Positive score → class 1 (positive class)
• Negative score → class 0 (negative class)
• The sign of the value determines the predicted class.
• The range is arbitrary (can be large positive or negative numbers).

Interpretation Example:

Scores: [4.1, -1.6, -3.9, 3.6]

→ Positive for 4.1 and 3.6, Negative for others.

Thresholding:

(gbrt.decision_function(X_test) > 0)

→ Converts scores into True/False predictions.

Limitations:

• Scores are not probabilities.

• Hard to interpret directly.
• Overfitted models can show high scores even when wrong.

[Link] Probabilities (predict_proba)

Definition:
Returns a probability estimate for each class.
For binary classification → shape = (n_samples, 2)

Example:

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

[[0.016, 0.984],
[0.843, 0.157],
[0.981, 0.019]]

• Each row sums to 1.

• Model predicts the class with the highest probability.

Interpretation:

• Probabilities near 0.5 = model uncertain.

• Probabilities near 0 or 1 = model confident.
• A calibrated model means if it predicts 70%, it’s correct ~70% of the time.

Advantages:

• Easier to understand.
• Good for applications needing risk estimation.

Comparison Table

Aspect decision_function() predict_proba()

Output Continuous scores Probabilities

Range -∞ to +∞ 0 to 1

Interpretation Confidence toward a class Likelihood of each class

Shape (Binary) (n_samples,) (n_samples, 2)

Intuitive? Harder Easier

Use Case Ranking, Margin Analysis Probability-based decisions

Uncertainty in Multiclass Classification

• Both methods work with multiple classes.

• Output shape: (n_samples, n_classes)
• Each column → confidence or probability for one class.
• Prediction = argmax (index of highest score or probability).
• Probabilities always sum to 1 for each row.

Example (3 classes):

[[0.107, 0.784, 0.109],

[0.789, 0.106, 0.105],
[0.102, 0.108, 0.789]]

→ Predicted classes: [1, 0, 2]

Summary

• decision_function() → raw confidence scores (unbounded).

Dr Pranjala Tiwari, Associate Professor SJCIT

ADVANCED MACHINE LEARNING AND DEEP LEARNING

• predict_proba() → normalized probabilities (0–1).

• Calibration → how well predicted probabilities reflect reality.
• Use both for analyzing model confidence and uncertainty.
• For interpretability, prefer predict_proba.
• For model comparison or ranking, decision_function can be useful.

Dr Pranjala Tiwari, Associate Professor SJCIT