EL

PT
N
Module 10 : Machine Learning Lecture 23B: Regression
• Introduction to linear regression

• Assumptions in linear regression

EL
• Introduction to multi-linear regression

PT
• Introduction to ridge regression

N
• Introduction to lasso regression

2
 Simple linear regression
How can we predict the value of a continuous variable in a supervised setting?

Simple linear regression is a statistical method that allows you to summarize and study the relationship
between two continuous (quantitative) variables.

Linear regression is a foundational statistical method used for modeling the relationship between one or
more independent variables (predictors) and a dependent variable (response).

EL
It is called "simple" because it deals with only one independent variable.

PT
The relationship is represented by a linear equation, typically in the form of y=mx+b,
where:

N
•y is the dependent variable (the one you are trying to predict),

•x is the independent variable (the one you are using to make predictions),

•m is the slope of the line (representing the relationship between x and y),

•b is the y-intercept (the value of y when x is 0).

3
 Simple linear regression

Types of Linear Regression:

Simple Linear Regression: When there's only one independent variable.

Multiple Linear Regression: When there are multiple independent variables.

EL
PT
N
4
 Simple linear regression

Assumptions of Linear Regression:

Linearity: The relationship between the independent and dependent variables is linear.

Independence: The observations are independent of each other.

EL
Homoscedasticity: The variance of the residual is constant across all levels of the independent
variables.

PT
Normality: The residuals are normally distributed.

N
No multicollinearity: The independent variables are not too highly correlated with each other.

5
 Process of Linear Regression:
Data Collection:
Gather data on the variables of interest.
Data Preprocessing: Clean the data, handle missing values, and encode categorical
variables if necessary.

Model Building:

EL
Select the appropriate variables for the model.
Fit the linear regression model to the data.

PT
Model Evaluation:

N
Check for the assumptions of linear regression.
Evaluate the goodness-of-fit using metrics like MAE, MSE, RMSE, etc.
Assess the significance of the coefficients.

Prediction and Inference:

Use the fitted model to make predictions on new data.
Infer the relationship between the independent and dependent variables.

6
The mathematics behind linear regression involves fitting a linear model to a dataset to find the relationship
between the independent variable(s) (features) and the dependent variable (target).

In simple linear regression, there's only one independent variable, while in multiple linear regression, there are
multiple independent variables.

Let's focus on simple linear regression for simplicity. Suppose we have a dataset with n data points, denoted as

EL
(xi​,yi​), where xi represents the independent variable and yi represents the corresponding dependent variable. The
goal is to find a line that best fits the data, represented by the equation:

PT
N
•y is the dependent variable (target).
•x is the independent variable (feature).
•m is the slope of the line (the coefficient that represents the relationship between x and y).
•b is the y-intercept (the point where the line intersects the y-axis).

7
The objective of linear regression is to find the values of m and b that minimize the difference between the predicted
values (mx+b) and the actual values (y).

This is typically done by minimizing the mean squared error (MSE) between the predicted and actual values.

The mean squared error (MSE) is calculated as:

EL
PT
N
To minimize the MSE, we can use a method called least squares estimation. This involves
finding the values of m and b that minimize the sum of the squared differences between the
observed and predicted values of y. The formulas for calculating the least squares estimates of
m and b are as follows:

8
To minimize the MSE, we can use a method called least squares estimation. This involves finding the values of m
and b that minimize the sum of the squared differences between the observed and predicted values of y. The
formulas for calculating the least squares estimates of m and b are as follows:

EL
Where:

PT
•ഥ
𝒙 is the mean of the independent variable (x).
•ഥ
𝒚 ​ is the mean of the dependent variable (y).

N
These formulas give us the values of m and b that minimize the MSE, thus providing the best-
fitting line through the data.

Once m and b are determined, the linear regression model can be used to predict the
dependent variable (y) for new values of the independent variable (x) using the equation
y = mx+b.

9
X = [1, 2, 3, 4, 5] # Independent variable

y = [2, 3, 4, 5, 6] # Dependent variable

EL
PT
N
10
 Evaluation Metrics
Mean Squared Error (MSE): Measures the average of the squares of the errors (residuals).

Root Mean Squared Error (RMSE): The square root of the MSE.

EL
Mean Absolute Error (MAE): The average of the absolute errors.

PT
N
11
 Evaluation Metrics

Mean Absolute Error (MAE):

MAE measures the average of the absolute errors or residuals. Mathematically, it is defined as:

EL
PT
Where:

N
•n is the number of data points.
•yi is the actual value of the target variable for the ith data point.
•ෝ
𝒚i is the predicted value of the target variable for the ith data point.

MAE gives equal weight to all errors regardless of their magnitude. It provides a
more straightforward interpretation since it's in the same units as the target
variable.

12
 Evaluation Metrics

Mean Squared Error (MSE):

MSE measures the average of the squares of the errors or residuals. Mathematically, it is defined as:

EL
PT
Where:

N
•n is the number of data points.
•yi is the actual value of the target variable for the ith data point.
•ෝ
𝒚i is the predicted value of the target variable for the ith data point.

MSE penalizes larger errors more heavily than smaller ones due to the squaring
operation. The lower the MSE, the better the model performance.

13
 Evaluation Metrics

Root Mean Squared Error (RMSE):

RMSE is the square root of the MSE and is particularly useful because it returns the error metric to the same
scale as the target variable. Mathematically, it is defined as:

EL
PT
Where:

N
•n is the number of data points.
•yi is the actual value of the target variable for the ith data point.
•ෝ
𝒚i is the predicted value of the target variable for the ith data point.

RMSE has the advantage of being interpretable in the same units as the target
variable. It's commonly used when the goal is to understand the magnitude of
the errors in the same scale as the target variable.

14
 Evaluation Metrics

In summary:

MSE is the average of squared errors, penalizing larger errors more heavily.

EL
MAE is the average of absolute errors, providing a more interpretable measure.

PT
RMSE is the square root of MSE, providing an interpretable measure in the same scale as the
target variable.

N
These metrics help assess the performance of regression models and compare different models to
choose the best one for a given problem.

15
 Extensions of Linear Regression:
Polynomial Regression: Extends linear regression to accommodate polynomial relationships.

Ridge Regression and Lasso Regression: Techniques for regularization to prevent overfitting.

EL
Logistic Regression: Used for binary classification problems.

Generalized Linear Models (GLMs): Extension of linear regression to accommodate non-normally

PT
distributed dependent variables.

N
Linear regression is widely used in various fields such as economics, finance, biology, social
sciences, and engineering for modeling and prediction tasks. Its simplicity, interpretability, and
effectiveness make it one of the most commonly used statistical techniques.

16
 Multiple linear regression
Multiple linear regression is an extension of simple linear regression that involves more than one
independent variable. In simple linear regression, there is only one predictor variable, but in multiple
linear regression, there are multiple predictor variables.

The general form of a multiple linear regression equation with k independent variables is:

EL
PT
where:
•ෝ
𝒚​ ​ is the predicted value of the dependent variable,

N
•b0​ is the y-intercept,
•b1​,b2​,…,bk are the coefficients associated with the independent variables x1​,x2​,…,xk​.
•x1​,x2​,…,xk​ are the independent variables.

17
 Model Training: Multiple linear regression
To train a multivariate linear regression model:

1) Data Collection: Gather data on the dependent and independent variables.

2) Data Preprocessing: Clean the data, handle missing values, encode categorical variables
if necessary.

3) Model Fitting:

EL
• Select the appropriate independent variables for the model.
• Fit the model to the training data using methods like ordinary least squares (OLS) or

PT
gradient descent.

4) Model Evaluation:

N
• Evaluate the model's performance on a separate validation or test dataset using metrics
like R-squared, MSE, RMSE, etc.
• Assess the significance of the coefficients and interpret their impact on the dependent
variable.

5) Prediction:
Use the trained model to make predictions on new data.

18
 Assumptions: Multiple linear regression
The assumptions of multivariate linear regression are similar to those of simple linear regression:

Linearity: The relationship between the independent and dependent variables is linear.
Independence: The observations are independent of each other.

Homoscedasticity: The variance of the residuals is constant across all levels of the independent variables.

EL
Normality: The residuals are normally distributed.

PT
No multicollinearity: The independent variables are not too highly correlated with each other.

N
19
The goal of multiple linear regression is to find the values of b0​,b1​,b2​,…,bk​ that minimize the difference between the
ෝ​ values and the actual y values from the dataset.
predicted 𝒚

This is often done using the method of least squares.

The formula for the coefficients b0​,b1​,b2​,…,bk​ can be calculated using linear algebra:

EL
where:
•b is the vector of coefficients,
•X is the matrix of independent variables,

PT
•y is the vector of dependent variable values.

N
Multiple linear regression assumes that there is a linear relationship between the
dependent variable and the independent variables, and it is sensitive to the presence of
multicollinearity (high correlation) among the independent variables.

In practice, software packages like Python's scikit-learn, R, or other statistical tools are
commonly used to perform multiple linear regression analysis on datasets, making it
easier to implement and interpret.

20
 Regularization
Regularization:

• In practice, multicollinearity (high correlation between independent variables) can lead to unstable
coefficient estimates.

• Regularization techniques like Ridge regression and Lasso regression are often used to mitigate this

EL
issue by adding a penalty term to the cost function.

PT
Multivariate linear regression is widely used in fields like economics, finance, marketing,

N
and social sciences for modeling complex relationships between multiple variables and
making predictions based on data.

It serves as the foundation for more advanced regression techniques and machine learning
algorithms.

21
 Ridge regression
Ridge regression, also known as Tikhonov regularization, is a linear regression technique that
extends ordinary least squares (OLS) regression by adding a regularization term. The regularization
term helps prevent overfitting, especially in situations where there is multicollinearity (high
correlation) among the independent variables.

In ridge regression, the standard OLS objective function is modified to include a penalty term that

EL
discourages large coefficients. The objective function for ridge regression is given by:

PT
N
where:
•J (b) is the objective function to be minimized,
•b is the vector of regression coefficients,
•xi​ is the vector of predictor variables for the ith observation,
•yi​ is the observed value for the ith observation,
•p is the number of predictors,
•λ is the regularization parameter (also known as the tuning parameter or shrinkage
parameter).

22
The term is the regularization term. The regularization parameter λ controls the strength of the
regularization.

When λ=0, ridge regression reduces to ordinary least squares, and as λ increases, the impact of the
regularization term becomes more significant.

The ridge regression solution can be obtained by minimizing the objective function using techniques like
gradient descent or linear algebra methods.

EL
The regularization term tends to shrink the coefficients towards zero, which can help prevent overfitting by
penalizing large coefficients.

PT
Ridge regression is particularly useful when dealing with multicollinearity, where predictor

N
variables are highly correlated. It stabilizes the coefficient estimates and reduces their
variance, improving the model's generalization performance.

In practice, the choice of the regularization parameter λ is crucial, and it is often determined
using techniques such as cross-validation. Ridge regression is implemented in various
machine learning libraries, including scikit-learn in Python and glmnet in R.

23
 Advantages of Ridge Regression:
Mitigates Overfitting: Ridge regression reduces the variance of the model by shrinking the
coefficients, which helps prevent overfitting, especially in the presence of multicollinearity.

Improves Stability: Ridge regression provides more stable and reliable estimates of
coefficients compared to ordinary least squares, particularly when dealing with correlated
independent variables.

EL
Works Well with High-Dimensional Data: Ridge regression is effective in high-dimensional
settings where the number of features (independent variables) is much larger than the

PT
number of observations.

N
Ridge regression is a valuable tool in machine learning for improving the stability and
generalization performance of linear regression models, especially in situations where
multicollinearity is present.

By adding a penalty term to the cost function, ridge regression effectively balances the trade-
off between bias and variance, leading to more robust and interpretable models.

24
 Lasso regression
• Lasso regression, which stands for Least Absolute Shrinkage and Selection Operator, is another
regularization technique used in linear regression.

• Similar to ridge regression, lasso regression also helps mitigate the problem of multicollinearity
and prevent overfitting.

EL
• However, lasso regression introduces a different type of penalty that can lead to sparsity in the
resulting model.

PT
• As mentioned earlier, multicollinearity occurs when independent variables in a regression model
are highly correlated with each other.

N
• This can lead to instability in the estimates of coefficients and decrease the interpretability of the
model.

25
 Lasso regression
Lasso regression addresses the problem of multicollinearity by adding a penalty term to the cost
function of the linear regression model. The penalty term in lasso regression is the L1 norm of the
coefficient vector:

EL
PT
•w are the coefficients (slopes) of the linear regression model.
•yi are the observed values of the dependent variable.

N
•ෝ
𝒚i ​ are the predicted values of the dependent variable.
•p is the number of independent variables.
•α is the regularization parameter (also known as the lasso parameter or penalty parameter),
which controls the strength of the regularization.

• Larger values of α lead to greater shrinkage of coefficients towards zero.

26
 Sparsity in Lasso Regression:
Unlike ridge regression, which shrinks coefficients toward zero, lasso regression has the property of
producing sparse models.

This means that the lasso can effectively select a subset of the most important features by driving
some coefficients to exactly zero.

EL
In other words, the lasso can perform feature selection as part of the model-fitting process.

PT
N
27
 Advantages of Lasso Regression
Feature Selection: Lasso regression automatically selects a subset of the most relevant features, making the
resulting model more interpretable and efficient.

Handles High-Dimensional Data: Lasso regression works well in high-dimensional settings where the number of
features is much larger than the number of observations.

EL
Improves Model Interpretability: By reducing the number of features, lasso regression can lead to simpler and
more interpretable models.

PT
Lasso regression is a valuable tool in machine learning for feature selection and improving

N
the interpretability of linear regression models.

By introducing a penalty term based on the L1 norm of the coefficients, lasso regression
encourages sparsity in the resulting model, leading to simpler and more efficient models that
are easier to interpret.

28
• O’Reilly Hands-on Machine Learning with Scikit – Learn, Keras & TensorFlow by Aurelien Geron

• Deep learning with Python (2nd Edition) by François Chollet

EL
PT
N
29
• We discussed the linear regression

• Assumptions in linear regression

• We discussed multi-linear regression

EL
• Assumptions in linear regression

PT
• We discussed the ridge regression

N
• We discussed the lasso regression

30
 N
PT
31
EL
 EL
PT
N
Module 10 : Machine Learning Lecture 24A: Logistic Regression
• Introduction to logistic regression

• Assumptions in logistic regression

EL
• Evaluation Metric of Logistic regression

PT
• Advantages and disadvantages of logistic regression

N
• Applications of Logistic regression in Mining

2
 Logistic regression
• Logistic regression is a statistical method used for binary classification problems, where the outcome
variable is categorical and has two classes.

• Despite its name, logistic regression is a classification algorithm rather than a regression algorithm.

• It's commonly used when the dependent variable is binary, representing outcomes such as 0 or 1, True or
False, Yes or No.

EL
• The logistic regression model is based on the logistic function (also called the sigmoid function), which has

PT
an S-shaped curve.

N
The logistic function is defined as:

3
 Logistic regression Workflow
Logistic regression works in the following steps:

Prepare the data:

The data should be in a format where each row represents a single observation, and each column
represents a different variable. The target variable (the variable you want to predict) should be
binary (yes/no, true/false, 0/1).

Train the model:

EL
We teach the model by showing it the training data. This involves finding the values of the model
parameters that minimize the error in the training data.

PT
Evaluate the model:

N
The model is evaluated on the held-out test data to assess its performance on unseen data.

Use the model to make predictions:

After the model has been trained and assessed, it can be used to forecast outcomes on new data.

4
 Logistic regression
In logistic regression, the logistic function is used to model the probability that a given input belongs to
a particular class.

The logistic regression model is represented as:

EL
PT
where:
•P(Y=1) is the probability of the dependent variable (Y) being in class 1,

N
•e is the base of the natural logarithm,
•β0​ is the intercept,
•β1​,β2​,…,βn​ are the coefficients associated with the independent variables x1​,x2​,…,xn​.

5
 • The logistic regression model predicts the log odds (logit) of the probability of the event Y=1.

• The log-odds are then transformed into probabilities using the logistic function.

• The coefficients β0​,β1​,…,βn​ are estimated using methods such as maximum likelihood estimation.

• The goal is to find the values of these coefficients that maximize the likelihood of the observed data.

EL
• Logistic regression is widely used in various fields, including medicine, finance, and social
sciences, for tasks such as predicting whether an email is spam or not, predicting whether a

PT
customer will make a purchase, or assessing the likelihood of a patient having a certain
medical condition.

N
• In practice, logistic regression is implemented in many programming languages and machine
learning libraries, including Python's scikit-learn, R, and others.

6
linear regression vs. logistic regression

EL
PT
N
7
 Simple linear regression
Feature: X = 1, 2, 3, 4, 5, 6
Class : Y = 0, 0, 0, 1, 1, 1

Hence, we have two classes : 0 and 1

After applying logistic regression:

EL
PT
N
8
 Logistic regression Curve

Class B

EL
PT
N
Class A

9
 Assumptions of logistic regression
Logistic regression makes several assumptions about the data and the model that are
important to consider when applying it to machine learning tasks:

Binary Outcome:

• Logistic regression is designed for binary classification tasks where the dependent variable
(or target variable) has only two possible outcomes or classes.

EL
• If there are more than two classes, alternative approaches such as multinomial logistic
regression or one-vs-rest classification may be more appropriate.

PT
Linearity of Log Odds:

N
• Logistic regression assumes that the relationship between the independent variables
(features) and the log odds of the dependent variable is linear.

• This assumption implies that the log odds of the outcome variable are a linear combination
of the predictor variables.

10
 Assumptions of logistic regression
Logistic regression makes several assumptions about the data and the model that are
important to consider when applying it in machine learning tasks:

Independence of Observations:

• Logistic regression assumes that the observations (data points) are independent of each
other.

EL
• In other words, the occurrence of one observation does not affect the occurrence of
another observation. This assumption is crucial for the validity of statistical inference and

PT
parameter estimation.

No Multicollinearity:

N
• Logistic regression assumes that there is little or no multicollinearity among the
independent variables.

• Multicollinearity occurs when two or more independent variables are highly correlated with
each other. High multicollinearity can lead to unstable parameter estimates and inflated
standard errors.

11
 Assumptions of logistic regression
Logistic regression makes several assumptions about the data and the model that are
important to consider when applying it in machine learning tasks:

Large Sample Size:

While logistic regression does not have strict sample size requirements like some other
statistical techniques, having a relatively large sample size is beneficial for obtaining
reliable parameter estimates and accurate predictions.

EL
Small sample sizes can lead to overfitting or unstable estimates, especially when the

PT
number of predictor variables is large.

Linearity in Logit:

N
Logistic regression assumes that the relationship between the independent variables
and the log odds of the dependent variable is linear.

This implies that the effect of changing one predictor variable is constant across all
levels of other predictor variables.

12
 Assumptions of logistic regression
Logistic regression makes several assumptions about the data and the model that are
important to consider when applying it in machine learning tasks:

No Outliers:

Logistic regression can be sensitive to outliers, especially if they have a large influence
on the model parameters.

EL
It's important to check for outliers and consider strategies such as robust regression
techniques or data transformation to mitigate their impact.

PT
Absence of Perfect Separation:

N
Perfect separation occurs when the values of one or more independent variables
perfectly predict the outcome variable, resulting in infinite parameter estimates.

Logistic regression may fail to converge or produce unreliable estimates in cases of

perfect separation. Techniques such as Firth's penalized likelihood or separation
diagnostics can be used to address this issue.

13
 Assumptions of logistic regression
While these assumptions mentioned in earlier slides provide a framework for applying logistic regression
effectively, it's essential to assess their validity in the context of the specific dataset and problem domain.

Violations of these assumptions can lead to biased estimates, poor model performance, or erroneous
conclusions.

EL
PT
N
14
 Evaluation Metrics
• The accuracy of a logistic regression model is a measure of its performance in
correctly predicting the class labels of the dataset.

• It's calculated as the ratio of the number of correctly predicted instances to the total
number of instances in the dataset. Mathematically, accuracy is defined as:

EL
PT
•

N
It's important to note that accuracy alone might not be sufficient for evaluating the
performance of a model, especially in cases of imbalanced datasets or when the costs
of false positives and false negatives are significantly different.

• In such cases, additional metrics like precision, recall, and F1 score may provide a
more comprehensive evaluation of the model's performance.

15
 Evaluation Metrics
• Precision is a metric used to evaluate the performance of a classification model,
particularly in binary classification tasks.

• It measures the proportion of correctly predicted positive instances (true positives) out of
all instances predicted as positive, including both true positives and false positives.
Mathematically, precision is defined as:

EL
PT
• Precision focuses on the accuracy of positive predictions made by the model. It is

N
especially useful when the cost of false positives (misclassifying a negative instance as
positive) is high, and we want to minimize the number of false positives.

• Precision values range from 0 to 1, where a higher value indicates better precision. It's
important to interpret precision along with other metrics such as recall and accuracy for a
comprehensive evaluation of the model's performance.

16
 Evaluation Metrics
• Recall, also known as sensitivity or true positive rate, is a metric used to evaluate
the performance of a classification model, particularly in binary classification tasks.

• It measures the proportion of correctly predicted positive instances (true positives)

out of all actual positive instances.

• Mathematically, recall is defined as:

EL
PT
•

N
Recall is useful when the cost of false negatives (misclassifying a positive instance
as negative) is high, and we want to minimize the number of false negatives.

• Recall values range from 0 to 1, where a higher value indicates better recall.

• It's important to interpret recall along with other metrics such as precision and
accuracy for a comprehensive evaluation of the model's performance.

17
 Evaluation Metrics
• The F1 score is a metric that combines both precision and recall into a single value,
providing a balanced evaluation of a classification model's performance, particularly in
binary classification tasks.

• It is the harmonic mean of precision and recall, calculated as:

• Mathematically, recall is defined as:

EL
PT
• The F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0. It is

N
a useful metric when you want to find an optimal balance between precision and recall,
especially when the classes are imbalanced or when the cost of false positives and false
negatives is similar.

• The F1 score is a useful metric for evaluating classification models, especially when
there is an imbalance between the classes or when both precision and recall are
important considerations.

18
 Advantages of Logistic regression
Logistic regression is a commonly used statistical technique for binary classification
problems in machine learning. Like any other algorithm, logistic regression comes with its
own set of advantages and disadvantages:

Advantages:

Simple and Interpretable: Logistic regression is a linear model that is relatively easy to

EL
understand and interpret. The coefficients obtained from logistic regression can be directly
interpreted in terms of feature importance.

PT
Efficient Training: Logistic regression can be trained efficiently even on large datasets. It's
computationally less intensive compared to more complex models like neural networks,

N
making it suitable for situations where computational resources are limited.

Probabilistic Predictions: Logistic regression outputs probabilities that an instance belongs

to a particular class rather than just binary predictions. This can be particularly useful when
you need to assess the confidence of your model's predictions.

19
 Advantages of Logistic regression
Logistic regression is a commonly used statistical technique for binary classification
problems in machine learning. Like any other algorithm, logistic regression comes with its
own set of advantages and disadvantages:

Advantages:

Less Prone to Overfitting: With proper regularization techniques like L1 or L2

EL
regularization, logistic regression can handle overfitting reasonably well, especially in
high-dimensional spaces.

PT
Feature Importance: Logistic regression provides a clear indication of the relative
importance of each feature in predicting the outcome. This can be valuable for feature

N
selection and understanding the underlying relationships between features and the target
variable.

20
 Disadvantages of Logistic regression
Logistic regression is a commonly used statistical technique for binary classification
problems in machine learning. Like any other algorithm, logistic regression comes with its
own set of advantages and disadvantages:

Disadvantages:

Limited Expressiveness: Logistic regression assumes a linear relationship between the

features and the log-odds of the response. This means it may not capture more complex

EL
relationships in the data. If the relationship is highly non-linear, logistic regression might
underperform compared to more flexible models like decision trees or neural networks.

PT
Assumption of Linearity: Logistic regression assumes that the relationship between the
independent variables and the logit transformation of the dependent variable is linear. If

N
this assumption is violated, the model's predictions may be inaccurate.

Binary Classification Only: Logistic regression is inherently designed for binary

classification problems. While there are extensions like multinomial logistic regression for
multi-class classification, logistic regression may not be suitable for regression tasks or
problems with more than two classes without modifications.

21
 Disadvantages of Logistic regression
Logistic regression is a commonly used statistical technique for binary classification
problems in machine learning. Like any other algorithm, logistic regression comes with its
own set of advantages and disadvantages:

Disadvantages:

Sensitive to Outliers: Logistic regression can be sensitive to outliers, especially if the

EL
outliers are present in the independent variables. Outliers can disproportionately influence
the parameter estimation process, leading to biased results.

PT
Feature Engineering Dependency: The performance of logistic regression heavily relies on
feature engineering. If informative features are not selected or if irrelevant features are

N
included, the model's predictive performance may suffer.

Overall, logistic regression is a valuable tool in a machine learning practitioner's toolbox,

particularly for binary classification tasks where interpretability and efficiency are
essential. However, it's essential to understand its limitations and select appropriate
models based on the specific characteristics of the dataset and problem at hand.

22
 Applications of Logistic Regression in Mining (1/5)
Logistic regression, despite its simplicity, finds various applications in the mining industry
due to its ability to handle binary classification problems efficiently. Here are five detailed
applications:

Exploration Targeting:

Problem: Mining companies often need to identify promising exploration targets to allocate
resources effectively.

EL
Application of Logistic Regression: Logistic regression can be used to analyze geological

PT
and geophysical data to predict the likelihood of mineralization in a particular area. For
example, it can utilize features such as rock type, geochemical anomalies, and geophysical

N
signatures to classify areas as prospective or non-prospective for specific minerals.

Detail: Historical exploration data, including drilling results and known mineral
occurrences, can serve as labeled data for training the logistic regression model. The
model then predicts the probability of mineralization in unexplored regions, guiding
exploration efforts towards high-potential areas.

23
 Applications of Logistic Regression in Mining (2/5)
Logistic regression, despite its simplicity, finds various applications in the mining industry
due to its ability to handle binary classification problems efficiently. Here are five detailed
applications:

Risk Assessment for Mine Safety:

Problem: Ensuring the safety of miners is a critical concern in the mining industry, with
various risks associated with different mining activities.

EL
Application of Logistic Regression: Logistic regression can be employed to assess the risk

PT
of accidents or incidents occurring during mining operations. Relevant features might
include factors such as geological conditions, equipment status, weather conditions, and

N
human factors.

Detail: By analyzing historical incident data, logistic regression models can identify
patterns and risk factors associated with accidents. The model can then predict the
likelihood of incidents occurring under specific conditions, allowing for proactive
measures to mitigate risks and improve safety protocols.

24
 Applications of Logistic Regression in Mining (3/5)
Logistic regression, despite its simplicity, finds various applications in the mining industry
due to its ability to handle binary classification problems efficiently. Here are five detailed
applications:

Mineral Deposit Classification:

Problem: Different types of mineral deposits require different mining techniques and
processing methods. Classifying mineral deposits accurately is crucial for efficient
resource extraction and processing.

EL
Application of Logistic Regression: Logistic regression can classify mineral deposits

PT
based on geological and geochemical characteristics. Features might include mineral
assemblages, alteration patterns, and structural attributes.

N
Detail: By training logistic regression models on labeled data representing different mineral
deposit types, such as porphyry copper, epithermal gold, or massive sulfide deposits, the
model can learn to distinguish between them. This classification aids in targeting
exploration efforts and planning mining operations accordingly.

25
 Applications of Logistic Regression in Mining (4/5)
Logistic regression, despite its simplicity, finds various applications in the mining industry
due to its ability to handle binary classification problems efficiently. Here are five detailed
applications:

Mineral Prospectivity Mapping:

Problem: Mining companies need to prioritize areas for exploration based on their potential
for hosting economically viable mineral deposits.

EL
Application of Logistic Regression: Logistic regression can be used to generate mineral
prospectivity maps by integrating various geological, geophysical, and geochemical

PT
datasets.

N
Detail: By analyzing known occurrences of mineral deposits along with geological features
associated with mineralization, logistic regression models can predict the probability of
similar occurrences in unexplored areas. This information helps in identifying prospective
regions for further exploration and investment.

26
 Applications of Logistic Regression in Mining (5/5)
Environmental Impact Assessment:

Problem: Mining operations can have significant environmental impacts, including

habitat destruction, water pollution, and air pollution.

Application of Logistic Regression: Logistic regression can be utilized to assess the

likelihood and severity of environmental impacts associated with proposed mining

EL
projects.

PT
Detail: By considering factors such as project location, terrain characteristics, proximity
to sensitive ecosystems, and proposed mitigation measures, logistic regression models

N
can predict the probability of various environmental impacts occurring.

This information aids in decision-making processes, allowing stakeholders to evaluate

and minimize the environmental footprint of mining activities.

In each of these applications, logistic regression serves as a valuable tool for leveraging
data to make informed decisions, optimize resource allocation, and mitigate risks in the
mining industry.

27
• O’Reilly Hands-on Machine Learning with Scikit – Learn, Keras & TensorFlow by Aurelien Geron

• Deep learning with Python (2nd Edition) by François Chollet

EL
PT
N
28
• We discussed logistic regression in detail

• We discussed the assumptions involved in logistic regression

• We discussed the evaluation metrics of logistic regression

EL
• We covered the Advantages and disadvantages of logistic regression,

PT
• And along with Applications of logistic regression in Mining

N
29
 N
PT
30
EL
 EL
PT
N
Module 10: Machine Learning Lecture 24B: K Nearest Neighbor
• Introduction to K-Nearest Neighbors

• K-Nearest Neighbors distance metrics

EL
• K-Nearest Neighbors parameters

PT
• Assumptions of K-Nearest Neighbors

N
• Advantages and disadvantages of K-Nearest Neighbors

• Applications of K-Nearest Neighbors in Mining

2
 K-Nearest Neighbors (KNN)
k-Nearest Neighbors (k-NN) is a supervised machine learning algorithm used for both classification
and regression tasks.

It is a simple, yet effective, algorithm that makes predictions based on the majority class (for
classification) or the average (for regression) of the k-nearest data points in the feature space.

EL
PT
N
3
 K-Nearest Neighbors (KNN)
K-Nearest Neighbors (KNN) is a simple yet powerful supervised machine learning algorithm used for
classification and regression tasks.

It is a non-parametric and lazy learning algorithm, meaning it doesn't make any assumptions about
the underlying data distribution and it doesn't learn explicitly during the training phase.

Here are the complete details of the KNN algorithm:

EL
Overview:

PT
KNN works based on the principle of similarity or distance. It classifies a data point based on how its

N
neighbors are classified.

The "K" in KNN represents the number of nearest neighbors considered when making predictions.

4
 K-Nearest Neighbors (KNN) Distance Metrics
The mathematics behind the K-Nearest Neighbors (KNN) algorithm involves primarily the calculation
of distances between data points and the method used for making predictions. Let's delve into the
mathematical details:

Distance Calculation:
KNN uses a distance metric to measure the similarity or dissimilarity between data points. The most

EL
commonly used distance metrics are Euclidean distance, Manhattan distance, and Minkowski
distance.

PT
Euclidean distance:
Euclidean distance between two points p and q in an n-dimensional space is calculated as:

N
where pi and qi are the ith coordinates of points p and q, respectively.

5
 K-Nearest Neighbors (KNN) Distance Metrics
Manhattan distance:
Manhattan distance, also known as City Block distance or L1 distance, between two points p and q
is calculated as the sum of the absolute differences of their coordinates:

EL
Minkowski distance:
Minkowski distance is a generalization of both Euclidean and Manhattan distances. For parameter
r, the Minkowski distance between two points p and q is calculated as:

PT
N
Euclidean distance is a special case of Minkowski distance when r=2, and Manhattan distance is
the case when r=1.

6
 K-Nearest Neighbors (KNN) Parameters
K: Number of neighbors to consider. Choosing an appropriate K value is crucial and can
significantly impact the performance of the algorithm.

Distance metric: The measure used to compute the distance between data points, such as
Euclidean distance, Manhattan distance, etc.

EL
PT
N
7
 K-Nearest Neighbors (KNN)
[Link] Phase:
1. The algorithm stores the entire training dataset in memory.
[Link] Phase:
1. Given a new, unseen data point, the algorithm identifies the k-nearest neighbors to that point in the feature
space.
2. For classification, the majority class among the k-neighbors is assigned to the new data point.
3. For regression, the average (or weighted average) of the target values of the k-neighbors is assigned to the
new data point.

EL
[Link] Metric:
1. The choice of distance metric (such as Euclidean distance, Manhattan distance, etc.) is crucial in determining
the neighbors. Euclidean distance is a common choice in practice.

PT
4. Choosing k:

N
1. The value of k is a hyperparameter that needs to be specified. A small k may be
sensitive to noise, while a large k may smooth out patterns in the data. Common
choices include odd numbers to avoid ties in voting for classification problems.
5. Scalability:
1. k-NN can be computationally expensive, especially with large datasets, as it requires
calculating distances to all training examples. Efficient data structures like KD-trees
or Ball trees are often used to speed up the search for nearest neighbors. the
majority class among the k-neighbors is assigned to the new data point for
classification

8
 How does K-NN work?
The functioning of K-NN can be elucidated through the following steps:

1. Determine the value of K, representing the number of neighbors to consider.

2. Compute the distance metric (Example: Euclidean distance) for K neighbors.

EL
3. Identify the K nearest neighbors based on the calculated Euclidean distance.

PT
4. Tally the occurrences of data points in each category among these K neighbors.

5. Allocate the new data points to the category with the highest neighbor count.

N
6. The model is now prepared for use.

9
 K-Nearest Neighbors (KNN)
Imagine we have a new data point and we need to assign it to the appropriate category. Please refer to
the image below:

EL
PT
N
10
 K-Nearest Neighbors (KNN)
Initially, we'll select the number of neighbors, opting for k=5.

Following that, we'll compute the Euclidean distance between the data points. The Euclidean distance
represents the distance between two points. Upon calculating the Euclidean distance, we identify the nearest
neighbors, with three closest neighbors belonging to category A and two closest neighbors belonging to
category B. Please refer to the image below for further clarification:

EL
PT
N
Observing that the three closest neighbors belong to category A, it is evident that this
new data point should be categorized as belonging to category A.

11
 K-Nearest Neighbors (KNN)
• k-NN is a non-parametric algorithm, meaning it doesn't make strong assumptions about the
underlying data distribution.

• It can adapt to complex decision boundaries and is particularly useful when the relationship
between input features and the target variable is not easily characterized by a simple
mathematical model.

EL
• While k-NN is straightforward and easy to understand, it might not perform well in high-
dimensional spaces or with datasets where irrelevant features are present.

PT
• Additionally, the choice of the distance metric and the value of k can significantly impact the

N
algorithm's performance.

• In practice, scikit-learn in Python and other machine learning libraries provide

implementations of k-NN for both classification and regression tasks.

12
 Assumptions of K-Nearest Neighbors (KNN)
The k-Nearest Neighbors (k-NN) algorithm in machine learning makes certain
assumptions and has some characteristics:

Instance-Based Learning:
k-NN is an instance-based or lazy learning algorithm. This means it doesn't explicitly
build a model during training. Instead, it memorizes the training instances and makes
predictions based on their similarity to new instances.

EL
Local Approximation:
The assumption behind k-NN is that similar instances tend to belong to the same class.

PT
Therefore, it uses the local approximation to predict the class label of a new instance
based on the labels of its k-nearest neighbors in the feature space.

N
Feature Similarity:
k-NN assumes that the similarity between instances can be effectively measured using a
distance metric such as Euclidean distance, Manhattan distance, or cosine similarity.
Instances that are closer in the feature space are considered more similar.

13
 Assumptions of K-Nearest Neighbors (KNN)
k Parameter Selection:
The performance of k-NN depends on the choice of the parameter k, which represents the
number of nearest neighbors to consider. A smaller value of k may result in more complex
decision boundaries, potentially leading to overfitting, while a larger value of k may result in
smoother decision boundaries but might miss local patterns.

Noisy Data Handling:

k-NN is sensitive to noisy data and outliers because it considers all training instances equally

EL
when making predictions. Noisy data can significantly affect the classification results and may
require preprocessing or noise reduction techniques.

PT
Computational Efficiency:

N
While k-NN is conceptually simple, it can be computationally expensive, especially for large
datasets, because it requires computing distances between the new instance and all training
instances. Efficient data structures such as KD-trees or ball trees are often used to speed up
the search for nearest neighbors.

Understanding these assumptions and characteristics is crucial for effectively applying k-NN
and interpreting its results in various machine learning tasks.

14
 Advantages of K-Nearest Neighbors (KNN)
Simplicity:
• KNN is easy to understand and implement, making it a great starting point for beginners in
machine learning.

No Training Phase:
• KNN is a lazy learning algorithm, meaning it doesn't require a training phase.

EL
• The model directly uses the training data for prediction, making it efficient for incremental
learning scenarios where new data points can be added without retraining the model.

PT
Non-Parametric:

N
• KNN makes no assumptions about the underlying data distribution, making it suitable for a
wide range of applications, including non-linear data.

15
 Advantages of K-Nearest Neighbors (KNN)
Flexibility:

• KNN can be used for both classification and regression tasks, making it versatile.

Interpretability:

EL
• Since predictions are based on nearby points in the feature space, KNN can provide insights
into the decision-making process, allowing users to interpret the results easily.

PT
Robust to Noise:

N
• KNN can perform well even in the presence of noisy data, as outliers can have less influence
when considering multiple neighbors.

16
 Disadvantages of K-Nearest Neighbors (KNN)
Computational Complexity:

• During prediction, KNN needs to calculate the distances between the query point and all
training points, which can be computationally expensive for large datasets, especially in high-
dimensional feature spaces.

Memory Intensive:

EL
• KNN stores all training data, which can consume a significant amount of memory, particularly

PT
for large datasets.

N
Need for Optimal K:

• The choice of the parameter K (number of neighbors) significantly affects the performance of
KNN.
• Selecting an inappropriate K value may lead to poor generalization or overfitting.

17
 Disadvantages of K-Nearest Neighbors (KNN)
Sensitive to Feature Scaling:

• Since KNN relies on distance metrics, features with larger scales may dominate the
distance calculation, leading to biased results. Therefore, feature scaling is often
necessary.

Imbalanced Data:

EL
• In classification tasks with imbalanced class distributions, KNN may bias predictions

PT
towards the majority class, especially when K is small.

Curse of Dimensionality:

N
• KNN's performance can degrade rapidly as the dimensionality of the feature space
increases. This phenomenon is known as the curse of dimensionality, where the volume of
the feature space grows exponentially with the number of dimensions, causing the nearest
neighbors to become less meaningful.

18
 Disadvantages of K-Nearest Neighbors (KNN)
• Understanding these advantages and disadvantages is crucial for effectively applying KNN
to different machine-learning tasks and scenarios.

• It's essential to consider these factors and perform proper experimentation and tuning to
achieve optimal results.

EL
PT
N
19
 Applications of K-Nearest Neighbors (KNN) in Mining
K-Nearest Neighbors (KNN) algorithm can be applied in various ways within the mining industry
to solve different problems. Here are five potential applications:

Exploration Targeting:

Problem:

• Identifying promising locations for mineral exploration based on geological and geochemical

EL
data.

PT
Solution:

N
Use KNN to analyze historical exploration data and geological features of known deposits to
predict potential new mineralization areas.

• By considering similar geological contexts and proximity to existing deposits, KNN can help
prioritize exploration efforts.

20
 Applications of K-Nearest Neighbors (KNN) in Mining
Rock Classification:

Problem:

• Classifying different types of rocks or mineral deposits based on their physical and chemical
properties.

EL
Solution:

PT
Utilize KNN to classify rock samples collected from mining sites based on their spectral signatures,
mineral composition, or other characteristics.

N
• KNN can learn from labeled training data to accurately classify new rock samples into predefined
categories, aiding in geological mapping and resource estimation.

21
 Applications of K-Nearest Neighbors (KNN) in Mining
Geotechnical Stability Assessment:

Problem:

• Assessing the stability of mining structures such as slopes, tunnels, and excavations to
prevent collapses and ensure worker safety.

EL
Solution:

PT
• Employ KNN to analyze geotechnical data including rock strength, fracture density, and
ground conditions to predict areas at risk of instability.

N
• By identifying similar geotechnical conditions and historical stability records, KNN can assist
in evaluating the stability of new mining operations and implementing preventive measures.

22
 Applications of K-Nearest Neighbors (KNN) in Mining

Predictive Maintenance:

Problem:

• Predicting equipment failures and optimizing maintenance schedules to minimize downtime

and reduce operational costs.

EL
Solution:

PT
• Apply KNN to analyze sensor data from mining equipment to identify patterns indicative of
potential failures or malfunctions.

N
• By learning from historical sensor readings and maintenance records, KNN can predict
equipment failures in advance, allowing maintenance teams to perform proactive maintenance
tasks and avoid unexpected breakdowns.

23
 Applications of K-Nearest Neighbors (KNN) in Mining

Environmental Monitoring:

Problem:

• Monitoring and managing environmental impacts of mining activities, such as water

pollution and habitat destruction.

EL
Solution:

PT
• Use KNN to analyze environmental monitoring data, including water quality
measurements, air pollution levels, and habitat characteristics.

N
• KNN can identify spatial patterns and similarities between monitoring sites, helping to
detect environmental hotspots, prioritize mitigation efforts, and comply with regulatory
requirements.

24
 Applications of K-Nearest Neighbors (KNN) in Mining

• These applications demonstrate how KNN can be leveraged within the mining industry to address various
challenges related to exploration, resource assessment, operational efficiency, safety, and environmental
stewardship.

• By utilizing machine learning algorithms like KNN, mining companies can enhance decision-making processes,
optimize operations, and mitigate risks associated with mining activities.

EL
PT
N
25
• O’Reilly Hands-on Machine Learning with Scikit – Learn, Keras & TensorFlow by Aurelien Geron

• Deep learning with Python (2nd Edition) by François Chollet

EL
• [Link]

PT
N
26
• We discussed K-Nearest Neighbors in detail

• We discussed K-Nearest Neighbors distance metrics

• We discussed K-Nearest Neighbors parameters

EL
• We discussed the assumptions of K-Nearest Neighbors

PT
• We discussed the advantages and disadvantages of K-Nearest Neighbors

N
• We discussed the 5 main Applications of K-Nearest Neighbors in Mining

27
 N
PT
28
EL
 EL
PT
N
Module 10 : Machine Learning Lecture 25A: Support Vector Machine
• Introduction to Support Vector Machine (SVM)
• Key concepts of Support Vector Machine (SVM)
• Assumptions in Support Vector Machine (SVM)
• Common kernel functions used in SVM include:

EL
• Linear Kernel
• Polynomial Kernel

PT
• Radial Basis Function (RBF) Kernel

N
• Advantages and disadvantages of Support Vector Machine (SVM)
• Applications of Support Vector Machine (SVM) in Mining

2
 Support Vector Machine
Support Vector Machine (SVM) is a powerful supervised learning algorithm primarily used for classification
tasks, although it can be extended for regression as well.

It belongs to the family of discriminative classifiers and is widely used in various fields, including computer
vision, text classification, bioinformatics, and more.

EL
PT
N
3
 Support Vector Machine
Overview:

• SVM aims to find the hyperplane that best separates data points of different classes in an n-dimensional space.

• If the data is linearly separable, SVM tries to find the hyperplane that maximizes the margin, i.e., the distance
between the hyperplane and the nearest data point from each class, known as support vectors.

• If the data is not linearly separable, SVM uses a technique called kernel trick to map the data into a higher-

EL
dimensional space where it can be linearly separated.

PT
N
4
 Key concepts of Support Vector Machine
let's delve deeper into the key concepts of Support Vector Machines (SVM) in detail:

1. Hyperplane:

•In SVM, a hyperplane is a decision boundary that separates data points of different classes in an n-dimensional space.

•For a binary classification problem, the hyperplane is a (n-1) dimensional subspace in the n-dimensional feature space.

•Mathematically, a hyperplane is represented as: w⋅x+ b = 0, where w is the weight vector perpendicular to the hyperplane,

EL
x is the input vector, and b is the bias term.

PT
N
5
2. Margin:

•The margin is the distance between the hyperplane and the nearest data point from each class.

•SVM aims to find the hyperplane that maximizes this margin.

•Larger margins usually imply better generalization to unseen data.

EL
PT
N
6
3. Support Vectors:

•Support vectors are the data points that lie closest to the decision boundary (hyperplane).

•These are the critical elements in defining the decision boundary and determining the margin.

•Support vectors influence the position and orientation of the hyperplane.

•The decision boundary is solely determined by the support vectors, making SVM memory efficient.

EL
PT
N
7
4. Kernel Trick:

• SVM can handle non-linear decision boundaries by mapping the

input space into a higher-dimensional feature space using kernel
functions.

• Kernel functions compute the dot product between the input

vectors in the higher-dimensional space without explicitly

EL
transforming them.

Common kernel functions include:

PT
1. Linear Kernel
2. Polynomial Kernel

N
3. Radial Basis Function (RBF) Kernel

The choice of kernel depends on the problem's characteristics and

the data distribution.

8
5. Optimization:

• SVM aims to find the optimal hyperplane that maximizes the margin while minimizing the
classification error.

• This optimization problem is often formulated as a quadratic programming (QP) or convex

optimization problem.

EL
• Techniques like gradient descent, sequential minimal optimization (SMO), or interior-point

PT
methods are commonly used to solve this optimization problem efficiently.

N
These key concepts form the foundation of Support Vector Machines and are crucial for
understanding how SVM works and how it achieves effective classification in both linearly
separable and non-linearly separable datasets.

9
 Assumptions of SVM

Support Vector Machines (SVMs) have certain assumptions and properties that underlie their
effectiveness.

While SVMs are quite versatile and can handle a variety of situations, understanding these
assumptions can help in applying them effectively. Here are some key assumptions and
properties of SVMs:

EL
Linear Separability (for Linear SVMs):

PT
The original formulation of SVM assumes that the classes can be separated by a linear decision
boundary (hyperplane) in the input space.

N
When the classes are not linearly separable, SVM aims to find the hyperplane that maximizes the
margin while minimizing the classification error, which might lead to a soft-margin SVM
formulation or the use of non-linear kernels.

10
 Assumptions of SVM
Margin Maximization:

• SVM aims to find the hyperplane that maximizes the margin, i.e., the distance between the
hyperplane and the nearest data point from each class (support vectors).

• Maximizing the margin helps SVM generalize well to unseen data and enhances its
robustness against outliers.

EL
PT
Kernel Function Selection:

• When using non-linear SVMs (e.g., with polynomial kernel, RBF kernel), the choice of

N
kernel function and its parameters (e.g., degree of polynomial, gamma for RBF) is crucial.

• The kernel function should be chosen based on the characteristics of the data and the
problem domain.

11
 Assumptions of SVM
Noisy or Outlier-Free Data:

SVMs are sensitive to noise and outliers in the data, especially in the case of hard-
margin SVM.

Outliers can significantly affect the position and orientation of the decision boundary.

EL
Techniques such as soft-margin SVM or robust kernel functions (e.g., RBF kernel) can
help mitigate the impact of outliers.

PT
Feature Scaling:

N
SVM performance can be influenced by the scale of the input features.

It's advisable to scale the features to a similar range (e.g., using standardization or
normalization) before training an SVM model to ensure that all features contribute
equally to the decision-making process.

12
 Assumptions of SVM
Binary Classification:

• SVM is inherently a binary classifier, meaning it separates data points into two classes.

• For multi-class classification, techniques such as one-vs-one or one-vs-all strategies can

be used, where multiple binary classifiers are trained and combined to make predictions
for multiple classes.

EL
Sparse Solution:

PT
• SVM often yields sparse solutions, meaning that only a subset of training data (support
vectors) contributes to defining the decision boundary.

N
• This property makes SVM memory efficient and allows it to handle high-dimensional data
efficiently.

Understanding these assumptions and properties can guide the appropriate application and
tuning of SVM models for different tasks and datasets. It's important to validate these
assumptions and adapt the SVM formulation accordingly to achieve optimal performance.

13
 Linear Kernel
• The linear kernel is one of the simplest kernel
functions used in Support Vector Machines (SVM).

• It represents the dot product between the input

vectors in the original feature space.

Mathematically, the linear kernel function is defined as:

EL
PT
Where:
xi and xj are input vectors in the original feature space.

N
• represents the dot product operation.

•The linear kernel computes a linear decision boundary,

which means it assumes that the classes can be
separated by a straight line (or hyperplane in higher
dimensions) in the input space.

14
 Linear Kernel
The advantages of the linear kernel include:

Computational efficiency: Linear kernels are

computationally less expensive compared to non-linear
kernels like polynomial or RBF kernels.

Interpretability: Linear SVMs with linear kernels provide

straightforward decision boundaries that are easier to

EL
interpret.

PT
However, the linear kernel is limited in its ability to capture
complex relationships in the data.

N
If the classes are not linearly separable, using a linear
kernel may result in poor classification performance.

In such cases, non-linear kernels like polynomial or RBF

kernels are often preferred, as they can map the data into
higher-dimensional spaces where non-linear relationships
can be captured.

15
 Polynomial Kernel
• The polynomial kernel is a popular kernel function
used in Support Vector Machines (SVM) for handling
non-linear classification problems.

• It maps the input vectors into a higher-dimensional

feature space using polynomial functions.

EL
The polynomial kernel function is defined as:

PT
N
Where:
•xi and xj are input vectors in the original feature space.
• ⋅ represents the dot product operation.
•c is a constant term (usually denoted as the coefficient of
the linear term).
•d is the degree of the polynomial.

16
 Polynomial Kernel

The polynomial kernel allows SVM to capture non-linear

decision boundaries by transforming the input space into a
higher-dimensional space where the classes might become
linearly separable.

Key points about the polynomial kernel:

EL
Degree (d): The degree of the polynomial determines the

PT
complexity of the decision boundary. Higher degrees allow
for more complex decision boundaries but may also lead to
overfitting.

N
Coefficient (c): The coefficient term c affects the importance
of higher-degree features compared to lower-degree
features. It helps control the influence of higher-order terms
in the polynomial.

17
 Polynomial Kernel
Advantages of the polynomial kernel include:

Flexibility: It can capture complex non-linear relationships in the

data.

Control over complexity: By adjusting the degree parameter, one

can control the complexity of the decision boundary.

EL
• However, choosing the appropriate degree and coefficient values
is crucial for achieving good classification performance.

PT
• A higher degree polynomial may lead to overfitting, while a lower

N
degree may result in underfitting.

• Additionally, the computational complexity increases with higher

degrees of polynomial kernels, which can impact training time,
especially for large datasets.

• Therefore, parameter tuning and cross-validation are essential

for optimizing the performance of SVM with polynomial kernels.

18
 Radial Basis Function (RBF) Kernel
The Radial Basis Function (RBF) kernel, also known as the
Gaussian kernel, is a widely used kernel function in Support
Vector Machines (SVM) for handling non-linear classification
problems.

It maps the input vectors into a higher-dimensional feature

space using a Gaussian function.

EL
The RBF kernel function is defined as:

PT
N
Where:
•xi​ and xj​ are input vectors in the original feature space.

•γ (gamma) is a hyperparameter that controls the spread of

the Gaussian function.

•∥xi​−xj∥ represents the Euclidean distance between the input

vectors.

19
 Radial Basis Function (RBF) Kernel
The RBF kernel allows SVM to capture complex non-linear decision
boundaries by implicitly mapping the input space into an infinite-
dimensional feature space.

It is called a radial basis function because its value depends only on the
distance between the input vectors, and it decreases radially as the
distance from a reference point (usually a support vector) increases.

EL
Key points about the RBF kernel:
Gamma (γ): The gamma parameter determines the influence of each

PT
training example.

N
A smaller value of gamma results in a larger range of influence and
smoother decision boundaries, while a larger value makes the decision
boundary more irregular and closely fitted to the training data. It
essentially controls the flexibility of the decision boundary.

Implicit Feature Mapping: Unlike the polynomial kernel, the RBF kernel
implicitly maps the data into an infinite-dimensional space, making it very
flexible in capturing complex decision boundaries.

20
 Radial Basis Function (RBF) Kernel
Advantages of the RBF kernel include:

• Flexibility: It can capture highly complex non-linear relationships

in the data.

• Versatility: RBF kernels can effectively handle a wide range of

data distributions.

EL
• However, choosing the appropriate value for the gamma
parameter is crucial for achieving good classification

PT
performance.

N
A poorly chosen gamma value can lead to overfitting or
underfitting.

• Additionally, the computational complexity of SVM with RBF

kernels increases significantly with the size of the dataset, as it
involves computing pairwise distances between all data points.

• Therefore, parameter tuning and model validation are essential

for optimizing the performance of SVM with RBF kernels.

21
 Advantages and Disadvantages of SVM
Support Vector Machines (SVMs) offer several advantages and disadvantages, which should be considered when
choosing this algorithm for a particular task. Here's a comprehensive list:
Advantages of SVM:

Effective in High-Dimensional Spaces:

• SVM performs well even in high-dimensional spaces, making it suitable for tasks with
many features, such as image classification or text categorization.

EL
Memory Efficient:

PT
• SVM uses a subset of training points (support vectors) to define the decision boundary.

N
This property makes SVM memory efficient, especially for large datasets.

Versatile:

• SVM supports different kernel functions, allowing it to handle various data distributions
and non-linear relationships. Common kernels include linear, polynomial, and radial basis
function (RBF).

22
 Advantages and Disadvantages of SVM
Advantages of SVM:

Robust to Overfitting:

• SVM is less prone to overfitting, especially in high-dimensional spaces, due to its ability
to maximize the margin between classes.

Effective with Small Datasets:

EL
• SVM can produce accurate results with relatively small training datasets, making it

PT
suitable for tasks where data collection is expensive or time-consuming.

N
Works Well with Non-linear Data:

• By using appropriate kernel functions, SVM can efficiently model complex, non-linear
decision boundaries.

Global Optimum:

• SVM aims to find the global optimum solution (i.e., the hyperplane with the maximum
margin), which leads to better generalization performance.

23
 Advantages and Disadvantages of SVM
Disadvantages of SVM:

Computationally Intensive:

• SVM can be computationally expensive, especially for large datasets, as it requires

solving a quadratic programming problem.

Sensitivity to Parameter Tuning:

EL
• SVM performance is sensitive to the choice of hyperparameters, such as the

PT
regularization parameter C and kernel parameters. Selecting appropriate values for these
parameters can be challenging and may require extensive experimentation.

N
Limited Interpretability:

• SVM models can be difficult to interpret, especially when using complex kernel functions
or in high-dimensional spaces.

• Understanding the relationship between input features and the decision boundary is not
always straightforward.

24
Disadvantages of SVM:

Slow Training Time:

• Training an SVM model can take a long time, especially for large datasets or when using
non-linear kernels.

• Additionally, the complexity of training time can increase significantly with the number of

EL
support vectors.

Not Suitable for Large Datasets:

PT
• SVM may not scale well to large datasets due to its computational complexity and

N
memory requirements.

No Probabilistic Output:

• SVM does not provide direct probabilistic interpretations of class membership, unlike
some other classifiers, such as logistic regression.

25
Disadvantages of SVM:

Sensitive to Noise:

SVM performance can degrade significantly in the presence of noisy data or outliers, especially when using a hard-
margin SVM formulation.

Understanding these advantages and disadvantages can help in making informed decisions about when to use SVM and

EL
how to mitigate its limitations for a particular machine-learning task.

PT
N
26
 Applications of SVM in Mining
Support Vector Machines (SVMs) find numerous applications in the mining industry due to their ability to handle
complex data and classify it with high accuracy. Here are five main applications of SVM in mining:

Mineral Identification and Classification:

• SVMs are used for mineral identification and classification in mining exploration and mineral
processing.

EL
• By analyzing spectroscopic data obtained from various sensors such as X-ray fluorescence

PT
(XRF), near-infrared (NIR), or hyperspectral imaging, SVM models can classify different minerals
or mineral compositions present in rock samples.

N
• This helps in identifying potential mineral deposits and optimizing mineral processing
operations.

27
 Applications of SVM in Mining

Fault Detection and Predictive Maintenance:

• SVMs are employed for fault detection and predictive maintenance in mining equipment
and machinery.

EL
• By analyzing sensor data such as vibration, temperature, or pressure readings from
mining equipment, SVM models can detect abnormal patterns indicative of equipment

PT
faults or failures.

• Early detection of faults allows for proactive maintenance interventions, minimizing

N
downtime and maximizing operational efficiency.

28
 Applications of SVM in Mining
Ore Grade Estimation:

• SVMs are utilized for ore grade estimation in mining operations.

• By analyzing geological data, drill core samples, or assay results, SVM models can predict the grade and quality
of ore deposits in different mining sites.

EL
• Accurate ore grade estimation enables mining companies to optimize resource extraction, mine planning, and
production scheduling, leading to cost savings and increased profitability.

PT
N
29
 Applications of SVM in Mining
Mine Safety and Risk Assessment:

• SVMs play a crucial role in mine safety and risk assessment by analyzing various factors such as geotechnical
data, geological conditions, operational parameters, and historical safety records.

• SVM models can predict potential hazards, assess risks associated with mining activities, and recommend
safety measures to prevent accidents and ensure worker safety in mines

EL
Stockpile Management and Inventory Control:

PT
• SVMs are used for stockpile management and inventory control in mining and material
handling operations.

N
• By analyzing data from stockpile surveys, volume measurements, and material
characteristics, SVM models can accurately estimate the volume, composition, and quality
of stockpiled materials such as ore, coal, or aggregates.

• This information helps in optimizing inventory levels, planning material movements, and
managing stockpile logistics efficiently.

30
 Applications of SVM in Mining

These are just a few examples of how SVMs are applied in the mining industry to address various challenges and
improve operational performance.

SVM’s ability to handle multidimensional data, nonlinear relationships, and classification tasks makes them valuable
tools for mining companies seeking to leverage data-driven approaches for decision-making and process

EL
optimization.

PT
N
31
• O’Reilly Hands-on Machine Learning with Scikit – Learn, Keras & TensorFlow by Aurelien
Geron

• Deep learning with Python (2nd Edition) by François Chollet

EL
• [Link]

PT
guide-for-beginners/

N
32
• We discussed Support Vector Machine (SVM) in detail.

• We discussed the key concepts of Support Vector Machine (SVM)

• We discussed assumptions in Support Vector Machine (SVM)

EL
• We discussed common kernel functions used in SVM include:

PT
• Linear Kernel
• Polynomial Kernel
• Radial Basis Function (RBF) Kernel

N
• We discussed the advantages and disadvantages of Support Vector Machine (SVM)

• We discussed the applications of Support Vector Machine (SVM) in Mining

33
 N
PT
34
EL
 EL
PT
N
Module 10 : Machine Learning Lecture 25B: Naïve Bayes Classifier
• Introduction to Naive Bayes classifier
• Mathematics behind Naive Bayes classifier
• Assumptions of Naive Bayes classifier
• Worked out an example of Naive Bayes classifier

EL
• Advantages and disadvantages of Naive Bayes classifier

PT
• Applications of Naive Bayes classifier in Mining

N
2
 Naive Bayes Classifier
The Naive Bayes classifier is a simple probabilistic classifier based on Bayes' theorem with
strong (naive) independence assumptions between the features.

It is widely used in various machine-learning applications, especially in text classification and

spam filtering.

EL
Let's break down the Naive Bayes classifier with some mathematics:

PT
Bayes' Theorem:
Bayes' theorem is a fundamental theorem in probability theory that describes the probability of

N
an event, based on prior knowledge of conditions that might be related to the event.

3
 Bayes Theorem
It is expressed mathematically as:

EL
Where:
•P(A∣B) is the probability of event A occurring given that event B has occurred.
•P(B∣A) is the probability of event B occurring given that event A has occurred.

PT
•P(A) and P(B) are the probabilities of events A and B occurring independently.

N
4
 Naive Bayes Classifier
Naive Bayes Classifier:

• The Naive Bayes classifier applies Bayes' theorem for classification tasks by assuming that the presence of a
particular feature in a class is independent of the presence of any other feature.

• This assumption simplifies the computation and allows the model to be trained efficiently even with a large
number of features.

EL
Mathematics behind Naive Bayes Classifier:

• Let's consider a classification task with a set of features X={x1​,x2​,...,xn​} and a set of classes

PT
C = {c1​,c2​,...,ck​}.

N
• The goal is to predict the most probable class cj​ given a set of features X.

• The Naive Bayes classifier calculates the probability of each class given the features using
Bayes' theorem:

5
 Naive Bayes Classifier

Since P(X) is constant for all classes, we can simplify the above equation to:

EL
Where:

PT
•P(cj ∣ X) is the posterior probability of class cj​ given the features X.
•P(X ∣ cj) is the likelihood of observing the features X given class cj.

N
•P(cj​) is the prior probability of class cj.

6
 Naive Bayes Classifier
Naive Independence Assumption:
The Naive Bayes classifier assumes that the features are conditionally independent given the class label cj.

Mathematically, this means:

EL
Parameter Estimation:
To classify new instances, the Naive Bayes classifier needs to estimate two types of

PT
probabilities:

N
[Link] Probability P(cj ​): The probability of each class occurring in the dataset, usually
estimated by the frequency of each class in the training set.

[Link] Probability P(xi ∣ cj ​): The probability of observing each feature given the class,
often estimated using maximum likelihood estimation or other methods depending on the
type of features.

7
 Naive Bayes Classifier
Classification:

Once the prior and likelihood probabilities are estimated, the Naive Bayes classifier predicts the class cj​ with
the highest posterior probability P(cj ∣X).

EL
ෝ is the predicted class label.
Where 𝒚

PT
Example:

N
• Let's say we have a binary classification problem with two features, x1 and x2​, and two
classes, c1 and c2.

• We can calculate the posterior probabilities using the Naive Bayes classifier and make
predictions based on the highest probability.

8
 Naive Bayes Classifier
Here are the algorithmic steps involved in implementing the Naive Bayes classifier:
Input:
Training dataset D consisting of n samples with m features and their corresponding class labels.

Initialization:
Calculate the prior probabilities P(cj​) for each class cj​ by counting the frequency of each class in the training
dataset.

For each feature xi and each class cj​:

EL
• Estimate the likelihood probabilities P(xi∣cj​) using appropriate probability estimation techniques.

PT
Training:
Store the calculated prior and likelihood probabilities for future predictions.

N
Prediction:
For a new sample xnew, calculate the posterior probability P(cj∣ xnew ​) for each class cj​:

Select the class with the highest posterior probability as the predicted class label for xnew​:

9
 Assumptions of Naive Bayes Classifier
The Naive Bayes classifier makes several key assumptions in order to simplify the calculation of probabilities and
make predictions. These assumptions are important to understand when using Naive Bayes in machine learning:

Feature Independence:

• Perhaps the most significant assumption of Naive Bayes is that features are conditionally
independent given the class label. This means that the presence of one feature is assumed

EL
to be unrelated to the presence of any other feature, given the class label.

• Despite this being a simplification and often not strictly true in real-world data, Naive Bayes

PT
can still perform well in practice.

N
Class-Conditional Feature Distributions:

• Naive Bayes assumes that each class has its own distribution of feature values. In other
words, the probability distribution of each feature given the class is assumed to be
independent of the distribution of other features given the same class.

10
 Assumptions of Naive Bayes Classifier
Predictive Features:
• The classifier assumes that the features used for prediction are relevant and informative for
the classification task.

• Features that are irrelevant or redundant may still be included in the model, but they should
ideally have minimal impact on the classification decision.

Data Quality:

EL
• Naive Bayes assumes that the training data is representative of the population and is of
sufficient quality. Poor-quality or biased data may lead to inaccurate predictions.

PT
Class Prior Probability:

N
• The classifier assumes that the prior probability of each class is known or can be estimated
accurately from the training data.

• If the class distribution is highly skewed or imbalanced, this assumption may not hold, and
additional techniques such as class weighting or resampling may be necessary.

11
 Assumptions of Naive Bayes Classifier
Continuous Features Assumed to Follow a Specific Distribution:

• In Gaussian Naive Bayes, it is assumed that continuous features follow a Gaussian

(normal) distribution within each class.

• This assumption may not always hold true in practice, especially for features with
complex distributions.

EL
Zero Conditional Probability Handling:

PT
Naive Bayes assumes that no conditional probability is zero. In practice, this might lead
to issues if a particular feature value does not occur in the training set for a given class.

N
• Techniques like Laplace smoothing are often used to address this issue by adding a
small constant to all counts.

Understanding these assumptions is crucial when applying Naive Bayes in real-world

scenarios, as violations of these assumptions can affect the performance of the classifier.
Despite its simplifications, Naive Bayes can be surprisingly effective, especially in text
classification and other high-dimensional domains, where its computational efficiency and
ability to handle large feature spaces are advantageous.

12
 Naive Bayes Working Example
Understanding the Naïve Bayes' Classifier:

To grasp the functioning of the Naïve Bayes' Classifier, consider the following scenario:

Imagine we possess a dataset detailing weather conditions alongside a target variable "Play,"
indicating whether one should engage in outdoor activities on a given day based on these
conditions.

EL
To address this, we undertake the following steps:

PT
1. Translate the dataset into frequency tables.
2. Construct a Likelihood table by computing the probabilities associated with the given

N
features.
3. Utilize Bayes' theorem to ascertain the posterior probability.

For instance, let's tackle the question: Given sunny weather, should the player engage in
Playing game or not?

To resolve this, let's examine the dataset provided in next slide:

13
 Naive Bayes Working Example
Dataset:

EL
PT
N
14
 Naive Bayes Working Example
Frequency table for the Weather Conditions:

EL
PT
Likelihood table weather condition:

N
15
 Naive Bayes Working Example
Applying Bayes theorem:

P(Yes|Sunny)= P(Sunny|Yes)*P(Yes)/P(Sunny) P(No|Sunny)= P(Sunny|No)*P(No)/P(Sunny)

P(Sunny|Yes)= 3/10= 0.3 P(Sunny|NO)= 2/4=0.5

P(Sunny)= 0.35 P(No)= 0.29

EL
P(Yes)=0.71 P(Sunny)= 0.35

PT
So P(Yes|Sunny) = 0.3*0.71/0.35 = 0.60 So P(No|Sunny)= 0.5*0.29/0.35 = 0.41

N
Observing the preceding calculation reveals that P(Yes|Sunny)>P(No|Sunny)

Consequently, on a sunny day, the player is recommended to participate in

the game.

16
 Advantages and Disadvantages of Naive Bayes Classifier
The Naive Bayes classifier has several advantages and disadvantages:

Advantages:

Simple and Fast:

• Naive Bayes is a simple and fast algorithm, making it easy to implement and
computationally efficient.

EL
• It scales well with large datasets and high-dimensional feature spaces.

PT
Efficient with Large Feature Spaces:

N
• Naive Bayes performs well even with a large number of features.

• It can handle datasets with thousands of features without significant computational

overhead.

Robust to Irrelevant Features:

• Naive Bayes is robust to irrelevant features. It can still produce good results even if
some features are not informative for the classification task.

17
 Advantages and Disadvantages of Naive Bayes Classifier
The Naive Bayes classifier has several advantages and disadvantages:

Advantages:

Handles Missing Data:

• Naive Bayes can handle missing data by simply ignoring the missing values during
training and prediction. This can be advantageous in real-world datasets where missing

EL
data is common.

PT
Effective for Text Classification:

N
• Naive Bayes is particularly effective for text classification tasks, such as spam filtering
and sentiment analysis. It performs well even with a relatively small amount of training
data.

Probabilistic Framework:

• Naive Bayes provides probabilistic predictions, allowing for easy interpretation of results
and uncertainty estimation.

18
 Advantages and Disadvantages of Naive Bayes Classifier
Disadvantages:

Assumption of Feature Independence:

• The assumption of feature independence may not hold true in many real-world datasets.
In cases where features are highly correlated, Naive Bayes may produce suboptimal
results.

EL
Sensitivity to Feature Distribution:

PT
• Naive Bayes assumes features follow a specific distribution within each class (e.g.,
Gaussian for continuous features). If this assumption is violated, the classifier's

N
performance may degrade.

Limited Expressiveness:

• Due to its simple probabilistic model, Naive Bayes may not capture complex
relationships between features and class labels as effectively as more sophisticated
algorithms like decision trees or neural networks.

19
 Advantages and Disadvantages of Naive Bayes Classifier
Disadvantages:

Zero Frequency Problem:

• Naive Bayes may encounter issues when a categorical feature value appears in the
testing dataset but not in the training dataset. This can lead to zero frequency counts
and affect the classifier's performance.

EL
Cannot Handle Numeric Data Well:

PT
• While Gaussian Naive Bayes can handle continuous numeric data, other variants like
Multinomial and Bernoulli Naive Bayes are designed for categorical data or discrete

N
features.

• Handling numeric data directly with these variants may require binning or other
preprocessing techniques.

20
 Advantages and Disadvantages of Naive Bayes Classifier
Disadvantages:

Requires Well-Balanced Classes:

• Naive Bayes tends to perform better when the class distribution is balanced. In datasets with highly imbalanced
classes, Naive Bayes may favor the majority class and produce biased predictions.

EL
Despite these limitations, Naive Bayes remains a popular choice for various classification tasks, especially in
scenarios where simplicity, speed, and efficiency are prioritized over model complexity and accuracy.

PT
N
21
 Application of the Naive Bayes classifier in mining
Applications of the Naive Bayes classifier in mining engineering:

Mineral Prospectivity Mapping:

Description:

• Mineral prospectivity mapping aims to identify areas with high potential for mineral

EL
deposits based on geological, geophysical, and geochemical data.

PT
Application:

• Naive Bayes classifiers can analyze spatial datasets containing geological features,

N
mineral occurrences, and exploration data to predict the likelihood of finding
economically viable mineral deposits in unexplored or underexplored regions.

• By considering factors such as geological formations, structural controls, and

mineralization indicators, Naive Bayes classifiers can assist in prioritizing exploration
targets and optimizing resource allocation in mineral exploration campaigns.

22
Applications of the Naive Bayes classifier in mining engineering:

Underground Mine Safety Monitoring:

Description:

• Underground mine safety monitoring involves detecting hazardous conditions and

ensuring the safety of workers in underground mining environments.

EL
Application:

PT
• Naive Bayes classifiers can analyze real-time sensor data from equipment, ventilation
systems, gas detectors, and personnel tracking devices to identify potential safety

N
hazards such as gas leaks, equipment malfunctions, or personnel emergencies.

• By classifying sensor data patterns associated with safety-critical events, Naive Bayes
classifiers can trigger alarms, initiate safety protocols, and provide early warnings to
miners and supervisors, thereby enhancing safety and accident prevention in
underground mines.

23
Applications of the Naive Bayes classifier in mining engineering:

Ore Grade Estimation:

Description:

• Ore grade estimation is essential for optimizing mineral processing operations, resource
planning, and mine economics by accurately quantifying the mineral content and quality of
ore.

EL
Application:

PT
• Naive Bayes classifiers can analyze multi-modal datasets comprising geological,

N
geochemical, and mineralogical data to predict ore grades and mineral recoveries in
mining operations.

• By modeling the relationship between exploration data, ore characteristics, and

processing parameters, Naive Bayes classifiers can provide estimates of ore grades,
mineral compositions, and metallurgical properties, supporting decision-making
processes related to ore extraction, processing, and ore reserve estimation.

24
Applications of the Naive Bayes classifier in mining engineering:

Water Management and Tailings Prediction:

Description:

• Water management and tailings prediction involve assessing water resources, minimizing
water consumption, and predicting the behavior of tailings facilities to mitigate
environmental risks.

EL
Application:

PT
• Naive Bayes classifiers can analyze hydrological, meteorological, and geological data to
predict water inflows, groundwater interactions, and tailings dam stability in mining

N
operations.

• By considering factors such as precipitation patterns, surface water runoff, and geological
characteristics, Naive Bayes classifiers can forecast water-related risks, optimize water
management strategies, and support decision-making regarding tailings disposal, dam
construction, and environmental remediation efforts.

25
Applications of the Naive Bayes classifier in mining engineering:

Energy Efficiency Optimization:

Description:

• Energy efficiency optimization aims to reduce energy consumption and greenhouse gas
emissions in mining operations by improving energy efficiency, implementing
renewable energy sources, and optimizing energy management practices.

EL
Application:

PT
• Naive Bayes classifiers can analyze energy consumption data, process parameters, and

N
operational variables to identify opportunities for energy savings, optimize equipment
utilization, and prioritize energy efficiency measures in mining operations.

• By classifying energy usage patterns, equipment performance metrics, and energy

efficiency indicators, Naive Bayes classifiers can support decision-making regarding
energy audits, equipment upgrades, and renewable energy integration, contributing to
sustainable mining practices and cost reduction initiatives.

26
• O’Reilly Hands-on Machine Learning with Scikit – Learn, Keras & TensorFlow by Aurelien
Geron

• Deep learning with Python (2nd Edition) by François Chollet

EL
• [Link]

PT
N
27
• We discussed the Naive Bayes classifier in detail mathematically

• We discussed the Assumptions of the Naive Bayes classifier

EL
• We solved one working example of the Naive Bayes classifier

PT
We discussed the Advantages and disadvantages of the Naive Bayes classifier

• Finally, we discussed the applications of the Naive Bayes classifier in Mining

N
28
 N
PT
29
EL