Machine Learning using Python – III

IPH2445

Course Notes

Department of Physics
Marian College Kuttikkanam (Autonomous)

Academic Year 2026-27

IPH2445: Machine Learning using Python – III Chapter 1

1 Introduction to Machine Learning Classification

1.1 Learning Outcomes
After studying this chapter, students will be able to:

1. Explain the concept of classification in machine learning.

2. Distinguish between supervised and unsupervised learning.

3. Identify binary and multiclass classification problems.

4. Recognize real-world applications of classification algorithms.

5. Understand the workflow of a machine learning classification system.

1.2 Introduction to Machine Learning

Machine Learning (ML) is a branch of Artificial Intelligence (AI) that enables computers
to learn patterns from data and make decisions or predictions without being explicitly
programmed.
Traditionally, computer programs follow fixed instructions written by programmers.
In contrast, machine learning systems learn from examples and improve their performance
as more data become available.
Machine learning is widely used in:

• Medical diagnosis

• Weather prediction

• Recommendation systems

• Autonomous vehicles

• Image recognition

• Speech recognition

• Fraud detection

Key Point
A machine learning system learns patterns from data and uses those patterns to
make predictions or decisions.

A typical machine learning workflow is:

Data Collection → Data Preparation → Model Training → Testing →

Prediction → Evaluation

1
IPH2445: Machine Learning using Python – III Chapter 1

1.3 What is Classification?

Definition
Classification is the process of assigning an input sample to one of several predefined
categories or classes.

The output of a classification model is a discrete label rather than a continuous

numerical value.

Example
A hospital wants to determine whether a patient has diabetes based on age, blood
sugar level and BMI.
Possible classes:

• Diabetic

• Non-Diabetic

Table 1: Sample Patient Dataset

Blood Sugar Level Age Diagnosis
180 45 Diabetic
95 30 Non-Diabetic
210 60 Diabetic

1.4 Classification versus Regression

Machine learning tasks are generally divided into classification and regression.

Table 2: Classification vs Regression

Classification Regression
Predicts categories or labels Predicts numerical values
Output is discrete Output is continuous
Spam Detection House Price Prediction
Disease Diagnosis Temperature Forecasting

1.5 Types of Machine Learning

Machine learning can be divided into:
1. Supervised Learning
2. Unsupervised Learning
3. Reinforcement Learning
In this chapter we focus on supervised and unsupervised learning.

2
IPH2445: Machine Learning using Python – III Chapter 1

1.6 Supervised Learning

Definition
Supervised learning is a machine learning approach in which the model is trained
using labeled data.

A labeled dataset contains:

• Input features
• Correct output labels

Example

Table 3: Student Dataset

Hours Studied Result
2 Fail
4 Pass
6 Pass

Applications
• Email Spam Detection
• Disease Diagnosis
• Face Recognition
• Credit Approval
• Handwritten Digit Recognition

1.7 Unsupervised Learning

Definition
Unsupervised learning is a machine learning approach in which the data contain
no labels and the algorithm discovers hidden structures.

Applications
• Customer Segmentation
• Market Basket Analysis
• Recommendation Systems
• Anomaly Detection
• Image Compression

3
IPH2445: Machine Learning using Python – III Chapter 1

1.8 Supervised versus Unsupervised Learning

Table 4: Comparison of Learning Paradigms

Feature Supervised Learning Unsupervised Learning
Training Data Labeled Unlabeled
Goal Predict labels Discover patterns
Output Known classes Clusters
Evaluation Easy Difficult
Example Disease Diagnosis Customer Segmentation

1.9 Binary Classification

Definition
Binary classification assigns data into one of two possible classes.

Examples:

• Spam / Not Spam

• Disease / No Disease

• Pass / Fail

• Approve / Reject

1.10 Multiclass Classification

Definition
Multiclass classification assigns data into one of three or more classes.

Examples:

• Handwritten Digit Recognition (0–9)

• Animal Classification

• Language Identification

• Fruit Classification

1.11 Real-World Applications of Classification

1. Medical Diagnosis

2. Email Spam Detection

3. Fraud Detection

4
IPH2445: Machine Learning using Python – III Chapter 1

4. Face Recognition
5. Autonomous Vehicles
6. Astronomy and Astrophysics
7. Particle Physics Experiments

1.12 Summary
• Machine learning enables computers to learn from data.
• Classification predicts categories rather than numerical values.
• Supervised learning uses labeled data.
• Unsupervised learning uses unlabeled data.
• Binary classification involves two classes.
• Multiclass classification involves three or more classes.

1.13 Review Questions

Short Answer Questions
1. Define machine learning.
2. What is classification?
3. Differentiate between classification and regression.
4. What is supervised learning?
5. What is unsupervised learning?
6. Define labeled data.
7. Define unlabeled data.
8. What is binary classification?
9. What is multiclass classification?
10. Give two applications of classification.

Descriptive Questions
1. Explain the workflow of a machine learning system.
2. Discuss classification with suitable examples.
3. Compare supervised and unsupervised learning.
4. Explain binary and multiclass classification.
5. Describe applications of classification in science and engineering.

5
IPH2445: Machine Learning using Python – III Chapter 1

Application-Oriented Questions
1. Explain why diabetes prediction is a classification problem.
2. Suggest a suitable learning method for customer segmentation.
3. Design a spam email classifier and identify inputs and outputs.
4. Formulate student pass/fail prediction as a classification problem.
5. Explain why handwritten digit recognition is a multiclass classification task.

2 Logistic Regression Fundamentals: Probability and

Sigmoid Function
2.1 Learning Outcomes
After studying this section, students will be able to:

1. Explain the need for Logistic Regression.

2. Distinguish between Linear Regression and Logistic Regression.
3. Interpret probability-based predictions.
4. Understand the mathematical form of the sigmoid function.
5. Relate sigmoid outputs to classification decisions.
6. Apply Logistic Regression concepts to real-world classification problems.

2.2 Introduction
In the previous section, classification problems were introduced as tasks that assign data
points to predefined categories. A natural question arises: How can a machine learning
algorithm decide whether an object belongs to one class or another?
One of the most widely used algorithms for solving binary classification problems is
Logistic Regression. Despite its name, Logistic Regression is primarily a classification
algorithm rather than a regression algorithm.
Logistic Regression predicts the probability that a data sample belongs to a particular
class. The predicted probability is then converted into a class label using a decision
threshold.
Typical applications include:

• Disease diagnosis
• Email spam detection
• Credit risk assessment
• Customer churn prediction
• Fraud detection

6
IPH2445: Machine Learning using Python – III Chapter 1

2.3 Why Not Use Linear Regression for Classification?

Linear Regression predicts continuous numerical values.
The linear regression model is expressed as

y = mx + c
where

• m = slope

• x = input variable

• c = intercept

For classification problems, outputs must represent probabilities between 0 and 1.

However, Linear Regression can produce values such as

y = −2.3
or

y = 3.7
which cannot represent probabilities.

Key Point
Classification requires outputs between 0 and 1. Logistic Regression solves this
problem by transforming linear outputs into probabilities using the sigmoid func-
tion.

2.4 Need for Logistic Regression

Consider a hospital attempting to predict whether a patient has diabetes.
Possible outcomes are:

• 0 = Non-Diabetic

• 1 = Diabetic

Instead of directly predicting the class label, Logistic Regression predicts the proba-
bility that a patient belongs to the diabetic class.
Examples:

Probability Prediction
0.90 Diabetic
0.75 Diabetic
0.45 Non-Diabetic
0.10 Non-Diabetic

Thus, Logistic Regression converts classification into a probability estimation problem.

7
IPH2445: Machine Learning using Python – III Chapter 1

2.5 Probability and Classification

Probability measures the likelihood of an event occurring.
The probability of an event lies between

0≤P ≤1
where

• P = 0 indicates impossibility.

• P = 1 indicates certainty.

• Intermediate values indicate varying likelihoods.

Examples:

Probability Interpretation
0.95 Very likely
0.75 Likely
0.50 Uncertain
0.25 Unlikely
0.05 Very unlikely

Logistic Regression uses these probabilities to determine class membership.

2.6 Decision Threshold

To convert probabilities into class labels, a threshold value is selected.
The most commonly used threshold is

0.5
Decision rule:

P ≥ 0.5 ⇒ Class 1

P < 0.5 ⇒ Class 0

For example:

Probability Predicted Class Interpretation

0.85 1 Positive
0.62 1 Positive
0.48 0 Negative
0.15 0 Negative

8
IPH2445: Machine Learning using Python – III Chapter 1

2.7 The Sigmoid Function

The sigmoid function is the mathematical foundation of Logistic Regression.
It converts any real-valued input into a value between 0 and 1.
The sigmoid function is given by
where

• σ(z) = sigmoid output

• z = input value

• e = Euler’s number (e ≈ 2.718)

The output always satisfies

0 < σ(z) < 1

making it suitable for probability estimation.

2.8 Properties of the Sigmoid Function

The sigmoid function has the following properties:

1. Output ranges between 0 and 1.

2. The curve is S-shaped.

3. Large positive values approach 1.

4. Large negative values approach 0.

5. The midpoint occurs at σ(0) = 0.5.

Some sample values are shown below.

z σ(z)
-4 0.018
-2 0.119
0 0.500
2 0.881
4 0.982

2.9 Interpretation of the Sigmoid Curve

The sigmoid curve gradually transforms linear outputs into probabilities.

• Negative inputs correspond to probabilities near 0.

• Positive inputs correspond to probabilities near 1.

• Values near zero correspond to uncertain classifications.

9
IPH2445: Machine Learning using Python – III Chapter 1

The decision boundary occurs at

σ(z) = 0.5
which corresponds to

z=0
This boundary separates the two classes.

2.10 Example Calculation

Suppose

z=2
Then
1
σ(2) =
1 + e−2
1
=
1 + 0.1353

= 0.881
Thus,

P = 0.881
Since

0.881 > 0.5

the observation is assigned to Class 1.

2.11 Applications of Logistic Regression

Logistic Regression is widely used in:

1. Medical diagnosis

2. Credit approval systems

3. Email spam detection

4. Customer churn prediction

5. Sentiment analysis

6. Fraud detection

7. Quality control in manufacturing

10
IPH2445: Machine Learning using Python – III Chapter 1

2.12 Summary
• Logistic Regression is a classification algorithm.

• It predicts probabilities rather than numerical values.

• Linear Regression is unsuitable for classification because outputs are unbounded.

• Logistic Regression uses the sigmoid function to constrain outputs between 0 and
1.

• A threshold value converts probabilities into class labels.

• The sigmoid function produces an S-shaped curve suitable for binary classification.

2.13 Review Questions

Short Answer Questions
1. What is Logistic Regression?

2. Why is Linear Regression unsuitable for classification?

3. Define probability.

4. What is a decision threshold?

5. Write the sigmoid function.

6. What is the range of sigmoid outputs?

7. What happens when z = 0?

8. Why is the sigmoid function useful in classification?

9. Give two applications of Logistic Regression.

10. What is the meaning of a probability value of 0.9?

Descriptive Questions
1. Explain the need for Logistic Regression.

2. Compare Linear Regression and Logistic Regression.

3. Discuss the properties of the sigmoid function.

4. Explain how probabilities are converted into class labels.

5. Describe the role of Logistic Regression in machine learning classification.

11
IPH2445: Machine Learning using Python – III Chapter 1

Application-Oriented Questions
1. A hospital predicts a diabetes probability of 0.82 for a patient. Determine the
predicted class using a threshold of 0.5.

2. A bank uses Logistic Regression to assess loan risk. Explain how probability esti-
mates assist decision-making.

3. Calculate the sigmoid output for z = 1 and interpret the result.

4. Explain how Logistic Regression can be used for email spam detection.

5. Design a simple binary classification problem that can be solved using Logistic
Regression.

3 Logit Function, Odds, Log-Odds and Decision Bound-

ary
3.1 Learning Outcomes
After studying this section, students will be able to:

1. Calculate odds and log-odds from probabilities.

2. Explain the concept of the logit function.

3. Interpret logit transformation in Logistic Regression.

4. Relate odds, probabilities and log-odds.

5. Determine decision boundaries for binary classification.

6. Analyze classification decisions using Logistic Regression.

3.2 Introduction
In the previous section, Logistic Regression was introduced as a classification algorithm
that predicts probabilities using the sigmoid function. However, a fundamental question
remains:

How does Logistic Regression transform a linear combination of input features into a
probability?

The answer lies in the concepts of odds, log-odds, and the logit function. These
concepts form the mathematical foundation of Logistic Regression and allow probabilities
to be modeled using linear equations.

12
IPH2445: Machine Learning using Python – III Chapter 1

3.3 Probability Revisited

Recall that probability measures the likelihood of an event occurring.

0≤P ≤1
where:

• P = 0 indicates impossibility.

• P = 1 indicates certainty.

• Intermediate values represent varying likelihoods.

For example,

P = 0.80
means there is an 80% chance that the event occurs.
While probabilities are useful, Logistic Regression does not model probabilities di-
rectly. Instead, it models odds.

3.4 Odds
Definition
Odds represent the ratio of the probability that an event occurs to the probability
that it does not occur.

Mathematically,
P
Odds =
1−P
where:

• P = probability of success

• (1 − P ) = probability of failure

Example 1
Suppose

P = 0.75
Then
0.75 0.75
Odds = = =3
1 − 0.75 0.25
This means the event is three times more likely to occur than not occur.

13
IPH2445: Machine Learning using Python – III Chapter 1

Example 2
Suppose

P = 0.20
Then
0.20
Odds = = 0.25
0.80
The event is less likely to occur than not occur.

3.5 Interpretation of Odds

Probability Odds Interpretation
0.10 0.11 Very unlikely
0.25 0.33 Unlikely
0.50 1 Equally likely
0.75 3 Likely
0.90 9 Very likely

Notice that odds can vary from

0→∞
Unlike probabilities, odds are not restricted to the interval [0, 1].

3.6 Log-Odds
The odds ratio is always positive and can become very large. This makes it inconvenient
for linear modeling.
To overcome this difficulty, Logistic Regression applies the natural logarithm to the
odds.

Definition
The logarithm of the odds is called the log-odds.

Mathematically,
P
 
Log-Odds = ln
1−P
The log-odds can take any value from

−∞ to +∞
which makes them suitable for linear equations.

14
IPH2445: Machine Learning using Python – III Chapter 1

3.7 Example of Log-Odds

Suppose

P = 0.80
First compute the odds:
0.80
=4
0.20
Then

Log-Odds = ln(4) = 1.386

Similarly, if

P = 0.20
then
0.20
= 0.25
0.80
and

ln(0.25) = −1.386
Notice:

• Positive log-odds correspond to probabilities greater than 0.5.

• Negative log-odds correspond to probabilities less than 0.5.

• Zero log-odds correspond to probability 0.5.

3.8 The Logit Function

Definition
The logit function is the logarithm of the odds ratio.

The logit transformation is given by

P
 
logit(P ) = ln
1−P
The logit function transforms probabilities from the interval

(0, 1)
to the interval

(−∞, +∞)
This transformation allows a linear model to be constructed.

15
IPH2445: Machine Learning using Python – III Chapter 1

3.9 Logistic Regression Equation

Logistic Regression assumes that the log-odds vary linearly with the input variables.
For a single feature x,
P
 
ln = β0 + β1 x
1−P
where:

• β0 = intercept

• β1 = coefficient

• x = input feature

For multiple variables,

P
 
ln = β0 + β1 x1 + β2 x2 + · · · + βn xn
1−P
This equation is called the Logistic Regression model.

3.10 Relationship Between Logit and Sigmoid Functions

The logit and sigmoid functions are inverses of one another.
Logit transforms

P → Log-Odds
while sigmoid transforms

Log-Odds → P
Thus,
1
P =
1 + e−z
where

z = β0 + β 1 x
The sigmoid function converts linear outputs into probabilities.

3.11 Decision Boundary

A classification model must determine where one class ends and another begins.

Definition
The decision boundary is the line, curve or surface that separates different classes.

For Logistic Regression, the decision boundary occurs when

P = 0.5

16
IPH2445: Machine Learning using Python – III Chapter 1

At this point,

Odds = 1
and

ln(1) = 0
Therefore,

β0 + β 1 x = 0
This equation defines the decision boundary.

3.12 Example: One-Dimensional Decision Boundary

Suppose the Logistic Regression model is

z = −4 + 2x
The decision boundary occurs when

z=0
Thus,

−4 + 2x = 0

x=2
Therefore:

• x < 2 belongs to Class 0.

• x > 2 belongs to Class 1.

The point x = 2 is the decision boundary.

3.13 Decision Boundary in Two Dimensions

For two features,

z = β0 + β1 x1 + β2 x2
The decision boundary is obtained by setting

z=0
giving

β 0 + β 1 x1 + β 2 x2 = 0
which represents a straight line in a two-dimensional feature space.
This line separates the two classes predicted by the model.

17
IPH2445: Machine Learning using Python – III Chapter 1

3.14 Applications
Odds, log-odds and decision boundaries are widely used in:

1. Medical diagnosis

2. Credit risk analysis

3. Customer behavior prediction

4. Fraud detection

5. Marketing analytics

6. Insurance risk assessment

3.15 Summary
• Odds measure the likelihood of success relative to failure.

• Log-odds are obtained by taking the logarithm of the odds.

• The logit function transforms probabilities into values ranging from −∞ to +∞.

• Logistic Regression models the log-odds as a linear function of input variables.

• The sigmoid function converts log-odds back into probabilities.

• Decision boundaries separate predicted classes.

• The Logistic Regression decision boundary occurs when P = 0.5.

3.16 Review Questions

Short Answer Questions
1. Define odds.

2. Define log-odds.

3. What is the logit function?

4. Write the mathematical expression for odds.

5. Write the mathematical expression for log-odds.

6. Why are log-odds used in Logistic Regression?

7. What is a decision boundary?

8. What happens when the probability is 0.5?

9. What is the value of the logit when P = 0.5?

10. State two applications of Logistic Regression.

18
IPH2445: Machine Learning using Python – III Chapter 1

Descriptive Questions
1. Explain the concept of odds with suitable examples.

2. Derive the logit function from probability and odds.

3. Discuss the role of log-odds in Logistic Regression.

4. Explain the relationship between sigmoid and logit functions.

5. Describe the concept of a decision boundary.

Application-Oriented Questions
1. A patient has a disease probability of 0.8. Calculate the odds and log-odds.

2. Calculate the logit value corresponding to a probability of 0.6.

3. Determine the decision boundary for the model z = −6 + 3x.

4. Explain how decision boundaries are used in spam email classification.

5. A bank predicts a loan default probability of 0.3. Compute the odds and interpret
the result.

4 Logistic Regression using Scikit-Learn

4.1 Learning Outcomes
After studying this section, students will be able to:

1. Understand the workflow of implementing Logistic Regression using Scikit-Learn.

2. Load and explore datasets in Python.

3. Split datasets into training and testing sets.

4. Train a Logistic Regression model.

5. Perform predictions using trained models.

6. Interpret model outputs and prediction probabilities.

7. Evaluate the performance of a Logistic Regression classifier.

8. Apply Logistic Regression to the Iris dataset.

19
IPH2445: Machine Learning using Python – III Chapter 1

4.2 Introduction
In the previous sections, we studied the mathematical foundations of Logistic Regression,
including probability, odds, log-odds, sigmoid functions and decision boundaries.
In practical machine learning applications, these calculations are performed automat-
ically using machine learning libraries. One of the most widely used libraries for machine
learning in Python is Scikit-Learn.
Scikit-Learn provides efficient implementations of many machine learning algorithms,
including Logistic Regression.
The typical workflow consists of:

Data Collection → Data Preparation → Model Training → Prediction → Evaluation

This section demonstrates the implementation of Logistic Regression using the Iris
dataset.

4.3 Introduction to Scikit-Learn

Scikit-Learn is an open-source machine learning library built on top of:

• NumPy

• SciPy

• Matplotlib

It provides tools for:

• Classification

• Regression

• Clustering

• Dimensionality Reduction

• Model Evaluation

• Data Preprocessing

Scikit-Learn is imported using:

import sklearn

20
IPH2445: Machine Learning using Python – III Chapter 1

4.4 The Iris Dataset

The Iris dataset is one of the most famous datasets in machine learning.
It contains measurements of iris flowers belonging to three species:

1. Iris Setosa

2. Iris Versicolor

3. Iris Virginica

The dataset contains four features:

1. Sepal Length

2. Sepal Width

3. Petal Length

4. Petal Width

The target variable is the flower species.

Table 5: Features of the Iris Dataset

Feature Description
Sepal Length Length of sepal (cm)
Sepal Width Width of sepal (cm)
Petal Length Length of petal (cm)
Petal Width Width of petal (cm)

The objective is to classify a flower into one of the three species.

4.5 Loading the Iris Dataset

The dataset can be loaded directly from Scikit-Learn.

from [Link] import load_iris

iris = load_iris()

X = [Link]
y = [Link]

Here,

• X contains the input features.

• y contains the target labels.

21
IPH2445: Machine Learning using Python – III Chapter 1

4.6 Exploring the Dataset

The dimensions of the dataset can be examined using:

print([Link])
print([Link])

Output:

(150, 4)
(150,)

This indicates:

• 150 flower samples

• 4 features per sample

The target classes are:

print(iris.target_names)

Output:

[’setosa’ ’versicolor’ ’virginica’]

4.7 Splitting the Dataset

Machine learning models must be evaluated using unseen data.
The dataset is divided into:

• Training Set

• Testing Set

The training set is used for learning.

The testing set is used for evaluation.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(

X,
y,
test_size=0.2,
random_state=42
)

Here:

• 80% of the data is used for training.

• 20% is used for testing.

22
IPH2445: Machine Learning using Python – III Chapter 1

4.8 Training the Logistic Regression Model

The Logistic Regression model is imported from Scikit-Learn.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=200)

[Link](X_train, y_train)

The fit() method trains the model using the training data.
Mathematically, the model learns the coefficients

β0 , β1 , β2 , β3 , β4
that best separate the flower classes.

4.9 Making Predictions

After training, predictions can be made using:

y_pred = [Link](X_test)

The output contains predicted class labels.

Example:

[1 0 2 1 1]

where:

• 0 = Setosa

• 1 = Versicolor

• 2 = Virginica

4.10 Predicting a New Flower

Suppose a flower has measurements:

• Sepal Length = 5.1 cm

• Sepal Width = 3.5 cm

• Petal Length = 1.4 cm

• Petal Width = 0.2 cm

Prediction:

23
IPH2445: Machine Learning using Python – III Chapter 1

sample = [[5.1,3.5,1.4,0.2]]

prediction = [Link](sample)

print(prediction)

Output:

[0]

This corresponds to Iris Setosa.

4.11 Prediction Probabilities

Logistic Regression predicts probabilities before assigning classes.
These probabilities can be viewed using:

model.predict_proba(sample)

Example output:

[[0.98 0.01 0.01]]

Interpretation:

• 98% probability of Setosa

• 1% probability of Versicolor

• 1% probability of Virginica

The class with the highest probability is selected.

4.12 Understanding Model Coefficients

The coefficients learned by the model can be displayed.

print(model.coef_)

Each coefficient indicates the influence of a feature on the classification decision.

Positive coefficients increase the likelihood of a class.
Negative coefficients decrease the likelihood of a class.

24
IPH2445: Machine Learning using Python – III Chapter 1

4.13 Model Evaluation

The simplest evaluation metric is accuracy.
Correct Predictions
Accuracy =
Total Predictions
Using Scikit-Learn:

from [Link] import accuracy_score

accuracy = accuracy_score(y_test,y_pred)

print(accuracy)

Example output:

0.97

This indicates an accuracy of

97%

4.14 Complete Program

from [Link] import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from [Link] import accuracy_score

iris = load_iris()

X = [Link]
y = [Link]

X_train, X_test, y_train, y_test = train_test_split(

X,y,test_size=0.2,random_state=42)

model = LogisticRegression(max_iter=200)

[Link](X_train,y_train)

y_pred = [Link](X_test)

print("Accuracy:",
accuracy_score(y_test,y_pred))

25
IPH2445: Machine Learning using Python – III Chapter 1

4.15 Laboratory Exercise

Aim
To implement Logistic Regression using Scikit-Learn and classify Iris flowers.
Procedure

1. Import the required libraries.

2. Load the Iris dataset.

3. Split the dataset into training and testing sets.

4. Train a Logistic Regression model.

5. Predict flower species.

6. Calculate accuracy.

7. Interpret the results.

Expected Outcome
Students should be able to:

• Train Logistic Regression models.

• Predict class labels.

• Interpret prediction probabilities.

• Evaluate model performance.

4.16 Summary
• Scikit-Learn provides an efficient implementation of Logistic Regression.

• The Iris dataset is commonly used for classification studies.

• Machine learning workflows involve data loading, training, testing and evaluation.

• The fit() method trains a model.

• The predict() method generates class labels.

• The predict_proba() method provides probabilities.

• Accuracy is a common performance metric.

26
IPH2445: Machine Learning using Python – III Chapter 1

4.17 Review Questions

Short Answer Questions
1. What is Scikit-Learn?

2. What is the Iris dataset?

3. What is the purpose of train-test splitting?

4. What does the fit() method do?

5. What does the predict() method do?

6. What is prediction probability?

7. What is accuracy?

8. Why is testing data necessary?

9. What are model coefficients?

10. Name two datasets commonly used for classification.

Descriptive Questions
1. Explain the workflow of implementing Logistic Regression using Scikit-Learn.

2. Describe the Iris dataset and its significance.

3. Discuss the role of training and testing datasets.

4. Explain how Logistic Regression predictions are generated.

5. Discuss the interpretation of model coefficients and probabilities.

Application-Oriented Questions
1. Build a Logistic Regression model to classify flowers using the Iris dataset.

2. Predict the class of a flower with given measurements and interpret the probability
outputs.

3. Compare model performance when using different train-test splits.

4. Investigate the effect of changing max_iter on model convergence.

5. Modify the program to classify a different dataset and compare the results.

27
IPH2445: Machine Learning using Python – III Chapter 1

5 Model Evaluation: Accuracy, Train-Test Split and

Overfitting
5.1 Learning Outcomes
After studying this section, students will be able to:

1. Explain the importance of model evaluation in machine learning.

2. Understand the purpose of train-test splitting.

3. Calculate and interpret classification accuracy.

4. Compare training and testing performance.

5. Identify signs of overfitting and underfitting.

6. Evaluate the generalization ability of a model.

7. Apply evaluation techniques to Logistic Regression models.

5.2 Introduction
Developing a machine learning model is only the first step in solving a classification
problem. A model that performs well on the training data may not necessarily perform
well on new, unseen data.
Therefore, machine learning practitioners must evaluate how effectively a model gen-
eralizes beyond the data used during training.
Model evaluation helps answer important questions:

• How accurate are the predictions?

• Does the model generalize well to unseen data?

• Is the model overfitting or underfitting?

• Can the model be trusted in real-world applications?

A reliable model should perform consistently on both training and testing datasets.

5.3 Why Model Evaluation is Important

Consider a student who memorizes answers to previous examination questions without
understanding the concepts.
The student may score perfectly on familiar questions but perform poorly on new
questions.
Similarly, a machine learning model may memorize training data instead of learning
meaningful patterns.
Model evaluation ensures that a classifier learns general relationships rather than
simply memorizing examples.

28
IPH2445: Machine Learning using Python – III Chapter 1

Key Point
The ultimate goal of machine learning is not to achieve perfect training accuracy
but to make accurate predictions on unseen data.

5.4 Training and Testing Data

To evaluate a model fairly, the dataset is divided into two parts:

1. Training Set
2. Testing Set

Training Set
The training set is used to learn model parameters.
For Logistic Regression, the coefficients are estimated using the training data.

Testing Set
The testing set is not used during training.
It is used only after the model has been trained to evaluate performance on unseen
data.

Training Data → Model Training

Testing Data → Model Evaluation

5.5 Train-Test Split

The process of dividing a dataset into training and testing subsets is called a train-test
split.
Common choices include:

Table 6: Common Train-Test Splits

Training Set Testing Set Usage
70% 30% Small datasets
80% 20% Most common
90% 10% Large datasets

In Scikit-Learn:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test =

train_test_split(
X,
y,
test_size=0.2,
random_state=42)

29
IPH2445: Machine Learning using Python – III Chapter 1

Here:

• 80% of data is used for training.

• 20% of data is used for testing.

5.6 Classification Accuracy

The most basic evaluation metric is accuracy.

Definition
Accuracy is the proportion of correctly classified observations among all observa-
tions.

Mathematically,
Number of Correct Predictions
Accuracy =
Total Number of Predictions
or
TP + TN
Accuracy =
TP + TN + FP + FN
where

• TP = True Positives

• TN = True Negatives

• FP = False Positives

• FN = False Negatives

5.7 Example of Accuracy Calculation

Suppose a classifier makes predictions on 100 samples.

• Correct predictions = 92

• Incorrect predictions = 8

Then
92
Accuracy = = 0.92
100
Therefore,

Accuracy = 92%
The model correctly classifies 92 out of every 100 observations.

30
IPH2445: Machine Learning using Python – III Chapter 1

5.8 Accuracy using Scikit-Learn

Accuracy can be calculated using:

from [Link] import accuracy_score

accuracy = accuracy_score(y_test,y_pred)

print("Accuracy =", accuracy)

Example output:

Accuracy = 0.9667

This corresponds to

96.67%
classification accuracy.

5.9 Training Accuracy and Testing Accuracy

Two different accuracies are often measured:

1. Training Accuracy

2. Testing Accuracy

Training Accuracy
Performance on training data.

train_accuracy =
[Link](X_train,y_train)

Testing Accuracy
Performance on unseen data.

test_accuracy =
[Link](X_test,y_test)

A good model should achieve similar training and testing accuracies.

31
IPH2445: Machine Learning using Python – III Chapter 1

5.10 Underfitting
Definition
Underfitting occurs when a model is too simple to capture the underlying patterns
in the data.

Characteristics:
• Low training accuracy
• Low testing accuracy
• Poor learning performance
Example:
Training Accuracy = 60%
Testing Accuracy = 58%
The model has not learned enough from the data.

5.11 Overfitting
Definition
Overfitting occurs when a model learns the training data too well, including noise
and random fluctuations.

Characteristics:
• Very high training accuracy
• Significantly lower testing accuracy
• Poor generalization
Example:
Training Accuracy = 99%
Testing Accuracy = 75%
The model memorizes training examples instead of learning general patterns.

5.12 Illustration of Overfitting

Suppose a student memorizes answers to previous examinations.
The student performs exceptionally well on known questions but struggles with new
questions.
Similarly, an overfitted machine learning model performs well on training data but
poorly on unseen data.

Key Point
A model should learn patterns rather than memorize data.

32
IPH2445: Machine Learning using Python – III Chapter 1

5.13 Good Model Performance

An ideal model should exhibit:

• High training accuracy

• High testing accuracy

• Small difference between training and testing performance

Example:

Training Accuracy = 95%

Testing Accuracy = 93%

Such a model is likely to generalize well.

5.14 Methods to Reduce Overfitting

Several approaches can reduce overfitting:

1. Increase training data.

2. Remove irrelevant features.

3. Use regularization techniques.

4. Perform cross-validation.

5. Simplify the model.

For Logistic Regression, regularization is commonly used.

5.15 Example: Evaluating a Logistic Regression Model

from [Link] import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

iris = load_iris()

X = [Link]
y = [Link]

X_train, X_test, y_train, y_test =

train_test_split(
X,
y,
test_size=0.2,
random_state=42)

model = LogisticRegression(max_iter=200)

33
IPH2445: Machine Learning using Python – III Chapter 1

[Link](X_train,y_train)

train_acc =
[Link](X_train,y_train)

test_acc =
[Link](X_test,y_test)

print("Training Accuracy =",train_acc)

print("Testing Accuracy =",test_acc)

Sample Output:

Training Accuracy = 0.975

Testing Accuracy = 0.967

The small difference indicates good generalization.

5.16 Summary
• Model evaluation measures the effectiveness of machine learning models.

• Training and testing datasets are used to assess performance fairly.

• Accuracy measures the proportion of correct predictions.

• Training accuracy measures learning performance.

• Testing accuracy measures generalization ability.

• Underfitting occurs when the model is too simple.

• Overfitting occurs when the model memorizes training data.

• A good classifier performs well on both training and testing datasets.

5.17 Review Questions

Short Answer Questions
1. Why is model evaluation important?

2. Define train-test split.

3. What is classification accuracy?

4. Write the formula for accuracy.

5. What is training accuracy?

6. What is testing accuracy?

34
IPH2445: Machine Learning using Python – III Chapter 1

7. Define underfitting.

8. Define overfitting.

9. What causes overfitting?

10. State two methods for reducing overfitting.

Descriptive Questions
1. Explain the importance of model evaluation.

2. Discuss the concept of train-test splitting.

3. Explain how classification accuracy is calculated.

4. Compare underfitting and overfitting.

5. Discuss methods used to improve model generalization.

Application-Oriented Questions
1. A classifier correctly predicts 180 out of 200 observations. Calculate its accuracy.

2. A model achieves 99% training accuracy and 72% testing accuracy. Explain the
likely problem.

3. Design an experiment to evaluate a Logistic Regression classifier using train-test

splitting.

4. Compare two classifiers with testing accuracies of 85% and 92% and discuss which
should be preferred.

5. Explain how increasing training data can reduce overfitting in a classification prob-
lem.

35