BRAINWARE UNIVERSITY

[PCC-CSM601] [Machine Learning and its Applications]

MODULE- 4 (CLASS NOTES)

Model Selection and Evaluation in Machine Learning

1. What is Model Selection?

Model selection is the process of choosing the most appropriate machine learning model for a given problem by
systematically comparing different algorithms and configurations. It involves evaluating models based on:

• Problem type: Classification, regression, clustering, etc.

• Data characteristics: Size, dimensionality, noise level, and feature types.
• Performance metrics: Accuracy, precision, recall, F1-score, RMSE, etc.
• Model behavior: Underfitting vs. overfitting tendencies.
• Practical considerations: Computational efficiency, scalability, and interpretability.

The goal is to identify the model that not only performs best on both training and unseen data but also aligns with
real-world constraints and project goals. Model selection is critical to ensuring that the deployed solution is both
effective and reliable.

2. Importance of Model Selection in Machine Learning

Model selection is a fundamental step in building reliable and high-performing machine learning systems. It
involves identifying the most appropriate algorithm based on the problem type, data characteristics, and learning
objectives. Choosing the right model directly affects the system’s ability to learn patterns, generalize to unseen
data, and operate efficiently within resource constraints.

1. Accuracy and Predictive Performance

Each ML model has its strengths depending on data structure and the type of problem:

• Linear models work well with linearly separable data.

• Tree-based models handle non-linear relationships and categorical variables.
• Support Vector Machines (SVMs) perform well in high-dimensional spaces.

Selecting the right model improves predictive accuracy and minimizes errors.

2. Training Efficiency and Scalability

Models vary in computational complexity:

• Naive Bayes, k-NN, and Logistic Regression train quickly on smaller datasets.
• Ensemble methods and deep learning models are powerful but resource-intensive.

An efficient model ensures faster training, better handling of large datasets, and quicker inference during
deployment.

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY

3. Interpretability

In many ML tasks, understanding how the model makes decisions is essential for debugging and feature analysis:

• Simple models (e.g., Decision Trees, Linear Regression) offer transparency.

• Complex models (e.g., Random Forests, Neural Networks) may require tools like SHAP or LIME for
interpretation.

Model selection helps balance performance with explainability.

4. Suitability to Data Type

Different ML algorithms are better suited to different data formats:

• Time-series models handle temporal patterns.

• Distance-based models work well with continuous features.
• Probabilistic models are suitable when uncertainty modeling is needed.

The right model aligns with the structure and distribution of the training data.

5. Computational Resource Constraints

Machine learning often involves large datasets and iterative training. Resource-efficient models are crucial when:

• Hardware limitations exist (e.g., low-memory systems).

• Training must be repeated frequently with new data (e.g., online learning). Choosing the right model ensures
feasibility within resource boundaries.

6. Generalization vs. Overfitting

A good model captures patterns in data without memorizing noise:

• Underfitting occurs with overly simple models.

• Overfitting occurs with overly complex models.

Model selection involves identifying models with the right bias-variance trade-off to ensure robust performance
on unseen data.

7. Adaptability to Evolving Data

In real-world machine learning pipelines, data distributions may shift over time:

• Incremental learning models or those that support retraining with partial data are better for dynamic
environments.
• Rigid models may degrade in performance if data evolves.

Selecting adaptable models ensures long-term relevance.

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)
 BRAINWARE UNIVERSITY

8. Cost and Development Time

Some models require:

• Manual feature engineering

• Extensive hyperparameter tuning
• Longer training cycles

Simple yet effective models reduce development effort and tuning cycles, speeding up experimentation and
deployment.

3. Procedure of Model Selection in Machine Learning

1. Problem Understanding

• Clearly define the objective (e.g., classification, regression, clustering).

• Identify the input and output variables.
• Determine the success criteria using appropriate performance metrics (e.g., accuracy, MSE, F1-score).

2. Data Preprocessing

• Data Cleaning: Handle missing values, outliers, and noise.

• Feature Engineering: Create meaningful features, encode categorical variables.
• Data Scaling: Normalize or standardize data if required by the algorithm.
• Splitting: Divide data into training, validation, and test sets.

3. Choose Algorithms Based on Problem Type

• For classification: Logistic Regression, Decision Trees, Random Forest, SVM, KNN, Neural Networks.
• For regression: Linear Regression, Ridge, Lasso, SVR, Gradient Boosting.
• For clustering: K-means, DBSCAN, Hierarchical Clustering.
• Consider algorithm suitability based on data type, size, linearity, and interpretability needs.

4. Model Training with Different Algorithms

• Train multiple candidate models on the training dataset.

• Use consistent data splits and pipelines to ensure fair comparison.
• Record training time and convergence behavior.

5. Performance Evaluation Using Cross-Validation

• Use k-fold cross-validation to assess the generalization of each model.

• Evaluate models based on selected metrics.
• Observe performance consistency across folds to detect overfitting or instability.

6. Hyperparameter Tuning

• Use techniques like Grid Search, Random Search, or Bayesian Optimization.

• Fine-tune model parameters (e.g., depth of tree, number of estimators, regularization strength).
• Optimize for validation performance, not test set.

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY
7. Model Comparison and Selection

• Compare models based on:

• Cross-validation scores.
• Simplicity and interpretability.
• Training time and resource usage.

• Select the model that best balances accuracy, efficiency, and generalization.

8. Final Testing on Unseen Data

• Evaluate the selected model on the test dataset (data not seen during training or validation).
• Confirm real-world performance.
• Analyze errors and consider retraining if needed.

4. Theory on Evaluating Indices in Machine Learning

Evaluating indices (also called performance metrics or evaluation metrics) are used to assess how well a machine
learning model performs on a given task. The choice of evaluation metric depends on the type of learning problem—
whether it's classification, regression, clustering, etc. These indices provide a quantitative basis for model comparison,
selection, and optimization.

4.1 Evaluation Metrics for Linear Regression Models (MSE, MAE, RMSE)
A. Mean Squared Error (MSE)

Definition:
Mean Squared Error (MSE) calculates the average of the squares of the errors between the actual and predicted
values. It measures the quality of an estimator by comparing predicted values (ŷ) with actual values (y).

Formula:
MSE = (1/n) * Σ(y_i - ŷ_i)^2

Example:
Let actual values y = [3, -0.5, 2, 7] and predicted values ŷ = [2.5, 0.0, 2, 8].
Errors = [0.5, -0.5, 0, -1]
Squared Errors = [0.25, 0.25, 0, 1]
MSE = (0.25 + 0.25 + 0 + 1)/4 = 0.375

Characteristics:
- Penalizes larger errors more due to squaring.
- Sensitive to outliers.
- Good when large errors are undesirable.

Advantages:
- Differentiable, making it suitable for optimization algorithms like gradient descent.
- Amplifies large errors, making it useful when large deviations are especially problematic.

Disadvantages:
- Not robust to outliers due to squaring.
- The unit of error is the square of the original unit, which can reduce interpretability.

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY
B. Mean Absolute Error (MAE)

Definition:
Mean Absolute Error (MAE) measures the average magnitude of the absolute errors between actual and predicted
values. Unlike MSE, it does not square the errors.

Formula:
MAE = (1/n) * Σ|y_i - ŷ_i|

Example:
Using the same values y = [3, -0.5, 2, 7] and ŷ = [2.5, 0.0, 2, 8],
Absolute Errors = [0.5, 0.5, 0, 1]
MAE = (0.5 + 0.5 + 0 + 1)/4 = 0.5

Characteristics:
- Treats all errors equally.
- More robust to outliers compared to MSE.
- Error is in the same unit as the target variable.

Advantages:
- Interpretable since the error is in the original unit.
- Robust to outliers, making it useful when data may contain anomalies.

Disadvantages:
- Not differentiable at zero, which may pose challenges in optimization.
- Does not emphasize large errors like MSE does.

C. Root Mean Squared Error (RMSE)

Definition:
Root Mean Squared Error (RMSE) is the square root of the Mean Squared Error. It combines the benefits of MSE
with the interpretability of MAE.

Formula:
RMSE = √MSE

Example:
Using the MSE = 0.375, RMSE = √0.375 ≈ 0.612

Characteristics:
-Provides error in the same unit as the target variable.
- Penalizes large errors due to squaring before taking the root.

Advantages:
- Combines sensitivity to large errors with output in the original unit.
- Often used when both interpretability and penalization of large errors are required.

Disadvantages:
- Sensitive to outliers like MSE.
- Can be harder to compute than MAE in certain environments.

4.2 R-squared (R²) and Adjusted R-squared in Regression Analysis

1. R-squared (R²) – Coefficient of Determination
2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)
 BRAINWARE UNIVERSITY
Definition: R-squared represents the proportion of the variance in the dependent variable that is predictable from the
independent variables.

Formula:
R² = 1 - (SS_res / SS_tot)
Where:
• SS_res = Sum of Squares of Residuals = Σ(y_i - ŷ_i)²
• SS_tot = Total Sum of Squares = Σ(y_i - ȳ)²
• y_i = Actual values
• ŷ_i = Predicted values
• ȳ = Mean of actual values

Interpretation:

• R² = 1: Perfect prediction.
• R² = 0: Model explains none of the variance.
• R² < 0: Model performs worse than using the mean.
Example: If R² = 0.85, then 85% of the variation in the dependent variable is explained by the model.

2. Adjusted R-squared

Why Needed: R-squared increases with more features, even if they are not useful. Adjusted R-squared penalizes the
addition of irrelevant features.
Formula:
Adjusted R² = 1 - [(1 - R²) * (n - 1) / (n - k - 1)]
Where:
• n = number of observations
• k = number of independent variables
• R² = R-squared value

Interpretation:

• Adjusted R² increases only when the new variable improves the model.
• It helps in feature selection by discouraging overfitting.

Key Differences Between R² and Adjusted R²

Feature R-squared (R²) Adjusted R-squared

Measures Explained variance Explained variance adjusted for number
of predictors
Affected by extra features Always increases or stays Increases only if new variable improves
same model
Ideal for Initial performance check Comparing models with different
numbers of variables
Value Range 0 to 1 (can be negative) Can be less than R², but not more

When to Use Which?

• Use R² to understand how well your model explains the variation in the data.
• Use Adjusted R² when comparing models with different numbers of predictors to avoid overfitting.

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY
4.3 Evaluation Metrics for Logistic Regression Models

Performance metrics are essential for assessing the effectiveness of machine learning models, particularly in
classification tasks. These metrics help evaluate how well the model distinguishes between classes and identifies errors
in predictions. Depending on the nature of the problem and the dataset (balanced or imbalanced), different metrics can
provide insights into different aspects of model performance.

Accuracy:

Accuracy is one of the most common and simplest metrics used for classification models. It measures the overall
percentage of correct predictions made by the model. Accuracy is calculated as the ratio of correctly predicted
instances (both true positives and true negatives) to the total number of instances:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

While accuracy is a useful metric in many scenarios, it can be misleading, especially for imbalanced datasets.

Precision:

Precision, also known as positive predictive value, measures the proportion of true positive predictions among all
instances predicted as positive by the model:

Precision = TP / (TP + FP)

Precision is important when the cost of false positives is high, such as in medical diagnostics.

Recall (Sensitivity or True Positive Rate):

Recall measures the proportion of actual positive instances that were correctly predicted as positive:

Recall = TP / (TP + FN)

Recall is crucial when the cost of false negatives is high, such as in fraud detection or disease diagnosis.

F1-Score:

F1-Score is the harmonic mean of precision and recall, providing a balance between them:

F1-Score = 2 × (Precision × Recall) / (Precision + Recall)

It is especially useful in scenarios with imbalanced classes.

Confusion Matrix:

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY
A Confusion Matrix is a table that visualizes the performance of a classification model. It shows the counts of true
positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). It helps in calculating various
performance metrics and identifying where the model makes errors.

Log-Loss (Logarithmic Loss):

Log-Loss measures the performance based on predicted probabilities:

Log-Loss = -1/N * Σ[y_i * log(ŷ_i) + (1 - y_i) * log(1 - ŷ_i)]

Lower values indicate better performance. It penalizes incorrect predictions with higher confidence more.

Specificity (True Negative Rate):

Specificity measures the proportion of actual negatives that were correctly identified:

Specificity = TN / (TN + FP)

It is important when false positives have significant costs.

ROC Curve:

The ROC Curve is a graphical representation of model performance across thresholds. It plots:
- True Positive Rate (TPR or Recall) on the Y-axis
- False Positive Rate (FPR) on the X-axis

FPR = FP / (FP + TN)

An ideal curve bows to the top-left corner. It helps to assess the trade-off between sensitivity and specificity.

AUC (Area Under the Curve):

AUC quantifies the overall ability of the model to discriminate between classes:

- AUC = 1: Perfect classification

- AUC = 0.5: No discriminatory power (random guessing)
- AUC < 0.5: Worse than random

A higher AUC indicates better overall performance and is useful for imbalanced datasets.

The Receiver Operating Characteristic (ROC) curve is a fundamental tool for evaluating the performance of
classification models, especially in binary classification tasks like logistic regression. The ROC curve is a graphical
representation that illustrates the capability of a model to distinguish between the two classes as the discrimination
threshold varies. Rather than using a single threshold (like 0.5) to determine class labels from predicted probabilities,
ROC curves evaluate how the true positive rate (sensitivity) and false positive rate change across all possible thresholds.
This makes it a powerful technique for assessing model performance independent of any particular threshold.
On the ROC graph, the x-axis represents the False Positive Rate (FPR), which is the ratio of actual negatives incorrectly
predicted as positives, calculated as:
FPR = FP / (FP + TN)
The y-axis represents the True Positive Rate (TPR) or Recall, which measures the ratio of actual positives correctly
predicted as positives, calculated as:
TPR = TP / (TP + FN)
By varying the threshold from 0 to 1, we get different combinations of TPR and FPR, and each point on the ROC

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY
curve corresponds to one such threshold.

A model that performs well will have a ROC curve that bows towards the top-left corner of the plot, indicating a high
TPR and low FPR. A model with no discrimination power (e.g., random guessing) will have a ROC curve along the
diagonal line from (0,0) to (1,1), where the TPR equals the FPR at all threshold levels. In contrast, a model whose ROC
curve is below the diagonal may be performing worse than random, indicating possible issues like label inversion or
model flaws.

The Area Under the Curve (AUC) is a scalar value that quantifies the overall performance of the model based on the
ROC curve. It calculates the area beneath the ROC curve and provides a single metric to summarize the model's ability
to rank positive instances higher than negative ones. The AUC value ranges from 0 to 1, where a value of 1.0 indicates
a perfect model, and 0.5 corresponds to a model with no discrimination capacity (i.e., random guessing). An AUC less
than 0.5 suggests that the model may be consistently predicting in the wrong direction.
AUC is particularly useful in situations where there is class imbalance, or when the cost of false positives and false
negatives differs significantly. Since it aggregates performance across all classification thresholds, it is not influenced
by the threshold chosen for decision-making, unlike metrics such as accuracy or precision. Thus, AUC-ROC is often
used to compare multiple classifiers, especially during model selection and evaluation phases.
In practical applications, such as medical diagnostics, fraud detection, or spam filtering, the ROC and AUC provide
intuitive and threshold-independent insights into a model's behavior. For example, a medical test with high TPR ensures
that most sick patients are identified, but if its FPR is also high, it may wrongly label many healthy individuals as ill.
By analyzing the ROC curve, practitioners can balance sensitivity and specificity, choosing an optimal threshold suited
to the real-world constraints of the problem domain.

4.4 Hyperparameters in Decision Trees

Hyperparameters are the configuration settings used to control the learning process of machine learning models. In
Decision Trees, hyperparameters are values set before training that govern the structure and behavior of the tree.
Unlike parameters (such as feature split thresholds), which are learned from data, hyperparameters are manually set
and directly influence how the model learns patterns. Key hyperparameters in decision trees include aspects like the
maximum depth of the tree, the minimum number of samples required to split a node, and the criterion used for
measuring the quality of a split (such as Gini impurity or information gain). For example, setting a high maximum
depth may allow the tree to fit the training data very closely, but may lead to overfitting and poor generalization to
unseen data. On the other hand, limiting the depth can simplify the model and improve its robustness.

Why Tune Hyperparameters in Decision Trees?

Tuning hyperparameters is essential to improve the performance and generalization ability of decision trees. A poorly
configured tree may either overfit the data (capturing noise and fluctuations in the training set) or underfit it (failing
to capture important patterns). For instance, if a decision tree is allowed to grow without constraints, it may become
too deep and specific, resulting in excellent training accuracy but low test accuracy. Conversely, a very shallow tree
might miss critical decision paths, resulting in poor accuracy even on the training data. By carefully tuning
hyperparameters, we aim to find a sweet spot that achieves good performance on both the training and unseen data.
Moreover, hyperparameter tuning helps to optimize computational efficiency and ensures that the model remains
interpretable and scalable.

Types of Hyperparameters in Decision Trees

There are several important hyperparameters that can be configured in a decision tree model. Below are the most
commonly used ones:

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY
a. Criterion: This defines how the quality of a split is measured. Two common criteria are Gini Impurity and
Information Gain (Entropy). For classification tasks, Gini tends to be faster, while Entropy is more informative
but computationally more expensive. For example, if we are classifying flowers based on petal size, the
criterion determines how we choose the best feature to split the data.
b. Max Depth: It limits the depth of the tree. A deeper tree can capture more detail but may overfit. For example,
setting a max depth of 3 forces the tree to make decisions within three layers, thus simplifying the model.
c. Min Samples Split: This is the minimum number of samples required to split an internal node. If set too high,
the tree might not split when it should, causing underfitting. For instance, if the minimum is 20, a node must
contain at least 20 samples to consider a further split.
d. Min Samples Leaf: Specifies the minimum number of samples required to be in a leaf node. This prevents the
tree from creating leaves with very few samples, which are likely to be noisy.
e. Max Features: Controls the number of features considered when looking for the best split. For example, if you
have 10 features but set max_features to 5, only 5 features will be randomly selected for consideration at each
split, introducing variability and reducing overfitting.
f. Min Weight Fraction Leaf: Defines the minimum weighted fraction of the sum total of weights required to
be at a leaf node, helpful for datasets with imbalanced sample weights.

These hyperparameters must be balanced based on the complexity of the dataset, number of samples, and the task at
hand (classification or regression).

Methods for Hyperparameter Tuning in Decision Tree

There are several systematic approaches to hyperparameter tuning. These methods help identify the most suitable
values by evaluating different combinations of hyperparameters.

a) Grid Search

Grid Search is a brute-force method that exhaustively evaluates all possible combinations of specified hyperparameter
values. For example, if we have 3 values for max_depth and 3 values for min_samples_split, Grid Search will train and
evaluate the model on all 9 combinations using cross-validation. Although computationally expensive, it guarantees
that the best parameters (from the provided range) will be found. However, it can be impractical when the parameter
space is large.

b) Randomized Search

Randomized Search, instead of trying all combinations, samples a fixed number of random combinations from the
parameter space. This method is more efficient in terms of time and often produces results comparable to Grid Search.
For example, instead of testing all 100 combinations, we might randomly test 20 of them and find a good enough set
of hyperparameters.

c) Bayesian Optimization

Bayesian Optimization builds a probabilistic model of the objective function and uses it to select promising
hyperparameter values based on past evaluation results. It is more intelligent than random or grid search and can
converge on the optimal parameters faster. For example, after a few iterations, the algorithm may realize that max_depth
values above 10 do not improve performance and avoid testing them further. Libraries like Optuna, Hyperopt, and
Scikit-Optimize support this approach.

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY
Pruning in Decision Trees

Pruning is an essential technique in decision tree learning used to improve model performance by reducing
its complexity. During the construction of decision trees, excessive splitting can lead to overfitting, where
the tree perfectly classifies training data but fails to generalize well to new, unseen data. Pruning helps address
this issue by cutting back branches or nodes that have little predictive value, thereby simplifying the tree and
improving its ability to generalize.

There are two primary types of pruning: pre-pruning and post-pruning. Pre-pruning involves halting the tree's
growth early, based on predefined conditions such as maximum depth, minimum samples per split, or a
threshold for impurity reduction. This strategy aims to prevent the tree from becoming overly complex in the
first place. However, it runs the risk of underfitting, as it may stop the tree from capturing important patterns
in the data. On the other hand, post-pruning involves growing a full tree and then removing branches or nodes
that do not contribute significantly to the model's accuracy. This is typically done using a cost-complexity
measure, where a penalty is applied for tree complexity.

In practice, pre-pruning helps control tree size during training, which is particularly useful for large datasets
with noisy data, whereas post-pruning is more effective when dealing with complex trees prone to overfitting.
Both methods are aimed at finding the optimal balance between simplicity and accuracy. Pre- pruning is more
computationally efficient, while post-pruning requires additional computation but can result in a more refined
model. By using these pruning strategies, decision trees can become more interpretable and perform better
on unseen data.

Implementing Hyperparameter Tuning in a Decision Tree

The process of hyperparameter tuning in Decision Trees follows a systematic workflow:

i. Define the Problem and Prepare Data: Before tuning, ensure the data is clean and split into training and
validation (or test) sets. The nature of the data (e.g., size, number of features, imbalance) influences the tuning
strategy.
ii. Choose Hyperparameters to Tune: Based on the problem complexity, choose a subset of key
hyperparameters. For classification problems, max_depth, min_samples_split, and criterion are commonly
tuned. For regression trees, similar parameters apply.
iii. Select Tuning Method: Depending on resource availability and dataset size, choose between Grid Search,
Random Search, or Bayesian Optimization.
iv. Run Cross-Validation: Use cross-validation to evaluate each hyperparameter combination. This ensures that
the model's performance is generalized and not biased by a single train-test split.
v. Analyze Results and Choose Best Parameters: After testing, analyze the results based on evaluation metrics
(e.g., accuracy, F1-score). The best parameters are those that perform well across all folds in cross- validation.
vi. Train Final Model: Retrain the Decision Tree using the entire training data and the best-found
hyperparameters. Finally, test the model on a separate test set to evaluate its real-world performance.

Example: Suppose we are building a Decision Tree to classify whether an email is spam or not. We might begin by
testing max_depth values of 5, 10, and 15, and min_samples_leaf values of 1 and 3. By running cross-validation for all
combinations, we may discover that max_depth=10 and min_samples_leaf=3 offer the best performance. We then use
these settings to train our final model.

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)

 BRAINWARE UNIVERSITY

5. Model Validation and Deployment in Machine Learning

5.1 Model Validation:
Model validation is a crucial step in the machine learning workflow, as it helps assess how well a model generalizes to
new, unseen data. The purpose of model validation is to ensure that the model does not just perform well on the training
data but also has the ability to make accurate predictions on unseen data, thus avoiding overfitting. Overfitting occurs
when a model memorizes the training data and performs poorly on new data, while underfitting happens when the
model is too simple to capture the underlying patterns. Common methods for model validation include train-test split,
cross-validation, and bootstrapping.

i. Train-Test Split: This is the simplest approach to validation, where the dataset is divided into two sets: a
training set and a test set. The model is trained on the training set and evaluated on the test set to see how well
it generalizes to new data. A typical split is 80-20 or 70-30, where 80% of the data is used for training and 20%
for testing.
ii. Cross-Validation: Cross-validation, particularly k-fold cross-validation, involves splitting the dataset into k
equally sized folds. The model is trained on k-1 folds and validated on the remaining fold. This process is
repeated k times, each time using a different fold for validation, and the final performance is averaged over all
k runs. Cross-validation provides a more reliable estimate of model performance as it evaluates the model on
multiple subsets of the data.
iii. Performance Metrics: To evaluate the model's performance during validation, various metrics can be used
depending on the type of problem (classification or regression). For classification tasks, common metrics
include accuracy, precision, recall, F1-score, and ROC-AUC. For regression tasks, metrics like mean
squared error (MSE), mean absolute error (MAE), and R-squared are often employed.

5.2 Model Deployment:

Once the model is validated and its performance is satisfactory, the next step is model deployment. Deployment
involves integrating the trained model into a production environment where it can make predictions on new,
incoming data. The deployment process ensures that the model is not only accurate but also scalable, reliable, and
easy to update when necessary. There are several strategies for deploying machine learning models:

• Batch Deployment: In batch deployment, the model is used to make predictions on a large set of data at
once. This method is typically used when real-time predictions are not necessary. For example, in predictive
maintenance, the model can run periodically to forecast potential equipment failures over a month or quarter.
• Real-Time Deployment: In real-time deployment, the model makes predictions instantly as new data comes
in. This approach is used in applications such as fraud detection in banking, where predictions need to be
made in real-time as transactions occur.
• Model as a Service (MaaS): One popular deployment method is exposing the model via a web service
(usually using an API). This allows the model to be accessed remotely by other applications or services for
making predictions. Frameworks such as Flask, FastAPI, or TensorFlow Serving are commonly used to
deploy models as APIs, allowing them to scale across different systems.
• Monitoring and Maintenance: After deployment, it is essential to continuously monitor the model’s
performance to ensure it maintains accuracy over time.

2025-26 Prepared by: Department of CSE-AI (Brainware University, Barasat)