Machine Learning

Thien Huynh-The
Department of Computer and Communications Engineering
HCMC University of Technology and Education

December 26, 2025

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 1 / 78

AI, ML, and DL: Definitions
• Artificial Intelligence (AI): Machines performing tasks requiring human intelligence. Encompasses
rule-based systems and advanced machine learning.
• Machine Learning (ML): A subfield of AI. Systems learn from data without explicit programming.
Algorithms learn patterns and make predictions. Examples: Linear Regression, SVMs.
• Deep Learning (DL): A subfield of ML using deep neural networks (multiple layers) to analyze data and
learn complex patterns. Examples: CNNs, RNNs.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 2 / 78

Machine Learning: Learning from Experience

• Definition: A computer program is said to learn from experience E with respect to some
class of tasks T and performance measure P, if its performance at tasks in T, as measured
by P, improves with experience E.
• Example: Email Spam Filtering
• T: Classifying emails as “spam” or “not spam.”
• P: Accuracy in classifying emails.
• E: A dataset of emails labeled as “spam” or “not spam.” The ML system learns patterns
from this labeled data to improve its spam detection accuracy.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 3 / 78

Learning from Experience: Task

T (Tasks): The specific problems the ML system is designed to solve. Examples:

• Classification: Assigning an input to one of several predefined categories (e.g., spam
filtering, image classification).
• Regression: Predicting a continuous value (e.g., house prices, stock prices).
• Clustering: Grouping similar inputs together (e.g., customer segmentation).

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 4 / 78

Tasks: Classification, Regression, and Clustering
Feature Classification Regression Clustering
Task Assign an input to one of a set Predict a continuous numerical Group similar inputs to-
of pre-defined categories. value. gether into clusters.
Output Discrete class label (e.g., Continuous numerical value Assignment of each input to
“spam,” “not spam,” “cat,” (e.g., house price, temperature). a specific cluster.
“dog”).
Training Data Labeled data (input-output Labeled data (input-output Unlabeled data (inputs
pairs). pairs). only).
Goal Maximize the accuracy of classi- Minimize the error between pre- Maximize similarity within
fication. dicted and actual values. clusters and minimize simi-
larity between clusters.
Examples Email spam detection, image House price prediction, stock Customer segmentation,
classification (cat vs. dog), price forecasting, temperature document categorization,
medical diagnosis. prediction. anomaly detection.
Algorithms Logistic Regression, Support Linear Regression, Polynomial K-Means, DBSCAN, Hierar-
Vector Machines (SVMs), De- Regression, Support Vector Re- chical Clustering, Gaussian
cision Trees, Random Forests, gression (SVR), Neural Net- Mixture Models (GMMs).
Naive Bayes. works.

Table: Comparison of ML Tasks

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 5 / 78
Tasks: Classification, Regression, and Clustering

Key Difference: Data Labels and Learning Paradigm

• Supervised Learning:
• Uses labeled data (input-output pairs).
• Learns a mapping from inputs to outputs to predict new outputs.
• Examples:
• Classification (e.g., spam detection).
• Regression (e.g., house price prediction).
• Unsupervised Learning:
• Uses unlabeled data (inputs only).
• Finds patterns or structure in the data (e.g., clustering).
• Example: Customer segmentation.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 6 / 78

Learning from Experience: Experience

The “experience” in machine learning refers to the data used to train the model. The nature
and quality of this data are crucial for the model’s performance.
Data Types:
• Labeled Data: Each data point is associated with a corresponding output or label. Used
in supervised learning.
• Examples: (input, output) pairs like (image, “cat”), (email, “spam”), (house features, price).
• Use Cases: Classification, Regression.
• Unlabeled Data: Data points without any associated outputs or labels. Used in
unsupervised learning.
• Examples: A collection of images without labels, customer purchase histories.
• Use Cases: Clustering, dimensionality reduction, anomaly detection.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 7 / 78

Learning from Experience: Experience (cont

Data Characteristics: The quality of the experience (data) significantly impacts the learning
process. Important characteristics include:
• Quantity: More data generally leads to better performance, especially for complex models.
• Quality: Clean, accurate, and consistent data is essential. Noisy or erroneous data can hinder learning.
• Relevance: The data should be relevant to the task being learned. Irrelevant data can confuse the model.
• Bias: Data can contain biases that reflect existing societal or other prejudices. These biases can be
learned by the model, leading to unfair or inaccurate predictions.
• Representation: The data should be representative of the real-world scenarios the model will encounter. If
the training data is not representative, the model may not generalize well to new data.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 8 / 78

Learning from Experience: Performance

P (Performance Measure): A metric used to evaluate how well the system performs on the
tasks. Examples:
• Accuracy: The proportion of correctly classified instances.
• Precision: The proportion of true positives among the predicted positives.
• Recall: The proportion of true positives among the actual positives.
• Mean Squared Error (MSE): Average squared difference between predicted and actual values (for
regression).

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 9 / 78

Performance Metrics for Classification
Confusion Matrix: A table summarizing the performance of a classification model.

Predicted
Actual Positive Negative
Positive True Positive (TP) False Negative (FN)
Negative False Positive (FP) True Negative (TN)

• Accuracy: Fraction of correctly classified instances.

TP + TN
Accuracy =
TP + TN + FP + FN
• Precision: Fraction of true positives among predicted positives.

TP
Precision =
TP + FP

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 10 / 78

Performance Metrics for Classification (cont)
Confusion Matrix: A table summarizing the performance of a classification model.

Predicted
Actual Positive Negative
Positive True Positive (TP) False Negative (FN)
Negative False Positive (FP) True Negative (TN)

• Recall (Sensitivity): Fraction of true positives among actual positives.

TP
Recall =
TP + FN
• F1-Score: Harmonic mean of precision and recall.

2 × Precision × Recall
F1-Score =
Precision + Recall

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 11 / 78

Performance Metrics for Regression
• Mean Squared Error (MSE): Average squared difference between predicted and actual values.
n
1X
MSE = (yi − ŷi )2
n i=1
where yi is the actual value, ŷi is the predicted value, and n is the number of data points.
• Root Mean Squared Error (RMSE): Square root of MSE.
√
RMSE = MSE
• Mean Absolute Error (MAE): Average absolute difference between predicted and actual values.
n
1X
MAE = |yi − ŷi |
n i=1
• R-squared (Coefficient of Determination): Represents the proportion of variance in the dependent
variable that is predictable from the independent variables.
Pn
(yi − ŷi )2
R 2 = 1 − Pi=1
n
i=1 (yi − ȳ )
2

where ȳ is the mean of the actual values.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 12 / 78
Overview of a ML Framework

Training Phase:
• Data Collection: Gathering raw training data.
• Feature Extraction/Engineering: Transforming
raw data into meaningful features (Xtrain ). Prior
knowledge can be incorporated.
• Model Training: Applying a learning algorithm
(Classification, Regression, Clustering, etc.) to
Xtrain and ytrain to train a model.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 13 / 78

Overview of a ML Framework

Test Phase:
• Data Collection: Gathering raw test data.
• Feature Extraction/Engineering: Applying the
same feature extraction to raw test data to get
Xtest .
• Model Evaluation/Prediction: Using the trained
model to predict on Xtest and get ytest .

Key Considerations:
• Feature engineering is crucial.
• Test data should be representative.
• Model evaluation is essential.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 14 / 78

Feature Engineering: Introduction

What is Feature Engineering?

Feature engineering is the process of transforming raw data into features that better represent
the underlying problem to the predictive models, resulting in improved model accuracy on
unseen data. It is a crucial step in machine learning, often more impactful than the choice of
model itself.
• Why is it important?
• Improves model performance.
• Simplifies model complexity.
• Enhances model interpretability.
• Two Main Components:
• Feature Extraction
• Feature Selection

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 15 / 78

Feature Extraction

Feature Extraction: Creating New Features

Feature extraction involves creating new features from raw data. This can involve:
• Manual Feature Engineering: Using domain expertise to create features.
• Automated Feature Extraction: Using algorithms to automatically generate features.
• Examples:
• Text Data: Bag-of-words, TF-IDF, word embeddings.
• Image Data: Edge detection, SIFT, HOG, CNN features.
• Time Series Data: Rolling averages, Fourier transforms, autocorrelation.
• Dimensionality Reduction: Feature extraction can also be used to reduce the
dimensionality of the data while preserving important information (e.g., PCA).

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 16 / 78

Feature Extraction: Time Domain Examples
Time domain features are calculated directly from the raw time series data. They capture
characteristics of the signal’s amplitude and variation over time.
• Mean: Average value of the signal.
N
1 X
Mean = xi
N i=1
• Standard Deviation (Std Dev): Measure of the signal’s dispersion around the mean.
v
u
u 1 X N
Std Dev = t (xi − Mean)2
N − 1 i=1

• Root Mean Square (RMS): Measure of the signal’s magnitude.

v
u
u1 X N
RMS = t x2
N i=1 i

• Zero-Crossing Rate: Number of times the signal crosses zero.

• Peak-to-Peak Amplitude: Difference between the maximum and minimum values of the signal.
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 17 / 78
Feature Extraction: Frequency Domain Examples
Frequency domain features are calculated after transforming the signal into the frequency
domain using techniques like the Fourier Transform, reveal its frequency content.
• Discrete Fourier Transform (DFT): Decomposes the signal into its constituent frequencies.
N−1
X
Xk = xn e −j2πkn/N
n=0

where xn is the time domain signal, Xk is the frequency domain representation, and j is the imaginary unit.
• Power Spectral Density (PSD): Describes how the power of a signal is distributed across different
frequencies.
|Xk |2
PSD =
N
• Dominant Frequency: The frequency with the highest power in the PSD.
• Spectral Centroid: The “center of mass” of the signal’s power spectrum (where fk are the frequencies).
PN/2
fk |Xk |2
Spectral Centroid = Pk=1 N/2
k=1 |Xk |
2

• Spectral Bandwidth: Measure of the width of the frequency band occupied by the signal.
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 18 / 78
Feature Selection
Feature Selection: Choosing the Best Features
Feature selection involves selecting a subset of the most relevant features from the existing set
of features. This can help to:
• Improve model performance by reducing overfitting.
• Simplify the model, making it faster and easier to interpret.
• Reduce computational cost.
• Methods:
• Filter Methods: Select features based on statistical measures (e.g., correlation, chi-squared
test).
• Wrapper Methods: Evaluate subsets of features by training a model on them (e.g., recursive
feature elimination).
• Embedded Methods: Feature selection is done as part of the model training process (e.g.,
LASSO regularization).

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 19 / 78

Feature Selection: Filter Methods

Filter methods select features based on statistical measures calculated from the data. They
evaluate each feature independently of the chosen learning algorithm.
• Key Characteristics:
• Evaluate features independently.
• Computationally efficient.
• Do not consider the interaction with the learning algorithm.
• Prone to selecting redundant features.
• Common Metrics/Methods:
• Correlation: Measures the linear relationship between two variables.

Cov(X , Y )
Correlation(X , Y ) =
σX σY
where Cov(X , Y ) is the covariance between X and Y , and σX and σY are the standard deviations of
X and Y , respectively. Select features with high correlation to the target variable.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 20 / 78

Feature Selection: Filter Methods
• Common Metrics/Methods:
• Chi-Squared Test (χ2 ): Measures the independence between categorical variables. Used for feature
selection in classification problems with categorical features.
X (Oi − Ei )2
χ2 =
Ei
where Oi are the observed frequencies and Ei are the expected frequencies. Select features with high
χ2 values (indicating dependence on the target variable).
• Analysis of Variance: In regression with categorical features, ANOVA selects features with
significantly higher variance between groups than within groups.
• Information Gain/Mutual Information: Selects features that provide the greatest reduction in
uncertainty (entropy) about the target variable.
• Advantages: Computationally fast, scalable to high-dimensional datasets.
• Disadvantages: Ignores feature dependencies, may select redundant features.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 21 / 78

Other Feature Engineering Techniques

Beyond Extraction and Selection

Besides extraction and selection, other important feature engineering techniques include:
• Data Cleaning: Handling missing values, outliers, and inconsistencies.
• Data Transformation: Scaling, normalization, and encoding categorical variables.
• Scaling: Standardizing or normalizing numerical features.
• Encoding: Converting categorical variables into numerical representations (e.g., one-hot
encoding, label encoding).
• Feature Creation/Construction: Combining existing features to create new ones (e.g.,
creating interaction terms).

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 22 / 78

Data Transformation: Scaling

Scaling: Bringing Features to a Similar Scale

Scaling transforms numerical features to a similar range of values. This is important because
features with larger values can dominate those with smaller values, which can negatively
impact the performance of some machine learning algorithms.
• Why is scaling important?
• Prevents features with larger magnitudes from dominating others.
• Improves the convergence speed of gradient-based optimization algorithms.
• Improves the performance of distance-based algorithms (e.g., k-NN, SVM).

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 23 / 78

Data Transformation: Scaling
Common Scaling Techniques:
• Min-Max Scaling (Normalization): Scales features to a range between 0 and 1.
x − xmin
x′ =
xmax − xmin
where x is the original value, xmin is the minimum value, and xmax is the maximum value of the
feature.
• Standardization (Z-score normalization): Scales features to have a mean of 0 and a standard
deviation of 1.
x −µ
x′ =
σ
where x is the original value, µ is the mean, and σ is the standard deviation of the feature.
• Robust Scaling: Scales features using median and interquartile range (IQR), making it less
sensitive to outliers.
x − Median
x′ =
IQR
where IQR is the interquartile range (Q3-Q1).
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 24 / 78
Feature Engineering: Summary

Key Takeaways
• Feature engineering is a crucial step in building effective machine learning models.
• It involves transforming raw data into meaningful features through extraction, selection,
cleaning, transformation, and creation.
• Domain expertise is often essential for effective feature engineering.
• It’s an iterative process that requires experimentation and evaluation.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 25 / 78

In-Class Assignment

Topics Covered:
• Discovery evaluation protocols:
• k-fold cross validation
• Leave one out cross validation
• Feature selection methods:
• Wrapper method
• Embedded method
• Other Engineering techniques:
• Encoding method

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 26 / 78

Introduction to Regression
Regression: Predicting Continuous Values
Regression is a supervised learning task where the goal is to predict a continuous numerical
output (target variable) based on input features.
• Examples:
• Predicting house prices based on features like size, location, and number of bedrooms.
• Forecasting stock prices based on historical data and market indicators.
• Estimating temperature based on weather patterns.
• Types of Regression:
• Linear Regression
• Polynomial Regression
• Support Vector Regression (SVR)
• Decision Tree Regression
• Random Forest Regression
• Neural Network Regression

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 27 / 78

Linear Regression
Linear Regression: Modeling Linear Relationships
Linear regression assumes a linear relationship between the input features and the target
variable.
• Simple Linear Regression: One input feature.
y = β0 + β1 x
where y is the predicted output, x is the input feature, β0 is the intercept, and β1 is the
slope.
• Multiple Linear Regression: Multiple input features.
y = β0 + β1 x1 + β2 x2 + · · · + βn xn
where x1 , x2 , . . . , xn are the input features, and β1 , β2 , . . . , βn are their corresponding
coefficients.
• Goal: Find the optimal values for the coefficients (βs) that minimize the difference
between the predicted and actual values (e.g., using Mean Squared Error).
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 28 / 78
Other Regression Methods

Beyond Linear Relationships

When the relationship between the input features and the target variable is non-linear, other
regression methods are more appropriate.
• Polynomial Regression: Models non-linear relationships using polynomial functions.
• Support Vector Regression (SVR): Uses support vectors to define a margin of
tolerance around the predicted values.
• Decision Tree Regression: Partitions the feature space into regions and predicts a
constant value within each region.
• Random Forest Regression: An ensemble method that combines multiple decision trees
to improve prediction accuracy.
• Neural Network Regression: Uses neural networks to learn complex non-linear
relationships.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 29 / 78

Linear Regression: Introduction

Linear regression models the relationship between a dependent variable y and one or more
independent variables x using a linear equation.
• Types of Linear Regression:
• Simple Linear Regression: One independent variable.

y = β0 + β1 x

• Multiple Linear Regression: Multiple independent variables.

y = β0 + β1 x1 + β2 x2 + · · · + βn xn

• Goal: Find the optimal coefficients (βs) that minimize the error between predicted and
actual values.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 30 / 78

Data Preprocessing for Linear Regression

Several preprocessing steps are often necessary to ensure the best performance of a linear
regression model:
• Handling Missing Values: Imputation (mean, median), removal.
• Outlier Detection and Treatment: Removal, transformation (e.g., log transformation).
• Feature Scaling: Important when features have different scales (e.g., standardization,
min-max scaling).
• Feature Encoding: Converting categorical variables into numerical representations (e.g.,
one-hot encoding).
• Feature Selection/Engineering: Selecting relevant features and creating new ones.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 31 / 78

Evaluating Linear Regression: Mean Squared Error (MSE)

Mean Squared Error (MSE): A Common Evaluation Metric

MSE measures the average squared difference between the predicted and actual values.
• Formula:
n
1X
MSE = (yi − ŷi )2
n
i=1

where yi is the actual value, ŷi is the predicted value, and n is the number of data points.
• Interpretation:
• A lower MSE indicates better model performance.
• MSE is sensitive to outliers due to the squared term.
• Other Metrics: RMSE (Root Mean Squared Error), MAE (Mean Absolute Error),
R-squared.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 32 / 78

Assumptions of Linear Regression

Key Assumptions
Linear regression models are based on several assumptions. Violating these assumptions can
affect the model’s validity.
• Linearity: The relationship between the independent and dependent variables is linear.
• Independence of Errors: The errors (residuals) are independent of each other.
• Homoscedasticity: The variance of the errors is constant across all levels of the
independent variables.
• Normality of Errors: The errors are normally distributed.
• No Multicollinearity: The independent variables are not highly correlated with each
other.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 33 / 78

Estimating Weights in Linear Regression

Finding the Best Fit Line

In linear regression, we aim to find a function that best describes the linear relationship
between input features (x) and the output (y ):

ŷ = f (x) = w1 x1 + w0 = xT w
where:
• ŷ is the predicted output.
 
• x = x1 is the input vector (including a constant 1 for the bias term).
1
 
• w = w1 is the weight vector (w1 is the weight for x1 , and w0 is the bias).
w0

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 34 / 78

Loss Function and Minimization
Minimizing the Error
We want to minimize the error between the predicted values (ŷ ) and the actual values (y ). A
common way to do this is by minimizing the Mean Squared Error (MSE). The loss function is:
N
1 X
L(w) = (yi − xT
i w)
2
2N
i=1

where:
• N is the number of data points.
• yi is the actual output for the i-th data point.
• xi is the input vector for the i-th data point.
We want to find the weights w that minimize this loss:

w∗ = argmin L(w)
w

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 35 / 78

Matrix Form and Solution
Matrix Form and Normal Equation
The loss function can be expressed in matrix form:
1
L(w) = ∥Xw − y∥2
2N
   T
y1 x1
 y2  xT2 
where y =  .  is the vector of actual outputs, X =  .  is the design matrix (each row is an input vector).
   
 ..   .. 
yN xTN
To find the minimum, we take the derivative of the loss function with respect to w and set it to zero:

∂L(w) 1
= XT (Xw − y) = 0
∂w N
Solving for w gives the normal equation:

w = (XT X)−1 XT y

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 36 / 78

Handling Non-Invertibility

Pseudo-Inverse
If XT X is not invertible (singular), we use the pseudo-inverse:

w = (XT X)† XT y
where (XT X)† is the Moore-Penrose pseudo-inverse of XT X.
The pseudo-inverse always exists and provides the least-squares solution, minimizing the
squared error even when the matrix is not invertible.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 37 / 78

Linear Regression Example: Data Preparation
Data:
• Input (Height - x)
• Output (Weight - y)
147 49
   
150 50
   
153 51
   
155 52
   
158 54
x=
  , y =  
160
 56
 
163 58
   
165 59
   
168 60
170 62

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 38 / 78

Linear Regression Example: Calculating XTX and (XTX)-1
Calculations:
• Design Matrix (X): Adding a column of ones for the bias term:

147 1
 
150 1
 
153 1
 
155 1
 
158 1
X=
160

 1
163 1
 
165 1
 
168 1
170 1

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 39 / 78

Linear Regression Example: Calculating XTX and (XTX)-1

Calculations:
• XT X: P 2 P   
x xi 252765 1599
XT X = P i =
xi N 1599 10
• (XT X)−1 :
 
T −1 0.1656 −26.477
(X X) ≈
−26.477 4188.72

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 40 / 78

Linear Regression Example: Calculating XTy and w

Calculations:
• XT y: P   
T xi yi 81077
X y= P =
yi 541
• Weights (w):
 
0.612
w = (XT X)−1 XT y ≈
−41.31

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 41 / 78

Linear Regression Example: The Equation and Prediction

The Linear Regression Equation:

ŷ = 0.612x − 41.31
Example Prediction (Height = 160):

ŷ = 0.612 × 160 − 41.31 ≈ 56.61

Conclusion:
A person with a height of 160 cm is predicted to have a weight of approximately 56.61 kg.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 42 / 78

Evaluating Linear Regression: Predictions
New Test Data:
• Height (xtest )
• Actual Weight (ytest )
   
173 63
175 64
   
xtest 178 , ytest = 66
=   
180 67
183 68
Predictions (ŷtest ): Applying the equation to each height:

ŷ1 = 0.612 × 173 − 41.31 ≈ 64.476

ŷ2 = 0.612 × 175 − 41.31 ≈ 65.696
ŷ3 = 0.612 × 178 − 41.31 ≈ 67.536
ŷ4 = 0.612 × 180 − 41.31 ≈ 68.756
ŷ5 = 0.612 × 183 − 41.31 ≈ 70.596

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 43 / 78

Evaluating Linear Regression: Mean Squared Error (MSE)
Calculating MSE:
n
1X
MSE = (yi − ŷi )2
n
i=1

1
MSE = [(63 − 64.476)2 + (64 − 65.696)2 + (66 − 67.536)2
5
+ (67 − 68.756)2 + (68 − 70.596)2 ]
1
≈ [2.179 + 2.876 + 2.350 + 3.084 + 6.749]
5
1
≈ [17.238]
5
≈ 3.448

Result: The Mean Squared Error for this test data is approximately 3.448.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 44 / 78

Linear Regression: Calculation Summary
Given input data x and corresponding outputs y, the weights w for the linear regression
equation ŷ = xT w can be calculated as follows:
1. Prepare the Data:
• Create the design matrix X by adding a column of ones to x for the bias term.
• Ensure x and y are column vectors.
2. Calculate XT X: Multiply the transpose of X by X.
3. Calculate the Inverse of XT X: Calculate (XT X)−1 . If XT X is singular (not invertible), use the
pseudo-inverse (XT X)† .
4. Calculate XT y: Multiply the transpose of X by y.
5. Calculate the Weights w:
w = (XT X)−1 XT y
T † T
(or w = (X X) X y if using the pseudo-inverse).
6. The Linear Regression Equation: The equation is then ŷ = xT w, or in scalar form (for simple linear
regression): ŷ = w1 x + w0 .
Key Points:
• The pseudo-inverse is used when (XT X) is not invertible.
• This method minimizes the sum of squared errors.
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 45 / 78
Python Code - Manual Calculation

from __future__ import division, print_function, unicode_literals

import numpy as np
import [Link] as plt

# height (cm)
X = [Link]([[147, 150, 153, 155, 158, 160, 163, 165, 168, 170]]).T
# weight (kg)
y = [Link]([[49, 50, 51, 52, 54, 56, 58, 59, 60, 62]]).T
# Visualize data
[Link](X, y, ’ro’)
[Link]([140, 190, 45, 75])
[Link](’Height (cm)’)
[Link](’Weight (kg)’)
[Link]()

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 46 / 78

Python Code - Manual Calculation

# Building Xbar
one = [Link](([Link][0], 1))
Xbar = [Link]((one, X), axis = 1)
# Calculating weights of the fitting line
A = [Link](Xbar.T, Xbar)
b = [Link](Xbar.T, y)
w = [Link]([Link](A), b)
print(’w = ’, w)
# Preparing the fitting line
w_0 = w[0][0]
w_1 = w[1][0]
x0 = [Link](145, 185, 2)
y0 = w_0 + w_1*x0

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 47 / 78

Python Code - Manual Calculation

# Drawing the fitting line

[Link](X.T, y.T, ’ro’) # data
[Link](x0, y0) # the fitting line
[Link]([140, 190, 45, 75])
[Link](’Height (cm)’)
[Link](’Weight (kg)’)
[Link]()

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 48 / 78

Python Code - Using Library

from sklearn import datasets, linear_model

# fit the model by Linear Regression
regr = linear_model.LinearRegression(fit_intercept=False)
# fit_intercept = False for calculating the bias
[Link](Xbar, y)

# Compare two results

print( ’Solution found by scikit-learn : ’, regr.coef_ )
print( ’Solution found by (5): ’, w.T)

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 49 / 78

Student Assignment: Code Comparison

Objective: To compare the results of manual linear regression calculations with those
obtained using the scikit-learn library.
• Students will implement linear regression from scratch using matrix operations (as derived
in class).
• Students will then use scikit-learn’s LinearRegression class to fit the same data.
• By comparing the resulting weight vectors, students will:
• Verify the correctness of their manual calculations.
• Gain practical experience with a widely used machine learning library.
• Develop a deeper understanding of the underlying mathematical principles of linear regression.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 50 / 78

Introduction to Classification

Classification is a supervised learning task where the goal is to predict a categorical output (a
class label) based on input features.
• Examples:
• Email spam detection (spam/not spam).
• Image classification (cat/dog/bird).
• Medical diagnosis (disease/no disease).
• Common Classification Algorithms:
• Decision Trees
• Random Forests
• Support Vector Machines (SVMs)
• k-Nearest Neighbors (k-NN)
• Neural Networks
• Logistic Regression (often used for binary classification)

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 51 / 78

Decision Trees
Decision trees create a tree-like structure of decisions based on feature values. Each internal
node represents a feature, each branch represents a decision rule, and each leaf node
represents an outcome (class label).
• How it works: Recursively partitions the data based on feature values to create homogeneous subsets
(i.e., subsets with mostly the same class label).
• Advantages: Easy to understand and interpret, can handle both categorical and numerical data.
• Disadvantages: Prone to overfitting, can be sensitive to small changes in the data.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 52 / 78

Random Forests

Random forests are an ensemble learning method that combines multiple decision trees to
improve prediction accuracy and reduce overfitting.
• How it works: Creates multiple decision trees on random subsets of the data and
random subsets of features. The final prediction is made by aggregating the predictions of
all trees (e.g., by majority voting).
• Advantages: High accuracy, robust to overfitting, handles high-dimensional data well.
• Disadvantages: More complex than single decision trees, less interpretable.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 53 / 78

Support Vector Machines (SVMs)
SVMs find the optimal hyperplane that best separates data points of different classes.
• How it works: Maps data points to a high-dimensional space and finds a hyperplane with the largest
margin between classes. Can use kernel functions to handle non-linearly separable data.
• Advantages: Effective in high-dimensional spaces, relatively memory efficient.
• Disadvantages: Can be computationally intensive for large datasets, sensitive to kernel choice and
hyperparameters.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 54 / 78

k-Nearest Neighbors (k-NN)

k-NN: Classifying Based on Neighbors

k-NN classifies a data point based on the majority class among its k nearest neighbors in the
feature space.
• How it works: Calculates the distance between the new data point and all training data
points. Selects the k nearest neighbors and assigns the most frequent class among them.
• Advantages: Simple to implement, no training phase.
• Disadvantages: Computationally expensive for large datasets, sensitive to feature scaling
and the choice of k.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 55 / 78

Neural Networks
Neural networks are complex models inspired by the structure of the human brain. They consist
of interconnected layers of neurons that learn complex non-linear relationships in the data.
• How it works: Multiple layers of interconnected nodes (neurons) process information through weighted
connections. Learning occurs by adjusting these weights based on the data.
• Advantages: Can learn highly complex patterns, achieve high accuracy on various tasks.
• Disadvantages: Can be computationally expensive to train, require large amounts of data, can be difficult
to interpret (black box).

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 56 / 78

Decision Trees: Introduction and Nominal Data

Decision Trees: A Tree-Based Approach to Classification

Decision trees are a powerful and interpretable classification method that uses a tree-like
structure to represent decisions and their possible outcomes.
Nominal Data:
• Nominal data represents categories or labels with no inherent order.
• Examples: Colors (red, blue, green), types of fruit (apple, banana, orange), weather
conditions (sunny, cloudy, rainy).
• Used as attributes (features) in decision trees to split the data.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 57 / 78

Decision Trees: Concept and Principle
A decision tree consists of:
• Nodes: Represent decisions based on feature values.
• Branches: Represent the outcomes of the decisions.
• Leaves: Represent the final outcomes or class labels.
The principle behind decision tree construction is to recursively partition the data into subsets
that are as “pure” as possible with respect to the target variable. “Purity” means that the
subsets contain mostly instances of a single class.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 58 / 78

Building a Decision Tree: Picking the Root Attribute
The core of decision tree construction lies in selecting the most informative attribute for
splitting the data at each node. This process aims to create subsets of data that are
increasingly “pure” (i.e., contain instances primarily of a single class).
• The Goal: To maximize the separation between classes after the split.
• How it Works:
• For each attribute:
• Calculate the entropy of the current node (before the split).
• Calculate the entropy of each child node that would result from splitting on that attribute.
• Calculate the information gain achieved by splitting on that attribute.
• Select the attribute with the highest information gain as the splitting attribute for the
current node.
• Example: With fruit data (apple, banana, orange) and features like color (red, yellow, orange) and size
(small, medium, large), we calculate information gain for each feature. The feature with higher gain is
chosen for the split.
• Recursive Process: This splitting process repeats for each child node until a stopping criterion is met
(e.g., pure nodes or maximum depth).
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 59 / 78
Entropy

Entropy measures the impurity or disorder of a set of data. In the context of classification, it
measures the degree to which a set of instances is mixed with different class labels.
• Formula:
Xc
H(S) = − pi log2 (pi )
i=1

where:
• S is the set of instances.
• c is the number of classes.
• pi is the proportion of instances in S that belong to class i.
• Interpretation:
• H(S) = 0 if all instances belong to the same class (pure set).
• H(S) is maximum when the instances are evenly distributed across all classes (most impure set).

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 60 / 78

Information Gain

Information gain measures the reduction in entropy achieved by splitting the data on a
particular attribute.
• Formula:
X |Sv |
Gain(S, A) = H(S) − H(Sv )
|S|
v ∈Values(A)

where:
• S is the set of instances.
• A is the attribute.
• Values(A) is the set of possible values of attribute A.
• Sv is the subset of S where attribute A has value v .
• Selecting the Root Node: The attribute with the highest information gain is chosen as
the root node for splitting.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 61 / 78

Decision Tree Example: Data

Outlook Temp. Humidity Wind Play?

• Outlook
• S(unny)
S H H W - • O(vercast)
S H H S - • R(ainy)
O H H W +
R M H W +
• Temperature
R C N W + • H(ot)
R C N S - • M(edium)
O C N S + • C(ool)
S M H W - • Humidity
S C N W + • H(igh)
R M N W + • N(ormal)
S M N S + • L(ow)
O M H S +
O H N W + • Wind
R M H S - • S(trong)
• W(eak)

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 62 / 78

Decision Tree Example: Initial Entropy

Initial Entropy (S):

• Total instances (N) = 14
• Positive instances (+) = 9
• Negative instances (-) = 5
• Calculation
9
p(+) = 14 ≃ 0.643
5
p(−) = 14 ≃ 0.357
H(S) = −p(+) log2 (p(+)) − p(−) log2 (p(−)) ≃ 0.940

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 63 / 78

Decision Tree Example: Information Gain for Outlook
Information Gain for Outlook:
• Outlook = Sunny (S): H(S⟨sub⟩Sunny⟨/sub⟩) ≃ 0.971 (5 instances: 2+, 3-)
• Outlook = Overcast (O): H(S⟨sub⟩Overcast⟨/sub⟩) = 0 (4 instances: 4+, 0-)
• Outlook = Rainy (R): H(S⟨sub⟩Rainy⟨/sub⟩) ≃ 0.971 (5 instances: 3+, 2-)
where:
• H(): Represents the entropy function.
• S⟨sub⟩Sunny⟨/sub⟩: Represents the subset of the data where the Outlook is “Sunny.”
• S⟨sub⟩Overcast⟨/sub⟩: Represents the subset of the data where the Outlook is “Overcast.”
• S⟨sub⟩Rainy⟨/sub⟩: Represents the subset of the data where the Outlook is “Rainy.”
 
5 4 5
Gain(S, Outlook) = H(S) − H(SSunny ) + H(SOvercast ) + H(SRainy )
14 14 14
 
5 4 5
≃ 0.940 − × 0.971 + ×0+ × 0.971
14 14 14
≃ 0.246
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 64 / 78
Decision Tree Example: Information Gain for Other Attributes
Information Gain for Other Attributes:
• Temperature:
• H(S⟨sub⟩Hot⟨/sub⟩) = 1 (4 instances: 2+, 2-)
• H(S⟨sub⟩Medium⟨/sub⟩) ≃ 0.918 (6 instances: 4+, 2-)
• H(S⟨sub⟩Cool⟨/sub⟩) ≃ 0.811 (4 instances: 3+, 1-)
Gain(S, Temperature) ≃ 0.029
• Humidity:
• H(S⟨sub⟩High⟨/sub⟩) ≃ 0.985 (7 instances: 3+, 4-)
• H(S⟨sub⟩Normal⟨/sub⟩) ≃ 0.592 (7 instances: 6+, 1-)
Gain(S, Humidity) ≃ 0.151
• Wind:
• H(S⟨sub⟩Strong⟨/sub⟩) = 1 (6 instances: 3+, 3-)
• H(S⟨sub⟩Weak⟨/sub⟩) ≃ 0.811 (8 instances: 6+, 2-)
Gain(S, Wind) ≃ 0.048
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 65 / 78
Decision Tree Example: Choosing the Root Node

Comparing Information Gains:

• Gain(S, Outlook) ≃ 0.246
• Gain(S, Temperature) ≃ 0.029
• Gain(S, Humidity) ≃ 0.151
• Gain(S, Wind) ≃ 0.048
Conclusion:
Outlook has the highest information gain. Therefore, Outlook is selected as the root node of
the decision tree.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 66 / 78

Building the Decision Tree: Next Steps

After determining “Outlook” as the root node, we proceed by creating subsets of the data
based on its values (Sunny, Overcast, Rainy) and repeating the information gain calculation for
each subset.
1. Create Subsets:
• Sunny Subset: Instances where Outlook = Sunny.
• Overcast Subset: Instances where Outlook = Overcast.
• Rainy Subset: Instances where Outlook = Rainy.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 67 / 78

Building the Decision Tree: Splitting Each Subset

2. Splitting Each Subset:

For each subset created in the previous step:
1. Calculate the entropy of the subset H(Ssubset ).
2. For each remaining attribute (Temperature, Humidity, Wind):
2.1 Calculate the information gain if we split the subset on that attribute:
Gain(Ssubset , Attribute).
3. Choose the attribute with the highest information gain as the splitting attribute for that
branch.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 68 / 78

Building the Decision Tree: Recursive Process and Example

3. Recursive Process:
The splitting process is repeated recursively for each newly created branch until a stopping
criterion is met:
• Stopping Criteria:
• All instances in a subset belong to the same class (pure node).
• No more attributes to split on.
• A predefined maximum tree depth is reached.
• The number of instances in a node falls below a threshold.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 69 / 78

Outlook Subset

Example: Splitting the “Sunny” Branch:

If we are working with the “Sunny” subset, we would calculate the information gain for
splitting on “Temperature,” “Humidity,” and “Wind” within that subset and choose the
attribute that yields the highest information gain to create further branches. Outlook =

Temp. Humidity Wind Play?

H H W -
Sunny H H S -
M H W -
C N W +
M N S +

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 70 / 78

Splitting the “Sunny” Branch
Temp. Humidity Wind Play?
H H W -
Outlook = Sunny Subset: H H S -
M H W -
C N W +
M N S +

Calculating Information Gain:

• Gain(S⟨sub⟩Sunny⟨/sub⟩, Humidity) ≃ 0.971 - (3/5 * 0 + 2/5 * 0) = 0.971
• Gain(S⟨sub⟩Sunny⟨/sub⟩, Temperature) = 0.571
• Gain(S⟨sub⟩Sunny⟨/sub⟩, Wind) = 0.019
Humidity is chosen for the next split.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 71 / 78

Splitting the “Rainy” Branch

Temp. Humidity Wind Play?

M H W +
Outlook = Rainy Subset: C N W +
C N S -
M N W +
M H S -
Calculating Information Gain:
• Gain(S⟨sub⟩Rainy⟨/sub⟩, Wind) ≃ 0.971 - (3/5 * 0.918 + 2/5 * 0) = 0.419
• Gain(S⟨sub⟩Rainy⟨/sub⟩, Temperature) = 0.019
• Gain(⟨sub⟩Rainy⟨/sub⟩, Humidity) = 0.019
Wind is chosen for the next split.

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 72 / 78

The Complete Decision Tree

Outlook

Sunny Overcast Rainy

Humidity Play = + Wind

High Normal Strong Weak

Play = - Play = + Play = - Play = +

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 73 / 78

Python Code - Classification with DT
import pandas as pd
from sklearn.model_selection import train_test_split
from [Link] import DecisionTreeClassifier
from [Link] import accuracy_score, classification_report
from [Link] import plot_tree
import [Link] as plt

# Load the dataset

file_path = ’/content/sample_data/[Link]’
df = pd.read_csv(file_path)

# Inspect the first few rows of the dataset

print("First few rows of the dataset:")
print([Link]())

# Separate features and labels

X = [Link][:, :-1] # Features: all columns except the last
y = [Link][:, -1] # Labels: the last column
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 74 / 78
Python Code - Classification with DT

# Split the data into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, s

# Create and train the Decision Tree classifier

clf = DecisionTreeClassifier(random_state=42)
[Link](X_train, y_train)

# Make predictions on the test set

y_pred = [Link](X_test)

# Evaluate the classifier

accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.2f}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred))

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 75 / 78

Python Code - Classification with DT

# Visualize the Decision Tree

[Link](figsize=(12, 8))
plot_tree(clf, feature_names=[Link], class_names=clf.classes_, filled=True)
[Link]("Decision Tree Visualization")
[Link]()

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 76 / 78

Python Code - Different Classification Algorithms

# Initialize classifiers
classifiers = {
"SVM": SVC(kernel=’linear’, random_state=42),
"k-NN": KNeighborsClassifier(n_neighbors=5),
"Neural Network": MLPClassifier(hidden_layer_sizes=(10,), max_iter=1000, random_state=4
}

# Train and evaluate each classifier

for name, clf in [Link]():
print(f"\n=== {name} Classifier ===")
[Link](X_train, y_train) # Train the classifier
y_pred = [Link](X_test) # Make predictions
accuracy = accuracy_score(y_test, y_pred) # Calculate accuracy
print(f"Accuracy: {accuracy:.2f}")
print("Classification Report:")
print(classification_report(y_test, y_pred))

Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 77 / 78

Student Assignment: Code Comprehension and Experimentation
Objective: Deepen understanding of the machine learning workflow through code analysis and
experimentation.
• Code Dissection: Students will meticulously analyze the provided Python code,
explaining the purpose of each line and its role within the overall machine learning
framework. This includes:
• Data loading and preprocessing.
• Model instantiation and training.
• Prediction and evaluation.
• Data Manipulation and Feature Engineering: Students will explore the impact of data
changes and feature engineering:
• Experiment with different train/test splits (e.g., 80/20, 70/30, k-fold cross-validation).
• Implement basic feature engineering techniques (e.g., feature scaling, creating interaction terms if
applicable).
• Analyze how these changes affect model performance.
• Algorithm Comparison: Students will extend the code to include and compare the
performance of other classification algorithms from scikit-learn (e.g., Logistic Regression,
Support Vector Machines, Random Forests).
Thien Huynh-The - HCMUTE Machine Learning December 26, 2025 78 / 78