HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION

FACULTY OF INTERNATIONAL EDUCATION

Report Project of Machine Learning

Building a Student Performance prediction model using machine
learning algorithms
COURSE: 02FIE

SEMESTER 2 – YEAR 2024-2025

Student: Phạm Nguyễn Hoài Bảo 22146005

Lê Vũ Xuân Phương 22146045

Instructor: ME. Vũ Quang Huy

Ho Chi Minh City, May 2025

1
CHAPTER 1: OVERVIEW
1.1. Introduction to the Project
In the current digital age, data has become an invaluable resource.
Effectively exploiting and utilizing data can bring significant competitive
advantages across various fields. Machine Learning (ML), a subfield of
artificial intelligence, enables computers to learn from data without
explicit programming. ML models can discover hidden patterns, make
predictions, or assist in decision-making, thereby solving complex real-
world problems. This project focuses on building and evaluating
predictive models using popular machine learning algorithms such as
Logistic Regression, K-Nearest Neighbors, Linear Regression, Decision
Trees, and especially the ensemble technique (Stacking) to improve
prediction performance.

1.2. Reasons for Choosing the Project

 High Applicability: Predictive models have wide-ranging
applications in many industries, including finance (stock price
prediction, credit risk), healthcare (disease diagnosis, treatment
effectiveness prediction), marketing (customer behavior prediction),
and many others. Mastering the techniques for building predictive
models is a key skill in the field of data science.
 Practical Implementation of Basic Algorithms: The project
allows for hands-on practice and a deeper understanding of how
basic machine learning algorithms work, their advantages and
disadvantages, forming a solid foundation for researching more
complex algorithms.
 Performance Enhancement with Stacking: Stacking is a
powerful method to combine multiple weaker models into a
stronger one, significantly improving the accuracy and
generalization capability of predictive models. Learning and
implementing Stacking provides insights into ensemble methods.
 Opportunity to Learn the ML Workflow: The project offers an
opportunity to practice the entire machine learning model building
workflow, from data collection, preprocessing, algorithm selection,
model training, result evaluation, to performance optimization.

1.3. Expected Project Scope

2
To complete this project, the following tasks will be undertaken:

 Theoretical Basis Research: In-depth study of the machine

learning algorithms used (Logistic Regression, k-NN, Linear
Regression, Decision Tree) and the Stacking technique, including
their operating principles, mathematical formulas, advantages,
disadvantages, and important parameters.
 Data Collection and Exploration: Searching for and collecting a
suitable dataset for the prediction problem, performing Exploratory
Data Analysis (EDA) to understand the data's structure and
characteristics.
 Data Preprocessing: Handling missing data, detecting and treating
outliers, encoding categorical variables, and scaling data to prepare
it for model training.
 Model Building and Training: Implementing predictive models
using the researched algorithms, training them on the preprocessed
dataset.
 Result Evaluation and Comparison: Using appropriate
evaluation metrics (e.g., RMSE, R-squared) to assess the
performance of each model and compare them, especially against
the Stacking model.
 Analysis and Conclusion: Providing observations on the achieved
results, identifying the strengths and weaknesses of each method,
and proposing future improvement directions.

CHAPTER 2: THEORETICAL BASIS

In this chapter, we present the machine learning algorithms used to
analyze and predict student performance in the Student Performance
dataset. Each algorithm is described in terms of its working principle,
strengths and weaknesses, and its application to our problem.

2.1. Logistic Regression

2.1.1. Introduction

3
Despite its name, "Logistic Regression" is a classification model used to
predict the probability of a data sample belonging to a specific class. It is
commonly used for binary classification problems (two classes) but can
also be extended to multi-class classification. Logistic Regression uses
the sigmoid function to map the output of a linear model to a probability
value between (0, 1).

2.1.2. Formula and Definition

The Logistic Regression model is based on the sigmoid function, with the
formula:

Where:

 P(Y=1∣X) is the probability that the dependent variable Y

belongs to class 1 (positive class) given the independent variables
X.
 z=beta_0+beta_1X_1+dots+beta_nX_n is the linear combination of
input variables (X_i) and weights (beta_i).
 beta_0 is the intercept.
 beta_1,dots,beta_n are the regression coefficients (weights)
corresponding to the input variables.

2.1.3. Advantages and Disadvantages

 Advantages:

o Simple, easy to understand, and straightforward to

implement.
o Effective for datasets with a linear relationship between
variables and the target class.
o Provides probabilities for each class, helping to assess the
confidence of predictions.
o Less affected by noise and can detect multicollinearity
between variables.

 Disadvantages:

4
 o Assumes a linear relationship between input variables and
the log-odds of the target variable.
o Not suitable for complex, non-linear classification problems.
o Sensitive to outliers if not handled carefully.
o Not robust when there are too many highly correlated
independent variables.

2.1.4. Hyperparameters and Their Meanings

 penalty: The type of regularization applied (e.g., 'l1', 'l2', 'elasticnet',

'none'). Regularization helps prevent overfitting.
 C: Inverse of regularization strength. Smaller C values indicate
stronger regularization.
 solver: Algorithm to use for optimization (e.g., 'liblinear', 'newton-
cg', 'lbfgs', 'sag', 'saga'). The choice of solver depends on the data
size and type of regularization.
 max_iter: Maximum number of iterations for the optimization
algorithms.

2.1.5. Practical Applications

 Healthcare: Predicting the risk of diseases (diabetes,

cardiovascular disease) based on risk factors.
 Finance: Assessing customer credit risk (likelihood of defaulting).
 Marketing: Predicting the likelihood of customers purchasing a
product or responding to an advertising campaign.
 Spam Detection: Classifying emails as spam or not.

2.2. K-Nearest Neighbors (KNN)

2.2.1. Introduction

KNN is a non-parametric, instance-based machine learning algorithm

used for both classification and regression problems. Its operating
principle is to find the k data points closest to a new data point in the
feature space. Then, for classification, it assigns the new point to the most
common class among the k neighbors; for regression, it predicts the
average (or weighted average) value of the k neighbors.

2.2.2. Formula and Definition

The k-NN algorithm does not have a "formula" for model training in the
traditional sense, as it does not learn an explicit function. Instead, it stores
the entire training dataset. When a new data point arrives, the distance

5
between this point and all points in the training set is calculated. The most
common distance metric is Euclidean distance: ​

Where:

 p and q are two points in an n-dimensional space.

 p_i and q_i are the i-th coordinates of points p and q.

2.2.3. Advantages and Disadvantages

 Advantages:

o Simple, easy to understand, and straightforward to

implement.
o No assumptions about data distribution (non-parametric).
o Effective for both classification and regression problems.
o Highly adaptable when data has complex, non-linear patterns.

 Disadvantages:

o Computationally and memory intensive for large datasets

because it needs to calculate distances to all training points.
o Sensitive to irrelevant features and noisy data.
o Performance degrades as the number of data dimensions
(features) increases (the "curse of dimensionality").
o Requires careful selection of the k value; suboptimal k can
lead to overfitting or underfitting.

2.2.4. Hyperparameters and Their Meanings

 n_neighbors: The number of neighbors k to use for prediction.

This is the most important parameter.
 weights: How weights are assigned to neighbors. Can be 'uniform'
(all neighbors have equal weight) or 'distance' (closer neighbors
have higher weight).
 metric: The distance metric used (e.g., 'euclidean', 'manhattan',
'minkowski').

2.2.5. Practical Applications

6
  Recommendation Systems: Recommending products to users
based on the preferences of similar users.
 Pattern Recognition: Handwriting recognition, facial recognition.
 Healthcare: Diagnosing diseases based on symptoms of similar
patients.
 Finance: Fraud detection.

2.3. Linear Regression

2.3.1. Introduction

Linear Regression is a supervised machine learning algorithm used to

model the linear relationship between a dependent variable (output
variable) and one or more independent variables (input variables). The
goal is to find the best-fitting line (or plane, hyperplane) for the data, such
that the sum of squared errors between the actual and predicted values is
minimized.

2.3.2. Formula and Definition

The simple linear regression model (one independent variable) has the
formula:

Where:

 y is the dependent variable (target variable).

 x_i are the independent variables (features).
 β0 is the intercept, the value of Y when all X_i are 0.
 βi are the regression coefficients, representing the change in Y for
a one-unit change in X_i, holding other variables constant.
 ϵ(epsilon) is the random error term, representing the unexplained
variance by the model.

The algorithm's objective is to minimize the cost function, typically the

Mean Squared Error (MSE):

Where:

 m is the number of training samples.

7
  y_i is the actual value of the i-th sample.
 y^i is the predicted value of the i-th sample.

2.3.3. Advantages and Disadvantages

 Advantages:

o Simple, easy to understand, and straightforward to

implement.
o Provides good interpretability of the relationship between
variables.
o Effective for problems where the relationship between
variables is linear.

 Disadvantages:

o Assumes a linear relationship between variables, cannot

model complex non-linear relationships.
o Sensitive to outliers, which can distort the regression
coefficients.
o Sensitive to multicollinearity – when independent variables
are highly correlated with each other.
o Assumes independence of residuals and their normal
distribution.

2.3.4. Hyperparameters and Their Meanings

 fit_intercept: Boolean, whether to calculate the intercept (beta_0).

Defaults to True.
 normalize: Boolean, whether to normalize the independent
variables before regression. Often replaced by StandardScaler. (In
newer scikit-learn versions, this parameter has been deprecated or
discouraged; instead, dedicated preprocessing scalers should be
used.)

2.3.5. Practical Applications

 Economics: Predicting house prices based on area, number of

bedrooms, location.
 Finance: Stock price prediction.
 Science: Modeling relationships between variables in experiments.
 Sales: Forecasting future sales.

2.4. Decision Tree

8
2.4.1. Introduction

A Decision Tree is a non-parametric, supervised machine learning

algorithm used for both classification and regression problems. It operates
by creating a tree-like model of decisions, where each internal node
represents a test on an attribute, each branch represents the outcome of
the test, and each leaf node represents a class label or a predicted value.

2.4.2. Formula and Definition

Decision Trees build a tree by splitting the dataset into smaller subsets
based on attributes. This splitting process is called "recursive
partitioning." The attribute chosen for splitting at each node is based on
criteria such as Information Gain or Gini Impurity for classification
problems, or Variance Reduction for regression problems.

Information Gain: Based on entropy.

Where:

o S is the current dataset.

o A is the attribute chosen for splitting.
o p_i is the proportion of samples belonging to class i.
o Values(A) is the set of possible values for attribute A.
o S_v is the subset of S where attribute A has value v.

Gini Impurity: Measures the impurity of a set.

The goal is to choose the attribute to split that reduces Gini Impurity the
most.

2.4.3. Advantages and Disadvantages

 Advantages:

9
 o Easy to understand and interpret, intuitive (can visualize the
tree).
o Can handle both numerical and categorical data.
o Requires less data preprocessing (no need for complex
scaling or outlier handling).
o Can model non-linear relationships well.

 Disadvantages:

o Prone to overfitting, especially when the tree is too deep.

o Unstable: small changes in the data can lead to very different
tree structures.
o Difficult to handle datasets with many categorical variables
with many levels.
o Creates orthogonal decision boundaries, which may not be
optimal for complex relationships.

2.4.4. Hyperparameters and Their Meanings

 criterion: The function to measure the quality of a split ('gini' or

'entropy' for classification, 'squared_error' for regression).
 max_depth: The maximum depth of the tree. Helps control
overfitting.
 min_samples_split: The minimum number of samples required to
split an internal node.
 min_samples_leaf: The minimum number of samples required to
be at a leaf node.
 random_state: Sets the random seed for reproducible results.

2.4.5. Practical Applications

 Medical Diagnosis: Deciding whether a patient has a disease

based on symptoms.
 Risk Management: Assessing loan risk.
 Customer Segmentation: Dividing customers into groups based
on purchasing behavior.
 Pattern Recognition: Image classification.

2.5. Stacking (Ensemble Model)

10
2.5.1. Introduction

Stacking (Stacked Generalization) is an advanced machine learning

ensemble technique that combines multiple base models (first-level
models) by training a "meta-learner" (or second-level model) on the
outputs of the base models. The goal is to leverage the strengths of each
base model and overcome their weaknesses, thereby creating a more
robust model with better generalization capabilities.

2.5.2. Formula and Definition

The Stacking process involves the following steps:

Phase 1 (Training Base Models):

o The training dataset is divided into k folds (e.g., k=5).

o For each fold:

 Train the base models (e.g., Logistic Regression, k-

NN, Decision Tree) on the remaining k−1 folds.
 Use the trained models to make predictions on the
held-out fold. These predictions will form the "meta-
features" dataset for the meta-learner.

o Additionally, the base models are also trained on the entire

original training set to make predictions on the independent
test set.

Phase 2 (Training Meta-learner):

o Create a new dataset for the meta-learner. This dataset has as

inputs the predictions of the base models from Phase 1 (i.e.,
"meta-features"), and as output the actual target variable.
o Train the meta-learner (e.g., Linear Regression, Logistic
Regression, or even a more complex model like Gradient
Boosting) on this "meta-features" dataset.
o When predicting on new data:

11
  The base models (trained on the full original data)
make predictions on the new data.
 These predictions are used as input for the meta-
learner.
 The meta-learner makes the final prediction.

2.5.3. Advantages and Disadvantages

 Advantages:

o Often achieves higher performance than any single base

model.
o Good generalization ability, helping to reduce overfitting.
o Flexible in choosing base models and meta-learners.

 Disadvantages:

o More complex to implement and debug compared to other

ensemble techniques (like Bagging or Boosting).
o Computationally expensive and time-consuming due to
training multiple models.
o Harder to interpret due to the combination of multiple
models.
o Care must be taken to avoid data leakage between base
models and the meta-learner (using K-fold cross-validation
for meta-feature generation is crucial).

2.5.4. Hyperparameters and Their Meanings

The hyperparameters for Stacking are primarily the hyperparameters of

each base model and the meta-learner. Additionally ,[Link].
StackingRegressor (or StackingClassifier) has the following parameters:

 estimators: A list of tuples, where each tuple contains the name

and object of a base model.
 final_estimator: The object of the meta-learner model.
 cv: The number of folds for cross-validation to generate meta-
features.
 passthrough: Boolean, if True, the original features are also passed
as input to the final_estimator along with the predictions of the
base models.

2.5.5. Practical Applications

12
  ML Competitions: Stacking is a very popular technique often
used to achieve top results in competitions like Kaggle.
 Finance: Stock market prediction, credit risk assessment.
 E-commerce: Recommendation systems, revenue forecasting.
 Biomedicine: Gene analysis, complex medical diagnosis.

CHAPTER 3: TRAINING DATA

In this chapter, we introduce the dataset used for analysis and model
training. The dataset pertains to the academic performance of students
and includes various socio-demographic and school-related factors.

3.1 Overview of the Dataset

Dataset name: Student Performance Data Set

Source: UCI Machine Learning Repository

Files used: [Link] (Mathematics subject)

Number of records (rows): 395 student

Number of attributes (columns): 33 features + 1 target variable

The dataset contains data collected from secondary school students in

Portugal. Each record represents information about a single student,
including personal background, family situation, educational support, and
academic results.

3.2 Key Features

The dataset includes the following types of variables:

Demographic:

sex: gender (male/female)

age: student age (15 to 22)

address: urban/rural

famsize: family size

13
 Pstatus: parents living together or apart

Parental and Family Background:

Medu and Fedu: mother’s and father’s education levels

Mjob and Fjob: mother’s and father’s job types

guardian: student’s legal guardian (mother/father/other)

School and Study-related:

school: school name

studytime: weekly study time

failures: number of past class failures

schoolsup: extra educational support

famsup: family educational support

paid: extra paid classes

internet: access to Internet at home

Behavioral and Social:

goout: frequency of going out with friends

Dalc and Walc: alcohol consumption on weekdays and weekends

health: current health status

Academic Performance:

G1, G2, G3: grades for period 1, 2, and final grade respectively (0–
20)

3.3 Target Variable

The primary variable of interest is:

G3: Final grade in Mathematics, ranging from 0 (fail) to 20

(excellent)

14
This variable will be used for:

Regression tasks: Predicting the exact grade (continuous value)

Classification tasks: Converting into categories (e.g.,

Fail/Pass/Excellent)

Example classification mapping:

G3 ≤ 9 → Fail

10 ≤ G3 ≤ 14 → Pass

G3 ≥ 15 → Excellent

3.4 Data Preprocessing (Summary)

Before applying machine learning models, the dataset is preprocessed:

Handling missing values: This dataset has no missing values.

Encoding categorical variables: Convert non-numeric fields

using One-Hot Encoding.

Feature scaling: Standardize numerical features for algorithms

sensitive to scale (e.g., KNN).

Train-test split: Typically split into 80% training and 20% testing
subsets.

Table 3.1: Data Information [Link]()

15
CHAPTER 4: MODEL TRAINING ALGORITHM
FLOWCHART
This chapter outlines the overall workflow for the student performance
prediction project. It provides a high-level visual and descriptive
representation of the main steps involved in data processing and applying
machine learning algorithms.
4.1 Project Flowchart

16
17
4.2 Description of Each Stage
4.2.1. Data Collection

The dataset [Link] is loaded from the UCI repository.

It contains real-world academic data for high school students.

4.2.2. Data Preprocessing

Convert categorical features into numerical format using encoding.

Standardize numerical features when required (especially for k-

NN).

Create new labels for classification based on final grade (G3)

4.2.3. Train-Test Split

Split the dataset into training and testing subsets to evaluate model
generalization.

Commonly use 80% for training and 20% for testing.

4.2.4. Model Selection and Training

Train multiple machine learning models:

Logistic Regression (classification)

k-NN (classification/regression)

Linear Regression (regression)

Decision Tree (classification/regression)

Stacking Ensemble (combination of models)

4.2.5. Model Evaluation

Use metrics appropriate to task type:

Classification: Accuracy, Precision, Recall, F1-score

Regression: RMSE, MAE, R²

18
4.2.6. Results Visualization

Compare models using visual tools:

Confusion matrices

Bar plots of accuracy/RMSE

Scatter plots for regression results

CHAPTER 5: TRAINING PROGRAM CODE

This chapter presents the complete Python code used in the student
performance prediction project. The implementation uses common
libraries such as pandas, numpy, scikit-learn, and matplotlib.

5.1. Library Declarations

19
5.2. Data Loading and Preprocessing

5.3 Model Training

1. Logistic Regression

lr = LogisticRegression()

[Link](X_train, y_train)

y_pred_lr = [Link](X_test)

2. k-Nearest Neighbors (k-NN)

knn = KNeighborsClassifier(n_neighbors=5)

[Link](X_train, y_train)

y_pred_knn = [Link](X_test)

20
3. Linear Regression (For Regression Task)

X_reg = [Link](['G3', 'pass'], axis=1)

y_reg = df['G3']

Xr_train, Xr_test, yr_train, yr_test = train_test_split(X_reg, y_reg, test_size=0.2,

random_state=42)

Xr_train = scaler.fit_transform(Xr_train)

Xr_test = [Link](Xr_test)

lr_reg = LinearRegression()

lr_reg.fit(Xr_train, yr_train)

y_pred_reg = lr_reg.predict(Xr_test)

4. Decision Tree

dt = DecisionTreeClassifier()

[Link](X_train, y_train)

y_pred_dt = [Link](X_test)

5. Stacking Ensemble

estimators = [

('lr', LogisticRegression()),

('knn', KNeighborsClassifier(n_neighbors=5)),

('dt', DecisionTreeClassifier())

stack_model = StackingClassifier(estimators=estimators,
final_estimator=LogisticRegression())

stack_model.fit(X_train, y_train)

y_pred_stack = stack_model.predict(X_test)

21
5.4 Model Evaluation

5.5 Visualization

CHAPTER 6: ANALYSIS AND EVALUATION OF

MODEL RESULTS
This chapter presents the results obtained from the models implemented
in Chapter 5. The performance of each algorithm is evaluated based on
metrics such as accuracy, confusion matrix, classification report, and
RMSE (for regression). The comparison allows us to identify which
model is most effective in predicting student performance.

22
6.1 Classification Results (Pass/Fail)
Observation:
The Stacking Ensemble outperformed all individual models, achieving
the highest accuracy. It combines the strengths of Logistic Regression,
k-NN, and Decision Tree, leading to better generalization.

 True Negatives (TN): 60

 False Positives (FP): 8

 False Negatives (FN): 7

 True Positives (TP): 65

This matrix shows that the ensemble model is effective at both

identifying students who will pass and those at risk of failing.

Classification Report Example (Stacking Ensemble)

Precision: High precision indicates that when the model predicts a

student will pass, it's usually correct.

Recall: High recall means it correctly identifies most passing students.

F1-Score: Balanced measure showing strong performance in both classes.

23
6.2 Regression Results (Final Grade G3 Prediction)

RMSE indicates that the average prediction error of final grades is

approximately 1.75 points.

This is a reasonable result given the grade scale (0–20), though

classification was more accurate.

6.3 Model Comparison Summary

6.4 Conclusion of Evaluation

Stacking Ensemble is the best-performing classification model.

Linear Regression gives decent grade predictions but is not as effective

as classification models for pass/fail decisions.

The combination of preprocessing, feature selection, and ensemble

learning significantly improves prediction reliability.

24
CHAPTER 7: CONCLUSION
7.1 Summary of the Study
This project aimed to apply machine learning techniques to predict
student academic performance using a real-world dataset. Various
classification and regression models were developed to determine
whether a student would pass or fail and to estimate their final grade.

Key models explored included:

Logistic Regression

k-Nearest Neighbors (k-NN)

Decision Tree

Stacking Ensemble

Linear Regression (for predicting final grades)

Each algorithm was implemented, evaluated, and compared to assess its

predictive power. The Stacking Ensemble model yielded the highest
accuracy in classification tasks, proving the effectiveness of combining
multiple models.

7.2 Key Findings

The Stacking Ensemble model achieved the best classification
accuracy (~87%), outperforming individual models like Logistic
Regression and k-NN.

Linear Regression provided reasonable estimates of student final

grades, with a Root Mean Squared Error (RMSE) of ~1.75.

Data preprocessing (handling missing values, encoding

categorical variables) and feature selection significantly
influenced model performance.

The project demonstrated the potential of machine learning in

educational data mining and early warning systems.

25
7.3 Limitations
The dataset was relatively small, which may limit generalizability.

The analysis focused on supervised learning only, without exploring

unsupervised techniques or deep learning models.

Some potentially important features (e.g., student motivation, personal

circumstances) were not included in the dataset.

7.4 Future Work

Explore additional algorithms such as Random Forest, XGBoost,
and Neural Networks for performance improvement.

Integrate more socio-emotional or behavioral data to enhance

prediction accuracy.

Build a real-time student performance monitoring dashboard

using web technologies.

Expand the dataset with longitudinal data from multiple schools or

semesters.

7.5 Final Remarks

The results show that machine learning can effectively support
educational institutions in predicting student outcomes. By identifying at-
risk students early, targeted interventions can be implemented to improve
learning outcomes. This project demonstrates both the potential and
practical steps for integrating AI tools in education.

CHAPTER 8: REFERENCES
1. Géron, A. (2019). Hands-On Machine Learning with Scikit-Learn,
Keras, and TensorFlow: Concepts, Tools, and Techniques to Build
Intelligent Systems (2nd ed.). O'Reilly Media.
2. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion,
B., Grisel, O., ... & Duchesnay, E. (2011). Scikit-learn: Machine
Learning in Python. Journal of Machine Learning Research, 12,
2825-2830.
3. Chollet, F. (2017). Deep Learning with Python. Manning
Publications Co.

26
 4. Kaggle. (n.d.). House Sales in King County, USA. Retrieved from
[Link]
5. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of
Statistical Learning: Data Mining, Inference, and Prediction (2nd
ed.). Springer.

CHAPTER 9: APPENDIX
This appendix contains the figures, charts, and data tables mentioned in
the previous chapters, clearly numbered and named for easy reference.

27
28
29