CREDIT RISK
PREDICTION
Build a model that can predict credit risk
using a dataset consisting of data on loans
received and those rejected.
MOCH ALFAN MIFTACHUL HUDA
[Link]/IN/ALFANMIFTACHUL
BUSINESS
UNDERSTANDING
The project aims to build a predictive model for assessing credit risk,
collaboration between ID/X Partners and lending companies with
technology solutions that improve the credit decision-making process.
face challenges in efficient credit risk evaluation, with the hope that
model development will increase efficiency, reduce the risk of bad debts,
and improve customer experience, providing innovative and sustainable
solutions.
DATA UNDERSTANDING
Data analysis process where we try to understand the dataset to be used before doing further
analysis. The purpose of this stage is to gain deep insight into the structure, characteristics, type data
numerical or categorical and content of the data
DATA UNDERSTANDING
It's important to understand the data that has
a relationship to credit risk and group them to
better understand the data , such as:
Features Related to the Loan Amount
Payment-Related Features
Features Related to Credit and Debt
Features Related to Loan Conditions
Features Related to Borrower Profile
DATA PREPARATION
Select Columns that have a relationship with credit risk
Use The describe ( ) function to provides a useful statistical summary for each
column in the DataFrame, which can help in understanding the distribution of
data, identifying outliers, and gaining early insights into the characteristics of
the data.
SEARCH MISSING DATA
search for missing or inappropriate data worth Nan to facilitate
further data processing where to handle it can delete or fill in
according to the character of the data can be in the form of
mean, median, or modus.
IDENTIFY DATA SKEWNESS Find the slope of numeric data to
determine the blank data fill based on
mean, median, or mode
the skewness is close to zero or the
distribution is symmetric, you might
choose to fill in the missing values with
the mean or median.
skewness is positive or the distribution is
skewed to the right, the median may be
more suitable because the median is
less sensitive to outliers than the mean.
skewness is negative or the distribution is
skewed to the left, mode may be more
suitable because it is the most frequently
occurring value and is not affected by
the long tail on the other side of the
distribution.
HANDLING MISSING DATA
DATA PREPROCESSING
To normalize data using the Min-Max Scaling method. Normalization is performed on certain
selected columns. The Min-Max Scaling method changes the values in each column so that
the range of values is between 0 and 1, by performing this normalization, the data will have a
uniform scale across columns, which can improve the performance of scale-sensitive
machine learning models, such as distance-based algorithms and algorithms that use
kernel functions.
CATEGORY VALUE CONVERSION
Use LabelEncoder to transform term, grade, home_ownership and sub_grade category data
and store it in a new variable or column and then delete the column before encoding it for
use in machine learning
LABELING
The process generates a new column called 'label', where each row of data will have a label
value indicating whether the borrower has a good (1) or bad (0) status, based on criteria
defined in the classify_loan_status function. Where there are several statuses and to make it
easier we use fully paid and current status as good status, besides it is considered bad.
Once the data is labeled, we can use this data to train a machine learning model, where the
model will learn to predict labels for data that has not been labeled based on patterns
present in the data that has been trained. Thus, the labeling process is an important initial
stage in the development of machine learning models for classification or prediction.
MODELING
To start the modeling stage we will divide the data into training and testing data
where the results are:
training data: 373028
test data: 93257
LOGISTIC REGRESSION MODEL
Logistic Regression models are initialized and
trained using training data. Then, the model is
used to make predictions on the test data. The
prediction results are used to calculate the
accuracy of the model, which indicates how
well the model predicts the target class.
In addition, classification reports are also
created to provide further insights into model
performance, including evaluation metrics
such as precision, recall, and f1-score for each
target class. The final step, results accuracy
and classification reports are printed to the
console for further analysis. Thus, the program
provides a comprehensive understanding of
the performance of Logistic Regression models
in predicting target classes from test data.
LOGISTIC REGRESSION RESULTS
The evaluation results of the Logistic Regression model showed an accuracy of 87.49%. In
the classification report, the model has a precision of 0.89 and a recall of 0.98 for borrowed
classes that have a good status (1), indicating the model's ability to predict positive classes.
However, model performance is lower for lending classes that have a poor status (0), with a
precision of 0.46 and a recall of 0.10. This suggests that models tend to be less accurate in
predicting negative class. Thus, although the overall accuracy of the model is quite high,
there is a need for improvement especially in predicting negative classes.
NUMBER OF SAMPLES Label 1, which indicates loans with good status, has
a total of 408,965 samples, while label 0, which
indicates loans with bad status, has only a total of
57,320 samples. The imbalance ratio between the
sample number of label 1 and label 0 is
approximately 7.13. That is, there is a significant
imbalance in the number of samples between the
positive class and the negative class.
This imbalance can affect the performance of
machine learning models, Because models tend to
be more likely to predict the majority class (in this
case, grade 1) and may not be good enough at
predicting the minority class (grade 0). Therefore,
special handling is necessary, such as
oversampling or undersampling, to rebalance the
dataset before training the model to avoid
unwanted bias in the resulting model.
EVALUATION MODEL
After the model is trained with the training data,
predictions are made on the test data and evaluated
using accuracy metrics. In addition, a classification
report is created that presents evaluation metrics
such as precision, recall, and f1-score for each class.
The accuracy results and classification report are then
printed to the console, providing an understanding of
the model's performance considering the imbalance
in the sample number between positive and negative
classes.
Thus, this model Provides an understanding of model
performance that considers imbalances in sample
numbers between positive and negative classes. Thus,
the model can provide fairer and more reliable
predictions for cases where there is an imbalance in
the number of samples between the target classes.
EVALUATION RESULTS
The evaluation results of the Logistic Regression model showed an accuracy of 61.08%. In the classification
report, the precision for class 0 (loans with bad status) is 0.22, indicating that only 22% of the predicted bad
loans actually have a bad status. However, the recall for class 0 was 0.83, indicating that the model was
able to identify most borrowers that actually had poor status. For class 1 (loans with good status), precision
reached 0.96, indicating that most of the loans that were predicted to be good really had good status.
However, the recall for class 1 is 0.58, indicating the model's limitations in identifying borrowing that is
actually good. The F1-score for class 0 is 0.34, while for class 1 it is 0.72, indicating better performance in
predicting class 1 compared to class 0. Thus, the evaluation results show that the model has a fairly good
performance in predicting good borrowing, but limited in predicting bad borrowing, which can be caused
by an imbalance in the number of samples between the classes in the data.
OVERSAMPLING
Modeling implementation using Logistic
Regression with handling class
imbalances using SMOTE (Synthetic
Minority Over-sampling Technique)
method. This method creates new
synthetic samples from minority
classes to balance the distribution of
classes in the training data.
The accuracy results and classification
report are then printed to the console.
Thus, these steps illustrate a complete
modeling process, including handling
class imbalances using the SMOTE
method to improve the performance of
Logistic Regression models.
OVERSAMPLING RESULTS
The evaluation results of the Logistic Regression model by handling class imbalances using the SMOTE
method showed an accuracy of 66.36%. In the classification report, the precision for class 0 (loans with bad
status) is 0.23, indicating that only 23% of the predicted bad loans actually have a bad [Link], the
recall for class 0 is 0.73, indicating the model's ability to identify most borrowers that actually have a poor
status.
For class 1 (loans with good status), precision reaches 0.94, indicating a high level of accuracy in predicting
good loans. However, the recall for class 1 is 0.65, suggesting that models tend to be less effective in
identifying borrowing that is actually good. The F1-score for class 0 is 0.35, while for class 1 it is 0.77,
indicating better performance in predicting class 1 compared to class 0. Thus, the evaluation results show
that the model has better performance in predicting good borrowing, but there are still limitations in
predicting bad borrowing, despite using SMOTE oversampling techniques to deal with class imbalances.
MODEL TRAINING Model training uses training data that has been
oversampled using the SMOTE method. After the
model is trained, predictions are made on the
test data using the predict() method. Then, the
model is evaluated using accuracy metrics using
the accuracy_score() function and classification
reports using the classification_report() function,
which presents evaluation metrics such as
precision, recall, and f1-score for each class in
the test dataset.
This program aims to provide an understanding
of the performance of Logistic Regression
models that have been trained by dealing with
class imbalances using the SMOTE method. The
program can be used to predict target classes
from new, previously unseen data and evaluate
model performance using relevant evaluation
metrics.
MODEL TRAINING RESULTS
Overall, the evaluation results show that the
Logistic Regression model that has been trained
by handling class imbalances using the SMOTE
method still has limitations in predicting minority
classes (loans with poor status). Although the
accuracy of the model has improved, the
precision, recall, and f1-score for class 0 (poor
status) is still low compared to class 1 (good
status). Therefore, to improve its accuracy, we
will use hyperparameters
HYPERPARAMETER
perform hyperparameter tuning for Logistic
Regression models using the Grid Search Cross-
Validation method. First, a Logistic Regression
model is initialized. The range of C parameter
values is specified as [0.001, 0.01, 0.1, 0.5, 1, 5]. Grid
Search Cross-Validation is then performed using
GridSearchCV, using the model, parameter
range, 5-fold cross-validation, and accuracy
score as parameters.
After the hyperparameter tuning process is
completed, the best parameters of the model
are obtained with grid_search.best_params_
and displayed to the screen. Next, the best model
is evaluated on the test data, and the accuracy
of the best model is printed onto the screen. This
process helps us find the optimal value for
parameter C which can improve the model's
performance in predicting the target class.
HYPERPARAMETER RESULTS
These results show that after searching several C parameter values in the Logistic Regression model using
GridSearchCV, the best parameter found was C=0.5. When using the model with the best parameters to make
predictions on test data, the accuracy is around 66.64%. This accuracy describes how well the model predicts
classes on test data. Although the accuracy does not reach a very high level, it still provides better
performance than random models. In addition to accuracy, model evaluation can be expanded by considering
other metrics such as precision, recall, and f1-score for each class.
XGBOOST
If the accuracy value is still not sufficient, we can
use XGBoost (Extreme Gradient Boosting), chosen
because it is a very popular ensemble algorithm
and is appreciated for its excellent performance in
predictive modeling. This algorithm can handle
problems with large datasets and a high number of
features, while maintaining the ability to handle
imbalanced datasets. Additionally, XGBoost has
integrated regularization that helps prevent
overfitting, as well as providing options for dealing
with class imbalance. With its high scalability and
flexibility in parameter tuning, XGBoost is often a
strong choice for various prediction problems in the
fields of artificial intelligence and data science.
XGBOOST
The XGBoost model has an accuracy of 97.23%,
indicating a high level of accuracy in predictions. In
classification, precision and recall for class 0 are
0.99 and 0.79, while for class 1 they are 0.97 and
1.00. These results show excellent performance in
identifying both classes, with a slight decrease in
precision for class 0. With high F1-score values for
both classes, the XGBoost model is proven to be
effective in modeling this prediction problem.
ROC CURVE AND PRECISION-RECALL CURVE
The ROC Curve and Precision-Recall Curve provide additional insight into the performance of
the classification model, especially when encountering class imbalance as seen in the XGBoost
classification results you provided. Although the accuracy of the XGBoost model reached 97.2%,
both curves provide additional information about the model performance especially regarding
the minority class (class 0 in this case).
ROC curve shows a value of 1 indicates better performance. In the Precision-Recall curve, the
ideal operating point is in the upper right corner (Precision=1, Recall=1), which indicates that the
perfect model has perfect precision and recall.