Course: MSc DS

Advanced Machine Learning

Module: 1
Preface

In the expansive domain of Machine Learning, it is essential to

explore algorithms and methodologies in more depth in order
to construct resilient and effective models. This thorough
course is designed with a framework consisting of five modules,
with the objective of guiding learners through fundamental as
well as cutting-edge subjects in the field of Advanced Machine
Learning.
Module 1 of the course titled "Foundations of Decision Trees
and Random Forests"provides a comprehensive analysis of the
underlying principles and complexitiesassociated with these
tree-based methodologies. This module places significant
emphasis on highlighting the advantages as well as the
possible drawbacks of usingthese approaches. In this module,
we go into the examination, depiction, andassessment of these
models, establishing robust groundwork for the ensuing topics.
Module 2, titled "Exploring Model Parameters and
Hyperparameters," provides acomprehensive understanding
of the fundamental aspects of machine learningmodels. In this
paper, we want to clarify the differentiation between
parameters andhyperparameters and provide several
approaches for tweaking and optimising [Link] doing so, we
enable practitioners to develop models that exhibit both
robustnessand generalisability.
The Module 3 of the course, titled "Advanced Regression
Models", delves into the constraints of conventional linear
regression and examines more sophisticated alternatives,
including Ridge and LASSO regression techniques. This lesson
provides valuable insights on how to effectively approach
difficulties like multicollinearity while also reaffirming
important modelling assumptions.
The Module 4, titled "Boosting, Bagging, and Beyond,"
introduces ensemble approaches that aim to improve model
stability and accuracy. This study explores themethodologies of
adaptive boosting and gradient boosting, elucidating their
practicalimplementations and inherent benefits.
In conclusion, Module 5, titled "Time Series Analysis in Machine
Learning", delves into the intricacies of sequential data
dynamics. By using decomposition, modelling, and
sophisticated forecasting approaches, individuals will acquire
the necessary abilities to address the complexities posed by
datasets that are time-dependent.

Learning Objectives:
1. Understand the basic terms and ideas used in decision-
making.
2. Understand the creation and visualisation of decision tree
models for analysingdata.
3. Understand the fundamental ideas and benefits of random
forest algorithms.
4. Create and evaluate random forest models adapted to various
datasets.
Structure:
1.1 Introduction to Decision Trees: Basics and Terminology
1.2 Creating and Visualising Decision Tree Models
1.3 Introduction to Random Forests: Concept and Benefits
1.4 Building Random Forest Models for Various Datasets
1.5 Interpretation and Evaluation of Decision Tree and Random
Forest Results
1.6 Summary
1.7 Keywords
1.8 Self-Assessment Questions
1.9 Case Study
1.10 References
1.1 Introduction to Decision Trees: Basics and Terminology
Decision trees are a foundational method within the realm of
machine learning, renowned for their inherent simplicity and
interpretability. Decision trees fundamentally divide data into
subsets by considering the values of input attributes. The
aforementioned process persists in a recursive manner,
resulting in the formation of a model characterised by a tree
structure, which represents various choices.
The primary constituents of a decision tree include the following
elements:
● Nodes: These entities denote a distinct characteristic or
aspect that the algorithm scrutinises. The first node, serving
as the starting point of the tree, is referred to as the "root
node." The term "it" refers to the whole of the dataset prior
to any division or separation.
● Edges/Branches: These nodes provide the result of a test
conducted on a certain property, which then determines the
next node to be traversed or leads to a terminal node. Nodes
are interconnected entities that facilitate the process of
decision-making by enabling the flow of data via a
hierarchical structure.
● Terminal Nodes or Leaf Nodes: The terminal nodes of the
decision tree represent the ultimate objectives, where no
more partitioning takes place. The outputs of machine
learning models typically consist of the result, which is often
represented as a class label in classification tasks or a
 continuous value in regression tasks.
● Depth: The longest route from the root node to a leaf is
referred to as the length of such a path. This observation
highlights the intricate nature of the tree.
One of the main advantages of decision trees is in their
inherent ability to be easily interpreted by humans. Unlike
some other algorithms, they do not function as a "black box."
Their decision-making process may be visualised and
comprehended by those who do not possess an extensive
understanding of machine learning. The use of a tree structure
enables the visualisation of the sequential decision-making
process that culminates in a certain conclusion. This
characteristic renders tree structures very advantageous in
situations where the capacity to comprehend and understand
the decision-making process is of utmost importance.
Nevertheless, the inherent simplicity of these entities is
accompanied by some constraints. The phenomenon of
decision trees being prone to overfitting is a well-documented
issue, wherein these models have a tendency to include
irrelevant or noisy information from the dataset, therefore
reducing their ability to generalise to new, unseen data.
Ensemble approaches, such as random forests, are used in this
context to enhance the robustness and accuracy of the
outcomes by using many trees. However, prior to delving into
more complex methodologies, it is essential to possess a
comprehensive comprehension of the fundamental principles
behind decision trees.

1.2 Creating and Visualising Decision Tree Models

The construction of decision trees involves partitioning the
dataset into subsets by considering the values of input
characteristics. The objective is to generate sets that exhibit a
high degree of homogeneity in relation to the target variable. In
order to construct a decision tree model, it is necessary to first
choose criteria that may be used to assess the homogeneity or
impurity of a given dataset. Frequently used metrics include the
Gini impurity, entropy, and classification error.
The method starts by assessing the whole dataset at the root
node. The algorithm assesses each attribute in order to identify
the optimal split that maximises the homogeneity of the
resultant subgroups. After selecting a feature and a split point,
the dataset is partitioned, and this procedure is iteratively
applied to each subset, resulting in the creation of new nodes
within the tree. The process of iterative division continues until
a specified stopping condition is satisfied, such as the
attainment of a certain maximum depth for the tree or the
point at which further splitting ceases to provide any more
value.
Decision tree models excel in the domain of visualisation. Visual
representations facilitate the comprehension of the model's
logic by stakeholders, even those without technical
competence. In the conventional representation of a tree
structure, the visualisation starts at the apex node positioned at
the uppermost level, while subsequent branches diverge in a
downward direction. Every individual node provides specific
information on the characteristic under consideration and the
corresponding value of the criteria. On the other hand, each
branch symbolises the result or outcome of that particular
assessment. The choice or expected result is shown at the leaf
nodes located at the bottom of the tree.
Software libraries, such as Scikit-learn in Python, provide pre-
existing functionalities that enable the creation and
visualisation of decision trees. The effectiveness of these
visualisations may be augmented by the use of tools such as
Graphviz, which facilitates the creation of more complex and
detailed graphical depictions.
In brief, the process of constructing a decision tree model
entails repeatedly partitioning the dataset by leveraging
feature values, with the objective of achieving optimal
uniformity within each subset. The model that is produced as a
consequence of this process has an intrinsic visual character,
rendering it amenable to graphical representation.
Consequently, it becomes readily accessible and interpretable
to a wide range of individuals.
1.3 Introduction to Random Forests: Concept and Benefits
Random forests are an influential ensemble learning technique
that leverages the straightforwardness and interpretability of
decision trees. The fundamental concept behind random
forests is the amalgamation of predictions generated by many
decision trees in order to create a model that is more resilient
and precise. Random forests address the issue of overfitting
often seen in single decision trees by capitalising on the
collective power of several trees and effectively averaging their
individual faults.
The occurrence of "randomness" inside random forests may be
attributed to two main factors. To begin with, it is important to
note that throughout the forest, every individual tree
undergoes training using a random subset of the data. This
process involves sampling with replacement, which is often
referred to as bootstrapping. This mechanism guarantees that
each tree is exposed to a somewhat distinct viewpoint of the
data, hence fostering diversity in their decision-making
processes. Furthermore, in the process of partitioning, only a
stochastic subset of characteristics is taken into account at each
node, as opposed to evaluating all the features that are
accessible. The use of randomisation in the model serves to
avoid the dominance of any one trait, hence fostering more
variety among the trees.

There are several advantages associated with the use of random

forests:
● Robustness to Overfitting: Although individual decision
trees have a tendency to overfit the training data, the
practice of averaging in random forests aids in achieving a
higher level of generalisation when applied to unknown
data.
● Enhanced Precision: Random forests frequently exhibit
higher accuracy in comparison to individual decision trees,
owing to their ensemble nature.
● Feature Importance: Random forests have the capability
to assess the significance of characteristics, hence offering
valuable insights into the relative influence of variables in
predictive modelling.
● Versatility: This algorithm is applicable for both
regression and classification problems and is capable of
effectively processing huge datasets with high
dimensionality.
● Parallelizable: Simultaneous construction of trees enables
random forests to exhibit scalability and efficiency.
Random forests improve the performance of decision trees by
including randomness and ensemble methods. They use the
potential of communal decision-making to provide models that
exhibit both accuracy and resistance to overfitting.
1.4 Building Random Forest Models for Various Datasets
Random forests, as an ensemble learning technique, exhibit
great versatility, accommodating a wide range of datasets,
spanning from basic numerical tables to intricate, high-
dimensional data. The process of constructing a random forest
model involves the generation of several decision trees in the
training phase, followed by the aggregation of their predictions
in the test phase. The following provides an overview of the
process:
● Data Preparation: As with every machine learning model,
the first stage involves data preparation. The process
includes the management of missing values, the encoding of
categorical variables, and the division of data into training
and test sets. The process of standardising or normalising
data may provide significant benefits, particularly when
dealing with variables that exhibit disparate sizes.

● Bootstrap Sampling: In the process of constructing each tree,

a bootstrap sample, which is a randomly selected subset of
the data with replacement, is used. This methodology
guarantees that each tree inside the model is subjected to a
distinct subset of the dataset, hence promoting variety
across the models.
● Tree Construction with Random Features: During the
construction of each tree, the process of splitting at each
node does not include the consideration of every
 characteristic. Only a randomly selected subset of
characteristics is considered. This methodology guarantees
that the model does not excessively depend on any one
characteristic, hence reducing the risk of overfitting.
● Ensemble Prediction: After the construction of the forest,
predictions are generated by combining the outputs of all
individual trees. In the context of classification, the final class
is determined by a majority vote across all trees, whereas in
regression, the average prediction across all trees is used.
● Evaluation: The evaluation of the random forest model is
conducted using the test dataset. Typical performance
measures used in classification tasks include accuracy,
precision, recall, and the F1 score. In regression assignments,
mean-squared error and R-squared are often employed
metrics.
The flexibility of random forests is a notable characteristic.
Regardless of the kind of information being analysed, whether
it involves consumer purchase behaviours, medical records,
financial indicators, or picture data, the methodology used
remains identical. Optimisation of speed may be further
enhanced by fine-tuning advanced factors like the number of
trees, the depth of trees, or the number of characteristics
analysed at each split. Random forests are widely used by data
scientists because of their exceptional flexibility, resilience, and
high accuracy, making them an essential component in their
analytical repertoire.

1.5 Interpretation and Evaluation of Decision Tree and

Random Forest Results
Decision trees and random forests are well recognised for their
high accuracy and adaptability, and they further have the
benefit of being interpretable. The comprehension and
assessment of outcomes are crucial in order to extract practical
lessons and guarantee the dependability of forecasts.
● Decision Trees: A decision tree may be described as a
structure that resembles a flowchart. The algorithm starts at
the root node and proceeds by assessing the values of
several attributes. At each node, it employs binary decision-
making processes, finally culminating in a final conclusion or
classification at the leaf nodes.
● Interpretation: The process of understanding the decision-
making process may be facilitated by tracing the trajectory
from the root to a certain leaf. The relevance of a
characteristic is positively associated with its position in the
tree, wherein more significant judgements are taken at
higher levels.
● Assessment: Metrics like accuracy, precision, recall, and the
F1 score are often used to assess decision trees in the
context of classification tasks. In regression tasks, one might
use mean-squared error or R-squared as evaluation metrics.
 Overfitting is a prevalent issue in machine learning,
characterised by a situation where the model exhibits great
performance on the training data but fails to generalise well
to unknown data.
● Random Forests: Random forests, functioning as a collection
of decision trees, have the ability to collectively contribute
their knowledge and insights.
● Interpretation: The interpretability of random forests is
significantly more complex due to their aggregated form
compared to individual trees. Nevertheless, the feature
relevance scores provided by the model demonstrate the
extent to which certain traits have contributed significantly
to the decision-making process across all trees. The out-of-
bag (OOB) error, which is an intrinsic measure in random
forests, provides valuable information on the accuracy of
predictions without the need for a distinct validation sample.
● Evaluation: Like decision trees, accuracy, precision, recall,
and the F1 score are often used metrics in the field of
classification. The use of mean squared error or R-squared is
common in regression analysis. Furthermore, the variability
seen among the predictions made by different trees might
provide valuable information on the level of confidence
shown by the model.

It is crucial to thoroughly assess both models in order to

have a comprehensive
understanding of the underlying patterns in the data. Thorough
assessment not only guarantees the dependability of forecasts
but also substantiates the insights of the model, hence
facilitating the adoption of data-driven decision-making.

1.6 Summary
❖ Module 1 provides an in-depth exploration of decision trees
and random forests, imparting students with a complete
comprehension of these potent machine-learning
methodologies. Commencing with the fundamental
principles, students acquire knowledge of the fundamental
framework and vocabulary associated with decision trees.
They get an understanding of how choices emanate from
root nodes and progress towards leaves, contingent upon
the values of the features involved. Although these trees can
be understood, they may sometimes be susceptible to
overfitting. Random forests are an ensemble approach that
combines many decision trees in order to generate models
that are both resilient and accurate. The use of the
combined knowledge of numerous trees contributes to the
mitigation of overfitting and the improvement of prediction
accuracy.

❖ As the module advances, students have practical experience

in constructing these models using a variety of datasets,
enabling them to see the versatility and adaptability of the
 methods. Additionally, the curriculum places significant
emphasis on the critical element of understanding and
assessing the outcomes. By examining the decision routes
inside individual trees and comprehending the significance
of feature importance scores in random forests, learners
acquire the ability to extract practical and applicable insights.
❖ Ultimately, the module highlights the need for a thorough
assessment, which guarantees the production of forecasts
that are not only precise but also dependable and capable of
being acted upon.

1.7 Keywords
● Decision Trees: Supervised machine learning techniques are
often used for performing classification and regression
problems.

● Random Forests: The use of an ensemble technique that

amalgamates numerous decision trees in order to enhance
accuracy and mitigate overfitting.
● Node: A decision node in a decision tree that splits into
terminal nodes (leaves) according to the values of certain
features.
● Overfitting: A modelling mistake characterised by the
algorithm's tendency to capture and incorporate noise
instead of the underlying patterns in the data, resulting in
satisfactory performance on the training data but worse
performanceon new, unexplored data.
● Feature Importance: A measure used in random forests to
assess the relative importance of various input variables in
predicting the target variable.
● Ensemble Methods: Techniques that combine predictions
from multiple machine learning algorithms to produce more
accurate and robust models

1.8 Self-Assessment Questions

1. What are the main structural and functional differences
between a decision treeand a random forest?
2. What do the nodes and leaves in a decision tree
represent, and how does adecision tree decide where to
divide a node?
3. What is bootstrapping in the context of random forests,
and why is itadvantageous?
4. Describe a situation in which an overfitting of a decision
tree could occur. Howcan a random forest be used to lessen
this issue?
5. Describe the value of a feature in a random forest model.
How may it be used toenhance the performance or
interpretability of the model?
1.9 Case Study
Title: Machine Learning to Improve
Lending DecisionsIntroduction:
The use of machine learning models in the banking industry has
seen a notable increase, aiming to enhance operational
efficiency and decision-making procedures. Determining loan
acceptance is a pivotal choice that has significant importance
for banks. Erroneous judgements have the potential to result in
financial losses, whilst excessively cautious actions might
impede the generation of potential income.
Case Study:
LendFlex, a prominent financial company, has been using
conventional credit scoring techniques for the evaluation of
loan applications. The aforementioned strategies have been
effective in previous instances, but the shifting demographics
of borrowers and the intensifying competition have prompted
them to pursue more advanced models capable of properly
forecasting borrower conduct.
Background:
Historically, LendFlex has used a multifaceted approach,
including credit scores, job records, and personal interviews as
criteria for determining loan outcomes. The acquisition of a
dataset including historical loan applicants, encompassing
diverse attributes and the corresponding loan repayment status,
has been facilitated by the emergence of machine learning
techniques. LendFlex posits that the use of decision trees and
random forests has the potential to provide a more intricate
comprehension of borrower risk in comparison to their existing
methodologies.
Your Task:
You have been entrusted with developing and assessing
decision trees and random forest models as a senior data
scientist at LendFlex. The objective is to assess and contrast the
performance of these techniques with conventional
approaches in order to ascertain their suitability as a principal
instrument for making loan approval decisions.
Questions to Consider:
1. In terms of accuracy and interpretability, a comparison may
be made between the decision tree and random forest
models.
2. Which characteristics are considered to be the most
significant by the random forest model for predicting loan
default?
3. What strategies may be used to optimise the fine-tuning of
models in order to mitigate the occurrence of false negatives,
specifically in the context of forecasting loan defaults?
4. When evaluating the business ramifications, it is necessary
to choose the most suitable model (conventional, decision
tree, or random forest) to suggest for
 LendFlex.
Recommendations:
When conducting model evaluation, it is important to take into
account both the prediction capabilities and interpretability of
the models. If the random forest demonstrates a substantial
improvement in performance compared to the decision tree
and conventional techniques, it may be justifiable to accept the
compromise in interpretability. However, in cases when the
improvements in speed are minimal, it may be more
advantageous to choose the simpler decision tree model
because of itsclear decision-making process.
Conclusion:
Machine learning, namely decision trees and random forests,
provide a potentially advantageous approach for financial
organisations such as LendFlex to enhance the accuracy and
efficiency of their loan approval procedures. By using these
models, LendFlex has the ability to make better-educated
choices, mitigate risks, and even enhance income.

1.10 References
1. Kurilovas, E., 2019. Advanced machine learning approaches
to personalise learning: learning analytics and decision
making. Behaviour & Information Technology, 38(4), pp.410-
421.
2. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J. and
Garrabrant, S., 2019. Risks from learned optimisation in
 advanced machine learning systems. arXiv preprint
arXiv:1906.01820.
3. Chakraborty, D. and Elzarka, H., 2019. Advanced machine
learning techniques for building performance simulation: a
comparative analysis. Journal of Building Performance
Simulation, 12(2), pp.193-207.
4. Hearty, J., 2016. Advanced machine learning with Python.
Packt Publishing Ltd.
5. Roy, K.S., Roopkanth, K., Teja, V.U., Bhavana, V. and Priyanka,
J., 2018. Student career prediction using advanced machine
learning techniques. International Journal of Engineering &
Technology, 7(2.20), pp.26-29.