A NOVEL IMPUTATION MODEL FOR IMPROVING THE

PERFOMANCE OF MACHINE LEARNING ALGORITHMS

BY:

ZIA ULLAH SU-19-01-001-018

MUHAMMAD USMAN SU-19-01-001-013

SUPERVISED BY:
DR. MUHAMMAD ASIF KHAN

DEPARTMENT OF COMPUTER SCIENCE & IT

SARHAD UNIVERSITY OF SCIENCE AND INFORMATION

TECNOLOGY PESHAWAR

2019-2023

i
 CERTIFICATE OF APPROVAL

This is to certify that this Research report, entitled “A novel imputation model for improving
the performance of machine learning algorithms” was conducted by Zia Ullah and
Muhammad Usman under the supervision of Dr. Muhammad Asif Khan. This FYP report is
submitted to the Department of Computer Science & IT, Sarhad University of Science &
Information Technology in the partial fulfillment of the requirements for the degree of
Bachelor of Computer Science.

Examination Committee

1. Internal Supervisor:

Name:………………………….. Signature:………………………

2. Project Supervisor:

Name:…………………………… Signature:…………………….

i
 DEDICATION

We dedicate this achievement to our parents & supervisor as well as our colleagues,
who encouraged and supported us during whole tenure. Without their support and
sincere advises, it could not possible to complete within given period.

ii
i
 PLAGIARISM UNDERTAKING

We Zia Ullah and Muhammad Usman hereby declare that content of this FYP report is our
own work and that it is the result of work done during the period of registration. To the best
of our knowledge, it contains no material previously published or written by another person
nor material which to a substantial extent has been accepted for the award of any other
degree or diploma of the university or other institute of higher learning, except where due
acknowledgement has been made in the text.

Signed:……………………………… Signed:……………………….
Zia Ullah Muhammad Usman
Date: 30/Oct/2023 Date: 30/Oct/2023

iii
i
 ACKNOWLEDGMENT

We are thankful to the Department of CS&IT of Sarhad University of Science and

Information Technology Peshawar for their kind encouragement in doing this project and for
completing it [Link] are grateful to them.
We convey our thanks to Dr Muhammad Asif Khan for providing us the proper guidance,
necessary support and information at the right time during the progressive reviews. We wish
to express our thanks to all Teaching and Non-teaching staff members of the Department of
CS&IT who were helpful in many ways for the completion of the project

iv
i
 ABSTRACT
Missing data are unavoidable in the real-world application of machine learning, and their
non-optimal processing may decrease the quality of data-driven models. Handling missing
values in given data is a hot research area. We proposed a novel imputation model to handle
missing values and outliers in the data. The proposed model out performs existing approaches
in term of accuracy and other standard performance measure.

v
i
 List of Tables

Table1.1: Dataset frequency:………………………………….…..………………….….….34

Table1.2: Frequency for the weather conditions:……………….…………...….…..………34

Table 1.3: Likelihood weather condition:…………………………………….……..….…...35

vi
i
 List of Figures

Figure 1.1: Histogram:…………………………………………………………………….………….5

Figure 1.2:Box plots:…………………………………………….………………....………..5
Figure 1.3: Heatsmaps for missing data:……………………………….……………….……………6
Figure 1.4:Pair Plots:………………………………………………………………….……………6
Figure 1.5: Correlation Matrix:……………………………………...………………….…………7
Figure 1.6:Time Series Plots:…………………………………………..……………….…………7
Figure 1.7:Bar Charts for Categorical Data:………………….…………………….……………8
Figure 1.8:General structure of a decision tree:…………………………….…...…..………23
Figure 1.9: Decision tree:…………………………………………………….…....……...…25
Figure 1.10: KNN Classification:……………………………….…………….…….…...….29
Figure 1.11 : Classification for new data points:………………….……………….………..29
Figure 1.12: Working of KNN:……………………………………….………….….………30
Figure 1.13: Euclidean distance between A1 and B2:………………….………….…..……31
Figure 1.14: KNN Algorithm:…………………………………………...…………………..31
Figure 3.1: The schematic diagram of the proposed ImputeNew…….………………………….55
Figure 4.1: Accuracy using decision tree(j48):……………...……….………….…..………59
Figure 4.2: Accuracy using KNN (IBK):……….…………………..………………….……60
Figure 4.3: Accuracy using BayesNet:……….…….………………………………..………60
Figure 4.4: Accuracy using RandomForest:….………………………………..…….………61

vii
i
 Table of Contents

CHAPTER 1: 1INTRODUCTION.........................................................................................1
1.1 Understanding Missing Data................................................................................................1
1.2 Key considerations................................................................................................................2
1.3 Exploratory Data Analysis (EDA)........................................................................................3
1.4 Visualizing Missing Data.....................................................................................................3
1.5 Handling Outliers and Anomalies........................................................................................4
1.6 Traditional Imputation Methods...........................................................................................8
1.7 Advanced Imputation Techniques........................................................................................9
1.8 Handling Missing Categorical Data:...................................................................................11
1.9 Mean Absolute Error (MAE) or Mean Squared Error (MSE)............................................12
1.10 Best Practices and Considerations....................................................................................14
1.1 Time-Series Data.................................................................................................................17
1.12 Temporal Ordering............................................................................................................17
1.13 Common Time-Series Analysis Techniques.....................................................................18
1.14 Challenges in Time-Series Analysis.................................................................................18
1.15 Dealing with Time-Series Data.........................................................................................19

1.16 Different Machine Learning Algorithm............................................................................22

1.17 Attribute Selection Measures............................................................................................25

1.18 Pruning: Getting an Optimal Decision tree.......................................................................26

1.19 Implementation of Decision Tree.....................................................................................27

1.20 K-Nearest Neighbor (KNN) Algorithm for Machine Learning........................................27

1.21 Naïve Bayes Classifier Algorithm....................................................................................32

1.22 Bayes' Theorem:................................................................................................................32

CHAPTER 2: BACKGROUND.............................................................................................38

2.1 Litrature Review........................................................................................................................................38

viii
i
CHAPTER 3: A NOVEL DATA IMPUTATION MODEL...............................................54

3.1 Introduction.........................................................................................................................54
3.2 Conceptual Framework of Proposed Novel Data Imputation Model (ImputeNew)...........54
3.3 Components of ImputeNew................................................................................................54
3.4 Understanding the Imputation Formula: Step-by-Step Explanation...................................56
3.5 Advantages of ImputeNew..................................................................................................56
3.6 Practical Application of ImputeNew...................................................................................57
CHAPTER 4: RESULTS AND DISCUSSION....................................................................58
4.1 Imputation
Formula.............................................................................................................58
4.2 Methodology.......................................................................................................................58
4.3 Imputation Methods............................................................................................................58
CHAPTER 5: CONCLUTION AND FUTURE WORK.....................................................62
5.1
Conclution....................................................................................................................................................62
5.2 Future Work................................................................................................................................................62
REFRENCES..........................................................................................................................63

ix
i
 CHAPTER 1
INTRODUCTION

Data preprocessing is a crucial step in the data analysis and machine learning pipeline. It
involves cleaning and transforming raw data into a format that is suitable for analysis or
model training. One common challenge in real-world datasets is the presence of missing
values. Missing data can arise due to various reasons such as data entry errors, sensor
malfunctions, or simply the absence of information.
Imputation, the process of estimating or filling in missing values, is a fundamental
aspect of data preprocessing. The goal of imputation is to enhance the quality of the dataset
by providing a more complete and accurate representation of the underlying information.
However, imputing missing values requires careful consideration to avoid introducing biases
or distorting the true nature of the data.
In this chapter, we will delve into the methods and techniques for handling missing
values during the data preprocessing phase. We will explore both traditional and advanced
imputation strategies, discussing their strengths, limitations, and suitable scenarios for
application. Additionally, we will address the importance of understanding the nature of
missingness and how it can impact the validity of imputation methods.

1.1 Understanding Missing Data

Understanding missing data is crucial for making informed decisions when handling
and imputing missing values. Missing data can occur for various reasons, and understanding
the nature of missingness helps in choosing appropriate imputation methods and interpreting
results accurately. There are three main types of missing data:

1.1.1 Missing Completely at Random (MCAR)

Definition: The missingness of data is unrelated to the observed or unobserved
values. It occurs randomly and is not dependent on any variable, observed or unobserved.
Implications: When data is MCAR, the missing values are a random subset of the data
and are not systematically related to other variables. This simplifies the imputation process
and allows for the use of simpler imputation methods without introducing bias.
In MCAR, the probability of missing data is the same for all observations, and it is unrelated
to both observed and unobserved values. It occurs purely by chance.

1
Example: Imagine conducting a survey on a random sample of people, and due to an error in
data entry, some responses are missing. If the likelihood of a response being missing is the
same for all individuals, regardless of their answers to survey questions, it is MCAR.

1.1.2 Missing at Random (MAR)

In MAR, the probability of missing data depends on observed values but not on
unobserved values. The missingness is systematic and related to other variables in the dataset.
Example: Consider a study on income where people with higher incomes are less likely to
disclose their exact earnings. Here, the likelihood of income being missing depends on other
observed variables like education or occupation.

1.1.3 Missing Not at Random (MNAR)

In MNAR, the probability of missing data depends on unobserved values, and the
missingness is not predictable based on the observed data alone.
Example: Consider a clinical trial where patients with severe side effects of a drug
are less likely to report their symptoms accurately. Here, the likelihood of reporting side
effects depends on the severity of those unobserved symptoms.
Understanding the type of missingness in your data helps in choosing appropriate
imputation methods and mitigating biases introduced during the analysis. Each type of
missing data requires a tailored approach for imputation or handling in statistical analyses.

1.2 Key considerations

Patterns of Missing Data: Examining the patterns of missing data can provide insights
into the type of missingness. For example, if certain variables consistently have missing
values together, it may indicate a specific pattern.

1.2.1 Missing Data Mechanisms

Understanding whether missing data follows MCAR, MAR, or MNAR informs the
choice of imputation methods. Analyzing the relationships between missingness and
observed variables helps identify the mechanism.
1.2.2 Impact on Analysis
Consideration of the impact of missing data on the intended analysis is essential.
Ignoring missing values or using inappropriate imputation methods can lead to biased results
and erroneous conclusions.

2
 In summary, understanding the nature of missing data is a critical first step in the data
preprocessing pipeline. It guides the selection of appropriate imputation techniques, helps
avoid biases in subsequent analyses, and ensures the reliability of results obtained from
datasets with missing values.

1.3 Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is a crucial phase in the data analysis process that
involves visually and statistically exploring the characteristics of a dataset. It helps analysts
and data scientists gain insights into the structure, patterns, and potential issues within the
data. When dealing with missing values, EDA plays a vital role in understanding the
distribution and patterns of missing data. Here are key aspects of EDA related to missing
data.

1.3.1 Identifying Patterns and Trends in Missing Data

Visual Inspection: Use graphical tools such as heatmaps, where missing values are
represented by distinct colors, to visualize the distribution of missing data across variables.
Summary Statistics: Calculate the percentage of missing values for each variable to
understand the extent of missingness.

1.4 Visualizing Missing Data

Missingness Heatmaps: Create heatmaps that visually represent missing values in the
dataset. These heatmaps can help identify clusters of missing values and patterns within the
data.
Bar Charts: Use bar charts to display the percentage of missing values for each
variable. This can provide a quick overview of the completeness of the dataset.

1.4.1 Handling Missing Values in Relation to Other Variables

Correlation Analysis: Explore correlations between missing values in different
variables. Understanding relationships between missing data in multiple variables can inform
the imputation strategy.
Scatter Plots: Create scatter plots to visualize relationships between variables with
missing values and other relevant variables. This can help identify potential patterns or
dependencies.

3
1.4.2 Temporal Analysis for Time-Series Data
Time Series Plots: For time-series data, analyze missing values over time.
Understanding whether missingness follows a temporal pattern is crucial for selecting
appropriate imputation methods.

1.4.3 Comparing Missingness Across Subgroups

Grouped Analysis: Explore missing data patterns across different subgroups or
categories. This is particularly relevant when dealing with categorical variables, as
missingness may vary across groups.

1.5 Handling Outliers and Anomalies

Outlier Detection: Explore the presence of outliers or anomalies in the data, as they
may be related to missing values. Addressing outliers before imputation can improve the
imputation process.

1.5.1 Data Imputation Validation

Impute and Compare: Impute missing values using chosen imputation methods and
compare the imputed dataset with the original dataset. Assess whether imputed values align
with expectations and maintain the integrity of the data.
By incorporating these EDA techniques, analysts can develop a comprehensive
understanding of the missing data patterns and make informed decisions about the most
appropriate imputation strategies. EDA not only facilitates the handling of missing values but
also provides valuable insights into the overall quality and characteristics of the dataset.
Exploratory Data Analysis (EDA) involves various visualizations and analyses to understand
the structure and patterns within a dataset.

a. Histograms
Histograms provide a visual representation of the distribution of a [Link] shown
in figure 1.1

4
 Figure 1.1: Histogram

b .Box Plots
Box plots help visualize the distribution and identify outliers in numerical [Link]
shown in Figure 1.2

Figure 1.2: Box plots

5
c .Heatmaps for Missing Data
A heatmap can be used to visualize missing values in a [Link] shown in Figure 1.3

Figure1.3: Heatsmaps for missing data

d. Pair Plots
Pair plots visualize relationships between multiple variables in a [Link] shown in
Figure 1.4

Figure 1.4: Pair Plots

6
e. Correlation Matrix
Correlation matrices provide insights into the relationships between [Link]
shown in Figure 1.5

Figure 1.5: Correlation Matrix

f. Time Series Plots

For time-series data, visualize trends and patterns over [Link] shown in Figure 1.6

Figure 1.6: Time Series Plots

7
g. Bar Charts for Categorical Data
Bar charts help visualize the distribution of categorical variables.
These examples demonstrate how EDA techniques can be applied to gain insights into
different aspects of the dataset. Customize these examples based on your specific dataset and
research questions. As shown in Figure 1.7

Figure 1.7: Bar Charts for Categorical Data

1.6 Traditional Imputation Methods

Traditional imputation methods involve filling in missing values with estimated or
calculated values based on the available data. Here are some common traditional imputation
methods:

1.6.1 Mean, Median, or Mode Imputation

Method: Replace missing values with the mean, median, or mode of the observed
values for the respective variable.
Use Case: Applicable for numerical variables with a relatively symmetric distribution.

1.6.2 Forward and Backward Filling

Method: Propagate the last observed value forward to fill missing values (forward
filling) or use the next observed value to fill missing values (backward filling).
Use Case: Suitable for time-series data where values are often consecutive.

8
1.6.3 Linear Interpolation
Method: Interpolate missing values based on a linear relationship between observed
values.
Use Case: Applicable when values follow a trend and have a linear relationship.

1.6.4 Regression Imputation

Method: Predict missing values using regression models based on other variables.
Use Case: Suitable when a relationship exists between the variable with missing values and
other observed variables.

1.6.5 Random Imputation

Method: Replace missing values with random values from the distribution of
observed values.
Use Case: Applicable when the missingness is completely at random.
It's important to note that these traditional imputation methods have their strengths and
limitations. The choice of method depends on the nature of the data, the missing data
mechanism, and the research question at hand. Additionally, imputation should be performed
cautiously, considering potential biases introduced by the chosen method.

1.7 Advanced Imputation Techniques

Advanced imputation techniques go beyond traditional methods and leverage more
sophisticated approaches, often involving statistical models or machine learning algorithms.
These methods are particularly useful when dealing with complex relationships or datasets
with non-random missingness. Here are some advanced imputation techniques:

1.7.1 Multiple Imputation

Method: Generate multiple plausible values for each missing data point, resulting in
multiple complete datasets. Perform analyses on each imputed dataset and combine the
results.
Strengths:
a) Accounts for uncertainty associated with imputation.
b) Suitable for various missing data mechanisms.

9
Limitations:
a) Requires assumptions about the missing data mechanism.
b) Computationally more intensive.

1.7.2 K-Nearest Neighbors (KNN) Imputation

Method: Impute missing values by considering the values of their K-nearest neighbors
in the observed data.
Strengths:
a) Captures non-linear relationships.
b) Adapts well to the local structure of the data.
Limitations:
a) Sensitive to the choice of the number of neighbors (K).
b) Computationally more intensive.

1.7.3 Matrix Factorization (e.g., Singular Value Decomposition)

Method: Decompose the data matrix into latent factors and impute missing values based
on the product of these factors.
Strengths:
a) Effective for high-dimensional data.
b) Captures underlying patterns.
Limitations:
a) Assumes linearity and may not capture complex relationships.
b) Sensitive to the choice of the number of latent factors.

1.7.4 Deep Learning-based Imputation

Method: Use neural networks to predict missing values based on observed data.
Strengths:
a) Captures complex relationships.
b) Can handle non-linear patterns.
Limitations:
a) Requires a sufficient amount of data.
b) Computationally intensive and may require tuning.

10
1.7.5 Miss Forest
Method Random Forest-based imputation that builds a separate forest for each
variable with missing values.
Strengths:
a) Handles non-linear relationships.
b) Robust to outliers.
Limitations:
a) Computationally intensive.
These advanced imputation techniques offer more flexibility and sophistication in handling
missing data. However, it's important to carefully consider the specific characteristics of the
dataset and the assumptions of each method. Additionally, model performance should be
evaluated, and results should be interpreted in the context of the research question.

1.8 Handling Missing Categorical Data

Handling missing categorical data requires specific techniques tailored to the nature
of categorical variables. Here are some common approaches to deal with missing categorical
data:

1.8.1 Mode Imputation

Method Replace missing values with the mode (most frequent category) of the
variable.
Use Case Applicable when the missingness is assumed to be random and not
associated with other variables.

1.8.2 Creating a New Category

Method Introduce a new category explicitly representing missing values.

Use Case Useful when missingness is not random and there might be information in
the missing values.

1.8.3 Using Predictive Models

Method Train a predictive model to predict the missing categorical values based on
other variables.
Use Case Suitable when relationships between variables can be modeled.

11
1.8.4 Probabilistic Imputation

Method: Assign missing values probabilistically based on the distribution of

observed values.
Use Case: Suitable when there is uncertainty about the imputed values.

1.8.5 Using the Previous/Next Value

Method: Fill missing values with the previous or next observed value.
Use Case: Applicable when categorical values tend to remain stable over consecutive
observations.

1.8.6 Using Cross-Tabulation

Method: Utilize cross-tabulation with other variables to estimate missing values.

Use Case: Effective when there are strong associations between categorical variables.

When handling missing categorical data, it's important to choose a method based on the
characteristics of the dataset and the nature of missingness. The choice should be guided by
the research context and the potential impact of imputation on subsequent analyses.

1.8.7 Evaluation of Imputation Methods

The evaluation of imputation methods is crucial to assess their performance and

choose the most suitable approach for handling missing data. Here are some common
evaluation metrics and considerations for assessing the effectiveness of imputation methods:

1.9 Mean Absolute Error (MAE) or Mean Squared Error (MSE)

Metric: MAE or MSE can be used to measure the difference between the imputed
values and the true values for the observed data.
Considerations: Lower MAE or MSE values indicate better imputation accuracy.

1.9.1 Root Mean Squared Error (RMSE)

Metric: RMSE is a variant of MSE that provides the square root of the average
squared differences between imputed and true values.
Considerations: Like MAE/MSE, lower RMSE values are indicative of better
imputation accuracy.

12
1.9.2 Correlation Coefficient

Metric: Measure the correlation between imputed and true values to assess the degree
of linear association.
Considerations: Higher correlation coefficients suggest better imputation accuracy.

1.9.3 Proportion of Missing Information (PMI)

Metric: PMI measures the proportion of missing information in the imputed dataset
compared to the complete dataset.
Considerations: Lower PMI values indicate better imputation performance.

1.9.4 Imputation Accuracy by Variable

Metric: Assess imputation accuracy separately for each variable, comparing imputed
and true values.
Considerations: Evaluate the performance of imputation methods on individual
variables to identify potential variable-specific challenges.

1.9.5 Cross-Validation

Approach: Implement cross-validation to evaluate imputation methods on multiple

subsets of the data, ensuring robustness and generalizability.
Considerations: Cross-validation helps assess how well the imputation method
performs on different data partitions.

1.9.6 Sensitivity Analysis

Approach: Conduct sensitivity analyses by varying assumptions or parameters in the

imputation process.
Considerations: Sensitivity analyses provide insights into the robustness of
imputation methods under different conditions.

1.9.7 Comparison with Baseline Methods

Approach: Compare the performance of advanced imputation methods with baseline
methods (e.g., mean imputation) to assess improvement.
Considerations: Understanding the relative performance of advanced methods
compared to simpler approaches is essential.

13
1.9.8 Domain-Specific Considerations

Considerations: Take into account domain-specific requirements and knowledge.

Some imputation methods may be more suitable for certain types of data or missing data
mechanisms.

1.9.9 Visual Inspection

Approach: Visualize the imputed values against the true values using plots, such as
scatter plots or time-series plots.
Considerations: Visual inspection can provide a qualitative assessment of the
imputation accuracy and reveal potential patterns or outliers.

1.9.10 Imputation Impact on Downstream Analyses

Approach: Evaluate the impact of imputation on the results of downstream analyses

(e.g., regression models, clustering).
Considerations: Assess how imputation choices influence the validity and reliability
of subsequent analyses.
When evaluating imputation methods, it's essential to consider the specific context of the
dataset, including its characteristics, missing data mechanisms, and the goals of the analysis.
Combining multiple evaluation metrics and methods provides a comprehensive
understanding of imputation performance.

1.10 Best Practices and Considerations

Handling missing data is a critical step in the data preprocessing pipeline, and
effective imputation requires careful consideration of various factors. Here are some best
practices and considerations when dealing with imputation:

1.10.1 Understand the Nature of Missing Data

Investigate the patterns and mechanisms of missing data (MCAR, MAR, MNAR).
Consider the implications of missingness for the analysis and results.

1.10.2 Explore Descriptive Statistics

Examine descriptive statistics before and after imputation to understand the impact on
central tendencies and variability.

14
Consider the distribution of imputed values compared to observed values.

1.10.3 Use Multiple Imputation for Uncertainty Estimation

Implement multiple imputation to account for the uncertainty associated with imputed
[Link] results from multiple imputed datasets for more robust analyses.
1.10.4 Evaluate Imputation Performance

Employ appropriate evaluation metrics (MAE, MSE, RMSE, correlation) to assess the
performance of imputation methods.
Consider domain-specific metrics if available.

1.10.5 Choose Imputation Methods Based on Data Characteristics

Select imputation methods based on the characteristics of the data, including variable
types, distribution, and relationships.
Choose methods that are suitable for the missing data mechanism.

1.10.6 Consider Variable-Specific Imputation Strategies

Tailor imputation strategies to the specific characteristics of each variable (e.g.,

numerical, categorical).
Utilize domain knowledge to guide imputation decisions.

1.10.7 Handle Time-Series Data Appropriately

Use time-aware imputation methods that preserve the temporal structure of the data.
Consider the impact of imputation on time-dependent analyses.

1.10.8 Be Transparent and Document Decisions

Clearly document the imputation methods applied, including any assumptions or

transformations made.
Provide details on the reasoning behind imputation choices.

1.10.9 Conduct Sensitivity Analyses

Perform sensitivity analyses to assess the robustness of imputation methods under

different conditions.
Vary assumptions or parameters to understand their impact on results.
15
1.10.10 Compare Advanced Methods with Baseline Approaches

Compare the performance of advanced imputation methods with simpler baseline

approaches (e.g., mean imputation) to assess added value.
Understand the trade-offs between complexity and performance.

1.10.11 Imputation in the Context of Downstream Analyses

Consider how imputation choices may influence the results of subsequent analyses.
Evaluate the impact of imputation on the validity and reliability of downstream
analyses.

1.10.12 Handle Imputation in Conjunction with Outlier Detection:

Address outliers before imputation if outliers are present in the dataset.

Outliers can significantly influence imputation results, especially in mean-based
methods.

1.10.13 Impute Missingness Indicators

Consider creating indicators for missingness in variables to account for the fact that
values are imputed.
This helps downstream analyses distinguish between observed and imputed values.

1.10.14 Regularly Update Imputation Strategies

Periodically revisit and update imputation strategies as more data becomes available
or as research questions change.
New data may provide additional information to improve imputation accuracy.

1.10.15 Validate Imputed Data

If possible, compare imputed data with external sources or expert knowledge to

validate the plausibility of imputed values.
This is particularly important when imputing categorical or domain-specific variables.
Remember that there is no one-size-fits-all solution for imputation, and the best approach
depends on the unique characteristics of the dataset and the goals of the analysis. Regularly
validate and document imputation decisions to ensure transparency and reproducibility in the

16
data analysis process.
1.11 Time-Series Data
Time-series data consists of observations or measurements collected and recorded
over successive points in time. This type of data is common in various fields, including
finance, economics, environmental science, and engineering. Time-series analysis involves
studying the patterns, trends, and behaviors within the data to make predictions or derive
meaningful insights. Here are key concepts related to time-series data:
Components of Time-Series Data:

1.11.1 Trend
The long-term movement or direction in the data, indicating a consistent upward,
downward, or stable pattern.

1.11.2 Seasonality
Regular, repeating fluctuations or patterns within a fixed time interval, often
associated with specific seasons, months, or days of the week.

1.11.3 Cyclic Patterns

Longer-term patterns that are not necessarily fixed to a specific time interval,
representing repetitive, non-seasonal fluctuations.

1.11.4 Irregularity or Noise

Unpredictable and random variations that cannot be attributed to the trend,
seasonality, or cyclic patterns.

Key Characteristics of Time-Series Data

1.12 Temporal Ordering

Observations are recorded in a chronological sequence, and the order of observations
is crucial for analysis.
1.12.1 Autocorrelation
The correlation between a time series and a lagged version of itself. It measures the
extent to which past observations influence future observations.

17
1.12.2 Stationarity
A time series is considered stationary if its statistical properties (mean, variance,
autocorrelation) remain constant over time. Stationarity is often assumed for modeling
purposes.
1.12.3 Seasonal Decomposition
Breaking down a time series into its individual components, such as trend,
seasonality, and residuals, to better understand its structure.

1.13 Common Time-Series Analysis Techniques

1.13.1 Moving Averages

Smoothing technique that calculates averages of subsets of consecutive data points to
identify trends.

1.13.2 Exponential Smoothing

A family of forecasting methods that assigns exponentially decreasing weights to past
observations.

1.13.3 Autoregressive Integrated Moving Average (ARIMA)

A popular time-series forecasting model that combines auto regression, differencing,
and moving averages.

1.13.4 Seasonal-Trend decomposition using LOESS (STL)

A method for decomposing time-series data into trend, seasonality, and residual
components using locally weighted scatterplot smoothing.

1.13.5 Long Short-Term Memory (LSTM)

A type of recurrent neural network (RNN) commonly used for modeling long-range
dependencies in time-series data.

1.14 Challenges in Time-Series Analysis

1.14.1 Non-Stationarity
Many real-world time series exhibit non-stationary behavior, which may require
transformations to achieve stationarity.

18
1.14.2 Outliers and Anomalies
Identification and handling of outliers or anomalies are critical for accurate modeling
and forecasting.

1.14.3 Variable Selection

Determining which variables to include in a time-series model, considering potential
lagged effects and interactions.

1.14.4 Model Evaluation

Evaluating the performance of time-series models often involves using metrics such
as Mean Absolute Error (MAE), Mean Squared Error (MSE), or Root Mean Squared Error
(RMSE).

1.14.5 Handling Missing Data

Time-series data may contain missing values, and imputation techniques should be
chosen carefully to preserve temporal patterns.
Understanding the characteristics and nuances of time-series data is essential for selecting
appropriate analysis techniques and building accurate models. The choice of modeling
approach depends on the specific patterns observed in the data and the goals of the analysis.

1.15 Dealing with Time-Series Data

Dealing with time-series data involves several steps to ensure proper analysis,
modeling, and interpretation. Here is a comprehensive guide on how to handle time-series
data:

1.15.1 Data Exploration and Visualization

Plot the time series to visually inspect trends, seasonality, and any apparent patterns.
Examine summary statistics to understand the central tendency and variability of the
data.

1.15.2 Temporal Aggregation

Consider aggregating the data to a lower frequency (e.g., daily to monthly) for a
broader perspective and to reduce noise.

1.15.3 Check for Stationarity

19
 Perform tests for stationarity, such as the Augmented Dickey-Fuller (ADF) test.
If non-stationary, apply differencing or transformations to achieve stationarity.

1.15.4 Decompose the Time Series

Decompose the time series into its components (trend, seasonality, and residuals)
using methods like STL or seasonal decomposition of time series (STL).

1.15.5 Handling Missing Values

Address missing values using appropriate imputation methods, considering the
temporal structure of the data.
Avoid imputing missing values in a way that introduces bias or disrupts temporal
patterns.

1.15.6 Feature Engineering

Create lag features to capture temporal dependencies.
Extract additional time-related features, such as day of the week, month, or season.

1.15.7 Model Selection

Choose a modeling approach based on the characteristics of the time series.
Common models include ARIMA, SARIMA, Prophet, and machine learning models
like LSTM.

1.15.8 Train-Test Split

Split the data into training and testing sets, ensuring that the testing set follows the
training set in time.

1.15.9 Parameter Tuning

Perform hyper parameter tuning for chosen models to optimize performance.
Consider using techniques like grid search or random search.

1.15.10 Model Training

Train the selected model on the training set, accounting for seasonality and any other
temporal patterns.

20
1.15.11 Model Evaluation
Evaluate the model on the test set using appropriate metrics (MAE, MSE, RMSE).
Consider visualizing predicted values against actual values.

1.15.12 Error Analysis

Analyze errors to understand where the model performs well and where it struggles.
Identify any systematic biases or patterns in the residuals.

1.15.13 Forecasting
Use the trained model to make forecasts for future time points.
Assess the uncertainty of forecasts and provide confidence intervals if possible.

1.15.14 Monitoring and Updating

Regularly monitor model performance and update the model as new data becomes
available.
Adjust model parameters or retrain if patterns in the time series change.

1.15.15 Consider External Factors

Incorporate external factors (covariates) that may influence the time series.
Ensure that external factors are aligned with the temporal structure.

1.15.16 Documentation
Document all preprocessing steps, modeling choices, and parameter configurations
for reproducibility.
Clearly explain any assumptions or transformations applied to the data.

1.15.17 Interpretation
Interpret the results in the context of the specific domain and objectives.
Understand the implications of the model's predictions for decision-making.

1.15.18 Regular Updates

Revisit the time-series analysis periodically to incorporate new data and update
models accordingly.
Adjust forecasting strategies based on evolving patterns.

21
1.15.19 Seek Expert Advice
Consult with experts in time-series analysis or domain-specific knowledge for
validation and additional insights.
By following these steps, you can effectively handle and analyze time-series data,
providing valuable insights and predictions. The specific approach may vary based on the
characteristics of the data and the goals of the analysis.

1.16 Different Machine Learning Algorithm

1.16.1 Decision Tree Classification Algorithm

o Decision Tree is a Supervised learning technique that can be used for

 Both classification and Regression problems, but mostly it is preferred for

solving Classification problems.
o It is a tree-structured classifier, where

 internal nodes represent the features of a dataset,

 branches represent the decision rules and
 Each leaf node represents the outcome.
o In a Decision tree, there are two nodes, which are the

 Decision Node and

 Leaf Node.
 Decision nodes are used to make any decision and have multiple branches,
whereas
 Leaf nodes are the output of those decisions and do not contain any further
branches.
o The decisions or the test are performed on the basis of features of the given dataset.

o Decision tree is a graphical representation for getting all the possible solutions to a
problem/decision based on given conditions.
 It is called a decision tree because, similar to a tree,
 That starts with the root node, which expands on further branches and
constructs a tree-like structure.
o In order to build a tree, we use the CART algorithm, which

 stands for Classification and Regression Tree algorithm.

22
 o A decision tree simply asks a question, and based on the answer (Yes/No), it further
split the tree into sub-trees.

Figure 1.8: General structure of a decision tree

Note: A decision tree can contain categorical data (YES/NO) as well as numeric data.

Why use Decision Trees?

There are various algorithms in Machine learning, so choosing the best algorithm for the
given dataset and problem is the main point to remember while creating a machine learning
model. Below are the two reasons for using the Decision tree:

o Decision Trees usually mimic human thinking ability while making a decision, so it is
easy to understand.
o The logic behind the decision tree can be easily understood because it shows a tree-
like structure.

1.16.2 Decision Tree Terminologies

 Root Node: Root node is from where the decision tree starts. It represents the entire
dataset, which further gets divided into two or more homogeneous sets.

 Leaf Node: Leaf nodes are the final output node, and the tree cannot be segregated
further after getting a leaf node.

 Splitting: Splitting is the process of dividing the decision node/root node into sub-
nodes according to the given conditions.

23
  Branch/Sub Tree: A tree formed by splitting the tree.

 Pruning: Pruning is the process of removing the unwanted branches from the tree.

 Parent/Child node: The root node of the tree is called the parent node, and other
nodes are called the child nodes.

How does the Decision Tree algorithm Work?

1. In a decision tree, for predicting the class of the given dataset, the algorithm starts from
the root node of the tree.
2. This algorithm compares the values of root attribute with the record (real dataset)
attribute and, based on the comparison, follows the branch and jumps to the next node.
3. For the next node, the algorithm again compares the attribute value with the other sub-
nodes and move further.
4. It continues the process until it reaches the leaf node of the tree.
5. The complete process can be better understood using the below algorithm:

o Step-1: Begin the tree with the root node, says S, which contains the complete
dataset.
o Step-2: Find the best attribute in the dataset using Attribute Selection Measure
(ASM).
o Step-3: Divide the S into subsets that contains possible values for the best attributes.

o Step-4: Generate the decision tree node, which contains the best attribute.

o Step-5: Recursively make new decision trees using the subsets of the dataset created
in step-3. Continue this process until a stage is reached where you cannot further
classify the nodes and called the final node as a leaf node.

Example: Suppose there is a candidate who has a job offer and wants to decide whether he
should accept the offer or Not.

1. So, to solve this problem, the decision tree starts with the root node (Salary attribute by
ASM).
2. The root node splits further into the next decision node (distance from the office) and one
leaf node based on the corresponding labels.
3. The next decision node further gets split into one decision node (Cab facility) and one

24
 leaf node.
4. Finally, the decision node splits into two leaf nodes (Accepted offers and Declined offer).

Consider the below diagram:

Figure 1.9: Decision tree

1.17 Attribute Selection Measures

While implementing a Decision tree, the main issue arises that how to select the best
attribute for the root node and for sub-nodes. So, to solve such problems there is a technique
which is called as Attribute selection measure or ASM. By this measurement, we can easily
select the best attribute for the nodes of the [Link] are two popular techniques for ASM,
which are:

i Information Gain

ii Gini Index

1.17.1 Information Gain

o Information gain is the measurement of changes in entropy after the segmentation of a

dataset based on an attribute.
o It calculates how much information a feature provides us about a class.

o According to the value of information gain, we split the node and build the decision

25
 tree.
o A decision tree algorithm always tries to maximize the value of information gain, and
a node/attribute having the highest information gain is split first.
o It can be calculated using the below formula:

Information Gain = Entropy(S) - [(Weighted Avg.) * Entropy (each feature)

Entropy: Entropy is a metric to measure the impurity in a given attribute. It specifies

randomness in data. Entropy can be calculated as:

Entropy(s) = -P(yes)log2 P(yes)- P(no) log2 P(no)

Where,

o S= Total number of samples

o P(yes)= probability of yes

o P(no)= probability of no

1.17.2 Gini Index

o Gini index is a measure of impurity or purity used while creating a decision tree in the
CART (Classification and Regression Tree) algorithm.
o An attribute with the low Gini index should be preferred as compared to the high Gini
index.
o It only creates binary splits, and the CART algorithm uses the Gini index to create
binary splits.
o Gini index can be calculated using the below formula:

Gini Index = 1- ∑jPj2

1.18 Pruning: Getting an Optimal Decision tree

Pruning is a process of deleting the unnecessary nodes from a tree in order to get the optimal
decision tree.

26
 A too-large tree increases the risk of Over fitting, and a small tree may not capture all the
important features of the dataset.
 Therefore, a technique that decreases the size of the learning tree without reducing
accuracy is known as Pruning.
 There are mainly two types of tree pruning technology used:

o Cost Complexity Pruning

o Reduced Error Pruning

1.18.1 Advantages of the Decision Tree

o It is simple to understand as it follows the same process which a human follow while
making any decision in real-life.
o It can be very useful for solving decision-related problems.

o It helps to think about all the possible outcomes for a problem.

o There is less requirement of data cleaning compared to other algorithms.

1.18.2 Disadvantages of the Decision Tree

o The decision tree contains lots of layers, which makes it complex.

o It may have an Over fitting issue, which can be resolved using the Random Forest
algorithm.
o For more class labels, the computational complexity of the decision tree may increase.

1.19 Implementation of Decision Tree

Implementation of the Decision tree steps will be:

o Data Pre-processing step

o Fitting a Decision-Tree algorithm to the Training set

o Predicting the test result

o Test accuracy of the result (Creation of Confusion matrix)

o Visualizing the test set result.

1.20 K-Nearest Neighbor (KNN) Algorithm for Machine Learning

27
 o K-Nearest Neighbour is one of the simplest Machine Learning algorithms based on
Supervised Learning technique.
o K-NN algorithm assumes the similarity between the new case/data and available cases
and put the new case into the category that is most similar to the available categories.
o K-NN algorithm stores all the available data and classifies a new data point based on
the similarity.
 This means when new data appears then it can be easily classified into a well
suite category by using K- NN algorithm.
o K-NN algorithm can be used for Regression as well as for Classification but mostly it
is used for the Classification problems.
o K-NN is a non-parametric algorithm, which

 means it does not make any assumption on underlying data.

o It is also called a lazy learner algorithm because it does not learn from the training set
immediately instead it stores the dataset and at the time of classification, it performs
an action on the dataset.
o KNN algorithm at the training phase just stores the dataset and when it gets new data,
then it classifies that data into a category that is much similar to the new data.

Example: Suppose, we have an image of a creature that looks similar to cat and dog, but we
want to know either it is a cat or dog.

 So for this identification, we can use the KNN algorithm, as it works on a similarity
measure.
 The KNN model will find the similar features of the new data set to the cats and dogs
images and based on the most similar features it will put it in either cat or dog
category.

28
 Figure 1.10: KNN Classification

Why do we need a K-NN Algorithm?

Suppose there are two categories, i.e., Category A and Category B, and we have a new data
point x1, so

o this data point will lie in which of these categories.

o To solve this type of problem, we need a K-NN algorithm.
o With the help of K-NN, we can easily identify the category or class of a particular
dataset.

Figure 1.11 : Classification for new data points

How does K-NN work?

29
The K-NN working can be explained on the basis of the below algorithm:

o Step-1: Select the number K of the neighbors

o Step-2: Calculate the Euclidean distance of K number of neighbors

o Step-3: Take the K nearest neighbors as per the calculated Euclidean distance.

o Step-4: Among these k neighbors, count the number of the data points in each
category.
o Step-5: Assign the new data points to that category for which the number of the
neighbor is maximum.
o Step-6: The model is ready.

Suppose we have a new data point and we need to put it in the required category.

Figure 1.12: Working of KNN

o Firstly, we will choose the number of neighbors, so we will choose the k=5.

o Next, we will calculate the Euclidean distance between the data points.

o The Euclidean distance is the distance between two points – already studied in
geometry.
o It can be calculated as written in below the diagram:

30
 Figure 1.13: Euclidean distance between A1 and B2
o By calculating the Euclidean distance we got the nearest neighbors.

Consider the below image:

Figure1.14: KNN Algorithm

o As we can see the 3 nearest neighbors are from category A, hence this new data point
must belong to category A.

How to select the value of K in the K-NN Algorithm?

Below are some points to remember while selecting the value of K in the K-NN algorithm:

o There is no particular way to determine the best value for "K", so we need to try some

31
 values to find the best out of them.
o The most preferred value for K is 5.

o A very low value for K such as K=1 or K=2, can be noisy and lead to the effects of
outliers in the model.
o Large values for K are good, but it may find some difficulties.

1.20.1 Advantages of KNN Algorithm

o It is simple to implement.

o It is robust to the noisy training data

o It can be more effective if the training data is large.

1.20.2 Disadvantages of KNN Algorithm

o Always needs to determine the value of K which may be complex some time.

o The computation cost is high because of calculating the distance between the data
points for all the training samples.

1.21 Naive Bayes Classifier Algorithm

o Naive Bayes algorithm is a supervised learning algorithm, which is based on Bayes

theorem and used for solving classification problems.
o It is mainly used in text classification that includes a high-dimensional training
dataset.
o Naive Bayes Classifier is one of the simple and most effective Classification
algorithms which helps in building the fast machine learning models that can make
quick predictions.
o It is a probabilistic classifier, which means it predicts on the basis of the probability of
an object.
o Some popular examples of Naive Bayes Algorithm are spam filtration, Sentimental
analysis, and classifying articles.

32
Why is it called Naive Bayes?

The Naive Bayes algorithm is comprised of two words Naive and Bayes, Which can be
described as:

o Naive: It is called Naive because it assumes that the occurrence of a certain feature is
independent of the occurrence of other features.
 Such as if the fruit is identified on the bases of color, shape, and taste, then
red, spherical, and sweet fruit is recognized as an apple.
 Hence each feature individually contributes to identify that it is an apple
without depending on each other.
o Bayes: It is called Bayes because it depends on the principle of Bayes' Theorem.

1.22 Bayes' Theorem:

o Bayes' theorem is also known as Bayes' Rule or Bayes' law, which is used to
determine the probability of a hypothesis with prior knowledge.
 It depends on the conditional probability.
o The formula for Bayes' theorem is given as:

Where,

P(A|B) is Posterior probability: Probability of hypothesis A on the observed event B.

P(B|A) is Likelihood probability: Probability of the evidence given that the probability of a
hypothesis is true.

P(A) is Prior Probability: Probability of hypothesis before observing the evidence.

P(B) is Marginal Probability: Probability of Evidence.

Working of Naive Bayes' Classifier:

Working of Naive Bayes' Classifier can be understood with the help of the below example:

33
 Suppose we have a dataset of weather conditions and corresponding target variable
"Play".
 So using this dataset we need to decide that whether we should play or not on a particular
day according to the weather conditions.

So to solve this problem, we need to follow the below steps:

1. Convert the given dataset into frequency tables.

2. Generate Likelihood table by finding the probabilities of given features.
3. Now, use Bayes theorem to calculate the posterior probability.

Problem: If the weather is sunny, then the Player should play or not?

Table1.1: Dataset Frequency

Outlook Play

0 Rainy Yes

1 Sunny Yes

2 Overcast Yes

3 Overcast Yes

4 Sunny No

5 Rainy Yes

6 Sunny Yes

7 Overcast Yes

8 Rainy No

9 Sunny No

10 Sunny Yes

11 Rainy No

12 Overcast Yes
34

13 Overcast Yes
 Table1.2: Frequency for the weather conditions

Weather Yes No

Overcast 5 0

Rainy 2 2

Sunny 3 2

Total 10 5

Table 1.3 : Likelihood weather condition

Weather No Yes

Overcast 0 5 5/14= 0.35

Rainy 2 2 4/14=0.29

Sunny 2 3 5/14=0.35

All 4/14=0.29 10/14=0.71

1.22.1 Applying Bayes' theorem

P(Yes|Sunny)= P(Sunny|Yes)*P(Yes)/P(Sunny)

P(Sunny|Yes)= 3/10= 0.3

P(Sunny)= 0.35

P(Yes)=0.71

So P(Yes|Sunny) = 0.3*0.71/0.35= 0.60

P(No|Sunny)= P(Sunny|No)*P(No)/P(Sunny)

35
P(Sunny|NO)= 2/4=0.5

P(No)= 0.29

P(Sunny)= 0.35

So P(No|Sunny)= 0.5*0.29/0.35 = 0.41

So as we can see from the above calculation that P(Yes|Sunny)>P(No|Sunny)

Hence on a Sunny day, Player can play the game.

1.22.2 Advantages of Naive Bayes Classifier

o Naive Bayes is one of the fast and easy ML algorithms to predict a class of datasets.

o It can be used for Binary as well as Multi-class Classifications.

o It performs well in Multi-class predictions as compared to the other Algorithms.

o It is the most popular choice for text classification problems.

1.22.3 Disadvantages of Naive Bayes Classifier

o Naive Bayes assumes that all features are independent or unrelated, so it cannot learn
the relationship between features.

1.22.4 Applications of Naive Bayes Classifier

o It is used for Credit Scoring.

o It is used in medical data classification.

o It can be used in real-time predictions because Naive Bayes Classifier is an eager

learner.
o It is used in Text classification such as Spam filtering and Sentiment analysis.

1.22.5 Types of Naive Bayes Model

There are three types of Naive Bayes Model, which are given below:

o Gaussian

36
 The Gaussian model assumes that features follow a normal distribution. This means if
predictors take continuous values instead of discrete, then the model assumes that
these values are sampled from the Gaussian distribution.

o Multinomial

The Multinomial Naive Bayes classifier is used when the data is Multinomial
distributed. It is primarily used for document classification problems, it means a
particular document belongs to which category such as Sports, Politics, education, etc.

The classifier uses the frequency of words for the predictors.

o Bernoulli

The Bernoulli classifier works similar to the Multinomial classifier, but the predictor
variables are the independent Boolean's variables. Such as if a particular word is present or
not in a document. This model is also famous for document classification tasks.

37
 CHAPTER 2
BACKGROUND
2.1 Literature Review
Data mining is a contemporary method for resolving numerous challenging real-world
issues. This term, which is fairly self-explanatory, is well-known and frequently used. It
describes a process that changes as new technologies emerge. The most crucial phase in data
mining is data pre-processing, which guarantees the accuracy of the data and produces results
that can be trusted to produce judgments. Vive defines data preparation as the straightforward
conversion of raw data into an intelligible format. Figure 1 illustrates the main data pre-
processing processes, which include data cleansing, integration, transformation, data
reduction, and data discretization. Dealing with missing data is a crucial step in the pre-
processing of data. Data cleaning is the first step in the preprocessing of data, which includes
this process. In the initial stage of data pre-processing, procedures are used to identify data
that is insufficient, inaccurate, inconsistent, and corrupt and to change or remove it. [5] [1]
We describe missing data patterns and mechanisms in the section under "Missing
Data Patterns and Mechanisms" before discussing missing values. The section on "Missing
Values Approaches" empirically explores methods for handling missing values in the
literature and critically evaluates a number of implementations in various fields, with a heavy
emphasis on machine learning. We explore a number of performance metrics in the missing
values domain in the part titled "Performance metrics for missing data imputation," and we
analyze and evaluate the findings from earlier studies in the section titled "Computations."
The Iris data set was then used to create two machine learning algorithms in the section under
"Experimental evaluation on machine learning methods" before the outcomes were presented.
The "Conclusion and Future Work" section summaries the study and suggests possible lines
of inquiry for the future. [6] [2]
This study sought to systematically investigate the performance and applicability of
several machine learning algorithms commonly used in current researches under various data
missing scenarios, and whether these machine learning algorithms were more advantageous
than traditional methods, in order to provide basis and reference for the selection of suitable
missing data processing evaluation criteria for clinical decision making. [7] [3]
There are numerous approaches to imputation that aim to substitute estimates for the
missing data (Song&Sheppard, 2007). Imputation techniques can impact how well machine
learning algorithms work, but it is unknown how they will affect how fair the predictions that

38
are produced will be. We examine the effects of imputation algorithms on several fairness
and performance indicators in this paper in order to investigate the occurrence of missing data
in real-world applications and to answer the following research question: [8] [4]
Missing attribute values can influence data sets for a number of different reasons. Due
to their lack of relevance, some attribute values are not recorded. For instance, a doctor may
make a diagnosis without a patient undergoing certain tests, or a homeowner could be asked
to rate the quality of air conditioning even though their home was not fitted with a unit. These
empty attribute values are referred to as "do not care" conditions. [8]
This paper's main contribution is to guide applied researchers on how to deal with
missing data for supervised learning problems. First, we demonstrate how imputation
techniques can improve prediction accuracy when missing-data perturbation is present.
Second, we demonstrate how regularizing the classifier by include missing-data perturbation
before imputation can actually increase prediction accuracy. With missing-data perturbation
and k-nearest-neighbors (k-NN) imputation on the Adult dataset, we achieve results that are
on par with the state of the art. Finally, we demonstrate that when the data are not further
disrupted, classifiers trained on one-hot encoded data typically produce higher predicted
accuracy. On the Congressional Voting Records (CVRs) dataset, for instance, a
straightforward one-hot encoded random forest outperforms the state-of-the-art with no
missing-data perturbation. [10] [5]
One of the most agonize problems that the machine learning field faces is missing
values in datasets. Even while there is an enormous amount of data, the percentage of cases
with complete data may be rather small, and the majority of it is full of missing values, which
poses significant challenges for data mining and machine learning systems that are typically
not equipped to manage them. In addition to the fact that missing values reduce the quality of
the data, missing data are likely the most frequent cause of shortcomings and flaws in
learning systems. Therefore, managing missing values is thought to be a key component of
overall data quality. The worst thing that could possibly happen in machine learning is trying
to remove them and drastically reducing the amount of data one has accessible [1]. [11] [6]
Alternatives are frequently lacking in applications because the values they represent
for some decision attributes may be erroneous or missing. One can determine a probability
distribution over the potential suggestions when these values are represented by probability
distributions (Durban & Stewart, 2012). Combining MCDA and Decision under Uncertainty
approaches is an additional option [Dominick, 2006 & Gas-pars-Wieloch, 2015]. In this
work, we concentrate on the second sort of imperfection, where some qualities lack values for
39
some alternatives. There are essentially only two viable meanings in the MCDA context: In
the first, a missing value is a pertinent criterion that must be taken into consideration while
making the choice. As a result, the missing value is viewed as a wildly inaccurate rating. The
MCDA model propagates this uncertainty, and it may be examined whether it affects the
ability to distinguish between alternatives [Lahdelma-et-al, 1998]. In the second
interpretation, the missing value is viewed as an irrelevant criterion, meaning that when its
value is missing, the criterion is no longer important for this choice. The scenario is
demonstrated by the example below. [12] [7]
The goal of this research is to introduce the MIA implementation within BART for
binary classification, which may be used to classify datasets with missing values without the
requirement for imputing during the training phase, new data categorization, or dropping
incomplete cases. To test this model's efficacy in various situations and domains, both
simulated and real data were used. In comparison to BART and random forest models, which
operate on datasets completed by imputation, it is anticipated that BART.m offers more
flexibility in dealing with incomplete datasets and offers a higher level of classification
accuracy. [13] [8]
We seek to close this gap. On a large number of datasets with realistic missingness
circumstances, we compare a representative collection of imputation approaches with regard
to imputation quality and the effect on the predictive effectiveness of subsequent ML models.
We employ 69 completely observed datasets with numerical and categorical columns from
Open ML (Vanschoren-et-al; 2013) for our investigations. Each dataset has a downstream
ML task (regression, binary classification, or multi-class classification) attached to it. The
three missingness patterns (MCAR, MAR, and MNAR; see also Section 3) have various
fractions of missing data, which we artificially introduce in our studies. Then, in two
application situations, we evaluate both the imputation performance and impact on
downstream performance: 1) Test data with missing values, i.e. We test on corrupted data
because 1) we train on complete data, corrupt (and impute) only test data, and 2) both the
training and test data have missing values. [14] [9]
In order to replace missing data and compare accuracy, this study compares three
machine learning classifiers, decision tree, KNN, and Bayesian network. The outcome
demonstrates that the Bayesian network has the lowest value for each of the three parameters,
indicating that it is the best method for imputed missing data. However, other elements, such
as the percentages of missing values and dataset sizes, can have an impact on these error
estimators. Although the Bayesian approach consistently yields the lowest values, the
40
findings are only noteworthy for datasets of modest size and less than 20% missing values.
[5]
The closest neighbour (K) must be picked when applying the imputation procedure. K
was set to a value depending on the outcomes of the experiment, starting at K=1 and stopping
at K=5, with K=4 serving as the most accurate estimation of K. The KNN imputation
algorithm made use of the Euclidean distance formula. The next step was to implement the
RF miss Forest algorithm, a non-parametric imputation technique based on the random forest.
To predict the missing variables for each variable, Forest fts a random forest on the observed
values. Miss-Forest's training and prediction processes are repeated until a certain number of
iterations have been completed. After three iterations, the miss-Forest came to an end. When
the difference between the previously imputed values and the freshly imputed data grew for
the first time with regard to both variable types, the iterative stopping criterion was reached.
The system was able to be trained using more iterations and better quality data than it had
previously anticipated. [6]
In various missing contexts, the performances of missing data processing systems
varied to some extent. Overall, machine learning outperformed conventional methods in
imputation, especially in cases when the amount of missing data was substantial. EL
performed better than individual machine learning techniques, with neural networks coming
in second. According to the MAR (the ratio of missing proportion 2:1) mechanism, EL was
the best suited method for missing imputation, and its average sensitivity, AUC, and Kappa
values were 0.908, 0.924, and 0.596, respectively. [7]
The findings in this article are intended to raise awareness of the effects of imputation
strategies and missing data in general in fair Machine Learning pipelines and assist
researchers in identifying (perhaps) effective imputation strategies that perform best in
specific contexts. [8]
The second group of approaches to handle missing attribute values, in which missing
attribute values are taken into consideration during the main learning process, is exemplified
by a modification of the LEM2 (Learning from Examples Module, version 2) rule induction
algorithm. In this method, rules are induced from the original data set with missing attribute
values treated as "do not care" conditions or lost values. [9] [10]
The findings demonstrate that, in the context of missing-data disturbance, imputation
techniques can improve forecast accuracy. One-hot encoded models that were trained without
any perturbation outperform imputed models that were trained on non-perturbed data for both
datasets. In the case of the Adult dataset, imputation unmistakably enhances accuracy when
41
M CAR-perturbed data are present. The three classifiers trained on the one-hot encoded
CVRs dataset, however, all exhibit decent performance at various levels of perturbation.
When the classifiers are trained using MNAR-perturbed data, the basic pattern of
results is maintained.[10]
In this experiment, the most common value, the most common value constrained to a
concept, and the delete technique are some more popular approaches for handling the
problem of missing values. In this experiment, a predetermined fraction of the values from
the original datasets are changed at random with missed values. These are the percentages:
20%, 30%, and 50%. The experiment is run five times, and the average is taken into account
when calculating the results. In this study, ILA is applied to datasets with varying percentages
of missing values, and the outcomes are compared to those obtained when ILA4 is applied to
the same datasets. The results of applying ILA4 to the three datasets are shown in TABLE
IV, while the results of applying ILA to the same datasets—Monk1, Vote, and Balance—with
each of the three methods for handling the problem of missing values—the most common
value, the most common value restricted to a concept, and the delete strategy—are shown in
TABLES V, VI, and VII. [11]
For the non-monotone example, a basic technique works well, but not for the other
models. We have demonstrated that, in order to obtain a consistent result for monotone
models, one should use a normalized average as opposed to a simple average and that, in the
case of the Choquet integral, one should only take into account the extreme points. The
axiomatic technique and the closed form of the restriction operator are used to derive these
key principles. [12]
Imputation is helpful up to 50% of missingness, according to both the literature [44,
45] and the simulated results of our work. BART.m can be used in missing proportions up to
90% even though it does not use imputation. This model is more adaptable because of the
remarkable property of BART.m, which justifies its usage in any missing proportion. The
choice of the optimum imputation approach for each dataset is difficult since there is no one
imputation technique that works in all circumstances [27, 29]. Additionally, a reliable
imputation technique like miss-forest typically requires a lot of time, especially for datasets
with a larger sample size and more variables [49]. As a result, the BART.m model with no
imputation requirement becomes a more effective model. [13]
There isn't one optimum way to input category columns. However, the discriminate
DL technique gets at least rank three or better in 75% of the cases in various contexts. The
random forest imputation method yields results that are very comparable but slightly less
42
good. The k-NN imputation strategy works well for MCAR with 50% missing values and
MAR with 10% to 50% missingness, and in 75% of the cases, it achieves at least rank three
or better. In 50% of the cases, VAE earns a rank between two and four. GAIN consistently
performs poorly in most circumstances; in 75% of cases, it ranks fourth or lower. For the
more complicated situations with MNAR missingness pattern, interestingly, mean/mode
imputation performs better. [14]
According to the experimental findings, the SVM regression method had the highest
level of precision when applied to the SARS data set. The experiment, however, made no
mention of the use of any missing value patterns, ratios, or procedures. Additionally, utilizing
exponential families in feature space, Smola et al. [81] developed an SVM and Gaussian
processes for addressing missing data. Finding effective optimization techniques and
computing marginal distribution become challenges in this study's estimation with missing
variables. In a different strategy [82], the authors utilized an SVM regression to handle the
data and substituted the missing values with those obtained from using the SVM classifier
over the training set.
This section shows the accuracy and percentage of missing values as a consequence
of simulations performed on the 10 datasets. The accuracy of each method was compared
using the three parameters described in the preceding section, as shown in Table II below. By
looking at the lowest values, these three parameters, MAE, MSE, and RMSE, were estimated.
These three metrics are all negatively orientated, therefore the better the results, the lower the
[Link] most helpful criteria for assessing the effectiveness of forecasting techniques and
determining forecast accuracy are MAE, MSE, and RMSE. In general, the error difference
between the imputed values and actual values is used to measure all of these metrics. [17]
In addition to the widely used complete case analysis, we tested the following six
imputation techniques: multiple imputation using bootstrapping with expectation
maximization, multiple imputation using multiple correspondence analysis, multiple
imputation using latent class analysis, multiple hot deck imputation, and multivariate
imputation by chained equations with two different model specifications: logistic regression
and random forests. The techniques are evaluated using actual data from a survey-based study
conducted as part of the Norwegian opioid maintenance treatment programme.
There are three categories into which missing mechanisms are typically subdivided:
missing completely at random (MCAR), missing at random (MAR), and missing not at
random (MNAR).1,21 Briefly stated, MCAR suggests that the missing data mechanism is
unrelated to any variable's values, whether missing or observed; MAR suggests that the
43
mechanism is unrelated to the missing values but may be related to other observed values;
and MNAR suggests that the mechanism is related to the missing values.
The majority of missing data management techniques, including CCA, provide
accurate estimates when the data are MCAR.9 However, the MCAR definition is a highly
tight presumption that is rarely met in reality.3 The MAR hypothesis is more plausible. The
assumption is frequently made since many imputation algorithms handle this level of missing
structure in the data well22. However, because MNAR data cannot be verified, it is
challenging to pinpoint the missing processes and then respond to them. Bias is most likely to
occur during this time. Unfortunately, the MNAR situation happens frequently. Imagine, for
instance, that in a study on mental health, those who exhibit symptoms of depression are less
likely to reply to inquiries about their mental health; Analyzing the respondents' mental
health scores will produce results that are skewed towards a better score than the actual score
in the population being studied. [18]

This article's assessment and analysis concentrate on the problems encountered during
the MVI process, including the MVI techniques used and the evaluation schemes considered.
The analysis of the publications over the previous ten years reveals a number of issues with
the literature, including the best methods for MVI and its evaluation, the elements that can
negatively impact missingness imputation, and the influence of MVI methods on the
decision-making process. According to the analysis of the conclusions from the 191 MVI
articles that were chosen, the most often used MVI techniques over the past ten years (from
2010 to August 2021) were EMMVI, HDMVI, LLSMVI, LRMVI, MMVI, MICEMVI,
BPCAMVI, SVDMVI, ANNMVI, KMCMVI, FCMMVI, CARTMVI, KNNMVI, and
RFMVI. [33]

This study sought to systematically investigate the performance and applicability of

several machine learning algorithms commonly used in current researches under various data
missing scenarios, and whether these machine learning algorithms were more advantageous
than traditional methods, in order to provide basis and reference for the selection of suitable
missing data processing evaluation criteria for clinical decision making. [35]
This section shows the accuracy and percentage of missing values as a consequence
of simulations performed on the 10 datasets. The accuracy of each method was compared
using the three parameters described in the preceding section, as shown in Table II below. By
looking at the lowest values, these three parameters, MAE, MSE, and RMSE, were estimated.

44
These three metrics are all negatively orientated, therefore the better the results, the lower the
scores.
The most helpful criteria for assessing the effectiveness of forecasting techniques and
determining forecast accuracy are MAE, MSE, and RMSE. In general, the error difference
between the imputed values and actual values is used to measure all of these metrics. [5]

This has led to the conclusion that there is no obvious evidence that favors one
method over the other and that the precision and accuracy of machine learning imputation
methods depend greatly on the type of data being processed. The review showed that the
missing values approaches currently in use have numerous shortcomings. One of the major
flaws in the existing literature and something that should be taken into account in future
studies is the fact that RSME is mostly employed as an evaluation metric while metrics are
not primarily used jointly. Additionally, the majority of the examined works use datasets
from various domains that are smaller than those from real-world datasets, which frequently
include a very high diversity of features. Therefore, To investigate the potential of novel
approaches to handle missing data in big data in the actual world, more work is required. [6]

There were four major steps in the entire procedure: (1) Missing data were produced
via simulation under several missing situations (missing mechanisms, missing proportions,
and ratios of missing proportions of each group), based on the original complete data set. (2)
Missing data were imputed using machine learning and conventional techniques (a total of
eight methods). (3) By determining the sensitivity, AUC, and Kappa values of the prediction
models, the effectiveness of the imputation strategies was assessed and compared. (4)
Statistical tests were utilized to determine the statistical significance of the observed
performance differences. [7]
To make predictions based on our cleaned data, we employ three common
classification algorithms: Logistic Regression, Random Forest, and Linear Support Vector
Classifier. These algorithms were chosen because they are both straightforward and
"standard" Machine Learning techniques, requiring minimum parameter adjustment in
compared to other algorithms. The reprehensibility of our findings is aided by the fact that
they also have well-established implementations in sci-kit-learn (Pedregosa et al., 2011).
These three methods of categorization also cover a wide range of classification strategies.
This makes it possible to investigate if distinct Machine Learning algorithms react to
imputation schemes differently in terms of fairness, classical performance, or both. [8]

45
 Methods to deal with missing attribute values typically fall under one of two
categories: sequential methods (also known as prepossessing methods) or parallel methods
(methods where missing attribute values are taken into account during the main learning
process). Sequential methods include those that remove cases with missing attribute values,
replace missing attribute values with the attribute's most frequent value, assign all possible
values to the attribute's missing value, replace missing attribute values with the attribute's
mean for numerical attributes, assign the attribute's missing value the value from the closest
fit case, or replace missing attribute values with new values computed from a new dataset.

A modification of the LEM2 (Learning from Examples Module, version 2) rule

induction algorithm, in which rules are induced from the original data set, with missing
attribute values considered to be "do not care" conditions or lost values, is representative of
the second group of methods to handle missing attribute values, in which missing attribute
values are taken into account during the main process of acquiring knowledge. [9]

While implicit modelling techniques, which are algorithmic in nature and rely upon
on implicit assumptions, typically do not assume the underlying reason for missing data,
explicit modelling techniques make the assumption that the data are MAR. Implicit
techniques include hot deck imputation, in which missing values are replaced by "similar"
non-missing values, and random replacement, in which one example with missing data is
randomly replaced with another complete example arbitrarily picked. By calculating the k-
NN of an example with missing data and applying the mode of the k-neighbors to the missing
data, hot deck imputation may be achieved. Using this strategy, Batista and Mona-rd (2003)
discovered that k-NN imputation can handle missing data better than summary statistic
imputation and internal methods utilized by decision trees. [10]
The three approaches take less time to complete their tasks than ILA4 does. This is so
that the inductive algorithms in the three techniques can deal with the missing values during
induction, but ILA4 receives the ready dataset with the missing values substituted by values.

The accuracy of ILA4 and ILA using the three approaches is shown in Fig. 1. The findings
make it clear that, in the majority of circumstances, ILA4 is the method with the highest
accuracy. [11]

One can determine a probability distribution over the potential suggestions when
these values are represented by probability distributions [Durban and Stewart, 2012].
Combining MCDA and Decision Under Uncertainty approaches is an additional option

46
[Dominick, 2006; Ga spars-Wieloch, 2015]. In this work, we concentrate on the second sort
of imperfection, where some qualities lack values for some alternatives. There are essentially
only two viable meanings in the MCDA context: [12]
This study demonstrated the BART.m model's excellent ability to categorize binary
incomplete datasets. It is more practical because it does not try to determine the best
imputation technique. There is no requirement for a variable selection preprocess phase
because it can automatically identify and eliminate irrelevant variables without any further
work. It can be applied even when 90% of the data is absent and delivers high classification
accuracy in a matter of seconds. With all of these characteristics, BART.m develops into a
versatile approach that the general public can utilize without the requirement for specialized
knowledge about the prepossessing stages and assumptions of incomplete classification
models. BART.m is a model for classification that has been shown to be both effective and
efficient, and it incorporates statistics methods in machine learning algorithms [13]

Few imputation approaches for classification problems demonstrate declining

performance in slightly more than 25% of the settings. Their median imputation performance,
however, is consistently favourable and typically better than for regression tasks. In all
scenarios, the approaches' potential improvements are essentially the same. Regarding
regression tasks, random forest consistently has the best performance, followed by k-NN and
discriminating DL in 50% of the instances. Sadly, this deteriorates to MNAR from MCAR.
Surprisingly, GAIN performs better than VAE in many situations this time around and has
much more room for development, especially when the missingness fraction is significant.
[14]

Few imputation approaches for classification problems demonstrate declining

performance in slightly more than 25% of the settings. Their median imputation performance,
however, is consistently favourable and typically better than for regression tasks. In all
scenarios, the approaches' potential improvements are essentially the same. Regarding
regression tasks, random forest consistently has the best performance, followed by k-NN and
discriminating DL in 50% of the instances. Sadly, this deteriorates to MNAR from MCAR.
Surprisingly, GAIN performs better than VAE in many situations this time around and has
much more room for development, especially when the missingness fraction is significant. [6]
Gel-man and Hill (2007) posit several reasons data may be missing. They group
missing data into four types: missing completely at random (MCAR), missing at random
(MAR), missing that depends on unobserved predictors, and missing that depends on the

47
missing value itself. Missing values that depend on unobserved predictors and missing values
that depend on the missing value itself can also be considered missing not at random
(MNAR). These categories are meant to identify the characteristics of the data that will be
missing, not the missing value itself. MCAR occurs when any values of a variable have the
same probability of being missing. In other words, this is the case when the data values in the
dataset will be randomly missing and there will be no reason why a specific value is missing.
An example of this would be if a respondent decides not to answer a certain question in a
survey by rolling a die and letting the decision be based on a certain number on the die.
The occurrences of non-response are quite common in sampling surveys, and in Gel-
man and Hill’s study the mechanism of non-response is assumed as MCAR, (for more
detailed explanation, see Rubin, 1987). When missing data are MCAR, no specific clue could
be derived from the other responses as to what the missing value should be. MAR, or missing
at random, can be considered to be semi-M CAR. It occurs when the probability of any
variable instance to be missing is the same for all units. However, what distinguishes MAR
from MCAR is that with MAR the variable can be predicted from other available data. When
data are MAR, omitting cases with missing data is accepted because doing so will reduce the
bias of the inferences.
The last type of missing data is missing not at random. MNAR can be sub-categorized
into: missing values which depend on unobserved predictors and missing values that depend
on the missing value itself. In these cases the likelihood of a value being missing is dependent
on some value. A good example of this comes from medical studies--when any particular
treatment causes discomfort to a patient, the likelihood of that patient walking out or
dropping out will increase (Rubin, 1987).
Another consideration that Rubin (1987) has put into his classification method is
whether the missing data are ignorable or not. By “ignorable” he means that the whole
variable can be omitted or disregarded in the model building. In cases of MAR, the ignorable
missing data mechanism occurs when variables are less important or less related to the model
than are other variables. This assumption has the same underlying philosophy as the causal
framework, in which ignoring something can be done if sufficient evidence and information
have already been gathered. So, in these cases, few correlated variables can be omitted. For
example, suppose we want to predict someone’s athletic capability or performance. The
variable of favorite color would most likely not be related to the prediction model; therefore,
excluding this variable will probably have few negative effects on the prediction model's
accuracy. The three main categories of missing data are: MCAR or missing completely at random,

48
MNAR or missing not at random and MAR or missing at random. Having examined the primary
characteristics of missing data, methods for handling missing data will be discussed next.
The simplest mechanism for handling a missing value is to discard the observation
with the missing value. However, with large dimension data sets (common in data mining) a
significant portion of the observations may have missing values. In addition, the discarding-
data approach can lead to biased estimation and can cause larger standard errors due to
reduced sample size. According to Gel-man and Hill (2007), the discarding-data approach
can be divided into three categories: complete analysis, available-case analysis and non-
response weighting. Complete analysis refers to excluding any missing values of either input
or output data. This method can cause bias to the analysis when the missing units differ
systematically from the completely observed cases. Consider the case in which study
participants are less likely to report their weight if they are obese, deleting all missing value
observations from the set would then render the study biased towards non-obese participants
as they are more likely to provide that data.
To maintain the size of the data set, and possibly remove the bias from missing
values, missing data can be modeled and the missing data values imputed. Yet, there are still
difficulties in identifying whether the missing data are really missing at random (MAR) or the
missing data depend on unobserved predictors or the missing data themselves (MNAR).
According to McKnight al. (2007), the assumption regarding whether the missing data are
MAR or MNAR is crucial to determining the best course of action. Since determining
whether the data are MNAR is a subjective task, researchers should check other studies,
conduct follow-up surveys, interview participants, or remeasure the sample units. It is
important to note that while checking other references is the approach of choice in solving
this matter, not every research can provide good references.
A simple method for supplying missing values is single imputation. There are three
types of single imputation, based on the types of values: constant, randomly selected, and
non-randomly derived values (McKnight et al., 2007). Constant substitution refers to
replacing the missing values with constant values such as mean substitution (either the
arithmetic mean or the estimated mean of the population), median substitution, or zero
imputation. Random imputation, which uses random values, consists of two major
divergences: hot-deck and cold-deck imputation. The non-random imputations are derived
values from regression, conditional imputation, or data that have been previously recorded
from a subject. Despite different types of single variable imputations, these methods all have
something in common--they assume that the standard error for the estimate is low.

49
 Constant substitution: Due to the ease and simplicity of the following single
imputation method, the most used type for supplying missing values is constant replacement
(McKnight et al., 2007). One constant replacement method is mean imputation, which
consists of predicting the missing observation by simply filling in the missing values with the
mean of the observed values. However, this method is less desirable because it tends to
under-represent extreme values, which biases the analysis by yielding a variable with greater
central tendency than should be expected. This invalidates the estimates of variance and
covariance, affecting the internal validity of the work. Another single imputation method is
ML estimated mean substitution, which is based on the maximum likelihood (ML) algorithm.
The arithmetic of this method slightly enhances the traditional mean imputation
method regarding its sensitivity to the outliers’ values. The methods draw on the assumption
of normal distribution of the data. Although the ML substitution provides an estimate mean
of the population (µ) instead of the sample mean, this method is still considered a less
desirable method because the substantial deviations from the assumed normal distribution
provide poor estimation.
The third constant replacement method, median substitution, is used when the data are
not normally distributed, in which case the curve can be skewed, flat or peaked and cannot be
represented by the mean replacement. Median imputation tends to produce larger standard
errors, which is not optimal to avoid type I error. However, compared to the two previous
constant imputation methods, it is better at reducing type II error (McKnight et al., 2007).
The last common type of constant replacement method is replacing the missing values with a
value of 0 based on logical rules. If the missing data happen to be in the outcome variable and
the probability of the predictors fully depends on recorded variables, then the missing values
can be modeled by adding another parameter having value of 0 or 1.
The added parameter will have value 1 for recorded data and 0 for missing data. For
example, in the Ohio University student retention datasets, one data element is the
accumulated GPA. If the value of the current GPA is missing, the rule allows a substitution
from the previous quarter's GPA. If there were no recorded GPA in the previous quarters, the
accumulative GPA value would be set at zero.
Random imputation: Random imputation of a single variable is needed when more
than a small fraction of data has missing values. Random imputation involves replacing the
missing values with randomly generated values. Randomly generated values can come from
the available values in the current dataset, also known as hot-deck imputation, or from similar
datasets containing matching variables, also known as cold-deck imputation. In random
50
imputation, the estimation of suitable values for replacing the missing values is generated
based on the available data.
According to McKnight et al. (2007), there are different strategies for hot-deck
imputation. The first strategy is simple random imputation, by imputing the missing value of
any missing variable with randomized values based on the available data. If the missing data
is MCAR, then there is no method for defining the missing value. Thus, if the observable
values occur in the same proportion as the sampled population, supplying missing values
from this predicted population will not introduce bias to the variable. This approach is
considered to be a good starting point for preliminary data analysis.
The strategy is hot-deck within adjustment cells—that is, blocking the relevant
covariates and imputing the missing data based on the randomly generated values of the
available data. Yet another approach uses the nearest neighbor’s value in order to replace the
data. This method imputes the missing value with the closest criteria from the available data.
For example, if the ethnicity of a participant is missing from a group with a similar ethnicity,
the missing values will be imputed with the particular ethnicity in that group Matching and
hot-deck imputation determine each missing unit (y) with a value from similar value of
predictors (x) in the observed data. Matching can become challenging when the matching
vectors need to be built with a small amount of available data.
To solve this problem, random imputation of the five closest resolved cases or other
available information can be used. One can also predict the missing values based on several
other variables that are fully observed; thus, the predicted data can be matched and imputed
to the datasets. The most common problem that arises from this method is that it
underestimates the standard errors due to the decreased variability. This is caused by the
missing data being imputed by values that already exist in the dataset.
According to Sea-storm, Kaufman, & Lee (2002), hot-deck imputation preserves the
distribution of the original data and increases the variance compared to mean imputation.
Consequently, according to Mundform & Whit-comb (1998), the estimate of the prediction
accuracy would be too dependent to the randomly selected value, due to its variation from
one selection value to another. In their research, Mundform & Whit-comb were running 1000
repetitions for hot-deck imputation and took the average value of the 1000 results of each 99
entries to obtain the value used in his [Link]-deck imputation is similar to hot-deck
imputation, but another set or sample is used to impute the data. Although the purpose of this
method is to solve the problem that occurs in hot-deck imputation, it still may increase the
probability of type I error due to the small standard error (McKnight et al., 2007).
51
Nonrandom imputation: Nonrandom imputation can be divided into single condition and
multiple conditions. Single condition methods consist of: conditional mean, last value carried
forward, and next value carried backward. Multiple conditions are used when there are more
than one single variable needed to provide more information for each missing value case.
Conditional mean imputation is based on a single condition and uses a classification
variable to estimate the mean to substitute the missing values. It emphasizes the relationship
between the classification variable and the missing data. If the relationships are weak, the
mean imputation resembles the method used in the hot- deck imputation.
Last value carried forward (LVCF) replaces the missing data value with the previous
available data, from the same subject or research participant under a certain time. This is
based on the assumption that the most recent available observation is the best guess for
subsequent missing values. To use this method, a prior observed value for the observation
must be available. For example, if the academic record of a student is missing the GPA for a
term, we would substitute the GPA of the most recent term. Next value carried backward
(NVCB) uses a similar process as the LVCF, where the imputation of the missing values on
the early observation can be filled behind with the next available data. The use of these
methods is limited to a subject’s own data that are observed continuously in a certain time
period.
Multiple condition nonrandom value imputation uses regression and error. A better
result can be produced if the value for the missing variable can be predicted with aggression
against the observed cases. Random regression imputation uses a regression model to predict
the missing values. This strategy uses uncertainty by adding the prediction error into the
regression.
Overall, although the single imputation method can be easily implemented, several of
its weaknesses could lead to distortion of the variable distribution. This distortion could then
lead to underestimation of the standard deviation, which in turn would result in
underestimation of the standard errors, and thus increased type I errors.
Multiple imputation is a method of supplying multiple values for a missing value. By
utilizing Markov Chain Monte Carlo (MCMC) simulation, multiple values can be generated
(McKnight et al., 2007). MCMC is using computer simulation of Markov chains where the
posterior distribution of the statistical inference problem is the asymptotic( (Muller, 2003).
The imputed values can be analyzed for mean and [Link] statistics can then be used
to derive expected values and associated confidence intervals.

52
 CHAPTER 3

A NOVEL DATA IMPUTATION MODEL

Overview
In the realm of data analysis, missing data is a prevalent issue that can compromise
the integrity and accuracy of statistical analyses. Traditional methods of imputing missing
values, such as mean or median replacement, may oversimplify the underlying data
distribution and fail to capture its complexities. To address this challenge, a novel imputation
model, termed Impute-new, has been developed. This chapter elaborates on the conceptual
framework, rationale, and practical implementation of Impute-new.

3.1 Introduction
Missing data is a common issue encountered across various domains, from healthcare
to finance and beyond. Traditional methods of dealing with missing data often involve simple
imputation techniques such as mean, median, or mode replacement, which may oversimplify
the complexities of the dataset and introduce biases. In response to this, a more sophisticated
approach is required – one that leverages the underlying distribution and characteristics of the
data.

3.2 Conceptual Framework of Proposed Novel Data Imputation Model (Impute-new)

The Impute-new model is formulated as follows:

Impute-new = (mod (max, median)) + R × WF ……………………………. (1)

In the Impute-new model, the modulus (mod) operation of max with median (mod
(max, median)) introduces a transformation that emphasizes the relationship between extreme
values (max) and the dataset's central value (median). This operation helps to incorporate
both the variability and central tendency of the dataset into the imputation process.

3.3 Components of Impute-new

1. max (Maximum Value): This component represents the highest observed value in
the dataset. The maximum value gives insight into the upper bound or extreme values
within the data.

53
 2. median (Median Value): The median represents the middle value of the dataset
when arranged in ascending order. Unlike the mean, the median is robust to outliers
and provides a measure of central tendency that reflects the dataset's distribution
around its middle point.
3. R (Range): The range (R) is calculated as the difference between the maximum (max)
and minimum (min) values in the dataset. It captures the spread or variability of the
data.
4. WF (Weight-factor): The Weight-factor (WF) is a critical parameter that reflects the
proportion of missing values relative to the total number of existing values in the
dataset.
It is calculated as:
WF = Existing Values / Missing Values…………. (2)
Here, "Missing Values" refers to the count of missing data points, and "Existing
Values" refers to the count of non-missing (observed) data points.
The WF reflects the proportion of missingness within the dataset. A higher WF
indicates a larger percentage of missing values relative to existing values, suggesting a
greater need for adjustment in imputed values to compensate for the missing data.

Figure 3.1: The schematic diagram of the proposed ImputeNew

54
3.4 Understanding the Imputation Formula: Step-by-Step Explanation
1. Compute Parameters:
 Identify the maximum (max) and median values from the dataset.
 Calculate the range (R) as 𝑅 = max − min.
 Determine the Weight-factor (WF) using the ratio of missing values to
existing values.
2. Imputation Process:
 For each missing value, apply the Impute-new formula:
Impute-new = (mod (max, median)) + 𝑅 × 𝑊𝐹
 The mod function here computes the remainder when max is divided
by median, introducing a nonlinear transformation that incorporates
both extreme and central tendencies of the dataset.
 The multiplication by R × WF scales the impact of missing value
 based on the data's range and the prevalence of missing values.

3.5 Advantages of Impute-new

1. Preservation of Data Characteristics:
 By utilizing max, median, range (R), and Weight-factor (WF), Impute-new
preserves important statistical characteristics of the dataset during
imputation.
2. Adaptability:
 The model adapts to diverse datasets with varying distributions and scales
due to its reliance on fundamental statistical parameters.
3. Reduced Bias:
 Impute-new minimizes bias compared to simplistic imputation methods by
incorporating multiple facets of the data distribution.
4. Interpretability:
 The transparent formula of Impute-new facilitates understanding and
reprehensibility of the imputation process.

55
3.6 Practical Application of Impute-new
1. Data Preparation:
 Identify missing values within the dataset.
 Compute max, median, min, and calculate the range (R) and Weight-factor
(WF).
2. Imputation Procedure:
 Apply the Impute-new formula to replace missing values.
 Monitor the impact of imputation on subsequent analyses and interpret the
results accordingly.

56
 CHAPTER 4

RESULTS AND DISCUSSION

The increasing prevalence of missing data in datasets poses a significant challenge to data
analysis. Traditional imputation methods like mean, median, and mode have limitations,
especially when dealing with higher percentages of missing values. This research addresses
this gap by presenting a novel imputation formula capable of accommodating both small (5-
25%) and large (30-50%) missing value scenarios.

4.1 Imputation Formula

The proposed formula is designed to handle a broad range of missing value percentages,
leveraging both fractional values (0.1 to 0.99) and large numbers (1 to 100, 1000, 10000).
The unique feature of the formula lies in its adaptability to different scales, allowing for
effective imputation regardless of the data distribution.

4.2 Methodology
4.2.1 Data Preparation
The "iris" dataset is chosen as the test bed for evaluation.
Missing values are artificially introduced, ranging from 5% to 50% in increments of 5%.

4.3 Imputation Methods

Mean, median, mode, and the proposed formula are applied to impute missing values.

4.3.1 Algorithmic Evaluation

The imputed datasets are subjected to three machine learning algorithms: IBK, J48, and
Bayes-net.
Weka is employed as the primary tool for algorithmic testing.

4.3.2 Performance Comparison

Accuracy is measured for each imputation method across different missing value percentages.
The focus is on identifying the superior imputation method under varying conditions.

57
4.3.3 Results
Our formula consistently demonstrates comparable accuracy to traditional imputation
methods (mean, median, mode) for missing value percentages ranging from 5% to 25%.
The proposed formula outperforms traditional methods significantly in the range of 30% to
50%.

Figure 4.1: Accuracy using decision tree (j48)

58
Figure 4.2: Accuracy using KNN (IBK)

Figure 4.3: Accuracy using BayesNet

59
Figure 4.4: Accuracy using Random Forest

60
 CHAPTER 5
CONCLUTION AND FUTURE WORK
5.1 Conclusion
The ImputeNew model represents an innovative approach to handling missing data by
synthesizing key statistical measures into a coherent imputation framework. By leveraging
the maximum value, median, range, and WeightFactor, ImputeNew offers a nuanced solution
that respects the underlying distribution and variability of the dataset. This model holds
promise for enhancing the robustness and reliability of data analyses across various domains,
ultimately contributing to more informed decision-making based on comprehensive and
accurate data insights.

5.2 Future work

In future, enhancing the ImputeNew data imputation model could involve several
avenues for research. Firstly, integrating transformer-based deep learning architectures could
enhance the model's ability to capture intricate data patterns, especially in high-dimensional
and sequential data contexts. Secondly, exploring semi-supervised and self-supervised
learning methods could leverage large volumes of unlabeled data to improve the model's
performance and generalization.
Additionally, assessing the model's robustness against various missing data
mechanisms beyond conventional assumptions (MCAR, MAR, MNAR) and investigating
novel approaches to handle such scenarios would expand its real-world applicability. Lastly,
conducting rigorous experiments on diverse benchmark datasets across domains would
provide valuable insights into the model's effectiveness and areas for optimization. These
steps would advance the ImputeNew data imputation model into a more robust and adaptable
solution for missing data challenges in practical settings.

61
 REFRENCES

[1] Zainal Abidin, Nadzurah; Ismail, Amelia Ritahani; Emran, Nurul.

"Performance Analysis of Machine Learning Algorithms for Missing Value
Imputation", International Journal of Advanced Computer Science and
Applications, vol. 9, No. 6, 2018.
[2] M. B. Mohammed; H. S. Zulkafli; M. B. Adam; N. Ali; I. A. Baba.
"Comparison of Five Imputation Methods in Handling Missing Data in a
Continuous Frequency Table". AIP Conference Proceedings 2355, 040006
(2021); [Link]
[3] Hand, David J.; Adèr, Herman J.; Mellenbergh, Gideon J. (2008). Advising
on Research Methods: A Consultant's Companion. Huizen, Netherlands:
[4] Polit DF Beck CT (2012). Nursing Research: Generating and Assessing
Evidence for Nursing Practice, 9th ed. Philadelphia, USA: Wolters Klower
Health, Lippincott Williams.
[5] Abidin, N. Z., Ismail, A. R., & Emran, N. A. (2018). Performance analysis
of machine learning algorithms for missing value imputation. International
Journal of Advanced Computer Science and Applications, 9(6). [30]1

[6] Emmanuel, T., Maupong, T., Mpoeleng, D., Semong, T., Mphago, B., &
Tabona, O. (2021). A survey on missing data in machine learning. Journal
of Big Data, 8(1), 1-37. [32]2

[7] Wang, H., Tang, J., Wu, M., Wang, X., & Zhang, T. (2022). Application of
machine learning missing data imputation techniques in clinical decision
making: taking the discharge assessment of patients with spontaneous
supratentorial intracerebral hemorrhage as an example. BMC Medical
Informatics and Decision Making, 22(1), 1-14. [35]3

[8] Caton, S., Malisetty, S., & Haas, C. (2022). Impact of Imputation Strategies
on Fairness in Machine Learning. Journal of Artificial Intelligence
Research, 74, 1011-1035. [37]4

[9] Grzymala-Busse, J. W., & Grzymala-Busse, W. J. (2010). Handling

missing attribute values. Data mining and knowledge discovery handbook,
33-51. [74]5

[10] Shahzad, W., Rehman, Q., & Ahmed, E. (2017). Missing data imputation
using genetic algorithm for supervised learning. International Journal of

62
 Advanced Computer Science and Applications, 8(3). [80]6

[11] Abu-Soud, S. M. (2019, April). A novel approach for dealing with missing
values in machine learning datasets with discrete values. In 2019
International Conference on Computer and Information Sciences
(ICCIS) (pp. 1-5). IEEE. [82]7

[12] Labreuche, C., & Destercke, S. (2019, August). How to handle missing
values in multi-criteria decision aiding?. In 28th International Joint
Conference on Artificial Intelligence (IJCAI 2019) (pp. 1756-1763). [84]8

[13] Mehrabani-Zeinabad, K., Doostfatemeh, M., & Ayatollahi, S. M. T. (2020).

An efficient and effective model to handle missing data in
classification. BioMed Research International, 2020. [85]9

[14] Jäger, S., Allhorn, A., & Bießmann, F. (2021). A benchmark for data
imputation methods. Frontiers in big Data, 4, 693674. [89]10

[15] Santos, M. S., Pereira, R. C., Costa, A. F., Soares, J. P., Santos, J., &
Abreu, P. H. (2019). Generating synthetic missing data: A review by
missing mechanism. IEEE Access, 7, 11651-11667.

63