Unit - II Data Preprocessing CO-6K Machine Learning – 316316

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Unit - II Data Preprocessing

2.1 Data Cleaning :Introduction to Data Cleaning, Identifying and Handling Noisy
Data, Removing Duplicates and Inconsistencies, Standardizing and Normalizing Data,
Handling Outliers

Introduction to Data Cleaning.

Data Cleaning is the process of identifying, correcting, or removing incorrect,

incomplete, duplicate, or irrelevant data from a dataset to improve its quality and
accuracy before analysis or machine learning.

 Data preprocessing is the first step in any data analysis or machine learning pipeline.

 Raw data (log file, transactions, audio /video recordings, etc) is often noisy,
incomplete and inconsistent which can negatively impact the accuracy of model.

 The goal of data cleaning is to ensure that the data is accurate, consistent and free of
errors.

 Clean datasets also important in EDA (Exploratory Data Analysis)

Benefits of Data Cleaning

1. Error-Free Data - Data cleaning removes wrong, invalid, and duplicate values from the
dataset so the data becomes correct and reliable.

2. Improved Data Quality - After data cleaning, the data becomes accurate, relevant, and
meaningful, which improves the overall quality of the dataset.

3. Better Accuracy and Efficiency - Clean data helps machine learning models learn
correct patterns, which increases prediction accuracy and reduces processing time.
-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 1 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4. Complete Data- Data cleaning handles missing or incomplete values so the dataset
becomes complete and usable for analysis.

5. Maintains Data Consistency- Data cleaning ensures the same format and structure of
data throughout the dataset.

How to Perform Data Cleaning / Steps in Data Cleaning

 The process of data cleaning begins by identifying common issues such as missing
values, duplicate records, and outliers.
 Performing data cleaning involves a systematic process to identify and remove
errors from a dataset.
 The following steps are essential to perform data cleaning:

1. Remove Unwanted Observations - This step involves eliminating duplicate records,

irrelevant entries, or redundant data that add noise to the dataset

2. Fix Structural Errors- In this step, data formats and variable types are standardized to
maintain consistency across the dataset.

3. Manage Outliers - Outliers are extreme values that can affect results. These values are
detected and handled either by removal or transformation.

4. Handle Missing Data- Missing data is handled using methods such as imputation,
deletion, or advanced techniques to maintain data accuracy and integrity.

Identifying and Handling Noisy Data

The first step in data cleaning is to assess the quality of your data. This involves checking
for:
 Missing Values: Identify any blank or null values in the dataset. Missing values can be
due to various reasons such as incomplete data collection, data entry errors, or data loss
during transmission.

 Incorrect Values: Check for values that are outside the expected range or are
inconsistent with the data type. For example, a date field with an invalid date or a
numeric field with non-numeric characters.

 Inconsistencies in Data Format: Verify that the data format is consistent throughout
the dataset. For instance, ensure that dates are in the same format (e.g., YYYY-MM-
DD) and that categorical variables have consistent labels.

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 2 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
By identifying these issues early, you can determine the extent of cleaning required and
plan your approach accordingly.
For example,

The faults in the DataFrame are as follows:

1. Duplicate Rows: Rows 1 and 6 are duplicates, indicating a potential data duplication
issue.
2. Missing Values: Row 7 has a missing value in the "Name" column, which could affect
analysis and interpretation.
3. Inconsistent Date Format: The "Date" column contains dates in the format "YYYY-
MM-DD", which is consistent, but it's important to ensure consistency across all date
entries.
4. Possible Outlier: The score of 100 in row 7 could be considered as an outlier,
depending on the context of the data and the scoring system used.

Removing Duplicates and Inconsistencies

Duplicate records can skew analysis results and lead to incorrect conclusions.
Deduplication involves:
 Identifying Duplicate Entries: Use techniques such as sorting, grouping, or hashing to
identify duplicate records.
 Removing Duplicate Records: Once duplicates are identified, remove them from the
dataset to ensure that each data point is unique and accurately represented.
 Identifying Redundant Observations: Look for duplicate or identical records that do
not add any new information.
 Eliminating Irrelevant Information: Remove any variables or columns that are not
relevant to the analysis or do not provide any useful insights.
Irrelevant data can clutter your dataset and lead to inaccurate analysis. Removing data that
does not contribute meaningfully to your analysis helps streamline the dataset and improve
its overall quality. This step involves:

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 3 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Standardizing and Normalizing Data

Standardizing Data

Standardizing data means making the data consistent by fixing differences in formats,
names, and representations. This helps in accurate and reliable data analysis. Standardizing
data removes confusion and improves the quality of the dataset

This step includes:

Standardizing data formats: Ensure that dates, times, and other values follow the same
format throughout the dataset.

Correcting naming differences: Make column names, variable names, and labels
consistent.

Ensuring uniform data representation: Use the same units, scales, and categories for all
related data.

Normalizing Data

Data normalization is the process of scaling numerical data so that all values fall within a
common range. Normalization makes data consistent and easier to compare during
analysis. This helps improve the performance of machine learning algorithm

Normalization typically involves:

 Scaling Numerical Values: Convert values to a fixed range, such as 0 to 1.

 Reducing Value Differences: Bring large and small values to a similar scale.
 Improving Model Accuracy: Helps algorithms learn patterns more effectively.

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 4 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Handling Outliers
Outliers are values that are very different from the other data values. They can affect the
final result of data analysis. Managing outliers is important for obtaining accurate and
reliable insights from the data.

Outliers can be:

 Remove Outliers: If outliers occur due to wrong data entry or are not useful, they
can be removed from the dataset.

 Transform Outliers: If outliers are correct but very large or small, they can be
changed using transformation methods to reduce their effect.

2.2 Handling Missing Values -Types of Missing Data, Identifying Missing Values,
Techniques to Handle Missing Data, Removing Missing Data, Mean, Median, and
Mode Imputation, Predictive Imputation (Regression, KNN), Using Algorithms that
Support Missing Values
Handling Missing Values in Machine Learning
Missing values appear when some entries in a dataset are left blank, marked as NaN, None or
special strings like "Unknown". If not handled properly, they can reduce accuracy, create
bias and break algorithms that require complete data.
 Keeps models accurate
 Prevents bias
 Maintains sample size
 Allows correct use of ML algorithms

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 5 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Reasons Behind Missing Values in the Dataset
Data can be missing from a dataset for several reasons and understanding the cause is
important for selecting the most effective way to handle it. Common reasons for missing
data include:
 Technical issues: Failed data collection or errors during data transmission.
 Human errors: Mistakes like incorrect data entry during data processing.
 Privacy concerns: Missing sensitive or personal information due to confidentiality
policies.
 Data processing issues: Errors that occur during data preparation.
By identifying the reason behind the missing data, we can better assess its impact whether
it's causing bias or affecting the analysis and select the proper handling method such as
imputation or removal.
Types of Missing Values
 MCAR (Missing Completely at Random): Missingness occurs randomly and is not
related to any variable in the dataset.
Ex: A few student attendance values are missing due to a system error.
 MAR (Missing at Random): Missingness depends on other observed variables, not on
the missing value itself.
Ex- Internship details are missing only for first-year students (But filled by final year
student), based on year of study
 MNAR (Missing Not at Random): Missingness is directly related to the value that is
missing
Ex - Students with very low marks do not report their marks.

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 6 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Identifying Missing Values
Detecting and managing missing data is important for data analysis. Some useful functions
for detecting, removing and replacing null values in Pandas DataFrame.
Functions:
Functions Descriptions
.isnull() Detect missing values
.notnull() Detect non-missing values
.info() Summary with missing counts
.isna() Same as isnull().
dropna() Remove missing rows/columns
fillna() Fill missing values
replace() Replace specific values
drop_duplicates() Remove duplicate rows
unique() Get unique values in a Series/DataFrame

Representation of Missing Values in Datasets

Missing values can appear in different forms, so using a consistent and well-documented
format is important for clean data handling.
Common representations include:
1. Blank cells: Empty entries in spreadsheets or CSV files.
2. Special values: Placeholders like "NA", "NaN", "NULL" or numbers such as -999.
3. Codes or flags: Labels like "MISSING" or "UNKNOWN" to indicate specific types of
missing data.

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 7 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Techniques/ Strategies to Handle Missing Data
Depending on the nature of the data and the missingness, several strategies can help
maintain the integrity of our analysis.
Techniques to Handle Missing Data
│
├── Removing Missing Data
├── Imputation Methods (Mean, Median and Mode Imputation )
├── Predictive Imputation (Regression, KNN)
└── Algorithms Supporting Missing Values
Creating a Sample Dataframe

Output:

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 8 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1. Removing Missing Data
Removing Rows with Missing Values
Removing rows with missing values is the easiest way to handle missing data. It makes the
dataset clean and easy to use. However, it can reduce the amount of data and may give
wrong results if the missing values are not random.
In this example, we are removing rows with missing values from the original DataFrame
(df) using the dropna() method and then displaying the cleaned DataFrame (df_cleaned).

Output -

Pros: simple, quick, results in a fully complete dataset

Cons: loss of data, possible bias

2. Mean, Median and Mode Imputation

Imputation - Imputation means filling missing values so that the dataset size remains the
same. However, the filled values may not always be correct. Missing values are filled using
simple statistical values
Mean, Median, and Mode Imputation:
In this method, missing values are replaced with the mean, median, or mode of the data. It
is easy to use but does not consider the relationship between different variables.

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 9 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Example -

Output –

Explanation: In this example, we are explaining the imputation techniques for handling
missing values in the 'Marks' column of the DataFrame (df). It calculates and fills missing
values with the mean, median and mode of the existing values in that column and then
prints the results for observation.
-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 10 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 df['Marks'].fillna(df['Marks'].mean()): Fills missing values in the 'Marks' column
with the mean value.
 df['Marks'].fillna(df['Marks'].median()): Fills missing values in the 'Marks'
column with the median value.
 df['Marks'].fillna(df['Marks'].mode(): Fills missing values in the 'Marks' column
with the mode value.
 .iloc[0]: Accesses the first element of the Series which represents the mode.

Pros: easy, fast, suitable for numeric variables

Cons: may distort the true distribution of the data

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 11 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Predictive Imputation (Regression, KNN)
Regression Imputation –
Regression Imputation is a technique used to handle missing data by predicting the
missing values using a regression model, based on the relationship between other
variables in the dataset.
📌 The column with missing values is treated as the dependent variable, and remaining
columns are treated as independent variables.
If two variables are related, then one variable can be used to predict the other.
Example:
More study hours and better attendance usually lead to higher marks.

Need of Regression Imputation

 Mean/Median imputation ignores relationships between variables
 Regression Imputation gives more accurate results
 Useful when variables are correlated

Steps Involved in Regression Imputation

1. Identify the column with missing values
2. Treat it as target(dependent) variable (Y)
3. Select other columns as predictor(independent) variables (X)
4. Build a regression model using complete data
5. Predict missing values
6. Replace missing values with predicted values
Example Dataset
Study Hours Attendance (%) Marks
5 80 60
7 85 70
6 90 ❓

👉 Marks value is missing

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 12 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Marks depend on Study Hours and Attendance
 A regression model is trained using available data
 The model predicts the missing Marks value

Python Example – Regression Imputation

import pandas as pd
from sklearn.linear_model import LinearRegression
# Create dataset
data = {
'StudyHours': [5, 7, 6],
'Attendance': [80, 85, 90],
'Marks': [60, 70, None]
}
df = [Link](data)
# Separate rows with and without missing values
train_data = df[df['Marks'].notnull()]
test_data = df[df['Marks'].isnull()]
# Create regression model
model = LinearRegression()
# Train model
[Link](train_data[['StudyHours', 'Attendance']], train_data['Marks'])

# Predict missing value

predicted_value = [Link](test_data[['StudyHours', 'Attendance']])

# Fill missing value

[Link][df['Marks'].isnull(), 'Marks'] = predicted_value
print(df)
📌 Predicted Marks ≈ 68

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 13 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Study Hours Attendance (%) Marks
5 80 60
7 85 70
6 90 68

Advantages of Regression Imputation

✔ Uses relationship between variables
✔ More accurate than mean/median imputation
✔ Preserves data trends
✔ Suitable for predictive analysis
Disadvantages of Regression Imputation
✖ Assumes linear relationship
✖ Prediction errors may occur
✖ Not suitable for very complex datasets

When to Use Regression Imputation

 When data is Missing at Random (MAR)
 When variables are strongly related
 When dataset size is small to medium

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 14 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
KNN Imputation
KNN Imputation is a predictive method used to handle missing data by finding the K
most similar data points (neighbors) and using their values to fill the missing value.

 Data points that are close to each other have similar values
 It does not use a mathematical equation
 It works on distance and similarity

K (Number of Neighbors)

 Number of nearest data points used for prediction

 Example: K = 2, K = 3

Nearest

 Based on distance between records

 Smaller distance → more similarity

Neighbors

 Closest data records

 Their values are used to predict the missing value

Steps Involved in KNN Imputation

1. Identify missing value

2. Measure distance between records
3. Select K nearest neighbors
4. Take average (for numerical data)
5. Replace missing value

Example Dataset
Person Height Weight Age
P1 158 52 21
P2 160 55 22
P3 162 57 23
P4 165 60 25
P5 161 56 ❓

❓ Age is missing for P5

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 15 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Python Program – KNN Imputation

import pandas as pd
from [Link] import KNNImputer

# Create dataset
data = {
'Height': [158, 160, 162, 165, 161],
'Weight': [52, 55, 57, 60, 56],
'Age': [21, 22, 23, 25, None]
}

df = [Link](data)

print("Original Dataset:")
print(df)

# Create KNN Imputer with K = 2

imputer = KNNImputer(n_neighbors=2)

# Apply KNN Imputation

df_imputed = imputer.fit_transform(df)

# Convert back to DataFrame

df_imputed = [Link](df_imputed, columns=[Link])

print("\nDataset After KNN Imputation:")

print(df_imputed)

Output After Imputation

Height Weight Age

158 52 21
160 55 22
162 57 23
165 60 25
161 56 22.5 ≈ 23

Advantages of KNN Imputation

✔ Works well for non-linear data

✔ No assumption about data distribution
✔ More accurate than mean/median
-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 16 Sutar Sir - 7385284210
Unit - II Data Preprocessing CO-6K Machine Learning – 316316
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Disadvantages of KNN Imputation

✖ Slow for large datasets

✖ Choice of K affects result

When to Use KNN Imputation

 Dataset is small to medium

 Data is non-linear
 Similar patterns exist

Basis Regression Imputation KNN Imputation

1. Basic Principle Uses a mathematical regression Uses similarity between data
equation points
2. Formula Usage Uses equation (Y = a + bX) No equation is used
3. Prediction Method Predicts using a straight line Predicts using nearest
neighbors
4. Type of Data Best for linear relationships Works well for non-linear
data
5. Speed Faster computation Slower for large datasets
6. Key Parameter Regression coefficients Value of K (number of
neighbors)
7. Accuracy Depends on strength of Depends on choice of K and
Dependence relationship distance

-----------------------------------------------------------------------------------------------------------------------------------------------------------
pg. 17 Sutar Sir - 7385284210