Sri Venkateswara College of Engineering and Technology (Autonomous) Chittoor
23ACA03 Machine Learning Lab
Aim:
2. Apply the following Pre-processing techniques for a given dataset.
a. Attribute selection b. Handling Missing Values c. Discretization d. Elimination of Outliers
Algorithm: Data Pre-processing Techniques
Step 1: Start
Begin the program execution.
Step 2: Import Required Libraries
Import pandas for data handling.
Import numpy for numerical operations.
Import SimpleImputer for handling missing values.
Import KBinsDiscretizer for discretization.
Step 3: Create the Dataset
Create a dataset containing attributes such as:
o Age
o Salary
o Experience
o Department
Store the dataset in a Pandas DataFrame.
Step 4: Display Original Dataset
Print the original dataset to understand the raw data.
Step 5: Attribute Selection
Select only the required numerical attributes:
o Age
o Salary
G SUJITHA Assistant Professor CSM
Sri Venkateswara College of Engineering and Technology (Autonomous) Chittoor
o Experience
Store the selected attributes in a new DataFrame.
Step 6: Handling Missing Values
Identify missing values in the selected attributes.
Replace missing values using the mean of each attribute.
Update the dataset with the filled values.
Step 7: Discretization
Choose the continuous attribute Age.
Divide the Age values into a fixed number of bins.
Convert continuous values into discrete categories.
Step 8: Elimination of Outliers
Calculate the first quartile (Q1) and third quartile (Q3).
Compute the Interquartile Range (IQR).
Identify outliers using:
o Lower limit = Q1 − 1.5 × IQR
o Upper limit = Q3 + 1.5 × IQR
Remove records that fall outside this range.
Step 9: Display Final Dataset
Print the processed dataset after removing outliers.
Step 10: Stop
End the program execution.
Program
# Import required libraries
import pandas as pd
import numpy as np
from [Link] import SimpleImputer
from [Link] import KBinsDiscretizer
# Step 1: Create a sample dataset
G SUJITHA Assistant Professor CSM
Sri Venkateswara College of Engineering and Technology (Autonomous) Chittoor
data = {
'Age': [20, 22, 25, [Link], 30, 120],
'Salary': [30000, 35000, [Link], 40000, 50000, 1000000],
'Experience': [1, 2, 3, 4, [Link], 50],
'Department': ['HR', 'IT', 'IT', 'HR', 'Finance', 'IT']
df = [Link](data)
print("Original Dataset:\n", df)
# a. Attribute Selection
# Select only numerical attributes
selected_df = df[['Age', 'Salary', 'Experience']]
print("\nAfter Attribute Selection:\n", selected_df)
# b. Handling Missing Values
# Replace missing values with mean
imputer = SimpleImputer(strategy='mean')
selected_df[['Age', 'Salary', 'Experience']] = imputer.fit_transform(selected_df)
print("\nAfter Handling Missing Values:\n", selected_df)
# c. Discretization
# Convert Age into 3 bins (Low, Medium, High)
discretizer = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='uniform')
selected_df['Age_Binned'] = discretizer.fit_transform(selected_df[['Age']])
G SUJITHA Assistant Professor CSM
Sri Venkateswara College of Engineering and Technology (Autonomous) Chittoor
print("\nAfter Discretization:\n", selected_df)
# d. Elimination of Outliers
# Using IQR method
Q1 = selected_df.quantile(0.25)
Q3 = selected_df.quantile(0.75)
IQR = Q3 - Q1
# Remove outliers
final_df = selected_df[~((selected_df < (Q1 - 1.5 * IQR)) |
(selected_df > (Q3 + 1.5 * IQR))).any(axis=1)]
print("\nAfter Eliminating Outliers:\n", final_df)
Output
OutputOriginal Dataset:
Age Salary Experience Department
0 20.0 30000.0 1.0 HR
1 22.0 35000.0 2.0 IT
2 25.0 NaN 3.0 IT
3 NaN 40000.0 4.0 HR
4 30.0 50000.0 NaN Finance
5 120.0 1000000.0 50.0 IT
After Attribute Selection:
Age Salary Experience
0 20.0 30000.0 1.0
1 22.0 35000.0 2.0
2 25.0 NaN 3.0
3 NaN 40000.0 4.0
4 30.0 50000.0 NaN
5 120.0 1000000.0 50.0
G SUJITHA Assistant Professor CSM
Sri Venkateswara College of Engineering and Technology (Autonomous) Chittoor
c:\Users\G Dinesh Babu\Desktop\ml\[Link]: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: [Link]
docs/stable/user_guide/[Link]#returning-a-view-versus-a-copy
selected_df[['Age', 'Salary', 'Experience']] = imputer.fit_transform(selected_df)
After Handling Missing Values:
Age Salary Experience
0 20.0 30000.0 1.0
1 22.0 35000.0 2.0
2 25.0 231000.0 3.0
3 43.4 40000.0 4.0
4 30.0 50000.0 12.0
5 120.0 1000000.0 50.0
After Discretization:
Age Salary Experience Age_Binned
0 20.0 30000.0 1.0 0.0
1 22.0 35000.0 2.0 0.0
2 25.0 231000.0 3.0 0.0
3 43.4 40000.0 4.0 0.0
4 30.0 50000.0 12.0 0.0
5 120.0 1000000.0 50.0 2.0
After Eliminating Outliers:
Age Salary Experience Age_Binned
0 20.0 30000.0 1.0 0.0
1 22.0 35000.0 2.0 0.0
2 25.0 231000.0 3.0 0.0
3 43.4 40000.0 4.0 0.0
4 30.0 50000.0 12.0 0.0
[Done] exited with code=0 in 362.988 seconds
G SUJITHA Assistant Professor CSM