Data Preprocessing
1. Importance of Data in Machine Learning
Explanation:
Machine learning models rely heavily on data quality.
Garbage In, Garbage Out (GIGO): Poor data quality leads to poor model performance.
Preprocessing ensures the data is clean, consistent, and ready for modeling.
Key Points :
Data should represent the problem accurately.
Preprocessing reduces noise and inconsistencies.
Example:
Imagine training a model to predict house prices with incomplete or inconsistent data (e.g., missing
square footage or prices in different units).
2. Handling Missing Values
Explanation:
Missing data can lead to inaccurate models.
Common strategies to handle missing data:
Remove rows/columns with missing values (if minimal).
Impute missing values using statistical techniques.
Techniques:
Remove Missing Data :
import pandas as pd
df = [Link]({'A': [1, 2, None], 'B': [4, None, 6]})
[Link]() # Remove rows with missing data
Imputation:
from [Link] import SimpleImputer
imputer = SimpleImputer(strategy='mean')
df['A'] = imputer.fit_transform(df[['A']])
In [1]:
import pandas as pd
df = [Link]({'A': [1, 2, None], 'B': [4, None, 6]})
In [2]:
df
Out[2]:
A B
0 1.0 4.0
1 2.0 NaN
2 NaN 6.0
In [3]:
from [Link] import SimpleImputer
imputer = SimpleImputer(strategy='mean')
df['A'] = imputer.fit_transform(df[['A']])
In [4]:
df
Out[4]:
A B
0 1.0 4.0
1 2.0 NaN
2 1.5 6.0
3. Data Scaling and Normalization
Explanation:
Different features may have values in varying ranges (e.g., age in years vs. income in thousands).
Scaling ensures all features contribute equally to the model.
Scaling : Adjust values to a specific range (e.g., 0-1).
Normalization: Rescales the data to have mean = 0 and standard deviation = 1.
Techniques:
Min-Max Scaling :
from [Link] import MinMaxScaler
scaler = MinMaxScaler()
scaled_data = scaler.fit_transform(df)
Standardization :
from [Link] import StandardScaler
scaler = StandardScaler()
standardized_data = scaler.fit_transform(df)
Example:
House price dataset with columns: Area (sq ft), Price ($). Normalize these columns to ensure a fair
contribution to the prediction model.
Why Do We Need Scaling and Normalization?
Machine learning models often use mathematical computations like distances (e.g., Euclidean distance) or
optimizations (e.g., gradient descent). If your dataset contains features with vastly different ranges or units, it
can negatively impact the model’s performance.
For example:
Imagine a dataset for predicting house prices :
Feature 1: Number of rooms (ranges between 1 to 10)
Feature 2: House price (ranges from 10,000 to 1,000,000)
In this scenario:
The model gives more importance to the feature with the larger range (house price), ignoring the
smaller-scale features (number of rooms).
Key Concepts
1. Data Scaling :
Scaling adjusts the range of your data to make sure all features contribute equally to the model.
Example: Bring all features to a range of [0, 1] or [-1, 1].
2. Normalization:
Normalization adjusts the distribution of data to have:
Mean = 0
Standard Deviation = 1
Example: Values become centered around zero, making them easier to work with for algorithms like
gradient descent.
When to Use Scaling vs. Normalization
Technique When to Use
Scaling When the range of values for features differs greatly (e.g., 1-10 vs. 1,000-10,000).
Normalization When the algorithm assumes data is normally distributed (e.g., neural networks, PCA).
Example to Understand Scaling and Normalization
Raw Dataset
Feature: Age (years) Feature: Salary ($)
25 50,000
35 80,000
45 120,000
55 200,000
Problem: "Salary" has a much larger range than "Age." Machine learning models may prioritize Salary over
Age, leading to biased results.
Techniques
1. Min-Max Scaling
Adjust values to a specific range, typically [0, 1].
Formula: Xscaled
X−Xmin
= Xmax − Xmin
Example:
For "Age," with values [25, 35, 45, 55]:
(Xmin , Xmax
= 25 = 55)
25−25
Scaled Age for 25: 55−25 =0
55−25
Scaled Age for 55: 55−25 =1
Resulting scaled data: | Age (scaled) | Salary (scaled) | |--------------|-----------------| | 0.0 | 0.0 | | 0.33 | 0.18 |
| 0.66 | 0.45 | | 1.0 | 1.0 |
Code Example:
from [Link] import MinMaxScaler
import pandas as pd
# Example DataFrame
data = {'Age': [25, 35, 45, 55], 'Salary': [50000, 80000, 120000, 200000]}
df = [Link](data)
# Min-Max Scaling
scaler = MinMaxScaler()
scaled_data = scaler.fit_transform(df)
print([Link](scaled_data, columns=[Link]))
print([Link](scaled_data, columns=[Link]))
2. Standardization (Normalization)
Adjusts data to have:
Mean = 0
Standard Deviation = 1
X−μ
Formula: Xnormalized = σ Where:
μ: Mean of the feature
σ: Standard deviation of the feature
Example:
For "Age," with values [25, 35, 45, 55]:
μ = 40, σ = 10
25−40
Normalized Age for 25: 10
=
−1.5
55−40
Normalized Age for 55: 10
= 1.5
Resulting normalized data: | Age (normalized) | Salary (normalized) | |------------------|---------------------| | -
1.5 | -1.3 | | -0.5 | -0.6 | | 0.5 | 0.0 | | 1.5 | 1.9 |
Code Example:
from [Link] import StandardScaler
# Example DataFrame
data = {'Age': [25, 35, 45, 55], 'Salary': [50000, 80000, 120000, 200000]}
df = [Link](data)
# Standardization
scaler = StandardScaler()
normalized_data = scaler.fit_transform(df)
print([Link](normalized_data, columns=[Link]))
Key Takeaways
1. Why scale and normalize?
To ensure all features contribute equally to the model.
Prevent dominance of features with larger ranges.
2. Which one to use?
Scaling : Preferred for algorithms based on distance (e.g., k-NN, SVM).
Normalization: Preferred for algorithms assuming normally distributed data (e.g., neural networks, PCA).
3. Practical Example:
Predicting house prices with features like "number of rooms" (1-10) and "price in dollars" (10,000-
1,000,000). Without scaling, the model might focus too much on price and ignore rooms.
In [5]:
from [Link] import StandardScaler
In [6]:
data = {'Age': [25, 35, 45, 55], 'Salary': [50000, 80000, 120000, 200000]}
In [7]:
df = [Link](data)
In [8]:
In [8]:
df
Out[8]:
Age Salary
0 25 50000
1 35 80000
2 45 120000
3 55 200000
In [9]:
# Standardization
scaler = StandardScaler()
normalized_data = scaler.fit_transform(df)
In [11]:
print([Link](normalized_data, columns=[Link]))
Age Salary
0 -1.341641 -1.110289
1 -0.447214 -0.577350
2 0.447214 0.133235
3 1.341641 1.554405
In [ ]:
4. Encoding Categorical Variables
Why Encode Categorical Variables?
Machine learning models work with numerical data, but datasets often contain categorical data (e.g.,
Gender: Male/Female, Country: India/USA).
Encoding converts these categories into numerical formats for processing by the model.
Types of Encoding
1. Label Encoding:
Assigns a unique integer to each category.
Simple and efficient for ordinal data (e.g., low, medium, high).
Example:
Column: ["Male", "Female", "Female", "Male"]
Encoded: [0, 1, 1, 0]
Code:
from [Link] import LabelEncoder
le = LabelEncoder()
data = ["Male", "Female", "Female", "Male"]
encoded = le.fit_transform(data)
print(encoded) # Output: [1, 0, 0, 1]
2. One-Hot Encoding:
Creates binary columns for each category.
Useful for nominal data (categories without order).
Example:
Column: ["Red", "Green", "Blue"]
Encoded:
Encoded:
Red Green Blue
1 0 0
0 1 0
0 0 1
Code:
import pandas as pd
data = ["Red", "Green", "Blue"]
df = [Link](data, columns=["Color"])
encoded = pd.get_dummies(df, columns=["Color"])
print(encoded)
Label Encoding vs. One-Hot Encoding
Aspect Label Encoding One-Hot Encoding
Data Representation Converts categories to integers (e.g., 0, 1, 2). Creates binary columns for each category.
When to Use Ordinal data (e.g., "Low", "Medium", "High"). Nominal data (e.g., "Red", "Green", "Blue").
Order Sensitivity May create false order relationships (e.g., "Red" < "Green"). Does not imply any order.
Impact on
Low impact (single column). Increases dimensionality (adds columns).
Dimensionality
Example Gender: Male=0 , Female=1 . Color: Red=1,0,0 , Green=0,1,0 .
Key Takeaways
Use Label Encoding for ordered data (e.g., education levels).
Use One-Hot Encoding for unordered data (e.g., color, gender, countries).
Be cautious with algorithms sensitive to magnitude (e.g., linear regression), as Label Encoding might
introduce bias.
Example:
A dataset with a column "Gender" (Male/Female) is converted to numbers: Male = 1, Female = 0.
In [12]:
data = ["Male", "Female", "Female", "Male"]
In [15]:
from [Link] import LabelEncoder
import pandas as pd
In [17]:
le = LabelEncoder()
encoded = le.fit_transform(data)
In [18]:
print(encoded)
[1 0 0 1]
In [19]:
colures = ["Red", "Green", "Blue"]
In [23]:
In [23]:
df = [Link](colures, columns=["Color"])
encoded = pd.get_dummies(df, columns=["Color"])
print(encoded)
Color_Blue Color_Green Color_Red
0 False False True
1 False True False
2 True False False
In [24]:
import pandas as pd
data = ["Red", "Green", "Blue"]
df = [Link](data, columns=["Color"])
encoded = pd.get_dummies(df, columns=["Color"])
print(encoded)
Color_Blue Color_Green Color_Red
0 False False True
1 False True False
2 True False False
5. Splitting the Dataset into Train and Test Sets
Why Do We Split the Dataset into Train and Test Sets?
1. Model Evaluation
The main goal of machine learning is to create models that can generalize well to unseen data.
Splitting the dataset helps evaluate how well the model performs on new, unseen data (test set), rather than
just memorizing the training data.
2. Avoid Overfitting
Overfitting occurs when a model learns the training data too well, including noise and details that don't
generalize to new data.
By testing on a separate dataset (test set), we can detect if the model is overfitting.
How It Works
1. Training Set:
Used to train the model.
The model learns patterns and relationships from this data.
2. Test Set:
Set aside for final evaluation.
Measures how well the model performs on unseen data.
Typical Split Ratios
70-80% for Training
20-30% for Testing
Example
Dataset
A dataset predicting house prices has:
A dataset predicting house prices has:
Features: Area (sq ft), Number of rooms, Age of house
Target: Price ($)
Splitting
Use train_test_split from scikit-learn:
from sklearn.model_selection import train_test_split
# Example data
X = [[1200, 3, 20], [1500, 4, 15], [1000, 2, 30], [1800, 4, 10]]
y = [250000, 300000, 200000, 400000] # Prices
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, rando
m_state=42)
print("Training Features:", X_train)
print("Test Features:", X_test)
print("Training Targets:", y_train)
print("Test Targets:", y_test)
Output :
Training Set: 75% of the data (used to train the model).
Test Set: 25% of the data (used for evaluation).
Key Takeaways
1. Splitting ensures the model is evaluated on unseen data.
2. Helps identify overfitting and improve generalization.
3. A good practice for building reliable and robust machine learning models.
Class Exercise
Provide a sample dataset (e.g., Titanic dataset or any other dataset with missing values, categorical
variables, and numerical features).
Ask students to:
1. Handle missing values.
2. Scale numerical features.
3. Encode categorical variables.
4. Split the dataset into train and test sets.
Pranjal Gajbhiye(AIE)
Happy Learning...
In [ ]: