Zeyad Omar
DATA PREPROCESSING
ML PRESENTATION
LinkedIn Updated , Nov 2024
[Link]
DATA
PREPROCESSING
Data Preprocessing is the process of transforming raw, messy
data into a clean, structured format suitable for analysis or
machine learning.
Key Goals:
Improve data quality.
Ensure consistency and relevance.
Enable better performance of models.
WHY IS IT ESSENTIAL
Garbage In, Garbage Out: Poor-quality data leads to poor
results.
Improves Model Performance: High-quality data ensures
accurate predictions.
Handles Challenges: Missing values, outliers, duplicates,
and inconsistent formats.
Prepares for Algorithms: Many models require scaled,
normalized, or encoded data.
KEY STEPS IN DATA
PREPROCESSING
1. Understanding the Data
2. Cleaning the Data
3. Transforming the Data
4. Feature Engineering and Selection
AGENDA
1. Data Collection
2. Data Types
3. Handling Missing Data
4. Duplicates
5. Detecting and Handling Outliers
6. Normalization and Standardization
7. Feature Scaling
8. Encoding Categorical Variables
9. Feature Engineering
10. Feature Selection
11. Imbalanced Datasets
DATA COLLECTION
Data collection is the process of gathering data from
various sources to use in analysis or machine learning
models.
Types of Data Sources
Structured Data:
Databases (SQL, NoSQL).
CSV, Excel files.
Unstructured Data:
Text files, PDFs, images, videos.
Logs and social media data.
Real-Time Data:
Sensors, IoT devices, and streaming APIs.
DATA TYPES
Data types define the nature of the data in a dataset.
Understanding data types is crucial for:
1. Selecting the right preprocessing techniques.
2. Applying appropriate statistical and machine learning models.
Types of Data
Numerical Data
Categorical Data
Time-Series Data
Text Data
Image Data
MISSING DATA
Why Does Missing Data Occur?
Human Error: Mistakes in data entry or recording.
Sensor/Device Issues: Data not captured due to malfunction.
Incomplete Surveys: Participants skip questions.
Merging Datasets: Missing values arise from mismatched records.
Strategies
Removing Missing Data
Imputation (Filling Missing Values)
Special Handling for Time-Series Data (Forward or backward filling)
DUPLICATES
Duplicates are identical or near-identical records in a dataset that can
arise from:
1. Repeated data entries.
2. Errors during data merging.
3. Data collection from multiple sources.
Problem
Skewed Results
Reduced Model Performance
Increased Storage Costs
OUTLIERS
Outliers are data points that deviate significantly from the majority of
the data.
Causes of Outliers:
Data entry errors (e.g., misplaced decimal points).
Natural variations (e.g., rare events).
Sampling issues or bias.
Detect Outliers
Visual Methods
Z-Score: Points with Z > 3 are potential outliers.
IQR Method
Handle Outliers
Remove Outliers
Cap Outliers
Transform Data
NORMALIZATION AND
STANDARDIZATION
Normalization:
Rescales data to a fixed range, typically [0, 1].
Standardization:
Centers data around the mean with a standard deviation of 1.
SCALING IMPORTANT
Improved Model Performance
Faster Convergence
Fair Feature Contribution
ENCODING
Categorical variables are features that represent categories or labels
instead of numerical values.
Examples:
Nominal Data: Categories with no inherent order (e.g., gender,
colors).
Ordinal Data: Categories with a meaningful order (e.g., education
level, rankings).
Why:
Machine Learning Models
Preserve Meaning
Avoid Bias
TYPES OF ENCODING
Label Encoding:
Assigns a unique integer to each category.
One-Hot Encoding:
Creates binary columns for each category
Ordinal Encoding:
Assigns numbers based on order for ordinal data.
Choosing the Right Encoding Technique
Label Encoding: Use for ordinal data.
One-Hot Encoding: Preferred for nominal data
Ordinal Encoding: Best for ordered categories where relationships
matter
FEATURE SCALING
CHOOSING THE RIGHT
TECHNIQUE
FEATURE ENGINEERING
TYPES OF FEATURE
ENGINEERING
TECHNIQUES FOR
FEATURE ENGINEERING
FEATURE SELECTION
TECHNIQUES
HANDLE IMBALANCE
IMBALANCED DATASETS
IMBALANCED DATA A
PROBLEM
HANDLE IMBALANCE
zeyad Omar
THANK YOU
FOR YOUR NICE ATTENTION