Data Preprocessing
Dr. Saima Khosa
Dr. Saima Khosa
Introduction to Data Preprocessing
• Emphasize its role in improving data quality and enhancing the performance of
machine learning models.
Dr. Saima Khosa
Forms of Data Preprocessing
Dr. Saima Khosa
Data Cleaning
• Data in the Real World Is Dirty: Lots of potentially incorrect data, e.g. instrument faulty, human or
computer error, transmission error
• incomplete: lacking attribute values, lacking certain attributes of interest
e.g., Occupation=“ ” (missing data)
• noisy: containing noise, errors, or outliers
• e.g., Salary=“−10” (an error)
• inconsistent: containing discrepancies in codes or names, e.g.,
• Age=“42”, Birthday=“03/07/2010”
• Was rating “1, 2, 3”, now rating “A, B, C”
• Intentional : misrepresented
• Jan. 1 as everyone’s birthday?
Dr. Saima Khosa
4
Incomplete (Missing) Data
• Data is not always available
• E.g. many tuples have no recorded value for several attributes, such as
customer income in sales data
• Missing data may be due to
• equipment malfunction
• inconsistent with other recorded data and thus deleted
• data not entered due to misunderstanding
• certain data may not be considered important at the time of entry
• not register history or changes of the data
Dr. Saima Khosa
5
How to Handle Missing Data?
• Ignore the tuple: usually done when class label is missing (when doing
classification)—not effective when the % of missing values per attribute
varies considerably
• Fill in the missing value manually: tedious + infeasible?
• Fill in it automatically with
• a global constant : e.g., “unknown”, a new class?
• the attribute mean for all samples belonging to the same class
Dr. Saima Khosa
6
[Link] Cleaning
1. Identifying and handling missing values:
Missing values are common in datasets and can adversely affect analysis and modeling.
Identifying and handling them effectively is a crucial step in data preprocessing.
1. Techniques:
1. Imputation: Fill missing values with a calculated value, such as the mean, median, or
mode of the feature.
2. Deletion: Remove rows or columns with missing values if they are few in number and
don't significantly impact the dataset's integrity.
3. Advanced methods: Utilize predictive modeling techniques to estimate missing
values based on other features in the dataset.
Dr. Saima Khosa
[Link] duplicates:
Duplicates occur when the same observations appear more than once in
the dataset. They can lead to biased analysis and skewed results.
1. Techniques:
[Link]: Identify duplicate records by comparing all or selected
features within the dataset.
[Link]: Delete duplicate records while ensuring that relevant
information is retained, typically based on a subset of features.
Dr. Saima Khosa
3. Dealing with outliers:
Outliers are data points that significantly deviate from the rest of the dataset. They
can distort statistical analysis and modeling results.
1. Techniques:
[Link]: Remove extreme values beyond a certain threshold to mitigate
their impact on analysis.
[Link]: Replace extreme values with less extreme values, such as the
nearest data point within a specified percentile range.
[Link] statistical measures: Use statistical measures like median and
interquartile range that are less influenced by outliers compared to mean and
standard deviation.
Dr. Saima Khosa
Data Transformation
Dr. Saima Khosa
[Link] categorical variables:
Categorical variables represent qualitative data with discrete categories.
Machine learning algorithms typically require numerical inputs, so
categorical variables need to be transformed into numerical form.
Techniques:
[Link]-hot encoding: Convert categorical variables into binary vectors,
where each category is represented by a binary feature.
[Link] encoding: Assign a unique numerical label to each category.
[Link] encoding: Encode categorical variables based on the target
variable's mean or other statistical measures.
Dr. Saima Khosa
Dr. Saima Khosa
Dr. Saima Khosa
Dr. Saima Khosa
[Link] numerical features:
Numerical features often have different scales,
which can lead to biased models. Scaling
transforms numerical features to a similar
scale, making comparisons more meaningful.
1. Techniques:
1. Standardization (Z-score
normalization): Transform features to
have a mean of 0 and a standard
deviation of 1.
2. Min-Max scaling: Scale features to a
fixed range, typically between 0 and 1.
3. Robust scaling: Scale features using
robust statistical measures that are
less affected by outliers.
Dr. Saima Khosa
[Link] datetime variables:
Datetime variables represent dates and times, which
may contain valuable information for analysis.
However, they need to be properly processed to
extract meaningful features.
1. Techniques:
1. Extracting date components: Split datetime
variables into separate components such as
year, month, day, hour, minute, and second.
2. Calculating time differences: Compute time
intervals between datetime variables, which
can be useful for time-series analysis or
feature engineering.
3. Encoding cyclic features: Handle cyclical
datetime features such as day of the week
or month using techniques like sin-cos
encoding to preserve their cyclical nature.
Dr. Saima Khosa
By applying appropriate
transformations to categorical These transformations
variables, numerical features, facilitate better model
and datetime variables, data performance and more
preprocessing ensures that the accurate predictions in data
dataset is suitable for machine science projects.
learning algorithms.
Dr. Saima Khosa
Data Normalization
Dr. Saima Khosa
Standardization: Explain how standardization transforms data to have a
mean of 0 and a standard deviation of 1.
Min-Max scaling: Discuss how min-max scaling scales data to a fixed
range, typically between 0 and 1.
Robust scaling: Explain robust scaling techniques like RobustScaler,
which are less affected by outliers.
Dr. Saima Khosa
Standardization:
Standardization transforms numerical
features to have a mean of 0 and a
standard deviation of 1. It ensures that
the features have a comparable scale,
making them suitable for algorithms
that are sensitive to feature scales.
1. Technique:
1. Calculate the mean (μ) and
standard deviation (σ) of each
feature.
2. Subtract the mean from each
value and divide by the standard
deviation (x - μ) / σ.
Dr. Saima Khosa
Min-Max scaling:
Min-Max scaling rescales numerical features to a fixed range, typically between 0
and 1. It preserves the relative relationships between data points and is suitable
for algorithms that require input features to be within a specific range.
Technique:
[Link] the minimum (min) and maximum (max) values of each feature.
[Link] each value to the range [0, 1] using the formula: (x - min) / (max -
min).
Dr. Saima Khosa
• Robust scaling scales numerical features
using robust statistical measures that are
less influenced by outliers. It is particularly
Robust useful when the dataset contains outliers
that can distort the scaling process.
scaling: • Technique:
• Calculate the median and interquartile
range (IQR) of each feature.
• Scale each value using the formula: (x -
median) / IQ
Dr. Saima Khosa
• Data normalization ensures that numerical features have consistent scales,
which helps algorithms converge faster and produce more reliable results. By
choosing the appropriate normalization technique based on the characteristics
of the dataset, data scientists can improve the performance and stability of
their machine learning models
Dr. Saima Khosa
Feature Engineering
Dr. Saima Khosa
Creating new features:
• Feature engineering involves creating new variables from existing data to provide additional insight
and improve the model's predictive power.
• Techniques:
• Polynomial features: Generate new features by considering polynomial combinations (e.g.,
square, cube) of existing features.
• Interaction terms: Create features that capture the interaction between two or more
variables, potentially uncovering effects that these variables may have on the response when
combined.
• Domain-specific features: Develop features based on industry knowledge, such as calculating
the body mass index (BMI) in healthcare data or customer lifetime value in retail.
Dr. Saima Khosa
Feature selection techniques:
• Filter methods: Use statistical techniques to select features based on their
relationship with the target variable, such as correlation or chi-squared test.
• Wrapper methods: Evaluate multiple models using subsets of features,
selecting the combination that results in the best model performance, like
forward selection or backward elimination.
• Embedded methods: Use algorithms that have built-in feature selection
methods during the training process.
Dr. Saima Khosa
Handling Imbalanced
Data
Dr. Saima Khosa
Introduction to Imbalanced Data:
• Data imbalance occurs when the classes in the dataset are not equally
represented, which is common in datasets where one class significantly
outnumbers the other(s).
• Example: In a disease diagnosis dataset, the number of people without the
disease may be much higher than the number of people with the disease.
Dr. Saima Khosa
Impact of Imbalanced Data:
• Problems Caused: Skewed data can lead to models that are biased towards
the majority class, resulting in poor classification performance on the minority
class.
• Significance: Ensuring good predictive performance for all classes, especially in
critical applications like medical diagnosis or fraud detection, is essential.
Dr. Saima Khosa
Techniques for Handling Imbalanced Data:
• Resampling Techniques:
• Oversampling: Increasing the number of instances in the minority class by
duplicating them.
• Undersampling: Reducing the number of instances in the majority class to
balance the dataset.
Dr. Saima Khosa
Synthetic Sample Generation (SMOTE):
• Explanation: SMOTE (Synthetic Minority Over-sampling Technique)
involves creating synthetic examples rather than duplicating existing ones.
• How It Works: By taking samples from the feature space for the minority
class and creating new synthetic examples along the lines joining any/all of
the k minority class nearest neighbors.
• Using Class Weights:
• Explanation: Adjusting the weights of different classes in the loss function
used by the classifier. Higher weights are assigned to the minority class,
and lower weights to the majority class, compensating for the imbalance.
Dr. Saima Khosa
Choosing the Right Technique:
1. Factors to Consider: The choice of method depends on the size of the
dataset, the degree of imbalance, the importance of each class, and the
specific problem at hand.
2. Practical Tips: Test multiple techniques to see which provides the best
performance improvement on your validation set. Always validate with a
balanced metric, such as F1-score, that considers both precision and
recall.
Dr. Saima Khosa
Conclusion:
1. Recap: Summarize the importance of addressing data imbalance and the
potential techniques.
2. Call to Action: Encourage the audience to consider these methods in their
projects to improve model fairness and accuracy.
Dr. Saima Khosa
Data Preprocessing
Pipeline
Dr. Saima Khosa
Introduction to Data Preprocessing Pipeline:
1. Explanation: A data preprocessing pipeline lays out a systematic series
of steps for transforming raw data into a format that's ready for analysis
and modeling. It automates repetitive tasks and ensures consistency and
efficiency in data preparation.
2. Importance: Helps avoid errors and variability in data transformation,
which can significantly impact the outcomes of predictive models.
Dr. Saima Khosa
Components of a Data Preprocessing
Pipeline:
• Data Cleaning: Includes removing duplicates, filling missing values, correcting errors, and handling outliers.
• Data Transformation: Involves normalizing data, encoding categorical variables, and transforming datetime
variables.
• Feature Engineering: Creating new features that increase the predictive power of the machine learning
models.
• Feature Selection: Selecting the most relevant features to improve model performance and reduce
computational cost.
• Data Splitting: Dividing data into training, validation, and test sets to ensure the model is trained and tested
on different data samples.
Dr. Saima Khosa
Building the Pipeline:
• Tools and Libraries: Mention the use of specific libraries such as scikit-learn
in Python, which offers utilities for building pipelines with transformers and
estimators.
• Steps to Build:
• Define each step of the transformation and feature engineering process.
• Chain these steps sequentially using pipeline tools.
• Ensure each step is parameterized to adjust transformations as need
Dr. Saima Khosa
Benefits of Using a Pipeline:
• Efficiency: Automates and simplifies the preprocessing steps, saving time
and reducing manual errors.
• Reproducibility: Ensures that the exact same steps are applied in the same
order, facilitating consistency across different runs and among different
users.
• Ease of Experimentation: Makes it easier to modify and validate different
preprocessing strategies to find the most effective approach.
Dr. Saima Khosa
Example of a Simple Pipeline:
• Illustration: Show a simple pipeline example that might include filling
missing values, encoding categorical data, and scaling features before
training a model.
• Code Example: Provide a snippet of Python code using scikit-learn to build a
basic pipeline.
Dr. Saima Khosa
Testing and Validation:
1. Explanation: Discuss how to test the pipeline to ensure it performs as
expected on new data.
2. Validation Techniques: Highlight the importance of using the validation
set to evaluate how the transformations affect model performance.
Dr. Saima Khosa