Data preprocessing is a critical phase in the data science pipeline.
Since real-world data is often
"noisy," incomplete, or inconsistent, preprocessing ensures that the data is transformed into a
format that a machine learning model can effectively interpret.
1. Data Cleaning
Data cleaning involves handling missing values, smoothing noisy data, and resolving inconsistencies.
Handling Missing Values: You can either remove rows with missing data (dropping) or fill
them in (imputation) using the mean, median, or mode.
o Example: In a dataset of house prices, if the "Number of Bedrooms" is missing for a
few entries, you might impute the median value of that column.
Handling Outliers: Identifying and treating data points that deviate significantly from the
rest.
o Example: Using a Z-score or Interquartile Range (IQR) to find a salary entry of
$1,000,000 in a dataset where the average is $50,000.
Impact: Reduces bias and prevents the model from learning incorrect patterns or being
skewed by "extreme" anomalies.
2. Data Transformation
This involves changing the format or structure of the data to make it suitable for specific algorithms.
Scaling (Normalization/Standardization): Bringing all numeric features to a similar scale
(e.g., 0 to 1).
o Example: Comparing "Age" (0–100) and "Annual Income" (0–200,000). Without
scaling, the model might "think" Income is more important simply because the
numbers are larger.
Encoding Categorical Data: Converting text-based categories into numbers.
o One-Hot Encoding: Creating binary columns for each category (e.g., "Color: Red"
becomes 1 or 0).
o Label Encoding: Assigning a unique integer to each category (e.g., Red=1, Blue=2).
Impact: Essential for distance-based algorithms like K-Nearest Neighbors (KNN) or SVM.
Without scaling, the model’s convergence can be incredibly slow or inaccurate.
3. Data Reduction
When dealing with massive datasets, reduction techniques help decrease the complexity while
preserving the integrity of the original data.
Dimensionality Reduction (PCA): Reducing the number of input variables (features) by
combining them into principal components.
o Example: If you have 50 different metrics for weather, Principal Component Analysis
(PCA) can compress them into 5 core features that represent 95% of the variance.
Numerosity Reduction: Reducing the volume of data by choosing representative samples or
clusters.
Impact: Reduces "the curse of dimensionality," prevents overfitting, and significantly lowers
the computational power required for training.
4. Feature Engineering
This is the process of creating new features or modifying existing ones to improve model
performance.
Feature Selection: Using statistical tests (like Chi-square) to keep only the most relevant
variables.
Feature Creation: Combining two variables to make a more meaningful one.
o Example: In a retail dataset, combining "Total Spend" and "Number of Visits" to
create a "Customer Loyalty Score."
Impact: This is often where the most significant performance gains occur. High-quality
features allow even simple models to perform exceptionally well.
df_clean = clean_dataframe(raw_df)
df_transformed = transform_features(df_clean)
df_engineered = create_date_features(df_transformed, 'order_date')
df_reduced = reduce_dimensions(df_engineered, variance_threshold=0.90)