UNIT II Data Preprocessing for Big Data
Data cleaning and preprocessing techniques,
Handling missing data,
Data transformation and normalization,
Dimensionality reduction methods.
Introduction to Data Preprocessing
Data preprocessing is a crucial step in the data analytics and machine learning pipeline.
In real-world applications,
raw data is rarely clean;
it is often noisy, incomplete, inconsistent, and unstructured,
which can negatively affect analysis results.
Need for Data Preprocessing
Poor-quality data can lead to incorrect analysis,misleading insights, and unreliable decisions.
Data preprocessing improves the accuracy, consistency, and usability of data, making it
suitable for effective analytics and model building.
Importance of Data Preprocessing
● Improves data quality
● Increases accuracy of analytical results
● Reduces errors and noise
● Ensures reliable decision-making
● Enhances performance of machine learning models
Major Steps in Data Preprocessing
1. Data Cleaning
Involves removing noise, correcting errors, and eliminating duplicate or irrelevant data to
ensure consistency and correctness.
2. Handling Missing Data
Missing values are managed using techniques such as:
● Deletion of records
● Mean/median/mode imputation
● Predictive methods
This ensures the dataset remains complete and usable.
3. Data Transformation
Data is converted into a suitable format for analysis. This includes:
● Encoding categorical variables
● Aggregation
● Scaling
● Feature construction
4. Data Normalization
Normalization scales numerical data to a common range, improving algorithm performance
and preventing features with large values from dominating results.
5. Dimensionality Reduction
Reduces the number of variables while retaining important information. Techniques like PCA
(Principal Component Analysis) help reduce complexity and improve efficiency.
Data Cleaning
Data cleaning is a fundamental step in data preprocessing that focuses on improving the
accuracy, consistency, and reliability of data.
Real-world datasets often contain errors, duplicates, noise, and inconsistencies, which must be
corrected before analysis.
1. Removing Duplicates
Duplicate records occur due to repeated data entry, data integration from multiple sources, or
system errors. These duplicates can bias analysis results.
● Identifying and removing duplicate rows ensures data uniqueness and prevents
over-representation of certain records.
2. Handling Noisy Data
Noisy data refers to random errors or meaningless values that distort true patterns.
Common techniques include:
● Binning: Smooths data by grouping values into intervals.
● Clustering: Groups similar data points and removes those that do not fit well into any
cluster.
● Smoothing: Uses statistical methods (e.g., moving averages) to reduce fluctuations.
3. Detecting Outliers
Outliers are extreme values that differ significantly from other observations and may affect
analysis.
Common methods include:
● Z-Score: Identifies values far from the mean.
● Interquartile Range (IQR): Detects values outside the normal range.
● Machine Learning Models: Use algorithms such as isolation forests or clustering to
detect anomalies.
4. Ensuring Data Consistency
Consistency ensures uniform representation of data across the dataset.
This includes:
● Correcting formatting issues (date formats, units)
● Fixing spelling and naming errors
● Standardizing categorical values (e.g., “M” vs “Male”)
Handling Missing Data
Missing data is a common issue in real-world datasets and must be addressed carefully during
data preprocessing to ensure accurate analysis and reliable results.
Causes of Missing Data
Missing values may occur due to:
● Sensor failures during data collection
● Human errors in data entry
● Data corruption or loss during transmission or storage
Methods for Handling Missing Data
1. Ignoring Missing Values
If only a small percentage of data is missing, the affected records or attributes can be
removed with minimal impact on analysis.
2. Imputation Techniques
Missing values can be replaced using statistical measures:
○ Mean for numerical data
○ Median when data contains outliers
○ Mode for categorical data
3. Prediction-Based Methods
Machine learning techniques such as regression and k-Nearest Neighbors (kNN) can
predict missing values based on patterns in existing data.
4. Interpolation for Time-Series Data
For sequential or time-based data, interpolation methods estimate missing values
using neighboring observations.
5. Using an ‘Unknown’ Category
For categorical attributes, missing values can be replaced with a separate category
like “Unknown” to preserve data records.
Conclusion
Handling missing data improves data quality and ensures more accurate analytics. The choice
of method depends on the nature of the data, the extent of missing values, and the analysis
objectives.
1. Deletion Techniques
a) Listwise Deletion
In this method, entire records (rows) containing missing values are removed from the dataset.
● Simple to apply
● Effective only when missing values are very few
● Can lead to loss of important data if many records are removed
b) Pairwise Deletion
Instead of removing complete records, this method uses all available data pairs during
analysis.
● Retains more data than listwise deletion
● Results may vary across different analyses
2. Simple Imputation Techniques
a) Mean Imputation
Missing numerical values are replaced with the mean of the available values in that attribute.
● Easy to implement
● Reduces data variability
● Sensitive to outliers
b) Median Imputation
Missing values are replaced with the median.
● Suitable when data contains outliers
● More robust than mean imputation
c) Mode Imputation
Missing values are replaced with the most frequent value.
● Commonly used for categorical data
● May introduce bias if one category dominates
3. Advanced Statistical Imputation Techniques
a) Regression Imputation
A regression model is built using complete data, and missing values are predicted based on
other variables.
● Preserves relationships among attributes
● More accurate than simple imputation
● Computationally expensive
b) Multiple Imputation
Multiple datasets are created by filling missing values with different estimated values. The
final result is obtained by averaging the outcomes.
● Handles uncertainty effectively
● Produces reliable statistical estimates
4. Distance-Based Techniques
k-Nearest Neighbors (kNN) Imputation
Missing values are estimated using values from the k most similar records.
● Maintains data patterns
● Sensitive to choice of k and distance metric
● Slower for large datasets
5. Time-Series Specific Techniques
a) Interpolation
Missing values are estimated using neighboring time points.
● Linear interpolation
● Polynomial or spline interpolation
b) Forward and Backward Filling
● Forward fill uses previous known value
● Backward fill uses next known value
Used mainly in sensor and sequential data.
6. Categorical Data Techniques
a) ‘Unknown’ Category
Missing values are replaced with a separate category such as Unknown.
● Prevents loss of data
● Keeps dataset size intact
b) Most Frequent Category
Replaces missing values with the most common category.
7. Machine Learning-Based Techniques
Advanced models are trained to predict missing values:
● Decision Trees
● Random Forest
● Neural Networks
● Autoencoders
These techniques offer high accuracy for complex datasets but require more computation and
expertise.
Data Transformation & Normalization
Data preprocessing is incomplete without data transformation and normalization, which
ensure that data is in a suitable form for analysis and modeling, especially for Big Data
Analytics.
1. Data Transformation
Data transformation converts data from its original format into a more suitable and
meaningful form. It helps in reducing complexity, improving quality, and making data
compatible with analytical models. Common techniques include:
a) Aggregation
● Combines data from smaller units into larger, summarized units.
● Example: Converting daily sales data into monthly or yearly sales totals.
● Benefit: Reduces dataset size and highlights trends over time.
b) Generalization
● Replaces specific, detailed values with higher-level concepts or ranges.
● Example: Converting age 27, 32, 41 into age groups 20–30, 30–40, 40–50.
● Benefit: Simplifies data and protects privacy (useful for anonymization).
c) Encoding
● Converts categorical data into numeric form for machine learning models.
● Techniques:
○ Label Encoding: Assigns unique numbers to each category
○ One-Hot Encoding: Creates binary columns for each category
● Example: “Red”, “Blue”, “Green” → 0, 1, 2 (label) or separate binary columns (one-hot)
● Benefit: Makes categorical data usable in algorithms that require numeric input.
1. Data Transformation Example
a) Aggregation
● Raw data: Every customer purchase logged daily.
● Transformation: Aggregate daily sales into monthly total sales for each product.
● Benefit: Helps identify trending products and seasonal demand patterns.
b) Generalization
● Raw data: Customer ages recorded as 22, 27, 35, 41…
● Transformation: Group ages into 20–30, 31–40, 41–50.
● Benefit: Simplifies customer segmentation for targeted marketing.
c) Encoding
● Raw data: Product categories like “Electronics”, “Clothing”, “Books”.
● Transformation:
○ Label Encoding: Electronics → 0, Clothing → 1, Books → 2
○ One-Hot Encoding: Creates binary columns for each category.
● Benefit: Enables machine learning models to process categorical features effectively.
2. Data Normalization Example
Scenario
Features like Price (10–1000 USD) and Quantity Sold (1–20) are used together in predictive
models. Without normalization, the model may give more importance to price because its
range is larger.
Techniques Applied
● Min-Max Scaling: Rescale Price and Quantity to 0–1 for uniform contribution.
● Z-Score Normalization: Center features around mean 0 and std deviation 1 for
clustering customers.
● Decimal Scaling: Quickly scale revenue numbers for dashboards.
Outcome
● Improved accuracy of recommendation systems
● Better customer segmentation and personalization
● Efficient trend analysis and demand forecasting
Dimensionality Reduction
In Big Data and machine learning, datasets often contain high-dimensional data—meaning
they have many features or variables. High dimensionality can lead to slow computations,
overfitting, and difficulty in visualization. Dimensionality reduction simplifies the data while
retaining the most important information.
Why Dimensionality Reduction is Needed
● High-dimensional data increases computational complexity.
● Many features may be redundant or irrelevant, adding noise.
● Visualization of data with more than 3 dimensions is difficult.
● Reducing dimensions improves model performance and interpretability.
Techniques for Dimensionality Reduction
1. Principal Component Analysis (PCA)
● PCA transforms original features into a smaller set of uncorrelated variables called
principal components.
● Retains maximum variance in the data.
● Use case: Reducing hundreds of features in customer or financial datasets.
2. t-Distributed Stochastic Neighbor Embedding (t-SNE)
● Non-linear technique mainly used for visualization of high-dimensional data in 2D or
3D.
● Preserves the local structure of data (similar points stay close).
● Use case: Visualizing clusters in image or text datasets.
3. Autoencoders (Deep Learning)
● Neural networks trained to compress data into a lower-dimensional representation
(encoding) and reconstruct it.
● Captures non-linear relationships in data.
● Use case: Image compression, anomaly detection, and large-scale sensor data.
4. Feature Selection
Reduces dimensions by selecting most relevant features:
● Filter methods: Select features based on statistical measures (e.g., correlation,
chi-square).
● Wrapper methods: Select features based on model performance (e.g., recursive
feature elimination).
● Embedded methods: Feature selection occurs during model training (e.g., Lasso
Regression).
● Use case: Predictive modeling, improving accuracy and interpretability.
Benefits of Dimensionality Reduction
● Faster computation due to fewer features
● Reduced noise by removing irrelevant/redundant variables
● Better visualization of high-dimensional data
● Improved model performance and reduced risk of overfitting
Case Study: Healthcare Data Analytics
Healthcare generates enormous amounts of data from hospitals, clinics, labs, and wearable
devices. However, this data is often large, inconsistent, and incomplete, making
preprocessing essential before analysis.
Problem
● Hospitals generate terabytes of patient records, including demographics, medical
history, lab results, prescriptions, and imaging data.
● Challenges include:
○ Missing values (e.g., unrecorded lab tests)
○ Inconsistent data (different units, spellings, or formats)
○ High dimensionality (hundreds of features per patient)
● Raw data is unsuitable for accurate disease prediction or analytics.
Data Preprocessing Steps
1. Handling Missing Values
● Techniques like mean/median imputation or kNN prediction were applied to fill
missing lab results or vital statistics.
2. Encoding Categorical Values
● Categorical features like disease types, drug names, and patient conditions were
converted to numeric form using:
○ Label Encoding
○ One-Hot Encoding
3. Normalization
● Lab results such as blood pressure, cholesterol, and glucose levels were normalized
using:
○ Min-Max Scaling
○ Z-score Normalization
● This ensured that all features contributed equally to predictive models.
4. Dimensionality Reduction (PCA)
● Original dataset had 200+ features (lab tests, demographics, vitals).
● Principal Component Analysis (PCA) reduced features to 20 key components that
retained most variance.
● Reduced computation and noise while improving model interpretability.
Outcome
● Preprocessed data improved machine learning model performance for disease
prediction.
● Accuracy increased by 30% compared to using raw data.
● Benefits included:
○ Faster computation
○ Reduced noise and redundancy
○ Better insights for hospital decision-making
Conclusion
This case study demonstrates how data preprocessing, normalization, encoding, and
dimensionality reduction are critical in Big Data Analytics for healthcare. Proper
preprocessing transforms messy, high-dimensional healthcare data into a reliable, structured
dataset, enabling accurate predictions and improved patient care.
ASSIGNMENT-2
1. Define data preprocessing. Why is it important in big data analytics?
1. Define Data Preprocessing. Why is it important in Big Data Analytics?
(6-Mark Answer)
Data preprocessing is the process of converting raw, noisy, incomplete, and inconsistent
data into a clean, structured, and suitable format for analysis and modeling. It involves steps
such as data cleaning, handling missing values, data transformation, normalization, and
dimensionality reduction.
In Big Data Analytics, preprocessing is essential because real-world big data is generated
from multiple heterogeneous sources and is often unreliable in its raw form.
Importance of Data Preprocessing in Big Data Analytics
1. Improves Data Quality
Preprocessing removes noise, duplicates, and inconsistencies, resulting in accurate and
reliable data.
2. Ensures Accurate Analysis and Results
Clean and well-prepared data leads to correct insights, predictions, and analytical
outcomes.
3. Handles Large and Complex Data
Big data is high in volume, velocity, and variety. Preprocessing helps manage and
organize such data efficiently.
4. Enhances Performance of Analytical Models
Machine learning and statistical models perform better and faster when trained on
preprocessed data.
5. Reduces Errors in Decision Making
Poor-quality data can lead to misleading conclusions. Preprocessing minimizes this
risk.
6. Supports Efficient Storage and Processing
Techniques like normalization and dimensionality reduction reduce data size and
computational cost.
2. What are the different techniques for handling missing data?
Q2. What are the different techniques for handling missing data?
(6-Mark Answer)
Missing data is a common issue in real-world datasets and can negatively affect data analysis
and machine learning results. Several techniques are used to handle missing data depending
on the nature and amount of missing values.
1. Deletion Techniques
In this method, records containing missing values are removed from the dataset.
● Listwise deletion removes entire rows with missing values.
● Pairwise deletion uses available data without deleting full records.
This technique is suitable when the percentage of missing data is very small.
2. Mean / Median / Mode Imputation
Missing values are replaced using statistical measures:
● Mean for numerical data
● Median when data contains outliers
● Mode for categorical data
This method is simple and widely used but may reduce data variability.
3. Regression Imputation
A regression model is built using complete data to predict missing values based on other
attributes.
This technique preserves relationships among variables but is computationally expensive.
4. k-Nearest Neighbors (kNN) Imputation
Missing values are estimated using values from the most similar records.
It maintains data patterns but can be slow for large datasets.
5. Interpolation Methods
For time-series data, missing values are filled using interpolation techniques such as linear or
spline interpolation.
This method maintains continuity in sequential data.
6. ‘Unknown’ Category for Categorical Data
Missing categorical values are replaced with a separate category like “Unknown” to avoid
data loss and preserve dataset size.
3. Explain Min-Max Normalization with its formula.
Q3. Explain Min-Max Normalization with its formula
Definition:
Min-Max Normalization is a data scaling technique used in data preprocessing to transform
numerical features into a fixed range, usually 0 to 1. This ensures that all features contribute
equally to analysis and machine learning models.
Purpose
● Prevents features with larger ranges from dominating the model.
● Speeds up convergence in machine learning algorithms.
● Preserves relative relationships between original values.
Advantages
● Simple and effective
● Preserves original distribution
● Scales data for algorithms sensitive to feature magnitude (e.g., KNN, Neural Networks,
SVM)
4. What is dimensionality reduction? Name two common methods.
5. Differentiate between data cleaning and data transformation with examples.
6. In a healthcare dataset, patient blood pressure values are missing for 15% of records.
Discuss different strategies you could use to handle these missing values. Which
would you choose and why?
Q6. Handling Missing Blood Pressure Values in a
Healthcare Dataset
Problem Statement
In a healthcare dataset, 15% of patient blood pressure (BP) values are missing. Missing
medical data can affect diagnosis accuracy, predictive models, and clinical decisions.
Therefore, appropriate handling of missing values is essential.
Strategies for Handling Missing Blood Pressure Data
1. Overview of Possible Techniques
Strategy Description Advantages Limitations
Ignore / Delete Remove records with Simple, no Data loss; not suitable
Records missing BP values computation when 15% is missing
Mean Imputation Replace missing BP Easy, fast Reduces variance;
with average BP value ignores patient
differences
Median Replace with median Robust to outliers Still oversimplifies
Imputation BP medical data
Mode Imputation Replace with most Useful for Not suitable for
frequent BP value categorical data continuous BP values
Regression Predict BP using age, Uses relationships Can introduce model
Imputation weight, heart rate in data bias
k-NN Imputation Use similar patients to More personalized Computationally
estimate BP expensive
Time-Series Estimate BP using Very accurate for Requires time-ordered
Interpolation previous/future monitored patients data
readings
Clinical Use medical Clinically Requires domain
Rule-Based guidelines meaningful expertise
Imputation
Since 15% of blood pressure values are missing, deleting records would cause data loss.
Predictive methods such as regression or k-NN imputation are preferred because they utilize
patient-specific information and produce medically meaningful and accurate estimates.
7. You are analyzing a bank loan dataset with 100+ features. Explain how
dimensionality reduction (PCA) could improve both efficiency and accuracy. Give a
real-world example.
Q7. Dimensionality Reduction Using PCA in a Bank Loan
Dataset
Introduction
In a bank loan dataset with 100+ features, many variables are often correlated, redundant, or
noisy. High dimensionality increases computational cost and may reduce model performance
due to the curse of dimensionality. Principal Component Analysis (PCA) is a dimensionality
reduction technique that transforms high-dimensional data into a smaller set of meaningful
features while preserving maximum variance.
What is PCA (Principal Component Analysis)?
PCA is an unsupervised statistical technique that converts original correlated features into a
new set of uncorrelated variables called principal components (PCs).
Key Characteristics of PCA
Aspec Description
t
Type Unsupervised learning
Output Orthogonal principal components
Goal Maximize variance with fewer dimensions
Nature Linear transformation
Result Reduced features with minimal information
loss
How PCA Works (Step-by-Step)
Ste Explanation
p
1 Standardize features (mean = 0, variance =
1)
2 Compute covariance matrix
3 Calculate eigenvalues & eigenvectors
4 Select top-k eigenvectors
5 Project data onto new feature space
Mathematical Insight (Theory)
● Covariance Matrix:
Σ=1nXTX\Sigma = \frac{1}{n} X^T XΣ=n1XTX
● Eigenvalue Equation:
Σv=λv\Sigma v = \lambda vΣv=λv
● Largest eigenvalues → most important components
How PCA Improves Efficiency
Problem Without PCA With PCA
Number of 100+ 10–20
Features
Computation Time High Low
Memory Usage High Reduced
Training Time Slow Faster
Model Complexity Very High Simplified
✔ PCA reduces redundant features → faster training & prediction
How PCA Improves Accuracy
Issue PCA Benefit
Multicollinearity Removes correlated variables
Noise Low-variance noise is discarded
Overfitting Reduced feature space lowers
overfitting
Generalization Improves performance on unseen data
✔ Cleaner features → better predictive accuracy
Real-World Banking Example
Problem
A bank analyzes loan approval using 120 features, such as:
● Income, credit score
● Transaction history
● Spending patterns
● Employment stability
● Previous loan defaults
Many features are highly correlated.
Applying PCA
Step Outcome
Original Features 120
PCA Applied Top 15
components
Variance Retained 95%
Redundant Removed
Features
Noise Reduced
Impact on Loan Default Prediction
Metric Before PCA After PCA
Model Accuracy 82% 89%
Training Time 12 sec 3 sec
Overfitting High Low
Interpretability Poor Improved
Example Interpretation of Components
Principal Component Meaning
PC1 Customer financial
strength
PC2 Credit behavior
PC3 Spending risk
PC4 Employment stability
Final Conclusion (Exam-Ready)
In a bank loan dataset with 100+ features, PCA improves efficiency by reducing
computational cost and improves accuracy by eliminating noise, multicollinearity, and
overfitting. By retaining only the most informative components, PCA enables faster and
more accurate loan default prediction.
8. In a retail sales dataset, product categories are stored as text (“Shirts”, “shirt”,
“SHIRTS”). What preprocessing steps would you apply to ensure consistency?
Q8. Preprocessing Steps to Ensure Consistency in Retail
Product Categories
Problem Statement
In a retail sales dataset, product categories are stored inconsistently as
“Shirts”, “shirt”, “SHIRTS”. Such inconsistencies cause incorrect grouping, wrong sales totals,
and poor model performance.
Preprocessing Steps to Ensure Consistency
1. Standardization of Text Case
Step Action
Convert all Lowercase or
text uppercase
Example:
● “Shirts”, “shirt”, “SHIRTS” → “shirts”
2. Remove Leading & Trailing Whitespaces
Issue Solution
Extra Apply trimming
spaces
Example:
● “ shirts ” → “shirts”
3. Correct Spelling Variations
Issue Solution
Typographical Spell check / dictionary
errors mapping
Example:
● “shrits” → “shirts”
4. Use Category Mapping (Data Dictionary)
Original Value Standard Value
Shirts shirts
shirt shirts
SHIRTS shirts
✔ Ensures uniform naming
5. Remove Special Characters (if any)
Issue Action
Special Regex
symbols cleaning
Example:
● “shirts!” → “shirts”
6. Handle Synonyms (Optional)
Term Standard Category
T-shirt shirts
Casual shirts
shirt
7. Encode Categories (After Cleaning)
Method Usage
Label Encoding ML models
One-Hot Regression /
Encoding classification
Example:
● shirts → 1
Before and After Example Table
Before Preprocessing
Product ID Category
101 Shirts
102 shirt
103 SHIRTS
After Preprocessing
Product ID Category
101 shirts
102 shirts
103 shirts
Why These Steps Are Important
Reason Benefit
Correct grouping Accurate sales
analysis
Avoid duplicates Reliable reports
ML compatibility Better model accuracy
Data integrity Consistent records
Final Conclusion (Exam-Ready)
To ensure consistency in retail product categories, preprocessing steps such as text
standardization, trimming, spelling correction, category mapping, and encoding are applied.
These steps ensure accurate analysis, reporting, and improved machine learning
performance.