Scribd
Upload
24 views3 pages

Data Cleaning Guide

The document outlines a comprehensive guide to data cleaning steps and techniques, including understanding data, handling missing values, duplicates, outliers, and encoding categorical variables. It emphasizes the importance of feature scaling, engineering, and validation, while also addressing specific considerations for classification and regression tasks. The final steps include ensuring no data leakage and saving the cleaned dataset in various formats.

Uploaded by

mmanikandan0005
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
24 views3 pages

Data Cleaning Guide

The document outlines a comprehensive guide to data cleaning steps and techniques, including understanding data, handling missing values, duplicates, outliers, and encoding categorical variables. It emphasizes the importance of feature scaling, engineering, and validation, while also addressing specific considerations for classification and regression tasks. The final steps include ensuring no data leakage and saving the cleaned dataset in various formats.

Uploaded by

mmanikandan0005
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Complete Data Cleaning Steps & Techniques

1. Understanding & Inspecting the Data

• Load the dataset (CSV, SQL, JSON, Parquet, etc.).

• Check data types (numeric, categorical, datetime, text).

• Explore statistics (mean, median, mode, std, ranges).

• Visualize with histograms, boxplots, and correlation heatmaps.

2. Handling Missing Data

• Drop missing values if small and random.

• Impute numerical values (mean, median, KNN, regression).

• Impute categorical values (mode, 'Unknown', frequency encoding).

• Use advanced methods like MICE or interpolation for time-series.

3. Handling Duplicates

• Identify duplicates on all or key columns.

• Remove only if redundant.

4. Outlier Detection & Treatment

• Methods: Z-score, IQR, Isolation Forest, DBSCAN.

• Treatment: Winsorization, transformations, or removal.

5. Data Type Conversion

• Convert codes to categories.

• Parse dates properly.

• Ensure numeric fields are stored as numbers.

6. Encoding Categorical Variables

• One-hot encoding (Nominal).

• Label encoding (Ordinal).

• Target or mean encoding (beware of leakage).

• Frequency encoding for high-cardinality features.


7. Feature Scaling

• Standardization (Z-score).

• Normalization (Min-Max scaling).

• Robust scaling for outliers.

• Keep consistent between train/test.

8. Feature Engineering & Transformation

• Numerical: log/sqrt transformations, polynomial features.

• Categorical: combine rare classes.

• Datetime: extract time components, time differences.

• Text: cleaning, TF-IDF, embeddings.

9. Handling Class Imbalance (Classification Only)

• Resampling: oversampling (SMOTE), undersampling.

• Algorithmic: class weights, focal loss.

• Evaluation: F1-score, Precision-Recall, ROC-AUC.

10. Multicollinearity Check

• Use correlation matrix or VIF.

• Remove or combine correlated features (VIF > 10).

11. Data Splitting

• Train-test split (70-30, 80-20).

• Stratified split for classification.

• Time-series split without shuffling.

12. Final Dataset Validation

• Ensure no data leakage.

• Check class distribution.

• Confirm realistic feature ranges.

• Save cleaned dataset (CSV, Parquet, SQL).


Summary

• Universal steps: Inspect → Clean → Fix types → Handle outliers → Encode → Scale →
Engineer → Validate.

• Classification-specific: handle imbalance, stratified split, class-aware metrics.

• Regression-specific: treat outliers, scale features, check heteroscedasticity.

You might also like