Scribd
Upload
2 views12 pages

Essential Data Cleaning Techniques

The document provides a comprehensive guide on various data cleaning techniques, including handling missing values, removing duplicates, fixing data types, cleaning text data, handling outliers, standardizing values, validating data, and fixing inconsistent data. It emphasizes the importance of data cleaning for accurate insights and effective decision-making, noting that it constitutes a significant portion of a data analyst's responsibilities. A final checklist is included to ensure data integrity before analysis.

Uploaded by

Mohd Aktharpasha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views12 pages

Essential Data Cleaning Techniques

The document provides a comprehensive guide on various data cleaning techniques, including handling missing values, removing duplicates, fixing data types, cleaning text data, handling outliers, standardizing values, validating data, and fixing inconsistent data. It emphasizes the importance of data cleaning for accurate insights and effective decision-making, noting that it constitutes a significant portion of a data analyst's responsibilities. A final checklist is included to ensure data integrity before analysis.

Uploaded by

Mohd Aktharpasha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Pooja Pawar

Handling Missing Values


isna() – Detect missing (NaN) values

notna() – Identify non-null entries

dropna() – Remove rows or columns

fillna() – Replace missing values with


constants or statistics

interpolate() – Fill gaps using trends

mean() – Numeric imputation

median() – Robust imputation for skewed


data

mode() – Categorical imputation


Pooja Pawar
Removing Duplicates
duplicated() – Detect duplicate rows

drop_duplicates() – Remove duplicate rows

subset= – Check duplicates in specific


columns

keep=‘first’ – Keep the first occurrence

keep=‘last’ – Keep the last occurrence

value_counts() – Identify repeated values

groupby() – Aggregate data by duplicate


keys

reset_index() – Rebuild index after cleaning

Pooja Pawar
Fixing Data Types
dtype – Inspect column data types

astype() – Convert data types

to_datetime() – Fix and standardize date


columns

to_numeric() – Convert numeric text

errors='coerce' – Handle invalid values

[Link] – Extract year

[Link] – Extract month

tz_localize() – Assign timezone to date time


data
Pooja Pawar
Cleaning Text Data
[Link]() – Normalize text to lowercase

[Link]() – Enforce consistent uppercase


text

[Link]() – Remove leading and trailing


spaces

[Link]() – Fix inconsistent or incorrect


text

[Link]() – Identify patterns or keywords

[Link]() – Break compound strings into


parts

[Link]() – Validate text length consistency

[Link]() – Extract patterns using regex


Pooja Pawar
Handling Outliers
describe() – Review value distribution

quantile() – Define IQR-based boundaries

between() – Filter values within valid


ranges

clip() – Cap extreme values

mean() – Reference central tendency

std() – Measure data spread

abs() – Remove sign-based noise

[Link]() – Apply conditional


replacements
Pooja Pawar
Standardizing Values
replace() – Correct inconsistent labels

map() – Apply controlled value mapping

round() – Standardize decimal precision

apply() – Apply custom transformation


logic

astype() – Normalize data types

mul() – Convert values to new units

div() – Scale values proportionally

clip() – Normalize values to a range

Pooja Pawar
Validating Data
unique() – Check allowed values

nunique() – Validate value cardinality

isin() – Enforce domain constraints

between() – Validate numeric ranges

duplicated() – Verify key uniqueness

isna().sum() – Review missing value


distribution

is_monotonic_increasing – Check sequence


order

assert – Enforce data rules


Pooja Pawar
Fixing Inconsistent Data
loc[] – Apply conditional updates

where() – Replace values conditionally

mask() – Inverse conditional replacement

merge() – Correct data using references

update() – Overwrite incorrect values

combine_first() – Fill from trusted source

drop() – Remove invalid records

query() – Filter data using conditions

Pooja Pawar
Final Pre-Analysis
Checklist
No unexpected nulls

No duplicate primary keys

Correct data types

Clean text fields

Valid ranges

Business rules satisfied

Pooja Pawar
Why Data Cleaning
Matters
Clean data means:

Accurate insights

Reliable dashboards

Better decisions

Fewer stakeholder questions

Trust in analytics

Data Cleaning = 70% of a Data Analyst’s job


Master it, and analysis becomes easy.
Pooja Pawar
Follow for more
data analytics
content

Pooja Pawar

You might also like