Scribd
Upload
2 views3 pages

Data Preprocessing & Data Wrangling

Data preprocessing involves cleaning, transforming, and organizing raw data for effective analysis and model building, addressing issues like noise and lack of structure. Data wrangling is a subset of preprocessing focused on preparing data into an analysis-ready format through steps like collection, cleaning, transformation, integration, reduction, and validation. Key techniques include data transformation, normalization, encoding, and aggregation, which help summarize large datasets, reduce complexity, and improve analysis efficiency.

Uploaded by

sayyedzeeshan696
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as ODT, PDF, TXT or read online on Scribd
2 views3 pages

Data Preprocessing & Data Wrangling

Data preprocessing involves cleaning, transforming, and organizing raw data for effective analysis and model building, addressing issues like noise and lack of structure. Data wrangling is a subset of preprocessing focused on preparing data into an analysis-ready format through steps like collection, cleaning, transformation, integration, reduction, and validation. Key techniques include data transformation, normalization, encoding, and aggregation, which help summarize large datasets, reduce complexity, and improve analysis efficiency.

Uploaded by

sayyedzeeshan696
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as ODT, PDF, TXT or read online on Scribd

Data Preprocessing & Data Wrangling

What is data preprocessing and why do we need it?

Data preprocessing is the cleaning, transforming and organising raw data into suitable format so it
can be effectively analysed and be used to build models

Raw data is often not suited for final product as it may have noise and not have a structure, this
makes data preprocessing an essential step in DS life cycle.

Why do we need it?


• Improve Data quality
• Reduces noise
• Transforms data into usable format
• Increases accuracy of result
• Reduces the processing time
• Required for machine learning algorithms

What is data wrangling?

Data wrangling is a part of data preprocessing that focuses on cleaning, transforming, and
structuring raw data into an analysis-ready format.

If data is cleaned, transformed, noise is reduced, and data from multiple sources is combined, then
data wrangling has been successfully performed.

Key process in data wrangling

• Transforming data → format/structure conversion


• Reducing noise → removing errors, outliers, inconsistencies
• Combining data → integrating multiple data sources

Steps involved in data wrangling:


• Data collection
◦ Collecting raw data from multiple sources like databases, files or external datasets.
• Data cleaning
◦ Reducing the noise
◦ Removing inconsistencies and replacing or removing missing data.
• Data transformation
◦ Transforming the data into a structured format.
• Data integration
◦ Combining the data from multiple sources.
• Data reduction
◦ Removing unnecessary data or repeated data
• Data validation
◦ Checking the accuracy, consistency and completeness
◦ Ensuring the data in ready for use in model building or analysis.

What is data transformation?

Data transformation is an essential step involved in data wrangling where the data
from one format is transformed into another suitable format for the given task.

Why is it needed?
• Many task requires data in a specific format.
• Raw data may not have a structure which can be used for the given task.
• Transformation increases Data consistency and model efficiency.
• The quality of data increases significantly

Common data transformation techniques:


• Normalising
◦ Remapping values into a common scale
• Encoding
◦ Converting categorial data into numerical form
• Aggregation
◦ Summarising data (sales in a day -> total sales in a week)
• Formatting
◦ Changing the data types or formats (date,time,text).
• Feature Construction
◦ Creating new attributes from existing ones.

What is data aggregation?

Data aggregation is the process of collecting and summarizing data from multiple
records or sources to produce a combined and meaningful result.
It is commonly used during data preprocessing and data transformation to reduce data
size and improve analysis.

Purpose of Data Aggregation


• To summarize large datasets
• To reduce data complexity
• To identify trends and patterns
• To improve efficiency of analysis

Common Aggregation Operations


• Sum – total values
• Average (Mean) – central value
• Count – number of records
• Minimum / Maximum – smallest and largest values

Example:
• Daily sales data aggregated into monthly sales totals
• Student marks aggregated to calculate average score

You might also like