UNIT:01

Introduction to Data Wrangling

Data wrangling is a crucial initial step in any data analysis, visualization, or machine
learning project. It involves transforming raw data into a format that is more suitable
for analysis. This process includes a series of steps, such as cleaning, transforming, and
enriching the data to ensure that it is structured and reliable.

Definition of Data Wrangling

Data wrangling, also known as data munging, is the process of converting raw data into
a structured and clean format that is ready for analysis. It encompasses various
activities aimed at handling inconsistencies, missing data, and ensuring that the data
meets the necessary quality standards for reliable analysis.

Importance of Data Wrangling

The quality of your analysis is highly dependent on the quality of your data. Even
sophisticated algorithms can yield poor results if the input data is flawed. Here are a few
reasons why data wrangling is essential:

1. Ensures Data Quality:

o High-quality data is accurate, consistent, and complete. Data wrangling

helps identify and rectify issues such as errors, inconsistencies, and
outliers.

2. Enhances Analysis:

o Clean and structured data allows analysts and data scientists to focus on
deriving insights rather than spending time dealing with data issues.

3. Reduces Complexity:

o By transforming data into a more usable form, data wrangling simplifies

complex datasets, making them easier to understand and analyze.

4. Improves Model Performance:

o Properly wrangled data improves the performance of machine learning

models, as these models rely on clean, normalized, and standardized data.

5. Facilitates Reproducibility:

o Data wrangling often involves documenting each transformation step,

which makes the process reproducible and transparent for other analysts
or stakeholders.
Common Data Wrangling Tasks

Data wrangling involves a series of tasks, each aimed at handling specific types of issues
found in raw data:

1. Data Cleaning:

o Removing Duplicates: Eliminating repeated entries to avoid skewed

analysis.

o Handling Missing Values: Addressing gaps in data using techniques like

imputation or removal.

o Correcting Errors: Fixing inaccuracies in data such as incorrect spellings,

erroneous values, etc.

2. Data Transformation:

o Normalizing and Standardizing: Adjusting data values to a common scale.

o Encoding Categorical Variables: Converting categories into numerical

values for machine learning models.

o Aggregating Data: Summarizing data to capture trends, such as computing

monthly sales from daily records.

3. Data Enrichment:

o Adding Contextual Information: Incorporating additional data, such as

weather information for sales data or demographic information for
customer records.

o Creating New Features: Generating new variables from existing data, such
as calculating customer lifetime value or creating time-based features.

4. Data Integration:

o Merging Datasets: Combining data from different sources to create a

unified dataset.

o Resolving Conflicts: Addressing discrepancies between data from

different sources.

5. Data Validation:

o Consistency Checks: Ensuring that data adheres to defined rules, such as

dates being in a valid range or numerical fields not containing text.

o Cross-Validation: Comparing data against external sources or historical

trends to verify accuracy.
Tools and Techniques for Data Wrangling

Data wrangling can be performed using various tools and techniques, depending on the
complexity of the data and the specific requirements of the analysis:

1. Programming Languages:

o Python: Widely used for data wrangling with libraries like Pandas, NumPy,
and Dask.

o R: Provides powerful packages like dplyr, tidyr, and [Link] for data
manipulation.

2. Visualization Tools:

o Tableau/Power BI: Useful for visual data cleaning and validation.

o Matplotlib/Seaborn: Used in Python for visual exploration of data

distributions and anomalies.

3. Data Profiling Tools:

o Pandas Profiling: Automatically generates comprehensive data profiling

reports in Python.

o DataPrep: A Python library for easy EDA and data cleaning.

o Great Expectations: A tool for validating, documenting, and profiling your

data.

4. Spreadsheets:

o Excel/Google Sheets: Often used for manual data cleaning and exploration,
especially for smaller datasets.

Real-World Examples of Data Wrangling

1. Business Analytics:

o A retail company collects data from multiple sources, including sales

records, customer interactions, and online behavior. Data wrangling is
used to clean and merge these datasets, enabling comprehensive analysis
of customer behavior and sales trends.

2. Healthcare:

o Medical researchers use data wrangling to clean and integrate patient

records from various hospitals. This allows for consistent analysis of
treatment outcomes across different demographic groups.
 3. Finance:

o Financial analysts wrangle data from stock markets, economic indicators,

and company reports to build predictive models for investment strategies.

4. Marketing:

o Marketing teams clean and merge data from CRM systems, social media
platforms, and email campaigns to better understand customer
engagement and optimize marketing efforts.

Challenges in Data Wrangling

Data wrangling can be a complex and time-consuming process, often requiring a deep
understanding of the data and domain-specific knowledge. Common challenges include:

1. Handling Large Datasets:

o Working with large volumes of data can be resource-intensive and

require optimization techniques to ensure efficient processing.

2. Dealing with Unstructured Data:

o Unstructured data like text, images, or social media feeds need specialized
techniques for cleaning and transformation.

3. Data Privacy and Security:

o Ensuring that sensitive information is handled appropriately, especially

when dealing with personal or financial data.

4. Automating the Wrangling Process:

o Creating automated, reusable pipelines for data wrangling can be

challenging but is essential for scalability and efficiency.

5. Changing Data Sources:

o Data sources can change over time, requiring ongoing adjustments to data
wrangling processes to maintain data quality and consistency.

Need for Data Cleanup

Data cleanup, also known as data cleansing or data cleaning, is an essential step
in the data wrangling process. It involves identifying and correcting errors,
inconsistencies, and inaccuracies in the data to ensure that it is of high quality
and suitable for analysis. High-quality data is crucial for making reliable and
accurate decisions, as poor data can lead to misleading conclusions and
ineffective models.

Common Issues in Raw Data

Raw data often comes with a variety of issues that need to be addressed before
it can be used effectively. These issues can arise from different sources such as
data entry errors, system errors, or inconsistencies between different data
sources. Here are some common problems found in raw data:

1. Incomplete Data:

o Definition: Missing values in a dataset, which can occur when no data

value is stored for a variable in an observation.

o Causes:

 Data entry errors or omissions.

 Non-response in surveys.

 Data loss during transmission or storage.

o Impact: Incomplete data can lead to biased analysis and affect the
performance of machine learning models.

o Example: In a dataset of customer information, some entries may lack

email addresses or phone numbers.

2. Inconsistent Data:

o Definition: Data that lacks uniformity in format, type, or value, making it

difficult to compare or aggregate.

o Causes:

 Different data entry standards (e.g., date formats like

MM/DD/YYYY vs. DD/MM/YYYY).

 Variations in categorical data (e.g., "Male" vs. "M" vs. "male").

 Data collected from multiple sources with different formats.

o Impact: Inconsistent data can lead to errors in data aggregation, filtering,

and analysis.
 o Example: In a dataset of product sales, the product category might be
recorded as "electronics" in one entry and "Electronics" in another.

3. Duplicate Data:

o Definition: Repeated entries for the same observation, which can result
from data entry errors or merging datasets without proper checks.

o Causes:

 Manual data entry or batch processing errors.

 Multiple data sources recording the same information.

o Impact: Duplicates can skew statistical analysis, leading to incorrect

insights.

o Example: A customer’s information is recorded twice in a CRM system,

resulting in inflated customer count and sales figures.

4. Erroneous Data:

o Definition: Data that contains incorrect or implausible values.

o Causes:

 Typographical errors during data entry.

 Misconfigurations in data collection systems.

o Impact: Erroneous data can distort analysis and lead to incorrect

conclusions.

o Example: An age field containing negative values or a transaction date

that is set in the future.

5. Outliers:

o Definition: Extreme values that deviate significantly from other

observations in the dataset.

o Causes:

 Data entry mistakes.

 Genuine but rare events (e.g., a sudden surge in sales due to a

marketing campaign).

o Impact: Outliers can skew statistical measures such as mean and standard
deviation and affect the performance of machine learning models.
 o Example: A transaction value of $1,000,000 in a dataset where the
average transaction value is $50.

Impacts of Unclean Data

Unclean data can have several negative consequences, impacting both analysis
and decision-making processes:

1. Inaccurate Analysis:

o Analysis based on flawed data can lead to incorrect conclusions and

poor decision-making. For example, missing or incorrect data in sales
records can result in a miscalculation of total sales and revenue
projections.

2. Increased Processing Time:

o Dealing with unclean data requires additional preprocessing time

and effort, delaying analysis and reporting. This is particularly
problematic in industries where timely insights are critical, such as
finance or healthcare.

3. Reduced Model Performance:

o Machine learning models trained on unclean data often perform

poorly. Models may overfit to noisy data, miss important patterns, or
generate biased predictions, leading to ineffective outcomes.

4. Misleading Visualizations:

o Data visualizations based on unclean data can be misleading, causing

stakeholders to misinterpret trends or relationships in the data.

5. Inconsistent Reports:

o When data quality issues are not addressed, different reports based
on the same data can yield conflicting results, undermining trust in
the data and the insights derived from it.

Steps to Identify Data Quality Issues

Identifying data quality issues is a critical part of the data wrangling process. It
involves systematically examining the data to uncover and understand potential
problems.

1. Data Profiling:

o Definition: The process of reviewing the data to understand its

structure, content, and quality. This includes analyzing metadata (e.g.,
 column names, data types) and generating descriptive statistics (e.g.,
mean, median, mode, range).

o Techniques:

 Summary Statistics: Calculating metrics like mean, median,

standard deviation, and count of missing values for each
column.

 Data Distribution Analysis: Visualizing the distribution of

values using histograms or box plots to identify skewness,
outliers, and anomalies.

 Relationship Analysis: Assessing correlations and

dependencies between variables to identify inconsistencies or
unexpected patterns.

2. Data Validation:

o Definition: The process of checking data for accuracy and consistency

against predefined rules or standards.

o Techniques:

 Domain Checks: Ensuring that values fall within an acceptable

range (e.g., age between 0 and 120).

 Format Checks: Verifying that data adheres to a specific

format (e.g., email addresses contain an "@" symbol).

 Business Rule Validation: Ensuring that data complies with

business rules (e.g., order date must be earlier than delivery
date).

3. Exploratory Data Analysis (EDA):

o Definition: An approach for analyzing data sets to summarize their

main characteristics, often using visual methods.

o Techniques:

 Visualization: Use of graphs and plots (e.g., scatter plots, bar

charts) to explore data distributions and identify anomalies.

 Cross-Tabulation: Analyzing the relationship between two or

more categorical variables to identify inconsistencies or
unexpected patterns.
  Descriptive Statistics: Calculating statistics like mean, median,
variance, and skewness to understand the data’s central
tendency and dispersion.

4. Automated Data Quality Checks:

o Definition: Using software tools to automatically identify and report

data quality issues.

o Tools:

 Pandas Profiling: Automatically generates a detailed report of

data quality issues in a pandas DataFrame.

 Great Expectations: A Python library for creating, testing, and

documenting data expectations.

 Data Prep: A Python library for EDA and data cleaning.

Examples of Data Cleanup in Different Domains

1. Healthcare:

o Scenario: Patient records from different hospitals are merged for a

study. The dataset has missing values for patient age and
inconsistent formats for test results.

o Cleanup Process:

 Fill missing age values using imputation (e.g., median age).

 Standardize test result units (e.g., converting mg/dL to

mmol/L).

 Remove duplicate patient records.

2. Finance:

o Scenario: A financial analyst is working with a dataset of daily stock

prices. The dataset has missing values on weekends and erroneous
data entries with negative stock prices.

o Cleanup Process:

 Remove weekends and holidays to focus on trading days.

 Correct negative prices using external validation or remove

affected entries.

3. Retail:
 o Scenario: A retail company wants to analyze customer purchase
behavior. The dataset contains duplicated entries and inconsistent
product categories.

o Cleanup Process:

 Remove duplicate customer entries based on unique

identifiers.

 Standardize product categories (e.g., "electronics" vs.

"Electronics").

 Impute missing purchase amounts with the average

purchase value.

4. Marketing:

o Scenario: A marketing team is analyzing email campaign data. The

dataset has missing email addresses and inconsistent response
rates.

o Cleanup Process:

 Remove records with missing email addresses.

 Normalize response rate data by standardizing the format

and converting to a consistent scale.

Best Practices for Data Cleanup

1. Document the Cleanup Process:

o Keep detailed records of the steps taken to clean the data, including
justifications for handling specific issues. This ensures
reproducibility and transparency.

2. Use Automated Tools Where Possible:

o Leverage tools and libraries to automate repetitive tasks, such as

missing value imputation or data validation. This can save time and
reduce errors.

3. Iterate and Validate:

o Data cleaning is often an iterative process. Perform initial cleanup,

validate the results, and refine as needed.

4. Involve Domain Experts:

 o Collaborate with domain experts to understand the context of the
data and make informed decisions during the cleanup process.

5. Create Reusable Cleaning Pipelines:

o Develop reusable scripts or pipelines for data cleaning, especially

when dealing with frequently updated datasets. This ensures
consistency and efficiency.

Data Cleanup Basics

Data cleanup is a foundational component of the data wrangling process. It ensures that
the data used for analysis is accurate, consistent, and reliable.

Removing Missing Values

Missing values can severely impact data analysis and decision-making. It is crucial to
identify and address them appropriately.

 Identification:

o Use functions like isnull() in Python or [Link]() in R to detect missing

values in datasets.

o Generate summary statistics to determine the proportion of missing

values per column.

 Strategies for Handling Missing Values:

o Deletion:

 Row Deletion: Remove entire rows with missing values if they

constitute a small portion of the dataset.

 Column Deletion: Remove columns with a high percentage of

missing values, particularly if they are not essential to the analysis.

o Imputation:

 Mean/Median/Mode Imputation: Fill in missing values using the

mean, median, or mode of the non-missing values in the column.
This is a simple yet effective method, particularly for numerical
data.

 Predictive Imputation: Use machine learning algorithms (e.g.,

regression models) to predict and fill in missing values based on
other features.
 o Forward/Backward Fill:

 This technique is commonly used in time-series data, where the

previous or subsequent value is used to fill in gaps (e.g.,
fillna(method='ffill') in Python).

Handling Outliers
Outliers are data points that deviate significantly from other
observations, and they can distort analysis and statistical measures.

 Identification:

o Visual Methods: Use box plots or scatter plots to visually detect outliers.

o Statistical Methods: Calculate Z-scores or apply the Interquartile Range

(IQR) method:

 Z-score method: A data point is considered an outlier if its Z-score

is greater than 3 or less than -3.

 IQR method: Identify outliers as values lying beyond

Q1−1.5×IQRQ1 - 1.5 \times IQRQ1−1.5×IQR or Q3+1.5×IQRQ3 +
1.5 \times IQRQ3+1.5×IQR.

 Strategies for Handling Outliers:

o Capping: Limit extreme values to a defined maximum or minimum

threshold to reduce their impact on analysis.

o Transformation: Apply transformations (e.g., log, square root) to reduce

the influence of outliers.

o Removal: Exclude outliers if they are determined to be errors or if they

do not provide relevant information for analysis.

Dealing with Duplicates

Duplicate entries can lead to inflated counts and distorted analysis,
making it vital to identify and remove them.

 Identification:

o Use functions like duplicated() in Python or unique() in R to find duplicate

records in a dataset.

o Analyze key columns to determine duplicates, especially in unique

identifiers like IDs or emails.
  Removal:

o Exact Duplicate Removal: Use methods to drop exact duplicates while

retaining one instance of the record.

o Near-Duplicate Handling: Define criteria for near-duplicates (e.g., slight

variations in text) and consolidate them into a single entry.

Correcting Data Types

Ensuring that data types are correctly assigned is crucial for accurate analysis and
modeling.

 Identification:

o Use functions to check the data types of each column (e.g., dtypes in
Pandas).

 Common Issues:

o Numeric vs. Categorical: Numerical data may be stored as strings,

leading to errors in calculations.

o Date Formats: Date values may be misformatted, requiring conversion to

a proper date type.

 Conversions:

o Use functions like astype() in Python or [Link]() in R to convert data

types as needed.

o Apply functions like pd.to_datetime() in Pandas for converting strings to

datetime objects.

1.3.5 Data Standardization and Normalization

Standardization and normalization are techniques used to scale and

prepare data for analysis.

 Normalization:

o Definition: Adjusting values to a common scale, typically [0, 1].

o Min-Max Scaling: Xnorm=X−XminXmax−XminX_{\text{norm}} = \frac{X -

X_{\text{min}}}{X_{\text{max}} -
X_{\text{min}}}Xnorm​ =Xmax​ −Xmin​ X−Xmin​ ​

o Application: Useful when you want to compare data points from different
ranges (e.g., income vs. age).
  Standardization:

o Definition: Transforming data to have a mean of 0 and a standard

deviation of 1.

o Z-Score Standardization: Z=X−μσZ = \frac{X - \mu}{\sigma}Z=σX−μ​

o Application: Effective for normally distributed data, allowing

comparisons across different scales.

Best Practices for Data Cleanup

1. Document the Cleaning Process: Keep detailed records of cleaning steps to
ensure reproducibility and understanding of decisions made.

2. Iterative Approach: Data cleaning is often an iterative process. Perform initial

cleaning, validate, and refine.

3. Collaboration with Domain Experts: Engage with subject matter experts to

ensure that data cleaning aligns with business requirements and contextual
understanding.

4. Use Automated Tools: Leverage libraries and tools to automate repetitive tasks,
reducing manual errors and saving time.

5. Visualize Data After Cleanup: Use visualizations to validate the effectiveness of

data cleaning efforts and ensure that anomalies have been addressed.

Practical Examples
1. Example of Removing Missing Values:

o Dataset: Customer information with missing phone numbers.

o Process: Use mean imputation to fill missing phone number values based
on customer demographics.

2. Example of Handling Outliers:

o Dataset: Sales data with a few extremely high transaction values.

o Process: Identify outliers using box plots, cap extreme values at the 95th
percentile.

3. Example of Removing Duplicates:

o Dataset: Email campaign results with duplicate entries for some

customers.
 o Process: Identify duplicates based on email addresses and retain unique
entries.

4. Example of Correcting Data Types:

o Dataset: Product catalog with price stored as strings.

o Process: Convert the price column to numeric data types for proper
analysis.

5. Example of Normalizing Data:

o Dataset: User activity data with varying scales for different metrics.

o Process: Apply Min-Max scaling to normalize user engagement scores.

Data cleanup is a critical step in the data wrangling process, directly impacting the
quality and reliability of subsequent analysis. By systematically addressing missing
values, outliers, duplicates, and data types, analysts can ensure that their datasets are
primed for accurate insights. Utilizing best practices and appropriate tools enhances the
efficiency and effectiveness of the data cleanup process, ultimately leading to better
decision-making.

Normalizing and Standardizing Data

Normalization and standardization are vital techniques in data preprocessing that
prepare datasets for analysis and machine learning. These processes help ensure that
the data is on a consistent scale, which is crucial for many algorithms that are sensitive
to the scale of input features.

Definitions
 Normalization:

o The process of scaling individual data points to a common range, usually

[0, 1].

o It is particularly useful for datasets where features have different units or

vastly different scales.

 Standardization:

o The process of transforming data to have a mean of 0 and a standard

deviation of 1.

o This technique is effective when the data follows a Gaussian (normal)

distribution.
Why Normalize or Standardize Data?
1. Improved Model Performance:

o Many machine learning algorithms, such as k-means clustering and

gradient descent-based methods, are sensitive to the scale of the input
data. Normalizing or standardizing helps improve their performance and
convergence speed.

2. Ensuring Equal Weight:

o When features are on different scales, some may disproportionately

influence the model’s predictions. Normalization and standardization
help to mitigate this issue.

3. Facilitating Comparisons:

o Normalized or standardized data allows for more straightforward

comparisons between different features or datasets.

4. Handling Outliers:

o Standardization is less affected by outliers than normalization, making it a

preferred choice when outliers are present.

Methods of Normalization
1. Min-Max Scaling:

o This technique rescales the feature to a fixed range, typically [0, 1].

o Formula: Xnorm=X−XminXmax−XminX_{\text{norm}} = \frac{X -

X_{\text{min}}}{X_{\text{max}} -
X_{\text{min}}}Xnorm​ =Xmax​ −Xmin​ X−Xmin​ ​

o Advantages:

 Preserves the relationships between data points.

 Useful for algorithms requiring bounded input (e.g., neural

networks).

o Disadvantages:

 Sensitive to outliers, as they can skew the min and max values.

2. Robust Scaling:

o This technique scales features using statistics that are robust to outliers,
such as the median and interquartile range (IQR).
 o Formula: Xrobust=X−medianIQRX_{\text{robust}} = \frac{X -
\text{median}}{IQR}Xrobust​ =IQRX−median​

o Advantages:

 Less affected by outliers compared to min-max scaling.

o Disadvantages:

 Does not necessarily bound the data.

Methods of Standardization
1. Z-score Standardization:

o This method transforms the data such that it has a mean of 0 and a
standard deviation of 1.

o Formula: Z=X−μσZ = \frac{X - \mu}{\sigma}Z=σX−μ​ Where μ\muμ is

the mean and σ\sigmaσ is the standard deviation.

o Advantages:

 Useful for algorithms that assume normally distributed data (e.g.,

linear regression).

 Reduces the effect of outliers.

o Disadvantages:

 Not bounded; standardized data can take on any value.

2. Robust Z-score:

o A variant that uses the median and median absolute deviation instead of
mean and standard deviation to reduce the influence of outliers.

o Formula: Zrobust=X−medianMADZ_{\text{robust}} = \frac{X -

\text{median}}{\text{MAD}}Zrobust​ =MADX−median​ Where MAD is
the median absolute deviation.

o Advantages:

 More robust to outliers compared to standard Z-score.

When to Use Normalization vs. Standardization

 Use Normalization:
 o When the dataset has features on different scales and requires a bounded
range.

o In algorithms that rely on distance metrics (e.g., k-nearest neighbors) or

neural networks.

o When the data contains outliers that you wish to limit in influence.

 Use Standardization:

o When the data follows a Gaussian distribution or you want to achieve a

zero mean and unit variance.

o In algorithms that assume normality (e.g., linear regression, logistic

regression).

o When outliers are present, and you want to mitigate their effect while
retaining them in the dataset.

Implementation Examples:

[Link] in Python (Pandas):

import pandas as pd

# Sample data
data = {'feature1': [1, 2, 3, 4, 5], 'feature2': [10, 20, 30, 40,
50]}
df = [Link](data)

# Min-Max Normalization
df_normalized = (df - [Link]()) / ([Link]() - [Link]())

2. Standardization in Python (Pandas):

import pandas as pd

# Sample data
 data = {'feature1': [1, 2, 3, 4, 5], 'feature2': [10, 20, 30, 40, 50]}

df = [Link](data)

# Z-score Standardization

df_standardized = (df - [Link]()) / [Link]()

[Link] Scikit-learn for Scaling:

from [Link] import MinMaxScaler, StandardScaler

# Sample data

data = [[1, 10], [2, 20], [3, 30], [4, 40], [5, 50]]

# Normalization

min_max_scaler = MinMaxScaler()

normalized_data = min_max_scaler.fit_transform(data)

# Standardization

standard_scaler = StandardScaler()

standardized_data = standard_scaler.fit_transform(data)

Best Practices for Normalization and Standardization

1. Know Your Data: Understand the distribution of your data and the presence of
outliers before deciding on normalization or standardization.
 2. Apply Consistently: Apply the same normalization or standardization method across
all training and test datasets to avoid data leakage and inconsistencies.

3. Document Your Process: Maintain clear documentation of the scaling method used
and the rationale behind choosing a particular technique.

4. Visualize Results: Use histograms, box plots, and scatter plots to visualize the effects
of normalization or standardization on the data.

5. Test Different Methods: Experiment with both normalization and standardization in

modeling to see which yields better performance for your specific use case.

Normalizing and standardizing data are critical steps in the data wrangling process that
enhance the effectiveness of data analysis and machine learning models. By ensuring that
data is on a consistent scale, analysts can improve model performance and ensure accurate
comparisons across features. Understanding when and how to apply these techniques is
essential for effective data preparation.

Data Transformation Techniques

Data transformation involves converting data from one format or structure to another. This
is a vital process in data wrangling, enhancing the quality and usability of data for analysis,
visualization, and machine learning. This module discusses various data transformation
techniques, their significance, and applications.

Importance of Data Transformation

Data transformation plays a critical role in the data wrangling process by:

1. Improving Data Quality: It helps rectify inconsistencies, errors, and discrepancies

within datasets, leading to more reliable analysis.

2. Facilitating Analysis: Transforming data into a suitable format makes it easier to

analyze and interpret, ensuring that the dataset aligns with analytical techniques.

3. Enabling Data Integration: Data from multiple sources often have different formats.
Transformation standardizes these formats, allowing for seamless integration.

4. Enhancing Model Performance: Machine learning models often perform better with
well-structured data, as certain algorithms require features to be on similar scales or
distributions.

Common Data Transformation Techniques:

1. Data Aggregation
 Definition: Data aggregation involves summarizing a dataset by grouping data
points based on one or more keys and applying aggregate functions (such as sum,
mean, count, etc.).

 Purpose:

o Reduces the size of large datasets, making them easier to analyze.

o Retains important summary statistics that provide insights into trends

and patterns.

 Applications:

o Monthly or yearly sales reports from daily sales data.

o Summarizing survey responses by demographic categories.

2. Data Encoding

 Definition: Data encoding converts categorical variables into a numerical format,

allowing algorithms to interpret categorical data effectively.

 Types:

o Label Encoding: Assigns a unique integer to each category.

 Purpose: Simple and efficient for ordinal categories (where order

matters).

 Example: "Low" = 1, "Medium" = 2, "High" = 3.

o One-Hot Encoding: Creates binary columns for each category, indicating

the presence (1) or absence (0) of each category.

 Purpose: Useful for nominal categories where no inherent order

exists.

 Example: For the color feature (red, green, blue), it generates three
binary columns.

3. Data Scaling

 Definition: Data scaling adjusts the range of feature values to ensure that they
contribute equally to distance calculations in algorithms, particularly for those
sensitive to feature magnitudes.

 Techniques:

o Min-Max Scaling: Rescales features to a fixed range, typically [0, 1].

  Formula: Xscaled=X−XminXmax−XminX_{\text{scaled}} = \frac{X -
X_{\text{min}}}{X_{\text{max}} -
X_{\text{min}}}Xscaled​ =Xmax​ −Xmin​ X−Xmin​ ​

 Use Case: Useful for algorithms that require bounded input, such
as neural networks.

o Z-score Standardization (Standardization): Transforms data to have a

mean of 0 and a standard deviation of 1.

 Formula: Z=X−μσZ = \frac{X - \mu}{\sigma}Z=σX−μ​ Where

μ\muμ is the mean and σ\sigmaσ is the standard deviation.

 Use Case: Suitable for algorithms that assume normally distributed

data, such as linear regression.

4. Data Discretization

 Definition: Data discretization involves converting continuous variables into

discrete categories or bins.

 Purpose:

o Simplifies the model by reducing the number of unique values.

o Enhances interpretability by converting continuous ranges into

categorical groups.

 Applications:

o Grouping ages into categories like "child," "teen," "adult," and "senior."

o Creating income brackets for analysis.

5. Feature Engineering

 Definition: Feature engineering is the process of creating new features from

existing ones to improve model performance.

 Techniques:

o Polynomial Features: Involves creating new features by raising existing

features to a power or creating interaction terms (e.g., X1×X2X_1 \times
X_2X1​ ×X2​ ).

o Date/Time Features: Extracting components such as year, month, day,

day of the week, etc., from datetime variables.

o Text Features: Converting text data into numerical representations, such

as term frequency or using embeddings.
 6. Mathematical Transformations

 Definition: Mathematical transformations involve applying functions to modify

the distribution of data.

 Common Functions:

o Log Transformation: Reduces the skewness of right-skewed data, making

it more normally distributed.

 Use Case: Effective for datasets with exponential growth patterns.

o Square Root Transformation: Can also help reduce skewness, particularly

for count data.

o Box-Cox Transformation: A family of power transformations that can

stabilize variance and make data more normally distributed.

Data transformation techniques are fundamental to data wrangling, enabling analysts to

prepare datasets for effective analysis and modeling. Understanding when and how to
apply these techniques enhances data quality and improves the performance of
analytical models. Each technique serves specific purposes and is chosen based on the
characteristics of the dataset and the requirements of the analysis.