DATA SCIENCE
UNIT 3 STUDY NOTES
Data Preprocessing
CO3: Apply data preprocessing techniques to clean, transform, and prepare raw data for analysis.
3.1 Need for Data Preprocessing
Raw data collected from surveys, sensors, web scraping, or existing databases is almost never analysis-ready. It
typically contains errors, gaps, inconsistencies, and mismatched formats. Data preprocessing is the set of techniques
used to convert this raw data into a clean, consistent, structured form suitable for exploration and modelling.
Why Raw Data Is Rarely Usable As-Is
Problem Example Consequence if Ignored
Missing values Some rows have no 'age' recorded Statistics like mean/average get skewed or
calculations fail
Duplicate records Same customer entered twice Inflated counts, biased averages
Inconsistent formats Dates as '01-02-2024' and '02/01/24' Sorting/filtering by date breaks
Noisy / incorrect data Age recorded as 250 Outliers distort analysis and models
Data from multiple Sales data in two systems with different Cannot be combined directly
sources column names
The Common Saying: 'Garbage In, Garbage Out'
No matter how sophisticated the analysis or model, its output is only as reliable as the data fed into it. In real industry
pipelines, data preprocessing typically consumes 60-80% of a data scientist's total project time — making it one of
the most important skills in this course.
Learning Outcome for 3.1
You should be able to identify at least four categories of data quality issues in a raw dataset and explain the
specific downstream consequence each one causes if left unaddressed.
3.2 Data Cleaning
Data cleaning is the broad process of detecting and correcting (or removing) inaccurate, corrupted, incorrectly
formatted, or irrelevant parts of a dataset. Handling missing values and duplicates (3.3, 3.4) are specific cleaning
tasks; this section covers the wider picture.
Typical Cleaning Tasks
• Correcting inconsistent categorical labels: 'Male', 'M', 'male' should all be standardised to one form.
• Fixing structural errors: typos in category names, trailing whitespace, mismatched capitalisation.
• Validating ranges: flagging values outside a plausible range (e.g. negative age, marks above 100).
• Removing irrelevant columns/rows that don't contribute to the analysis goal (e.g. an internal tracking ID with no
analytical value).
import pandas as pd
df = pd.read_csv("[Link]")
# Standardising inconsistent category labels
df["gender"] = df["gender"].[Link]().[Link]()
df["gender"] = df["gender"].replace({"m": "male", "f": "female"})
# Flagging implausible values
invalid_rows = df[(df["age"] < 0) | (df["age"] > 100)]
print(invalid_rows)
# Dropping a column that adds no analytical value
df = [Link](columns=["internal_tracking_id"])
Learning Outcome for 3.2
You should be able to detect inconsistent labels, implausible values, and irrelevant columns in a real dataset,
and write Pandas code to correct or remove them.
3.3 Handling Missing Values
Why Values Go Missing
• Missing Completely at Random (MCAR): no pattern — e.g. a sensor glitch.
• Missing at Random (MAR): missingness relates to another observed variable — e.g. income is more often
unanswered by a specific age group.
• Missing Not at Random (MNAR): missingness relates to the value itself — e.g. people with very high income
choose not to disclose it.
Understanding why data is missing matters — it determines whether removing or imputing rows will bias the
analysis.
Detecting Missing Values
[Link]().sum() # count of missing values per column
[Link]().mean() * 100 # percentage missing per column
Strategies for Handling Missing Values
Strategy When to Use Pandas Example
Drop the row Very few missing rows (e.g. <2%) and [Link]()
dropping won't bias results
Drop the column A column is mostly missing (e.g. >60%) and [Link](columns=['col'])
not critical
Mean/Median Numeric column, missing values are MCAR df['age'].fillna(df['age'].median())
imputation
Strategy When to Use Pandas Example
Mode imputation Categorical column df['branch'].fillna(df['branch'].mode()[0])
Forward/backward fill Time-series data where nearby values are [Link](method='ffill')
similar
Common Mistake
Blindly using [Link]() on the entire DataFrame can silently delete a large fraction of your dataset if even
one column has scattered missing values. Always check isnull().sum() first and decide column-by-column.
Learning Outcome for 3.3
You should be able to detect and quantify missing values, reason about why they may be missing, and
choose/apply an appropriate imputation or removal strategy with justification.
3.4 Removing Duplicate Records
Duplicate rows — identical or near-identical records — commonly arise from repeated form submissions, merging
data from multiple sources, or logging errors. Left unaddressed, they inflate counts and bias summary statistics.
[Link]().sum() # count of fully duplicated rows
df[[Link]()] # view the duplicate rows
df = df.drop_duplicates() # remove exact duplicate rows
df = df.drop_duplicates(subset=["email"]) # remove duplicates based on one key
column
df = df.drop_duplicates(subset=["email"], keep="last") # keep the most recent entry
Exact vs Near Duplicates
• Exact duplicates: every column matches — straightforward to detect with duplicated().
• Near duplicates: the same real-world entity recorded slightly differently, e.g. 'Riya Sharma' vs 'riya sharma ' —
these require cleaning (3.2) first, such as trimming whitespace and standardising case, before duplicate
detection will catch them.
Learning Outcome for 3.4
You should be able to detect exact duplicate rows using a full-row or subset-of-columns check, and explain
why near-duplicates require prior cleaning before they can be caught.
3.5 Data Transformation
Data transformation converts data into a form better suited for analysis or modelling — changing scale, encoding
categories numerically, or reshaping values.
Normalization and Standardization
Technique Formula (concept) Result Range
Min-Max Normalization (x - min) / (max - min) Scales values to [0, 1]
Technique Formula (concept) Result Range
Standardization (Z-score) (x - mean) / std Centred at 0, unit variance
# Min-Max normalization
df["marks_norm"] = (df["marks"] - df["marks"].min()) / (df["marks"].max() -
df["marks"].min())
# Standardization
df["marks_std"] = (df["marks"] - df["marks"].mean()) / df["marks"].std()
Encoding Categorical Data
Most numeric/statistical operations and ML algorithms cannot work directly with text categories — they need to be
encoded as numbers.
# Label encoding — assigns each category an integer (use only for ordinal categories)
df["grade_encoded"] = df["grade"].map({"C": 0, "B": 1, "A": 2, "A+": 3})
# One-hot encoding — creates a separate 0/1 column per category (use for nominal
categories)
df = pd.get_dummies(df, columns=["branch"])
Why This Choice Matters
Label-encoding a non-ordinal category (e.g. branch: MCA=0, MBA=1, BCA=2) wrongly implies an order or
numeric relationship between categories. Use one-hot encoding for categories with no natural order.
Learning Outcome for 3.5
You should be able to normalise/standardise a numeric column, and choose correctly between label encoding
and one-hot encoding for a given categorical column.
3.6 Data Integration
Data integration combines data from multiple sources — different files, databases, or systems — into a single, unified
dataset for analysis. This is common when, for example, student academic records and placement records are stored
in separate systems.
Common Integration Operations
import pandas as pd
academics = pd.read_csv("[Link]") # columns: student_id, cgpa
placements = pd.read_csv("[Link]") # columns: student_id, company, package
# Merge (like a SQL JOIN) on a common key
merged = [Link](academics, placements, on="student_id", how="inner")
# Types of joins, same concept as SQL:
# how="inner" -> only students present in both files
# how="left" -> all academic records, placement info where available
# how="outer" -> all records from both, missing values filled with NaN
# Concatenating datasets with the same columns (e.g. two years of the same report)
year1 = pd.read_csv("2023_data.csv")
year2 = pd.read_csv("2024_data.csv")
combined = [Link]([year1, year2], ignore_index=True)
Challenges in Integration
• Schema mismatch: the same attribute may be named differently across sources (e.g. 'stud_id' vs 'student_id') —
needs renaming before merging.
• Semantic mismatch: the same column name can mean different things across sources (e.g. 'status' meaning
attendance in one file, admission status in another).
• Redundancy: overlapping data recorded in multiple systems, needing de-duplication after merging (see 3.4).
Learning Outcome for 3.6
You should be able to merge two related datasets using an appropriate join type, and identify
schema/semantic mismatches that must be resolved before integration.
3.7 Introduction to Data Wrangling
Data wrangling (also called data munging) is the broader, often iterative process of taking raw, messy data and
reshaping/reformatting/restructuring it into a form ready for analysis. It overlaps with cleaning, transformation, and
integration, but emphasises restructuring the data's layout, not just fixing individual values.
3.7.1 Data Formatting
Ensuring every column has a consistent, correct data type and representation.
# Converting a text date column into an actual datetime type
df["date"] = pd.to_datetime(df["date"], format="%d-%m-%Y")
# Converting a numeric column stored as text (e.g. due to a stray currency symbol)
df["price"] = df["price"].[Link]("₹", "").[Link](",", "").astype(float)
# Ensuring consistent text case across a column
df["city"] = df["city"].[Link]()
3.7.2 Data Manipulation
Restructuring the shape or organisation of the dataset itself — not just individual values — to suit the analysis being
performed.
# Creating a new derived column
df["result"] = df["marks"].apply(lambda m: "Pass" if m >= 40 else "Fail")
# Grouping and aggregating
branch_avg = [Link]("branch")["marks"].mean()
# Pivoting - reshaping rows into columns (wide format)
pivot = df.pivot_table(values="marks", index="student_id", columns="subject")
# Melting - reshaping columns into rows (long format), the inverse of pivoting
long_df = [Link](df, id_vars=["student_id"], var_name="subject", value_name="marks")
Formatting vs Manipulation — Quick Distinction
Aspect Data Formatting Data Manipulation
Focus Correctness/consistency of individual values Overall shape/structure of the dataset
Typical operation Type conversion, text cleanup Grouping, pivoting, deriving new columns
Example '01-02-2024' -> datetime object Wide table -> long table via melt()
Learning Outcome for 3.7
You should be able to distinguish data formatting from data manipulation, convert a column's type/format
correctly, and reshape a dataset using groupby, pivot_table, or melt as appropriate.
Unit 3 — Quick Revision Map
Sub-topic Core Idea Key Tool/Concept
3.1 Why raw data can't be used directly Data quality issues, 'garbage in, garbage
out'
3.2 Fixing inconsistent/incorrect values Label standardisation, range validation
3.3 Dealing with gaps in data isnull(), fillna(), dropna(), imputation
3.4 Removing repeated records duplicated(), drop_duplicates()
3.5 Reshaping value scale/representation Normalization, standardization, encoding
3.6 Combining multiple data sources merge(), concat(), join types
3.7 Restructuring data for analysis Type formatting, groupby, pivot, melt