UNIT 2 - Data Collection and

Preprocessing

BCO606A - INTRODUCTION TO DATA SCIENCE

NOTES
WHAT IS A DATABASE?
•A database is a collection of data that is organized so that its contents can easily be
accessed managed and updated
•To create a database, we need to have a Database Management Software (DBMS)

Problems that databases are made to address

1. Unnecessary duplication of data
2. Boredom and time wastage while searching for a record
3. Misleading reports due to poor data entry and organization
4. Poor update of records
Functions of DBMS

1. Allow users to delete or add records

2. Update and modify existing records
3. Organize data for easy access, retrieval and manipulation of records
4. Ensuring the security of data
5. Keep statistics of data in the database.
Database Models
Databases can be classified according the method used to organize data namely:
1. Flat file
2. Hierarchical
3. Network
4. Relational
5. Object Oriented Models
A data model is a collection of concepts and rules for the description of the structure of the database. Structure
of the database means the data types, the constraints and the relationships for the description or storage of
data respectively.
Database Models- Hierarchical models
•Data items are arranged in tree form. To access level two data items, you have to
first access level 1 data item called the root component.
What is API?
API stands for Application Programming Interface. It allows two applications to
communicate each other. It is the contract between client and server. Client sends a
request through the API and after performing the action the API will send back a
response to the client.
● Analogy: A Restaurant Menu
○ You (the client) are given a menu (the API documentation) with a list of items
(endpoints) you can order, along with a description of each (parameters).
○ You tell the waiter (the API request) your order.
○ The kitchen (the server) prepares the food.
○ The waiter brings you the food you ordered (the API response).
○ You don't need to know how the kitchen works, and the kitchen doesn't need to know
who you are. The menu and waiter provide a standard interface for this interaction.
● Technical Definition: A set of rules, protocols, and tools that allows different software
applications to communicate with each other. It defines the kinds of calls or requests that
can be made, how to make them, the data formats to use, and the conventions to follow.
Types of API
● REST (Representational State Transfer):
○ Architectural style, not a strict protocol.
○ Uses HTTP methods explicitly.
○ Stateless: Each request must contain all the information needed to process it.
○ Resources are represented as URIs.
○ Returns data in lightweight formats like JSON or XML.
○ Most common type of API on the web.
● SOAP (Simple Object Access Protocol):
○ A protocol with strict standards.
○ Uses XML for message format.
○ Built-in error handling and security features (WS-Security).
○ Generally considered more heavyweight and complex than REST.
● GraphQL:
○ A query language for APIs, developed by Facebook.
○ Allows the client to request exactly the data it needs, nothing more and nothing less.
○ Solves problems of over-fetching (getting too much data) and under-fetching (needing multiple requests).
○ Uses a single endpoint, typically .
What is Web Scraping?

● The automated process of extracting large amounts of data from websites.

● A scraper programmatically fetches web pages (like a browser does) and then parses the HTML code to extract
the desired information, converting unstructured website data into a structured format (like a CSV, JSON, or
database).
● Instead of a human manually copying and pasting names and prices from a product page into an Excel sheet, a
script does it automatically for thousands of pages in minutes.

Why Scrape? (Use Cases)

● Price Monitoring & Competition Analysis: Track prices of products across e-commerce sites.
● Market Research: Gather data on trends, customer reviews, and sentiment.
● Lead Generation: Collect public contact information from business directories.
● News & Sentiment Analysis: Aggregate articles and social media posts for financial or research purposes.
● Academic Research: Collect data from various public sources for analysis.
● Training Machine Learning Models: Create large datasets for AI models.
Key Tools & Libraries (Python Ecosystem)
Sensors and Logs in Data science

1. Data Generation: Sensors and logs create raw data.

2. Ingestion & Collection: Data is collected into a central system (e.g., cloud storage, a data lake like S3, a
streaming platform like Kafka).
3. Processing & Wrangling:
○ For Sensors: Cleaning, denoising, aggregating, and feature engineering on time-series data.
○ For Logs: Parsing, sessionizing, and transforming into a structured event-based format.
4. Storage: Processed data is stored in a query-optimized system (e.g., data warehouse like BigQuery,
Snowflake, or a time-series database like InfluxDB).
5. Analysis & Modeling: This is the core Data Science work:
○ Exploratory Data Analysis (EDA): Understanding distributions, patterns, and correlations.
○ Machine Learning: Building models for prediction, classification, or clustering.
○ Creating Dashboards & Reports: Visualizing key metrics for stakeholders
Handling Missing Values

1) Deletion Methods (Complete Case Analysis)

These are the simplest methods, but they rely on a strong assumption.

● Listwise Deletion: Discard any sample (row) that has any missing value.
○ Mathematical Implication: You are left with a complete dataset, but you assume the remaining data is a random
sample of the original population. This is only true if data is MCAR (Missing Completely At Random).
○ Risk: If the data is not MCAR, your remaining dataset D_complete becomes a biased sample, and any model
trained on it will produce biased estimates. The probability of a sample being kept is P(keep) = P(no
missingness in any feature), which can be very low in high-dimensional data, leading to significant
information loss.
● Pairwise Deletion: Used primarily in correlation/covariance matrix calculation. Uses all available pairs of observations.
○ The covariance between two features X_i and X_j is calculated using only the samples where both X_i and
X_j are present.
○ Problem: The covariance matrix calculated this way may not be positive semi-definite, which is a requirement
for many multivariate analyses. It also uses different sample sizes for different pairs, making the overall
structure inconsistent.
2) Single Imputation Methods

These methods fill in (impute) a single value for each missing data point.

● Mean/Median/Mode Imputation: Replace missing values with the mean μ (for continuous) or mode (for categorical) of the observed
values for that feature.
○ x_imputed = μ = (1/n_observed) * Σ x_observed
○ Drawbacks:
1. Underestimates Variance: The imputed value is a constant, reducing the natural spread of the data. Var(X_imputed) <
Var(X_true).
2. Distorts Covariance: The covariance between the imputed feature and any other feature is biased towards zero because
the imputed values are all at the center, dragging the correlation down. Cov(X_imputed, Y) ≈ (n_observed/n_total) *
Cov(X_observed, Y).
3. Creates False Certainty: The model treats the imputed value as if it were a real, measured value, which is not true.
● Regression Imputation (Conditional Imputation): A more sophisticated method. For a feature X_j with missing values:
○ Build a regression model X_j ~ f(X_1, X_2, ..., X_{j-1}, X_{j+1}, ..., X_p) using samples where X_j is not missing.
○ For a sample missing X_j, use the other features and the trained model to predict x_j_hat.
○ Use x_j_hat as the imputed value.
○ x_imputed = β_0 + β_1 * x_1 + ... + β_k * x_k (for linear regression)
○ Advantage: Preserves the relationship between variables better than mean imputation.
○ Drawbacks:
1. Still underestimates variance. The imputed values fall perfectly on the regression hyperplane, ignoring the error term ε of
the regression model. This implies no uncertainty around the imputed value.
2. The same false certainty problem exists.
3) Advanced Probabilistic Imputation: The EM Algorithm

The Expectation-Maximization (EM) algorithm provides a framework for finding maximum likelihood estimates when
data is incomplete. It's often used to estimate parameters (like mean μ and covariance matrix Σ) from incomplete data.

● Goal: Find the parameters θ (e.g., μ, Σ) that maximize the likelihood of the observed data.
● The Process (for a multivariate Gaussian assumption):
1. Initialization: Start with initial guesses for μ and Σ (e.g., using mean imputation).
2. Expectation (E-step): Given the current μ and Σ, calculate the conditional expectation of the missing data
given the observed data for each sample. This "fills in" the missing values not with a single number, but with
a distribution.
■ For a sample with missing components, the conditional distribution P(X_miss | X_obs, μ, Σ) is
itself a Gaussian. We compute its expectation E[X_miss | X_obs].
3. Maximization (M-step): Re-calculate the parameters μ and Σ using the "completed" data from the E-step
(often using the expected sufficient statistics).
■ μ_new = (1/N) * Σ E[X_i], Σ_new = (1/N) * Σ (E[X_i X_i^T] - μ_new μ_new^T)
4. Iterate: Repeat the E and M steps until the parameters μ and Σ converge.
● Advantage: Provides unbiased parameter estimates under the MAR assumption.
● Output: The final μ and Σ can be used to impute missing values (e.g., using the conditional mean from the final
E-step) or to perform further analysis.
4) Multiple Imputation (The Gold Standard)

Multiple Imputation (MI) is designed to fix the fundamental flaw of single imputation: the underestimation of uncertainty.

● Core Idea: Instead of imputing one value, impute M > 1 values (e.g., M=5). This creates M different complete datasets.
● The Process (often using MICE - Multiple Imputation by Chained Equations):
1. Create M Datasets: For each dataset m = 1 to M:
■ Fill missing values with a random draw (e.g., from mean + error). This is the "Proper Imputation" step.
■ For each variable with missing data, fit a model (e.g., regression) using the other variables and use it to
predict missing values, adding a random error term to the prediction to preserve variance.
■ Iterate this process over all variables multiple times (chained equations).
2. Analyze: Train your desired ML model on each of the M complete datasets. You now have M slightly different
models, with M different parameter estimates (e.g., M sets of coefficients).
3. Pool Results: Combine the results from the M models using Rubin's Rules.
■ The final parameter estimate Q (e.g., a coefficient) is the average of the M estimates:
Q̄ = (1/M) * Σ Q_m
■ The variance of Q is calculated as:
T = Ū + (1 + 1/M) * B
where:
■ Ū = (1/M) * Σ Var(Q_m) is the within-imputation variance (average uncertainty from each model).
■ B = (1/(M-1)) * Σ (Q_m - Q̄)^2 is the between-imputation variance (uncertainty due to the
missing data itself).
● Why it Works: MI correctly incorporates the uncertainty caused by the missing data. The final variance estimate T is larger
than a single imputation variance, accurately reflecting our less certain knowledge of the imputed values.
5) Model-Based Methods

Some algorithms have native, mathematically sound ways of handling missing data.

● Tree-Based Models (XGBoost, LightGBM):

○ They handle missing values during the split finding process. For a given feature, they learn a "default
direction" (left or right child node) for samples with missing values. This direction is chosen to
minimize the loss function (e.g., Gini impurity, MSE) just like any other split.
○ Advantage: No need for pre-imputation. The model uses the missingness itself as information.
● Probabilistic Models (e.g., Bayesian Networks):
○ The model defines a joint probability distribution P(X_1, X_2, ..., X_p). Missing data can be
handled by integrating it out. For inference on observed variables, we calculate the marginal
likelihood: P(X_obs) = ∫ P(X_obs, X_miss) dX_miss.
○ This is computationally expensive but mathematically pristine, as it accounts for all possible values
of the missing data, weighted by their probability.
Handling Outliers

Outliers are data points that deviate significantly from other observations. They can arise from measurement error, data entry error, or

genuine natural variation. The core challenge is determining whether an outlier is noise (to be removed) or signal (to be analyzed).

1. The Fundamental Step: Detection

You must find outliers before you can handle them. Methods are often univariate (per feature) or multivariate (considering feature

interactions).

A. Statistical & Graphical Methods

● Standard Deviation (Z-Score) Method:

○ Assumption: Data is ~Normally distributed.
○ Math: Calculate the Z-score for each data point x_i in a feature:
z_i = (x_i - μ) / σ
where μ is the mean and σ is the standard deviation.
○ Threshold: Typically, if |z_i| > 3, the point is considered an outlier. This threshold corresponds to ~99.7% of data lying
within μ ± 3σ under normality.
● Interquartile Range (IQR) Method:
○ Assumption: Non-parametric (does not assume a distribution).
1. Calculate Q1 (25th percentile) and Q3 (75th percentile).
2. IQR = Q3 - Q1.
3. Define the "fences":
■ Lower Fence: Q1 - 1.5 * IQR
■ Upper Fence: Q3 + 1.5 * IQR
○ Threshold: Any data point outside the fences (x_i < Lower Fence or x_i > Upper Fence) is considered a
mild outlier. Using 3 * IQR identifies extreme outliers.
Model-Based Methods

● DBSCAN (Density-Based Spatial Clustering):

○ Principle: Clusters are dense regions of points. Points in low-density regions are labeled as outliers.
○ Math: Requires two parameters: eps (neighborhood radius) and min_samples. A point is an outlier (noise) if
it has fewer than min_samples points within its eps radius.
● Isolation Forest:
○ Principle: Anomalies are "few and different," making them easier to isolate than normal points.
○ Math:
1. Build an ensemble of random decision trees.
2. For each tree, randomly select a feature and a split value between the min and max of that feature.
3. The number of splittings required to isolate a sample is its path length.
4. The anomaly score is calculated based on the average path length across all trees. A shorter path
length indicates a higher likelihood of being an outlier.
● One-Class SVM:
○ Principle: Learns a decision boundary that encompasses the "normal" data points. Points outside this
boundary are outliers.
○ Math: Maps data into a high-dimensional feature space and finds a maximal margin hyperplane that
separates the data from the origin.
Once detected, the choice of how to handle outliers has significant mathematical implications for your model.

A. Deletion (Trimming)

● Action: Remove the outlier samples from the dataset.

● When to Use: When you are certain the outliers are due to errors (e.g., sensor malfunction, data entry typo) and are not
representative of the population.
● Mathematical Impact:
○ Pros: Simplest method. Can significantly improve the performance of models sensitive to outliers (e.g., Linear
Regression, k-Means clustering).
○ Cons: Reduces sample size. If the outliers are not errors but genuine rare events, you are throwing away valuable
information and biasing your model towards the majority.

B. Imputation (Capping/Winsorizing)

● Action: Replace the outlier values with a less extreme value. This preserves the sample size.
● Methods:
○ Capping: Set a floor and a ceiling. Any value below the lower_bound is set to the lower_bound; any value above the
upper_bound is set to the upper_bound.
■ Example: Using IQR, set upper_bound = Q3 + 1.5*IQR, lower_bound = Q1 - 1.5*IQR.
 ○ Transformation: Apply a mathematical function that reduces the impact of extreme values.
■ Log Transformation: x_new = log(x). Effective for right-skewed data.
■ Square Root Transformation: x_new = sqrt(x).
■ Box-Cox Transformation: A more generalized, parameterized power transformation to stabilize variance and make data more normal.
● Mathematical Impact:
○ Pros: Preserves all samples. Reduces the variance and skewness induced by outliers, making the data more conformative to algorithms that
assume normality.
○ Cons: Distorts the original distribution and alters the relationships between variables. The model is no longer learning from the true values.

C. Using Robust Algorithms

The most elegant solution is often to choose an algorithm that is inherently resistant to outliers.

● Tree-Based Models (Random Forest, Gradient Boosting):

○ Why they are robust: They make splits based on sample proportions (e.g., Gini Impurity) rather than means/variances. A single extreme
value won't affect the split logic much.
● Model-Based Methods:
○ Use models with robust loss functions.
○ Example: Switch from Linear Regression to Huber Regressor.
■ Linear Regression uses Mean Squared Error (MSE) loss: L(y, y_hat) = (y - y_hat)^2. The squared term heavily penalizes
outliers, pulling the regression line towards them.
■ Huber Regressor uses Huber loss, which is quadratic for small errors but linear for large errors:
L_δ(y, y_hat) = { ½(y - y_hat)² for |y - y_hat| ≤ δ, { δ|y - y_hat| - ½δ² otherwise.
This means outliers are penalized less severely than in MSE, making the model more robust.
● Mathematical Impact:
○ Pros: No need to manually remove or alter data, preserving its integrity. The model itself is designed for the real world, which is often
messy.
○ Cons: You are limited to a specific class of algorithms.
Data Cleaning
Data cleaning (or cleansing) is the process of detecting, correcting, or removing corrupt, inaccurate, or
irrelevant parts of a dataset. It's the most time-consuming and critical step, often summarized by the
rule: "Garbage In, Garbage Out."

The Data Cleaning Checklist:

A typical data cleaning workflow addresses the following issues:`

1. Handling Duplicates
2. Handling Missing Values
3. Handling Outliers
4. Fixing Structural Errors
5. Standardizing/Normalizing Data
6. Validating Data
Handling Missing Values

Mean/Median Imputation

● Mean: x_imputed = μ = (1/n) * Σ_{i=1}^n x_i

Replaces missing values with the arithmetic average.
● Median: The value m such that P(X ≤ m) ≥ 0.5 and P(X ≥ m) ≥ 0.5.
The middle value, robust to outliers.

Regression Imputation

For a feature X_j with missing values, we model it as a function of other features:
X_j = β_0 + β_1 X_1 + ... + β_k X_k + ε

where ε ~ N(0, σ²). The imputed value for a missing X_j is its conditional expectation:
E[X_j | X_1, ..., X_k] = β_0 + β_1 x_1 + ... + β_k x_k

This preserves relationships but underestimates variance by ignoring the error term ε.
Probabilistic Imputation: The EM Algorithm

The Expectation-Maximization algorithm finds maximum likelihood estimates for parameters θ (e.g., μ, Σ) with missing data.

1. E-Step (Expectation): Compute the expected value of the log-likelihood function, given the current parameter estimate θ^{(t)} and the
observed data X_obs:
Q(θ | θ^{(t)}) = E_{X_miss | X_obs, θ^{(t)}} [ log L(θ; X_obs, X_miss) ]
This involves calculating the conditional distribution of the missing data given the observed data.
2. M-Step (Maximization): Find the new parameter estimate that maximizes the Q-function:
θ^{(t+1)} = argmax_θ Q(θ | θ^{(t)})
3. Iterate until convergence. The final θ is used for imputation.

Handling Outliers

IQR Method

● Q1 = F^{-1}(0.25), Q3 = F^{-1}(0.75), where F is the cumulative distribution function.

● IQR = Q3 - Q1
● Lower Bound: L = Q1 - k * IQR
● Upper Bound: U = Q3 + k * IQR (typically k=1.5 for outliers, k=3 for extremes)
● Any x_i such that x_i < L or x_i > U is considered an outlier.
Z-Score Method

● z_i = (x_i - μ) / σ
● A point is an outlier if |z_i| > c, where c is a threshold (often 3).

Robust Z-Score (Using Median and MAD)

● Median Absolute Deviation (MAD): MAD = median(|x_i - median(X)|)

● Robust Z-Score: z_i_{robust} = (x_i - median(X)) / (k * MAD)
where k ≈ 1.4826 (a constant scaling factor to make MAD consistent with the standard deviation for normal distributions).

Feature Engineering:

Binning/Discretization

Transforms a continuous variable X into a categorical variable with K bins.

● Equal-Width: Bin boundaries are at min(X) + i * (max(X) - min(X)) / K for i = 0, ..., K.

● Equal-Frequency: Bin boundaries are chosen so each bin contains approximately n / K observations.
Polynomial Features

Creates new features by raising existing features to a power and multiplying them together.

● For two features (a, b), polynomial features of degree 2 are: (1, a, b, a², ab, b²).
● This allows linear models to fit nonlinear relationships. The model becomes:
y = β_0 + β_1 a + β_2 b + β_3 a² + β_4 ab + β_5 b²

Interaction Features

Explicitly model the interaction between two features by creating a new feature x_new = x_i * x_j.

Feature Encoding:
Target Encoding (Smoothing)

Encodes a category by the average value of the target variable for that category, but regularized to avoid overfitting.

encoded_value = λ * mean(target | category) + (1 - λ) * mean(target)

where λ is a smoothing parameter that depends on the category's frequency. Common form:

encoded_value = \frac{n * \bar{y}_{category} + m * \bar{y}_{global}}{n + m}

where:

● n = number of samples in the category

● m = a smoothing hyperparameter (controls the strength of the prior \bar{y}_{global})

Feature Scaling
Data Transformation
Data transformation is the process of applying a mathematical function to each data point. This is often done to change
the distribution of a variable to make it more suitable for machine learning algorithms.

Common Transformation Functions:

● Log Transformation: x_new = log(x) or x_new = log(x + 1) (to handle zeros)

○ Use Case: Right-skewed data (e.g., income, revenue).
○ Math: Compresses large values and expands smaller ones. Helps stabilize variance and make data more
normally distributed.
● Square Root Transformation: x_new = sqrt(x)
○ Use Case: A weaker alternative to the log transform for right-skewed data and count data.
● Box-Cox Transformation: A parameterized transformation that finds the best lambda ( λ) to make the data as normal
as possible.
○ Formula: x_new = { (x^λ - 1)/λ, if λ ≠ 0; log(x), if λ = 0 }
○ Math: It maximizes the log-likelihood function to find the optimal λ. It assumes all data is positive.
● Yeo-Johnson Transformation: An extension of Box-Cox that works for both positive and negative data.
○ Formula: More complex, with different cases based on the value of x and λ.
Encoding Techniques
Machine learning algorithms require all input and output to be numerical. Encoding is the process of converting categorical
data into this numerical format.

The choice of encoding is critical and depends on the cardinality (number of unique values) and the nature of the categories
(ordinal vs. nominal).

A. Ordinal Encoding

● Use Case: Ordinal data - categories with a natural, meaningful order or ranking.
● Examples: Education Level ('High School' < 'Bachelor' < 'Master' < 'PhD'), Satisfaction Rating ('Poor' <
'Fair' < 'Good' < 'Excellent'), Size ('S' < 'M' < 'L').
● How it works: Each unique category is assigned an integer based on its rank.
○ {'Poor': 0, 'Fair': 1, 'Good': 2, 'Excellent': 3}
● Mathematical Implication: The model interprets the distance between categories as meaningful (e.g., the difference
between 'Good' (2) and 'Excellent' (3) is the same as between 'Fair' (1) and 'Good' (2)). This is only true if
the underlying scale is interval-based.
B. Label Encoding

● Use Case: Nominal data where there is no ordinal relationship and the number of categories is very low
(e.g., binary classification, target variable for classification).
● Examples: Target variable y for a classifier: {'Cat', 'Dog', 'Bird'} -> {0, 1, 2}. Binary features:
{'Yes', 'No'} -> {1, 0}.
● How it works: Assigns an arbitrary integer to each category. {'Red': 0, 'Blue': 1, 'Green': 2}.
● Pitfall: For nominal input features, this is generally bad practice. A model might incorrectly assume an
order (Green (2) > Blue (1)) or calculate a meaningless distance (Red to Blue is 1, Blue to Green is 1).
This can severely degrade model performance.
● Key Difference from Ordinal: The assignment is arbitrary, not based on a predefined rank.

C. One-Hot Encoding

● Use Case: Nominal data (no natural order). The gold standard for low-to-medium cardinality features.
● Examples: Country, Color, Product Category.
Creates new binary (0/1) columns for each category. A sample gets a 1 in the column corresponding to its category and 0

in all others.

● Original feature Color: ['Red', 'Blue', 'Green']

● After OHE:

It represents categorical variables as orthogonal vectors in a Euclidean space. This perfectly captures the fact that there
is no inherent order or distance between categories.
 ● The Dummy Variable Trap: If you have K categories, you only need K-1 new features to represent them completely.
The K-th feature is a linear combination of the others (e.g., if not Red and not Blue, it must be Green). Including all K
can cause multicollinearity in models like Linear Regression, making the model unstable. Most libraries
(pd.get_dummies, OneHotEncoder) can drop the first category to avoid this.

Introduction to Preprocessing Pipelines

A pipeline is a way to chain multiple data processing steps together in a single, cohesive object. This is a fundamental

best practice in machine learning.