Data Analysis — 16 Mark Q&A Notes

Permutations & Random Sampling | Challenges in Real-World Data Preprocessing

Q1. Explain Permutations and Random Sampling in Data Analysis with

Examples and Code. (16 marks)
Permutations and random sampling are fundamental techniques in data analysis used for resampling,
testing, and drawing representative subsets from data — essential in statistical inference, simulation, and
machine learning validation.

1. Permutations
A permutation is a rearrangement of the elements of a dataset in a different order. In data analysis,
permutations are widely used in:
• Permutation testing (a non-parametric hypothesis test) — checking if an observed
relationship/difference between groups is statistically significant by comparing it against many random
reshuffles of the data
• Shuffling data before splitting into training/test sets, to remove ordering bias
• Generating all possible arrangements for combinatorial analysis

Example: Using the 7 students' heights dataset: [164, 167.3, 170, 174.2, 178, 180, 186]. We can shuffle this
data randomly to remove any inherent order, or generate permutations to test if a subgroup's average height
is significantly different from the rest by chance.

Code:
import numpy as np
import pandas as pd

heights = [Link]([164, 167.3, 170, 174.2, 178, 180, 186])

df = [Link]({'Height (cm)': heights})

# Simple permutation (shuffle) of the dataset

shuffled = [Link](df['Height (cm)'])
print("Shuffled Heights:\n", shuffled)

# Permutation test example: compare two groups

group_A = [Link]([164, 167.3, 170])
group_B = [Link]([174.2, 178, 180, 186])
observed_diff = [Link](group_B) - [Link](group_A)

combined = [Link]([group_A, group_B])

diffs = []
for _ in range(1000):
[Link](combined)
new_A = combined[:len(group_A)]
new_B = combined[len(group_A):]
[Link]([Link](new_B) - [Link](new_A))

p_value = [Link]([Link](diffs) >= observed_diff)

print("Observed difference:", observed_diff)
 print("Permutation test p-value:", p_value)

Output interpretation: The p-value tells us how likely it is to see a difference as large as the observed one
purely by random chance. A low p-value (typically < 0.05) suggests the difference between groups is
statistically significant, not due to random shuffling.

2. Random Sampling
Random sampling is the process of selecting a subset of observations from a larger dataset such that every
observation has a known (often equal) chance of being selected. It ensures the sample is representative of
the population, avoiding bias.
Common types:
• Simple Random Sampling — every observation has equal probability of selection
• Stratified Sampling — population divided into subgroups (strata); samples drawn proportionally
• Systematic Sampling — every k-th observation is selected
• Sampling with/without Replacement — with replacement allows repeated selection (bootstrap);
without replacement ensures unique selections

Example: From the same heights dataset, randomly sample 4 out of 7 students to estimate the class's
average height without measuring everyone.

Code:
# Simple random sampling without replacement
sample = df['Height (cm)'].sample(n=4, random_state=42, replace=False)
print("Random Sample (no replacement):\n", sample)

# Random sampling with replacement (bootstrap sampling)

bootstrap_sample = df['Height (cm)'].sample(n=7, random_state=42, replace=True)
print("\nBootstrap Sample (with replacement):\n", bootstrap_sample)

print("\nOriginal Mean:", df['Height (cm)'].mean())

print("Sample Mean:", [Link]())

Output interpretation: The sample mean approximates the population mean without needing every
observation. Bootstrap sampling (with replacement) is used to estimate the variability of a statistic (like the
mean) by repeatedly resampling.

Applications, Advantages & Limitations

• Machine Learning: train-test splits, k-fold cross-validation, bootstrap aggregation (bagging)
• Surveys: sampling a subset of a population to estimate opinions cost-effectively
• A/B Testing: permutation tests validate whether a difference in conversion rates between two groups is
real or due to chance

Advantages: Reduces cost/time of studying an entire population; permutation tests make no assumption
about underlying data distribution (non-parametric).
Limitations: Small or poorly chosen samples may not represent the population well; permutation testing can
be computationally expensive for large datasets or many iterations.
Q2. Discuss the Challenges in Real-World Data Preprocessing with
Examples and Code. (16 marks)
Real-world data is rarely clean or ready for direct analysis. Data preprocessing — the step of transforming
raw data into an analyzable format — faces several practical challenges before univariate, bivariate, or
multivariate analysis can even begin.
Using the same student height dataset (extended with realistic imperfections) as a running example: [164,
167.3, None, 174.2, 178, 1800, 186]

Key Challenges
1. Missing Values
Real datasets often have gaps due to sensor failure, non-response, or data entry errors.
import numpy as np
import pandas as pd

heights = [164, 167.3, [Link], 174.2, 178, 1800, 186]

df = [Link]({'Height (cm)': heights})
print("Missing values:\n", [Link]().sum())

# Handling: fill with mean/median

df['Height (cm)'] = df['Height (cm)'].fillna(df['Height (cm)'].median())
print(df)

2. Outliers and Erroneous Entries

A value like 1800 cm is clearly a data-entry error (extra zero) rather than a real height, but the system won't
catch it automatically.
# Detecting outliers using IQR method
Q1 = df['Height (cm)'].quantile(0.25)
Q3 = df['Height (cm)'].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df['Height (cm)'] < Q1 - 1.5*IQR) | (df['Height (cm)'] > Q3 + 1.5*IQR)]
print("Outliers detected:\n", outliers)

3. Inconsistent Formats and Units

Heights might be recorded in cm by one source and inches by another; dates in DD/MM/YYYY vs
MM/DD/YYYY — requires standardization.

4. Duplicate Records
Same student's height recorded twice due to system glitches, inflating counts and skewing statistics.
df_dup = [Link]({'Height (cm)': [164, 167.3, 167.3, 174.2]})
print("Duplicates found:\n", df_dup.duplicated().sum())
df_clean = df_dup.drop_duplicates()

5. Noisy Data
Random errors or variance from measurement tools (e.g., a faulty measuring tape adding ±2 cm
inconsistently) that obscure true patterns.

6. Categorical Encoding Issues

Non-numeric data (like “Male”/“Female” in the Ad/Gender/Click-rate example) must be encoded numerically
(0/1, one-hot encoding) before use in regression or ML models — inconsistent labeling (“M”, “Male”, “male”)
complicates this.

7. Scale Differences
When combining multiple variables (e.g., height in cm and income in thousands), unscaled variables can
dominate models; requires normalization/standardization.
from [Link] import StandardScaler
scaler = StandardScaler()
df['Height_scaled'] = scaler.fit_transform(df[['Height (cm)']])
print(df)

8. Imbalanced or Insufficient Data

Some categories/groups (e.g., very few “Ad2” entries in the click-rate example) are underrepresented, biasing
multivariate models like regression.

9. High Dimensionality
In multivariate datasets with many variables, some may be irrelevant or redundant (multicollinearity), requiring
dimensionality reduction (PCA) before meaningful analysis.

Applications
• Healthcare: cleaning patient records (missing test results, inconsistent units) before analyzing blood
pressure trends
• Retail: removing duplicate transactions and correcting currency mismatches before sales-advertising
bivariate analysis
• Marketing: encoding categorical variables (ad type, gender) correctly before multivariate regression for
click-rate prediction

Advantages of Addressing These Challenges

• Improves accuracy and reliability of subsequent univariate/bivariate/multivariate analysis
• Prevents misleading conclusions caused by outliers or missing data
• Enables fair comparison across variables through scaling/normalization

Limitations
• Preprocessing is time-consuming — often 60-80% of a data analyst's effort
• Requires domain knowledge to distinguish true outliers from valid extreme values
• Over-cleaning (e.g., removing too many “outliers”) can remove genuine, useful data points

Conclusion
Data preprocessing directly impacts the validity of univariate, bivariate, and multivariate analyses. Using the
height dataset example, unresolved missing values, entry errors (1800 cm), or inconsistent units would distort
mean/variance calculations, corrupt correlation results, and bias regression coefficients — making rigorous
preprocessing a mandatory first step, not an optional one.