Data Analytics Tutorial

Complete Solutions

Indian Institute of Information Technology Allahabad

Department of Information Technology
IIIT Allahabad Data Analytics Tutorial Solutions

Contents
1 Unit 1: Data Preprocessing Techniques 2
1.1 Question 1: Mean Imputation and Variance Distortion . . . . . . . . . . 2
1.2 Question 2: Linear Interpolation Accuracy . . . . . . . . . . . . . . . . . 3
1.3 Question 3: Mean vs Median Imputation under Outliers . . . . . . . . . 4
1.4 Question 4: Min–Max Normalization and Distance Preservation . . . . . 4
1.5 Question 5: Custom Range Scaling . . . . . . . . . . . . . . . . . . . . . 5
1.6 Question 6: Z-Score under Distribution Drift . . . . . . . . . . . . . . . . 6
1.7 Question 7: IQR-Based Outlier Detection . . . . . . . . . . . . . . . . . . 6
1.8 Question 8: Robust Scaling using Median and IQR . . . . . . . . . . . . 7
1.9 Question 9: Mahalanobis Distance Computation . . . . . . . . . . . . . . 7
1.10 Question 10: Quantization Error in Equal-Width Binning . . . . . . . . . 9
1.11 Question 11: Equal-Frequency Binning Instability . . . . . . . . . . . . . 10
1.12 Question 12: Moving Average Smoothing . . . . . . . . . . . . . . . . . . 10
1.13 Question 13: Median Filtering Performance . . . . . . . . . . . . . . . . 11
1.14 Question 14: One-Hot Encoding and Sparsity . . . . . . . . . . . . . . . 12
1.15 Question 15: Ordinal Encoding Bias Quantification . . . . . . . . . . . . 12
1.16 Question 16: Sample Size Computation . . . . . . . . . . . . . . . . . . . 13
1.17 Question 17: Stratified Sampling Allocation . . . . . . . . . . . . . . . . 13
1.18 Question 18: Scaling-Induced Data Leakage . . . . . . . . . . . . . . . . 14
1.19 Question 19: PCA Variance Retention . . . . . . . . . . . . . . . . . . . 14
1.20 Question 20: Dimensionality Reduction Ratio . . . . . . . . . . . . . . . 15
1.21 Question 21: Class Imbalance Severity . . . . . . . . . . . . . . . . . . . 15
1.22 Question 22: SMOTE Oversampling Size . . . . . . . . . . . . . . . . . . 16
1.23 Question 23: Log Transformation Impact . . . . . . . . . . . . . . . . . . 16
1.24 Question 24: Box–Cox Variance Stabilization . . . . . . . . . . . . . . . . 17
1.25 Question 25: Integrated Numerical Preprocessing Pipeline . . . . . . . . 18

2 Unit 2: Association Rule Mining 19

2.1 Question 26: Apriori Algorithm Execution . . . . . . . . . . . . . . . . . 19
2.2 Question 27: Apriori Pruning and Candidate Reduction . . . . . . . . . . 20
2.3 Question 28: Support Threshold Sensitivity . . . . . . . . . . . . . . . . 21
2.4 Question 29: Apriori Computational Complexity . . . . . . . . . . . . . . 22
2.5 Question 30: FP-Tree Construction and Mining . . . . . . . . . . . . . . 22
2.6 Question 31: FP-Growth vs Apriori Efficiency . . . . . . . . . . . . . . . 23
2.7 Question 32: Conditional FP-Tree Mining . . . . . . . . . . . . . . . . . 24
2.8 Question 33: FP-Tree Compression Ratio . . . . . . . . . . . . . . . . . . 24
2.9 Question 34: Downward Closure Bound Computation . . . . . . . . . . . 25
2.10 Question 35: Closed Frequent Itemset Identification . . . . . . . . . . . . 25
2.11 Question 36: Maximal Frequent Itemset Detection . . . . . . . . . . . . . 26
2.12 Question 37: Confidence and Lift Calculation . . . . . . . . . . . . . . . 26
2.13 Question 38: Conviction Computation . . . . . . . . . . . . . . . . . . . 27
2.14 Question 39: Rule Filtering using Interestingness . . . . . . . . . . . . . 27
2.15 Question 40: Numerical Interpretation of Association Rules . . . . . . . . 27

1
IIIT Allahabad Data Analytics Tutorial Solutions

1 Unit 1: Data Preprocessing Techniques

1.1 Question 1: Mean Imputation and Variance Distortion
Problem: Daily internet usage (in GB) of students residing in BH1 is partially missing
and recorded as [4.2, 5.1, NaN, 6.3, 5.9, NaN, 6.5].

(a) Compute the mean using only non-missing observations

Step 1: Identify non-missing values:

Non-missing values = [4.2, 5.1, 6.3, 5.9, 6.5]

Count = 5 observations
Step 2: Calculate sum:

Sum = 4.2 + 5.1 + 6.3 + 5.9 + 6.5 = 28.0 GB

Step 3: Calculate mean:

28.0
x̄ = = 5.6 GB
5
Answer: Mean = 5.6 GB

(b) Replace missing values using mean imputation

After replacing NaN values with 5.6:
Answer: [4.2, 5.1, 5.6, 6.3, 5.9, 5.6, 6.5]

(c) Compute variance before and after imputation

Step 1: Variance BEFORE imputation (n=5):
n
1X
σ2 = (xi − x̄)2
n i=1
1
= [(4.2 − 5.6)2 + (5.1 − 5.6)2 + (6.3 − 5.6)2 + (5.9 − 5.6)2 + (6.5 − 5.6)2 ]
5
1
= [1.96 + 0.25 + 0.49 + 0.09 + 0.81]
5
3.6
= = 0.72
5
Step 2: Variance AFTER imputation (n=7):
1
σ 2 = [(4.2 − 5.6)2 + (5.1 − 5.6)2 + (5.6 − 5.6)2 + (6.3 − 5.6)2
7
+ (5.9 − 5.6)2 + (5.6 − 5.6)2 + (6.5 − 5.6)2 ]
1
= [1.96 + 0.25 + 0 + 0.49 + 0.09 + 0 + 0.81]
7
3.6
= ≈ 0.514
7

2
IIIT Allahabad Data Analytics Tutorial Solutions

Step 3: Calculate percentage change:

0.514 − 0.72
Percentage change = × 100 = −28.6%
0.72
Answer: Variance decreased by 28.6%
Note: Mean imputation artificially reduces variance by adding values at the center of
the distribution.

1.2 Question 2: Linear Interpolation Accuracy

Problem: Electricity consumption (kWh) of BH2 over six equally spaced days is [210, 215,
NaN, NaN, 235, 240].

(a) Estimate the missing values using linear interpolation

Step 1: Identify boundary values:

• Before missing: Day 2 = 215 kWh

• After missing: Day 5 = 235 kWh

• Number of intervals between them: 3

Step 2: Calculate step size:

235 − 215 20
Step size = = ≈ 6.67 kWh
3 3
Step 3: Interpolate missing values:

Day 3 = 215 + 6.67 = 221.67 kWh

Day 4 = 221.67 + 6.67 = 228.33 kWh

Answer: Missing values: 221.67 kWh and 228.33 kWh

(b) Compute Mean Absolute Error (MAE) if actual values are 225 and 230
Step 1: Calculate absolute errors:

Error at Day 3 = |221.67 − 225| = 3.33 kWh

Error at Day 4 = |228.33 − 230| = 1.67 kWh

Step 2: Calculate MAE:

3.33 + 1.67 5.0
MAE = = = 2.5 kWh
2 2
Answer: MAE = 2.5 kWh

3
IIIT Allahabad Data Analytics Tutorial Solutions

1.3 Question 3: Mean vs Median Imputation under Outliers

Problem: Water consumption (litres) is [180, 190, 200, NaN, 210, 920].

(a) Impute the missing value using mean and median separately
Step 1: Calculate MEAN of non-missing values:
180 + 190 + 200 + 210 + 920 1700
Mean = = = 340 litres
5 5
Step 2: Calculate MEDIAN of non-missing values:
Sorted: [180, 190, 200, 210, 920]
Median (middle value) = 200 litres
Answer: Mean imputation: 340 litres; Median imputation: 200 litres

(b) Compute absolute deviation from true value (assuming true value = 205)
Step 1: Deviation with MEAN imputation:
|340 − 205| = 135 litres
Step 2: Deviation with MEDIAN imputation:
|200 − 205| = 5 litres
Answer: Mean deviation: 135 litres; Median deviation: 5 litres
Note: Median imputation is 27 times better than mean when outliers are present!
(135/5 = 27)

1.4 Question 4: Min–Max Normalization and Distance Preser-

vation
Problem: CPU utilization (%) is [5, 20, 40, 80, 160].

(a) Normalize data to [0, 1]

Step 1: Identify min and max:
min = 5, max = 160, Range = 155
Step 2: Apply normalization formula: x′ = x−min
max − min
5−5
5→ = 0.000
155
20 − 5 15
20 → = = 0.097
155 155
40 − 5 35
40 → = = 0.226
155 155
80 − 5 75
80 → = = 0.484
155 155
160 − 5 155
160 → = = 1.000
155 155

4
IIIT Allahabad Data Analytics Tutorial Solutions

Answer: [0.000, 0.097, 0.226, 0.484, 1.000]

(b) Compute Euclidean distance between 20 and 160 before and after
Step 1: Distance BEFORE normalization:
d = |160 − 20| = 140
Step 2: Distance AFTER normalization:
d′ = |1.000 − 0.097| = 0.903
Step 3: Verify relationship:
d 140
d′ = = = 0.903 ✓
Range 155
Answer: Before: 140; After: 0.903
Note: Min-max normalization preserves relative distances scaled by the range.

1.5 Question 5: Custom Range Scaling

Problem: Network traffic ranges from 100 MB to 6100 MB.

(a) Normalize 3100 MB to [−1, 1]

Step 1: Apply formula: x′ = 2 × x−min
max − min
−1
3100 − 100
x′ = 2 × −1
6100 − 100
3000
=2× −1
6000
= 2 × 0.5 − 1
=1−1=0
Answer: 3100 MB → 0

(b) Verify symmetry around zero

Step 1: Check midpoint:
100 + 6100 6200
Midpoint = = = 3100 MB ✓
2 2
Step 2: Check minimum:
0
100 → 2 × − 1 = −1 ✓
6000
Step 3: Check maximum:
6000
6100 → 2 × − 1 = 2 − 1 = +1 ✓
6000
Answer: Symmetry verified: min→ −1, midpoint→ 0, max→ +1

5
IIIT Allahabad Data Analytics Tutorial Solutions

1.6 Question 6: Z-Score under Distribution Drift

Problem: Latency before upgrade: µ = 40, σ = 5; after upgrade: µ = 60, σ = 20.
Compute z-score of latency 70 for both distributions.

(a) Compute z-scores

Step 1: Z-score BEFORE upgrade:
x−µ 70 − 40 30
z= = = = 6.0
σ 5 5
Interpretation: 70 is 6 standard deviations above mean (extremely high!)
Step 2: Z-score AFTER upgrade:
70 − 60 10
z= = = 0.5
20 20
Interpretation: 70 is only 0.5 standard deviations above mean (quite normal)
Answer: Before: z = 6.0; After: z = 0.5

(b) Compute relative deviation change

Step 1: Calculate percentage change:
0.5 − 6.0 −5.5
Change = × 100 = × 100 = −91.67%
6.0 6.0
Answer: Relative deviation decreased by 91.67%
Note: The same value appears much more ”normal” after upgrade due to increased
mean and variability.

1.7 Question 7: IQR-Based Outlier Detection

Problem: Noise levels (dB): [46, 47, 48, 49, 50, 90].

(a) Compute Q1 , Q3 , IQR

Step 1: Find Q1 (25th percentile):
Position = 0.25 × (6 + 1) = 1.75
Q1 = 46 + 0.75 × (47 − 46) = 46 + 0.75 = 46.75
Step 2: Find Q3 (75th percentile):
Position = 0.75 × (6 + 1) = 5.25
Q3 = 50 + 0.25 × (90 − 50) = 50 + 10 = 60
Step 3: Calculate IQR:
IQR = Q3 − Q1 = 60 − 46.75 = 13.25
Answer: Q1 = 46.75, Q3 = 60, IQR = 13.25

6
IIIT Allahabad Data Analytics Tutorial Solutions

(b) Determine if 90 is an outlier

Step 1: Calculate upper bound:

Upper bound = Q3 + 1.5 × IQR = 60 + 1.5 × 13.25 = 60 + 19.875 = 79.875

Step 2: Compare:
90 > 79.875
Answer: YES, 90 is an outlier

1.8 Question 8: Robust Scaling using Median and IQR

Problem: Temperature (◦ C): [29, 30, 31, 32, 120].

(a) Compute median and IQR

Step 1: Find median:
Sorted: [29, 30, 31, 32, 120]
Median (middle value) = 31◦ C
Step 2: Find Q1 and Q3 (using first 4 values for robustness):
29 + 30
Q1 = = 29.5
2
31 + 32
Q3 = = 31.5
2
Step 3: Calculate IQR:

IQR = 31.5 − 29.5 = 2◦ C

Answer: Median = 31◦ C, IQR = 2◦ C

(b) Compute robust-scaled value of 120

Step 1: Apply formula: x′ = x−median
IQR

120 − 31 89
x′ = = = 44.5
2 2
Answer: Robust-scaled value = 44.5
Note: 120 is 44.5 IQR units above the median, clearly an outlier!

1.9 Question 9: Mahalanobis Distance Computation

Problem: Sports data with Speed (m/s) and Heart Rate (bpm).

7
IIIT Allahabad Data Analytics Tutorial Solutions

Speed (m/s) Heart Rate (bpm)

6 110
7 115
8 118
9 121
15 198

(a) Compute mean vector and covariance matrix

Step 1: Calculate mean vector µ:
6 + 7 + 8 + 9 + 15 45
µspeed = = = 9 m/s
5 5
110 + 115 + 118 + 121 + 198 662
µHR = = = 132.4 bpm
5 5
 
9
µ=
132.4
Step 2: Calculate deviations:

Point 1: [6 − 9, 110 − 132.4] = [−3, −22.4]

Point 2: [7 − 9, 115 − 132.4] = [−2, −17.4]
Point 3: [8 − 9, 118 − 132.4] = [−1, −14.4]
Point 4: [9 − 9, 121 − 132.4] = [0, −11.4]
Point 5: [15 − 9, 198 − 132.4] = [6, 65.6]

Step 3: Calculate covariance matrix Σ:

9 + 4 + 1 + 0 + 36 50
Var(Speed) = = = 10
5 5
501.76 + 302.76 + 207.36 + 129.96 + 4303.36 5445.2
Var(HR) = = = 1089.04
5 5
67.2 + 34.8 + 14.4 + 0 + 393.6 510
Cov(Speed, HR) = = = 102
5 5
 
10 102
Σ=
102 1089.04
 
10 102
Answer: µ = [9, 132.4], Σ =
102 1089.04

(b) Compute Mahalanobis distance of last observation [15, 198]

Step 1: Calculate determinant:

det(Σ) = 10 × 1089.04 − 1022 = 10890.4 − 10404 = 486.4

Step 2: Calculate inverse:

 
−1 1 1089.04 −102
Σ =
486.4 −102 10

8
IIIT Allahabad Data Analytics Tutorial Solutions

Step 3: Calculate Mahalanobis distance:

   
15 − 9 6
x−µ= =
198 − 132.4 65.6

D2 = (x − µ)T Σ−1 (x − µ) ≈ 8.96

√
D= 8.96 ≈ 2.99
Answer: Mahalanobis distance ≈ 2.99
Note: A distance of 3 suggests the last observation may be a multivariate outlier.

1.10 Question 10: Quantization Error in Equal-Width Binning

Problem: Noise ranges from 35 dB to 135 dB.

(a) Construct 5 equal-width bins

Step 1: Calculate bin width:

Range = 135 − 35 = 100 dB

100
Bin width = = 20 dB
5
Step 2: Define bins:

Bin 1: [35, 55)

Bin 2: [55, 75)
Bin 3: [75, 95)
Bin 4: [95, 115)
Bin 5: [115, 135]

Answer: 5 bins with width 20 dB each

(b) Compute maximum possible quantization error

Step 1: Quantization error concept:
• Values in a bin are typically represented by the bin center

• Maximum error occurs at bin edges

Step 2: Calculate:
Bin width 20
Max quantization error = = = 10 dB
2 2
Answer: Maximum quantization error = 10 dB

9
IIIT Allahabad Data Analytics Tutorial Solutions

1.11 Question 11: Equal-Frequency Binning Instability

Problem: Footfall data: [80, 95, 120, 150, 400, 800].

(a) Perform equal-frequency binning using 3 bins

Step 1: Each bin contains 6/3 = 2 elements:

Bin 1: [80, 95], width = 95 − 80 = 15

Bin 2: [120, 150], width = 150 − 120 = 30
Bin 3: [400, 800], width = 800 − 400 = 400

Answer: Bin widths: 15, 30, 400

(b) Compute coefficient of variation

Step 1: Calculate mean width:
15 + 30 + 400 445
w̄ = = ≈ 148.33
3 3
Step 2: Calculate standard deviation:
r
(15 − 148.33)2 + (30 − 148.33)2 + (400 − 148.33)2
s=
r 3
17776.89 + 14002.09 + 63357.69
=
3
√
r
95136.67
= = 31712.22 ≈ 178.08
3
Step 3: Calculate coefficient of variation:
s 178.08
CV = = ≈ 1.20 = 120%
w̄ 148.33
Answer: CV ≈ 120% (high instability)
Note: High CV indicates extreme variability in bin widths!

1.12 Question 12: Moving Average Smoothing

Problem: Temperature readings: [30, 31, 29, 32, 130, 33, 34].

10
IIIT Allahabad Data Analytics Tutorial Solutions

(a) Apply moving average with window size 3

Step 1: Calculate moving averages:
Index 0: undefined (need 3 points)
30 + 31 + 29 90
Index 1: = = 30.00
3 3
31 + 29 + 32 92
Index 2: = ≈ 30.67
3 3
29 + 32 + 130 191
Index 3: = ≈ 63.67
3 3
32 + 130 + 33 195
Index 4: = = 65.00
3 3
130 + 33 + 34 197
Index 5: = ≈ 65.67
3 3
Index 6: undefined
Answer: [−, 30.00, 30.67, 63.67, 65.00, 65.67, −]

(b) Compute percentage spike reduction

Step 1: Original spike at index 4:
Original value = 130
Step 2: Smoothed value:
Smoothed value = 65.00
Step 3: Calculate reduction:
130 − 65 65
Reduction = × 100 = × 100 = 50%
130 130
Answer: Spike reduced by 50%

1.13 Question 13: Median Filtering Performance

Problem: Using Question 12 data: [30, 31, 29, 32, 130, 33, 34].

(a) Apply median filter (k = 3)

Step 1: Calculate medians:
Index 0: undefined
Index 1: median(30, 31, 29) = 30
Index 2: median(31, 29, 32) = 31
Index 3: median(29, 32, 130) = 32
Index 4: median(32, 130, 33) = 33
Index 5: median(130, 33, 34) = 34
Index 6: undefined
Answer: [−, 30, 31, 32, 33, 34, −]

11
IIIT Allahabad Data Analytics Tutorial Solutions

(b) Compute absolute error at spike

Step 1: At spike position (index 4):

Original value = 130

Filtered value = 33
Absolute error = |130 − 33| = 97

Answer: Absolute error = 97

Note: Median filter completely removes the spike! Much better than moving average.

1.14 Question 14: One-Hot Encoding and Sparsity

Problem: Hostel ∈ {BH1, BH2, BH3, BH4, BH5}.

(a) Compute total new features

Step 1: One-hot encoding:
• Each category becomes a binary feature

• 5 categories = 5 binary features

Answer: 5 new features

(b) Compute sparsity for 12,000 records

Step 1: Calculate total entries:

Total entries = 12,000 × 5 = 60,000

Step 2: Calculate non-zero entries:

Non-zero entries = 12,000 (one 1 per row)

Step 3: Calculate zero entries:

Zero entries = 60,000 − 12,000 = 48,000

Step 4: Calculate sparsity:

48,000
Sparsity = = 0.8 = 80%
60,000
Answer: Sparsity = 80%

1.15 Question 15: Ordinal Encoding Bias Quantification

Problem: Levels {Low, M edium, High} encoded as {1, 2, 3}.

12
IIIT Allahabad Data Analytics Tutorial Solutions

(a) Compute mean encoded value

1+2+3 6
Mean == =2
3 3
Answer: Mean encoded value = 2

(b) Compute pairwise artificial distances

d(Low, Medium) = |1 − 2| = 1
d(Medium, High) = |2 − 3| = 1
d(Low, High) = |1 − 3| = 2

Answer: Pairwise distances: 1, 1, 2

Note: Ordinal encoding creates artificial metric relationships that may not reflect true
semantic distances!

1.16 Question 16: Sample Size Computation

Problem: Wi-Fi logs contain 90,000 records. Compute minimum sample size for 95%
confidence and 3% margin of error.
Step 1: Use sample size formula:
Z 2 × p × (1 − p)
n=
E2
Step 2: Identify parameters:

Z = 1.96 (for 95% confidence)

p = 0.5 (maximum variance assumption)
E = 0.03 (3% margin of error)

Step 3: Calculate:
(1.96)2 × 0.5 × 0.5
n=
(0.03)2
3.8416 × 0.25
=
0.0009
0.9604
=
0.0009
≈ 1067.11

Answer: Minimum sample size n ≈ 1068 records

1.17 Question 17: Stratified Sampling Allocation

Problem: Sports = 30%, Non-sports = 70%.

13
IIIT Allahabad Data Analytics Tutorial Solutions

(a) Allocate samples for n = 500

Step 1: Proportional allocation:

nsports = 500 × 0.30 = 150

nnon-sports = 500 × 0.70 = 350

Answer: Sports: 150, Non-sports: 350

(b) Compute sampling fraction per stratum

Step 1: Sampling fraction:
n 500
f= =
N N
where N is the total population size.
Answer: Sampling fraction = 500 N
(same for both strata)
Note: Proportional stratified sampling maintains the same sampling fraction across
all strata.

1.18 Question 18: Scaling-Induced Data Leakage

Problem: Training max = 85, test value = 120.
Step 1: Normalize using training statistics (assuming min = 0):
120 − 0 120
x′test = = ≈ 1.41
85 − 0 85
Step 2: Compute overshoot:

Overshoot = (1.41 − 1) × 100 = 0.41 × 100 = 41%

Answer: Normalized test value = 1.41; Overshoot = 41%

Note: This demonstrates data leakage! The scaler should only use training data statis-
tics.

1.19 Question 19: PCA Variance Retention

Problem: Eigenvalues: [7, 4, 2, 1, 0.5].
Step 1: Calculate total variance:

Total = 7 + 4 + 2 + 1 + 0.5 = 14.5

14
IIIT Allahabad Data Analytics Tutorial Solutions

Step 2: Calculate cumulative variance:

7
PC1: = 0.483 = 48.3%
14.5
7+4 11
PC1-2: = = 0.759 = 75.9%
14.5 14.5
7+4+2 13
PC1-3: = = 0.897 = 89.7%
14.5 14.5
7+4+2+1 14
PC1-4: = = 0.966 = 96.6%
14.5 14.5
14.5
PC1-5: = 1.000 = 100%
14.5
Step 3: Find minimum components for 92%:

PC1-4 gives 96.6% ¿ 92%

Answer: Need 4 components to retain at least 92% variance

1.20 Question 20: Dimensionality Reduction Ratio

Problem: Original features = 15, retained = 5.
Step 1: Compression ratio:
15
Ratio = =3:1
5
Step 2: Reduction percentage:
15 − 5 10
Reduction = × 100 = × 100 = 66.67%
15 15
Answer: Compression ratio = 3:1; Reduction = 66.67%

1.21 Question 21: Class Imbalance Severity

Problem: Infected = 25, Healthy = 975.
Step 1: Imbalance ratio:
975
Ratio = = 39 : 1
25
Step 2: Minority prevalence:
25 25
Prevalence = = = 0.025 = 2.5%
25 + 975 1000
Answer: Imbalance ratio = 39:1; Minority prevalence = 2.5%

15
IIIT Allahabad Data Analytics Tutorial Solutions

1.22 Question 22: SMOTE Oversampling Size

Problem: Balance dataset from Question 21 using SMOTE.
Step 1: Current distribution:
• Minority (Infected): 25
• Majority (Healthy): 975
Step 2: SMOTE balancing:
• Generate synthetic samples to match majority class
• Target: 975 infected samples
• Need to generate: 975 − 25 = 950 synthetic samples
Step 3: Final size:
Total = 975 (Infected) + 975 (Healthy) = 1950
Answer: Final dataset size = 1950 records

1.23 Question 23: Log Transformation Impact

Problem: Traffic: [5, 50, 500, 5000].

(a) Apply log10

log10 (5) ≈ 0.70

log10 (50) ≈ 1.70
log10 (500) ≈ 2.70
log10 (5000) ≈ 3.70
Answer: [0.70, 1.70, 2.70, 3.70]

(b) Compute variance reduction ratio

Step 1: Variance BEFORE transformation:
5 + 50 + 500 + 5000
x̄ = = 1388.75
4
(5 − 1388.75)2 + (50 − 1388.75)2 + (500 − 1388.75)2 + (5000 − 1388.75)2
σ2 =
4
≈ 5,185,416
Step 2: Variance AFTER transformation:
0.70 + 1.70 + 2.70 + 3.70
ȳ = = 2.20
4
2 (0.70 − 2.20)2 + (1.70 − 2.20)2 + (2.70 − 2.20)2 + (3.70 − 2.20)2
σlog =
4
≈ 1.67

16
IIIT Allahabad Data Analytics Tutorial Solutions

Step 3: Variance reduction ratio:

5,185,416
Ratio = ≈ 3,105,638 : 1
1.67
Answer: Variance reduced by ratio ≈ 3,000,000 : 1 (massive reduction!)

1.24 Question 24: Box–Cox Variance Stabilization

Problem: Original data variance = 625. Assume λ = 0.5 and compute transformed
variance trend.

(a) Compute transformed values for data point x = 25

Step 1: Box-Cox transformation formula for λ = 0.5:
√
x0.5 − 1 x−1
y= =
0.5 0.5
Step 2: For x = 25:
√
25 − 1 5−1 4
y= = = =8
0.5 0.5 0.5
Answer: Transformed value = 8

(b) Variance trend

Step 1: Effect of λ = 0.5 (square root):

• Compresses large values more than small values

• Reduces right skewness

• Stabilizes variance (reduces it)

Step 2: Variance trend:

Variance DECREASES after Box-Cox transformation

Answer: Variance decreases (stabilizes)

Note: Box-Cox with λ = 0.5 is equivalent to square root transformation, which com-
presses large values and stabilizes variance.

17
IIIT Allahabad Data Analytics Tutorial Solutions

1.25 Question 25: Integrated Numerical Preprocessing Pipeline

Problem: Given missing values, outliers, skewness, categorical features, and imbalance,
assign numerical preprocessing steps in correct order.
Correct Order and Justification:

Step 1: Handle Missing Values (Imputation)

Justification: Cannot process data with NaN values. Must impute first.
Numerical effect: Enables computation of statistics and transformations.

Step 2: Detect and Handle Outliers

Justification: Outliers affect scaling and transformations.
Numerical effect: Prevents extreme values from skewing normalization (e.g.,
min-max would map outliers to 1).

Step 3: Encode Categorical Features

Justification: ML algorithms require numerical input.
Numerical effect: One-hot encoding creates binary features; ordinal encoding
creates integer features.

Step 4: Transform Skewness (Log/Box-Cox)

Justification: Many algorithms assume normality; scaling works better on
normalized data.
Numerical effect: Reduces variance by orders of magnitude (e.g., Q23: vari-
ance reduced 3M×).

Step 5: Scale/Normalize Features

Justification: Features on different scales bias distance-based algorithms.
Numerical effect: Brings all features to same range (e.g., [0,1] or [-1,1]).

Step 6: Handle Class Imbalance (SMOTE/Undersampling)

Justification: Should be done after all feature engineering is complete.
Numerical effect: Changes dataset size; synthetic samples inherit scaled/transformed
features.

Step 7: Dimensionality Reduction (PCA)

Justification: Final step before modeling; operates on fully preprocessed data.
Numerical effect: Reduces feature count while retaining variance (e.g., Q19:
4 PCs retain 96.6%).

Answer: Order: Imputation → Outliers → Encoding → Skewness → Scaling

→ Imbalance → PCA
Note: This order prevents data leakage and ensures each step operates on properly
prepared data.

18
IIIT Allahabad Data Analytics Tutorial Solutions

2 Unit 2: Association Rule Mining

Transaction Database (for Questions 26, 30, 31)

TID Items
T1 A, B, C
T2 A, C, D
T3 B, C, E
T4 A, B, C, D
T5 A, B, E
T6 B, C, D
T7 A, C
T8 A, B, C, E
T9 A, B, D
T10 B, C

Table 1: Transaction database with 10 transactions

2.1 Question 26: Apriori Algorithm Execution

Problem: Using the transaction database above, minimum support = 40% (4 transac-
tions).

(a) Generate L1 , L2 , L3 using Apriori

Step 1: Count 1-itemsets:

A: T1, T2, T4, T5, T7, T8, T9 = 7

B: T1, T3, T4, T5, T6, T8, T9, T10 = 8
C: T1, T2, T3, T4, T6, T7, T8, T10 = 8
D: T2, T4, T6, T9 = 4
E: T3, T5, T8 = 3

Step 2: Generate L1 (support ≥ 4):

L1 = {A, B, C, D}

Step 3: Generate candidates C2 and count:

AB: T1, T4, T5, T8, T9 = 5

AC: T1, T2, T4, T7, T8 = 5
AD: T2, T4, T9 = 3
BC: T1, T3, T4, T6, T8, T10 = 6
BD: T4, T6, T9 = 3
CD: T2, T4, T6 = 3

19
IIIT Allahabad Data Analytics Tutorial Solutions

Step 4: Generate L2 (support ≥ 4):

L2 = {AB, AC, BC}
Step 5: Generate candidates C3 :
• From L2 , we can form: ABC
• All 2-subsets (AB, AC, BC) are in L2 ✓
Step 6: Count ABC:
ABC: T1, T4, T8 = 3 < 4
Step 7: Generate L3 :
L3 = {} (empty, no 3-itemsets meet threshold)
Answer: L1 = {A, B, C, D}, L2 = {AB, AC, BC}, L3 = {}

(b) List all candidate itemsets at each level

C1 = {A, B, C, D, E}
C2 = {AB, AC, AD, BC, BD, CD}
C3 = {ABC}
Answer: C1 : 5 candidates, C2 : 6 candidates, C3 : 1 candidate

(c) Compute total number of database scans

Step 1: Scans required:
• Scan 1: Count C1
• Scan 2: Count C2
• Scan 3: Count C3
Answer: 3 database scans

2.2 Question 27: Apriori Pruning and Candidate Reduction

Problem: Candidate supports: AB=3, AC=4, BC=2, BD=1; min-support=3.

(a) Identify pruned candidates

Step 1: Compare each candidate with threshold:
AB = 3 ≥ 3 ✓ (kept)
AC = 4 ≥ 3 ✓ (kept)
BC = 2 < 3 × (PRUNED)
BD = 1 < 3 × (PRUNED)
Answer: Pruned candidates: BC, BD

20
IIIT Allahabad Data Analytics Tutorial Solutions

(b) Compute reduction percentage

Step 1: Calculate:

Total candidates = 4
Pruned = 2
2
Reduction = × 100 = 50%
4
Answer: Reduction percentage = 50%

2.3 Question 28: Support Threshold Sensitivity

Problem: Total transactions = 200.

(a) Compute min-support count for 30% and 10%

Step 1: For 30% threshold:

Min count = 200 × 0.30 = 60 transactions

Step 2: For 10% threshold:

Min count = 200 × 0.10 = 20 transactions

Answer: 30%: 60 transactions; 10%: 20 transactions

(b) Compute candidate growth ratio

Step 1: Compare thresholds:

• Lower threshold (10%) allows more items to be frequent

• More frequent items → more candidate combinations

Step 2: Growth ratio estimate:

60
Ratio = =3
20
Answer: Candidate growth ratio ≈ 3× (lowering threshold increases can-
didates)
Note: Lower support thresholds exponentially increase the number of frequent itemsets
and candidates.

21
IIIT Allahabad Data Analytics Tutorial Solutions

2.4 Question 29: Apriori Computational Complexity

Problem: Items = 8, Transactions = 1000. Compute worst-case number of candidate
3-itemsets.
Step 1: Worst case assumption:

• All 8 items are frequent in L1

• All possible 2-itemsets are in L2

Step 2: Calculate combinations:

 
8 8! 8×7×6 336
C3 = = = = = 56
3 3! × 5! 3×2×1 6

Answer: Worst-case: 56 candidate 3-itemsets

2.5 Question 30: FP-Tree Construction and Mining

Problem: Using the transaction database, min-support = 40% (4 transactions).

(a) Construct FP-tree

Step 1: Find frequent items and order by frequency:

B: 8 (highest)
C: 8
A: 7
D: 4

Frequency order: B ¿ C ¿ A ¿ D
Step 2: Build FP-tree:

• Root node

• Process each transaction in frequency order

• T1 (A,B,C): B → C → A

• T2 (A,C,D): C → A → D

• T3 (B,C,E): B → C

• T4 (A,B,C,D): B → C → A → D

• ... (continue for all transactions)

Step 3: FP-tree structure (simplified):

22
IIIT Allahabad Data Analytics Tutorial Solutions

Root
B:6
C:5
A:3
D:1
C:2
A:2
D:1

Answer: FP-tree constructed with root and branches ordered by frequency

B-C-A-D

(b) Compute conditional pattern base size for item D

Step 1: Find all paths containing D:

• T2: A, C, D → prefix: {A, C}

• T4: A, B, C, D → prefix: {B, C, A}

• T6: B, C, D → prefix: {B, C}

• T9: A, B, D → prefix: {B, A}

Step 2: Conditional pattern base for D:

{BCA} : 1
{CA} : 1
{BC} : 1
{BA} : 1

Answer: Conditional pattern base size = 4 paths

2.6 Question 31: FP-Growth vs Apriori Efficiency

Problem: Compare database scans for the given transaction database.

(a) Apriori scans

Step 1: From Question 26:

• Scan 1: Count C1

• Scan 2: Count C2

• Scan 3: Count C3

Answer: Apriori requires 3 scans

23
IIIT Allahabad Data Analytics Tutorial Solutions

(b) FP-Growth scans

Step 1: FP-Growth process:
• Scan 1: Count item frequencies
• Scan 2: Build FP-tree
• No more scans needed (mining happens on tree)
Answer: FP-Growth requires 2 scans

(c) Scan reduction percentage

3−2 1
Reduction = × 100 = × 100 ≈ 33.3%
3 3
Answer: Scan reduction = 33.3%
Note: FP-Growth is more efficient as it avoids repeated database scans by using the
compressed FP-tree structure.

2.7 Question 32: Conditional FP-Tree Mining

Problem: Conditional base {A : 3, B : 2, D : 1}, min-support=2.
Step 1: Identify frequent items in conditional base:
A: 3 ≥ 2 ✓
B: 2 ≥ 2 ✓
D: 1 < 2 ×
Step 2: Frequent items: {A, B}
Step 3: Check combination AB:
• Need to verify if A and B co-occur ≥ 2 times in conditional base
• Without co-occurrence data, we can only confirm individual items
Answer: Frequent patterns: {A, B} (and potentially {AB} if they co-occur
≥ 2 times)

2.8 Question 33: FP-Tree Compression Ratio

Problem: Original DB = 50,000, FP-tree nodes = 4,000.
Step 1: Calculate compression ratio:
50,000
Compression ratio = = 12.5 : 1
4,000
Answer: Compression ratio = 12.5:1
Note: FP-tree achieves massive compression by sharing common prefixes!

24
IIIT Allahabad Data Analytics Tutorial Solutions

2.9 Question 34: Downward Closure Bound Computation

Problem: Given support(ABC) = 15.
Step 1: Apply downward closure (Apriori) property:

• If itemset is frequent, all subsets must be frequent

• If subset is infrequent, all supersets must be infrequent

Step 2: Lower bounds for subsets:

support(AB) ≥ 15
support(AC) ≥ 15
support(BC) ≥ 15
support(A) ≥ 15
support(B) ≥ 15
support(C) ≥ 15

Step 3: Upper bounds for supersets:

• Any itemset containing ABC cannot have support > 15

• E.g., support(ABCD) ≤ 15, support(ABCE) ≤ 15

Answer: All subsets: support ≥ 15; All supersets: support ≤ 15

2.10 Question 35: Closed Frequent Itemset Identification

Problem: support(A) = 50, support(AB) = 50.
Step 1: Definition of closed itemset:

• An itemset is closed if no immediate superset has the same support

Step 2: Analysis:

• A has support 50

• AB (superset of A) also has support 50

• Therefore, A is NOT closed

• If no superset of AB has support 50, then AB is closed

Answer: AB is CLOSED (assuming no larger itemset has support 50); A

is NOT closed

25
IIIT Allahabad Data Analytics Tutorial Solutions

2.11 Question 36: Maximal Frequent Itemset Detection

Problem: Frequent sets {A, B, AB, ABC}.
Step 1: Definition of maximal frequent itemset:

• A frequent itemset is maximal if no immediate superset is frequent

Step 2: Analysis:

A ⊂ AB ⊂ ABC ⇒ A not maximal

B ⊂ AB ⊂ ABC ⇒ B not maximal
AB ⊂ ABC ⇒ AB not maximal
ABC has no frequent superset ⇒ ABC is maximal

Answer: ABC is the ONLY maximal frequent itemset

2.12 Question 37: Confidence and Lift Calculation

Problem: support(A) = 0.4, support(B) = 0.5, support(A,B) = 0.3.

Compute Confidence(A→B)
Step 1: Formula:
support(A ∩ B)
Confidence(A → B) =
support(A)
Step 2: Calculate:
0.3
Confidence =
= 0.75 = 75%
0.4
Answer: Confidence = 0.75 or 75%

Compute Lift(A→B)
Step 1: Formula:
support(A ∩ B)
Lift(A → B) =
support(A) × support(B)
Step 2: Calculate:
0.3 0.3
Lift = = = 1.5
0.4 × 0.5 0.2
Answer: Lift = 1.5
Note: Lift ¿ 1 indicates positive correlation between A and B!

26
IIIT Allahabad Data Analytics Tutorial Solutions

2.13 Question 38: Conviction Computation

Problem: confidence(A→B) = 0.8, support(B) = 0.6.
Step 1: Formula:

1 − support(B)
Conviction(A → B) =
1 − confidence(A → B)

Step 2: Calculate:
1 − 0.6 0.4
Conviction = = = 2.0
1 − 0.8 0.2
Answer: Conviction = 2.0
Note: Conviction = 2 means the rule makes errors at half the rate expected if A and
B were independent.

2.14 Question 39: Rule Filtering using Interestingness

Problem: support = 12%, confidence = 85%, lift = 0.9.
Step 1: Compute leverage:

• Leverage = support(A,B) - support(A) × support(B)

• From lift: support(A,B) = lift × support(A) × support(B)

• 0.12 = 0.9 × support(A) × support(B)

• support(A) × support(B) = 0.12 / 0.9 = 0.133

Leverage = 0.12 − 0.133 = −0.013

Step 2: Interpret:

• Lift ¡ 1 → NEGATIVE correlation

• Leverage ¡ 0 → Items occur together LESS than expected

Answer: Leverage = -0.013; Rule is NOT useful (negative correlation)

Note: Despite high confidence (85%), the rule is misleading because items actually
repel each other (lift ¡ 1).

2.15 Question 40: Numerical Interpretation of Association Rules

Problem: support = 0.25, lift = 1.8.
Step 1: Understanding lift:
Observed co-occurrence
Lift =
Expected co-occurrence (if independent)

27
IIIT Allahabad Data Analytics Tutorial Solutions

Step 2: Calculate:

Observed = support(A, B) = 0.25

Observed 0.25
Expected = = ≈ 0.139
Lift 1.8
Step 3: Ratio:
Observed 0.25
= ≈ 1.8 : 1
Expected 0.139
Answer: Observed:Expected = 1.8:1 (items co-occur 1.8× more than if
independent)
Note: This means A and B appear together 80% more often than random chance would
predict!

— END OF SOLUTIONS —

28