Statistical Experiments and

Significance Testing, Regression and

Prediction Module -4

- Multi – Arm Bandit Algorithm.

- Power and Sample size.
- Factor Variables in Regression.
- Interpreting the Regression Equation.
- Regression Diagnostics.
- Polynomial and Spline Regression

- Textbook : Chapter 3 and Chapter 4

Multi-Arm Bandit Algorithm
The Multi-arm Bandit Algorithm is an alternative approach to testing, particularly for web testing, that
facilitates explicit optimization and rapid decision making compared to traditional statistical
experiment design.
Key Concepts
• Traditional A/B Testing vs. Bandits: A traditional A/B test is conducted, and results are acted
upon only after the experiment concludes. Bandit algorithms, conversely, track outcomes
continuously, allowing researchers to begin leveraging favourable results earlier in the process.
• Analogy: The algorithm is conceptualized using an imaginary slot machine (a multi-arm bandit)
where each arm has a different payoff. This machine serves as an analogy for a multitreatment
experiment.
• Goal: The primary objective is to maximize payoff by identifying and settling on the best "arm"
(treatment) sooner rather than later.
Key Terms for Multi-Arm Bandits
• Multi-arm bandit: An imaginary slot machine with multiple arms for the
customer, serving as an analogy for a multitreatment experiment.
• Arm: Represents a treatment in an experiment (e.g., "headline A in a web test").
• Win: The experimental equivalent of a win at the slot machine (e.g., "customer
clicks on the link").

• Summary : The Multi-Arm Bandit Algorithm is a key concept in practical

statistics for data scientists that deals with the exploration–exploitation
trade-off — a balance between trying out new options (exploration) and using
the best-known option (exploitation).
Concept

Imagine a row of slot machines (“one-armed bandits”) in a casino.

Each machine gives a different — but unknown — probability of payout.

Your goal: maximize your total reward over time.

But you don’t know which machine is best — so you must:

➢ Explore: Try each machine to learn its payout rate.

➢ Exploit: Play the machine that seems best so far.

This dilemma is exactly what the Multi-Arm Bandit (MAB) problem models.
Definition

You have k arms (options).

Each arm i has an unknown reward distribution with an expected value μᵢ.
At each round t, you:
1. Choose an arm Aₜ to pull.
2. Receive a reward Rₜ.
3. Update your estimates to maximize the long-term reward.

Applications in Data Science

1. A/BTesting Optimization — dynamically assign users to webpage versions (ads, layouts, etc.) while
reducing loss from poor performers.
2. Online Advertising — choose which ad to show for the highest click-through rate (CTR).
3. Recommendation Systems — decide which item to recommend to increase engagement.
4. Clinical Trials — allocate patients to better-performing treatments faster.
Example (A/B Testing Context)
Suppose:
• Ad A: Click rate = 5%
• Ad B: Click rate = 4%
• Ad C: Unknown
Instead of testing each ad for a fixed period, MAB algorithms adaptively allocate more traffic to the better-performing
ad — improving conversion rate while still learning.
Power and Sample Size
Sample size calculations are performed to determine if a hypothesis test will detect a real difference
between treatments (Power).
Key Terms for Power and Sample Size
• Power: The probability of detecting a given effect size with a given sample size. It is the
probability of detecting a specified effect size with specified sample characteristics (size and
variability).
• Effect size: The minimum size of the effect you hope to detect in a statistical test (e.g., "a 20%
improvement in click rates").
• Significance level: The statistical significance level at which the test will be conducted.
Key Ideas for Power Calculation
To calculate power or required sample size, you must specify four parts:
1. Sample size.
2. Effect size you want to detect.
3. Significance level (alpha) at which the test will be conducted.
4. Power (the required probability of detection).
Intuitive Alternative Approach (Steps)
An intuitive alternative approach to sample size estimation involves bootstrapping and
permutation testing:
1. Start with a hypothetical data set representing the best guess about the resulting data
(e.g., a box representing a .200 hitter).
2. Create a second sample by adding the desired effect size (e.g., a box reflecting a 33%
rate if the desired effect is a 10% boost).
3. Draw a bootstrap sample of size n from each box.
4. Conduct a permutation or formula-based hypothesis test and record if the difference
between the bootstrap samples is statistically significant.
5. Repeat the preceding two steps many times; the frequency of significant results is the
estimated power.
Example: Sample Size Calculation
If current click-through rates are 1.1% and a researcher seeks a 50% boost (to 1.65%).
• An initial trial might yield Box A: 19 ones and Box B: 34 ones from 2,000 draws each.
The difference (34-19) is not statistically significant.
• Using statistical software to achieve 80% power (power=0.8) for an effect size resulting
from a 50% boost, the required sample size is calculated as approximately 116,602.393
impressions. If seeking a 50% boost (0.0123 to 0.011 effect size), a sample size of almost
120,000 impressions is required.
These are very important regression concepts
in “Practical Statistics for Data Scientists.”

• Factor Variables in Regression

→ Dummy Variable Representation
→ Factor Variables with Many Levels
→ Ordered Factor Variables
Regression and Prediction
• Regression is a supervised learning technique used to model the
relationship between a dependent variable (Y) and one or more
independent variables (X).

The goal is to predict Y for given X values using an estimated model:

Factor Variables in Regression
A factor variable (also called a categorical variable) represents data
that belongs to distinct categories, rather than numeric values.
Examples:
• Gender: Male / Female
• Education: High School / Bachelor’s / Master’s
• Region: North / South / East / West
These cannot be directly used in regression because regression models
require numerical input.
So, we convert these categorical (factor) variables into numeric form
using dummy variables (or one-hot encoding).
 Dummy Variables Representation
A dummy variable is a binary variable (0 or 1) used to represent categories.
If a factor variable has k levels, we use k − 1 dummy variables to avoid multicollinearity
(known as the dummy variable trap).
Example: Education Level

Education Dummy_Bachelor Dummy_Master

High School 0 0
Bachelor’s 1 0
Master’s 0 1

Here:
• “High School” is the reference category (baseline).
• The regression intercept (𝛽0 )corresponds to the mean outcome for the reference group.
• Coefficients of dummy variables (𝛽1 , 𝛽2 )show the difference in outcome compared to the
reference group
Regression Model Example

𝑆𝑎𝑙𝑎𝑟𝑦 = 𝛽0 + 𝛽1 (Bachelor) + 𝛽2 (Master) + 𝜀

Interpretation:

• 𝛽0 :Mean salary for High School graduates

• 𝛽1 :Difference in mean salary between Bachelor’s and High School

• 𝛽2 :Difference in mean salary between Master’s and High School

import pandas as pd
import [Link] as sm

data = [Link]({
'education': ['High School', 'Bachelor', 'Master', 'High School', 'Bachelor'],
'salary': [30, 45, 60, 35, 50]
})

# Convert categorical variable to dummy variables

edu_dummies = pd.get_dummies(data['education'], drop_first=True)
X = sm.add_constant(edu_dummies)
model = [Link](data['salary'], X).fit()
print([Link]())

This model automatically treats “High School” as the reference level.

 Factor Variables with Many Levels
When a categorical variable has many categories, dummy encoding can create:
• Too many new variables → high-dimensional data
• Risk of overfitting
• Hard-to-interpret models
Example: Region
• If Region has 20 categories, you’ll get 19 dummy variables.
• This makes the model complex and increases computation.
Possible Solutions
• Combine categories – Merge rare categories (e.g., group small regions together).
• Regularization – Use Lasso or Ridge regression to shrink less important coefficients.
• Hierarchical modeling – Treat levels as random effects (mixed models).
• Target / Mean encoding – Replace categories with average outcome values (used in ML).
Ordered Factor Variables
Definition
Some categorical variables have a natural order — but not a numeric
distance between levels.
Examples:
• Education Level: High School < Bachelor < Master < PhD
• Satisfaction: Low < Medium < High
• These are called ordered factor variables (or ordinal variables).

For example, the variable

BldgGrade is an ordered factor variable. Several of the types of grades are
shown in Table 4-1. While the grades have specific meaning, the numeric value is
ordered from low to high, corresponding to higher-grade homes.
Interpreting the Regression Equation:
- A regression equation shows how each predictor affects the
outcome, controlling for other variables.
 Correlated Predictors
• In multiple regression, the predictor variables are often correlated with each other.
• When predictors (independent variables) are correlated with each other — called
multicollinearity — interpreting individual regression coefficients becomes difficult or
misleading.
Example
You’re predicting house price using:
• size (in square feet)
• number_of_rooms
Naturally, larger houses tend to have more rooms, so size and rooms are positively
correlated.
Because both variables explain similar variation in price, the regression model struggles
to determine which predictor truly explains the change.
Correlated Predictors:
When two or more predictor variables (independent variables) in a regression model are
highly correlated, it becomes difficult for the model to determine how much each variable
individually contributes to predicting the outcome.
This situation is called correlation among predictors (or collinearity).

Collinearity Vs Multicollinearity
These are the two most overlapping concepts from statistics. Collinearity is when there is
a linear relationship or association between two variables or features. The term
multicollinearity is formed for the linear association between two or more variables.
What Happens When Predictors Are Correlated ?

Effect Description
Small changes in data can drastically
Unstable Coefficients
change coefficients.
Coefficients may get negative even when
Wrong Signs
true effect is positive.
p-values become large → insignificant
High Standard Errors
coefficients.
You can’t easily say, “Holding others
Hard to Interpret
constant…” when variables are related.
# Example in Python

import pandas as pd
import numpy as np
import [Link] as sm

# Create correlated predictors

[Link](0)
size = [Link](1500, 300, 100)
rooms = size / 300 + [Link](0, 0.5, 100) # correlated with size
price = 50000 + 120*size + 10000*rooms + [Link](0, 10000, 100)

data = [Link]({'price': price, 'size': size, 'rooms': rooms})

# Check correlation
print("Correlation between size and rooms:", [Link]([Link](), 2))

# Fit regression model

X = sm.add_constant(data[['size', 'rooms']])
model = [Link](data['price'], X).fit()
print([Link]())
OUTPUT

Interpretation
• Both variables are correlated (r ≈ 0.96).
• Neither size nor rooms has a significant p-value, even though both clearly affect price.
• The model struggles to separate their individual effects.
How to Handle Correlated Predictors ?

Strategy Explanation
If two predictors measure the same thing (e.g., “size”
Remove one of the correlated variables
and “rooms”), drop one.

Combine predictors Create a composite variable (e.g., average or ratio).

Use Ridge or Lasso regression to shrink correlated

Regularization methods
coefficients.

Transform correlated predictors into uncorrelated

Principal Component Regression (PCR)
components.
Correlated Predictors and Multicollinearity
• Impact of Correlation: High correlation among predictors makes coefficient interpretation difficult. For example, in the
housing data, the coefficient for Bedrooms may be negative because larger homes often have more bedrooms and
larger SqFtTotLiving—if SqFtTotLiving already captures the overall size, Bedrooms only accounts for the residual
variation, which might be associated with smaller living area relative to the number of rooms.
• Multicollinearity: An extreme case of correlation indicating redundancy among predictor variables. It happens if a
variable is included multiple times, if P-1 dummies are created from a factor variable (which is generally fine), or if two
variables are nearly perfectly correlated.

Multicollinearity is an extreme form of correlation among predictors — where variables are so highly correlated that
they provide redundant information.

Causes of Multicollinearity
1. Including the same variable twice (e.g., total area and area in sq. meters).
2. Creating too many dummy variables from a factor variable — e.g., using all P dummies instead of P–1.
3. Nearly perfect correlation between two continuous variables (e.g., Celsius and Fahrenheit temperature).

Effects:
• Coefficients become unstable or undefined.
• The regression matrix 𝑋 ′ 𝑋becomes nearly singular (non-invertible).
• Model may still predict well, but coefficients are meaningless.
Confounding Variables
A confounding variable is an important predictor that has been omitted from the regression model.
Because it’s missing, the model misattributes its effect to other variables, making the estimated coefficients
misleading or biased.

Definition
A confounding variable is a variable that influences both the predictor and the response but is not included
in the regression model.
When this happens, the estimated effect of included predictors becomes distorted — this is called omitted
variable bias.

Example: The Housing Price Case (King County Data)

Let’s consider predicting house price using only:
• SqFtTotLiving (size of the house)
• Bedrooms (number of bedrooms)

However, an important variable — location (Zip Code) — is omitted.

In real life, location strongly affects price — houses in premium areas cost more for the same size.
What Happens When Location Is Omitted

When ZipCode (location) is not included, the model may show:

• A negative coefficient for Bedrooms — even though larger houses (with more
bedrooms) should cost more.
Why?
Because:
• High-priced areas (urban zones) often have smaller but more expensive houses.
• Low-priced areas (rural zones) often have larger, cheaper houses.

The model wrongly interprets this location-driven pattern as if more bedrooms reduce
price.

That’s a confounding effect — the omitted “location” variable distorts the relationship
between Bedrooms and Price.
Main Effects and Interactions

Main Effect
A main effect represents the individual (direct) relationship between a predictor and the
response variable, assuming all other predictors stay constant.
Example:
“For every extra square foot of living area, price increases by $118 )on average).”

Interaction Effect
An interaction occurs when the effect of one predictor depends on the level of another
predictor.

In other words:
The relationship between predictor A and the outcome changes depending on predictor B.
Regression Diagnostics
After fitting a regression model, diagnostics are used to check whether the model
assumptions are valid and whether the predictions are reliable.

A good model should satisfy:

1. Linearity — The relationship between predictors and the outcome should be linear.
2. Independence of errors — Residuals should not be correlated.
3. Homoscedasticity — The variance of residuals should be constant.
4. Normality of residuals — Residuals should be approximately normally distributed.
5. No multicollinearity — Predictors should not be highly correlated.
Regression Diagnostics
Regression diagnostics assess how well a model fits the data, primarily relying on
analysing residuals.
Defination:
A residual is the difference between the actual (observed) value and the predicted
value from your regression model.
Residual = 𝑌actual − 𝑌෠ predicted
where:
• 𝑌actual = the true observed value
• ෡𝑌predicted = the value predicted by your regression model

Meaning:

Residuals measure the error or leftover variation that your model couldn’t explain.
• If residual = 0 → model prediction is perfect.
• If residual is large (positive or negative) → model is off by a large margin.
So, they represent the "unexplained" part of your data.
Outliers in Regression

An outlier is a data point that does not fit the general pattern of the relationship
between predictors (X) and the response variable (Y).

In regression analysis, an outlier is an observation whose actual Y value is very far from
the predicted Y (ŷ) — that is, the model didn’t predict it well.

What Makes an Outlier Important?

Outliers can:
• Indicate data entry or measurement errors.
• Reveal unusual or exceptional cases (e.g., a luxury or damaged property).
• Influence regression results, sometimes strongly affecting coefficients.

Hence, detecting and understanding them is crucial before drawing conclusions.

Example (from the book: King County housing data)
• The book shows a property deed (Figure 4-4) representing the largest negative residual.
• That house sold for much less than the model predicted.
• Hence, its residual was large and negative — a downward outlier.
Such a property might have:
• Legal issues (e.g., foreclosure or “statutory warranty deed”), or
• Poor condition not captured by other predictors.
Heteroskedasticity, Non-Normality, and Correlated Errors
• Heteroskedasticity means the variance of residuals(Difference between actual
and predicted values) is not constant across the range of fitted (predicted) values.
• Non-normality occurs when residuals don’t follow a normal distribution —
they may be skewed or have heavy tails.
• “Heteroskedasticity indicates non-constant error variance and often signals
missing model structure. Non-normality of residuals suggests outliers or incorrect
functional form.
Partial residual plots are a key diagnostic for detecting nonlinearity between
predictors and the outcome.”
• Correlated errors occur when the residuals (errors) in a regression model are not
independent — that is, the error for one observation is related to the error for
another. – result in “misleading inferences.”
Polynomial and Spline Regression :
These techniques are used to fit relationships where the response and predictor are
not linearly related.
What Is Polynomial Regression?

Polynomial Regression is an extension of linear regression that allows for curved

relationships between the predictor variable and the outcome.

Instead of using just 𝑋as the predictor, we include higher powers of X:

𝑌 = 𝛽0 + 𝛽1 𝑋 + 𝛽2 𝑋 2 + 𝛽3 𝑋 3 + ⋯ + 𝜖

Why Use It?

A simple linear model (𝑌 = 𝑏0 + 𝑏1 𝑋 )assumes a straight-line relationship)).
But in many real-world datasets — such as house price vs. square footage, age vs.
income, speed vs. fuel efficiency — the relationship is curved.

Polynomial regression helps to:

• Capture nonlinear trends while keeping the model linear in coefficients.
• Improve model fit when residual plots show curvature.
Example (from the book)
• Predictor: SqFtTotLiving (square feet of house)
• Response: SalePrice
A linear model might underpredict for small houses (<1000 sq ft) and overpredict for mid-sized homes (2000–
3000 sq ft).
This curvature can be corrected by adding 𝑋 2 (square footage squared) as a term.
So, the new model is:
𝑺𝒂𝒍𝒆𝑷𝒓𝒊𝒄𝒆 = 𝜷𝟎 + 𝜷𝟏 𝑺𝒒𝑭𝒕𝑻𝒐𝒕𝑳𝒊𝒗𝒊𝒏𝒈 + 𝜷𝟐 ൫𝑺𝒒𝑭𝒕𝑻𝒐𝒕𝑳𝒊𝒗𝒊𝒏𝒈)𝟐 + 𝝐

Figure 4-10 (Explained)

In the partial residual plot:
• The solid line (polynomial regression fit) follows the
dashed smoothed curve (nonparametric smoother)
much more closely.
 Splines
Splines are a flexible and smooth way to fit nonlinear relationships in regression
without resorting to high-degree global polynomials.
Instead of fitting one big polynomial across the whole dataset, splines fit different
polynomials in separate sections of the data and join them smoothly at certain points
called knots.
Mathematically,
A spline is a piecewise continuous polynomial that is smooth at the “knots.”
* Knots divide the range of 𝑋 into intervals.
* Within each interval, the model fits a polynomial (usually cubic).
* At each knot, the pieces are joined smoothly — that is, their value and slope match.
Generalized Additive Models (GAMs)
A Generalized Additive Model (GAM) is an extension of the linear regression model
that allows for nonlinear relationships between predictors and the outcome — but
keeps additivity (each predictor has its own smooth effect).
In the plot:
• The gray scatter points show actual data.
• The dark line )solid) shows the GAM’s
smooth fitted curve.
• If we added a dashed line for a simple linear
regression, we’d see the GAM fits curvature
better — but without sharp bends
(i.e., avoiding overfitting).
Practical Remark

In Practical Statistics for Data Scientists, these topics emphasize that:

• Model diagnostics ensure validity and trust in regression results.

• Polynomial and spline models enhance predictive performance when

relationships are nonlinear.

• Visualization (residual plots, partial residuals, fitted curves) is crucial

to interpret and validate model behavior.
 Summary

Concept Goal Example Tool

Optimize decisions under

Multi-Arm Bandit ε-Greedy, Thompson Sampling
uncertainty

Determine data needed for reliable

Power & Sample Size [Link]
tests

Factor Variables Encode categories for regression pd.get_dummies()

Interpreting Regression Understand variable effects [Link]

Diagnostics Validate model assumptions Residual plots, Q–Q plot

np.column_stack(),
Polynomial/Spline Regression Model nonlinear relationships
[Link]()