CORRELATION AND SIMPLE
REGRESSION INFERENCE
DAY-15
Correlation Analysis
Linear Regression
Model Diagnostics
TODAY’S Accuracy Assessment
SESSION
Pearson Correlation
The Pearson correlation coefficient measures the strength of linear relationships between two
continuous variables:
Key Properties:
Range: -1 to +1
Sensitive to outliers
Measures only linear association
Our Example: With r = 0.80 between study hours and
exam scores, we observe a strong positive linear relationship.
Confidence Intervals for Correlation
Due to the non-normal sampling distribution of r, we use Fisher's z-transformation:
Example: For r = 0.80, n = 30: z' = 1.099, SE = 0.192
95% CI for r: approximately [0.62, 0.90]
Spearman Rank Correlation
Spearman's ρ measures monotonic relationships
by correlating the ranks of X and Y values rather
than the raw data.
Calculation Process:
[Link] both X and Y variables
[Link] Pearson correlation on ranks
When to Use Spearman:
Monotonic but non-linear relationships
Ordinal data or heavy outliers
Robust alternative to Pearson
Choosing Between Pearson and Spearman
Golden Rule: Always visualise your data first! If you observe curvature or influential outliers, consider
Spearman or data transformation before relying on Pearson.
Visual Pitfalls in Correlation Analysis
Correlation coefficients can be misleading without proper visualisation. Here are common traps to avoid:
Simpson's Paradox
A fascinating statistical phenomenon where the relationship between variables reverses when data is
aggregated across groups.
Protection Strategy: Always stratify your analyses by potential confounding variables and compare
within-group patterns to overall trends before drawing conclusions.
Simple Linear Regression Model
Moving from correlation to prediction, we fit the linear model:
Interpreting Regression Coefficients
Using the same data but different sample sizes dramatically illustrates how n affects confidence
interval width.
Practical Meaning: Our model suggests that each additional hour of study is associated with a 4-point
increase in exam score, on average. Students who don't study at all are predicted to score 50 points.
Caution: The intercept is only meaningful when X=0 falls within your data range.
Extrapolation beyond observed values requires careful justification!
Breaking Down Residual Components
R² in Simple Regression
In simple linear regression, the coefficient of determination equals the squared correlation
coefficient:
Practical Example:
SST = (n−1)sy² = 2900
Correlation r = 0.80
Therefore: R² = 0.64
This means 64% of the variation in Y is explained by the linear relationship with X.
Computing Standard Error Example
Slope Standard Error Formula
The standard error of the slope estimate provides the foundation for hypothesis testing and
confidence intervals.
Slope Standard Error Formula
The standard error of the slope estimate provides the foundation for hypothesis testing and
confidence intervals.
Slope Inference Example
Calculations
Standard Error:
SE(b₁) = 5 × √(0.36/28) = 0.568
t-statistic:
t = 4/0.568 ≈ 7.04
95% Confidence Interval:
4.00 ± 2.048 × 0.568
[2.84, 5.16]
Mean Response vs Prediction
Residual Diagnostics
Bootstrap Confidence Intervals for RMSE
Model-Agnostic Approach
1. Sample rows with replacement (size n)
2. Fit regression on bootstrap sample
3. Compute RMSE on predictions
4. Repeat 2000+ times
5. Use 2.5th-97.5th percentiles for 95% CI
Example Result: RMSE CI ≈ [4.9, 8.2]
Compare with traditional χ²-based intervals for robust inference.
Micro-Lab A: Computing Correlations
Expected Correlation Results
Pearson Correlation
r ≈ 0.80 (p < 1×10⁻⁶)
95% CI: [0.62, 0.90]
Strong positive linear relationship with high statistical significance
Pearson Correlation
r ≈ 0.80 (p < 1×10⁻⁶)
95% CI: [0.62, 0.90]
Strong positive linear relationship with high statistical significance
Micro-Lab B: Simple Linear Regression
Regression Results Example
Regression Results Example
Micro-Lab C: Bootstrap RMSE Analysis
Bootstrap Procedure
Generate B = 2000 bootstrap resamples
Fit model and compute RMSE for each
Calculate percentile confidence interval
Compare with χ²-based parametric CI
Expected Result: RMSE ≈ 6.1; Bootstrap 95% CI [4.9, 8.2]; χ² CI [4.8, 8.0]
Reporting Template & Common Pitfalls
Scores increase by 4.0 points per study hour (95% CI 2.84–5.16, p < 1×10⁻⁶). R² = 0.64.
Diagnostics show approximate linearity and constant variance; one high-leverage point checked, no undue influence.
Typical error (RMSE) ≈ 6.1 with 95% CI ~ [4.8, 8.0].
Problem Set 3: Weekly Assignment
ANY
DOUBTS?