Slides 2 Notes

2026-03-27

Here is a conceptual flow of the “Splines and Penalization” topic, focusing on the mathematical foundations
and the key takeaways from the data results:
1. The Need for Non-Linearity
Standard Ordinary Least Squares (OLS) models (y = Xβ + ϵ) fail when data exhibits cyclical patterns (like
the VIX index). A single straight line cannot capture these shifts, leading to poor in-sample fit (pp. 2-3).
While we could break the sample into pieces manually, this risks having too few observations per segment
and introduces discontinuity (jumps) between segments (p. 4).
2. Regression Splines
Regression splines solve these issues by using basis functions to empirically arrive at a flexible function f (X)
(p. 7).
Piecewise Linear Basis
The goal is to fit a “broken line” where segments meet at specific break points called knots.
• The Model: For knots at points {a, b}, we use indicator functions Ia and Ib :

f (xi ) = β0 + β1 xi + β2 (xi − a)Ia + β3 (xi − b)Ib

(p. 9)
• Takeaway: This ensures segments are connected end-to-end, providing a better fit for processes unfolding
over time (p. 11).
Piecewise Cubic & Natural Splines
To achieve greater smoothness (avoiding “kinks” at knots), we use cubic polynomials.
• Continuity Constraints: We constrain the first and second derivatives to be the same on both sides of a
knot (p. 13).
• The Model:
K
X
f (xi ) = β0 + β1 xi + β2 x2i + β3 x3i + θj (xi − xj )3+
j=1

(p. 14)
• Natural Cubic Splines: To prevent instability at the boundaries (where data is sparse), we impose
linearity beyond the boundary knots (p. 17).
3. The Bias-Variance Trade-off
Choosing the “best” model is difficult because a better R2 doesn’t always mean a more instructive model
(p. 20).
• The Concept: MSE consists of two parts: M SE(ŷ) = V ar(ŷ) + Bias(ŷ)2 (p. 27).
• The Trade-off: By shrinking coefficients, we increase bias but decrease variance. We trade accuracy for
stability to achieve a lower overall MSE (pp. 28-29).

1
4. Penalized Regression
Rather than manually tuning knots, we add a mathematical penalty to the loss function (p. 24).
Ridge Regression (L2 Penalty)
• The Model: Minimizes the Sum of Squared Residuals (SSR) plus a penalty on the squared coefficients:
 
Xn X p p
X
β̂ = arg min  (yi − β0 − xij βj )2 + λ βj2 
β
i=1 j=1 j=1

(p. 30)
• Matrix Form: β̂ = (X ⊤ X + λI)−1 X ⊤ y (p. 31).
• Takeaway: As the tuning parameter λ increases, coefficients shrink toward zero but never reach exactly
zero (pp. 34, 38).
LASSO Regression (L1 Penalty)
• The Model: Uses the sum of absolute values as the penalty:
 
Xn Xp p
X
β̂ = arg min  (yi − β0 − xij βj )2 + λ |βj |
β
i=1 j=1 j=1

(p. 35)
• Takeaway: LASSO shrinks coefficients faster than Ridge. Crucially, it can produce exact zero coefficients,
effectively performing variable selection by discarding irrelevant predictors (pp. 37-38).
Comparison of Ridge and LASSO Regression
This table summarizes the core differences you are likely to be tested on, emphasizing the mathematical
impact of their respective penalties.

Feature Ridge Regression (L2 Penalty) LASSO Regression (L1 Penalty)

Pp Pp
Penalty Term λ j=1 βj2 λ j=1
Coefficient Shrinkage Shrinks coefficients toward zero, Shrinks coefficients toward zero
but they never reach exactly zero and can produce exact zeros
(unless λ = ∞) (pp. 34, 38). (pp. 37-38).
Variable Selection No. All variables remain in the Yes. It acts as a selection operator
model, though their impact may by discarding variables with zero
be minimal (p. 38). coefficients (p. 38).
Solution Type Analytical/Linear. Can be solved Non-linear. Requires a quadratic
using the matrix formula: programming or numerical
β̂ = (X ⊤ X + λI)−1 X ⊤ y (p. 31). solution (p. 35).
Rate of Shrinkage Slower shrinkage for the same λ Faster shrinkage for the same λ
(p. 37). (p. 37).
Geometric Shape The constraint region is a The constraint region is a
circle/sphere (β12 + β22 ≤ C), diamond (β1 | + |β2 | ≤ C) .The
making it unlikely to hit the axes OLS error ellipses are very likely
(p. 40). to hit one of these corners first,
which is why LASSO is able to
“zero out” coefficients entirely.

Key Exam Takeaways

2
 • The Tuning Parameter (λ): In both models, if λ = 0, you get the standard OLS result. As λ → ∞, the
coefficients → 0 (p. 35).
• Why use them? To navigate the Bias-Variance Trade-off. We accept a little bias (by shrinking the
“true” OLS coefficients) to significantly reduce variance, leading to a lower overall Mean Squared Error
(MSE) and a more stable model (pp. 28, 31).
• The “Selection” Edge: If your exam asks which method is better for a model with 100 predictors where
only 5 are actually important, the answer is LASSO, because it can zero out the 95 irrelevant ones
(p. 38).
The K + 4 Parameters Rule
When you use a piecewise cubic spline with K interior knots, the total number of coefficients (parameters)
you need to estimate is K + 4 (p. 14). [1] 1. Why K + 4? (The Intuition) Think of it as starting with a
standard cubic polynomial and adding one “extra” piece of flexibility for every knot you add: [2]
• The Base (+4): A single cubic polynomial is f (x) = β0 + β1 x + β2 x2 + β3 x3 . This requires 4 parameters
(p. 14).
• The Knots (+K): Every time you add a knot, you are essentially adding one more “hinge” or “bend” to
the curve. Each interior knot (j = 1 to K) adds 1 additional parameter (θj ). [3, 4, 5]
2. The Mathematical Formula This is the basis expansion formula shown on page 14 of your slides (p. 14):
K
X
f (xi ) = β0 + β1 xi + β2 x2i + β3 x3i + θj (xi − xj )3+
| {z }
j=1
Base Cubic (4 parameters) | {z }
Knot Adjustments (K parameters)

• (xi − xj )3+ : This is a “truncated power basis.” It is equal to (xi − xj )3 if xi > xj , and 0 otherwise.
• Total Count: 4 (base) + K (knots) = K + 4 parameters (p. 14). [4, 6]
3. How Constraints Reduce Parameters
You might wonder why it’s not more. If you had K knots, you’d have K + 1 separate regions. If each region
was a completely independent cubic, you’d need 4 × (K + 1) parameters. [7]
• However, because we force the pieces to be continuous and have continuous 1st and 2nd derivatives at
every knot, we “use up” 3 constraints per knot.
• The Math: 4(K + 1) − 3K = K + 4. [5, 8]
Comparison Note: Natural Cubic Splines
If your exam asks about Natural Cubic Splines, remember that they are more restrictive. Because we force the
ends to be linear (not cubic), we remove 4 degrees of freedom (2 at each end), bringing the total parameters
down from K + 4 to just K. [8, 9]
You’ve hit on the exact reason why Penalized Regression is the logical next step after learning about basic
splines.
Without penalization, you are correct: choosing knots is a “manual” process prone to indiscriminate decision-
making. If you pick too many knots, you overfit (the curve is too “wiggly”); if you pick too few, you underfit
(it’s too “stiff”). [1, 2] Penalized regression mediates this by transforming the problem from “Where do I
put the knots?” to “How much do I value smoothness versus accuracy?” It does this through three specific
mechanisms:
1. Automation via the Smoothing Parameter (λ)
Instead of obsessing over exact knot placement, you place a large number of initial knots (e.g., one at every
data point or a dense grid). The penalized model then uses a tuning parameter λ to decide how much to
“shrink” the influence of those knots. [3, 4, 5]
• If λ is high: The penalty dominates, smoothing out the curve by shrinking coefficients toward linearity.
• If λ is low: The model focuses on fitting every data point, allowing more “wiggles”. [1, 5, 6]

3
2. Statistical Objectivity (The Loss Function)
Penalized regression provides a mathematical “referee” through the objective function:

min [SSR(β) + λ · Penalty(β)]

β

The “indiscriminate” choice of parameters is now governed by cross-validation (GCV) or criteria like AIC/BIC.
These tools automatically select the λ that minimizes predictive error, providing the “proper statistical
guidance” you mentioned was missing. [1, 7, 8, 9]
3. Sparse Selection (LASSO specific)
While Ridge regression (often used in “Smoothing Splines”) keeps all knots but shrinks their impact, LASSO
can set knot coefficients to exactly zero. This effectively “deletes” unnecessary knots from the model,
performing automatic knot selection based only on where the data actually shows a significant change in
trend. [4, 10, 11]
Summary for your exam

Aspect [1, 2, 4, 12] Manual Splines Penalized Splines

Knot Selection Analyst chooses locations Place many; model selects/shrinks
(Subjective) (Objective)
Guidance “Eyeball” fit or trial & error Statistical criteria (GCV, AIC,
BIC)
Risk High risk of overfitting or “sharp” Balanced via the Bias-Variance
angles trade-off (p. 28)

In the context of penalized regression and splines, Elastic Net acts as the final “fail-safe” that addresses the
specific weaknesses of using Ridge or LASSO individually.
It comes into play primarily when your data has highly correlated predictors (multicollinearity), a situation
where LASSO often fails by arbitrarily picking just one variable and discarding others that might be equally
important. [1, 2]
1. The Core Innovation: “The Grouping Effect”
The most important takeaway for your exam is the grouping effect.
• LASSO’s Limitation: If two variables are 95% correlated, LASSO will pick one at random and set the
other to zero.
• Elastic Net’s Solution: It uses its L2 (Ridge) component to ensure that highly correlated variables have
similar coefficient magnitudes, while its L1 (LASSO) component still allows for sparsity. It essentially
says: “If these variables are related and important, keep them all in the model together”. [2, 3, 4, 5, 6,
7]
2. Mathematical Formula
Elastic Net minimizes a loss function that includes both penalties, weighted by a mixing parameter (α) and a
total penalty strength (λ). [2, 8]
  
Xp Xp
β̂ = arg min SSR(β) + λ α |βj | + (1 − α) βj2 
β
j=1 j=1

• α (Mixing Parameter): Determines the “balance.” If α = 1, it is pure LASSO; if α = 0, it is pure Ridge.

• λ (Regularization Strength): Controls the overall amount of shrinkage applied to the model. [7, 9, 10]

4
3. When to Use It (Summary for Exam)

Scenario [1, 8, 11, 12, 13, 14] Recommended Method Why?

p > n (More predictors than Elastic Net LASSO can only select at most n
observations) variables; Elastic Net can select
more.
Multicollinearity Elastic Net It stabilizes the model by
grouping related variables rather
than selecting one randomly.
Sparsity Needed LASSO or Elastic Net Both can set coefficients to exactly
zero, whereas Ridge cannot.

Exam Tip: If a question asks which model is most robust or flexible, the answer is almost always Elastic Net,
as it includes Ridge and LASSO as special cases that can be “discovered” through cross-validation. [9, 15]
To explain the bias-variance tradeoff in a machine learning exam, focus on how model complexity
influences two distinct types of error to determine a model’s ability to generalize to new data. [1, 2]
1. Define the Core Components
A model’s total expected error on unseen data is the sum of three parts: [3, 4]
• Bias (Bias2 ): Error from overly simplistic assumptions (e.g., using a straight line to fit a curve). High
bias leads to underfitting, where the model misses important patterns in the training [Link] measures
the difference between our model’s average prediction and the true value. It represents the error caused
by simplifying a complex real-world problem.: Think of a simple straight OLS line trying to fit cyclical
data (like the VIX). That model has high bias because it’s too rigid to see the pattern. High bias leads
to underfitting.
• Variance: Error from excessive sensitivity to small fluctuations in the training data. High variance
leads to overfitting, where the model “memorizes” random noise as if it were a real [Link] of a
very complex, ‘wiggly’ spline that hits every single data point perfectly. That model has high variance
because if you changed just one data point, the whole curve would shift wildly to chase it.
• Irreducible Error (σ 2 ): Unavoidable noise inherent in the data itself that no model can eliminate. [1, 3,
4, 5]
2. Explain the “Tradeoff”
The tradeoff is a balancing act: as you change model complexity, bias and variance move in opposite directions.
[2, 4]
• Increasing Complexity: (e.g., adding more features or increasing polynomial degree) decreases bias but
increases variance.
• Decreasing Complexity: (e.g., simplifying the model) increases bias but decreases variance. [1, 6]
• The goal of Penalized Regression (Ridge and LASSO) is to find the ’Sweet Spot.’By increasing the
penalty (λ), we intentionally increase the bias (we make the model simpler and flatter). However,
because the coefficients are smaller, we drastically decrease the variance (the model becomes more
stable).
The Result: We are trading a little bit of accuracy for a lot of stability. If the reduction in variance is greater
than the increase in bias, the Total MSE goes down, giving us a more reliable model for the future.
3. Identify Signs of Each Problem In an exam, you may be asked to diagnose these based on error rates:
[7, 8]
• High Bias (Underfitting): High training error AND high test error. The errors are usually similar in
size.

5
 • High Variance (Overfitting): Very low training error but high test error. There is a significant “gap”
between the two. [7, 8, 9]
4. Summary Table for Quick Reference

Feature [1, 7, 8, 9, 10, 11] Underfitting (High Bias) Overfitting (High Variance)
Complexity Too simple Too complex
Training Error High Low
Test Error High High
Gap (Train vs Test) Small Large
Fixes Increase complexity, add features Regularization, more data,
simplify model