Splines regressions

September 28, 2017

• Reading:
Svante Wold (1974). Spline functions in data analysis. Technimetrics 16
(1), 1-11.

1
 Moving Beyond Linearity

• The linearity assumption is good in many machine learning problems.

• However, there are other methods that offer a lot of flexibility,

without losing the ease and interpretability of linear models:
• Polynomial regression
• Step functions
• Regression and smoothing splines
• Local regression
• Generalized additive models (GAMs)

2
Polynomial Regression

• Replace the standard linear model with a polynomial

function:

3
Polynomial Regression

• For large enough degree d, a polynomial regression allows us

to produce an extremely non-linear curve.

• We do this by creating new variables X1 = X, X2 = X2, etc. and

then treat as multiple OLS linear regression.

• In general, we are not really interested in the coefficients,

but instead the fitted function values at any value x0:

4
Polynomial Regression

መ 0 ) is a linear function of the 𝛽መ𝑗 , we can get a simple

• Since 𝑓(𝑥
expression for pointwise variances Var[𝑓መ 𝑥0 ] at any value
of 𝑥0 .

• It is unusual to use d greater than 3 or 4 because for large

values of d, the polynomial curve can become overly flexible
and can take on very strange shapes.

• Note that we can also use cross-validation to choose d.

5
Step Functions

• Using polynomial functions of the features as predictor in a

linear model imposes a global structure on the non-linear
function of X.

• To avoid imposing such a global structure, we can create

transformations of a variable by cutting the variable into
distinct regions.

• In particular, we use step functions to break the range of X

into bins, where we fit a different constant in each bin.

6
Step Functions

• This amounts to converting a continuous variable into an ordered

categorical variable.

• In greater detail, we create cut points (or knots) c1, c2,….,cK in the
range of X and then construct K + 1 new variables:

where I(.) is an indicator function that returns a 1 if the condition is

true and 0 otherwise.

7
Step Functions

• Note that for any value of X, C0(X) + C1(X) + … + CK(X) = 1,

since X must be exactly in one of the K + 1 intervals.

• We then use OLS estimation to fit a linear model using these

K + 1 new variables:

• For a given value of X, at most one of C1, C2,…,CK can be non-

zero.

• 𝛽𝑗 represents the average increase in the response for X in

𝑐𝑗 ≤ 𝑋 ≤ 𝑐𝑗+1 relative to X < c1.

8
Step Functions

• Unless there are natural breakpoints in the predictors, piece-

wise constant functions can miss the action.

9
 Basis Functions
• Polynomial and piece-wise constant regression models are special
cases of a basis function approach.

• The idea is to have at hand a family of functions or transformations

that can be applied to a variable X: b1(X),….,bK(X)

• Instead of fitting a linear model in X, we fit the following model:

• Note that the basis functions b1(.),….,bK(.) are fixed and known.

10
Basis Functions

• For polynomial regression, the basis functions are 𝑏𝑗 𝑥𝑖 =

𝑗
𝑥𝑖

• For piece-wise constant functions, the basis functions are

𝑏𝑗 𝑥𝑖 = 𝐼(𝑐𝑗 ≤ 𝑥𝑖 ≤ 𝑐𝑗+1 )

• Note that we can use OLS to estimate the unknown

regression coefficients.

• Thus, all of the inference tools for linear models (standard

errors, F-statistics, etc.) are available in this setting.

11
Regression Splines

• Regression splines are a flexible class of basis functions that

extend upon the polynomial regressions and piece-wise
constant regression approaches.

• They involve dividing the range of X into K distinct regions;

within each region, a polynomial function is fit to the data.

• These polynomials are constrained so that they join

smoothly at the region boundaries (or knots).

• Provided that the interval is divided into enough regions, this

can produce an extremely flexible it.

12
Piecewise Polynomials

• Instead of fitting a high-degree polynomial over the entire

range of X, piece-wise polynomial regression involves fitting
separate low-degree polynomials over different regions of X.

• Here, the beta coefficients differ in different parts of the

range of X; the points where the coefficients change are
called knots.

• Example: A piecewise cubic polynomial with a single knot at

a point c takes the following form:

13
Piecewise Polynomials (cont.)

• Each of the polynomial functions can be fit using OLS applied

to simple functions of the original predictor.

• Using more knots leads to a more flexible piecewise

polynomial.

• If general, if we place K different knots through the range of

X, then we end up fitting K + 1 different polynomials.

• It is better to add constraints to the polynomials (e.g.

continuity).

• Splines have the maximum amount of continuity

14
Piecewise Polynomials (cont.)

15
Piecewise Polynomials (cont.)

• Each constraint that we impose effectively frees up one

degree of freedom, by reducing the complexity of the
resulting piecewise polynomial fit.

• The general definition of a degree-d spline is that it is a

piecewise degree-d polynomial, with continuity in
derivatives up to degree d – 1 at each knot.

• Thus, a linear spline is obtained by fitting a line in each

region of the predictor space defined by the knots, requiring
continuity at each knot.

16
Linear Splines

17
Cubic Splines

18
Natural Splines

• A natural spline is a regression spline with additional

boundary constraints.

• The function is required to be linear at the boundary (in the

region where X is smaller than the smallest knot, or larger
than the largest knot).

• This additional constraint means that natural splines

generally produce more stable estimates at the boundaries.

19
Natural Splines (cont.)

• A natural cubic spline extrapolates linearly beyond the

boundary knots. This adds 4 = 2 x 2 extra constraints, and
allows us to put more internal knots for the same degrees of
freedom as a regular cubic spline.

20
Choosing the Location of Knots

• The regression spline is most flexible in regions that contain

a lot of knots, because in those regions the polynomial
coefficients can change rapidly.

• One option is to place more knots in places where we feel

the function might vary most rapidly, and to place fewer
knots where it seems more stable.

• In practice, it is common to place knots in a uniform fashion.

For example, one strategy is to decide K, the number of
knots, and then place them at appropriate quantiles of the
observed X.

21
Choosing the Number of Knots

• One option is to try out different numbers of knots and see

which produces the best looking curve.

• However, a more objective approach is to use cross-

validation.

• The procedure is repeated for different number of knots K;

then the value of K giving the smallest RSS is chosen.

• Splines allow us to place more knots, and hence flexibility,

over regions where the function f seems to be changing
rapidly, and fewer knots where f appears more stable.

22
 Smoothing Splines
• We create regression splines by specifying a set of knots, producing
a sequence of basis functions, and then use OLS to estimate the
spline coefficients.

• What we really want is a function g that makes RSS small and

smooth. Thus, consider the following criterion for fitting a smooth
function g(x) to some data (known as a smoothing spline):

where λ is a nonnegative tuning parameter.

23
 Smoothing splines
• The first term is a loss function (RSS), which tries to make g(x)
match the data at each xi.

• The second term is a roughness penalty and controls how wiggly

g(x) is; this is modulated by the tuning parameter λ.

• The larger the value of λ, the smoother g will be. The smaller the
value of λ, the more wiggly the function.

• As λ  ∞, the function g(x) becomes linear.

24
 Smoothing Splines (cont.)
• It turns out that the solution is a natural cubic spline, with a knot at
every unique value of xi.

• The tuning parameter λ controls the level of roughness (i.e. the

effective degrees of freedom).

• Smoothing splines avoid the knot-selection issue, leaving a single λ

to be chosen.

• The vector of n fitted values (for a particular choice of λ) can be

ෝ 𝜆 = 𝑺𝜆 𝒚, where 𝑺𝜆 is the n x n smoother matrix.
written as 𝒈

25
Smoothing Splines

• The effective degrees of freedom are trace(𝑺𝜆 ), which

equals:

which is the sum of the diagonal elements of the matrix 𝑺𝜆 .

• We use LOOCV to find λ. The LOOCV error is given as:

26
Smoothing Splines (cont.)

27
Local Regression

• Local regression is a different approach for fitting flexible

non-linear functions, which involves computing the fit at a
target point x0 using only the nearby training observations.

• It is a memory-based procedure because we need all the

training data each time we wish to compute a prediction.

• The span plays a role like that of the tuning parameter λ in

smoothing splines; it controls the flexibility of the non-linear
fit.

28
Local Regression (cont.)

• The smaller the value of the span s, the more local and
wiggly will be our fit.

• A very large value of s will lead to a global fit to the data

using all of the training observations.

• We can use cross-validation to choose s or specify it directly.

• Another choice to be made includes how to define the

weighting function K, and whether to fit a linear, constant, or
quadratic regression.
29
Local Regression (cont.)

30
Local Regression (cont.)

• With a sliding weight function, we fit separate linear fits over

the range of X by weighted least squares.

31
Generalized Additive Models

• Generalized additive models (GAMs) allow for flexible

nonlinearities in several variables, but retains the additive
structure of linear models.

• GAMs can be applied with both quantitative and qualitative

responses.

• In particular, we replace each linear component of the

multiple linear regression model with a (smooth) non-linear
function.

32
 Generalized Additive Models (cont.)

• It is called an additive model because we calculate a separate fj for

each Xj, and then add together all of their contributions.

• We can use any of the previously discussed methods (smoothing

splines, local regression, polynomial regression, etc.) as building
blocks for fitting an additive model.

33
Generalized Additive Models (cont.)

• GAMs allow us to fit a non-linear fj to each Xj, so that we can

automatically model non-linear relationships that standard
linear regression will miss.

• This means that we do not need to manually try out many

different transformations on each variable individually.

• The non-linear fits can potentially make more accurate

predictions for the response Y.

34
Generalized Additive Models

• Because the model is additive, we can still examine the

effect of each Xj on Y individually while holding all of the
other variables fixed.

• Thus, if we are interested in inference, GAMs provide a

useful representation.

• The smoothness of the function fj for the variable Xj can be

summarized via degrees of freedom.

35
Generalized Additive Models

• The main limitation of GAMs is that the model is restricted

to be additive.

• With many variables, important interactions can be missed.

However, we can manually add interaction terms to the
GAM model by including additional predictors of the form Xj
x Xk.

• Although we have not yet covered classification problems,

note that GAMs can also be used in situations where Y is
qualitative.

36
 Summary
• Polynomial regression, step functions, regression splines, smoothing
splines, local regression, and generalized additive models.

37