Chapter 5: Regression
Methods
Function fitting
One of the most commonly used data science techniques—fitting data
with functions or function fitting
Function fitting - Predict the value (or class) of a dependent attribute y,
by combining the predictor attributes X into a function, y = f(X).
Most common ones are
linear regression for numeric prediction
logistic regression for classification
Regression
“Regress” - To return to a previous, usually worse or less
developed state.
or
To have a tendency to approach or go back to a statistical mean.
All the samples regressed toward a population mean. Therefore,
this trend was called regression by Galton
5.1 LINEAR REGRESSION
One of the oldest data science methodologies
Come up with a function that explains and predicts the value of the
target variable when given the values of the predictor variables
How it Works
If one would like to know the effect of the number of rooms in a house
(predictor) on its median sale price (target) - fig 5.1
Each data point on the chart corresponds to a house
On average, increasing the number of rooms tends to also increase
median price
This general statement can be captured by drawing a straight line
through the data
The problem in linear regression is, therefore, finding a line (or a
curve) that best explains this tendency
A Simple Regression Model
If there are two predictors, then the problem is to find a surface (in a 3-D space).
With more than two predictors, visualization becomes difficult and one has to
revert to a general statement where the dependent variables are expressed as a
linear combination of independent variables:
Consider the problem with one predictor. Clearly, one can fit an infinite number of
straight lines through a given set of points such as the ones shown in Fig. 5.1
Which one is the best?
A metric is needed, one that helps quantify the different straight line fits through
the data.
Once this metric is found, then selecting the best line becomes a matter of finding
the optimum value for this quantity.
Error function
A commonly used metric is the concept of an error function.
Suppose one fits a straight line through the data. In a single predictor
case, the predicted value, ŷ, for a value of x that exists in the dataset is
then given by eqn (5.2):
Error = actual target value minus predicted target value (5.3):
Aggregate error
Eqn. (5.3) defines the error at a single location (x, y) in the dataset.
Aggregate error: Compute the error for all existing points
Some errors will be positive, and others will be negative.
The difference can be squared to eliminate the sign bias and an
average error for a given fit can be calculated as (5.4):
where n = no. of points in the dataset
Let J = total squared error
For a given dataset, the best combination of (b0, b1) can then be
found, which minimizes the total error, e.
This is a classical minimization problem, solved by methods of
calculus
Values of b can be found, which minimize the total error J.
Specifically, one can take partial derivatives of J with respect to b1 and
b0 and set them equal to zero.
Chain rule of differential calculus gives us:
Partial derivative w.r.t. b0: