Least-Squares Method

“Choose as a “best-fitting” line the line that minimizes the sum of squared deviations of the
observed values from those predicted.”

n n n
i.e. SSE =  e   ( y  yˆ )   [ y  ( ˆ
i 1
i
2

i 1
i i
2

i 1
i 0  ˆ1 xi )]2

-to find values of ̂ 0 and ̂1 so that those errors are the smallest (i.e. we have shortest distances
between actual y values and the points on the regression line), we need to differentiate SSE
w.r.t. ̂ 0 and ̂1 and set their derivatives to be equal to 0:

1
-solving these 2 N.E.’s in 2 unknowns, will give us the solution to our ̂ 0 and ̂1 :

where ˆ0  y  ˆ1 x and

 n  n 
n   xi   yi 
 xi yi   i 1  i 1 
ˆ1  i 1 n

  x  x  y  y   S
i i xy

x  x 
2 2
 n  S
  xi 
i xx
n

 xi   i 1 
2

i 1 n

And so Least-squares/best-fitting/regression/prediction line/equation: ŷ  ˆ0  ˆ1 x

- It can be shown that ̂ 0 and ̂1 obtained using Least-Squares Method, are so-
called
B – “Best”

L – “Linear” NOTE: ˆ1 ~ N ( 1 ,

2
) and ˆ0 ~ N (  0 ,
 2
x i
2

)
S xx nS xx
U – “Unbiased”

E – “Estimators”

2
 Ex. Assume n = 10 students constitute a random sample from a population of first year
students who have already entered the university or will do so in the immediate
future. We want to find out if we can use student’s math test score (x) administered
prior to the entrance to university to estimate/predict final grade in Calculus (y)

Student 1 2 3 4 5 6 7 8 9 10
Math test score (x) 39 43 21 64 57 47 28 75 34 52
Final calculus grade (y) 65 78 52 82 92 89 73 98 56 75

- First, we need to do scatter plot (scatter diagram), i.e. plot of x vs y, to see if we

have a linear relationship between x and y.

- Second, we may now proceed with calculations of the least-squares line.

n n n n
To do that, we will need following:  xi ,
i 1
 yi ,
i 1
 xi yi and
i 1
x
i 1
i
2
.

3
 S xy
 ˆ1  
S xx

ˆ0  y  ˆ1 x 

 the least-squares line is given by:

And so if we want to predict student’s calculus grade knowing that his/her math test score was
50, then:

4
 Estimation of Variance

- recall, that under the model assumptions, we assumed that y’s (or  ’s) have
constant variance  2 , V x.
- since this is the population model, we are talking about population variance and as
such it is unknown as well.

n n n
SSE =  ei2   ( yi  yˆi )2   [ yi  ( ˆ0  ˆ1 xi )]2 =
i 1 i 1 i 1

5
 ( S xy ) 2 SSE
i.e. we get SSE  S yy  and from there s 2 
S xx n2

NOTE: - s is called the standard error of estimation and it estimates how much the y
values vary around the population line E(y).

- from Empirical Rule, we know that approximately 95% of all observations should lie
within 2 std. deviations of the mean.

Ex. (cont’d)

( S xy ) 2
SSE  S yy 
S xx

So, SSE =

SSE
Hence s2  =
n2

(NOTE: see discussion on CV’s )