Predictive Analytics

Regression and Classification

Module 3

Sourish

Chennai Mathematical Institute

Experiment

An experiment performed to assess the relative effect of three

toxins and a control on the liver of a certain species of trout.
The are about the amounts of deterioration (in standard units)
of the liver in each sacrificed fish.

Toxin 1 Toxin 2 Toxin 3 Control

28 33 18 11
23 36 21 14
14 34 20 11
27 29 22 16
Framing the Model

An experiment performed to assess the relative effect of three

toxins and a control on the liver of a certain species of trout.
The are about the amounts of deterioration (in standard units)
of the liver in each sacrificed fish.
I The above experiment can be modeled as

Yij = θi + εij , i = 1, 2, · · · , k and j = 1, 2, · · · , ni

where

I E(εij ) = 0 and Var (εij ) = σi2 < ∞ for all i, j, and

Cov (εij , εi 0 j 0 ) = 0, ∀ i, j
Framing the Model

I The above experiment can be modeled as

Yij = θi + εij , i = 1, 2, · · · , k and j = 1, 2, · · · , ni

where

I E(εij ) = 0 and Var (εij ) = σi2 < ∞ for all i, j, and

Cov (εij , εi 0 j 0 ) = 0, ∀ i, j

I σi2 = σ 2 for all i (homoscedasticity).

I εij ∼ N (0, σ 2 )
Framing the Model

I The above experiment can be modeled as

Yij = θi + εij , i = 1, 2, · · · , k and j = 1, 2, · · · , ni

where

I Can we express this model into linear model form?

y = Xβ + 

I Then we will have an analytic solution to above model.

Finding Solutions
We define the design matrix X as follows:
 
Toxin 1 Toxin 2 Toxin 3 Control
 1 0 0 0 
 
 1 0 0 0 
 
 1 0 0 0 
 
 1 0 0 0 
 
 0 1 0 0 
 
 0 1 0 0 
 
 0 1 0 0 
 
 0
X = 1 0 0  
 0 0 1 0 
 
 0 0 1 0 
 
 0 0 1 0 
 
 0 0 1 0 
 
 0 0 0 1 
 
 0 0 0 1 
 
 0 0 0 1 
0 0 0 1
Finding Solutions
We have the response vector y as follows:
 
28
23
 
14
 
27
 
33
 
36
 
34
 
29
y =
18

 
21
 
20
 
22
 
11
 
14
 
11
16
Finding Solutions

We can find the X T X as follows

 
4 0 0 0
T
0 4 0 0
X X =
0

0 4 0
0 0 0 4

and (X T X )−1 is
1 
4 0 0 0
0 1
0 0
(X T X )−1 = 
0
4
1

0 4 0
1
0 0 0 4
Finding Solutions
We can find the X T X as follows
P4 
j=1 y1j
P4 y 
2j 
X T y = Pj=1

 4j=1 y3j 

P4
j=1 y4j

Hence the solution is:

 1 P4   
4 Pj=1 y1j ȳ1
1 4 y   
2j  ȳ2 
β̂ = (X T X )−1 X T y =  41 Pj=1 =

4 ȳ3 
 4 j=1 y3j  
1 P4 ȳ4
4 j=1 y4j

So the model yields group sample mean as the solution for

(θ1 , θ2 , θ3 , θ4 ).
Ask yourself: How to model global mean?

I The above experiment is be modeled as

Yij = θi + εij , i = 1, 2, · · · , k and j = 1, 2, · · · , ni

where

I Each θi is the group mean. But how can we model a global

mean of all data?

I Take a pause of 10 minutes and try for yourself to develop

the model with global mean.
A model with global mean

I We can try

Yij = µ + θi + εij , i = 1, 2, · · · , k and j = 1, 2, · · · , ni

where

I But we are going to face a problem. Can you identify the

problem?

I Pause the video and think about it for 10 minutes.

A model with global mean
Our response vector y is still same. The change we see is in
the design matrix X . Let us see the changed design matrix.
 
Intercept Toxin 1 Toxin 2 Toxin 3 Control

 1 1 0 0 0  

 1 1 0 0 0  

 1 1 0 0 0  

 1 1 0 0 0  

 1 0 1 0 0  

 1 0 1 0 0  

 1 0 1 0 0  
X =
 1 0 1 0 0  

 1 0 0 1 0  

 1 0 0 1 0  

 1 0 0 1 0  

 1 0 0 1 0  

 1 0 0 0 1  

 1 0 0 0 1  
 1 0 0 0 1 
1 0 0 0 1
A model with global mean

The X T X matrix is
 
16 4 4 4 4
4 4 0 0 0
XTX = 
 
4 0 4 0 0

4 0 0 4 0
4 0 0 0 4

I Now if you look at carefully the first column of X T X is

direct sum of 2nd, 3rd, 4th and 5th column. So X T X is not
invertible.

I That means solution does not exits if we create dummy

variable for each labels of categorical variables.
Thank You