Data Transformation
Objectives:
Understand why we often need to transform our data
The three commonly used data transformation techniques
Additi e effects and m
Additive
multiplicative
ltiplicati e effects
Application of data transformation in ANOVA and
regression.
regression
Xuhua Xia
Why Data Transformation?
The assumptions of most parametric methods:
Homoscedasticity
Normality
Additi it
Additivity
Linearity
D
Data transformation
f
i is
i usedd to make
k your data
d
conform to the assumptions of the statistical methods
Illustrative examples
Xuhua Xia
Homoscedasticity and Normality
Homoscedasticity and Normality
The data deviates from both
homoscedasticity and normality.
Wont it be nice if we would
make data look this way?
Xuhua Xia
Xuhua Xia
Homoscedasticity
Types of Data Transformation
The logarithmic transformation
Th square-roott transformation
The
t
f
ti
The arcsine transformation.
Data transformation can be done conveniently in
EXCEL.
Alternatives: Ranks and nonparametric methods.
ID
1
2
3
4
5
6
7
8
9
n
Mean
Var
t
df
P
Equal Var.?
Kurtosis
Skewness
P(Zg1)
P(Zg2)
Xuhua Xia
ID Group
Group
Group
1 1 Group
2 2
11
2.72
20.09
1.31
3.05
2.13
22
7.39
54.60
4.02
33
7 13
39
54
54.60
60
27.39
2.13
4 02
4.02
44
20.09
148.41
3.05
5.01
55
20.09
148.41
3.05
5.01
66
20.09
148.41
3 05
3.05
5 01
5.01
77
54.60
403.43
4.02
6.00
88
54.60
403.43
4.02
6.00
9
148.41
1096.63
9
5.01
7.00
n
9
9
Mean
3.08
5.01
Var
1.30
1.47
t
-3.48
3 48
df
16
P
0.0031
Equal Var
Var.?
?
P=
P
0 8687
0.8687
Kurtosis
-0.35
-0.30
Skewness
0.16
0.02
P(Zg1)
0.82
0.97
P(Zg2)
0.659393 0.684475
Xuhua Xia
The two groups of data seem to differ
greatly in means, but a t-test shows that
the means do not differ significantly from
each other - a surprising result.
p of data differ greatly
g
y in
The two ggroups
variance, and both deviate significantly
from normality. These results invalidate
the t-test.
t-test
We calculate two ratios: var/mean ratio
and Std/mean ratio ((i.e., coefficient of
variation).
Var/mean
V
/
C.V.
Group1 Group2
56 420 416.891
56.420
416 891
1.230
1.230
Log transformation
Log-transformation
Xuhua Xia
Log-Transformed Data
ID
Group 1
Group 2
2.72
20.09
7.39
54.60
7.39
54.60
20.09
148.41
20.09
148.41
20 09
20.09
148 41
148.41
54.60
403.43
54.60
403.43
148.41
1096.63
9
9
37.26
275.33
2102.35 114784.50
-2.09
2 09
16
0.0530
P=
P
0.0000
4.89
4.89
2.12
2.12
0.0046
0.0046
0.0144
0.0144
NewX = ln(X+1)
Log-Transformed Data
ID
1
2
3
4
5
6
7
8
9
The transformation is successful
b
because:
The variance is now similar
Deviation from normality is now
nonsignificant
The t-test revealed a highly significant
difference in means between the two
groups
n
Mean
Var
t
df
P
Equal Var.?
Kurtosis
Skewness
P(Z 1)
P(Zg1)
P(Zg2)
Xuhua Xia
Group 1
Group 2
1.31
3.05
2 13
2.13
4 02
4.02
2.13
4.02
3.05
5.01
3.05
5.01
3.05
5.01
4.02
6.00
4.02
6.00
5 01
5.01
7 00
7.00
9
9
3.08
5.01
1 30
1.30
1 47
1.47
-3.48
16
0.0031
P=
0.8687
-0.35
-0.30
0.16
0.02
0 82
0.82
0 97
0.97
0.659393 0.684475
NewX = ln(X+1)
Transform back:
X = e NewX 1
C
Compare
this
thi mean with
ith the
th original
i i l
mean. Which one is more preferable?
Calculate the standard error, the
degree of freedom, and 95% CL
(t0.025,16 = 2.47).
2 47)
Normal but Heteroscedastic
Normal but Heteroscedastic
ID
1
2
3
4
5
6
7
8
9
Any transformation that you use is likely to
change
h
normality.
li Fortunately,
l t-test and
d ANOVA
are quite robust for this kind of data. Of course,
you can also
l use nonparametric
i tests.
Xuhua Xia
n
Mean
Var
t
df
P
Equal Var.?
Kurtosis
Skewness
Level 1
Mean
Level 2
Mean
Factor A
Level 1
Level 2
1.313
2.450
2.127
3.240
2.127
3.343
3.049
3.976
3 049
3.049
3 578
3.578
3.049
4.507
4.018
4.666
4.018
5.324
5.007
6.060
3.084
4.127
2.030
2.927
2 805
2.805
3 599
3.599
2.751
3.837
3.968
4.933
3.766
4.576
3.589
4.781
4.570
5.719
4.562
5.983
5 956
5.956
6 868
6.868
3.778
4.803
Xuhua Xia
Group 2
1
4
4
8
8
8
12
12
15
9
13
1.5
3.216338
16
0.0054
P=
-0.28571
0
9
8
20.25
0.0013
-0.76582
0
The t-test, however,
d t t significant
detects
i ifi t
difference in means. You
can use nonparametric
methods to analyse data
for comparison, and you
are like to find t-test to be
more powerful.
The two variances are
significantly different.
different
Xuhua Xia
Additivity
Factor B
Group 1
11
12
12
13
13
13
14
14
15
What experimental design is
this?
Compare the group means
means.
Is there an interaction effect?
Multiplicative Effects
Factor B
Level 1
Mean
Additivity
Addi
i i means that
h the
h difference
diff
between levels of one factor is
consistent for different levels of
another factor.
Level 2
Mean
Xuhua Xia
Factor A
Level 1
Level 2
2.718
10.589
7.389
24.530
7.389
27.316
20.086
52.304
20 086
20.086
34 795
34.795
20.086
89.660
54.598 105.306
54.598 204.262
148.413 427.365
37.262 108.458
6.616
17.678
15 524
15.524
35 570
35.570
14.665
45.408
51.884 137.739
42.222
96.110
35.185 118.225
95.498 303.596
94.819 395.790
385 215 960.457
385.215
960 457
82.403 234.508
Compare
C p the ggroupp means. Is
there an interaction effect?
Does this data set meet the
assumption of additivity?
When the assumption of
additivity is not met, we have
difficulty
y in interpreting
p
g main
effects.
Now calculate the ratio of group
means. What did you find?
Multiplicative Effects
Factor B
Level 1
Mean
Level 2
Mean
Factor A
Level 1
Level 2
2.718
10.589
7.389
24.530
7.389
27.316
20.086
52.304
20 086
20.086
34 795
34.795
20.086
89.660
54.598 105.306
54.598 204.262
148.413 427.365
37.262 108.458
6.616
17.678
15 524
15.524
35 570
35.570
14.665
45.408
51.884 137.739
42.222
96.110
35.185 118.225
95.498 303.596
94.819 395.790
385 215 960.457
385.215
960 457
82.403 234.508
For Factor A, we see that Level 2 has a
mean about 2.88
2 88 times as large as that for
Level 1. For factor B, Level 2 has a mean
about 2.18 times as large as that for Level
1).
If yyou know the value for Level 1 of
Factor A, you can obtain the value for
Level 2 of Factor A by multiplying the
known value by 2.88. Similarly, you can
do the same for Factor B.
We say that the effect of Factors A and B
are multiplicative, not additive.
Xuhua Xia
Log-transformation
Factor A
Factor
Level 2
Factor B
B Level
Level 11
1.31
2.718
10.589
1.313
2.450
2 13
2.13
7.389
24.530
2.127
3.240
7.389
27.316
2.127
3.343
20.086
52.304
3.049
3.976
Level 11
Level
203.049
20.086
3 086
049
343.578
34.795
3 795
578
3.049
4.507
20.086
89.660
4.018 105.306
4.666
54.598
4.018 204.262
5.324
54.598
5.007 427.365
6.060
148.413
Mean
3.084 108.458
4.127
Mean
37.262
2.030
2.927
6.616
17.678
2 524
805
3 570
599
152.805
15.524
353.599
35.570
2.751
3.837
14.665
45.408
3.968 137.739
4.933
51.884
Level 22
3.766
4.576
Level
42.222
96.110
3.589 118.225
4.781
35.185
4.570 303.596
5.719
95.498
4.562 395.790
5.983
94.819
3855.956
385.215
215 960.457
9606.868
457
Mean
3.778 234.508
4.803
Mean
82.403
Now log-transform the data. Compare
the means. Is the assumption of
additivity met now?
Original Data
37.262
2102.351
108.458
17878.648
82.403
12400.091
234.508
80241.944
3.084
1.302
4.127
1.268
3.778
1.235
4.803
1.385
Mean
Variance
T
Transformed
f
d d
data
t
Xuhua Xia
Square-Root Transformation
ID
1
2
3
4
5
6
7
8
9
Why log-transformation can
change
h
the
th multiplicative
lti li ti
effects to additive effects?
Z = XY
ln(( Z ) = ln(( X ) + ln((Y )
Xuhua Xia
Mean
Var
Var/Mean
Std/M
Std/Mean
Xuhua Xia
Group 1
1
4
4
9
9
9
16
16
6
25
10.333
56.500
5.468
0 727
0.727
Group 2
9
16
16
25
25
25
36
36
49
26.333
152.500
5.791
0 469
0.469
The two groups of data differ much
in variance.
Calculate two ratios: var/mean
ratio and Std/mean ratio (i.e.,
coefficient of variation).
variation)
Does your calculation suggest logtransformation? When is logtransformation appropriate?
Use square-root transformation
when
h different
diff
t groups have
h
similar
i il
Variance/Mean ratios
Notice the means, which do not
coincide with the most frequent
observations
b
ti
Square-Root Transformation
ID
1
2
3
4
5
6
7
8
9
Group 1
1.171
1.17
2.094
2.09
2.094
3 069
3.06
3.069
3 069
3.06
16
4.05
16
4.05
25
5.04
10.333
3.07
56.500
1.412
Mean
Var
Group 2
3.069
16
4.05
16
4.05
25
5 04
5.04
25
5.04
25
5 04
5.04
36
6.03
36
6.03
49
7.03
26.333
5.04
152.500
1.475
Quiz on Data Transformation
Square-root
transformation:
1
2
0
2
3
0
5
1.4
1.8
0.600
2.776
-0.266
3.066
X '= X + 3/ 8
The variance is now
almost identical between
the two groups
Transform the means
back to the original scale
and compare these means
with the original means:
X = ( X ' )2
3
8
Xuhua Xia
n
Mean
Var
SE
T
LowerL
UpperL
The data set is rightskewed for each
group.
Calculate the
variance/mean ratio
and C.V. for each
group, and decide
what transformation
you should use.
Do the transformation
and convert the means
back to the original
scale
scale.
4
2
0
2
4
Mean
12
10
8
6
Mean, Lower, Upp
per
When you have multiple
groups, a Variance vs Mean
or a Std vs Mean p
plot can
help you to decide which data
transformation to use. The
graph on the left shows that
the Var/Mean ratio is almost
constant What
constant.
transformation should you
use?
Confidence Limits
Mean, Lo
ower, Uppe
er
8
V
Variance
4
2
4
1
0
2
5
1.8
2.2
0.663
2.776
-0.042
3.642
Xuhua Xia
With Multiple Groups
Group
2
3
6
9
4
5
8
6
2
5
4
11
5
5
4.8
7.2
5.2
7.2
1.020 1.200
2.776 2.776
1.969 3.868
7.631 10.532
4
2
0
-2
2
0
Mean
Before transformation
12
10
8
6
4
2
0
-2
0
Mean
After transformation
With the skewness in our data, do confidence
limits on the right make more sense? Why?
Xuhua Xia
Xuhua Xia
Arcsine Transformation
Group1 Group2 Group1 Group2
84.20 92.30 66.579 73.890
88 90 95.10
88.90
95 10 70.539
70 539 77.211
77 211
89.20 90.30 70.814 71.854
83.40 88.60 65.957 70.267
80.10 92.60 63.507 74.215
81.30 96.00 64.378 78.463
85.80 93.70 67.863 75.463
Mean
84.70 92.66 67.091 74.480
Var
12.29
6.73
8.017
8.226
SE
1 325 0.980
1.325
0 980
1 070
1.070
1 084
1.084
LowerL
81.457 90.258 64.472 71.828
UpperL
87.943 95.056 69.709 77.133
T
Transform
f
back
b k
NewMean
84.847 92.841
LowerL
81.428 90.273
UpperL
87.974 95.041
Xuhua Xia
Data Transformation Using SAS
X ' = arcsin( X )
Data Mydata;
y
;
Used for proportions
p the
Compare
variances before and
after transformation
Do you know how to
transform the means
and C
C.L.
L back to the
original scale?
X = (sin X ' )
input
p x;
newx=log(x);
newx=sqrt(x+3/8);
newx=arsin(sqrt(x));
2
cards;
Xuhua Xia
Natural logarithm
transfromation
Square-root
S
transformation
Arcsine
transformation