Chapter 2: Looking at Data

Relationships
• Given ≥2 quantitative variables, we often wish to
look at the relationship between them
– Visualizing association: scatterplots
– Numerically describing association: correlation
– Model for association/prediction: regression
– Note: correlation does not imply causation

• Also may wish to do so for categorical variables

– Two-way tables 1
 Association between two variables
• Examples: height and weight, smoking habits
and life expectancy, age and bone density, gender
and political affiliation
• Association: knowing the value of one variable
tells us something about the value of the other
• Independence: knowing the value of one variable
tells us nothing about the value of the other
• Note: we assume both variables are measured on
the same individuals 2
 Association and causation
• Key fact: association NOT the same as causation
• Caution: associations often spurious (non-causal)
due to other variables “lurking” in the background
• Example: countries with more TV sets have
higher life expectancy rates
• Does owning a TV increase life expectancy?
• In statistics, may just explore association or go
further and try to assess causal relationship
3
Nations: Internet users (%)
vs life expectancy (years)

Does internet use increase life expectancy?

4
 Types of variables
• Response variable y measures outcome of interest
• Changes in explanatory variable x may explain and
sometimes may cause changes in y

• Examples: response variable? explanatory variable?

– Different amounts of alcohol given to mice, changes in
body temperatures noted
– Drop in body temperature after observed amount of
alcohol consumed
– GRE scores and graduate GPA
– Age of child and child’s height
5
 Scatterplots
Example 1:
Mean heights of
children in
Kalama, Egypt
surveyed at age
18 to 29 months

Plot: y versus x
predictor?
response?
6
 Example 1: Mean heights of
children (Kalama, Egypt) vs. age

7
Example 2: State mean SAT math score
vs. %high school seniors taking SAT

8
 Example 3: Nations
infant mortality and per capita GDP

9
 In a scatter plot, look for …
• Relationship form: linear, exponential, …
• Direction of association: does y increase or
decrease with increase in x (positive or negative
association)
• Strength of association: do the points follow the
relationship form closely or are they scattered?
• Outliers: deviations from overall relationship
• How would you characterize our examples? 10
Which relationship is stronger?

11
 Correlation
• Visual inspection important but also useful to have
numerical summary measures
• (Pearson) correlation r a numerical measure of the
strength, direction of linear relationship between
quantitative variables x, y
• Data: x, y measured on n individuals
• Examples:
– x = weight, y = blood pressure
– x = age, y = bone density
– x = GDP, y = infant mortality rate
Properties of Correlation Coefficient r
• -1 ≤ r ≤ 1

• r > 0 for positive association, r < 0 for negative

• r ≈ 0 implies no or weak linear relationship

• r ≈ ±1 implies very strong linear relationship

• r unit free, unaffected by linear transformation

(e.g. z = ax + b)

• If we swap x and y, r does not change 13

 Properties of r (continued)
• r strongly affected by outliers

• r captures strength of linear relationship

• y and x may have very strong non-linear (e.g.

quadratic) relationship but r ≈ 0

• r = ±1 only if points lie exactly on a straight line

• In SPSS: Analyze → Correlate → Bivariate

14
Examples of correlation coefficients

15
Formula for correlation coefficient r

1  xi − x   yi − y 
r= ∑   
n − 1  sx   s y 


Pearson product-moment
correlation coefficient

16
 Calculating r
• Calculate sample means, SDs for x and y

• Standardize: for each observation for each

subject, subtract mean, divide by SD (standardize)

• For each subject, take product of standardized x’s

and y’s

• Sum up the products, divide the sum by n-1

• Almost always use statistical package e.g. SPSS

17
 Optional example:
Calculating r for classifying fossils (1)

Lengths (cm) of leg and upper arm bones of

Archaeopteryx, extinct animal thought to
connect modern birds with dinosaurs

Femur: 38 56 59 64 74
Humerus: 41 63 70 72 84
18
 Calculating r for fossils (2)
• Let x = femur length, y = humerus length
• Sample size n = 5
• Step 1: Find mean, SD for x, y

Femur: 38 56 59 64 74
Humerus: 41 63 70 72 84

Femur: X = 58.2 cm Sx = 13.20 cm

Humerus: Y = 66.0 cm SY = 15.89 cm 19
 Calculating r for fossils (3)
Step 2: Use means, SDs from step 1 to
calculate standardized scores for each x and y:

Standard score Standard score

Value (X − X) Value (Y − Y )
of x SX of y SY

38 (38 - 58.2) / 13.20 = -1.53 41 (41 - 66.0) / 15.89 = -1.57

56 (56 - 58.2) / 13.20 = -0.17 63 (63 - 66.0) / 15.89 = -0.19
59 (59 - 58.2) / 13.20 = 0.06 70 (70 - 66.0) / 15.89 = 0.25
64 (64 - 58.2) / 13.20 = 0.44 72 (72 - 66.0) / 15.89 = 0.38
74 (74 - 58.2) / 13.20 = 1.20 84 (84 - 66.0) / 15.89 = 1.13
20
 Calculating r for fossils (4)
Step 3: r is approximate average of
products of standardized scores; as with
SD, “average” by dividing by (n – 1)
(−1.53 × −1.57) + (−0.17 × −0.19) + 0.06 × 0.25 + 0.44 × 0.38 + 1.20 × 1.13
𝑟𝑟 =
4
2.41 + 0.03 + 0.02 + 0.17 + 1.36
𝑟𝑟 =
4

3.98
𝑟𝑟 = = 0.995
4

Conclusion?
21
 Key fact: r = 0 does not imply x
and y are not associated

• r = 0 only means there is no linear

relationship

• There could be a (strong) non-linear

relationship

22
More examples of correlation coefficients

Source: Wikipedia
 Transformations
• When variables are skewed or relationships are
non-linear, can be helpful to transform variables
• Typically seek to make
– variable distribution more normal (avoid outliers)
– relationship between variables more linear
• Logarithm, square root, ranks are common
transformations
• Note: logarithm, square root only make sense if
variable > 0
• There are other correlation measures that capture
24
non-linear relationships
 Example: Nations
Infant mortality and per capita GDP

r = -0.435

25
 Example: Nations
log(infant mortality) and log(per capita GDP)

r = -0.888

26
27
 Summary (MMC Sections 2.1-2.3)
• Scatterplots: look for form, direction, strength,
possible outliers
• Correlation coefficient r: numerical measure that
captures direction, strength of linear relationship
– sign of r: direction of linear relationship
– absolute value of r: strength of linear relationship
– correlation is not the same as causation

• Always plot the data, visually inspect relationship,

and assess appropriateness of linear model
• Transformations 28
Correlation is not causation!

29
Correlation is not causation!

Messerli (2012). New England Journal of Medicine 367:1562-1564.

Correlation is not causation!

[Link]
 Linear regression
• Straight line which “best” describes how response
variable y changes with explanatory variable x
• Equation of straight line: y = b0 + b1x
b0: intercept b1: slope
• Goal: choose b0, b1 to give line that “best” fits data
• Can use line to predict y for different values of x
• Note: here distinguish y and x; switching roles
(response, explanatory) changes regression line
32
Example: height and weight
 Example: height and weight

𝑟𝑟 = 0.48

𝑦𝑦� = 64.8

𝑠𝑠𝑦𝑦 = 13.3

𝑥𝑥̅ = 159.6

𝑠𝑠𝑥𝑥 = 9.0
Example: height and weight

𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿

𝑦𝑦 = −49.02 + 0.713𝑥𝑥
 Properties of the regression line
• Models response (dependent) variable y …
… using an explanatory (independent) variable x
• Equation of straight line: y = b0 + b1x
b0: intercept, predicted value of y when x is zero
b1: slope, change in predicted y per unit change in x

• Passes through the point (𝑥𝑥,̅ 𝑦𝑦)

�
• Swapping x and y gives different lines
• Use method of least squares to obtain “best” fit 36
Calculating the slope and intercept
• Formula for the slope
𝑠𝑠𝑦𝑦
𝑏𝑏1 = 𝑟𝑟
𝑠𝑠𝑥𝑥

• Formula for the intercept

𝑏𝑏0 = 𝑦𝑦� − 𝑏𝑏1 𝑥𝑥̅

 Why these formulas?
• These formulas give the line that make observed 𝑦𝑦 and
predicted 𝑦𝑦� values close
• Observed 𝑦𝑦 values are observations in the data
• Predicted 𝑦𝑦� values are estimated using regression line
• Prediction errors (residuals) are the differences 𝑦𝑦 − 𝑦𝑦�
• Our line gives the smallest sum of squared differences
2
� 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖

“Method of least squares”

 Fat gain vs non-exercise activity
Error = Residual
Predicted = ŷ

Regression line minimizes the sum of squared errors

39
 Example: Mean heights of
children (Kalama, Egypt) vs. age

40
Example: Mean heights of children
(Kalama, Egypt) vs. age
x =
23.5 months, y 79.85 cm,
sx =
3.606 months, s y 2.302 cm,
r = 0.9944

Slope b1 =

Intercept b0 =
41
Example: Mean heights of children
(Kalama, Egypt) vs. age
x =
23.5 months, y 79.85 cm,
sx =
3.606 months, s y 2.302 cm,
r = 0.9944

Slope b1 = .9944 × 2.302/3.606 = 0.635

Intercept b0 = 79.85 - 0.635 × 23.5 = 64.93

42
 Example 1: Mean heights of
children (Kalama, Egypt) vs. age
Positively
associated
(b1 > 0)
Linear
relationship?
Guess r?

43
 Example: Mean heights of children
(Kalama, Egypt) vs. age
• Regression line:
Height (cm) = 64.93 + 0.635 × Age (months)

• At 18 months predict:
ŷ=
• If observed height at 18 months is 76.1 cm …
• Residual or error = 76.1 - 76.36 = -0.26 cm 44
 Interpretation of regression line
• Slope b1 and correlation coefficient r of same
sign; capture direction of (linear) association

• b1 average change in height given 1 month

change in age

• b0 average height at birth

45
 Taking predictions too far …
• Regression line for Kalama data:
Height (cm) = 64.93 + 0.635 × Age (months)

• At 32 months predict:
Height = 64.93 + 0.635 × 32 = 85.25 cm

• At 30 years (360 months) predict:

Height = 64.93 + 0.635×360 = 293.53 cm, >9 feet!

• Why might this be a poor prediction?

• Extrapolation (beyond range of data) dangerous!46

 47
[Link]
 Note: use a statistical package
• The software will evaluate the least squares
regression line, just have to know where to look
in the output

• Residuals, predicted values can be printed and

plotted

• SPSS → Analyze → Regression → Linear

• We will discuss regression again in chapter 10 in

context of statistical inference 48
 Correlation and regression
• slope b1 and correlation coefficient r of same sign
• r2, square of the correlation coefficient, is fraction of
variation in y explained by linear relationship with x
• r2 summarizes goodness of fit of linear model
• higher r2 indicates better fit of linear model
• r2 varies between 0 and 1
• r2 = .989 for Kalama data means 98.9% variation in
mean height explained by linear relationship with
49
age
 r2: proportion of variation of y
explained by linear relationship with x

(a) less scatter, smaller prediction error, r2=.989

(b) more scatter, larger prediction error, r2=.849 50
Height and weight
(r2 = 0.23)
Average height and average weight
2
(r = 0.94)

Averaging inflates correlation:

maintains signal, decreases noise
 Height and weight
when height > 175 cm (r2 = 0.06)

Analyzing restricted range

usually deflates correlation
 Summary: regression and correlation
• Correlation, regression useful tools to summarize
linear relationship between quantitative variables
• Least squares most common method to fit linear
regression line to data
• Regression model useful for prediction
• Slope b1 predicted change in y for unit change in x
• Intercept b0 is predicted value of y when x=0
• Correlation r and slope b1 of same sign
54
• r = b1 if x, y both measured in standardized units
 Regression and correlation: limitations
• Correlation measures only linear relationship
• Fitting regression line only makes sense if x to y
relationship approximately linear; plot the data
first!
• While regression can be useful for prediction,
beware of extrapolation
• Correlation, regression sensitive to extreme
values
• Correlation does not imply causation 55
 Residual plots
• Residual = y – ŷ (observed y minus predicted y)
• Residual plot: scatter plot of residual versus x
• Residual plots help assess the fit of linear
regression model to data
– good fit: random spread of points about the zero
line, with no obvious pattern
– bad fit: clear patterns or trends emerge

• Residuals always add to zero and have mean zero

56
Example: mean heights of children
(Kalama, Egypt) vs. age

Regression plot Residual plot 57

Height and weight residual plot
Other examples of residual plots
 Outliers and influential observations
• Outlier: observation that lies outside the overall
pattern: x-outlier, y-outlier, joint outliers
• Influential observation: observation whose
removal markedly changes results of statistical
analysis
• In linear regression, y-outliers often result in large
residuals, x-outliers often influential
• Influential observations “pull” the regression line
• Least squares regression is not robust to the
presence of outliers why? 60
Example: Gesell score against age
(months) at first word

61
Example: Gesell development score
• With complete data, r2 = .41
– looks like age at first word a good predictor of score

• Child 18 looks like an outlier …

– what happens if we remove child 18?
– deleting child 18, r2 = .11 and regression line shifts
– child 18 is “influential”

• Child 19 is also an outlier

– deleting child 19 does not change regression line much
– child 19 is not “influential”
62
 Lurking variables (1)
• Other variables may influence x-y relationship
• Lurking variable: variable (not included among
the explanatory and response variables) that
influences relationships of those variables
• Example: x = height, y = math test score in
school children
• Suppose we observe strong positive correlation
• Does being tall improve math test scores?
63
 Lurking variables (2)
• Does being tall improve math test scores?
• Probably not. Lurking variable: age
• Age may influence both height, math test score
• If ignore age (z), may incorrectly conclude height
(x) partially determines math score (y)

64
Annual imports and health care spending

r = .97

65
Annual imports and health care spending
• x = annual imports ($) to USA 1990-2001
• y = annual health spending ($) in USA 1990-2001
• No obvious causal relationship between x, y
• Highly correlated: r = .97
• Lurking variable: z = year of reporting; both x, y
grew rapidly over this time period
• Any two variables that both increase over time will
show correlation 66
 Summary
• (Pearson) correlation, linear regression are
measures of linear relationship
• Examine fit of linear model to data using scatter
plots of data and of residuals
• Look for influential observations
• Correlation and regression may be misleading if
ignore important lurking variables
• Averaging tends to increase correlation,
restricting range to decrease correlation 67
 Two-way tables …
• So far, we have focused on quantitative variables
• Two-way tables summarize relationships for
categorical variables, cross tabulate information
on two variables
• Categorical variable analog to scatter plot for
quantitative variables
• To interpret tables, often useful to calculate
percentages
 Two-way table
• Summarize distribution
Academic Small of each variable
Center Hospital Total – How many patients died?
Recover 760 390 1150
– How many were seen at
Die 40 10 50 each site?
Total 800 400 1200
• Summarize joint
Academic Small distribution
Center Hospital Total • To make reading easier,
Recover 63.3% 32.5% 95.8% we may calculate
Die 3.3% 0.8% 4.2% overall percentages …
Total 66.7% 33.3% 100.0%
 Two-way table
• Summarize distribution
Academic Small of each variable
Center Hospital Total – How many patients died?
Recover 760 390 1150
– How many were seen at
Die 40 10 50 each site?
Total 800 400 1200
• Summarize joint
Academic Small distribution
Center Hospital Total • To make reading easier,
Recover 95.0% 97.5% 95.8% we may calculate
Die 5.0% 2.5% 4.2% overall percentages …
Total 100% 100% 100%
• … or row and column
percentages
Lurking variables and Simpson’s paradox
• Academic medical centers less skilled than
small hospitals?
• Lurking variable: simple vs complicated cases
• How does this impact interpretation of our
findings?
 Simpson’s paradox
Simple Cases Complicated Cases

Academic Small Academic Small

Center Hospital Center Hospital
(n = 400) (n=380) Total (n = 400) (n = 20) Total
Recover 99% 98% 99% Recover 91% 80% 90%
Died 1% 2% 1% Die 9% 20% 10%
Total 100% 100% 100% Total 100% 100% 100%

Academic Small
Center Hospital
(n = 800) (n = 400) Total
Recover 95% 97.5% 95.8%
Die 5% 2.5% 4.2%
Total 100% 100% 100.0%
Lurking variables and Simpson’s paradox
• Academic medical centers less skilled than
small hospitals?
• Lurking variable: simple vs complicated cases
• How does this impact interpretation of our
findings?
• Simpson’s paradox: an association that holds
for all subgroups can reverse direction when
subgroups are combined
 Simpson’s paradox: US median wage
• Since 2000, median US wage has risen ~1%
adjusted for inflation
• Over same period, the median wage for all groups:
– high school dropouts,
– high school graduates with no college education,
– people with some college education, and
– people with Bachelor’s or higher degree
have decreased.
• How can both be true: overall wages have risen,
but wages within every subgroup have fallen?
 Simpson’s paradox: US median wage
• Explanation:
– there are now many more college graduates (who get
higher-paying jobs) than there were in 2000
– wages for college graduates have fallen at much slower
rate (down 1.2%) than for those of lower educational
attainment (e.g. 7.9% for high school dropouts)
– growth in proportion of college graduates swamps the
wage decline for specific groups
 Summary
• Two-way tables for categorical variables
• Lurking variables and Simpson’s paradox

76