Chapter 2 - Observational Studies
Controlled experiments are different from observational studies.
In a controlled experiment, the investigators decide who will be in the
treatment group and who will be in the control group.
In an observational study, it is the subjects who assign themselves to
the different groups: investigators just observe what happens.
Let us clarify the word control in the following two perspectives:
- A control is a subject who did not get the treatment.
- A controlled experiment is a study where the investigators
decide who will be in the treatment group and who will not.
Question: based on the above definitions, can you think of any
examples that are observational studies?
For example, the studies of the effect on smoking is going to be
observational for sure. This is because no one is going to simply
smoke for 10 years just to satisfy a study or to meet the condition of
an experiment!
In reality, some people are smokers and they have been for 10 years.
Some people are not. They work with the investigators and
statisticians under randomized experiments. It can even be
double-blinded. Even though these conditions of controlled
experiment are met, the study of smoking effect is necessarily
observational.
Observational studies are a powerful tool, as the smoking example
shows. However, they can also be misleading.
To see if confounding is a problem, it may help to find out how the
controls were selected. The main issue is: was the control group
really similar to the treatment group - apart from the exposure of
interest? If there is confounding, something has to be done about it,
although perfection cannot be expected.
Statisticians talk about controlling for confounding factors in an
observational study. This is another use of the word control.
In the example of smoking case, one technique is to make
comparisons of smaller and more homogeneous groups.
- For example, one comparison of death rates among smokers
and non-smokers could be misleading, because smokers are
disproportionately male and men are more likely than women to
have heart disease.
- Another way is to look at Age. Elder people have different
smoking habits. Hence, the smokers and non-smokers are
compared by age as well as by sex. Good observational
studies control for confounding variables.
Let us take a look at another example: the Coronary Drug Project.
The Coronary Drug Project was a randomized, controlled
double-blind experiment, whose objective was to evaluate five drugs
for the prevention of heart attacks. The subjects were middle-aged
men with heart trouble. If the 8,341 subjects, 5,552 were assigned at
random to the drug groups and 2,789 to the control group. The drugs
and the placebo (lactose) were administered in identical capsules.
The patients were followed for 5 years.
One of the drugs on the test was clofibrate, which reduces the levels
of cholesterol in the blood. Unluckily, this treatment did not save any
lives.
About 20% of the clofibrate group died over the period of follow up,
compared to 21% of the control group. A possible reason for this
failure was suggested — many subjects in the clofibrate group did
not take their medicine.
Subjects who took more than 80% of their prescribed medicine (or
placebo) were called “adheres” to the protocol. For the clock rate
group, the 5-year mortality rate among the adheres was only 15%,
compared to 25% among the non-adheres (Table 1). This looks like
strong evidence for the effectiveness of the drug. However, caution is
in order. This particular comparison is observational not experimental
— even though the data were collected while an experiment was
going on.
Many adherents were different from non-adheres in other ways,
besides the amount of the drug they took.
To find out, the investigators compared adheres and non-adheres in
the control group. Remember, the experiment was double-blind. The
controls did not know whether they were taking an active drug or the
placebo; neither did the subjects in the clofibrate group. The
psychological basis for adherence was the same in both groups.
In the control group too, the adheres did better. Only 15% of them
died during the 5-year period, compared to 28% among the
non-adheres. The conclusion:
- clofibrate does not have an effect.
- Adheres are different from non-adheres.
Let us take a look at an example about gender bias in graduate
admissions.
Next, we will look at an observational study from Graduate Division at
the University of California, Berkeley.
This is a study on gender bias in admissions by the Graduate
Division at the University of California, Berkeley. During the
investigation period, there were 8,442 men who applied for admission
to graduate school and 4,321 women. About 44$ of the men and
35% of the women were admitted. Taking percents adjusts for the
difference in numbers of male and female applicants: 44 out of every
100 men were admitted, and 35 out of every 100 women were
admitted.
It seems like this university prefer men, 44 to 35. Is this conclusion
correct?
Each major did its own admissions to graduate work. By looking at
them separately, the university should have been able to identify the
ones which discriminated against the women. At that point, a puzzle
appeared.
Major by major, there did not seem to be any bias against women.
Some majors favored men, but some favored women. On the whole,
if there was any bias, it ran against the men. What was going on?
Over a hundred majors were involved. However, the six largest majors
together accounted for over one-third of the total number of applicants to
the campus. And the pattern for these majors was typical of the whole
campus. Table 2 shows the number of male and female applicants, and
the percentage admitted, for each of these majors.
In each major the percentage of female applicants who were admitted is
roughly equal to the percentage for male applicants. The only exception
is major A, which appears to discriminate against men. It admitted 82% of
the women but only 62% of the men. The department that looks most
biased against women is E. It admitted 28% of the men and 24% of the
women. This difference only amounts to 4 percentage points. However,
when all six majors are taken together, they admitted 44% of the male
applicants, and only 30% of the females. The difference is 14 percentage
points.
This seems paradoxical, what happened?
1) The first two majors were easy to get into. Over 50% of the men applied
to these two majors.
2) The other four majors were much harder to get into. Over 90% of the
women applied to these four majors.
The men were applying to the easy majors, the women to the harder ones.
There was an effect due to the choice of major, confounded with the effect
due to gender.
The statistical lesson: relationships between percentages in subgroups can be
reversed when the subgroups are combined. This is called Simpson’s
paradox.
Since the original summarized results in Table 2 is confounded with the effect
due to gender, the table is difficult to make meaningful conclusion.
Alternatively, a statistician might summarize the above table by computing
one overall admissions rate for men and another for women, but adjusting for
the gender difference in application rates. The procedure would be to take
some kind of average admission rate separately for the men and women. An
ordinary average ignores the differences in size among the departments.
Instead, a weighted average of the admission rates could be used, the
weights being the total number of applicants (male and female) to each
department, see the next slide.
The above results come out to be 39%. We can do the same math about the
weighted average admission rate for the women to be
The weighted average admission rate for the women works out to be 43%.
Hence, the final conclusion is: The weighted average admission rate for men
is 39%, while the weighted average admission rate for women is 43%. The
weighted averages control for the confounding factor — choice of major.
These averages suggest that if anything, the admissions process is biased
against the men!