Improved Learning in a Large-Enrollment Physics Class

Louis Deslauriers et al.

Science 332, 862 (2011);
DOI: 10.1126/science.1201783

This copy is for your personal, non-commercial use only.

If you wish to distribute this article to others, you can order high-quality copies for your
colleagues, clients, or customers by clicking here.

Downloaded from [Link] on October 8, 2012

Permission to republish or repurpose articles or portions of articles can be obtained by
following the guidelines here.

The following resources related to this article are available online at

[Link] (this information is current as of October 8, 2012 ):

Updated information and services, including high-resolution figures, can be found in the online
version of this article at:
[Link]
Supporting Online Material can be found at:
[Link]
A list of selected additional articles on the Science Web sites related to this article can be
found at:
[Link]
This article cites 14 articles, 1 of which can be accessed free:
[Link]
This article has been cited by 4 articles hosted by HighWire Press; see:
[Link]
This article appears in the following subject collections:
Education
[Link]

Science (print ISSN 0036-8075; online ISSN 1095-9203) is published weekly, except the last week in December, by the
American Association for the Advancement of Science, 1200 New York Avenue NW, Washington, DC 20005. Copyright
2011 by the American Association for the Advancement of Science; all rights reserved. The title Science is a
registered trademark of AAAS.
REPORTS
Acknowledgments. This work was supported by the Belgian Postdoctoral Fellows, and S.L. Senior Research SOM Text
Fonds National de la Recherche Scientifique (FNRS), Associate at the FNRS. M.I.G., V.L., and K.F. are Fig. S1
European Commission, Mind Science Foundation, supported by the Wellcome Trust. Table S1
McDonnell Foundation, French-Speaking Community References
Concerted Research Action (ARC 06/11-340), Fondation Supporting Online Material
Léon Frédéricq, and National Institutes of Health. [Link]/cgi/content/full/332/6031/858/DC1 22 December 2010; accepted 6 April 2011
M.-A.B. and O.G. are Research Fellows, M.B. and C.S. Materials and Methods 10.1126/science.1202043

Improved Learning in a Large-Enrollment from any particular practice but rather from the
integration into the overall deliberate practice
framework.
Physics Class This study was carried out in the second term
of the first-year physics sequence taken by all
undergraduate engineering students at the Uni-
Louis Deslauriers,1,2 Ellen Schelew,2 Carl Wieman*†‡
versity of British Columbia. This calculus-based
course covers various standard topics in electric-
We compared the amounts of learning achieved using two different instructional approaches
ity and magnetism. The course enrollment was
under controlled conditions. We measured the learning of a specific set of topics and
850 students, who were divided among three
objectives when taught by 3 hours of traditional lecture given by an experienced highly rated

Downloaded from [Link] on October 8, 2012

sections. Each section had 3 hours of lecture per
instructor and 3 hours of instruction given by a trained but inexperienced instructor using
week. The lectures were held in a large theater-
instruction based on research in cognitive psychology and physics education. The comparison
style lecture hall with fixed chairs behind benches
was made between two large sections (N = 267 and N = 271) of an introductory undergraduate
grouping up to five students. The students also had
physics course. We found increased student attendance, higher engagement, and more than
weekly homework assignments, instructional labo-
twice the learning in the section taught using research-based instruction.
ratories, and tutorials and recitations where they
solved problems; this work was graded. There were
he traditional lecture approach remains the The deliberate practice concept encompasses the two midterm exams and a final exam. All course

T prevailing method for teaching science at

the postsecondary level, although there
are a growing number of studies indicating that
educational ideas of constructivism and formative
assessment. In our case, the deliberate practice takes
the form of a series of challenging questions and
components were common across all three sections,
except for the lectures, which were prepared and
given independently by three different instructors.
other instructional approaches are more effective tasks that require the students to practice physicist- During week 12, we studied two sections
(1–8). A typical study in the domain of physics like reasoning and problem solving during class whose instructors agreed to participate. For the
demonstrates how student learning is improved time while provided with frequent feedback. 11 weeks preceding the study, both sections were
from one year to the next when an instructor The design goal was to have the students taught in a similar manner by two instructors (A
changes his or her approach, as measured by stan- spend all their time in class engaged in deliberate and B), both with above average student teaching
dard concept-based tests such as the Force Con- practice at “thinking scientifically” in the form of evaluations and many years experience teaching
cept Inventory (9) or the instructor’s own exams. making and testing predictions and arguments this course and many others. Both instructors lec-
In our studies of two full sessions of an advanced about the relevant topics, solving problems, and tured using PowerPoint slides to present con-
quantum mechanics class taught either by tra- critiquing their own reasoning and that of others. tent and example problems and also showed
ditional or by interactive learning style, students All of the activities are designed to fit together demonstrations. Meanwhile, the students took
in the interactive section showed improved learn- to support this goal, including moving the sim- notes. “Clicker” (or “personal response system”)
ing, but both sections, interactive and traditional, ple transfer of factual knowledge outside of class questions (average 1.5 per class, range 0 to 5)
showed similar retention of learning 6 to 18 months as much as possible and creating tasks and feed- were used for summative evaluation (which was
later (10). Here, we compare learning produced back that motivate students to become fully en- characterized by individual testing without dis-
by two contrasting instructional methods in a gaged. As the students work through these tasks, cussion or follow-up other than a summary of the
large-enrollment science course. The control group they receive feedback from fellow students (12) correct answers). Students were given participa-
was lectured by a motivated faculty member with and from the instructor. We incorporate multi- tion credit for submitting answers.
high student evaluations and many years of experi- ple “best instructional practices,” but we believe Before the experiment, a variety of data were
ence teaching this course. The experimental group the educational benefit does not come primarily collected on the students in the two sections
was taught by a postdoctoral fellow using instruc-
tion based on research on learning. The same
selected learning objectives were covered by both Table 1. Measures of student perceptions, behaviors, and knowledge.
instructors in a 1-week period.
The instructional design for the experimental Control section Experimental section
section was based on the concept of “deliberate Number of students enrolled 267 271
practice” (11) for the development of expertise. Mean BEMA score (13) (week 11) 47 T 1% 47 T 1%
Mean CLASS score (14) (start of term) 63 T 1% 65 T 1%
1 (agreement with physicist)
Carl Wieman Science Education Initiative, University of British
Columbia, Vancouver, BC, Canada. 2Department of Physics and Mean midterm 1 score 59 T 1% 59 T 1%
Astronomy, University of British Columbia, Vancouver, BC, Canada. Mean midterm 2 score 51 T 1% 53 T 1%
*On leave from the University of British Columbia and the Attendance before experiment* 55 T 3% 57 T 2%
University of Colorado. Attendance during experiment 53 T 3% 75 T 5%
†To whom correspondence should be addressed. E-mail: Engagement before experiment* 45 T 5% 45 T 5%
gilbertwieman@[Link]
‡This work does not necessarily represent the views of the
Engagement during experiment 45 T 5% 85 T 5%
Office of Science and Technology Policy or the United States *Average value of multiple measurements carried out in a 2-week interval before the experiment. Engagement also varies over
government. location in the classroom; numbers given are spatial and temporal averages.

862 13 MAY 2011 VOL 332 SCIENCE [Link]

 REPORTS
(Table 1). Students took two midterm exams (iden- with student-student discussion (CQ), small-group had not taught this class before and were not
tical across all sections). In week 11, students took active learning tasks (GT), and targeted in-class familiar with the students. Before the first class,
the Brief Electricity and Magnetism Assessment instructor feedback (IF). Before each of the three they solicited two volunteers enrolled in the course
(BEMA), which measures conceptual knowledge 50-min classes, students were assigned a three- or to pilot-test the materials. The volunteers were
(13). At the start of the term, students took the four-page reading, and they completed a short true- asked to think aloud as they reasoned through the
Colorado Learning Attitudes about Science Survey false online quiz on the reading. To avoid student planned questions and tasks. Results from this
(CLASS) (14), which measures a student’s per- resistance, at the beginning of the first class, several testing were used to modify the clicker questions
ceptions of physics. During weeks 10 and 11, we minutes were used to explain to students why the and tasks to reduce misinterpretations and adjust
measured student attendance and engagement in material was being taught this way and how the level of difficulty. This process was repeated
both sections. Attendance was measured by count- research showed that this approach would increase before the second class with one volunteer.
ing the number of students present, and engage- their learning. During the week of the experiment, engage-
ment was measured by four trained observers in A typical schedule for a class was the following: ment and attendance remained unchanged in the
each class using the protocol discussed in the CQ1, 2 min; IF, 4 min; CQ2, 2 min; IF, 4 min; CQ2 control section. In the experimental section, student
supporting online material (SOM) (15). The results (continued), 3 min; IF, 5 min; Revote CQ2, 1 min; engagement nearly doubled and attendance in-
show that the two sections were indistinguishable CQ3, 3 min; IF, 6 min; GT1, 6 min; IF with a creased by 20% (Table 1). The reason for the
(Table 1). This in itself is interesting, because the demonstration, 6 min; GT1 (continued), 4 min; attendance increase is not known. We hypothe-
personalities of the two instructors are rather dif- and IF, 3 min. The time duration for a question or size that of the many students who attended only
ferent, with instructor A (control section) being activity includes the amount of time the students part of a normal class, more of them were captured

Downloaded from [Link] on October 8, 2012

more animated and intense. spent discussing the problem and asking numer- by the happenings in the experimental section and
The experimental intervention took place dur- ous questions. There was no formal lecturing; decided to stay and to return for the subsequent
ing the 3 hours of lecture in the 12th week. Those however, guidance and explanations were provided classes.
classes covered the unit on electromagnetic waves. by the instructor throughout the class. The instructor The test was administered in both sections in
This unit included standard topics such as plane responded to student-generated questions, to results the first class after the completion of the 3-hour
waves and energy of electromagnetic waves and from the clicker responses, and to what the in- unit. The control section had covered the material
photons. The control section was taught by in- structor heard by listening in on the student- related to all 12 of the questions on the test. The
structor A using the same instructional approach student discussions. Students’ questions commonly experimental section covered only 11 of the 12
as in the previous weeks, except they added in- expanded upon and extended the material covered questions in the allotted time. Two days before
structions to read the relevant chapter in the text- by the clicker questions or small-group tasks. The the test was given, the students in both sections
book before class. The experimental section was material shown on the slides used in class is given were reminded of the test and given links to the
taught by two instructors who had not previously in the SOM, along with some commentary about postings of all the material used in the experi-
taught these students. The instructors were the the design elements and preparation time required. mental section: the preclass reading assignments
first author of this paper, L.D., assisted by the At the beginning of each class, the students and quizzes; the clicker questions; and the group
second author, E.S. Instructor A and L.D. had were asked to form groups of two. After a clicker tasks, along with answers to all of these. The
agreed to make this a learning competition. L.D. and question was shown to the class, the students students were encouraged by e-mail and in class
instructor A agreed beforehand what topics and discussed the question within their groups (which to try their best on the test and were told that it
learning objectives would be covered. A multiple- often expanded to three or more students) and would be good practice for the final exam, but their
choice test (see SOM) was developed by L.D. submitted their answer using clickers. When the performance on the test did not affect their course
and instructor A that they and instructor B agreed voting was complete, the instructor showed the grade. Few students in either section finished in less
was a good measure of the learning objectives results and gave feedback. The small-group tasks than 15 min, with the average being about 20 min.
and physics content. The test was prepared at the were questions that required a written response. The test results are shown in Fig. 1. For the
end of week 12. Most of the test questions were Students worked in the same groups but submitted experimental section, 211 students attended class
clicker questions previously used at another individual answers at the end of each class for to take the test, whereas 171 did so in the control
university, often slightly modified. Both sections participation credit. Instructor A observed each of section. The average scores were 41 T 1% in the
were told that they would receive a bonus of 3% these classes before teaching his own class and control section and 74 T 1% in the experimental
of the course grade for the combination of par- chose to use most of the clicker questions devel- section. Random guessing would produce a score
ticipating in clicker questions, taking the test, and oped for the experimental class. However, Instruc- of 23%, so the students in the experimental sec-
(only in the experimental section) turning in group tor A used these only for summative evaluation, tion did more than twice as well on this test as
task solutions, with the apportionment of credit as described above. those in the control section.
across these tasks left unspecified. L.D. and E.S. together designed the clicker The test score distributions are not normal
In contrast to instructor A, the teaching experi- questions and small-group tasks. L.D. and E.S. (Fig. 1). A ceiling effect is apparent in the experi-
ence of L.D. and E.S. had been limited to serving
as teaching assistants. L.D. was a postdoctoral re- Fig. 1. Histogram of student
searcher working in the Carl Wieman (third author scores for the two sections.
of this paper) Science Education Initiative (CWSEI)
and had received training in physics education
and learning research and methods of effective
pedagogy while assisting with the teaching of six
courses. E.S. had a typical physics graduate student
background except for having taken a seminar
course in physics education.
The instructional approach used in the experi-
mental section included elements promoted by
CWSEI and its partner initiative at the University
of Colorado: preclass reading assignments, pre-
class reading quizzes, in-class clicker questions

[Link] SCIENCE VOL 332 13 MAY 2011 863

REPORTS
mental section. The two distributions have little A concern frequently voiced by faculty as they Science, Technology, Engineering, and Mathematics (STEM)
overlap, demonstrating that the differences in consider adopting active learning approaches is Education Project, The National Academies Board on Science
Education, 2008]. [Link]/bose/
learning between the two sections exist for es- that students might oppose the change (18). A Froyd_Promising_Practices_CommissionedPaper.pdf
sentially the entire student population. The stan- week after the completion of the experiment and 4. J. E. Froyd, “Evidence for the efficacy of student-active
dard deviation calculated for both sections was exam, we gave students in the experimental sec- learning pedagogies” (Project Kaleidoscope, 2007).
about 13%, giving an effect size for the difference tion an online survey (see SOM); 150 students [Link]/documents/[Link]
5. R. R. Hake, Am. J. Phys. 66, 64 (1998).
between the two sections of 2.5 standard de- completed the survey. 6. J. K. Knight, W. B. Wood, Cell Biol. Educ. 4, 298
viations. As reviewed in (4), other science and For the survey statement “I really enjoyed the (2005).
engineering classroom studies report effect sizes interactive teaching technique during the three 7. M. Prince, J. Eng. Educ. 93, 223 (2004).
less than 1.0. An effect size of 2, obtained with lectures on E&M waves,” 90% of the respon- 8. L. Springer, M. E. Stanne, S. S. Donavan, Rev. Educ. Res.
69, 21 (1999).
trained personal tutors, is claimed to be the largest dents agreed (47% strongly agreed, 43% agreed) 9. D. Hestenes, M. Wells, G. Swackhamer, Phys. Teach. 30,
observed for any educational intervention (16). and only 1% disagreed. For the statement “I feel I 141 (1992).
This work may obtain larger effect sizes than would have learned more if the whole physics 10. L. Deslauriers, C. Wieman, Phys. Rev. ST Phys. Educ. Res.
in this previous work because of the design and 153 course would have been taught in this high- 7, 010101 (2011).
11. K. A. Ericsson, R. Krampe, C. Tesch-Romer, Psychol. Rev.
implementation that maximized productive en- ly interactive style.” 77% agreed and only 7%
100, 363 (1993).
gagement. The clicker questions and group tasks disagreed. Thus, this form of instruction was 12. M. K. Smith et al., Science 323, 122 (2009).
were designed not only to require explicit expert well received by students. 13. L. Ding, R. Chabay, B. Sherwood, R. Beichner, Phys. Rev.
reasoning but also to be sufficiently interesting In conclusion, we show that use of deliberate ST Phys. Educ. Res. 2, 010105 (2006).
14. W. K. Adams et al., Phys. Rev. ST Phys. Educ. Res. 2,

Downloaded from [Link] on October 8, 2012

and personally relevant to motivate students to practice teaching strategies can improve both
010101 (2006).
fully engage. Another factor could be that pre- learning and engagement in a large introductory 15. Materials and methods are available as supporting
vious work primarily used end-of-term tests, and physics course as compared with what was ob- material on Science Online.
the results on those tests reflect all the learning tained with the lecture method. Our study com- 16. B. Bloom, Educ. Res. 13, 4 (1984).
that students do inside and outside of class, for pares similar students, and teachers with the same 17. R. H. Bauernfeind, C. J. Olson, Phi Delta Kappan 55, 271
(1973).
example, the learning that takes place while doing learning objectives and the same instructional 18. G. K. Allen, J. F. Wedman, L. C. Folk, Innovative High.
homework and studying for exams. In our inter- time and tests. This result is likely to generalize to Educ. 26, 103 (2001).
vention, the immediate low-stakes test more direct- a variety of postsecondary courses. Acknowledgments: This work was supported by the University
ly measured the learning achieved from preclass of British Columbia through the Carl Wieman Science
Education Initiative.
reading and class itself, in the absence of sub-
sequent study. References and Notes
1. R. J. Beichner et al., in Research-Based Reform of Supporting Online Material
We are often asked about the possible con- University Physics, E. F. Redish, P. J. Cooney, Eds. [Link]/cgi/content/full/332/6031/862/DC1
tributions of the Hawthorne effect, where any (American Association of Physics Teachers, College Park, Materials and Methods
change in conditions is said to result in improved MD, 2007). SOM Text
performance. As discussed in citations in the SOM, 2. C. H. Crouch, E. Mazur, Am. J. Phys. 69, 970 References
(2001).
the original Hawthorne plant data actually show 3. J. E. Froyd, “White paper on promising practices in
no such effect, nor do experiments in educational undergraduate STEM education” [Commissioned paper for 16 December 2010; accepted 5 April 2011
settings (17). the Evidence on Promising Practices in Undergraduate 10.1126/science.1201783

864 13 MAY 2011 VOL 332 SCIENCE [Link]