Using Student Performance Predictions in a Computer

Science Curriculum
A.T. Chamillard
Computer Science Department
University of Colorado at Colorado Springs
Colorado Springs, CO 80933-7150
719-262-3150
chamillard@[Link]

ABSTRACT is certainly useful for some purposes, it is also reasonable to use

Professors often develop anecdotal guidelines about how each more formal statistical analysis techniques to develop models to
student’s past performance in their academic major relates to their help predict student performance and to identify key courses.
performance in later courses. While these guidelines can be Prior efforts to predict student performance in various majors and
useful, a more formal statistical analysis of these relationships can institutions are documented in the literature. Thomas examined
provide valuable insight into predicted student performance, performance prediction in an introductory physics course [11],
which can help professors guide their students to focus on indicating that prior performance is more effective for predicting
potential areas of difficulty. In addition, such analyses can student performance in that course than a diagnostic test. Felder et
identify which courses are key indicators of later performance in al [7] predicted performance in an introductory chemical
the major. This additional insight into the relationships between engineering course using a variety of factors, including
the courses in the curriculum can help professors implement performance in previous courses. Other researchers have built and
curriculum changes and measure the effects of those changes. In validated models predicting introductory computing aptitude [8]
this paper, we present the results of such an analysis for computer and used high school performance data and other demographic
science majors at the U.S. Air Force Academy. factors to predict introductory programming performance [2, 3,
10] and retention in the computer science major [4]. The results
Categories and Subject Descriptors presented here differ from previous performance prediction
efforts because they include formal predictive models for each
K.3.2 [Computers and Education]: Computer and Information
required class in the computer science curriculum and provide
Science Education – computer science education, curriculum.
insights that span the entire computer science curriculum rather
than focusing on a selected course or class year.
General Terms In addition to offering some predictive capabilities, the results of
Measurement, Performance, Experimentation.
a formal statistical analysis can also provide educators with
insight into the relationships between the courses in the
Keywords curriculum. This insight can facilitate curriculum modification
Student performance, performance prediction, predictive efforts in a number of ways. For example, given limited resources
modeling, computer science curriculum. to accomplish curriculum changes, educators can focus their
efforts on those courses that have or should have the strongest
1. INTRODUCTION effect on student performance in later courses. As curriculum
changes are implemented, changes in the statistical relationships
Predicting student performance in a particular course, or even on
between student performance in different courses can be
assessments within a course, is a difficult but useful undertaking.
monitored to ensure the desired effects are achieved.
Given such predictions, a professor can help focus student effort
on potential problem areas for particular students given their In this paper, we present the results of a formal statistical analysis
performance in previous courses. In addition, curriculum using data for 285 computer science majors spanning 7 class
committees can use prediction results to guide changes to the years at the U.S. Air Force Academy, called USAFA hereafter.
curriculum and evaluation of the effects of those changes. While some of the specific insights resulting from this analysis
may only be applicable to the computer science major at USAFA,
Typically, student performance predictions and identification of
the analysis approach, use of the results, and implementation
the key courses in an academic major are based on anecdotal
issues are more generally applicable to both other academic
evidence (if they are developed at all). While anecdotal evidence
disciplines and other educational institutions.

Permission to make digital or hard copies of all or part of this work for
The following sections briefly discuss the required courses
personal or classroom use is granted without fee provided that copies are included in the computer science major at USAFA, present our
not made or distributed for profit or commercial advantage and that copies analysis approach, provide the results of our statistical analysis,
bear this notice and the full citation on the first page. To copy otherwise, or discuss how the results can be used in practice, and present our
republish, to post on servers or to redistribute to lists, requires prior specific conclusions.
permission and/or a fee.
ITiCSE’06, June 26–28, 2006, Bologna, Italy.
Copyright 2006 ACM 1-59593-055-8/06/0006...$5.00.

260
2. REQUIRED COURSES gaining further insight into the relationships between computer
In this section we present the required major’s courses taken by science courses and providing a mechanism for monitoring the
computer science majors at USAFA. Students also take optional impact of curriculum changes.
courses to complete the major, but we only consider required 3.1 Building the Models
computer science courses in our analysis here. The computer Linear regression models can be used as approximations of the
science program at USAFA was CSAB-accredited during the functional relationship between a predicted value and a set of
period under analysis (it is now accredited by ABET), and topic predictor variables [9]. We used linear regression to build our
coverage closely follows ACM curriculum guidelines [1]. predictive models for each assessment and overall performance in
We note that the content and sequence of major’s courses at each course.
USAFA is continually examined and modified as necessary. The In linear regression, the predictive model is of the form
courses listed here were taken by the students included in the
dataset, but the course contents, the semester, and even the year in y=β + β x + K + β x , where y is the predicted value,
0 1 1 i i
which they are taken are subject to change as the major evolves.
A summary of the required courses is provided in Table 1. each x i is a predictor variable, and each β i is a coefficient
In general, all students in the dataset were required to take all of calculated using linear regression. The regression coefficients are
the above courses. There is one exception to this rule, however: calculated using a linear least squares fit to the data. We note that,
CS 463, the Introduction to Database Management course, which in general, larger coefficients (positive or negative) indicate a
was only a required course for 4 out of the 7 class years included stronger predictive effect from the associated predictor variable.
in the dataset.
3.2 Analyzing the Models
3. ANALYSIS APPROACH To determine how well a predictive model fits the data used, we
In this section we describe the statistical analysis techniques we need some measure of how well the model captures the variance
used to analyze performance data and to generate our predictive in the data. For linear regression, the standard measure of fit is the
models. We present our technique for using linear regression to 2
Multiple Correlation Coefficient Squared, R , which measures
generate the predictive models, discuss our analysis of the how much of the variance in the predicted value is captured by
resulting models and their associated parameters, and describe the 2
correlations we generated for further analysis. the predictive model. R ranges from 0.0 to 1.0, with a
magnitude near 1.0 indicating that the model explains most of the
For the purposes of this paper, course assessments are defined as variance in the data, which in turn implies that the model provides
graded activities in the course, such as programming assignments good predictive power.
and exams. Student performance on the course assessments is
defined as the student’s overall percentage on those assessments. The linear regression technique we used assumes that the errors
Overall student performance in a particular course is captured by (e.g., residuals) in the model are independent, have zero mean,
both the student’s overall course percentage and the letter grade constant variance, and follow a normal distribution [6]. These
the student earned in the course. assumptions can be checked using plots of the standardized
residuals against the predicted response variables.
The primary goal of our statistical analysis was to develop
predictive models for each of the course assessments and the The structure we would expect to find in these plots, given our
overall student performance in each course using student assumptions about the errors in the model, is essentially a
performance in prior courses as predictor variables. We can then horizontal line with residual values scattered randomly above and
use those predictive models to predict student performance in a below zero. If we find that the “spread” of the residuals increases
particular course so we can guide students to focus their studies or decreases as the value of the response or predictor variable
on specific potential problem areas. Additional goals included increases, we should suspect that the variance is not constant.
While more formal statistical techniques have been proposed for
Table 1. Required Computer Science Courses
Course Course Title Semester Year
CS 225 Fundamentals of Computer Science (CS1) Fall Sophomore
CS 359 Programming Paradigms Spring Sophomore
CS 326 Foundations of Computer Science Fall Junior
CS 351 Computer Organization and Architecture I Fall Junior
CS 356 Computer Organization and Architecture II Spring Junior
CS 380 Algorithms and Data Structures Spring Junior
CS 453 Software Engineering I Fall Senior
CS 463 Introduction to Database Management Fall Senior
CS 483 Operating Systems Fall Senior
CS 454 Software Engineering II Spring Senior
CS 467 Computer Networks Spring Senior

261
checking these assumptions, visual inspection of the residual plots grade data from transcripts provided by the registrar, but those
is generally sufficient for recognizing serious violations of the transcripts do not contain detailed course data.
assumptions [6]. To summarize, when predicting performance in each course we
used previous course grades as the predictor variables. We did,
3.3 Generating Correlations however, attempt to predict performance on all assessments in
Although the predictive models can yield valuable insight into the that course despite the fact that we typically did not have
relationships between courses in the major, it may also be complete course assessment data for all students in the dataset.
enlightening to measure these relationships directly. We can use
There were a number of courses for which none of the models
the Pearson correlation coefficient (r) to examine the linear
built for the assessments, overall percentage, or course grade had
relationship between two variables. The coefficient ranges from
reasonable predictive power. Those courses included
–1.0 to 1.0, with a coefficient magnitude close to 1.0 indicating a
Programming Paradigms (CS 359), Computer Organization and
strong linear relationship and a magnitude close to 0.0 indicating
Architecture II (CS 356), Introduction to Database Management
a weak linear relationship.
(CS 463), and Software Engineering I (CS 453) and II (CS 454).
To gain additional insight into the relationships between the
We were able to build one or more predictive models with
courses in the major, we calculated correlation coefficients for
reasonable predictive power for the following courses:
pairings of the course grades for each course. Analysis of those
Foundations of Computer Science (CS 326), Computer
correlation coefficients is provided in the following section.
Organization and Architecture I (CS 351), Algorithms and Data
Structures (CS 380), Operating Systems (CS 483), and Computer
4. ANALYSIS RESULTS Networks (CS 467). While space limitations preclude presentation
of the actual models, we note that we developed 7 models with
4.1 Dataset Description reasonable predictive power. Six of those models predicted course
The dataset used for the statistical analysis presented in this paper percentage or course grade, while the seventh model predicted
consists of 285 computer science majors from the Class of 1995 performance on the Operating Systems programming
through the Class of 2001. Of these 285 students, 175 students assignments.
graduated with a computer science degree; the other 110 students
either changed to a different major or left USAFA without
graduating. When building the predictive models for a particular
4.3 Correlation Results
course, we included all students who took that course, whether or While the predictive models discussed above provide some
not they ultimately graduated with a computer science degree. insight into the relationships between the courses in the major,
directly calculating correlation coefficients between the course
The dataset contains 117 measurements, including percentages for grades can also be enlightening. The resulting coefficients ranged
the assessments in each course, the overall percentage in each from 0.29 to 0.73, providing interesting insights into several pair-
course, and the course letter grade in each course. We encoded wise course relationships. We also note that if the relatively
the course grade using standard GPA (Grade Point Average) common anecdotal argument that "stronger students generally do
values for letter grades (A = 4.0, A- = 3.7, B+ = 3.3, etc.). well across the curriculum and weaker students generally do
poorly" were true, these coefficients would likely be higher;
4.2 Predicting Performance another argument for using statistical analysis rather than
For the predictive models we built, we only considered predictive anecdotal observation to evaluate student performance.
2 It is interesting to note that the strongest three correlations occur
models with an R greater than or equal to 0.500 as sufficiently
powerful to be potentially useful and, therefore, to merit further between courses that students take in the same semester. Students
discussion. None of the residual plots for any of the models take CS 351 and CS 326 (r=0.73) in the fall semester of their
discussed below contained patterns indicating violation of the junior year, CS 356 and CS 380 (r=0.72) in the spring semester of
linear regression assumptions. their junior year, and CS 463 and CS 483 (r=0.72) in the fall
semester of their senior year. We believe the correct interpretation
We did not develop any models for our CS1 course because there of this result is that students having a “good” semester do well in
are no previous courses to provide performance data. For the most or all of their computer science classes in that semester,
remaining 10 courses, we built 99 models, using the percentages while students having a “bad” semester do poorly in most or all of
for the course assessments (assignments, exams, etc.), overall their computer science classes in that semester. We note that,
course percentages, and course grades as the predicted values. For given the structured environment at USAFA, the semesters in
the predictor variables, we used only course grades from courses which particular courses are taken is a requirement, not a
completed prior to the course for which we were building suggestion.
predictive models. Our initial goal was to use all assessments for
prior courses in the models as well, but incomplete data precluded One of the weaker correlations (r=0.47) involves the two courses
that approach. Our linear regression approach requires that the in the software engineering capstone sequence (CS 453 and CS
data for a particular student contain all predictor variables from 454). This could be due, at least in part, to the fact that 50% of a
previous courses for that student to be included in the modeling, student’s grade in both CS 453 and CS 454 is based on a group
which would lead to significant data losses in the modeling. In project rather than on individual work. This grading policy
fact, we only had complete data for one student (out of 285) for reduces the effect that a particular student’s past performance
all the courses being analyzed. We were able to gather course would have on that student’s grade in these courses. It is
interesting to note that CS 453 and CS 454 are essentially treated
as a year long course in terms of material coverage and project

262
work [5], but there is only a weak correlation between the course assignments be completed before the students who are having
grades for these courses. Students are moved into different groups trouble can be identified. Using the predictive models lets us
between CS 453 and CS 454, however, so this weak correlation provide early focus to those students predicted to have problems
may also be indicative of the significant effect that group in this area rather than waiting until those students have
performance has on individual course grades in these courses. performed poorly on one or more assessments before providing
that focus.
5. USING THE RESULTS It is clear that there is some danger associated with using
predictive models to predict student performance in a course. One
5.1 Student Focus such danger is that an instructor who tells a student their precise
Most of the models with reasonable predictive power can be used expectations of that student's performance may negatively
to predict a student’s overall performance in a particular course influence the student's performance, particularly if those
given their performance in previous courses. For example, the expectations are for poor performance in the course or on
predictive model for the Overall Percentage in CS 483 (Operating particular course assessments. Rather than providing a predicted
2 grade to each student at the beginning of the course – such a
Systems) had the strongest predictive power, with R = 0.579
(see Figure 1). A student with a C- (1.7) in CS 380 (Algorithms prediction could even be viewed by students as a "contract" for a
and Data Structures), CS 356 (Computer Organization and particular grade – it would be more prudent for instructors to use
Architecture II), and CS 326 (Foundations of Computer Science) the predicted grades to identify the students who may need to
would have a predicted CS 483 percentage of 64%. This is almost expend extra effort on the course or particular assignments in the
certainly a failing grade (D) in the course, so the student could be course. Those students can then be approached for a general
warned that, based on their past performance, additional effort discussion about their predicted performance in the course
may be required to successfully complete the course. We note that ("Based on your grades in CS 356, CS 380, and CS 359, you'll
using only the information that the student earned C- grades in the probably have to spend some extra time on this course") rather
3 previous courses, both the student and the instructor might than with a specific grade prediction ("Based on your grades in
believe that this student could expect a C- from this course as CS 356, CS 380, and CS 359, you'll probably get a D in this
well. The predictive model, however, shows that the student is at course"). While this approach doesn't completely alleviate
higher risk for failure in the course than anecdotal examination of concerns about students living up (or down) to instructor
the previous course grade information indicates; although the expectations, it does provide a way to provide early focus to those
difference between C- and D is small, it is significant because it is students who may need it the most.
also the difference between passing and failing the course.
This information should not be used as a threat, of course;
5.2 Curriculum Changes
The statistical results from this dataset and, more generally, from
instead, it should be used to help motivate the student toward
the overall analysis approach can also be used to help identify
expending additional effort to try to succeed in the course. This
potential curriculum changes and to measure the effects of those
same technique can be used for the other predictive models
changes.
discussed above.
The predictive models we generated identified several “key
Predictive models can be used even more effectively to help
courses” in the major; specifically, those that seem to provide the
guide student efforts when the predicted variable is a course
strongest predictive effect in the predictive models for later
assessment rather than an overall course grade. For example, the
courses. In the 7 models discussed above, the CS 359
predictive model for the CS 483 programming percentage (see
(Programming Paradigms) course grade appears as a statistically
Figure 1) indicates that CS 356 (Computer Organization and
significant predictor variable in 6 of those models: twice as
Architecture II), CS 380 (Algorithms and Data Structures), and
providing the most predictive effect, 3 times as providing the
CS 359 (Programming Paradigms) grades are significant
second-strongest effect, and once providing the third-strongest
predictors of performance in this course assessment. A CS 483
effect. The CS 225 (CS1) course grade appears as a statistically
student who did poorly in one or more of those classes could
significant predictor variable in 3 of the models: twice as
therefore be encouraged to expend additional effort on the
providing the most predictive effect and once as providing the
programming assignments.
second-strongest effect. All other course grades appeared in three
Continuing our CS 483 programming percentage example, it can or fewer of the models, with smaller predictive effects. While the
certainly be argued that effective instructors will notice through fact that CS 359 and CS 225 are sophomore classes indicates that
course assessment mechanisms which students are having trouble they were included in the modeling efforts for all junior and
with the programming assignments in the course and can offer senior courses, thereby giving us some expectation that they
suggestions to focus the efforts of those students as necessary. would appear in more predictive models, we note that their
The key distinction between that approach and using predictive predictive power in those models indicates that they are key
models is that the predictive models can be used to provide this courses in the computer science major. This is consistent with our
focus to the students at the beginning of the course, while the intuition about those courses; because they are the first two
latter approach requires that one or more programming courses taken by all computer science majors, these courses are

CS 483% = 4.900 * CS 380Grade + 3.982 * CS 356Grade + 2.530 * CS 326Grade + 44.684

CS 483 Programming % = 8.395 * CS 356Grade + 8.077 * CS 380Grade + 7.836 * CS 359Grade + 9.431
Figure 1. CS 483 (Operating Systems) Predictive Models

263
structured to cover foundational computer science topics that we Major’s GPA or final standing among the majors in the same
expect the students to need and use throughout the major. class year. It may also be useful to include other predictor
Our correlation results also yielded valuable information about variables (SAT or ACT scores, high school GPA, and so on) as
the relationships between the courses in the curriculum. For predictor variables in these models. Given predictions based on
example, we were surprised to find that the correlation between such models, we could provide students with sound feedback
the Computer Organization and Architecture I and II grades was early in the major.
relatively weak. This result is somewhat counterintuitive because The analysis approach and observations presented here are not
these courses represent a two-course sequence on computer limited to either the computer science major or to USAFA; in
organization and architecture. We can use these results as one fact, the author is now implementing this approach at a different
indication that the two courses may not be as tightly integrated as university. The approach is sufficiently general that it can be
we originally intended. We can therefore make curricular changes applied to any major at any institution, and other researchers
to tie these two courses more tightly together, re-accomplish our could conduct similar analyses to provide additional observations
statistical analysis after the changes have been implemented, and and recommendations.
examine the resulting relationship to gain insight into the
effectiveness of our changes (a higher correlation between the 7. REFERENCES
grades implies a stronger connection between the courses). [1] ACM/IEEE-CS Joint Curriculum Task Force. Computing
As indicated above, the predictive models and the correlation Curricula 1991. ACM Press, New York, NY, 1991.
coefficients are expected to change as curriculum changes are [2] Butcher, D.F., and Muth, W.A. Predicting performance in an
implemented (and as the dataset grows over time as well). Given introductory computer science course. Communications of
the dynamic nature of the computer science curriculum, it would the ACM, 28, 3 (Mar. 1985), 263-268.
be unreasonable to expect a static set of predictive models and
other statistical relationships using the approach described above. [3] Byrne, P., and Lyons, G. The effect of student attributes on
Instead, we envision an evolution of these relationships as success in programming. In Proceedings of the 6th Annual
curriculum changes occur and the size of the dataset increases. Conference on Innovation and Technology in Computer
After the initial analyses are completed, the predictive models and Science Education (ITiCSE 2001), Canterbury, UK, 49-52.
correlation coefficients can easily be updated on an annual basis. [4] Campbell, P.F., and McCabe, G.P. Predicting the success of
freshmen in the computer science major. Communications of
6. CONCLUSIONS the ACM, 27, 11 (Nov. 1984), 1108-1113.
Using student performance data from previous courses and [5] Chamillard, A.T., and Braun, K.A. The software engineering
predictive models that predict student performance in a particular capstone: Structure and tradeoffs. In Proceedings of the 33rd
course, professors can help focus student effort on potential SIGCSE Technical Symposium on Computer Science
problem areas in that course. Educators can also use this Education, Northern Kentucky, KY, February 27-March 3,
information to guide their implementation and evaluation of 2002, 227-231.
curriculum changes.
[6] Draper, N.R., and Smith, H. Applied Regression Analysis.
In this paper, we reported the use of statistical analysis techniques John Wiley & Sons, New York, NY, 1966.
to build such predictive models. While many of the generated
[7] Felder, R.M., Forrest, K.D., Baker-Ward, L., Dietz, E.J., and
models did not have sufficient predictive power to be useful, the
Mohr, P.H. A longitudinal study of engineering student
stronger models and other observations from the analysis provide
performance and retention: I. Success and failure in the
useful insight into the relationships between the various courses.
introductory course. Journal of Engineering Education, 82,
There are still numerous analyses that can also provide valuable (1993), 15-21.
information. While most of the models presented here use only
[8] Glorfeld, L.W., and Fowler, G.C. Validation of a model for
previous course grades as predictor variables, it is possible that
predicting aptitude for introductory computing. In The
more robust models and additional insights could be provided if
Papers of the Thirteenth SIGCSE Technical Symposium on
complete assessment data for each course were also available. To
Computer Science Education, Indianapolis, IN, 1982, 140-
gain these additional benefits, we suggest an ongoing data
143.
collection strategy where course assessment data for each course
is collected and archived every semester. This approach would [9] Montgomery, D.C., and Peck, E.A. Introduction to Linear
ensure that complete data is available for each student, providing Regression Analysis. John Wiley & Sons, New York, NY,
the opportunity to include assessment predictor variables in the 1982.
predictive models rather than limiting the predictor variables to [10] Taylor, H.G., and Mounfield, L.C. The effect of high school
previous course grades. As a separate but related extension, we computer science, gender, and work on success in college
could include elective courses in our analysis to evaluate the computer science. In Proceedings of the Twentieth SIGCSE
relationships between student performance in required and Technical Symposium on Computer Science Education,
elective courses and the assessments in those courses. Louisville, KY, 1989, 195-198.
It would also be interesting to determine whether or not it is [11] Thomas, E.W. Performance prediction and enhancement in
possible to predict probable success or failure in the major based an introductory physics course for engineers. Journal of
on performance in the early courses in the major. These Engineering Education, 82, (1993), 152-156.
predictions could simply predict “Yes” or “No” for graduation
with a computer science degree, or they could be used to predict

264