Computers & Education: X Reality 4 (2024) 100061

Contents lists available at ScienceDirect

Computers & Education: X Reality

journal homepage: [Link]/computers-and-education-x-reality

A Serious game for programming in higher education

Thomas Hainey *, Gavin Baxter
University of the West of Scotland, United Kingdom

A R T I C L E I N F O A B S T R A C T

Keywords: Programming is a highly difficult skill which is a constituent of many undergraduate programmes at Higher
Formative assessment Education (HE) level. With the advancement of games technology there is an increasing opportunity for edu­
Revision cators to provide innovative assessment tools for students on their courses which are highly immersive and
Computer game
graphically indicative of the times. This could potentially be in a supplementary capacity or to a greater extent
Serious game
Evaluation
inextricably linked to the learning outcomes and assessment outcomes. Notably serious games and Games-Based
Programming education Learning (GBL) have received high levels of attention from educationalists due to being motivational, novel
learning approaches. This paper will outline two empirical studies conducted to develop a game to teach pro­
gramming at HE level. The first study will gauge the acceptability of a computer game for teaching programming
and formulating content integration development requirements. The second study will outline the evaluation of
the developed game being placed in a module as a formative assessment tool to assist learners to revise for their
formal class test. Study one showed that acceptability of the game was high with 61 participants completing an
acceptability/content integration questionnaire. The game was designed to consolidate knowledge on rudi­
mentary and advanced programming concepts, data structures and algorithms. 48 participants evaluated the
game in study two with the results generally indicating that they enjoyed playing the game as a revision
alternative with 14% of participants rating it as very effective and 51% of participants as effective for allowing
them to prepare for their class test. The majority of participants also believed that games could be utilised in a
formative and summative assessment capacity on courses for independent study.

1. Introduction Boyle, & Wilson, 2016). There is however a lack of specific empirical
evidence in the serious games field in terms of game types/genres being
Computer Programming is a fundamental skill across many applied to different groups requiring particular pedagogical content
Computing Programmes at HE including: Computer Science, Software (Hays, 2005). Serious games are a potential solution to some of these
Engineering and Computer Games Development/Technology. Pro­ issues as they are regarded as a new innovative teaching approach
gramming is a highly useful skill that can lead to a very rewarding capable of engaging learners and revitalising interest in a difficult field
career, but two problematic issues exist: Firstly, programming is of study via immersive experiential learning. It has been noted that
perceived as a difficult, challenging skill to learn where lifetime devel­ research is lacking in terms of how computer games are used in teaching
opment and commitment is required taking almost a decade to turn a with regards to acceptance by learners and specifics regarding referred
novice programmer into an expert. Secondly, programming courses curricula and pedagogical content in terms of content integration
have some of the highest dropout rates where it is suggested that (Backland & Hendrix, 2013). Integration of content and a lack of
traditional teaching approaches such as lecturing, role-playing, lab tu­ development frameworks and models is a challenge for serious
torials and paper-based case-studies are not sufficiently effective in games/GBL in terms of validity. Nevertheless, educational institutions
suitably preparing graduates for employment. Serious games have been are encouraged to utilise novel, innovative, technological approaches to
empirically evaluated qualitatively or quantitatively in several areas engage learners, enhance the educational experience and improve
including: business, computing, social issues and health (Boyle et al., retention and pedagogical effectiveness.
2016; Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012) at various The academic literature has many examples of games for the pur­
levels of education including Primary Education (Hainey, Connolly, poses of teaching programming with a large number of GBL applications

* Corresponding author.
E-mail address: [Link]@[Link] (T. Hainey).

[Link]
Received 3 October 2023; Received in revised form 3 February 2024; Accepted 18 March 2024
Available online 18 April 2024
2949-6780/Crown Copyright © 2024 Published by Elsevier Ltd. This is an open access article under the CC BY license ([Link]
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

and games for the purposes of teaching aspects of Computer Science and Universities having a range of learners with vastly different previous
Software Engineering. A rich and diverse collection of games exist knowledge and experience. One additional problem is that traditional
including board games (Jordaan, 2018), online interactive tools (Ber­ teaching approaches such as lecturing, practical labs, paper-based
kling & Thomas, 2013), mobile phone applications (Boticki, Barisic, course works and role playing may not be sufficient to cater for every
Martin, & Drljevic, 2012), console games (Morsi, Richards, & Rizvi, type of learner in terms of preparation for careers in the computer
2010), multiplayer online games (Xiao & Miller, 2014), games-based games/software industry. The Scottish Government (2016) published a
construction environments (Doss, Juarez, Vincent, Doerschuk, & Liu, digital and learning teaching strategy report with the objectives of
2011), PC games (Connolly, Stansfield, & Hainey, 2007), collaborative ensuring the consideration of digital technology in curriculum and
or competitive gamification approaches (Sheth, Bell, & Kaiser, 2013), assessment delivery areas and to drive innovation in digital technology
games in collaborative virtual environments such as Second Life (Yap, for learning and teaching. Additionally, the Scottish Qualifications Au­
2011) and MMORPGs (Malliarakis, Satratzemi, & Xinogalos, 2014). thority (SQA) (2018) published a report written by young people across
While many of the games that are customised and specially imple­ Scotland setting out ideas and recommendations on the future of how
mented for teaching a highly specific topic, there are examples of assessment of qualifications take place where an identified key theme
extremely popular commercial games being adapted and utilised for was involving learners in the design of assessment and assessment sys­
teaching. For example: Super Mario Brothers to teach AVL Trees, rota­ tems. Lecturers are therefore encouraged at tertiary education level to
tions, balancing, searching, insertion and deletion of nodes (Wassila & use innovative digital technologies in teaching, learning and assessment
Tahar, 2012) and Bomberman for teaching procedural programming in C in the hope of improving student retention, engagement, pedagogical
(Chang, Chou, & Chen, 2010). While games in the literature cover a effectiveness and to maintain excellent and contemporary professional
broad and diverse range of topics in software engineering and computer practice. This study will investigate the acceptability and design stages
science. Some of the games specifically relating to programming edu­ of integration of games-based learning (GBL) or serious games (a
cation covered the following topics: particularly interactive digital technology) into a HE context for com­
puter programming education. The study will produce a framework to
• Object Oriented Programming Concepts - classes, inheritance and address the issues of integrating a computer game for programming
polymorphism (Lotfi & Mohammed, 2018; Alhazbi & Ismail, 2010; education into a real classroom context. This will be formulated by
Wong, Yatim, & Tan, 2015; Malliarakis et al., 2014). taking into account the problems associated with programming educa­
• Syntax, data types and algorithm design, variables, if-statements and tion and other applications that have been developed to address move
loops (Anderson & McLoughlin, 2007; Masso & Grace, 2011; Mal­ motivational learning utilising games and novel approaches. The re­
liarakis et al., 2014). quirements for the application will be derived from a survey method­
• Sequence, selection, iteration and functions/methods (Cagin, 2013; ology and the existing academic literature. A game will then be designed
Law, 2012) and developed from the first survey study and will be evaluated in a
• Programming logic and simple coding (Chang, 2010; Jiau, Chen, & classroom setting in an HE programming class to ascertain if it can be a
SSu, 2009; Finkelstein, Nickel, Harrison, Suma, & Barnes, 2009). useful formative assessment tool in a supplementary learning capacity.
• Algorithms and data structures (Browne & Anand, 2013; Chaffin,
Doran, Hicks, & Barnes, 2009; Lawrence, 2004; Shabana & Chen, 2. Research rationale, questions, methodologies and
2009; Wei, Chen, & Doong, 2009). developmental design
• Debugging code (Deitz & Buy, 2016; Miljanovic & Bradbury, 2017).
• Software and code visualisation (Marques, Levitt, & Nixon, 2012). 2.1. Research rationale
• Pointers (Karapinar, Zavarak, Senturk, Kara, & Erdogmus, 2012).
• Array and multi-dimensional array visualisation (Baker, Zhang, & University computing courses generally have at least one or perhaps
Caldwell, 2012). more programming modules and students tend to find these modules to
• Decomposition of complex programming tasks (Khenissi, Essalmi, & be challenging as programming has a reputation of being a difficult skill
Jemni, 2013). to learn. Year 1 and 2 of computing courses at university have the
highest levels of attrition (Chaffin et al., 2009; Jantan & Aljunid, 2012)
There are examples of a number of game-based construction and GBL with computer programming courses being a major contributory factor.
applications for the purposes of teaching programming indicating great Learners can struggle to grasp the most basic of programming concepts
graphical diversity and fidelity. Malliarakis et al. (2014) performed a and skills (Deitz and Buy (2016, p. 37). In an attempt to address this, this
review of a number of games including Catacombs (a 3D multiplayer study is a preliminary step in attempting to design and integrate a novel
game to teach variable declaration, nested conditionals, structures and digital teaching approach into a computer programming course based on
loops), Saving Sera (a 2D game to teach control structures and recur­ user acceptability and requirements at HE level and will be of interest to
sion), EleMental (a 3D game to teach recursion), Gidget (a web-based the following parties for the following reasons:
game to teach algorithm design and analysis) and Wu’s Castle (a 2D
game for teaching loops, arrays and parameters). Vahldick, Mendes, and • Educationalists in general as a framework will be presented on
Marcelino (2014) in their review of games to improve introductory integration of a new novel teaching and learning approach taking
computer programming competencies identified 40 games categorising into account content and assessment integration.
them based on the following criteria: type, platform, competence, topics • Computer scientists as the study investigates computer programming
and language. specifically identifying problematic areas of learning and where a
Teaching practical computer programming concepts in Higher Edu­ digital technology can be best applied for these areas.
cation (HE) can be a very challenging task for educationalists. There is a • GBL and serious game researchers as empirical evidence will be
diverse range of learners with regards to proficiency, confidence and presented in a practical application in terms of ascertaining re­
programming experience. These learners can come from Secondary quirements from learners and evaluating the serious game in terms of
Education (SE), Further Education (FE) institutions, be international or learning effectiveness.
mature learners. This problem is exacerbated by increasing pressures • Government and educational authorities specifically the Scottish
placed on universities in Scotland to move articulation levels from 49% Government and the Scottish Qualifications Authority (SQA) as the
to 75% in the next decade (Scottish Funding Council, 2016) meaning study will base the requirements for the GBL application on learner
that students can gain access to realistically the first 3 years of HE from a preferences.
college level route. This can lead to computer programming classes at

2
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

• Higher Education Institutions (HEIs) can use this study as a peda­ learners, what particular concepts the learners would like to see inte­
gogical blueprint for the integration of GBL into computer pro­ grated into that game and what type of game should be produced in
gramming education and to guide the incorporation of digital terms of genre, platform, and graphical fidelity. To answer these ques­
technologies into other courses potentially. tions, the most efficient method to obtain a large amount of data from
the participants was a survey/questionnaire methodology to perform
This paper will outline the full development cycle of a game to teach numerical analysis for potentially more generalizable results.
programming consisting of two empirical studies. Study one will gauge Research question 6 is attempting to ascertain the effectiveness of a
overall acceptability of a programming game and identify content and serious game for teaching programming concepts and to answer the
general requirements. Study two will allow participants to utilise the question of whether it can be used as a formative assessment mecha­
game as a formative assessment tool and evaluate the game ascertaining nism/tool in Higher Education. It was decided that an experimental
if it can be an effective formative assessment supplementary tool at HE post-test design would be utilised with non-parametric statistical tests
level. being utilised.
The objectives of the study are the following: This is primarily a quantitative research study with a positivist
paradigm being adopted due to the researchers Computer Science and
• Perform a literature review to ascertain what empirical evidence research background. The research questions and the selected method­
exists associated with serious games and programming education ologies for answering each research question are listed below.
and also to identify problems associated with traditional teaching
approaches for programming.
2.4. Research methodologies
• Perform a survey (empirical study 1) to ascertain if a serious game for
programming education will be accepted in a HE setting, what
Research question 1, 2 and 3 will be answered utilising archival
particular aspects of programming that students find to be difficult,
research using the following search terms: (“computer games” OR “video
what aspects of programming would be best to incorporate into a
games” OR “serious games” OR “simulation games” OR “games-based
serious game and what format and type of game would be more
learning” OR “online games” OR “MMOG” OR “MMORPG” OR “Virtual”
suitable to implement.
OR “Augmented”) AND (“program*” OR coding OR “computer science”)
• Utilise empirical study one and the literature to formulate a frame­
The following relevant electronic databases were searched: ACM,
work for content integration for programming education games and
EBSCO (including CINAHL Complete, eBook Collection, Education
directly integrate the pedagogical content into the game.
Source, Library, Information Science& Technology Abstracts, Psy­
• Develop a game with the programming content integrated into a
cARTICLES, PsycBooks, Psychology and Behavioural Sciences Collec­
gamified environment.
tion, SocINDEX), Ethos, IEEE, IngentaConnect, ScienceDirect and Web
• Perform an intervention, post-test experiment (empirical study 2) to
of Science and SpringerLink and the grey literature. There has been a
ascertain if the developed serious game can be utilised as a formative
near exponential increase in the publication of GBL studies over the past
or summative assessment tool in a programming class in HE.
five years, so it was necessary to limit the search to titles of papers and
abstracts when cross referencing the search terms. As a result, the
2.2. Research questions
number of papers found in each search engine are displayed in Table 1.
The papers were then identified, screened, considered for eligibility
This paper has the following research questions:
and then included in the final analysis utilising a preliminary PRISMA
methodology shown in Fig. 1. (see Fig. 2)
Research Question 1: What are the main problems associated with
Research question 4 and 5 will utilise the questionnaire methodology
traditional programming teaching approaches?
to reach a wide and diverse range of participants in an efficient cost
Research Question 2: How have computer games been used for the
effective manner. The questionnaire was constructed utilising a number
purposes of teaching computer programming in the past?
of open-ended, close-ended and matrix questions utilising appropriate
Research Question 3: What issues are identified in terms of pedagog­
academic guidelines (Blair, Czaja, & Blair, 2013) and modelled on a
ical content integration for computer games teaching programming? What
previous questionnaire produced to establish the motivations for playing
existing frameworks (if any) are there?
computer games (Hainey, Connolly, Stansfield, & Boyle, 2011b). The
Research Question 4: To what extent will a GBL approach be accepted
pedagogical content for the questionnaire was constructed utilising a
by learners for teaching computer programming?
number of programming books (including Dawson, 2014; Malik, 2014),
Research Question 5: What programming pedagogical content would
subject matter experts and utilising a review of the content of three HE
participants like to see in a game to teach programming at HE level?
programming modules at the university including: Introduction to Pro­
Research Question 6: Can a computer game be effective as a formative
gramming, Programming with Objects and Software Development for
assessment tool for computer programming in HE?
Games. The questionnaire was constructed in SurveyMonkey and
emailed to Creative Technology students at the University. The survey
2.3. Selection of research methodology
was available for a two week period in April of 2019. The full ques­
tionnaire is available in Appendix 1.
The research methodologies selected are archival research in the
form of a systematic literature review, survey research and experimental
Table 1
post-test design with non-parametric statistical analysis techniques. The
Systematic literature review relevant papers.
archival systematic review is more suitable to ascertain what problems
associated with teaching computer programming exist in the literature, Search Engine Total Papers Relevant Papers

how computer games have been utilised teaching computer program­ ACM 1233 29
ming, issues associated with pedagogical content associated with serious EBSCO 1484 6
Ethos 93 1
games and programming education and whether there are any existing
IEEE 776 251
frameworks. A systematic literature review of relevant databases is the IngentaConnect 1176 19
most efficient method to answer the first three research questions. ScienceDirect 2489 26
Research question 4,5, are attempting to ascertain a number of re­ Web of Science 2739 23
quirements for the proposed GBL environment/serious game to be SpringerLink 548 0
Totals 10538 364
produced in terms of whether that game will be accepted by the target

3
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Fig. 1. PRISMA Methodology for relevant paper inclusion.

Research question 6 will utilise a post-test experimental group design 3. Background research
and will employ a similar questionnaire to evaluate the game. The game
was made available to a second year programming module at University 3.1. Pedagogical difficulties associated with computer programming
in a supplementary, formative assessment tool capacity via email. After education
the game has been played the participants are invited to complete a post-
test evaluation questionnaire on SurveyMonkey. This questionnaire was Computer programming is a highly complex skill with many peda­
of a similar design as the previous questionnaire and is available in gogical issues and a considerable amount of time is required to effec­
Appendix 2. tively learn that skill and maintain it. Rigorous computer programming
has been introduced by the Department for Education (DfE) in recent
2.5. Developmental design years with programming being proposed as a skill that everyone should
have with [Link] and [Link] indicating that it is a glam­
This paper will outline the entirety of a first developmental cycle and orous path to employment (Lee et al., 2014). Hwang et al. (2010) note
includes the following stages: that almost all students enrolled in a department related to the computer
field (i.e. computer science or information management systems) are
• Ascertaining the pedagogical difficulties with programming. required to take a programming class at some point. Despite the fact that
• Gauging acceptability and development requirements (Empirical employment opportunities for Computer Scientists are rising, the num­
Study 1) ber of students enrolling on and graduating from Computer Science
• Formulating a general pedagogical content integration framework related degrees is declining with the majority of attrition taking place in
for a programming serious game. the first two years of a degree programme (Chaffin et al., 2009) and
• Implementation of the game fewer students are actually choosing a career in Computer Science or
• Evaluating the game as a formative assessment tool in a course Engineering (Marques et al., 2012). There is a disparity between
(Empirical Study 2) numbers of graduates, graduates in employment and unfilled positions
in related industries (Jordaan, 2018).

4
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Fig. 2. Content integration considerations for a game to teach computer programming.

The literature review identified numerous issues associated with programming education, some of the following topics are considered to
computer programming with the result being a “programming incompe­ be problematic for learners: algorithms due to abstract mathematical
tency” as graduates do not meet industry expectations in terms of theories, modelling complicated concepts and possibly describing
competence and capabilities (Kumar & Sharma, 2018). Sun, Phelps, complicated changes in dynamic data structures with the intent of
Egert, and Bayliss (2009) highlight that teaching introductory pro­ solving complex problems (Shabanah, Chen, Wechsler, Carr, & Weg­
gramming is a multifaceted problem encompassing: issues of abstrac­ man, 2010). Novice programmers also find it incredibly difficult to
tion, algorithmic thinking, software construction, and debugging decipher code and to determine what it is doing, i.e., programming
strategies. Kaučič and Asič (2011, p. 1) state that mastering program­ comprehension (Miljanovic & Bradbury, 2017). Yan (2009) notes that
ming related competencies demands “acquisition of previously completely these problems are not really associated with prior computing knowl­
unknown knowledge, problem solving skills and strategies, design strategies edge but because the subject is considered to be difficult, there is a lack
and approaches wrapped by implementation abilities”. Computer pro­ of problem solving skills and an appreciation of abstract concepts and
gramming courses are perceived as extremely difficult with high levels logical thinking (Khenissi et al., 2013). The difficulties are experienced
of attrition (Wei et al., 2009; Chaffin et al., 2009; Jantan & Aljunid, from both a learner and a teacher perspective as these problems are
2012; Lee et al., 2014). Learners can struggle to grasp the most basic of exacerbated with the addition of abstract concepts associated with Ob­
programming concepts and skills. For example Deitz and Buy (2016, p. ject Orientation which is the industry standard programming paradigm
37) state that “novice programmers often struggle with many concepts un­ where programs are constructed of interacting objects. Rudimentary
derlying computer programming” such as variables and the misinterpre­ programming education is not just about abstract Object-Oriented
tation of program states and variable states remains a frequent source of concepts and practical programming alone but rather the interactions
errors for novice programmers. Cagin (2013) summarised the literature and interweaving of the two. Writing a computer program can require a
and stated that there is a lack of rigorous empirical research associated great deal of implicit knowledge beyond the scope of simply being
with the difficulties experienced by introductory programmers. Some of familiar with a specific language (Hwang et al., 2010). Learners gener­
these difficulties are: ally begin introductory programming classes by producing text-based
applications which they consider to be relatively unimpressive. Mita­
• Confusion with branching and locations within programming logic mura, Suzuki, and Oohori (2012) points out two key points in relation to
where the explanation of different programming segments and the learning programming, firstly that the learning experience differs
combined whole is considered educationally beneficial. significantly in relation to academic background and secondly, that
• Difficulties in the composition and coordination of programming learners sometimes do not experience enjoyment with traditional
components. teaching approaches. Traditional teaching approaches in relation to
• Understanding error messages to appropriately debug programs. programming have a reputation for lacking engagement (Berkling &
• Thinking in programming syntax, as this requires thinking in an Thomas, 2013) and do not provide ways to think outside of the box
operational level of abstraction before the production of code in a (Jayasinghe & Dharmaratne, 2013). They provide low motivation and
specific language can take place. there can be an issue of generational gaps between teachers and students
• Difficulty in the visualisation of programming constructs for given (Khenissi et al., 2013). Kaučič and Asič (2011) highlighting that tradi­
problems. tional teaching approaches for programming education tend to be seen
• Learners require well-structured timely feedback in response to their as dull and uninteresting as a result of teachers prioritising theory rather
actions. than practical work. A number of approaches are highlighted to over­
come some of the issues with traditional education such as pair pro­
In terms of specific topics within Computer Science related to gramming, individual/collaborative work, read before write approach

5
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

and the use of games (Kaučič & Asič, 2011). De Aquino Leal and Ferreira Table 2
(2013) believe that difficulties learning programming are possibly due Attributes of traditional teaching approaches.
to deficient training at elementary level or misconceptions in relation to Attributes Lectures Role- Paper- Live-
the course they are entering as motivated users of computer technology play based case through
select a course based on this fact. Tan, Ting, and Ling (2009) performed studies case
studies
a survey asking 182 participants undertaking an undergraduate course
on computer programming about their level of understanding of certain Instructor has control of the HIGH MEDIUM LOW LOW
rudimentary and advanced concepts. The participants rated variables, learning, has to be an
effective speaker and
decision logic structures, iteration logic structures, error handling, ar­ communicate the
rays and parameters as the easiest concepts to understand and rated all intrinsic interest of a
of the object oriented principles (polymorphism, inheritance encapsu­ subject through
lation), pointers, abstract data types and recursion as the most difficult enthusiasm.
Participants are actively LOW HIGH MEDIUM LOW
concepts to understand.
involved in the exercise
It was considered important to gain a greater appreciation of the provide experiential
advantages, disadvantages and suitability of the main teaching ap­ learning and deeper
proaches utilised in software engineering and by implication program­ understanding to have a
ming education in this study. Table 2 attempts to summarise a review considerable lasting
effect.
performed by Hainey (2010) and Macphail, Hainey, and Connolly Can be time consuming and MEDIUM MEDIUM MEDIUM HIGH
(2012) citing the following sources: ADPRIMA, 2019, Andrianoff and therefore not always
Levine (2002), Bernstein and Klappholz (2001), Barrett (1997), Biggs practical.
(1999), Bonwell (1996), Cashin (1985), Cope and Horan (1996), Con­ Scales with larger numbers HIGH LOW HIGH HIGH
of students and/or
nolly, Stansfield, McLellan, Ramsay, and Sutherland (2004), Davis
quantity of material
(2001), Polack-Wahl (1999), Simsarian (2003) and Shaw and Dermoudy Can provide a model of HIGH MEDIUM LOW HIGH
(2005). how professionals
address problems and
questions.
3.2. Synthesis and presentation of model framework
Can provide elements of MEDIUM HIGH MEDIUM MEDIUM
variety, reality and
As well as considering the integration of learning content, there are a specificity.
number of questions and issues to be addressed when considering the Likelihood of transfer to the MEDIUM HIGH MEDIUM MEDIUM
basic requirements of a serious game identified in some of the following real world is improved.
Appeals to learners who HIGH HIGH LOW LOW
categories:
learn by listening.
Appeals to learners who HIGH LOW HIGH HIGH
3.2.1. Assessment learn by reading.
Assessment integration is one of the key challenges of serious games Appeals to learners who LOW HIGH LOW LOW
where it is necessary to consider what particular types of assessment learn from
demonstration.
mechanics are incorporated into a serious game. Assessment can be Appeals to learners who MEDIUM HIGH LOW LOW
categorised into formative and summative assessment. Conole and learn from discussion.
Warburton (2005) state that formative assessment is “when the evidence Appeals to learners who LOW HIGH HIGH MEDIUM
is actually used to adapt the teaching to meet student needs”. Summative learn from practice by
doing
assessment is defined by the QAA (2012) as “used to indicate the extent of
Presents a risk for learners LOW HIGH LOW LOW
a learner’s success in meeting the assessment criteria used to gauge the for example, tension or
intended learning outcomes of a module or programme.” Lotfi and embarrassment.
Mohammed (2018) address assessment in a mobile game for teaching Allows the learners to MEDIUM LOW LOW HIGH
Object Oriented Programming principles in the context of using the identify the mistakes of
previous developers, thus
assessment data to ensure that the learning outcomes of the game are reducing the possibility
actually met and raise the question of whether this assessment should of making the same
take place at the end of the game in a summative fashion or throughout mistake.
the game in a formative fashion. Provides a safe MEDIUM HIGH MEDIUM LOW
environment to increase
practice experience when
3.2.2. Type of game and player dynamic real life experiences are
One important distinction in relation to computer games is whether unavailable, for example
the game should be played in a single-player or a multi-player capacity highlighting client
and determining what mode is preferable to the players. Many serious frustration.
Immediate feedback is LOW HIGH LOW LOW
games are generally single-player mode due to the implementation provided.
complexity, however research indicates that players behave differently Can give new perspectives MEDIUM HIGH MEDIUM LOW
when playing in a single-player mode than in a multi-player mode (Xiao on situations.
& Miller, 2014; Siu, Guzdial, & Riedl, 2017). Learners can apply newly LOW MEDIUM HIGH HIGH
developed analytical
problem solving skills in
3.2.3. Type of programming language their own time to find a
When designing a game for programming education it is important to large number of solutions
ascertain what particular programming paradigm and language should for complex issues.
be the focus of the game e.g. should the game focus on Pascal, C++, C# Maintains the attention of LOW HIGH MEDIUM MEDIUM
the learner.
or Visual Basic? Programming languages have now gone through 5 Suited for teaching LOW MEDIUM MEDIUM MEDIUM
generations where one main distinction is High Level Languages (HLL) abstract, complex
and Low Level Languages (LLL) (Kawash & Collier, 2013). Alternatively (continued on next page)
as a result of programming education resulting in transferable skills it is

6
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Table 2 (continued ) 3.2.8. Acceptance and preferred content

Attributes Lectures Role- Paper- Live- Preferences, perceptions and attitudes to computer games for
play based case through educational purposes and particular subject areas are extremely
studies case important when considering the design, implementation and evaluation
studies of a serious game or GBL application. Establishing that the group of
subjects, or higher order recipient learners will accept a game for a particular subject is beneficial
thinking skills such as and is also very important to ascertain the receiving groups re­
values, motor skills,
quirements in terms of preferred content (Connolly, Stansfield, & Hai­
analysis and application
etc. ney, 2009; Backland & Hendrix, 2013).
Takes into account that LOW MEDIUM LOW LOW Pedagogical Content integration is a key problem that exists because
learners are at different every serious game/GBL application is generally a custom built learning
levels of understanding tool and general content integration frameworks are problematic to
and learn at different
paces.
develop. The literature review revealed very little recent research per­
Puts pressure on the learner LOW HIGH LOW LOW formed particularly addressing pedagogical content integration for
to perform. serious games. Aldrich (2004, 2005) categorises content into mainly
May require a debriefing MEDIUM HIGH MEDIUM MEDIUM simulation elements which are linear (a defined path from beginning to
session.
end), cyclical (addressing small competitive tasks combined to impact
Has to be well planned, MEDIUM HIGH MEDIUM MEDIUM
monitored and the overall environment), system (exposing the user to intertwined,
orchestrated or it may complex relationships governing system operation) and open-ended
lack focus. (presenting a subject with no set experience). Gomez-Martin,
Puts pressure on instructor HIGH HIGH MEDIUM LOW Gomez-Martin, and Gonzales-Calero (2009) define intrinsic and
(s) to perform.
Monitoring by a MEDIUM HIGH HIGH MEDIUM
extrinsic game content. Intrinsic content is a game in which learning
knowledgeable person to content is integrated within the framework of the game. In this type of
provide good feedback is game, learning and gaming are closely related and both happen simul­
required. taneously. Extrinsic content is a game in which the scenario is separate
It is easy for the learners to MEDIUM HIGH HIGH MEDIUM
or less integrated within the learning content. In this type of game,
see the relevance to their
own situation. learning and gaming are independent activities. These categories are not
Inappropriate results may MEDIUM HIGH MEDIUM LOW an either/or but a continuum of possible options that complement
occur if insufficient different content styles. This is very similar and in line with Winn’s
information provided. (2009) categorisations of intrinsic endogenous games where the game
play is informed by the learning content and pedagogical theory and
possible to have a programming game that is essentially program lan­ exogenous games where the learning content is added on top of suc­
guage agnostic (Law, 2012; Xiao & Miller, 2014). cessful game mechanics without significant modification. General
frameworks considering content integration also exist. Sauvé (2010)
3.2.4. Genre, format and level of graphical fidelity note in their content segmentation framework that an appropriate bal­
Computer games are an incredibly versatile medium with a variety of ance between game and learning time is required to maintain motiva­
different genres (Arsenault, 2009) including: action, adventure, shooter, tion and that major content segments should be defined in relation to
fighting and racing. These genres can be delivered in a number of learning objectives and target population. Echeverria et al. (2011)
different formats such as: PC, Console, Mobile Device and Virtual Reality propose a classroom games design framework with an educational
(Hainey, 2016) in different levels of graphical fidelity i.e. 2D or 3D dimension based on Bloom’s revised taxonomy focusing on cognitive
(Gerling, Birk, Mandryk, & Doucette, 2013). Genre, format and level of processes and types of knowledge. Tang and Hanneghan (2010) propose
graphics should be considered in relation to player preferences. the following guidelines for embedding pedagogical content into
game-play:
3.2.5. Recognised curriculum guidelines
A game created for the purposes of programming in HE should • should be in line with defined learning objectives and assessable
consider governing curriculum guidelines and bodies such as: The • in increasing order of difficulty and achievable
Institute of Electrical and Electronic Engineers (IEEE – [Link] • in the form of an intellectual exercise with minimal abstraction
[Link]), the British Computer Society (BCS – [Link] • applicable and readily transferable to a real world scenarios
org) and the Association for Computing Machinery (ACM – [Link] • Achievement should be based on Weiner’s ‘Attribution Theory’
[Link]). rather than pre-destined winning
• feedback should assist in success and error recognition
3.2.6. Integration of soft skills and abstract concepts • be embedded within the storytelling/narrative of the game whenever
The integration of soft skills (e.g. teamwork, interpersonal relation­ possible
ships) and abstract concepts can be highly complex particularly with a • use multiple representations where appropriate and be concise but
subject that has the concept of abstraction (hiding complexity and in­ avoid oversimplification
formation) as one of its key pillars (Hidalgo-Céspedes, Marín-Raventós, • contain challenges allowing learners to apply acquired knowledge
& Lara-Villagrán, 2014). and thus increase retention
• not have more than 7 key concepts in order to aid information recall.
3.2.7. Appropriate integration into a module or course
When utilising a game to teach in any capacity in a course it is Aspects to consider when integrating pedagogical content into a
necessary to consider how to integrate the game/application into that game for programming is summarised in Fig. 2.
course at the appropriate time (Ketamo, Kiili, Arnab, & Dunwell, 2013;
Sung, 2009).

7
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

4. Gauging acceptability and requirements (empirical study Table 3

one) Effectiveness of a game to teach rudimentary programming concepts.
Rudimentary Programming Concept Rank Mean SD
4.1. Participants and methodology
Conditionals 1st 4.10 0.77
Logical Operators 2nd 4.06 0.76
61 participants completed the Serious Games for Teaching Program­ Variable Types 2nd 4.06 0.83
ming Concepts questionnaire. 4 participants (6.6%) were female, 55 Loops 2nd 4.06 0.80
participants (90.2%) were male, 1 (1.6%) participant specified that they Objects 3rd 4.04 0.73
Arithmetic Operators 4th 3.96 0.83
were non-binary/third gender and 1 participant (1.6%) preferred not to Switch Statements 5th 3.94 0.80
say. Participants played games for 17.54 h per week (SD = 10.43) with a Declaring Variables 5th 3.94 0.81
range of 0–31 h. This is approximately 10 more hours per week than a Statements and Sequences 5th 3.94 0.79
general survey of the motivations for playing computer games in HE Algorithms 6th 3.92 0.92
Methods and Functions 7th 3.90 0.81
(Hainey, Connolly, Stansfield, & Boyle, 2011a) however; this is consis­
Variable Scope 8th 3.89 0.84
tent as participants were on Computer Games related degree pro­ Relational Operators 9th 3.85 0.78
grammes therefore it was expected that they would play computer Returning Types with Functions 9th 3.85 0.83
games more frequently. Participants performed 11.46 h (SD = 8.90) of Compound Conditions 10th 3.80 0.84
independent programming study per week with a range of 0–31 h. A Working with Constants 11th 3.79 0.87
Parameter Passing/Overloading Functions 12th 3.70 0.88
Wilcoxon-matched pairs signed rank test between the amount of hours Programming Syntax 12th 3.70 0.92
spent playing computer games and the amount of time spent performing Using Enumerators 13th 3.64 0.85
independent study of computer programming indicated that partici­ References 14th 3.62 0.87
pants play computer games for a significantly longer period in a week Pointers 15th 3.60 0.99
Smart Pointers 16th 3.55 0.95
rather than engaging in independent study (Z = − 3.025, p < 0.002).
Participants were an average of 21.77 years of age (SD = 4.84) with a
range of 17–48 years. Participants had been learning computer pro­
gramming for 3.11 years on average (SD = 2.33) with a range of 0–10 Table 4
years. 43 participants (70.5%) were on the Computer Games Develop­ Effectiveness of a game to teach advanced programming concepts.
ment degree stream, 9 participants (14.8%) were on Computer Games Advanced Programming Concepts Rank Mean SD
Technology and 9 (14.8%) were on Computer Animation Arts. 41 par­ Inheritance 1st 3.83 0.79
ticipants (67.2%) described themselves as a novice, 19 participants Defining classes, Constructors and Destructors 2nd 3.81 0.82
(31.1%) were intermediate level and 1 participants (1.6%) described Setting Member Access Levels, Data Members, Member 3rd 3.77 0.88
themselves as an expert programmer. Functions, Scope
Encapsulation 4th 3.74 0.79
The questionnaire was constructed utilising a number of rudimen­ Static Data Members/Static Member Functions 5th 3.72 0.85
tary programming books (including Dawson, 2014) and subject matter Polymorphism 6th 3.70 0.81
experts and utilising a review of the content of three HE programming Static Object Instantiation 7th 3.68 0.84
modules at UWS (University of the West of Scotland) including: Intro­ Mutator Accessor Functions 8th 3.66 0.84
Composition 9th 3.64 0.85
duction to Programming, Programming with Objects and Software
ADTs, Container Classes, Iterators, Interfaces 10th 3.63 0.91
Development for Games. The questionnaire was constructed in Survey­ Aggregation/Association 11th 3.62 0.85
Monkey and emailed to students on the Computer Games Devel­
opment/Technology and Computer Animation Arts Degree streams. The
survey was available for a two week period. The survey was analysed 4.5. Data structures and perceived effectiveness of games
utilising quantitative statistical analysis i.e. Mean, Standard Deviation,
and non-parametric statistical inferential tests where appropriate such The participants reported the following data structures to be the least
as Mann-Whitney U tests. The full questionnaire is in Appendix 1. difficult to learn arrays, stacks, standard vectors and multi-dimensional
arrays. The participants were also asked to rate the potential effective­
4.2. Acceptability of games for teaching computer programming in HE ness of a game for teaching the data structures. The results are displayed
in Table 5.
Participants were asked if they believed that a game to teach com­ The results indicated that a computer game would be most effective
puter programming concepts at HE was a good idea. 46 participants at teaching: arrays, queues, stacks, multi-dimensional arrays and
(75%) answered ‘yes’ to this question indicating that computer games
could be a potential teaching approach.
Table 5
Effectiveness of a game to teach data structures.
4.3. Rudimentary programming concepts and perceived effectiveness of
games Data Structure Rank Mean SD

Arrays 1st 3.94 0.77

Participants were asked to rate the potential effectiveness of com­ Queues 1st 3.94 0.84
puter games to teach rudimentary programming concepts. A game was Stacks 2nd 3.90 0.88
Multi-Dimensional Arrays 3rd 3.88 0.86
considered most effective to teach conditionals, logical operators, vari­
Standard Vector 4th 3.77 0.78
able types, loops, objects and arithmetic operators. The results are dis­ Game Trees and Minimax Trees 5th 3.73 0.96
played in Table 3. Bi-Directional Graphs 6th 3.65 0.91
Trees 6th 3.65 0.93
Hash Tables 7th 3.64 0.90
4.4. Advanced programming concepts and perceived effectiveness of
Graphs 8th 3.62 0.95
games Uni-Directional Graphs 9th 3.60 0.89
Bit Vectors 9th 3.60 0.90
Participants were also asked to rate the potential effectiveness of Priority Queues and Heaps 9th 3.60 0.92
computer games to teach advanced programming concepts. The results Binary Trees 9th 3.60 0.90
Binary Search Trees 9th 3.60 0.95
are shown in Table 4.

8
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Table 6 Table 8
Effectiveness of a game to teach algorithms. Preferred game format.
Algorithm Rank Mean SD Format Rank Mean SD

Basic Path Finding 1st 3.83 0.79 Personal Computer 1st 4.82 0.58
Bubble Sort 2nd 3.72 0.85 Tablet PC 2nd 4.02 0.75
Merge Sort 3rd 3.70 0.88 Mobile Phone 3rd 3.84 1.00
Heap Sort 4th 3.68 0.89 Augmented Reality 4th 3.31 1.16
Quick Sort 4th 3.68 0.89 Virtual Reality 5th 3.20 1.29
Binary Search 4th 3.68 0.91 Games Console 5th 3.20 1.12
Robust Path Finding 4th 3.68 0.84
Radix Sort 5th 3.62 0.90
Table 9
standard vectors. The results also indicated that computer games would Technical aspects of computer game to teach programming.
be least effective for binary search trees, binary trees, priority queues Technical aspect Rank Mean SD
and heaps and bit vectors. Control Mechanism and Interface 1st 4.58 0.58
Clear Goal Structure 2nd 4.53 0.63
4.6. Algorithms and perceived effectiveness of games Help and Scaffolding 3rd 4.22 0.67
Navigation Inside the Environment 4th 4.20 0.79
Narrative and Dialog 5th 3.89 0.83
The following three algorithms were reported as the most difficult to Sound 6th 3.71 0.87
learn: robust path finding, radix sort and basic path finding while the Collaboration 7th 3.49 0.87
least difficult algorithms were bubble sort, quick sort, heap sort and Graphics 8th 3.09 1.10
merge sort. The participants were asked to rate the potential effective­ Realism of Environment 9th 2.44 0.92
Realism of Characters in the Environment 9th 2.44 0.87
ness of computer games for teaching the searching and sorting algo­ Character Customisation 9th 2.24 1.00
rithms. The results are displayed in Table 6.

4.7. Suitability of game genre to teach programming Table 10

Preferred level of education.
Participants were asked to assess the suitability of game genres for Level of Education Rank Mean SD
potentially teaching programming concepts. The results are shown in Secondary Education 1st 4.44 0.62
Table 7. Primary Education 2nd 4.20 0.99
80% specified a single-player game and 20% specified multi-player. Further Education 3rd 4.07 0.89
Informal Private Education 4th 3.96 0.90
Participants were asked whether they thought the game should be 2D or
Higher Education 5th 3.84 1.13
3D. The majority of the participants stated that it did not matter if the Industrial Training 6th 3.69 1.33
game was in 2D or 3D (36 participants). Participants were also asked
what particular level of graphical fidelity they believed the game should
be. The majority of participants (71%) specified that they would prefer 4.8. Suitability of games for level of education
medium graphical fidelity. Participants were asked to rate the suitability
of different platforms/fromats for a game to teach programming con­ Participants were asked how suitable a game for teaching program­
cepts. The most suitable format for a game in this context was the PC. ming would be with regards to suitability for different levels of educa­
Table 8 shows the preferred format of participants. tion. The results are displayed in Table 10.
The participants were asked to rate the importance of the technical
aspects of a potential game where control mechanism and interface, 5. CODE: S.P.A.C.E: a game to teach programming
clear goal structure and help and scaffolding were considered to be the
most important aspects. The results are show in Table 9. 5.1. Game pedagogical content development

Table 7 The survey performed in empirical study 1 was utilised to determine

Game genre ranked by suitability. the requirements for the game in relation to pedagogical content that it
Game Genre Rank Mean SD
would be suitable to teach, the preferred format, genre of game and
overall acceptability. The majority of participants (75%) believed that a
Educational Game 1st 4.42 0.78
game would be suitable to teach programming in HE. The top ranking
Puzzle Game 2nd 4.36 0.94
Simulation Game 3rd 4.29 0.82 attributes were then used to produce the requirements for the game e.g.
Serious Game 4th 4.16 0.98 for the game genre in Table 7, the top attributes were the following:
Adventure Game 5th 4.07 0.91 Game Genre Rank Mean SD
Turn-Based Strategy Game (TBSG) 6th 4.07 0.86
Role Playing Game (RPG) 7th 3.91 1.10 Educational Game 1st 4.42 0.78
Text-Based Adventure Game 8th 3.89 1.01 Puzzle Game 2nd 4.36 0.94
Action Adventure Game 9th 3.84 0.98 Simulation Game 3rd 4.29 0.82
Platform Game 10th 3.73 1.07 Serious Game 4th 4.16 0.98
Board Game 10th 3.73 0.99 Adventure Game 5th 4.07 0.91
Real Time Strategy Game (RTSG) 11th 3.71 1.12
Action Game 12th 3.47 1.06
Stealth Game 13th 3.33 1.09 It was therefore decided that the game should be a mixture of these
Rhythm Game 14th 3.24 1.23
genres and it ended up being an educational, puzzle, simulation,
Survival Horror Game 15th 3.00 1.21
First Person Shooter Game 16th 2.98 1.29 adventure, story-based serious game.
Massively Multiplayer Online Game (MMOG) 17th 2.80 1.31 The game was developed in Unity 5 with the incorporation of
Fighting Game 18th 2.75 1.33 intrinsic and endogenous narrative content integration being the main
Sports Game 19th 2.69 1.14 philosophy meaning that the learning outcomes and the game goals are
Racing Game 19th 2.69 1.16
inextricably interconnected. Utilising a narrative based story provided a

9
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Fig. 3a. Menu screen.

Fig. 4a. Class tutorial screen.

game where the discourse provided immersion, engagement, motiva­
tion, and learning (Naul & Liu, 2020). The pedagogical content was
integrated based on student preferences in relation to what the game
would contain in the broad sense and the intricate learning content was
taken from two undergraduate programming courses. One at year 1 and
one at year 2 as these are considered to have the highest levels of
attrition in Computer Science related courses. The game is a story-based
space, puzzle adventure game requiring the player to navigate through
an asteroid field where they are then confronted with a number of
programming related questions. The development utilised some of the
following guidelines from Tang and Hanneghan (2010) to embed
pedagogical content into game-play:

• should be in line with defined learning objectives and assessable

•in increasing order of difficulty and achievable
•feedback should assist in success and error recognition
•be embedded within the storytelling/narrative of the game when­
ever possible
•contain challenges allowing learners to apply acquired knowledge
and thus increase retention
Fig. 4b. Class level game play.
•not have more than 7 key concepts in order to aid information
recall.
times. An example of the sequential game screen shots are as follows:
Fig. 3a and b illustrate the menu and a screen shot of the narrative
The game is divided into 5 levels which allow the player to select the
within the game. Fig. 4a and b shows the class subject matter tutorial
particular topic, there is then a screen that displays a tutorial and the
and the gameplay associated with the subject matter.
player then shoots there way though waves of enemy spacecraft to have
Level 1 was derived from Table 3 where the top rated preferred
the opportunity to answer the questions to proceed. They are then
content in terms of perceived effectiveness for teaching rudimentary
presented with a question and they have to shoot the correct answer 10
programming concepts were:
Rudimentary Programming Concept Rank Mean SD

Conditionals 1st 4.10 0.77

Logical Operators 2nd 4.06 0.76
Variable Types 2nd 4.06 0.83
Loops 2nd 4.06 0.80

Ten questions were devised on each on the topics of variable types,

conditionals, logical operators and loops taken from the two program­
ming modules illustrated in Fig. 5a, b, 6a and 6b.
Level 2 was derived from Table 4 where the top rated preferred
content for effectiveness for teaching advanced programming concepts
were:
Advanced Programming Concepts Rank Mean SD

Inheritance 1st 3.83 0.79

Defining classes, Constructors and Destructors 2nd 3.81 0.82
Setting Member Access Levels, Data Members, Member 3rd 3.77 0.88
Functions, Scope
Encapsulation 4th 3.74 0.79
Fig. 3b. Narrative content.

10
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Fig. 6b. Question answering screen.

Fig. 5a. Menu screen.

structures were:
Data Structure Rank Mean SD

Arrays 1st 3.94 0.77

Queues 1st 3.94 0.84
Stacks 2nd 3.90 0.88
Multi-Dimensional Arrays 3rd 3.88 0.86

There are 10 questions on: arrays, queues, stacks, vectors, and multi-
dimensional with the previous format. The tutorial content is shown in
Fig. 7a and one of the example questions is shown in Fig. 7b.
Level 4 was derived from Table 6 where the top rated preferred
content and the games potential effectiveness for teaching data struc­
tures were:
Algorithm Rank Mean SD

Basic Path Finding 1st 3.83 0.79

Bubble Sort 2nd 3.72 0.85
Merge Sort 3rd 3.70 0.88
Heap Sort 4th 3.68 0.89
Quick Sort 4th 3.68 0.89

Fig. 5b. Narrative content.

Level 4 therefore teaches: bubble, heap, quick, merge sort and path
finding with various questions on each. Fig. 8a and b illustrate the
narrative and the intrinsic, endogenous pedagogical content integrated
into the game.
Level 5 is a randomised combination of all of the previous levels.
In summary the game has three types of integrated, intrinsic,
endogenous narrative in relation to pedagogical content integration.

• Instructional pedagogical content in the form of tutorials that are

revealed in the narrative discourse of an admiral speaking to a pilot.
• Tested pedagogical content in the form of questions that are being
asked in the middle of the space shooter game where the player has
to shoot the correct answer 10 times to progress.
• Interface pedagogical content in the form of menu choices that are
still within the ethos of the game. An illustration of this is in Fig. 9.

The summary results which were placed into a requirements speci­

Fig. 6a. Shooting screen.
fication in Appendix.
Appendix 3.

6. Evaluation of CODE: S.P.A.C.E Study 2 (Empirical Study Two)

Therefore, there are ten questions each on classes, constructors, ac­
cess modifiers and inheritance. The types of questions are illustrated in
6.1. Procedure and methodology
Fig. 4a and b.
Level 3 was derived from Table 5 where the top rated preferred
The methodology utilised for evaluating the game was an experi­
content and the games potential effectiveness for teaching data
mental post-test intervention design where the post-test was a survey

11
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Fig. 7a. Tutorial narrative content.

Fig. 7b. Pedagogical endogenous intrinsic content.

which was based on the survey used for empirical study one. The survey 6.2. Participants
is available at Appendix 2. As in empirical study one the data was ana­
lysed utilising quantitative non-parametric statistical tests such as Mean, A second year undergraduate programming course called Software
Standard Deviation and Mann-Whitney U tests. Development for Games participated in the evaluation of CODE: S.P.A.C.
The instructions for carrying out the evaluation, CODE: S.P.A.C.E E. 48 participants completed the CODE: S.P.A.C.E evaluation question­
and a link to the evaluation questionnaire were distributed to students at naire (Appendix 2) after data cleansing. 38 participants (81.3%) were
the UWS undertaking a Computer Games Programme. After playing the male, 6 participants (12.5%) were female and 3 participants (6.3%)
game participants were invited to complete the CODE: S.P.A.C.E evalu­ preferred not to say. Participants played games for 16.95 h per week (SD
ation questionnaire. = 9.68) with a range of 0–31 h which is consistent with study one.
Participants performed 10.95 h (SD = 8.10) of independent

12
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Fig. 8a. Tutorial narrative content.

Fig. 8b. Intrinsic, endogenous content.

between males and females in relation to age, how long they played
computer games per week, how many hours a week they invested in
learning to program or the years spent learning to program. 37 partic­
ipants (77%) were from the Computer Games Development Programme
and 11 participants (23%) were from the Computer Games Technology
Programme.
Participants were asked to rank the approaches for teaching pro­
gramming in terms of how motivational they believed the approach to
be. The results are displayed in Table 11.
The results indicate that GBL/Serious Games and Computer Games
for Learning are potentially a highly motivational teaching approach as
they are second and third in the overall ranking after practical lab ses­
sions and individual course works. Participants were asked to rank the
approaches for teaching programming in terms of how effective they
believed the approach to be. The results are displayed in Table 12.
Fig. 9. Menu interface pedagogical content. Computer games for learning were rated as 5th and GBL and Serious
Games were rated as 6th in terms of overall effectiveness as teaching
programming study per week with a range of 0–31 h. Again this is approaches. While these approaches were not rated as highly as indi­
relatively consistent with study one. The average age of participants was vidual course works, practical lab sessions and tutorials it seems that
23.29 (SD = 4.78) with a range of 18–39. Participants had been learning there is clear trend in the results suggesting that more interactive,
programming for an average of 4.72 years (SD = 1.97) with a range of practical approaches are rated more highly than passive, less interactive
1–10 years. Mann-Whitney U tests indicated no significant differences

13
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

approaches. 22 participants (45.8%) classified themselves as a novice Table 12

programmer, 25 participants (52.1%) classified themselves as an inter­ Ranking of programming teaching approaches in relation to effectiveness.
mediate level programmers and only 1 participants (2.1%) classified Teaching Approach Ranking Mean SD
themselves as an expert level programmer. A Mann-Whitney U test
Individual Course Works 1st 4.27 0.76
indicated that intermediate programmers spent significantly more hours Practical Lab Sessions 2nd 4.08 0.87
per week (13.36, SD = 8.39) learning to program that novice pro­ Class Room Tutorials 3rd 3.96 0.82
grammers (7.32, SD = 5.16) (Z = − 2.583, p < 0.010). A Mann-Whitney Online Tutorials 4th 3.92 0.94
U test also indicated that intermediate level programmers had been Computer Game For Learning 5th 3.79 0.97
Games-based Learning/Serious Games 6th 3.77 0.97
learning to program for a significantly longer number of years than Group Based Course Works 7th 3.63 1.16
(5.46, SD = 1.73) than novice programmers (3.72, SD = 1.75) (Z = Lecturing Classroom 8th 3.55 1.16
− 3.135, p < 0.002). Role Playing 9th 3.04 1.02
Live Thru Case Studies 10th 2.98 0.81
Paper Based Case Studies 11th 2.75 0.89
6.3. Effectiveness of CODE:S.P.A.C.E for teaching rudimentary
programming concepts

Participants were asked to rate the effectiveness of the game to teach Table 13
Effectiveness of the game for teaching rudimentary programming concepts.
rudimentary programming concepts. The results are displayed in
Table 13. Rudimentary Programming Concept Ranking Mean SD
The results indicated that the game was rated as most effective at Variable Types 1st 3.89 1.07
teaching the following concepts: variable types, declaring variables, Declaring Variables 2nd 3.81 0.97
programming syntax, conditionals, loops, switch statements and Programming Syntax 2nd 3.81 0.91
Conditionals 3rd 3.78 0.95
methods and functions. The game was rated lowest at teaching the
Loops 3rd 3.78 1.08
following programming concepts: working with constants, smart Switch Statements 3rd 3.78 0.96
pointers, and pointers. Mann Whitney U tests indicated that there were Methods and Functions 4th 3.73 0.99
no significant differences between novice and intermediate level pro­ Objects 5th 3.70 0.97
Arithmetic Operators 5th 3.70 0.97
grammers in relation to the game’s perceived effectiveness.
Logical Operators 5th 3.70 0.94
Statements and Sequences 6th 3.69 0.89
6.4. Effectiveness of CODE:S.P.A.C.E for teaching advanced Returning Types with Operators 7th 3.62 0.95
programming concepts Variable Scope 8th 3.58 1.05
Using Enumerators 9th 3.54 1.02
Compound Conditions 10th 3.51 0.87
Participants were asked to rate the effectiveness of the game for Parameter Passing and Overloading Functions 10th 3.51 1.04
teaching advanced programming concept and the results are displayed Using References 11th 3.46 1.12
in Table 14. Relational Operators 12th 3.44 0.88
Algorithms 13th 3.42 1.00
The game was considered to be most effective at teaching the
Working with Constants 14th 3.38 0.92
following: defining classes, constructors and destructors, inheritance Smart Pointers 15th 3.35 1.21
and encapsulation. Pointers 16th 3.32 1.16

6.5. Effectiveness of CODE:S.P.A.C.E for teaching data structures

Table 14
Participants were asked to rate the effectiveness of the game for Effectiveness of the game to teach advanced programming concepts.
teaching algorithms and the results are displayed in Table 15. The re­ Advanced Programming Concepts Rank Mean SD
sults indicated that the game was most effective at teaching: arrays,
Defining Classes Constructors and Destructors 1st 3.78 1.00
queues, multi-dimensional arrays, stacks and standard vectors. Inheritance 2nd 3.74 1.07
Mann-Whitney U tests again indicated that there was no significant Encapsulation 3rd 3.70 0.97
differences associated with perceived effectiveness in relation to novice Setting Member Access Levels 4th 3.65 1.01
or intermediate programming level skill. Polymorphism 5th 3.62 1.06
Static Data Members and Static Data Functions 6th 3.59 1.01
Aggregation/Association 7th 3.51 1.12
6.6. Effectiveness of CODE:S.P.A.C.E for teaching algorithms Mutator Access Functions 8th 3.44 1.00
Abstract Data Types 9th 3.43 1.09
Static Object Instantiation 10th 3.42 1.13
The participants were asked to rate the games effectiveness for
Composition 11th 3.41 1.12
teaching algorithms and the results are displayed in Table 16.

Table 11 The results indicated that the game was most effective at teaching
Ranking of programming teaching approaches in relation to motivation. the bubble, quick and merge sort. Mann Whitney U tests again indicated
Teaching Approach Ranking Mean SD that there was no significant difference between novice and intermedi­
ate level programmers in relation to the games effectiveness in teaching
Practical Lab Sessions 1st 4.02 0.98
Individual Course Works 1st 4.02 0.81
algorithms.
Games-based Learning/Serious Games 2nd 4.00 0.80
Computer Game for Learning 3rd 3.98 0.86 6.7. Effectiveness of CODE:S.P.A.C.E for teaching particular skills
Classroom Tutorials 4th 3.73 0.98
Online Tutorial 5th 3.52 0.92
The participants were asked to rate the effectiveness of the game for
Lecturing Classroom 6th 3.31 1.01
Live Thru Case Studies 7th 3.27 0.89 teaching particular skills that were important for learning programming.
Role Playing 8th 3.26 1.03 The results are displayed in Table 17 showing that recollection, problem
Group Based Course Works 9th 3.23 1.06 solving and analysing and classifying are the top ranked skills.
Paper-based Case Studies 10th 2.71 1.09

14
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Table 15 Table 17
Effectiveness of the game to teach data structures. Effectiveness of the game to teach skills.
Data Structure Rank Mean SD Skills Rank Mean SD

Arrays 1st 3.92 0.98 Recollection 1st 3.78 1.22

Queues 2nd 3.78 0.98 Problem Solving 2nd 3.62 1.19
Multi-Dimensional Arrays 2nd 3.78 0.95 Analysing/Classifying 3rd 3.57 1.09
Stacks 3rd 3.72 1.00 Reflection 4th 3.49 1.19
Standard Vector 4th 3.69 0.95 Critical Thinking 5th 3.46 1.17
Priority Queues And Heaps 5th 3.51 1.10 Mathematics 6th 3.42 1.23
Binary Search Trees 5th 3.51 1.12 Management 7th 3.05 1.33
Trees 6th 3.50 1.08 Creativity 8th 3.03 1.21
Binary Trees 7th 3.49 1.12 Leading/Motivating 9th 2.58 1.34
Game Trees And Minimax Trees 8th 3.43 1.19 Collaboration/Teamwork 10th 2.43 1.34
Graphs 9th 3.42 1.25
Uni-Directional Graphs 9th 3.32 1.23
Bi-Directional Graphs 10th 3.30 1.29
Hash Tables 10th 3.30 1.20 Table 18
Bit Vectors 11th 3.28 1.28 Ratings of technical aspects of the game.
Technical aspect Rank Mean SD

6.8. Technical aspects of the game Clear Goal Structure 1st 4.46 0.56
Control Mechanism and Interface 2nd 4.08 0.92
Navigation Inside the Environment 3rd 4.05 0.88
The participants were asked to rate the importance of the technical
Help and Scaffolding 4th 3.81 1.17
aspects of the game. The results are presented in Table 18. Clear goal Sound 5th 3.38 1.11
structure and control mechanism and interface as well as navigation Narrative and Dialog 6th 3.30 1.33
inside the environment and help and scaffolding are considered Collaboration 7th 2.97 0.96
Graphics 8th 2.76 1.14
important in the environment. This is consistent with the requirements
Realism of the Characters in the Environment 10th 2.57 1.26
that were formulated in empirical study one. Realism of the Environment 11th 2.51 1.17
The clear goal structure, help and scaffolding refer to the highly Character Customisation 12th 2.20 1.23
detailed instructions, narrative, and detailed tutorial level to set the
scene within the environment. Fig. 10 shows an example of the tutorial
level and narrative to assist the player in understanding the goal struc­ The game was produced utilising the Unity 5 Games Engine and as a
ture and provides clear direction within the environment. The game has result could really be delivered on a number of platforms. Participants
an exceptionally smooth interface for navigation between the gaming were asked to rate the suitability of the game for different platforms of
and puzzle elements i.e. the enemy ships and the questions. The game delivery. The results are displayed in Table 19.
interface provides an intrinsic menu structure where the narrative al­ The results indicated that the game was most suitable for the PC,
ways situates the player in the role of ‘pilot’ of a ship. Intrinsic peda­ Tablet PC or Mobile Phone. The game was integrated as a formative
gogical menu structures can be viewed in Fig. 9. assessment revision supplementary tool into a specific module at HE
which has now run for a period of two years. In the first year of running
the students in the class did not have access to this game, however in the
6.9. Formative assessment effectiveness, type and format suitability second year of running the module they did. A Mann-Whitney U test was
performed between the two independent year groups to ascertain if the
Participants were asked how effective the game was as a preparation game made a statistically significant difference to the overall grades in
mechanism for their class test assessment. The results were generally the module and the grades in the class test worth 40%. The results of the
positive. 37 participants answered this question were 5 participants test indicated that there was no significant differences in overall grades
(14%) rated the game as very effective, 19 participants (51%) rated the (Z = − 1.073, p < 0.283) or the class test (Z = − 0.743, p < 0.453). This is
game as effective, 9 participants (24%) rated it as neutral, 3 participants possibly due to the small cohorts in each level and the fact that the
(8%) rated it as ineffective and 1 participant (3%) rated the game as very experimental design could have been stronger in relation to evaluation.
ineffective. Participants were also asked how they believed that a The best way to evaluate such a game would require a Randomised
computer game could be utilised within the context of a module at HE Controlled Trial (RCT) with 30 representative samples in each group to
level in either a formative or summative capacity or both. 37 partici­ produce a statistically significant comparison.
pants answered this question where 25 (68%) believed it could be used
in both a formative and summative way, 6 (16%) believed that it could 7. Conclusions
only be used in a formative capacity and 6 (16%) in a summative way.
20 participants (54%) believed that a game could in fact be an assess­ This section will revisit the original research questions and provide a
ment for a module where 17 participants (46%) believed that it could summarised answer for them.
not be used as an assessment for a module.
Research Question 1: What are the main problems associated with
Table 16 traditional programming teaching approaches?
Effectiveness of game to teach algorithms.
Many problems were found associated with computer programming
Algorithm Rank Mean SD
education. Generally speaking programming is perceived as difficult,
Bubble Sort 1st 3.81 0.97
not particularly motivational and with high levels of attrition. Tradi­
Quick Sort 2nd 3.78 1.03
Merge Sort 3rd 3.70 0.94 tional didactic approaches including lectures, paper-based based
Binary Search 4th 3.68 0.97 studies, live-through case studies and role playing are not seen as
Heap Sort 5th 3.62 1.01 effective. One of the findings of the questionnaire study was that prac­
Basic Path Finding 6th 3.59 1.17 tical lab exercises and course work are the most effective and motiva­
Robust Path Finding 7th 3.51 1.12
tional teaching approach to computer programming, however the
Radix Sort 7th 3.51 1.22

15
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

Fig. 10. Tutorial level and narrative within the game.

literature indicates that one problem associated with current approaches

Table 19
is lack of practical, industry-based experience representing a holistic
Platform suitability rating.
view of the software development or programming process. New more
Format Rank Mean SD interactive approaches such as games, GBL, serious games and even the
Personal Computer 1st 4.81 0.40 gamification of courses have to be more frequently utilised to act as a
Tablet PC 2nd 4.11 0.81 supplementary learning approach for traditional approaches to address
Mobile Phone 3rd 3.73 1.07
this situation.
Games Console 4th 3.61 1.08
Augmented Reality 5th 2.72 1.11 Tan et al. (2009) performed a similar questionnaire study asking 182
Virtual Reality 6th 2.53 1.18 undertaking an undergraduate course on computer programming about
their level of understanding of certain rudimentary and advanced con­
cepts. The study found that the following concepts were easiest to un­
derstand: variables, decision logic structures, iteration logic structures,

16
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

error handling, arrays and parameters. The hardest concepts to under­ it would appear that as with traditional approaches they are not entirely
stand were: object oriented principles (polymorphism, inheritance preferred by everyone. This study has expanded on the study performed
encapsulation), pointers, abstract data types and recursion which is by Ibrahim et al. (2011) by utilising the literature and the questionnaire
highly consistent with the findings of this particular study. This verifies to produce a framework for a programming serious game in terms of
that the results obtained in this study were generalizable and by backing content integration. This framework has also been utilised to produce an
up the findings of this study has proven that these results have wide implemented game to act as a formative assessment teaching and scaf­
reaching applications in Scottish HE overall. It is also apparent that the folding tool on two programming course at HE level. The results of
study performed by Tan et al. (2009) was performed almost a decade integration of this game into a module have been positive and this study
previously indicating that ten years on these issues have not been suf­ can be expanded to again verify that the results have wider applications
ficiently addressed or resolved. in Scottish HE and potentially internationally.

Research Question 2: How have computer games been used for the Research Question 5: What programming pedagogical content
purposes of teaching computer programming in the past? would participants like to see in a game to teach programming at HE
level?
314 papers were included in the final literature review. 187 of these
were papers with developed or existing games revealing that computer Participants indicated that the more simpler programming concepts
games have certainly been used for teaching computer programming in were suitable to be taught by a game which is possibly due to the fact
the past. These games are at various different levels of abstraction, uti­ that more simplistic concepts are easier to visualise in a gaming sce­
lising different platforms, implementation languages, levels of graphical nario. Table 20 shows the most popular rudimentary and advanced
fidelity and to teach a number of different topics including pointers, programming concepts as well as data structures and algorithms to be
search algorithms, logic, syntax, object orientation and data structures, incorporated into a computer game.
functions/methods and debugging. The IEEE search engine revealed the
majority of these games which is not particularly surprising with an Research Question 6: Can a computer game be effective as a
excess of 60 games being discovered. In general, the study has indicated formative assessment tool for computer programming in HE?
that computer games are a highly versatile medium for teaching com­
puter programming. A full analysis of these games and the other iden­ The results indicate that GBL/Serious Games and Computer Games
tified studies from Fig. 1 are beyond the scope of this study but can be a for Learning are potentially a highly motivational teaching approach
future research direction. with computer games for learning rated as 5th and GBL and Serious
Games were rated as 6th overall. The results indicated that that more
Research Question 3: What issues are identified in terms of peda­ interactive, practical approaches are preferable.
gogical content integration for computer games teaching program­ In relation to rudimentary programming concepts, the game was
ming? What existing frameworks (if any) are there? rated more effective at teaching: variable types, declaring variables,
programming syntax, conditionals, loops, switch statements and
A number of issues were identified in terms of considering peda­ methods and functions and objects which was consistent with the
gogical content integration for a game to teach computer programming. preferred content materials from study one. The game was rated lowest
The issues identified in the literature were some of the following: what at teaching the following programming concepts: working with con­
particular genre, format and type of game, whether the game should be stants, smart pointers and pointers.
single or multiplayer, assessment integration and feedback, adherence In relation to advanced programming concepts the game was rated
to the computer science curriculum, whether the game is to address high highly effective in being able to teach the following concepts: defining
level, low level programming languages or to be programming language classes, constructors and destructors, inheritance and encapsulation
agnostic, how are abstract concepts to be incorporated into the game which is again consistent with the preferred content identified by po­
and general acceptance of the game. While there are frameworks in the tential participants in study one. The game was rated highly in terms of
literature that address content integration such as the content segmen­ being able to teach the following data structures effectively: arrays,
tation framework (Sauvé, 2010), the classroom design framework queues, multi-dimensional arrays, stacks and queues which again is
(Echeverria et al., 2011) and the guidelines for embedding pedagogical relatively consistent in relation to the content that was identified for
content into game-play (Tang & Hanneghan, 2010), these frameworks incorporation. A similar finding was found in relation to algorithms with
are too general and not specific enough for considering the issues of the exception of the basic path finding algorithm possibly indicating that
incorporating specialised content into a game for teaching program­ the game produced does not intuitively lend itself to this particular
ming. This resulted in the development of a new preliminary framework concept.
for considering content integration for a game to teach computer pro­ No significant differences between novice and intermediate level
gramming shown in Figure 2. This framework has been crucial in the programmers in relation to the game’s effectiveness with regards to
development of the prototype game as a result and is a preliminary rudimentary programming concepts, advanced concepts, data structures
framework which can undergo further future refinement. or algorithms was detected. This is possibly due to the generic game
mechanics allowing practically any pedagogical content to be integrated
Research Question 4: To what extent will a games-based learning
approach be accepted by learners for teaching computer programming?
Table 20
Preferred Content to be Integrated.
75% of participants believed that computer games could be used to
Rudimentary Advanced Data structures Algorithms
teach computer programming concepts at HE level in Study One. Ibra­
him, Yusoff, Omar, and Jaafar (2011) found a similar positive result • Conditionals •Inheritance •Multi- •Basic path
where 60% of participants believed that they could learn better with •Logical •Defining classes, dimensional finding
operators constructors and arrays •Bubble sort
educational games being an option in the curriculum and 80% of par­ •Variable types destructors •Standard •Quick sort
ticipants believing that games make the subject more interesting and •Loops •Setting member access vectors •Heap sort
assist them to think critically which coincidentally is the 3rd most •Objects levels, data members, •Stacks •Merge sort
important skill identified by participants to be a successful programmer member functions and •Arrays
scope
in this study. In conclusion games are generally accepted by learners but

17
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

into it. The game has a tutorial narrative and shooter quiz with 4 incorporate the more complex and difficult subjects to be educationally
possible answers and one correct answer structure. This may mean that far more useful rather than easier. This indicates that the design is
optimistically it can be effective at teaching novice and intermediate entirely subjective and should take into account programming educa­
level programmers. Pessimistically this could be a further research study tionalists in the future.
direction to identify novice and intermediate level content and perform
separate evaluations with a control structure to ascertain the games 7.2. Implications and future research directions
perceived effectiveness at teaching different programming concepts/
levels. There are a number of implications and future recommendations for
The results indicated that the game was particularly effective at the future including:
developing the following important skills for learning to program: Greater appreciation and utilisation of the potential of the use of
recollection, problem solving, analysing/classifying, reflection and computer games, GBL, serious games, gamification, Games-Based Con­
critical thinking. The main difference from the first empirical study was struction Learning in computer science, computer games development
that recollection was rated as 8th most important, however problem and technology courses at a number of education levels. This can be
solving and critical thinking ranked highly in both study one and study achieved in three ways.
two. In terms of the technical aspects of the game it is evidence in both
study one and two that a clear goal structure, control mechanism and the • Educationalists themselves developing games by the use of their own
interface, navigation inside the environment and help and scaffolding games development skills, grants or subject matter experts. With the
were all rated highly indicating consistency. increasing simplicity of games development engines such as Unity,
14% rated the game as very effective and 51% rated the game as Unreal and Game Maker this will become more and more feasible
effective for their class test preparation. 68% believed it could be used in without having to possess great games programming knowledge.
both a formative and summative way, 54% believed that a game could in • Utilisation of students on these courses at a more advanced level
fact be an assessment for a module and 46% believed that it could not be reflecting on particular aspects of programming education that they
used as an assessment for a module. A comparison of class test grades found difficult and setting them assessments to develop game ap­
and overall module grades between students who had access to the game plications to assist students at a lower level.
and students who did not indicated that there was no significant dif­ • There are a vast number of games for teaching a number of different
ferences in overall grades or the class test. The overall results suggest areas and a particular concentration of computer games to teach
that a computer game for teaching programming concepts can be used aspects of programming. A taxonomy and catalogue of serious games
effectively as a formative and potentially summative assessment tool in general and for programming education should be created to make
that is preferable to a majority of students in a preliminary evaluation. use of already existing games.
There are a number of implications to be considered for the game to be
used in as a formative or summative assessment tool. Firstly, whether Levels of education, experience, knowledge and ability have to be
the game should be used in a formal or informal capacity i.e. whether the more deeply considered in good Higher Educational Practice in relation
results of the game are actually assessed and contribute towards a grade to computer programming courses and programmes. This is especially
in a formal capacity or whether it is used as a formative supplementary an issue when encountering issues such as direct entry and articulation
revision tool before in an informal capacity before formal assessment from FE (Further Education) institutions to HE institutions or simply
takes place. Secondly it is necessary to consider whether a game could in coming from SE (Secondary Education) institutions as the learning
fact be suitable for every type of learner within a class and it is important experience significantly differs in relation to programming Mitamura
to consider learner preferences and more importantly accessibility in et al., 2012 and is maybe deficient at earlier levels of education (De
relation to equality to provide equal opportunities to succeed. Aquino Leal & Ferreira, 2013). Traditional approaches can be seen as
lacking motivation, engagement and hence become ineffective (Berkling
7.1. Limitations & Thomas, 2013) with issues of generational gaps between teachers and
students (Khenissi et al., 2013). These issues raise a number of recom­
One primary limitation of games is that they will never replace di­ mendations for policy in computer programming courses in relation to
dactic traditional teaching approaches and it is highly important to be formative and summative assessment. Educators have to enhance their
realistic about this for the moment. Computer games have a number of responsibility in relation to formative assessment to suitably gauge level
negative connotations associated with them including that they are of experience, knowledge and skills. Computer games can assist with
conducive to negative personality traits and behaviours (Rosas et al., these issues in relation to being a formative assessment mechanism
2003). Similarly, to every other teaching approach computer games are which is potentially more engaging and ideally reduces any learner
subject to individual preferences, perceptions, motivations and atti­ embarrassment as they can be played on an individual level at home to
tudes. If for example an educationalist is attempting to teach mathe­ highlight problems and bring learners up to the same base level. Lec­
matical concepts with a computer game and the learner has a negative turers in HE have to perform more research into formative assessment
attitude to both mathematics and computer games – then this approach methods which are inclusive and involve learners in their development
will significantly reduce in effectiveness. to a far greater degree. The Scottish Qualifications Authority (2018, p.
Another limitation of computer games associated with attitudes is 11) in their future of assessment report compiled by young adults in
that games are primarily viewed as fun entertaining interactive pass Scotland had the following aspiration:
times where a highly challenging aspect is separating the fun from the "Assessment should be fair, and inclusive a “level playing field” for all,
educational engagement. This limitation was implicitly implied from the but to achieve this assessment needs to be personalised, with a broad variety
literature review with the idea of intrinsic, extrinsic and exogenous and of assessment methods that young people can choose from over the course of a
endogenous content integration. school year. We need to strive for equity, not equality."
These results were presented to a number of educationalists in the With regards to giving young adults the choice of assessment
field of serious games and programming at a workshop on Serious methods in education – it may very well be the case that one of the novel
Games in Cyber Security and it was discovered that there is a disparity teaching approaches that is selected may be computer games.
between the learner and an educationalist perspective in relation to how The following future research directions have been compiled from
a prototype game should be developed. Learners would prefer to have the synthesis of this study:
the most simplistic concepts incorporated into a prototype game where
educationalists believe that it would be more advantageous to

18
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

• This paper has presented a preliminary evaluation of the game even specifying level of programming ability as novice, interme­
assimilated into a HE module however a more detailed and stringent diary or expert.
experimental study could be performed in other modules and runs of • Finally, the content integration framework presented in this study
the same module to ascertain validity and utilise a Randomised was a preliminary framework which could undergo refined future
Controlled Trial (RCT) to gain deeper comparative insights of games development.
and traditional teaching approaches.
• Due to the size and complexity of this study it was not possible to CRediT authorship contribution statement
fully detail the analysis of all of the studies found in the systematic
literature review as it was beyond the scope of the study. Therefore, Thomas Hainey: Conceptualization, Formal analysis, Funding
further detailed analysis of the systematic literature review findings acquisition, Investigation, Methodology, Project administration, Re­
will be performed for the following reasons: sources, Software, Supervision, Validation, Visualization, Writing –
o to fully detail the developed and existing games where 187 papers original draft. Gavin Baxter: Writing – review & editing.
were found. It should be noted that there are duplicate games in
these papers as there can be several papers published associated
Declaration of competing interest
with one game for teaching programming.
o to fully detail the problems associated with programming where
The authors declare that they have no known competing financial
91 papers were found.
interests or personal relationships that could have appeared to influence
• A taxonomy of existing serious games will be developed for the use of
the work reported in this paper.
educationalists and serious game researchers particularly in the area
of computer programming. This would take some time to produce
Acknowledgements
but the idea would be that educationalists would be able to augment
their material with a number of supplementary serious games to
This work is funded by the Carnegie Trust – Grant Number:
assist learners.
RIG007432.
• Further research is required in the area of formative and summative
assessment mechanisms to ascertain how learners in computer pro­
gramming courses would like to be assessed and also an additional Appendix A. Supplementary data
framework could be developed in relation to formative assessment to
gauge levels of experience, knowledge, and skill. This is particularly Supplementary data to this article can be found online at [Link]
relevant in relation to articulation and direct entry into university org/10.1016/[Link].2024.100061.
courses.
• No significant differences were found between the game and non- References
game groups in relation to class tests and overall grades and this
ADPRIMA Instructional methods information website.(2019). Retrieved 6 August, 2019,
warrants further rigorous investigation. A more rigorous RCT should from [Link]
be conducted with a suitable representative sample in each group to Aldrich, C. (2004). Simulations and the future of learning: An innovative (and perhaps
allow parametric statistical tests to test for statistical significance. Revolutionary) approach to e-learning. San Francisco: Pfeiffer.
Aldrich, C. (2005). Learning by doing: A comprehensive guide to simulations computer
• A disparity was discovered between educationalists and learners and games, and pedagogy in e-learning and other educational experiences. Pfeiffer.
future research could collect the perceptions, attitudes, and moti­ Alhazbi, S., & Ismail, L. S. (2010). Supportive online learning environment to improve
vation of programming educationalists to obtain a different student’ satisfaction in object-oriented programming courses. In In Proceedings of the
2nd international Congress on engineering education, december 8 – 9, Kuala Lumpar,
perspective and increase understanding.
Malaysia.
• While this study has been primarily quantitative, additional quali­ Anderson, E. F., & McLoughlin, L. (2007). Critters in the classroom: A 3D computer
tative studies could be performed to obtain deeper analysis to adhere game-like tool for teaching programming to computer animation students. In In
Proceedings of SIGGRAPH ’07 San Diego, California, August 05 -09.
to pedagogical, evaluation and assessment standards.
Andrianoff, S., & Levine, D. (2002). Role playing in an object-oriented world. In In
• From a design and development perspective of the current prototype Proceedings of the 33rd SIGCSE technical Symposium on computer science education,
game there are four particular future research avenues that can be Kentucky, USA (pp. 121–125). ACM Press.
explored: Arsenault, D. (2009). Video game genre, evolution and innovation. Eludamos Journal of
Computer Game Culture, 3(Issue. 2), 149–176.
o Firstly, validation of the game in other levels of education and in Backland, P., & Hendrix, M. (2013). Educational games – are they worth the effort? A
other institutions to gather more conclusive empirical data to literature survey of effectiveness of serious games. In Proceedings of 5th international
support the validity of the approach. So for example allowing the Conference on Games and virtual Worlds for serious applications (VS - GAMES), 11-13
Sep 2013, Poole UK.
game to be used in Further Education and Secondary Education Baker, A., Zhang, J., & Caldwell, E. R. (2012). Reinforcing array and loop concepts
and possibly other Higher Education institutions. through a game-like module. In In proceedings of the 2012 17th international
o Secondly, the game could be translated into other programming conference on computer games (CGAMES), july 30 – Aug 1, Louisville, KY, USA.
Berkling, K., & Thomas, C. (2013). Gamification of a Software Engineering Course and a
languages as it is currently customised for teaching C#. Despite the detailed analysis of the factors that lead to its failure. In Proceedings of the 2013
fact that programming principles are transferrable it may be more international Conference on interactive collaborative learning (ICL), September 25 – 27,
effective to convert the game to the programming languages that Kazan, Russia.
Bernstein, L., & Klappholz, D. (2001). Getting software engineering into our guts.
the students are learning in their course such as C++, Java, Python
CrossTalk. The Journal of Defense Software Engineering, 25–26.
etc. Biggs, J. (1999). Teaching for quality learning at university. Buckingham: Open University
o Thirdly, the game could be expanded to be generic in relation to Press.
Blair, J., Czaja, R. F., & Blair, E. A. (2013). Designing surveys: A guide to decisions and
allowing the educationalists to add their own pedagogical content.
procedures. SAGE Publications, Inc.
This may mean the development of an editor tool to allow teachers Bonwell, C. C. (1996). Enhancing the lecture: Revitalizing a traditional format. In
to add questions for their students similar to the Dialog Editor that T. E. Sutherland, & C. C. Bonwell (Eds.), In Using active learning in college classes: A
was created for the Requirements Collection and Analysis Game range of options for faculty, new Directions for Teaching and learning No. 67 (pp.
31–44). Place: Wiley Periodicals Inc.
(Hainey, 2010). Boticki, I., Barisic, A., Martin, S., & Drljevic. (2012). Sortko: Learning sorting algorithms
o Fourthly, the game could have challenge and difficulty custom­ with mobile devices. In In proceedings of the seventh IEEE international conference on
isation implemented into it to allow for adjustment to program­ wireless, mobile and ubiquitous technology in education. March 27-30, Takamatsu,
Japan.
ming knowledge e.g. specifying a level of easy, medium or hard or Boyle, E. A., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., et al. (2016). An
update to the systematic literature review of empirical evidence of the impacts and

19
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

outcomes of computer games and serious games. Computers & Education, 94, Hidalgo-Céspedes, J., Marín-Raventós, G., & Lara-Villagrán, V. (2014). Playing with
178–192. metaphors: A methodology to design video games for learning abstract programming
Browne, K., & Anand, C. (2013). Gamification and serious game approaches for concepts. ITICSE, 14. Jun 21-25 2014, Uppsala, Sweden.
introductory computer science tablet software. In Gamification’13 proceedings of the Hwang, C.-S., Su, Y.-C., & Tseng, K.-C. (2010). Effects of computer game-based
first international conference on gameful design, research and applications, Toronto, instruction on students’ programming achievement in Taiwan. In In proceedings of the
Ontario, Canada (pp. 50–57). international conference on computational aspects of social networks (pp. 233–236).
Cagin, K. (2013). Empirical evidence that proves a serious game is an educationally Ibrahim, R., Yusoff, R. C. M., Omar, H. M., & Jaafar, A. (2011). Students perceptions of
effective tool for learning computer programming constructs at the computational using educational games to learn introductory programming. Computer and
thinking level. In PhD thesis, university of Greenwich. Information Science, 4(Issue No 1).
Cashin, W. E. (1985). Improving lectures idea paper No. 14. Manhattan. Kansas State Jantan, S. R., & Aljunid, S. A. (2012). An experimental evaluation of scaffolded
University, Center for Faculty Evaluation and Development. educational games design for programming. In In proceedings of the 2012 IEEE
Chaffin, A., Doran, K., Hicks, D., & Barnes, T. (2009). Experimental evaluation of conference on open systems (pp. 21–24). Malaysia: Kuala Lumpur.
teaching recursion in a video game. In In Sandbox ’09 proceedings of the 2009 ACM Jayasinghe, U., & Dharmaratne, A. (2013). Game based learning vs. Gamification from
SIGGRAPH symposium on video games (pp. 79–86). New Orleans, Louisiana. higher education students’ perspectives. In In proceedings of the 2013 IEEE
Chang, M.&K. (2010). Web-based multiplayer online role playing game (MORPG) for international conference on teaching, assessment and learning for engineering (TALE),
assessing students’ Java programming knowledge and skills. In In proceedings of the August 26 – 29, Bali, Indonesia.
third IEEE international conference on digital game and intelligent toy enhanced learning Jiau, H. C., Chen, J. C., & Ssu, K.-F. (2009). Enhancing self-motivation in learning
(DIGITEL). programming using game-based simulation and metrics. IEEE Transactions on
Chang, W.-C., Chou, Y.-M., & Chen, K.-C. (2010). Game-based digital learning system Education, 52(4).
assists and motivates C programming language learners. In In proceedings of the 6th Jordaan, D. B. (2018). Board games in the computer science class to improve students’
international conference on networked computing and advanced information management knowledge of the Python programming language. In In proceedings of the international
(pp. 10–16). conference on intelligent and innovative computing applications (ICONIC), december 6 –
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). 7. Maritus: Plain Magnien.
A systematic literature review of empirical evidence on computer games and serious Karapinar, Z., Zavarak, S., Senturk, A., Kara, R., & Erdogmus, P. (2012). A game to test
games. Computers & Education, 59(2), 661–686. pointers: Path finding. In In proceedings of the 2012 international conference on
Connolly, T. M., Stansfield, M. H., & Hainey, T. (2007). An application of games-based information technology based higher education and training (ITHET) (pp. 21–23).
learning within software engineering. British Journal of Educational Technology, 38 Kaučič, B., & Asič, T. (2011). Improving introductory programming with scratch?. In In
(Number 3), 416–428. Proceedings of the 34th international convention MIPRO, may 23 – 27, Opatija, Croatia.
Connolly, T., Stansfield, M., & Hainey, T. (2009). Towards the development of a games- Kawash, J., & Collier, R. (2013). Using video game development to engage
based learning evaluation framework. In T. Connolly, M. Stansfield, & L. Boyle undergraduate students of assembly language programming. In In SIGITE’13
(Eds.), Games-based learning advancements for multi-sensory human computer interfaces: Proceedings of the 14th annual ACM SIGITE conference on Information technology
Techniques and effective practices (pp. 251–273). Hershey, PA, USA: IGI Global. education (pp. 71–76).
Connolly, T. M., Stansfield, M. H., McLellan, E., Ramsay, J., & Sutherland, J. (2004). Ketamo, H., Kiili, K., Arnab, S., & Dunwell, I. (2013). Integrating games into the
Applying computer games concepts to teaching database analysis and design. In In classroom: Towards new teachership. In S. de Freitas, M. Ott, M. M. Popescu, &
proceedings of the international conference on computer games, AI, design and education, I. Stanescu (Eds.), New pedagogical approaches in game enhanced learning: Curriculum
reading, UK, november 2004. integration (pp. 534–537). IGI Global. [Link]
Conole, G., & Warburton, B. (2005). A review of computer-assisted assessment. ALT-J, ch007.
Research in Learning Technology, 13(1), 17–31. Khenissi, M. A., Essalmi, F., & Jemni, M. (2013). Presentation of a learning game for
Cope, C., & Horan, P. (1996). The role played case: An experiential approach to teaching programming languages education. In In Proceedings of the 13th International
introductory information systems development. Journal of Information Systems conference on advanced learning technologies. July 15 – 18.
Education, 8(2–3), 33–39. Kumar, B., & Sharma, K. (2018). A gamified approach to achieve excellence in
Davis, B. G. (2001). Tools for teaching. San Francisco, CA: Jossey-Bass Publishers. programming. In In Proceedings of the 4th international Conference on computing
Dawson, M. (2014). Beginning C++ through game programming. In 4th edition, Boston, sciences (ICCS), August, (pp. 30–31).
USA, CANGAGE learning. Law, R. (2012). Teaching programming using computer games: A program language
De Aquino Leal, A. V., & Ferreira, J. (2013). Teaching computer programming based on agnostic approach. In In Proceedings of the 6th European Conference of games-based
pattern with activities and collaborative games using concrete materials for high learning (ECGBL), Cork, Ireland (pp. 4–5).
school students. Proceedings of the IEEE Frontiers in Education Conference (FIE), 1, Lawrence, R. (2004). Teaching data structures using competitive games. IEEE
1604–1610. Transactions on Education, 47(4), 459–466.
Deitz, S., & Buy, U. (2016). From video games to debugging code. In In proceedings of Lee, M. J., Bahmani, F., Kwan, I., LaFerte, J., Charters, P., Horvath, A., et al. (2014).
IEEE/ACM 5th international workshop on games and software engineering (GAS), Austin, Principles of a debugging-first puzzle game for computing education. In In
TX, USA, may 16 2016. proceedings of the IEEE symposium on visual languages and human-centric computing
Doss, K., Juarez, V., Vincent, D., Doerschuk, P., & Liu, J. (2011). Work in progress – a (VL/HCC), july 28 August 1, Melbourne, VIC, Australia.
survey of popular game creation platforms used for computing education. In In Lotfi, E., & Mohammed, B. (2018). Teaching object oriented programming concepts
proceedings of 2011 frontiers in education conference (FIE), Oct 12 – 15, Rapid city, SD, through a mobile serious game. In In SCA’19 proceedings of the 3rd international
USA. conference on smart city applications, Tetouan, Morocco, October (pp. 10–11).
Echeverria, A., Garcia-Campe, C., Nussbaum, M., Gil, F., Villalta, M., Amestica, M., et al. Macphail, A., Hainey, T., & Connolly, T. M. (2012). Applying mLearning in software
(2011). A framework for the design and integration of collaborative classroom engineering education: A survey of mobile usage. In In proceedings of IADIS
games. Computers and Education, 57(1), 1127–1136. international conference mobile learning (pp. 107–114).
Finkelstein, S. L., Nickel, A., Harrison, L., Suma, E. A., & Barnes, T. (2009). cMotion; A Malik, D. (2014). C++ programming: Program design including data structures, 7 edition.
new game design to teach emotion recognition and programming logic to children CENGAGE Learning.
using virtual humans. In In proceedings of IEEE virtual reality conference, March 14 – Malliarakis, C., Satratzemi, M., & Xinogalos, S. (2014). Integrating learning analytics in
18, Lafayette, LA, USA. an educational MMORPG for computer programming. In In Proceedings of the 14th
Gerling, K. M., Birk, M., Mandryk, R. L., & Doucette, A. (2013). The effects of graphical international Conference on advanced learning technologies, july 7 – 10. Athens, Greece.
fidelity on player experience. In In Proceedings of the 13th international Conference on Marques, B. R. C., Levitt, S. P., & Nixon, K. J. (2012). Software visualisation through
making Sense of converging media (p. 229). video games. In In Proceedings of SAICSIT’12, October 01-03. Pretoria, South Africa.
Gomez-Martin, M. A., Gomez-Martin, P. P., & Gonzales-Calero, P. A (2009). Content Masso, N., & Grace, L. (2011). Shapemaker: A game-based introduction to programming.
Integration in Games-Based Learning Systems. In T. M. Connolly, & M. H. Stansfield In In Proceedings of the 16th international Conference on computer games (CGAMES).
(Eds.), Games-based Learning Advancement for Multisensory Human Computer July 27-30, Louisville, KY, USA.
Interfaces: Techniques and Effective Practices. Hershey: Idea-Group Publishing. Miljanovic, M. A., & Bradbury, J. S. (2017). RoboBUG: A serious game for learning
Hainey, T. (2010). Using games-based learning to teach requirements collection and debugging techniques. In In ICER ’17 proceeding of the 2017 ACM conference on
analysis at tertiary education level. Doctoral Dissertation. Retrieved April 2019 from international computing education research (pp. 93–100).
[Link] Mitamura, T., Suzuki, Y., & Oohori, T. (2012). Serious games for learning programming
pdf. languages. In In Proceedings of the 2012 IEEE international Conference on systems, Man
Hainey, T. (2016). Moving digital games for learning forward. On the Horizon, 24(1), and cybernetics (SMC), October 14 – 17, Seoul, South Korea.
132–136. Morsi, R., Richards, C., & Rizvi, M. E. (2010). Work in progress – BINX: A 3D XNA
Hainey, T., Connolly, T. M., Boyle, E. A., & Wilson, A.&R. (2016). A systematic literature educational game for engineering education. In In Proceedings of the 40th Frontiers in
review of games-based learning empirical evidence in primary education. Computers education conference. October 27 -30, Washington DC.
& Education, 102, 202–223. Naul, E., & Liu, M. (2020). Why story matters: A review of narrative in serious games.
Hainey, T., Connolly, T. M., Stansfield, M. H., & Boyle, E. A. (2011a). Evaluation of a Journal of Educational Computing Research, 58(3), 687–707. [Link]
game to teach requirements collection and analysis in software engineering at 10.1177/0735633119859904
tertiary education level. Computers & Education, 56(1), 21–35. Polack-Wahl, J. (1999). Incorporating the client’s role in a software engineering course.
Hainey, T., Connolly, T. M., Stansfield, M. H., & Boyle, E. A. (2011b). The differences in In Proceedings of the 30th SIGSCE technical symposium on computer science education
motivations of online game players and offline game players: A combined analysis of (pp. 73–77). Los Angeles, USA: ACM Press.
three studies at higher education level. Computers & Education, 57(4), 2197–2211. QAA. (2012). Understanding assessment: Its role in safeguarding academic standards and
Hays, R. T. (2005). The effectiveness of instructional games: A literature review and quality in higher education: A guide for early career staff. Retrieved 15 October 2016
discussion. DTIC Document.

20
T. Hainey and G. Baxter Computers & Education: X Reality 4 (2024) 100061

from [Link] Sung, K. (2009). Computer games and traditional CS courses. Communications of the ACM,
[Link] . 52(12), 74–78.
Rosas, R., Nussbaum, M., Cumsille, P., Marianov, V., Correa, M., Flores, P., et al. (2003). Tan, P.-H., Ting, C.-Y., & Ling, S.-W. (2009). Learning difficulties in programming
Beyond nintendo: Design and assessment of educational video games for first and courses: Undergraduates’ perspective and perception. In Proceedings of the 9th
second grade students. Computers & Education, 40, 71–94. international conference on computer technology and development (Vol. 1, pp. 42–46).
Sauvé, L. (2010). Effective educational games. In D. Kaufman, & L. Sauvé (Eds.), Washington, DC, USA: IEEE.
Educational gameplay and simulation environments: Case studies and lessons learned (pp. Tang, S., & Hanneghan, M. (2010). Designing educational games: A pedagogical
27–50). Portland, OR: Book News, Inc. approach in design and implementation of educational games: Theoretical and
Scottish Funding Council. (2016). Articulation. Retrieved 20 August 2019 from htt practical perspectives. In P. Zemliansky, & D. Wilcox (Eds.), Hershey, PA: IGI global,
p://[Link]/web/FILES/CMP_AccessandInclusionCommittee16February ISBN13: 9781615207824 (pp. 108–125).
_17022016/AIC16_07_Articulation_Policy_Update.pdf. Vahldick, A., Mendes, A. J., & Marcelino, M. J. (2014). A review of games designed to
Scottish Qualifications Authority (SQA). (2018). Young Scot report. Retrieved January improve introductory computer programming competencies. In Frontiers in education
2024 from [Link] conference (FIE) (Vol. 2014, pp. 1–7). IEEE.
[Link]. Wassila, D., & Tahar, B. (2012). Using Serious game to simplify algorithm learning. In In
Shabana, S., & Chen, J. X. (2009). Simplifying algorithm learning using serious games. In proceedings of the international conference on education and e-learning innovations. July
In the proceedings of the 14th Western Canadian conference on computing education, 1 – 3, Sousse, Tunisia.
Burnaby, British Columbia, Canada. May 1 – 2, 2009 (pp. 34–41). Wei, C.-S., Chen, Y., & Doong, J.-G. (2009). A 3D virtual world teaching and learning
Shabanah, S. S., Chen, J. X., Wechsler, H., Carr, D., & Wegman, E. (2010). Designing platform for computer science courses in second Life. In In proceedings of the
computer games to teach algorithms. In In Proceedings of the seventh international international conference on computational intelligence and software engineering.
Conference on information technology: New Generations on information technology: New Winn, B. M. (2009). The design, play, and experience framework. In R. E. Ferdig (Ed.),
generations, April 12 – 14, Las Vegas, NV, USA. Handbook of research on effective electronic gaming in education (Vol. III, pp.
Shaw, K., & Dermoudy, J. (2005). Engendering an empathy for software engineering. In , 1010–1024). Hershey PA: Information Science Reference.
Vol. 42. Proceedings of the 7th australasian computing education conference (ACE2005), Wong, Y. S., Yatim, M. H.b. M., & Tan, W. H. (2015). Learning object-oriented
Newcastle, Australia (pp. 135–144). programming with computer games: A game-based approach. In Proceedings of the 9th
Sheth, S., Bell, J., & Kaiser, G. (2013). A competitive-collaborative approach for European Conference on games-based learning (ECGBL), Steinkjer, Norway, October 8-9.
introducing software engineering in a CS2 class. In In proceedings of the 26th Xiao, D., & Miller, R. C. (2014). A multiplayer online game for teaching software
international conference on software engineering education and training (CSEE&T), may engineering practices. In In Proceedings of the first ACM conference on Learning @ scale
19 – 21, San Francisco, CA, USA. conference. Atlanta, Georgia, USA March 4 – 5.
Simsarian, K. (2003). Take it to the next stage: The roles of role playing in the design Yan, L. (2009). Teaching object-oriented programming with games. In Proceedings of the
process. In In proceedings of CHI 2003, April 5-10, 2003, Ft. Lauderdale, Florida, USA. sixth international conference on information technology: New generations (pp.
ACM 1-58113-637-4/03/0004. 969–974). IEEE Computer Society.
Siu, K., Guzdial, M., & Riedl, M. O. (2017). Evaluating singleplayer and multiplayer in Yap, J. (2011). Virtual world labyrinth: An interactive maze that teaches computing. In
human computation games. In In Proceedings of the 12th international Conference on In proceedings of the defence science research conference and expo (DSR). August 3 – 5,
the Foundations of digital games. Article No. 34. Singapore.
Sun, Q., Phelps, A. M., Egert, C. A., & Bayliss, J. D. (2009). Games in the classroom at the
rochester Institute of technology. IEEE Computer Society.

21