Education for Chemical Engineers 36 (2021) 54–64

Contents lists available at ScienceDirect

Education for Chemical Engineers

journal homepage: [Link]/locate/ece

Moodle and Socrative quizzes as formative aids on theory teaching in

a chemical engineering subject
E. Romero ∗ , L. García, J. Ceamanos
Chemical & Environm. Engineering Dpt., Architecture & Engineering College (EINA), Aragon Institute of Engineering Research (I3A), University of Zaragoza,
Mariano Esquillor s/n, 50018 Zaragoza, Spain

a r t i c l e i n f o a b s t r a c t

Article history: There is a majority opinion on the positive influence of using technologies in teaching. However, few
Received 3 June 2020 studies account for their effect on students’ final grades. Traditional theory teaching often shows a lack of
Received in revised form 31 January 2021 students’ motivation, engagement and self-efficacy. A way to improve these is to include quizzes, which
Accepted 2 March 2021
may allow students to gain in skills acquisition feedback and self-regulation, and to control their own
Available online 5 March 2021
way to construct knowledge. The suggested methodology embeds quizzes and tests through two ICT,
each one with a different strategy. One is the use of quick and real-time quizzes with Socrative, focused
Keywords:
in promoting motivation and engagement in the classroom. The second is the use of tests with Moodle
Traditional lecture
Blended learning
for a longer time scale (blocks of lessons). Both tests and quizzes were designed as formative activities:
Grades the students are able to gain feed-back, weaknesses identification and better programming of their work.
Outcomes Three ‘tools’ (one Moodle test; three Socrative quizzes; no tool/only traditional teaching) were rotated
Motivation among three different cohorts and blocks in a Chemical Engineering subject (n=49 students). Effects on
Moodle the students’ learning outcomes (grades) at the final theory exam were evaluated by blocks. A paired-data
Test one-factor ANOVA test showed no significant statistical differences in using Moodle tests or Socrative
Socrative quizzes or not. Some possible negative influences over data were identified and improvements for further
Quiz
study have been suggested. An intensification of quizzes and tests could be relevant. Although no signif-
ICT
icant improvement over grades was obtained, the use of both tools was positively valued for students
ANOVA
Facility Index and instructors. Surveys’ results showed that the proposed methodology may create a more attractive
and self-regulated educational environment. However, this study supports that students’ previous per-
ceptions about the tools may condition their final perceptions over the ICTs real aid for learning and their
future academic outcomes.
© 2021 Institution of Chemical Engineers. Published by Elsevier B.V. All rights reserved.

1. Introduction the changes that it brings about at a social and educational level.
Any university professor can check daily how involved students
The typical passivity of some students in classrooms and classes, are in the use of different social networks and live communication
especially those with theory content or, in general, traditional lec- software. The question that arises is: why not to take advantage of
tures and master classes, all of which are expository, is well known these tools for the benefit of university teaching? In other words,
to all instructors, especially at higher education. This may require why not reduce the technological gap between students and teach-
an action from faculty for engaging the students in the teaching ers/teaching? The use of the new ICT tools implies, in the first place,
activity. There are efforts to improve the classical lecture structure an approach to the technological reality of our students, which can
by new advanced processes (Ben-Av and Ben-Av, 2015). facilitate their larger involvement in the subjects (Romero et al.,
In turn, our society is immersed in a continuous evolution- 2019). Secondly, they represent a very important support for tra-
ary process, within which the field of technology is paradigmatic. ditional teaching, which can use many of these tools and software
Today’s students have very useful and powerful tools such as smart- programs, especially if they are free, facilitating and enhancing their
phones, tablets, laptops, etc. This reality is unquestionable, just like application and effects. For these reasons blended learning, as a
mixture of face-to-face instruction and e-learning, has spreaded
across the world and gained in popularity (Mirriahi et al., 2015;
Drysdale et al., 2013; Halverson et al., 2014), also in higher edu-
∗ Corresponding author. cation (Norberg et al., 2011; Smith and Hill, 2019). Nevertheless,
E-mail address: eromero@[Link] (E. Romero).

[Link]
1749-7728/© 2021 Institution of Chemical Engineers. Published by Elsevier B.V. All rights reserved.
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

some aspects remain unclear in the literature, as a lack of consid- et al., 2006), system functionality (Pituch and Lee, 2006) or self-
eration of the faculty perspective (Kaleta et al., 2007; Smith and efficacy (Teo, 2009). Utility and ease of use (Sumak et al., 2011) and
Hill, 2019) and a relatively little quantitative data capture about performance expectancy and social influence (Sumak et al., 2010)
blended learning use (Smith and Hill, 2019). are the most important factors that can influence the attitudes of
Successful university students are usually described as self- users toward using an e-learning technology.
regulated students (Nunez et al., 2011; Perry et al., 2008; Schunk But what is really important is the question: are both students’
and Zimmerman, 2008). One of the technological tools that learning and academic results improved with the use of techno-
improves the self-regulated learning and at the same time the tra- logical advances? Islam (2013) studied the use of Moodle in hybrid
ditional lecture is the use of quizzes, tests and surveys through the (traditional and e-learning mixed) courses, through chats, exams,
student response systems (SRSs). SRSs have experienced a huge wikis, news, discussion forums, reading materials, and submitting
evolution from bulky computer terminals to simple standalone assignments. The study showed that the students’ prior perceptions
electronic device (clickers) to downloaded applications in stan- about the usefulness and ease of use determine the subsequent per-
dard smartphones. Clickers are interesting devices for quizzing ceptions of the real aid for learning and the aid for participation and
and have shown a positive effect on active learning (Kennedy and motivation. The students’ opinion over their future academic per-
Cutts, 2005; Stowell and Nelson, 2007) but have strong drawbacks: formance is influenced, therefore, by their initial impressions about
high price and complicated deployment. Currently all the higher these systems such as Moodle.
education students ‘wear’ smartphones so, by using these devices, Moodle use has reported to be an efficient way to improve aca-
institutions can avoid purchase, maintenance and devices manage- demic achievement in some important specific aspects: declarative
ment for SRS. knowledge, general and on text use of learning strategies, deep
The electronic devices and technological supported teaching approach to learning, self-regulation and statistically significant
help students to improve their learning (Remon et al., 2017). Today differences in academic outcomes (Nunez et al., 2011). Novo-Corti
smartphones have larger data processing capacity than big comput- et al. (2013) showed a promotion of participation, motivation,
ers existing some years ago. Internet and cloud-based storage have competence and qualifications in the students by using a mixed
deeply changed the work of many professionals around the world e-learning (Moodle multiple choice tests) and face to face (tradi-
and already have introduced a new way to work also in schools and tional classroom exams) teaching. Yang et al. (2013) also provided
universities. a study with students’ significantly improved English learning skills
The joining of smartphones and tablets and software devel- using Moodle.
opment has become a productive area in many fields, also in The information collected during the use of Moodle to manage
Education. Programs and apps as Socrative and others allow teach- a course can be useful in relation to predicting the academic results
ers to make direct online and real-time quizzes, surveys and of the students. Romero et al. (2013) developed a specific Moodle
tests (assessments) in the classroom (Artal-Sevil et al., 2017). mining tool aimed at instructors to classify the students depend-
This improves student-teacher interactions, student motivation ing on their usage of courses in order to predict the students’ final
(Guerrero et al., 2016), engagement in the subject (Ferrandiz et al., academic marks. As an important tool for helping the traditional
2016; Lim, 2017), and feedback (students know their skills, and lecture, the use of quizzes was showed in the study as the main
teachers know how the students are understanding the lessons to determiner of final marks, though some other activities could help
make more efforts on some specific topics). The Socrative app has the instructor to decide whether to promote or reduce some of
experienced a large growth, including the university level, for the them to improve the outcomes. Moodle quizzes can also be used to
last years and currently is a leader in this type of software (Pryke, improve formative assessments (Cohen and Sasson, 2016).
2020), since it was developed in 2010 at MIT (Pentalog, 2020). Software apps for smartphones can be great tools to improve
Among the different technologies available for education, the the traditional lecture by making quizzes, tests and surveys. The
learning management systems (LMS) have become usual, espe- Socrative application has been used for years in education at all
cially at higher education. The COVID-19 disease has also urged levels. During 2012–2013 school year this software recorded 116
the whole teaching community to focus on e-learning and online million responses to questions and more than 3 million teachers
teaching. These systems allow instructors and universities to create and students created 278,000 quizzes, whereas in 2018 there were
and manage subjects, courses, or entire degrees and masters. This about 60 million student accounts and more than 3 billion student
is probably why today LMSs have become a common and essential responses recorded, and currently the application has nearly 3 mil-
tool for education (Alvarez et al., 2013). Most of educational cen- lion users worldwide and is available in 14 languages (Pentalog,
ters around the world use them as a way to achieve better quality 2020). It is a software that allows running of previously online pre-
education and to provide a new learner-centered teaching (Islam, pared tests by instructors. This e-learning tool runs via the web
2012). and native iOS and Android mobile apps. Students can take the
One of the available LMSs is Moodle. It has some advantages as tests on their own portable device (smartphone, tablet, laptop, etc.)
low memory using, low bandwidth consuming, a lot of functionali- and anywhere, not only at classroom. The use of their own devices
ties, robust construction and reliable use. Moodle is surely the most makes the Socrative application very interesting from the point of
widespread example of LMS. It has many years of use and a great view of new technologies assisted learning. It involves a ‘different’
development that allows multitasking within the educational field. use of the device (mostly used to talk, chat, play and connect to
One of the most interesting features to support traditional teach- social networks), which the students are completely familiar with.
ing is the evaluation of students through tests and quizzes. These The use of Socrative is not limited to a specific field of knowl-
quizzes can be designed in many different ways (multiple-choice, edge. Because the instructor can implement every discipline
short answer, etc.) and can be configured by multiple options. It is teaching goals by suitable questions and quizzes, any class can be
a tool that allows the teacher to determine the level of knowledge enhanced by the use of this application. In the university level, par-
acquisition and to observe the evolution of the students through ticularly for first-year students, there is a low interest in learning
the semester. the tough subjects. This is surely one of the main reasons for the
There are studies on the adoption and use of e-learning tech- usually high failure rates. For this reason, more and more teach-
nology, specifically focused on LMSs such as Moodle. Some of these ers try to use new technologies to stimulate students. Socrative is
studies assessed ease of use (Sumak et al., 2011), utility (Larsen an application with a friendly environment and allows to measure
et al., 2009), system quality (Islam, 2012), service quality (Roca in real time and online the level of knowledge, the students’ skills

55
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

and also to follow their evolution. In the first year, students may ficult to use. Main feature of Socrative is conducting surveys and
also have a certain refusal to show themselves in public. Socrative tests.
has the possibility of being used anonymously, so this can help to This study has been designed as a combination of e-learning
the students’ disinhibition when answering questions. In fact, the and face-to-face approach, as pretends Moodle tests and Socrative
work from Frias et al. (2016) found the students were more likely to quizzes to support the traditional theory teaching. There is still no
answer the questions from their own mobile devices than to solve general consensus on definitions for mixing both methodologies
them going to the blackboard, they recognized that this applica- (Smith and Hill, 2019). Some authors do not differentiate hybrid
tion helped them pass the course and they showed great interest from blended learning (Allen and Seaman, 2010). Percentages of
in extending its use to other subjects. Guerrero et al. (2016) stud- each can determine the classification of learning (fully face-to-
ied the effect of using Socrative on the participation of engineering face, web-enhanced, blended, hybrid, and fully e-learning/online)
students in class. The work showed that participation increased (Smith and Kurthen, 2007). This last study considers as blended a
when mobiles were used in theory class. Additionally, students course in which the instructor adds, beyond an online syllabus and
were willing to participate in activities and share their own results. a few online documents, some significant online learning activities
An increase in student engagement after using Socrative also as, specifically, online quizzes or a few online discussions.
was found by Lim (2017). Low level of engagement conducts to Regarding blended learning, it has become increasingly popu-
poor learning performance, so any way to increase the engagement lar in all levels (Glazer, 2012) and specifically at higher education
can address this problem. The way for improving performance (Namyssova et al., 2019). Several studies in different environments
can be through different teaching strategies, methods and/or tools. have been proven blended ways as more effective than pure face-
The study showed the use of Socrative to make polls, exercises, to-face teaching or pure online instruction. Means et al. (2010)
quizzes and games stimulated the discussion and favored the feed- found better students’ outcomes in blended learning compared to
back between instructor and students. The students reported to traditional instruction after reviewing 99 studies. Similar results
be felt more engaged with the new interactive model. The study were shown by Smith and Hill (2019) in a review of blended learn-
analyzed students’ feedback academic results, attendance records ing in higher education articles published from 2012 to mid-2017.
and instructor’s teaching evaluation scores. The results showed However, it remains unclear under which circumstances blended
enhancing of the students’ learning experience and, which is more learning environments are successful (Oliver and Trigwell, 2005;
significative, improving of their academic performance. Similar Van Laer and Elen, 2020). Studies suggest that blended learn-
results were found by Remon et al. (2017). ing challenges learners’ self-regulation and results indicate ‘fewer
However, the use of new technologies (mobile device applica- mis-regulators when more self-regulatory design features are inte-
tions) does not necessarily imply a satisfactory improvement in grated’ (Van Laer and Elen, 2020). These finding highlights the
the academic results of the students. In general, initial percep- value of integrating features that support self-regulation in blended
tions about usefulness and ease of use, and even social aspects, can learning environments. E-quizzes are considered an important tool
influence the attitudes of students toward using of technologies. for helping traditional lecture (Romero et al., 2013) and good tools
Chou et al. (2017) observed that students from secondary educa- for self-regulation because allow the students to gain insight in
tion who used the BYOD (bring your own device) approach with the knowledge and competences and to tune their work and time in
Socrative test had worse results in the formative tests than the stu- order to improve them and have been proven to ‘positively affect
dents with traditional instruction. The authors reported that this the effectiveness and attractiveness of blended learning’ (Spanjers
was due to the fact that some students (out of a total of 46) were et al., 2015).
not familiar with the BYOD approach. In the summative evaluation, This work has focused on the use of Moodle tests and Socrative
both learning pathways showed similar results. However, the study quizzes as a support for traditional theory teaching, which on many
showed a uniform growth of learning outcomes in students who occasions the students see as difficult, tedious and boring. The goal
used new technologies and scored higher in the segment of learn- of using technology has been to improve the face-to-face theory
ing retention. In addition, the BYOD learning practice enhanced the teaching, but maintaining the essence of it. No variation of tradi-
advanced students’ motivation and interest. The authors suggested tional theory teaching was addressed, only the inclusion of two
to schools and teachers exploring new creative ways to integrate technological aids to the students for gaining and retaining con-
traditional teaching and BYOD approach. These findings appear cepts and materials, with a small percentage of total activities. For
easily related to higher education as well, especially in those stu- this, different tests in Moodle and quick tests (quizzes) in Socra-
dents using smartphones for learning (BYOD approach) at first time. tive have been developed, with different teaching strategies. The
Computer-based learning environments can add more difficulties work described in this paper can be so considered as a blended
to students to achieve self-regulating processes while learning in K- learning design study. As a relevant goal, this study tries to evalu-
12 environment (Quintana et al., 2005) and also at higher education ate the specific effect of the two tools use on the students’ grades.
level (Azevedo et al., 2004; Graesser et al., 2005). These environ- It has been also considered crucial to know about perception and
ments may imply more effort by the students to decide time to opinions for the both selected ICT. For this reason, students’ and
learn, time to invest, when to change strategies, when to increase lecturers’ surveys have been carried out.
or save effort, etc. (Azevedo et al., 2005).
New technologies must be applied carefully with simple but
clever designed and scheduled activities. For these reasons, quizzes
and tests appears as interesting activities to be used with new ICTs 2. Methods
in order to students to gain more self-regulated learning. The two
tools, Moodle and Socrative, are easily accessible to instructors and 2.1. Target
students. Moodle is freely available for universities, instructors and
students. The developer of Socrative has made available to any user The study was carried out during Spring 2019 in a 6th semester
a free version that allows all the main and basic functions to be per- Chemical Engineering subject, ‘Industrial Chemical Processes’, at
formed, so that both students and teachers can use it without any the Engineering College from a university located at Northern
problem. Both tools can be used for multiple teaching tasks. In this Spain. Among different activities, the subject’s syllabus includes
aspect, Moodle, especially, is a very versatile system and has many seven lessons (out of eight) directly related with theory (1–2 and
options to interact with students, although sometimes can be dif- 4–8):

56
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

Lesson 1: Introduction. Relevant and distinctive characteristics of Table 1

Chart with the different ‘tools’ used by block and students’ group.
Chemical Industry.
Lesson 2: Basic knowledge of chemical reactors and separation Cohort Block Tool
operations. I Moodle
Lesson 3: Mass and energy balances problems. A II Socrative
Lesson 4: Processes without chemical reactions: separation of air III None
by cryogenic distillation, adsorption and membranes. I Socrative
Lesson 5: Processes with reaction: sulfuric acid production. B II None
Lesson 6: Energy: carbon combustion in fluidized bed, carbon gasi- III Moodle
fication, hydrogen, and fuel cells. I None
Lesson 7: Integrated processes: oil refining. C II Moodle
Lesson 8: Polymers production. III Socrative

The seven lessons are grouped into three blocks (homogeneous in

It also showed their individual score at the end of the quiz. After
extension and difficulty):
each Socrative quiz, the lecturers showed the students the right
• Block I: Lessons 1, 2 and 4. answers and explanations about them. The students could ask any
• Block II: Lessons 5 and 6. question and doubts both in Socrative and Moodle activities.
• Block III: Lessons 7 and 8. Quizzes and tests were not intended for the learners to evalu-
ate the students because the lecturers decided the students’ scores
were not included in the final marks. These activities were done
Every year approximately 150 students are enrolled in the
exclusively to help the students to gain self-assessment, feedback
subject ‘Industrial Chemical Processes’, and are gathered in three
and self-regulation, identifying their strengths and weaknesses,
groups (cohorts) (A, B, C). Each one had a different instructor, but
targeting areas that need efforts and programming their future
all of them applied the same materials, contents and activities, took
work. Assessment of these goals has been done only by surveys
the same exam and obtained grades under equal evaluation rules.
and evaluation of final theory marks.
The teachers met many times and worked together to ensure these
The application of the three different ‘tools’ (one ‘Moodle’ test,
facts. The students were not differentiated by group and all of them
three ‘Socrative’ quizzes, ‘None’ quiz/test) was rotated between the
where considered similar in the study. Possible differences between
different cohorts (A, B, C) in successive blocks (Table 1). As all the
groups can be balanced by suitable study planning (Section 2.2).
cohorts received the three ‘tools’, this avoids possible differences
among groups. The students that did not run all the three Socrative
2.2. Tools
quizzes and the Moodle test were rejected. The purpose of this is to
have homogeneous number of data among ‘tools’ for the statistical
Different assessment tests have been designed depending on
analysis. Finally, the number of students who participated in the
each of the two technological tools used. The Moodle tests were
complete study was 49 (from a total of 142) and they were balanced
prepared to evaluate groups of lessons (blocks). Although the Moo-
(A: 16, B: 16, C: 17). The exam had 16 open-ended questions. The
dle tests at the end of a block of lessons is a typical example of
effect of tools application or not on the theory exam results (scores)
summative assessment activity, they were used only to formative
were carried out by taking the score mean for each block for each
purposes, as students could know about their block’s skills and
student.
could prepare better the final exam. They were intended for the stu-
As an example of activities carried out in this study, Table 2
dents to self-evaluate their knowledge and work and to consolidate
shows one of the Socrative quizzes, including the questions asked
the concepts after one of the theory blocks. Then the students can
and the multiple choice answers.
check their level of understanding and work to improve or maintain
it to face the final theory evaluation (exam). The tests were carried
out after the end of the block. A period of 1–2 weeks prior to the 2.3. Facility Index
test was given to the students for preparation. The test contained 10
multiple choice questions. Socrative is configured very well on any In order to be stable, similar distribution of easy, intermediate,
(portable or not) device and has a very friendly, direct, fast and easy and difficult questions between blocks is required for the study. To
to use interface. Socrative quizzes allow to assess the students in ensure this issue a Facility Index (FI) analysis was done. FI repre-
direct (real-time) quizzes during the class or just afterwards. It is a sents the ratio of students that answer a question right and can be
useful tool that gives the students a quick feedback on their under- associated to the overall difficulty of the question (Gamage et al.,
standing level and allows them to concentrate in weak issues after 2019). The FI can be calculated from an equation ([Link], 2013)
a class. This second technique has also the goal of encouraging the simplified as Eq. (1):
active participation of the students in the classroom and improving xi − xi (min)
motivation. Thus, Socrative has been used in this work as a way of Fi = 100 · (1)
xi (max) − xi (min)
formative learning, to carry out quick tests in the classroom with
the students’ own devices (smartphones, tablets and laptops). Up to where Fi is the Facility Index for question i, xi (min) is the minimum
3 tests have been carried out in traditional lectures within a block, score at question i, xi (max) is the maximum score at question i, and
each one containing 4–5 multiple choice questions. The specific xi is the score mean value at question i. xi (min)=0 because there
way of working has been to carry out the tests at the end of the is not negative marking. The higher the Facility Index, the easier
classes. The goal was to determine the assimilation degree of the the question is. The FI values are interpreted in different ranges
contents exposed during the same class or, at most, two consecutive ([Link], 2019) (Table 3).
classes. The quizzes were previously announced (at the beginning
of the class) to the students with the aim for them to pay more 2.4. Statistics
attention to the instructor. Moodle tests automatically gave stu-
dents the right answers to questions at the end of the test. Socrative ANOVA is widely used for statistical analysis to determine
showed students if an answer was right or not after each question. whether three or more population means are different. Some

57
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

Table 2
Socrative quiz for Cohort A, used in Block II, corresponding to Lesson 5. Right answers are underlined.

No. Question Answers

1 In the production of sulphuric acid, H2 SO4 , by the contact method the reaction a) high temperatures
carried out is: SO2 (g) + 12 O2 (g)  SO3 (g), H0 = -98.3 kJ/mol, exothermic b) high pressure
equilibrium. Which operating conditions are used to increase the SO2 c) low pressure
conversion? Mark the correct answer/s. d) low temperatures

2 In a modern sulphuric acid plant, the absorption of SO3 in concentrated a) true

sulphuric acid uses two absorption towers. The advantage of double b) false
absorption process is that, since the SO3 absorption is carried out at different
temperatures, the conversion of SO2 increases.
1
3 Why a catalyst is used in the reaction: SO2 (g) + 2
O2 (g)  SO3 (g), H = -98.3 a) to achieve high SO2 conversion
kJ/mol? Mark the most appropriate answer. b) to achieve high reaction rate, because the
reaction is carried out at low temperatures
c) to avoid the reverse reaction, i.e. the shift of
SO3 toward SO2

4 In a modern sulphuric acid plant, a reactor contains four adiabatic fixed beds a) true
with V2 O5 catalyst to carry out the reaction: SO2 (g) + 12 O2 (g)  SO3 (g), H = b) false
-98.3 kJ/mol. Four beds are used to cool the exhaust gases from one bed before
entering the next bed. It is an exothermic equilibrium and is favored at low
temperatures.

Table 3 teachers about the use of these techniques and methodologies has
Facility Index (FI) value ranges and its interpretation.
been carried out by surveys. It was intended to gather both opinions
FI (%) Interpretation to conclude whether these tools are easy to use, easy to prepare,
<5 Extremely difficult or something wrong with the whether they improve the motivation, engagement and other prac-
question tical issues. The students’ survey was open to all the students and
6–10 Very difficult was filled also by students who did partially the test and quizzes
11–20 Difficult schedule and that were rejected for the statistical study (Section
21–34 Moderately difficult
2.4). These were students that usually did the Moodle test but only
35–65 About right for the average student
66–80 Fairly easy did one of the Socrative quizzes. Opinions from these students were
81–89 Easy considered also valuable, because they were in touch with the tools.
90–94 Very easy Table 4 shows the questions. Total number of participants in stu-
95–100 Extremely easy
dents’ survey was n=89. The students’ answers followed a Likert’s
numerical scale from 1 (the lowest value) to 7 (the highest value),
except Questions 3 and 4 which consist on the selection of one
authors have included the statistical analysis of the results to
specific technique, Question 7, which was intended for instructors’
endorse their findings in new ICT’s applying (Zainuddin et al., 2020),
organization and Questions 9 and 14, which were open-ended. The
including ANOVA (Molin et al., 2020; González et al., 2019). Thus, it
surveys were done at the end of the semester. A survey was also
is used to compare the groups to see if there is significant difference
prepared for teachers to comment on the experience of using tests
among them. In this work ANOVA was used to have more insight
in Moodle and quizzes with Socrative (Table 5). Some questions
into the differences among the three tools treatment (‘Moodle’ =
involved both tools and other questions were specific for each one.
traditional teaching including Moodle tests, ‘Socrative’ = traditional
Instructors prepared and proposed the different questions for both
teaching including Socrative quizzes, and ‘None’ = traditional teach-
surveys attending common topics about participants’ impressions
ing with no ICT). That is, the real effect of incorporating Moodle test
and opinions.
or Socrative quizzes to the traditional theory teaching or not. The
scores mean by individual was calculated in each cohort and each
block. The compared parameter in the ANOVA test was the tool’s 3. Results and discussion
mean of marks in the final theory exam by block.
The statistical study of the data must consider that the appli- 3.1. Facility Index of questions
cation of the tools has been carried out on a rotating basis in
each cohort, in different blocks of lessons. Therefore, each student The FI was calculated for every exam question for the different
will have a score mean for every ‘tool’. This means having non- cohorts (Table 6) using students’ scores. The instructors also valued
independent data, that is, paired. Therefore, the test that should be the difficulty of each question and marked it as Easy, Intermediate
used to study whether there are statistically significant differences or Difficult. These values were converted to FI values for compar-
by tools application is a typical analysis of variance test (ANOVA) ing to the FI student values. Instructors’ assessments and students’
with paired data (95% confidence interval). To verify that the usual scores FI values mostly agreed on. However, at the last questions
conditions for the ANOVA test application were right, several tests some differences can be found (Questions 13 and 16). Mean val-
were carried out, such as the Shapiro-Wilk test (normality) and the ues for FI-equivalent instructors’ values and mean (considering the
Mauchly test (sphericity) and the graphical representation by using three cohorts) FI values from exam scores were also calculated by
histogram (normality) and boxplot (extreme data). Free software ‘R’ block. All these numbers are in the 35–65 range, which points out
(R Core Team, 2020) was used for all the tests. the Questions by block were ‘About right for the average student’
(Table 2). It means the average exam were neither difficult nor easy
2.5. Surveys for the students. This is important for checking stability of the sta-
tistical study, as similar difficulty in each block can be assumed from
Besides the influence of the ICT tools use over the academic instructors’ FI-equivalent values (block I appears as slightly easier
outcomes (scores), an analysis of the perception of students and than others). However, some differences between blocks appear

58
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

Table 4
Students’ survey.

Q. no. Question/statement

1 Completing the Moodle and Socrative quizzes have helped you create interest and motivate for the subject.
2 Completing the Moodle and Socrative quizzes have helped you prepare the theory of the subject.
3 Which tool do you consider easier to use?
4 Which tool has been most useful to you for gaining knowledge?
5 The Moodle test has allowed you to better prepare (a part of) the subject.
6 The Moodle test has allowed you to carry out a more continuous study of (a part of) the subject.
7 Do you consider the application of the Moodle test adequate for 15 minutes of the problem class?
8 In general, do you consider the use of tests in Moodle useful for teaching?
9 Propose improvements in Moodle test. Comments.
10 The Socrative quizzes have contributed to making the theory classes more enjoyable.
11 The Socrative quizzes have helped you pay more attention to the instructor’s explanations.
12 The Socrative quizzes have helped you fix your knowledge.
13 In general, do you consider the use of quizzes in Socrative useful for teaching?
14 Propose improvements in Socrative quizzes. Comments.

Table 5
Instructors’ survey.

Q. no. Question/statement Possible answers

1 Which tool requires more preparation time? Make an estimate.

2 Which tool influences the planning of the subject the most? Moodle (time)
3 Which tool requires more time in analyzing the results? Socrative (time)
4 Which tool is easier to apply? Both equal
5 What tool do you think (a priori, before analyzing the results) that most influences the results of
the students in the subject? Why?

6 Disadvantages of using the Moodle tests.

7 Disadvantages of using Socrative quizzes.
8 Advantages of using Moodle tests
9 Advantages of using Socrative quizzes.
10 Have you used any of these tools freely in the subject in this course? Explain why.

Table 6
Facility Index (FI) analysis on the theory exam questions. Instructors classified questions as easy (E), intermediate (I) or difficult (D). Equivalences to FI values were considered
as E: 66–100%, I: 33–65%, D: 0–32%. Students’ FI (%) were calculated from the exam scores.

Q. no. Instructors’ assessment Students’ FI by cohort Block Instructors equiv. FI avg. by block Students’ FI avg. by block

A B C

1 E 99 73 73
2 I 85 62 65
3 D 64 57 61
I 61 63
4 I 37 22 38
5 E 75 47 68
6 E 78 66 62

7 I 58 49 54
8 I 35 27 39
9 D 36 27 10 II 50 43
10 E 63 52 42
11 I 63 49 41

12 E 82 62 73
13 I 15 9 21
14 D 15 14 10 III 50 35
15 I 46 37 44
16 I 35 22 35

when studying students’ FI values. As block number increases the average result of these questions in each Block: ‘Moodle’, ‘Socra-
mean FI value decreases. This is not consistent with the instruc- tive’ and ‘None’. Keep in mind that the scores (marks) correspond
tors’ assessment. At the Block I both FI results were really close. In to parts of the exam (Blocks). For example, for the ‘Moodle’ tool,
the Block II mean FI value from students’ scores at the exam was the cohort A students’ scores correspond to the means of the Block
slightly lower. And at Block III the difference is higher. I questions in the exam, while the cohort B students’ scores come
from Block III and the cohort C students’ marks are from Block II.
This is not necessarily a problem, since the three cohorts rotate
3.2. Statistics in the ‘tools’ application. If there was any cohort influence, theo-
retically it would be equally distributed among the tools. The study
The scores of each student in the exam questions corresponding considered that any of the Blocks was an equivalent potential target
to the block in which the student had performed either the Moodle for the tools used. The set of data associated with each of the tools
test, the Socrative quizzes or did not do any test were averaged, (‘Moodle’, ‘Socrative’, ‘None’) were statistically evaluated (Table 7).
resulting in one statistical piece of information for each student It can be seen how there are no important differences between
in each Block. Therefore, there are 3 groups that can influence the

59
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

Table 7
Statistical parameters for each data group for paired analysis.

Tool Number of scores Mean 95% conf. interval Standard deviation Variance

Moodle 49 4.793 [4.07, 5.52] 2.52 6.33

Socrative 49 4.867 [4.32, 5.41] 1.89 3.60
None 49 5.201 [4.54, 5.86] 2.31 5.32

Table 8 Table 9
Shapiro-Wilk normality test results for each data group (‘tool’). Results of paired-data one-factor ANOVA test.

Tool W p-value Sum. sq. F Num. df Denom. df p-value

Moodle 0.94363 0.0205 4.6 0.4178 2 96 0.6597

Socrative 0.95809 0.6797
None 0.98265 0.0791
Table 10
Average values (means) in each tool by block. Cohorts (A, B, C) involved in each block
the means. Actually, the average marks of the applied tools (test and each tool are included (between brackets).
in Moodle or quizzes in Socrative) are slightly lower than those Block Moodle Socrative None
corresponding to traditional teaching (no tools application). This
I 7.48(A) 5.54(B) 7.26(C)
a priori means that there is no positive influence on the applica- II 4.02(C) 5.48(A) 4.70(B)
tion of the different tests with technological tools. It could even be III 2.93(B) 3.65(C) 3.52(A)
inferred that the influence of its application is negative. To check
if there are statistically significant differences between the means,
an analysis of variance (one-factor ANOVA) was carried out. test for sphericity delivered a p-value = 0.182, the ANOVA test use
For running a paired-data ANOVA test, the assumption about was approved and results from it must be taken as significant.
normality of data is common. Fig. 1 shows a histogram plot for data. The paired-data ANOVA test did not observe any statistically
Socrative and None data groups appear normal while Moodle can significant difference among the three groups Moodle, Socrative
deviate from normality. Shapiro-Wilk test results show a deviation and None (Table 9, p-value = 0.66). Therefore, it is considered that
from normality only for Moodle scores data (Table 8). This kind the use of Moodle or Socrative tools does not have neither positive
of test was used as recommended with sample sizes lower than nor negative effect on the students’ academic results (grades) on
50 because at larger sample sizes the Shapiro-Wilk test becomes the theory exam. Differences between the two tools must be also
very sensitive even to a minor deviation from normality. Although dismissed.
the ANOVA test is very robust even when there is a certain lack of The average values of the scores used in this study correspond
normality and this requirement becomes more relaxed the larger to different blocks of lessons and in each block to different groups
the population is, it is true that very extreme data can completely of students. Table 9 shows the values depending on the blocks.
invalidate the conclusions of an ANOVA. When the distributions are When considering the score means block by block, there is no pos-
far from normal, a non-parametric test such as the Kruskal-Wallis itive influence from the Moodle or Socrative application in both
H test can be used (Kvam and Vidakovic, 2007), which requires block I and block III. In block II, apparently Socrative could mean
homoscedasticity. To check if there are extreme data (outliers), the an improvement over traditional teaching (‘None’ group). In fact,
statistical data of the studied groups (‘Moodle’, ‘Socrative’, ‘None’) this grade corresponds to cohort A, which obtained slightly higher
were graphed in a boxplot (Fig. 2). Only one outlier was identified average grades than the other groups (A: 5.49, B: 4.39, C: 4.98). If all
but this can be assumed for testing. It is known that null-hypothesis the scores are considered (Table 9), the study is more representa-
significance tests (NHST) as ANOVA are widely used in this kind of tive and theoretically compensates for the effects between groups
studies where means are compared (Cumming et al., 2007). One- of students (cohorts), and, as already shown, there was no signifi-
way (one-factor) ANOVA can be highly robust even when usual both cant effect with the use of the tools. It is worth to mention that the
requirements, normality and homoscedasticity, are not fitted, espe- values decrease with the block number in the three groups. Facil-
cially normality (Sahai et al., 2011; Blanca et al., 2017; Laerd, 2018). ity Index calculated values (Table 6) showed a difference between
In fact, one-way ANOVA is not very sensitive to deviations from instructors-based data and students’ scores in Blocks II and III. This
normality: simulations with a variety of non-normal distributions gap is more evident in the last Block. All these facts probably point
(including flat, highly peaked, highly skewed, and bimodal) found out that the students studied the successive lessons from more to
the proportion of false positives always around 5% or a little lower, less. In other words, it seems that the students did not have much
just as it should be (McDonald, 2014). This author recommends time to study the complete theory part of the subject and they had
staying on ANOVA unless the lack of normality is very extreme, a adequate time only to study the first lessons. For the last lessons
fact that does not happen in this study. ANOVA test is also robust (block II and specially block III) they probably did not have the
when populations are high, even the equal variance assumption necessary time to fix the knowledge well. This may represent an
is violated (Coombs et al., 1996) or the normality deviated (Cody, unexpected error in the study. There is also the fact that the aver-
2011). In this study n=49 for each group (cohort) so this can be age marks of all the students are slightly different depending on the
considered enough high. group (Table 10). The theory exam questions were not of multiple-
It is worth to mention that variances in this work are not choice kind but short answer type. In theory, the effect of taking
equal, especially for the Socrative tool (Table 7). In cases where scores from different groups is offset by considering all groups and
homoscedasticity is not met, the ANOVA test can be used with all blocks. However, if we add differences between groups to the
Welch correction (Welch, 1951). However, there is also other favor- effect of less time devoted to study in block II and specially in block
able fact to fairly use of ANOVA: the equal number of populations III, this may also incorporate some margin of error to the data and
among cohorts which compensates some differences in variance the statistical study. Nevertheless, mean scores in Block I for ‘Moo-
(Zimmerman, 2004). Also there exists a second condition, and dle’ (cohort A) and ‘None’ (cohort C) were very similar and score for
probably the most important, that must be fulfilled when running ‘Socrative’ (cohort B) was lower, which shows no influence of tools
one-factor paired-data ANOVA test: sphericity. As the Mauchly’s also in Block I.

60
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

Fig. 1. Data histogram: score distribution on the different tools.

3.3. Students’ survey Table 11

Students’ survey results. Scores follow a Likert scale between 1 (minimum) and 7
(maximum).
Students’ survey results are shown in Table 11. Students
assessed positively Moodle tests and Socrative quizzes to create Cohort A B C Average score
interest and motivation for the subject (Question 1 mark: 4.3/7) 05/10/2019 05/28/2019 05/16/2019
Date
and also in the preparation of the subject’s theory materials (Ques- 05/23/2019
tion 2 mark: 4.2/7). The comparison of two techniques has been No. surveys 30 19 40 89 (total)
briefly carried out in the survey. Students have preferred Moodle Attendees (%) 100 100 100
tests than Socrative quizzes because of its ease of use and they have Enrolled (%) 60 46.3 63.5
the perception of being more useful in the knowledge acquisition Question 1 4.7 4.8 3.8 4.3
(69.7%). The positive assessment to Moodle tests was reflected in Question 2 4.63 4.7 3.7 4.2
Questions 5 (4.2/7), 6 (4.0/7) and 8 (5.0/7). Question 8 exhibited the Question 5 4.43 4.9 3.7 4.2
Question 6 4.0 4.8 3.6 4.0
highest score in the whole survey. Some comments in Question 7
Question 8 4.97 5.7 4.6 5.0
(Moodle tests) were as follows: ‘It forces you to study the theory’; ‘It Question 10 5.48 4.6 4.6 4.9
helps you to fix the knowledge’; ‘It encourages attention in classes’; Question 11 5.29 4.6 4.4 4.7
‘The score obtained in the test should be considered in the subject’s Question 12 4.64 4.5 3.8 4.2
final mark’. Question 13 4.96 4.6 4.5 4.7

The use of Socrative quizzes have the aim of performing fast Question 3 60% 73.7% 60.5% 63.1%
tests in classroom to make the traditional lectures more enjoyable (Moodle)
Question 4 63% 89.5% 64.8% 69.7%
(Question 10 mark: 4.9/7) and also to obtain the students for pay-
(Moodle)
ing more attention to the teacher’s explanations (Question 11 mark:

61
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

Fig. 2. Boxplot with statistic values for the three ‘tools’ in paired-data.

4.7/7). This can be apparently considered as not consistent to a more studies (Munusamy et al., 2019; Guarascio et al., 2017) have con-
positive assessment about Moodle tests (Question 4). The students cluded that Socrative application promotes engagement of students
consider Socrative quizzes funny, attractive and useful for improv- in the classroom and enrich their learning experience. The prefer-
ing the attention to the instructor. They valued them as useful for ence for Socrative compared to the traditional student response
teaching (Question 13 mark: 4.7/7). However, the same question system (SRS) are based on the participation in class, the facility of
scored higher for Moodle (Question 8 mark: 5.0/7) and most of answering to classroom questions, easy to use and format enjoy-
the students prefer Moodle to Socrative as more useful in knowl- ment. Socrative enhances engagement probably due to anonymity
edge acquisition (Question 4: 69.7%). The most relevant aspect and favors students to be actively involved in class (McKenzie
for students about Moodle tests was that questions and answers and Ziemann, 2020). Mixing traditional and digital (ICT) teach-
(solutions) were accessible during the whole semester. These stu- ing/learning, as in blended learning approach, has shown positive
dents’ comments can explain the favorable results toward Moodle effects on students’ outcomes (Means et al., 2010), also in higher
because some of the students do not retained or missed the results education (Smith and Kurthen, 2007). Socrative also has been
of Socrative tests in order to study the subject when facing the reported as improving the students grades (González et al., 2019;
exam. The teachers kept the results (all the questions and the cor- Remon et al., 2017).
rect answers) of the Moodle tests accessible to the students through Students’ survey showed a positive perception about the utility
the Moodle system during the semester. However, the results of the that the Moodle tests (especially) and Socrative quizzes could have
Socrative tests were shown at the end of the face-to-face classes, but in the acquisition of knowledge in the subject. However, it was not
were not available for further study. In fact, several of the students consistent with the actual exam results (grades) (Section 3.2). Lack
asked the teachers to have the Socrative test questions and the cor- of positive effects of e-learning on grades is not new. In the appli-
rect answers to study before the exam. This fact was solved as soon cation of the Flipped Classroom teaching strategy, for example, it
as the teachers knew it, but probably later to be relevant. Therefore, has been observed that, although students significantly improved
it is possible that the students’ opinions, less favorable toward the the non-technical skills and abilities (capacity for action and inde-
use of Socrative, were related with the information available about pendence, assumption of responsibility, social skills, self-reflection,
the results of the respective tests. This is a proof of their concern class engagement), it did not lead to an improvement in the stu-
about the lack of this study material and its possible influence on dents’ final grades (technical skills) (Cieliebak and Frei, 2016).
the opinions of at least some of the students on the use of Socrative. However, as cited above, Socrative and blended learning has been
A comment repeated several times in the students’ surveys, both proven as positive tools for increasing students’ outcomes. In this
for Moodle and Socrative tests/quizzes, is about the students’ desire work, there could be particular causes explaining why quizzes/tests
for including the quizzes marks into their final subject’s score. do not show any effect on students’ grades. Maybe, for some stu-
Also, some students would like to carry out Moodle tests more dents, the moment when the tools were used became too far from
frequently, for example one at the end of every lesson. the exam to remember them. For other students, maybe the date
General findings in this work from students’ surveys about for using the tools was too close to the exam to properly revise
Socrative’ positive effect and opinions agree with literature. Some the related contents. In fact, FI and statistical data showed a lack of

62
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

study for the last lessons (Section 3.2). Also, no influence of quizzes uation apparently contrasts with the statistical data comparison
and test on grades could be related to a low profile implementa- results. This work is possibly yet another example of the fact that
tion. Because of ‘tools’ rotating among cohorts, only three Socrative students’ previous perceptions about usefulness and ease of use
quizzes and one Moodle test were carried out, and they were run condition their final perceptions (at the end of the semester) of
separately. Probably a higher number of quizzes/tests should be real aid for learning and aid for participation and motivation. The
necessary for perceiving a positive effect on the final marks. This is students’ opinion about their future academic outcomes is influ-
supported by some comments in students’ survey (see above). In enced, therefore, by their initial impression about these tools such
the work from Remon et al. (2017) the Socrative quizzes were joint as Moodle tests and Socrative quizzes.
to a specific interactive e-leaning teaching method using a tablet. Students have positively valued the two tools used, individu-
A possible synergistic effect could increase the positive results of ally, with Moodle being slightly better than Socrative. When both
e-quizzes. Also, the number of interactive (Socrative) questions (62 are compared, Moodle was clearly rated higher. It is possible that
questions for two academic years) was much higher in that work the opinion, less favorable toward the use of Socrative, was proba-
from Remon et al. (2017) than in the present study (12–13 questions bly related to the lack of information (questions and test solutions)
in one semester). available before the exam, which was not the case with Moodle.
Anyway, although the two tools did not show a significant effect
on the students’ final marks, they were favorably evaluated by both
3.4. Instructors’ survey
students and teachers in terms of their interest in applying them as
support for the traditional theory lectures. The students reported
Considering Question 1 (Table 5), two teachers (67%) answered
an increase in the motivation and interest toward the subject with
that the preparation of Moodle tests required more time than
the application of the two tools. They pointed out as a positive effect
Socrative quizzes, while one teacher (33%) answered that the two
in using Moodle that questions and answers were accessible dur-
techniques needed a similar preparation time. The answers of Ques-
ing the whole semester for studying. Students considered Moodle
tion 2 show a wide range. One teacher pointed out that the two
easier for using and they have the perception of being more use-
tools influenced similarly in the subject’s planning, other teacher
ful in the knowledge acquisition. Probably it would be necessary
answered that Socrative quizzes influenced more in the planning
to study in the future the incorporation of more Socrative tests
and the third teacher did not answer. The time required to ana-
to each block of lessons, because the three tests included in this
lyze the results (Question 3) was equal for the two techniques. The
study do not covered the total number of lessons in each block.
answer was the same for the three instructors. They also agreed
It would be a way to compensate this difference with the Moodle
on Question 4: Socrative tests were easier to apply than Moodle
tool application. Thus, a future strategy could be to run a test with
tests. Considering Question 5, two instructors (67%) answered that
Socrative every two hours of traditional teaching class for any block.
Moodle tests could influence more in academic marks, because it
This could have a possible positive effect both on the students’ per-
helps to the study of the subject. The third teacher answered that
ception of the Socrative quizzes usefulness and on the grades. The
the two techniques should influence equally. This last instructor
students also supported to do more tests with Moodle, probably at
concluded that both tools were complementary and supported a
the end of each lesson, reinforcing the positive valuation for Moo-
simultaneous use. About the inconveniences of Moodle tests (Ques-
dle tool. Instructors’ survey pointed out Moodle tests were more
tion 6) all the teachers coincided that this was a more difficult tool
difficult to implementation and required more time to deal with,
to use and less attractive than Socrative tests. It is worth mention-
but both tools were valuated as interesting for increasing students’
ing that Moodle tests required more training to be implemented.
motivation, engagement, and self-regulation.
Also, the higher preparation time for Moodle tests was pointed out.
The inconveniences of Socrative quizzes (Question 7) were the dif-
ficulty to apply for long teaching periods (more than two hours Declaration of Competing Interest
of class) and also some students’ difficulty for right answers iden-
tification when running the test. The advantages of Moodle tests The authors report no declarations of interest.
(Question 8) were: ‘It is already a known Learning Management
System for students and teachers’; ‘It helps for a continued study of Acknowledgements
the subject’; ‘It is easy for correction’. The advantages of Socrative
quizzes (Question 9) were: ‘It makes the classes enjoyable and stu- The authors wish to thank ‘Vicerrectorado de Política
dents pay more attention’; ‘It is fast for preparation and carrying Académica’ of the University of Zaragoza, for financial sup-
out and a fast feedback about students’ understanding is achieved’; port through the teaching innovation projects PIIDUZ 18 327 and
‘It is easy for correction’. PIIDUZ 19 404.

4. Conclusions References

Study of data (score mean for each student at each block, Allen, E.I., Seaman, J., 2010. Learning on demand. Online education in the United
States, 2009. Babson Survey Research Group [Link]
depending on tool) showed no statistical significative influence of Alvarez, A., Martin, M., Fernandez-Castro, I., Urretavizcaya, M., 2013. Blended tra-
Moodle tests or Socrative quizzes on the theory exam grades. Data ditional teaching methods with learning environments: experience, cyclical
suggested the students worked more for the first lessons than for evaluation process and impact with MAgAdI. Comput. Educ. 68, 129–140.
Artal-Sevil, J.S., Romero, E., Artacho, J.M., 2017. Quick surveys in classroom, Mobile
the last ones while facing the theory exam. This fact and the pos-
phone, a power teaching tool. Proceedings of the International Technology, Edu-
sible differences among student groups could negatively affect the cation, and Development INTED2017 Conference, 9282–9291.
data and the statistical study. Nevertheless, this possible error was Azevedo, R., Cromley, J.G., Fielding, I., Winters, F.I., Moos, D.C., Green, J.A., 2005.
Adaptive human scaffolding facilitates adolescents’ self-regulated learning with
probably not determinant, as mean scores for the tools did not show
hypermedia. Instr. Sci. 33, 381–412.
positive differences in Block I. Students’ survey revealed that Moo- Azevedo, R., Cromley, J.G., Seibert, D., 2004. Does adaptive scaffolding facilitate
dle tests and Socrative quizzes helped them for the preparation of students’ ability to regulate their learning with hypermedia? Contemp. Educ.
the subject’s theory materials. This fact implies a positive effect of Psychol. 29, 344–370.
Ben-Av, M.B., Ben-Av, R., 2015. Is the classical lecture structure dead? Proceed-
those tools in self-regulation. Students valued the feed-back that ings of the International Technology, Education, and Development INTED2015
tools add to their learning process. However, those students’ val- Conference, 1614.

63
E. Romero et al. Education for Chemical Engineers 36 (2021) 54–64

Blanca, M.J., Alarcón, R., Arnau, J., Bono, R., Bendayan, R., 2017. Non-normal data: is Munusamy, S., Osman, A., Riaz, S., Ali, S., Mraiche, F., 2019. The use of Socrative and
ANOVA still a valid option? Psicothema 29, 552–557. Yammer online tools to promote interactive learning in pharmacy education.
Chou, P.N., Chang, C.C., Lin, C.H., 2017. BYOD or not: a comparison of two assessment Curr. Pharm. Teach. Learn. 11, 76–80.
strategies for student learning. Comput. Hum. Behav. 74, 63–71. Namyssova, G., Tussupbekova, G., Helmer, J., Malone, K., Afzal, M., Jonbekova, D.,
Cieliebak, M., Frei, A.K., 2016. Influence of flipped classroom on technical skills 2019. Challenges and benefits of blended learning in higher education. Int. J.
and non-technical competences of IT students. 2016 IEEE Global Engineering Technol. Educ. 2, 22–31.
Education Conference (EDUCON2016), 1012–1016. Norberg, A., Dziuban, C.D., Moskal, P.D., 2011. A time-based blended learning model.
Cody, R., 2011. SAS® Statistics by Example. SAS Institute Inc., Cary, NC. On the Horizon 19, 207–216.
Cohen, D., Sasson, I., 2016. Online quizzes in a virtual learning environment as a tool Novo-Corti, I., Varela-Candamio, L., Ramil-Diaz, M., 2013. E-learning and face to face
for formative assessment. J. Technol. Sci. Educ. 6, 188–208. mixed methodology: evaluating effectiveness of e-learning and perceived sat-
Coombs, W.T., Algina, J., Oltman, D.O., 1996. Univariate and multivariate omnibus isfaction for a microeconomic course using the Moodle platform. Comput. Hum.
hypothesis tests selected to control type I error rates when population variances Behav. 29, 410–415.
are not necessarily equal. Rev. Educ. Res. 66, 137–179. Nunez, J.C., Cerezo, R., Bernardo, A., Rosario, P., Valle, A., Fernandez, E., Suarez, N.,
Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Lo, J., 2011. Implementation of training programs in self-regulated learning strategies
McMenamin, N., Wilson, S., 2007. Statistical reform in psychology. Is anything in Moodle format: results of an experience in higher education. Psicothema 23,
changing? Psychol. Sci. 18, 230–232. 274–281.
Drysdale, J.S., Graham, C.R., Halverson, L.R., Spring, K.J., 2013. An analysis of research Oliver, M., Trigwell, K., 2005. Can ‘blended learning’ be redeemed? E-learning 2,
trends in dissertations and theses studying blended learning. Internet High. 17–26.
Educ. 17, 90–100. Pentalog, 2020. Success Story. Socrative. Development of an E-learning Plat-
Ferrandiz, E., Puentes, C., Moreno, P.J., 2016. Engaging and assessing students form (Accessed 16 August 2020) [Link]
through their electronic devices and real time quizzes. Multidisc. J. Educ. Soc. website-developement-e-learning-platform.
Technol. Sci. 3, 173–184. Perry, N.E., Hutchinson, L., Thauberger, C., 2008. Talking about teaching self-
Frias, M.V., Arce, C., Flores-Morales, P., 2016. Uso de la plataforma [Link] regulated learning: scaffolding student teachers’ development and use of
para alumnos de química general (Using [Link] platform for students in practices that promote self-regulated learning. Int. J. Educ. Res. 47, 97–108.
general chemistry). Educ. Quím. 27, 59–66. Pituch, K.A., Lee, Y.K., 2006. The influence of system characteristics on e-learning
Gamage, S.H.P.E., Ayres, J.R., Behrend, M.B., Smith, E.J., 2019. Optimising Moodle use. Comput. Educ. 47, 222–244.
quizzes for online assessments. Int. J. STEM Educ. 6, art. no. 27. Pryke, S., 2020. The use of Socrative in university social science teaching. Learn.
Glazer, J.S., 2012. Blended learning: across the disciplines, across the academy. Sty- Teach. 13, 67–86.
lus, Sterling, VA. Quintana, C., Zhang, M., Krajcik, J., 2005. A framework for supporting metacognitive
González, A.G., Salgado, D.R., García Sanz-Calcedo, J., Cruz García, C., Barrios Muriel, aspects of online inquiry through software-based scaffolding. Educ. Psychol. 40,
J., López Pérez, O., Álvarez García, F.J., 2019. A teaching methodology for the real- 235–244.
time assessment of students’ competencies related to manufacturing subjects R Core Team, 2020. R: A Language and Environment for Statistical Computing. R
using technology based on electronic devices. Proc. Manuf. 41, 579–586. Foundation for Statistical Computing, Vienna, Austria [Link]
Graesser, A.C., McNamara, D.S., VanLehn, K., 2005. Scaffolding deep comprehen- org/.
sion strategies through Point&Query, AutoTutor and iSTART. Educ. Psychol. 40, Remon, J., Sebastian, V., Romero, E., Arauzo, J., 2017. Effect of using smartphones
225–234. as clickers and tablets as digital whiteboards on students’ engagement and
Guarascio, A.J., Nemececk, B.D., Zimmerman, D.E., 2017. Evaluation of students’ learning. Act. Learn. High. Educ. 18, 173–187.
perceptions of the Socrative application versus a traditional student response Roca, J.C., Chiu, C.M., Martinez, F.J., 2006. Understanding e-learning continuance
system and its impact on classroom engagement. Curr. Pharm. Teach. Learn. 9, intention: an extension of the technology acceptance model. Int. J. Hum. Comput.
808–812. Stud. 64, 683–696.
Guerrero, C., Jaume, A., Juiz, C., Lera, I., 2016. Use of mobile devices in the classroom Romero, C., Espejo, P.G., Zafra, A., Romero, J.R., Ventura, S., 2013. Web usage mining
to increase motivation and participation of engineering university students. IEEE for predicting final marks of students that use Moodle courses. Comput. Appl.
Latin Am. Trans. 14, 411–416. Eng. Educ. 21, 135–146.
Halverson, L.R., Graham, C.R., Spring, K.J., Drysdale, J.S., Henrie, C.R., 2014. A the- Romero, E., Artal-Sevil, J.S., Artacho, J.M., 2019. Traditional lecture in higher
matic analysis of the most highly cited scholarship in the first decade of blended education: how to enhance engagement through flipped learning and
learning research. Internet High. Educ. 20, 20–34. ICT. 13th International Technology, Education and Development Conference
Islam, A.K.M.N., 2012. The role of perceived system quality as the educators’ moti- (INTED2019), 4188–4197.
vation to continue e-learning system use. AIS Trans. Hum. Comput. Interact. 4, Sahai, H., Ageel, M.I., Khurshid, A., 2011. Analysis of variance model, effects of
25–44. departures from assumptions underlying. In: Lovric, M. (Ed.), International Ency-
Islam, A.K.M.N., 2013. Investigating e-learning system usage outcomes in the uni- clopedia of Statistical Science. Springer, Berlin, Heidelberg.
versity context. Comput. Educ. 69, 387–399. Schunk, D.H., Zimmerman, B.J., 2008. Motivation and self-regulated learning: theory,
Kaleta, R., Skibba, K., Joosten, T., 2007. Discovering, designing, and delivering hybrid research and applications. Lawrence Erlbaum Associates Publishers, Mahwah,
courses. In: Picciano, A.G., Dziuban, C.D. (Eds.), Blended learning: research per- NJ.
spectives. Sloan Consortium, Needham, MA, pp. 109–143. Smith, G.G., Kurthen, H., 2007. Front-stage and back-stage in hybrid e-learning face-
Kennedy, G.E., Cutts, Q.I., 2005. The association between students’ use of an elec- to-face courses. Int. J. e-Learn. 6, 455–474.
tronic voting system and their learning outcomes. J. Comput. Assist. Learn. 21, Smith, K., Hill, J., 2019. Defining the nature of blended learning through its depiction
260–268. in current research. High. Educ. Res. Dev. 38, 383–397.
Kvam, P.H., Vidakovic, B., 2007. Nonparametric statistics with applications to science Spanjers, I.A., Könings, K.D., Leppink, J., Verstegen, D.M., de Jong, N., Czabanowska,
and engineering. Wiley, Hoboken, NJ. K., van Merriënboer, J.G., 2015. The promised land of blended learning: quizzes
Laerd, 2018. One-Way ANOVA (Accessed 15 January 2021) [Link] as a moderator. Educ. Res. Rev. 15, 59–74.
com/statistical-guides/[Link]. Stowell, J.R., Nelson, J.M., 2007. Benefits of electronic audience response systems on
Larsen, T.J., Sorebo, A.M., Sorebo, O., 2009. The role of task-technology fit as users’ student participation, learning, and emotion. Teach. Psychol. 34, 253–258.
motivation to continue information system use. Comput. Hum. Behav. 25, Sumak, B., Hericko, M., Pusnik, M., 2011. A meta-analysis of e-learning technology
778–784. acceptance: the role of user types and e-learning technology types. Comput.
Lim, W.N., 2017. Improving student-engagement in Higher Education through Hum. Behav. 27, 2067–2077.
mobile-based interactive teaching using Socrative. Proc. of the 2017 IEEE Global Sumak, B., Polancic, G., Hericko, M., 2010. An empirical study of virtual learning
Engineering Education Conference (EDUCON2017), 404–412. environment adoption using UTAUT. Proceedings of the Second international
McDonald, J.H., 2014. Handbook of Biological Statistics, 3rd edition. Sparky House Conference on Mobile, Hybrid, and On-line Learning (ELML 2010), 17–22.
Publishing, Baltimore, MD. Teo, T., 2009. Modeling technology acceptance in education: a study of pre-service
McKenzie, M., Ziemann, M., 2020. Assessment of the web-based audience response teachers. Comput. Educ. 52, 302–312.
system Socrative for biomedical science revision classes. Int. J. Educ. Res. Open Van Laer, S., Elen, J., 2020. Adults’ self-regulatory behaviour profiles in blended learn-
1, 100008. ing environments and their implications for design. Technol. Knowl. Learn. 25,
Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K., 2010. Evaluation of evidence- 509–539.
based practices in online learning: a meta-analysis and review of online learning Welch, B.L., 1951. On the comparison of several mean values: an alternative
studies. U.S. Department of Education, Washington, DC [Link] approach. Biometrika 38, 330–336.
rschstat/eval/tech/evidence-based-practices/fi[Link]. Yang, Y.T.C., Chuang, Y.C., Li, L.Y., Tseng, S.S., 2013. A blended learning environment
Mirriahi, N., Alonzo, D., Fox, B., 2015. A blended learning framework for curriculum for individualized English listening and speaking integrating critical thinking.
design and professional development. Res. Learn. Technol. 23, 28451. Comput. Educ. 63, 285–305.
Molin, F., Haelermans, C., Cabus, S., Groot, W., 2020. The effect of feedback on Zainuddin, Z., Shujarat, M., Haruna, H., Chu, S.K.W., 2020. The role of gamified
metacognition – a randomized experiment using polling technology. Comput. e-quizzes on student learning and engagement: an interactive gamification
Educ. 152, 103885. solution for a formative assessment system. Comput. Educ. 145, 103729.
[Link], 2013. Quiz Statistics Calculations (Accessed 25 May 2020) [Link] Zimmerman, D.W., 2004. A note on preliminary tests of equality of variances. Br. J.
[Link]/dev/Quiz statistics calculations#Introduction. Math. Stat. Psychol. 57, 173–181.
[Link], 2019. Quiz Report Statistics (Accessed 25 May 2020) [Link]
[Link]/dev/Quiz report statistics.

64