Assignment Coversheet
Please complete and paste this coversheet to the front of your assignment. This
assignment is anonymous. Do not use your name.
Student candidate number:
Module Title: Design and Use of Technologies for Education (DUTE)
Assignment Title: Critique of Khan Academy’s technology design with respect to
its educational value
Word Count: 2,732
(Excluding abstract, footnotes, bibliography/references list, appendices, tables,
figures and title)
Thinking about the feedback on your draft assignment, please indicate what the key
points were:
For each point, state what action you took to respond to this feedback in preparing
the final version of your assignment:
Student Declaration:
By ticking this box, I confirm that all the work is my own unless collaboration has
been specifically authorised. I understand that any form of Academic Misconduct is
strictly prohibited, including the use of essay mills, homework help sites, plagiarism,
collusion, falsification, impersonation or any other action which might give me an
unfair advantage. ü
I am aware that my assignment may be used in anonymised form to help future
students.
Opt-out: do not allow material to be used in teaching future students ☐
1
At the top of Khan Academy's home page appears a strong statement about its product's value
and impact on driving learning outcomes: "For every student, every classroom. Real results".
In this essay, I aim to explore what the webpage's carefully chosen words mean by applying a
critical platform gaze. First, I will analyze KA's educational value associated with personalized
learning to grapple with the learning theory embedded in the user experience. Then, I wish to
explore the ways KA aspires to enhance learning/teaching through the provision of learning
analytics. Ultimately, I will extend the discussion to the evidence of its proclaimed impact. For
length constraints, my analysis will be mostly centered on the teacher-user experience. I will
base my discussion on information that anyone can access by using a free teacher account and
use relevant literature to critique it.
Unpacking KA's educational value
A buzz term usually used by edtech companies when promoting the benefits of their products
is personalized learning (PL). Despite the lacking consensus about the definition of that term
(UNESCO, 2022), it is commonly characterized as tackling three major problems in education:
the one-size-fits-all model, the fixed temporal divisions of learning and teachers' heavy
workloads (Pelletier, 2023). Under the section 'Why Khan Academy works', a description of
PL is outlined: "Students practice at their own pace, first filling in gaps in their understanding
and then accelerating their learning". This sentence alludes to mastery learning, the notion that
most students can attain a high level of proficiency when instruction is systematically
approached. It involves providing assistance to students when they encounter learning
difficulties and allowing them ample time to achieve mastery (Bloom, 1973). The KA course
pages display all covered skills with each represented by a square that is color-coded based on
the student's performance and mastery levels. By dividing lessons into bite-sized content for
instruction and assessment, KA employs what Watters (2021) refers to as the pedagogy of
teaching machines. Such a premium on the individualization of education requires a strong
needs-analysis orientation and consistent progress monitoring.
The analysis of PL on KA reveals a balance between limitations and opportunities. On a
positive note, teachers can craft class rosters, send assignments targeting specific skills, to the
entire group or individuals, based on their needs, and monitor progress through dashboards. If
combined in a feedback loop, the assignment tool and the progress reports can be powerful
2
instruments of personalization providing teachers with insights to task students accordingly.
However, the platform is not geared towards determining students' initial level of proficiency.
Student-account users are only asked to select their school year and the courses they would like
to study. The absence of prompts to set goals, a baseline assessment or further investigation
into what it is that they desire to become better at substantiates the claim that the market
imposes an idea of people having an intrinsic motivation to learn and being able to do so
autonomously (Bayne, 2020). Moreover, limited data decreases teachers' agency and opens
avenues to examine who determines what learners need and the extent to which the needs are
indeed personal.
It is paramount to remain critical of what is presented as PL by edtech companies. Holmes et
al (2018) argue that one way to grapple with the complexity of the term is by considering its
multiple dimensions with respect to why something is to be learned, how/what/when it is to be
learned, who is involved and where it takes place. From this perspective, KA offers few
possibilities mainly because the content and the learning outcomes stem from state-defined
curriculum, thus leaning towards one-size-fits all education. Moreover, the learning pathway
is curtailed by three types of learning objects (articles, videos and exercises) as if they were
capable of teaching all skills in the course catalog to any student. Even though KA pitches PL
as a proxy for their method efficacy, there is limited information on the website about their
interpretation of PL.
With regards to the methodology, KA displays overt elements of behaviorism in the learning
design. All practice exercises follow the same logic: the problem is presented on the screen
(stimulus), then the student works it out and gives an answer (response) and receives feedback
(reinforcement). If the answer is right, confetti bursts onto the screen and a positive message
pops up. If wrong, the student is encouraged to review a related article or video, use a hint and
try again. In 2011, the founder of Khan Academy, Salman Khan, presented the platform at a
TEDTalk entitled "Let's use video to reinvent education". His speech caused shouts of
acclamation, but rather than novelty, much of contemporary educational technologies have
roots in mid-20th-century artifacts (Watters, 2021). Sal's discourse around edtechs was a
reverberation of what Skinner (1958) had proposed. The author argued about the need to make
education more efficient in light of the effects of scalability on making learners into passive
receivers of instruction. Skinner (1958) also predicted that videos would be used to replace
traditional lectures and talked about teaching machines as enablers of PL for presenting level-
3
appropriate content to the student as a good tutor would do. For packaging most of Skinner's
ideas, there are limitations when applying mastery learning to more sophisticated skills through
a behaviorist approach.
In the education spectrum, the knowledge-transmission and the social constructivist models lie
on opposite sides. Freire (1972) posits that on one end there is the banking education treating
students as passive depositories of information, and on the other, stands problem-posing
education aiming to develop critical thinkers able to work in collaboration. All practice
exercises on KA are based on the drill skill model commonly used in behaviorist instructional
technologies with systematic practice of concepts to perfect a skill or procedure (Dede, 2008).
In order for the platform to compute the student performance and provide immediate feedback,
all activities require a particular and unanimous correct answer. This modus operandi
emphasizes individualization over social learning and is unable to provide opportunities to
develop higher-order thinking skills such as justifying a stand or decision and using the
knowledge and skills to produce original work. I do not view KA as a system to train students
on simply recalling facts and basic concepts, in fact, a number of activities touch upon higher
levels of the cognitive domain and involve testing and drawing connections among ideas.
However, recognizing the limitations of the platform is important to understand the role played
by the teachers in complementing the learning experience with other materials, activities and
pedagogies. To explore how KA supports the teachers in making pedagogical decisions, I now
turn to critique its elements of learning analytics.
The teacher dashboards
Learning analytics (LA) has been widely promoted as a means to gain insights on the learning
process. The application of LA is closely connected to tailoring educational experiences,
informing student self-direction and supporting planning and orchestration (Wise, 2019).
While modern technologies can quickly process large amounts of data, analyzing learner
information has long been part of teachers' routines. Skinner (1958) mentioned that the analysis
of paper strips with written answers produced by his teaching machines could inform content
refinement. In line with such a trend in edtech, KA takes pride in providing dashboards that
enable teachers and parents to "See at a glance whether a child or student is struggling or if she
hit a streak and is now far ahead of the class". The organization's discourse about LA is
4
entangled with its proclaimed educational value of personalization, and the data collected
shapes their perception of learning success.
KA offers three types of reports based on log data providing a summary of class performance
as a whole or detailed student profiles. The 'Scores' report displays students' performance on
the content assigned to them with color-coded boxes showing the percentage of correct
answers. It is possible to visualize the questions answered by students, the first response they
gave on each question and the number of attempts. The second report focuses on achievement
of mastery goals. The third report, activity overview, is the most robust and comprehends all
students' activities on KA. The information on this report is varied, comprising total learning
minutes and details of all skills the students did practice on including a breakdown of the
mastery level attained in each skill. The data a teacher can get through these dashboards has
the potential to inform the planning of lessons and consequently improve the teaching and the
learning as suggested by Wise (2019). More than simply providing information of the outcomes
(final grades), it is possible to map out individual and whole-class struggling areas by checking
most missed questions, have insights into student persistence through the number of attempts
and use of hints and identify distracted students who are not on-task by monitoring the learning
time. Although it is also important to measure the quality of learning products (Gasevic,
Dawson and Siemens, 2015), KA reports do not shed light on the strategy adopted by the
students or if they tried different strategies in each attempt. Insights into the student way of
reasoning about the practice problems could significantly help the teacher spot where and how
to best help the learners.
Embedding other modalities of LA into the KA experience might propel more accurate
pedagogical interventions. While logged data is insufficient to gain deeper insights into what
is influencing learning (Cukurova, Giannakos and Martinez-Maldonado, 2020), multimodal
data opens possibilities to evaluate complex cognitive abilities through the assessment of
student knowledge, affect, intentions and beliefs (Blikstein and Worsley, 2016). In my opinion,
combining gesture and affective state analyses could aggregate value to the scores report. Many
students have a negative attitude towards learning Math, but since education depends on
learners being willing to take risks (Biesta, 2005), technology could help teachers make
learning more pleasant and effective by understanding which students need emotional support
to develop a can-do attitude as well as spot learners who are prone to giving up when under
high levels of stress.
5
There is also room to improve the teacher dashboards when it comes to supporting timely
interventions. Orchestration involves monitoring multi-layered activities and deciding on
necessary adaptations in real time (Dillenbourg, 2013), but at the top of the activity overview
page, there is a disclaimer saying that KA may take 10 minutes to update student activity.
Considering that a regular class in Brazil lasts 50 minutes, and some of that is consumed by
students logging into the platform and doing the activities, the teacher agency gets delayed.
Moreover, considering that groups are large, the visualization of the reports and interpretation
of the data should be facilitated. While the score report is very straightforward in categorizing
the performances by different colors, the display of mastery levels is confusing and requires
many clicks to get granular insights. Although there is limited evidence as to how MMLA
supports actionable feedback (DiMitri et al., 2018), KA could deploy and test features that
provide more actionable insights and recommendations to teachers (UNESCO, 2022), such as
clustering students and giving assignment options. The teacher would still be the authority
making the final decisions as to how learners are assisted but improving the usability of the
skills report might bring human and machine interpretations closer.
In this section I have explored what KA offers to teachers in terms of learning analytics and
proposed ways that it could be improved. Next, I will critique some of the impact evidence
from real-classroom settings.
Evidence Vs discourse
KA claims that there is a growing body of research indicating the impact of its personalized
mastery learning platform. The "Impact" page contains links to different studies divided by
tiers of evidence quality: large-scale randomized-control trial, quasi-experimental studies and
correlational studies with statistical control. The earliest publication dates back to 2015, seven
years after the launch of KA, suggesting that the product had already reached a level of maturity
when impact evaluations started to be carried out. By the short descriptions of the studies it is
possible to infer that KA interprets enhancement as a quantitative change in learning that is
measured by test grades and more time on task (Kirkwood and Price, 2014), thus connecting
learner attainment with technology effectiveness (Cukurova and Luckin, 2018). Due to essay
6
constraints, my analysis will focus on the research conducted in Brazil since it is at the top of
the tier 1 list.
Randomized evaluations usually gain prestige and credibility for the scale and the promise of
finding causal evidence. The study conducted by Fernan, Finamor and Lima (2019) involved
8,302 students from 157 schools with one treatment and one control grade in each of them. The
treatment consisted of teachers receiving training about KA and receiving follow-up by
specialists to monitor the platform use throughout one academic year. The findings validate
Cukurova and Luckin's (2018) statement that robust and reliable evidence of the impact of
technology in education is still lacking. While KA positively influenced students' attitude
toward mathematics, this result did not translate into an overall positive treatment effect on
math proficiency. Moreover, in schools with poor infrastructure, the students spent
significantly less time on KA and created a score gap in comparison with students from schools
with better infrastructure (Ferman, Finamor and Lima, 2019). In sum, it is conceivable that
socioeconomic factors have an effect on the implementation and use of KA and on the
attainment of better learning outcomes and opportunities for public school students.
The Brazilian study revealed some important implementation hurdles. Ferman, Finamor and
Lima (2019) concluded that the implementation was mostly inadequate due to problems with
internet connectivity and maintenance of the school computers. 63% of the schools did not
have one computer per student and used a rotation system, which clashes with KA's proclaimed
benefit of personalization and endorses that proven to work is not the same as effective
implementation of edtechs (Slavin, 2017). Furthermore, since broadband access is quite limited
in low-income countries, edtechs are urged to provide features for offline access and ensure
compatibility with low-tech devices (UNESCO, 2022). Given that the study sheds light on what
hinders the generation of KA's intended impact, I do not understand why good practices or
minimum required standards are not widely promoted on the teacher-user interface. Context-
relevant information as to what works, to what ends, in which situation and for whom could
guide Brazilian teachers, policymakers and school leaders who desire to implement KA.
Simple and straightforward recommendations for optimal use embedded in the teacher-user
experience could prove significantly more helpful than the links to academic articles.
Furthermore, Khan Academy could consider funding studies to assess the effectiveness of its
courses across other subject areas.
7
KA also displays short testimonials of users attesting to how the platform changed their lives.
The anecdotal evidence consists of empirical and verifiable endorsement based on personal
experiences (Cukurova and Luckin, 2018), and KA does a good job in selecting strong
testimonials portraying the stories of people who used the platform and now donate money to
KA thus contributing to the learning of other people. Although this certainly has value, I do
not think that KA has been able to gather robust evidence showing real impact and learning
gains. Therefore, I believe that the affirmation "The evidence is clear — Khan Academy works"
found on the "Impact" page remains an unsubstantiated claim.
Conclusion
In this essay, I aimed to apply rigorous critical lenses to critique KA. In section 1, I examined
the platform's claimed educational value and problematized the meaning and practicality of PL.
I also emphasized that by putting a premium on individual work, KA offers limited learning
opportunities regarding higher-order thinking skills. In section 2, I analyzed the dashboards for
teachers and highlighted that some multimodalities could complement the logged data reports
provided that they also adopted a more user-friendly approach to trigger on-the-spot
interventions. Finally, in section 3 I reviewed the data on impact generated by KA use and
suggested that despite weak evidence about the product's efficacy, the organization could
intentionally spread good practices for optimal pedagogical outcomes.
In my view, KA is a great source of controlled practice and it succeeds in engaging learners
through gamification. It is certainly a powerful tool to supplement classroom teaching and it is
praiseworthy that most of its resources are free. However, there are opportunities to further
enhance the learning experience and collect evidence of its educational value in line with its
discourse.
References
Bayne, S. n. (2020). The manifesto for teaching online / Siân Bayne, Peter Evans, Rory Ewins,
Jeremy Knox, James Lamb, Hamish Macleod, Clara O'Shea, Jen Ross, Philippa Sheail,
Christine Sinclair ; illustrated by Kirsty Johnston. The MIT Press.
8
Biesta, G. Against learning: reclaiming a language for education in an age of learning.
Blikstein, P., & Worsley, M. (2016). Multimodal Learning Analytics and Education Data
Mining: using computational technologies to measure complex learning tasks. Journal of
Learning Analytics, 3(2), 220-238
Bloom, B. S. (1973). Recent developments in mastery learning. Educational psychologist,
10(2), 53-57. [Link]
Cukurova, M., Giannakos, M., & Martinez‐Maldonado, R. (2020). The promise and challenges
of multimodal learning analytics. British Journal of Educational Technology, 51(5), 1441-
1449. [Link]
Cukurova, M., & Luckin, R. (2018). Measuring the Impact of Emerging Technologies in
Education: A Pragmatic Approach. Springer, Cham.
Dede, C. (2008). Theoretical perspectives influencing the use of information technology in
teaching and learning. International handbook of information technology in primary and
secondary education, 43-62.
Dillenbourg, P. (2013). Design for classroom orchestration. Computers and education, 69, 485-
492. [Link]
Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018). From signals to knowledge: A
conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning,
34(4), 338-349.
Ferman, B., Finamor, L., & Lima, L. (2019). Are Public Schools Ready to Integrate Math
Classes with Khan Academy?
Freire, P. (1972). Pedagogy of the oppressed / Paulo Freire ; translated [from the Portuguese]
by Myra Bergman Ramos. Sheed and Ward.
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about
learning. TechTrends, 59, 64-71.
Holmes, W., Anastopoulou, S., & Mavrikis, E. (2018). Technology-enhanced Personalised
Learning: Untangling the Evidence. Robert Bosch Stiftung.
Kirkwood, A., & Price, L. (2014). Technology-enhanced learning and teaching in higher
education: what is ‘enhanced’ and how do we know? A critical literature review. Learning,
media and technology, 39(1), 6-36.
UNESCO. 2023. Global Education Monitoring Report Summary 2023: Technology in
education: A tool on whose terms? Paris, UNESCO.
Pelletier, C. (2023). Against Personalised Learning. International journal of artificial
intelligence in education. [Link]
9
Skinner, B. F. (1958). Teaching machines; from the experimental study of learning come
devices which arrange optimal conditions for self instruction. Science (American Association
for the Advancement of Science), 128(3330), 969-977.
[Link]
Slavin, R. E. (2017). Evidence-based reform in education. Journal of Education for Students
Placed at Risk (JESPAR), 22(3), 178–184.
Watters, A. (2021). Teaching machines. The MIT Press.
Wise, A. F. (2019). Learning analytics: Using data-informed decision-making to improve
teaching and learning. Contemporary technologies in education: Maximizing student
engagement, motivation, and learning, 119-143.
10