Short Answer Grading Using One-shot Prompting and Text Similarity

Scoring Model

Su-Youn Yoon
EduLab, Inc.,
Shibuya city, Tokyo, Japan
[Link]@[Link]

Abstract answer includes content X, then it receives a score

both analytic scores and final holistic scores. ASAG dataset. The question requests answers for
Short answer items typically consist of multi- three sub-questions, and the relevant text spans of
ple sub-questions, and providing an analytic each answer is marked by (a), (b), (c) in Figure
score and the text span relevant to each sub- 1. We will refer to the text span relevant to each
question can increase the interpretability of the sub-question as the ‘justification key’ and the score
automated scores. Furthermore, they can be of each sub-question as the ‘analytic score’ as in
used to generate actionable feedback for stu- Mizumoto et al. (2019).
dents. Despite these advantages, most studies
Providing analytic scores and justification keys
have focused on predicting only holistic scores
due to the difficulty in constructing dataset with is beneficial because it can improve the inter-
manual annotations. To address this difficulty, pretability and validity of automated scoring mod-
we used large language model (LLM)-based els. For instance, by comparing the system-
one-shot prompting and a text similarity scor- detected justification keys with the manually-
ing model with domain adaptation using small annotated justification keys, we can understand
manually annotated dataset. The accuracy and where the systems pay attention during score gener-
quadratic weighted kappa of our model were
ation and evaluate whether it is valid or not. Next,
0.67 and 0.71 on a subset of the publicly avail-
able ASAG dataset. The model achieved a sub- they can be used for systems to provide actionable
stantial improvement over the majority base- feedback; students can understand why their an-
line. swers are incorrect and how they can correct them.
However, it is difficult to train such models because
1 Introduction it requires additional human annotations, resulting
in a substantial cost increase.
Automated scoring systems can assess learners’
In this study, we will present an automated short
responses faster than human raters, with the result-
answer grading model that can produce analytic
ing scores being consistent over time. This has
scores and justification keys. To overcome the dif-
prompted strong demand for high-performing auto-
ficulty of constructing a large scale dataset with
mated scoring systems.
manual annotations, we will explore approaches
Items that elicit short responses are efficient
that can be trained using small training data.
method to evaluate students’ knowledge and have
been widely used in the education field. Many re- 2 Previous studies
searches have been conducted in developing auto-
mated short answer grading (ASAG) systems result- Previous studies in the ASAG field have explored
ing high performing systems with relatively small various approaches, such as traditional linguistic
training dataset. features with statistical models and neural mod-
Short response grading focuses on the content els. Among these possible approaches, the ap-
rather than the writing quality, and the possible propriate one in a particular scoring condition is
answers are restricted by the closed-form of the largely dependent on the availability of a manually
question (Burrows et al., 2015). Typically, the annotated, question-specific dataset. If sufficient
scoring rubric for one question consists of mul- amount of human-scored answers are available for
tiple sub-questions that are in the form of "if an each question as training data, data-driven mod-
Figure 1: Example of a question and student’s answer from the Automated Student Assessment Prize Short Answer
Scoring dataset

els such as statistical models using word n-gram tion of automated annotations had a positive impact,
features or embedding-based neural models have but this impact decreased as the manually-scored
shown promising performance(Heilman and Mad- data increased.
nani, 2013; Riordan et al., 2017; Sung et al., 2019). The researches in the second group consider
On the contrary, if only rubrics or small reference the ASAG to be a combination of two well-
samples are available, the models need to rely on known Natural Language Processing (NLP) tasks:
methods to calculate the semantic similarity be- answer-span (justification key) extraction and text-
tween the reference and the test answers. similarity scoring(Haller et al., 2022). First, ref-
Among the first group, Heilman and Madnani erence answers are created based on the rubrics
(2013) developed a model based on the word and or model answers. Next, they extract justification
character n-gram features and domain adaptation keys from a student’s answer using regular expres-
approach, and it achieved good performance. Rior- sions or methods used in the information retrieval.
dan et al. (2017) investigated several basic neural The similarity scores between student’s answers
architectures and the model comprised of the word and reference answers were calculated using var-
embeddings, LSTM layers, and attention could ious semantic similarity metrics such as bag-of-
achieve promising performance over a wide range words, content vector analysis, and word embed-
of tasks and datasets. More recently, Sung et al. ding. Based on the similarity scores, a final score
(2019) achieved the comparable performance to hu- that decides whether the justification key is seman-
man raters by fine-tuning pre-training BERT model tically equivalent to the concept in the reference
with domain adaptation. answers is assigned.
Most studies have focused on predicting only Similar to the second group, we aim to solve
holistic scores, but some recent studies have inves- the ASAG task using only a few manually-scored
tigated predicting analytic scores and justification answers and the two-task approaches. Especially,
keys. Mizumoto et al. (2019) trained a bidirec- we will explore LLM-based one-shot prompting in
tional LSTM model with an attention mechanism the context of the ASAG. Recent studies(Radford
to minimize the difference between the system at- et al., 2019; Brown et al., 2020; Chung et al., 2022)
tention span and manually annotated justification have demonstrated that large language models
keys. The model achieved good performance in (LLMs) with larger sizes of parameters can achieve
both tasks, and the joint-learning could improve substantial performance improvements over small
the performance of the holistic score prediction in pre-trained language models. Furthermore, un-
the low-resource scenario. Wang et al. (2021) used like pre-trained language models which require
a self-learning approach and replaced the automat- fine-tuning with a substantially large task-specific
ically detected justification keys with the manual dataset, LLMs can learn new tasks through in-
annotations. In the low-resource scenario, the addi- context learning using only a few demonstration
examples. In particular, GPT-3, the third generation train(80%), development(10%), and test(10%) set,
of the Generative Pre-trained Transformer model, respectively. From the train set, 64 answers (16 an-
has shown superior performance in few-shot learn- swers per question) were selected for the reference
ing, and it has achieved comparable or superior answer augmentation (hereafter, answer augmenta-
performance over fine-tuned pre-trained models on tion set)
diverse tasks(Brown et al., 2020).
LLM-based few-shot learning has already been Train
Development Test
used in educational tasks such as writing assistant, Total Answer
tutoring(Cao, 2023) or item generation(Park et al., augmenta-
2022), but relatively limited researches have con- tion
ducted in the automated scoring field. Mizumoto N 5,226 64 680 636
and Eguchi (2023) used GPT-3 to grade holistic
Table 1: Number of answers in each set
writing proficiency of non-native English learners’
essays, but the performance was not high.
Table 2 shows the distribution of the holistic
In this study, we will explore a combination
scores (score1) rated by a human rater. The ma-
of LLM-based one-shot prompting and sentence-
jority score was 0 and it was more than half of the
similarity scoring model. In particular, we aim to
entire data.
answer the following questions:
Human score 0 1 2 3
• How accurately can the model grade short
% 53 19 17 11
answers?
Table 2: Percentage of each score out of the total data
• Can unsupervised domain adaptation achieve
substantial improvement over the baseline
model without the domain adaptation? For each answer in the answer augmentation set,
we manually annotated the justification keys and
• How accurately can the model predict the jus- analytic scores. All four questions used in this
tification keys and analytic scores? study consisted of three sub-questions. We marked
the justification keys of each answer as (a), (b), (c)
The proposed model produces both justification in Figure 1. Next, we assigned a binary score for
keys and analytic scores, and it is essential to eval- each justification key. The score was 1 when the
uate the quality of these outputs. However, the data justification key matched one of the correct answers
used in this study included only holistic scores. Be- provided in the rubrics. If the answer received a
cause of this, we generated holistic scores based holistic score of 3 (perfect score) but the justifica-
on the analytic scores and indirectly evaluated the tion key was not included in the correct answers,
quality of the analytic scores through the holistic we still treated it as a correct answer. We assigned
score evaluation. For the justification key evalu- 1 and registered this justification key to the correct
ation, we used a small set of manually annotated answer list2 . For the remaining cases, we assigned
dataset. 03 . In addition, we stored the correct answer that
matched each justification key.
3 Data
We used the subset of the Automated Student As- 4 Method
sessment Prize Short Answer Scoring (hereafter, We used an LLM-based one-shot prompting and
ASAP-SAS) data set. It originally included ten a neural text similarity model to grade students’
questions from science, biology, English, and En- answers. First, we extracted a justification key for
glish language arts domains. We selected four ques- each sub-question using the one-shot prompting.
tions (two science and two biology1 ) because the Next, we calculated the similarity scores between
rubrics included the detailed explanations and the
2
correct responses. The correct answers in the rubrics did not cover all pos-
sible correct answers. In order to increase the efficiency of
The total number of answers for the four ques- finding missing correct answers, we annotated the answers
tions were 6, 542. The data was partitioned into with a holistic score of 3 and registered missing answers first.
3
For instance, in Figure 1, the analytic scores of (a), (b),
1
They were question 1, 2, 5, and 6. (c) were 0, 1, 0, respectively.
each justification key and the reference answers similarity scoring tasks.
using the Sentence-BERT(Reimers and Gurevych, The performance of SBERT can be further im-
2019) model and assigned an analytic score. Fi- proved by domain adaptation (Thakur et al., 2021).
nally, we generated a holistic score from all ana- For domain adaptation, we created two datasets: a
lytic scores. A detailed explanation for each step is dataset with manual annotations (hereafter, gold
provided in the subsection. dataset) and a dataset with automatically generated
labels(hereafter, silver dataset).
4.1 Justification key extraction For the gold dataset, we used the answer aug-
We used an one-shot prompt to extract justification mentation set. For each student’s answer in the
keys from the student’s answer. The prompt con- set, we extracted justification keys and created all
sisted of the instruction and one demonstration ex- possible combinations with the augmented refer-
ample. An example of one question and its prompt ence answers. The label for each combination was
is presented in Figure 2. generated in the semi-automated way. If the text
We extracted the text describing the sub- pairs of the combination was semantically equiva-
questions from the original question (underlined lent, the label was 1, otherwise it was 0. We used
sentences in Figure 2) and used it as an instruc- different steps for the correct answer group and the
tion. The demonstration example consisted of an incorrect answer group. If the justification key was
input and an output in JSON format. The input con- from the correct answer group, we compared its
tained a "student’s answer" field with one sample most similar reference answer with that of the refer-
answer as its value. The output contained justifica- ence text of the combination. If they were identical,
tion keys for each sub-question extracted from the we assigned 1; otherwise, we assigned 0. If the jus-
sample answer. tification key was from the incorrect answer group
and the reference text was from the correct answer
4.2 Reference Answer Augmentation
group, we assigned 0. For the remaining cases, we
We extracted correct answers from the rubrics as manually examined the combination and assigned 1
reference answers. The initial reference answers when they were semantically equivalent; otherwise,
were augmented using the manually annotated an- we assigned 0.
swer augmentation set. The justification keys were The silver dataset was generated using the train-
extracted and classified into two groups (correct vs. ing partition. All answers in the training set were
incorrect) based on the manual analytic scores. In split into sentences. For each sentence, we found
addition, for each correct answer in the augmented two answers from the reference answers: the most
set, we stored the most similar correct answer from similar correct answer and the most similar incor-
the initial reference answers. We used this informa- rect answer. A similarity score between the training
tion to expedite labeling of the gold dataset used in sentence and the reference answer was calculated
the domain adaptation. After removing duplicated using the pre-trained SBERT cross-encoder. We
keys, the justification keys were combined with the first selected one with the lower similarity score
initial reference answers. and assigned 0. For the remaining one, if the score
was higher than 0.5, the label was 1; otherwise it
4.3 Sentence-BERT and Domain Adaptation
was 0.
Sentence-BERT (hereafter, SBERT) is a modifi- Finally, we trained the pre-trained SBERT bi-
cation of the pre-trained BERT network, and it encoder model using both gold and silver dataset.
uses Siamese network and triplet loss to minimize
the distance between sentence pairs with similar 4.4 Analytic Score Generation
meaning s and maximize the distance between sen- At the inference, justification keys in the student’s
tence pairs with different meanings (Reimers and answer were extracted using the prompt. For each
Gurevych, 2019). The model derives semantically justification key, we created all possible pairs with
meaningful sentence embeddings, and the similar- the reference answers and calculated the similarity
ity score between two sentence embeddings (e.g., scores using the adapted SBERT model described
cosine similarity score) is used at the inference in the section 4.3. The pair with the highest simi-
time. The models trained on the natural language larity score was selected and the analytic score was
inference (NLI) dataset have showed the state-of- generated as follows: the analytic score was 1 only
the art performance in various unsupervised text
 Figure 2: one-shot prompt for the question in Figure 1

when the most similar reference answer was one the training script provided by Zeng et al. (2022)5 .
of the correct answers and the similarity score was For each question, the pre-trained bert-base-cased
higher than the 0.5. Otherwise, it was 0. model with a sequence classification layer on top
was fine-tuned using the training set. The detailed
4.5 Holistic Score Generation descriptions about the training was provided in
Holistic score was the sum of all analytic scores. Zeng et al. (2022) (hereafter, fine-tuned BERT
There was no penalty for incorrect answers. The model)
maximum score for all questions in this study was The proposed model used human scores for 64
3, so no extra points were awarded for answers answers, while the bench-marking model used the
containing more than 3 correct justification keys. human scores for the entire training set(5, 226).

5 Experiment 6 Results
We used the OpenAI’s GPT-3.5 ‘text-davinci-003’ Table 3 provides the performance of the auto-
model for the one-shot prompting. We used the mated scoring models on the test set. The human
text completion API and set the temperature to 0 to raters achieved a good agreement in this task, and
avoid the randomness in the output. both the exact agreement and quadratic weighted
For the SBERT model, we used the ‘all-mpnet- kappa were 0.92 and 0.96, respectively.
base-v2 model’ in the huggingface model reposi-
tory4 as the base model. accuracy wtkappa corr
For the domain adaptation, we used both a gold one- 0.68 0.73 0.73
dataset with the manual labels and a silver dataset shot+SBERT
with the automated labels. The size of the gold fine-tuned 0.77 0.88 0.88
dataset was 1, 218 text pairs, while the size of silver BERT
dataset was 33, 000 text pairs. The base SBERT
Table 3: accuracy, quadratic weighted kappa, and Pear-
model was trained using the combined dataset. son correlation coefficients between the human and sys-
As a bench-marking model, we fine-tuned pre- tem scores
trained language model (Devlin et al., 2018) using
4
[Link] The combination of one-shot prompting and
base-v2 5
The training script was downloaded from
[Link]
SBERT (one-shot+SBERT) achieved substantially adapted SBERT model with and without the an-
better performance over the majority baseline (ac- swer augmentation were lower than 0.02 in both
curacy = 0.53). However, there was a meaningful accuracy and the quadratic weighted kappa.
difference with the fine-tuned BERT model, and
the fine-tuned model achieved substantially better 7 Discussion
performance. We investigated the accuracy of detecting justi-
The one-shot+SBERT model used over 1, 000 fication keys using the answer augmentation set.
non-scored answers per question. In order to evalu- We compared manually annotated justification
ate the impact of non-scored answers on the model keys with those extracted by the GPT-3.5 model.
performance, we conducted an ablation test. While Based on the manual annotation, there were 216
using the same method for the justification key ex- justification keys, resulting in an average of 3.37
traction, we used following additional SBERT mod- justification keys per answer. We calculated the
els for the semantic similarity scoring: the number of words in the justification keys. The
average based on the manual annotations was 8.4,
• baseline: the baseline SBERT model without
while that based on the GPT model was 10.4. The
the reference answer augmentation
GPT model had a tendency to extract longer spans
• reference answer-augmented model: the base- and contained a few extra words such as conjunc-
line SBERT model with the reference answer tion markers at the beginning. Adding these extra
augmentation words did not have a big impact on the next step, so
we treated these cases as correct. More concretely,
• SBERT with the domain adaptation: the for each manual and system-based justification key
SBERT model adapted using the training set pairs, if the system key contained the entire manual
without the reference answer augmentation key or the word overlap between two keys was over
90%, we counted it as a correct case. The correct
Table 4 provides the results of the ablation test.
cases were 195 out of 216, resulting in an accuracy
accuracy wtkappa corr of 90.3%. The accuracy of the justification key de-
baseline 0.44 0.48 0.46 tection was high. Furthermore, most errors were
reference 0.52 0.56 0.62 from low-scoring answers where the justification
answer- keys did not contain correct content. These errors
augmented did not have an impact on the overall score.
model 8 Limitation
SBERT with the 0.67 0.71 0.74
domain adapta- As we mentioned in the early section, one of the
tion major limitation is the absence of the evaluation
one- 0.68 0.73 0.73 of justification keys and analytic scores. In future
shot+SBERT study, we will focus on this evaluation.

Table 4: Ablation test of the one-shot+SBERT model 9 Conclusion and Future Study
We developed automated short answer grading
The performance of the baseline model with-
models using only 16 answers per question, that
out any adaptation on the ASAG dataset was
was less than 5% of the bench-marking model used.
substantially worse than the final model (one-
The model was based on the one-shot prompting
shot+SBERT). The accuracy was 0.44 and it was
and Sentence-BERT model. The model achieved
even lower than the majority baseline. The aug-
a good performance, resulting in 27% of relative
mentation of the reference answers using a small
error reduction in accuracy over the majority base-
dataset could achieve a small improvement, but the
line.
major improvement was achieved by the domain
However, there was a large gap compared to
adaptation. The impact of the reference answer
the model trained on the large manually scored
augmentation was marginal when the model was
dataset. In order to use automated scoring mod-
adapted to the domain dataset; differences between
els to grade real tests, the models need to achieve
near-human performance, and the performance of
the current model was far behind. However, LLM- Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring
based prompting can be used in different scenarios. the potential of using an ai language model for auto-
mated essay scoring. Research Methods in Applied
For instance, the analysis of the justification key
Linguistics, 2(2):100050.
revealed that the automated justification key extrac-
tion was highly accurate. The LLM-based prompt- Tomoya Mizumoto, Hiroki Ouchi, Yoriko Isobe, Paul
ing can be used to efficiently and cost-effectively Reisert, Ryo Nagata, Satoshi Sekine, and Kentaro
Inui. 2019. Analytic score prediction and justifica-
enrich the annotations of the existing dataset, and tion identification in automated short answer scoring.
this dataset can be used to train the models to pro- In Proceedings of the Fourteenth Workshop on Inno-
vide not only the final score, but also additional in- vative Use of NLP for Building Educational Applica-
formation that can be used to increase the model’s tions, pages 316–325.
interpretability. Previous studies, such as Mizu- Yena Park, Geoffrey T LaFlair, Yigal Attali, Andrew
moto et al. (2019), have shown that a joint model Runge, and Sarah Goodwin. 2022. Interactive read-
trained on predicting both justification keys and ing—the duolingo english test.
holistic scores can improve the accuracy of auto- Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
mated scores. Dario Amodei, Ilya Sutskever, et al. 2019. Language
models are unsupervised multitask learners. OpenAI
blog, 1(8):9.
References
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert:
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sentence embeddings using siamese bert-networks.
Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind arXiv preprint arXiv:1908.10084.
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, et al. 2020. Language models are few-shot Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten
learners. Advances in neural information processing Zesch, and Chungmin Lee. 2017. Investigating neu-
systems, 33:1877–1901. ral architectures for short answer scoring. In Pro-
ceedings of the 12th workshop on innovative use of
Steven Burrows, Iryna Gurevych, and Benno Stein. NLP for building educational applications, pages
2015. The eras and trends of automatic short answer 159–168.
grading. International Journal of Artificial Intelli-
gence in Education, 25(1):60–117. Chul Sung, Tejas Dhamecha, Swarnadeep Saha, Tengfei
Ma, Vinay Reddy, and Rishi Arora. 2019. Pre-
Chen Cao. 2023. Leveraging large language model training bert on domain resources for short answer
and story-based gamification in intelligent tutor- grading. In Proceedings of the 2019 Conference on
ing system to scaffold introductory programming Empirical Methods in Natural Language Processing
courses: A design-based research study. arXiv and the 9th International Joint Conference on Natu-
preprint arXiv:2302.12834. ral Language Processing (EMNLP-IJCNLP), pages
6071–6075.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar-
ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Nandan Thakur, Nils Reimers, Johannes Daxenberger,
Wang, Mostafa Dehghani, Siddhartha Brahma, et al. and Iryna Gurevych. 2021. Augmented SBERT: Data
2022. Scaling instruction-finetuned language models. augmentation method for improving bi-encoders for
arXiv preprint arXiv:2210.11416. pairwise sentence scoring tasks. In Proceedings of
the 2021 Conference of the North American Chapter
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and of the Association for Computational Linguistics: Hu-
Kristina Toutanova. 2018. Bert: Pre-training of deep man Language Technologies, pages 296–310, Online.
bidirectional transformers for language understand- Association for Computational Linguistics.
ing. arXiv preprint arXiv:1810.04805.
Tianqi Wang, Hiroaki Funayama, Hiroki Ouchi, and
Stefan Haller, Adina Aldea, Christin Seifert, and Kentaro Inui. 2021. Data augmentation by rubrics for
Nicola Strisciuglio. 2022. Survey on automated short answer grading. Journal of Natural Language
short answer grading with deep learning: From Processing, 28(1):183–205.
word embeddings to transformers. arXiv preprint
arXiv:2204.03503. Zijie Zeng, Xinyu Li, Dragan Gasevic, and Guanliang
Chen. 2022. Do deep neural nets display human-like
Michael Heilman and Nitin Madnani. 2013. Ets: Do- attention in short answer scoring? In Proceedings of
main adaptation and stacking for short answer scor- the 2022 Conference of the North American Chap-
ing. In Second Joint Conference on Lexical and Com- ter of the Association for Computational Linguistics:
putational Semantics (* SEM), Volume 2: Proceed- Human Language Technologies, pages 191–205.
ings of the Seventh International Workshop on Se-
mantic Evaluation (SemEval 2013), pages 275–279.