CAN GENERATIVE LARGE LANGUAGE MODELS PERFORM ASR ERROR CORRECTION?

Rao Ma, Mengjie Qian, Potsawee Manakul, Mark Gales, Kate Knill

ALTA Institute, Machine Intelligence Lab, Department of Engineering, Cambridge University, UK

ABSTRACT and is jointly trained with the acoustic encoder. Due to the
ASR error correction is an interesting option for post process- limited availability of labelled speech training data, ASR sys-
arXiv:2307.04172v2 [[Link]] 29 Sep 2023

ing speech recognition system outputs. These error correc- tems struggle to generate rare words that have low frequency
tion models are usually trained in a supervised fashion using in the training corpus. Large quantities of text data covering a
the decoding results of a target ASR system. This approach wide range of domains are much easier to collect and process
can be computationally intensive and the model is tuned to than speech data. Therefore, text-based methods have been
a specific ASR system. Recently generative large language explored to improve the performance of speech recognition
models (LLMs) have been applied to a wide range of natural systems. One option is ASR error correction which automati-
language processing tasks, as they can operate in a zero-shot cally identifies errors within the ASR hypothesis and outputs
or few shot fashion. In this paper we investigate using Chat- the corrected transcription [7, 8, 9].
GPT, a generative LLM, for ASR error correction. Based on The development of the error correction model follows
the ASR N-best output, we propose both unconstrained and the trend of Natural Language Processing (NLP) technology.
constrained, where a member of the N-best list is selected, Early models were rule-based systems, which required care-
approaches. Additionally, zero and 1-shot settings are eval- fully designed features and human expertise [10]. With the
uated. Experiments show that this generative LLM approach emergence of recurrent networks and attention mechanisms,
can yield performance gains for two different state-of-the-art models with an E2E architecture became mainstream. These
ASR architectures, transducer and attention-encoder-decoder models usually adopt a similar structure where the bidirec-
based, and multiple test sets. tional encoder takes the ASR transcription as input and the
reference text is used as the training target. This approach has
Index Terms— ASR error correction, generative model,
shown promising performance on diverse datasets for ASR
large language model, speech recognition, zero-shot
models of different architectures [11, 12, 13, 14].
In the past few years, large-scale pre-trained language
1. INTRODUCTION
models became available. These are generally trained on
Automatic speech recognition (ASR) systems aim to tran- multi-domain text data that is several magnitudes more than
scribe human speech into readable text and are the key com- the prevailing ASR systems. For instance, BERT is pre-
ponent for human-computer interaction [1]. In recent years, trained on 3,300M words [15] and T5 is trained on 750GB
significant advancements have been made in this area. End- text [16]. Previous works [17, 18] developed methods to
to-end (E2E) systems such as LAS or RNN-T are effective at build an ASR error correction model based on the powerful
modelling long contexts within the utterance and show supe- T5 model. By fine-tuning from the pre-trained NLP model,
rior performance compared to the HMM-based counterparts implicit knowledge learned from huge amounts of text data
[2, 3, 4]. The training of ASR systems requires the avail- can be effectively transferred to the target error correction
ability of high-quality transcribed speech data, which can be task. Results indicate the importance of adopting the ASR
costly to obtain. Publicly available corpora usually contain at N-best list rather than the top one hypothesis as model input
most thousands of hours of annotated speech data. In contrast, for accessing richer context in the correction process.
the recently released ASR model, Whisper [5], is pre-trained Typically error correction models are trained in a super-
on around 680,000 hours of weakly supervised data collected vised fashion to effectively learn the error patterns made by
from the Internet. the ASR system. The training process requires first decod-
The decoder part of an RNN-T or a LAS model acts as ing large amounts of speech data with the ASR system of in-
a language model that estimates the probability of the gen- terest, and then using the erroneous hypotheses to train the
erated word sequence [6]. It learns from the reference text correction model. These two stages can be computationally
intensive to adopt in practice. Additionally, the error correc-
This paper reports on research supported by Cambridge University Press
& Assessment, a department of The Chancellor, Masters, and Scholars of
tion model is usually bound to a specific ASR system and a
the University of Cambridge. Mengjie Qian is supported by EPSRC Project particular domain. Therefore, when we switch the underlying
EP/V006223/1 (Multimodal Video Search by Examples). ASR system or apply it to a new domain, the corresponding
error correction model can be less effective and needs to be to use the ASR N-best list as input to the error correction
re-trained. To address the above issues, in this paper we pro- model while using the more powerful ChatGPT model rather
pose approaches to perform zero-shot or few-shot ASR error than T5 to perform the task.
correction. These novel methods are training-free and enable
plug-and-play support to an existing ASR system. 3. LLM-BASED ASR ERROR CORRECTION
Generative large language models (LLMs) such as Chat-
GPT have demonstrated remarkable performance of language In this section, we introduce our methods of utilising gener-
understanding on text processing tasks [19, 20, 21]. In our ative large language models for zero-shot or few-shot error
work, we examine its performance in identifying and correct- correction. Two types of tasks are discussed: unconstrained
ing errors on two state-of-the-art ASR architectures. Different error correction and N-best constrained error correction.
prompts to ChatGPT and both unconstrained and constrained
generation methods are compared on three standard ASR data 3.1. Unconstrained Error Correction
sets. Zero and 1-shot settings are evaluated as well. The re-
sults show that this efficient generative LLM, ChatGPT, error In the unconstrained error correction (uncon) setting, we ask
correction approach yields performance gains for both ASR ChatGPT to directly output the corrected hypothesis without
architectures. adding an explanation. This task can be relatively difficult to
perform as ChatGPT has no prior knowledge about the error
patterns of the ASR system and no access to the original utter-
2. BACKGROUND ance. Instead of the 1-best ASR transcription, therefore, we
input the N-best list obtained from the beam search decoding
Error correction models aim to fix errors in the ASR transcrip-
of the ASR model to ChatGPT. Hypotheses from the N-best
tion and are an interesting option for ASR post-processing. A
list can act as hints to help the model better detect and cor-
standard error correction model adopts an E2E structure, tak-
rect the errors [27, 28]. In the ablation study in Section 4.4
ing the ASR transcriptions as the model input and generating
we show that using a reasonable number of N is important
the corrected sentence. Several model variants incorporating
for the model to achieve good performance. When only the
additional inputs have been proposed [8, 9, 13]. [17] proposes
top one ASR hypothesis is used as input ChatGPT-based error
an N-best T5 error correction model that is fine-tuned from a
correction may degrade performance.
pre-trained T5 model. It leverages the ASR N-best hypothe-
ses as model input and demonstrates significant performance
gain over the model using the 1-best input. It also proposes 3.1.1. Zero-shot vs. 1-shot Prompts
an N-best constrained decoding approach in error correction, The prompt designed for the zero-shot uncon setting is illus-
which uses the combined scores of the ASR model and the T5 trated in Figure 2. In this prompt the hypotheses are sorted
model to find the best hypothesis in the N-best list. by the (descending) ASR posterior score. Furthermore, tags
like <hypothesis1> and </hypothesis1> are used
<BOS> come and pay us a visit <EOS>
to surround each N-best hypothesis. Other input formats such
as using numbers rather than tags or using plain sentences
Transformer Decoder
without the explicitly specified order were also examined and
showed degraded performance to our selected prompt. Con-
Transformer Encoder
sidering the complexity of this task, we additionally exper-
iment with the 1-shot setting to perform in-context learning.
Here, we give an example for ChatGPT to refer to before con-
text correction <BOS> come on past the visit <SEP> come and pay a visit <EOS>
ducting error correction (highlighted in orange in Figure 2).
This example is selected from a decoding result of the Trans-
Fig. 1. N-best T5 error correction model structure. ducer ASR model on the dev other set of LibriSpeech. It is
used in all our 1-shot error correction experiments. By show-
There has been rapid growth in the LLM literature, and ing both input and the desired output in the prompt, we hope
larger and better LLMs are constantly being released. LLMs to remind ChatGPT to match the sentence length of the given
continue to be scaled up in size and pre-trained on increas- hypotheses and only make edits to the detected errors.
ingly more data. With reinforcement learning from human
feedback (RLFH), they are capable of performing several
3.2. N-best Constrained Error Correction
NLP tasks in a zero-shot manner [22, 23]. For example,
LLMs such as ChatGPT have been applied to summary as- Unconstrained error correction is the standard approach to
sessment [24], and grammatical error correction [25, 26]. generate the corrected transcription based on the information
Their inherent ability to perform ASR post-processing tasks, from the given hypotheses. Results in [17, 18] suggest that
however, has been less explored. In this work, we follow [17] constraining the decoding space to the given N-best list leads
 zero-shot uncon zero-shot select
Input: Perform error correction on the top3 outputs generated by an Input: Perform language model rescoring based on the top3 outputs
Automatic Speech Recognition (ASR) system. The ASR hypotheses, generated by an Automatic Speech Recognition (ASR) system. The ASR
listed in order of their ASR posterior score, are as follows: hypotheses, listed in order of their ASR posterior score, are as follows:

<hypothesis1> he grasped the little passing </hypothesis1> <option1> he grasped the little passing </option1>
<hypothesis2> he grasped the little parcel </hypothesis2> <option2> he grasped the little parcel </option2>
<hypothesis3> he grasped the little puzzle </hypothesis3> <option3> he grasped the little puzzle </option3>
Please provide the corrected top1 ASR transcription of the given Please output the selected top1 ASR transcription as <option?> The
utterance only, do not add any explanations or other words. selected top1 ASR transcription </option?>.

ChatGPT: The selected top1 ASR transcription is <option2> he grasped

ChatGPT: he grasped the little parcel
the little parcel </option2>.

1-shot uncon
Input: Perform error correction based on the top3 outputs generated by an Automatic Speech Recognition (ASR) system. The ASR hypotheses
are listed in order of their ASR posterior score. You need to provide the corrected ASR hypothesis directly without any explanations. Here is an
in-context example:

<hypothesis1> but i will see wiltdale i hadn't come across biffon for a long time </hypothesis1>
<hypothesis2> but i will see wiltow i hadn't come across biffon for a long time </hypothesis2>
<hypothesis3> but i will see wilt thou i hadn't come across biffon for a long time </hypothesis3>
Your output: but i will see whelpdale i had not come across biffen for a long time
Please feel free to refer to this example. Please start:
<hypothesis1> he grasped the little passing </hypothesis1>
<hypothesis2> he grasped the little parcel </hypothesis2>
<hypothesis3> he grasped the little puzzle </hypothesis3>

ChatGPT: he grasped the little parcel

Fig. 2. Prompt design for (a) zero-shot unconstrained error correction, (b) zero-shot selective approach, and (c) 1-shot uncon-
strained error correction. Here we use a 3-best list generated by the ASR system as input to ChatGPT for illustration.

to performance gains in some cases. We propose two meth- the zero-shot uncon example in Figure 2 the Levenshtein dis-
ods, therefore, to constrain the output of ChatGPT to be a tance of the ChatGPT output to the 3-best ASR hypotheses is
hypothesis within the given N-best list, namely the selective 1, 0, 1 respectively. The second hypothesis will be selected,
approach and the closest mapping. therefore, as the corrected result for this utterance.

3.2.1. Selective Approach

4. EXPERIMENTS
With the selective approach (select), ChatGPT is asked to
select the most likely ASR transcription from all the candi- 4.1. Setup
dates rather than generate one from scratch. All the input
sentences are listed as <option1> ASR hypothesis We conduct experiments on ChatGPT (gpt-3.5-turbo-0613) to
</option1>, and ChatGPT is asked to return the selected study its performance on error correction for two ASR mod-
option in the format of <option?> The selected els. One is a Conformer-Transducer [29] model containing
ASR transcription </option?>. This method is 12 encoder layers. The model was trained on 960 hours Lib-
similar to language model rescoring to some extent, however, riSpeech data with SpecAugment [30] and speed perturbation
it performs the selection in one go. More importantly, Chat- applied, following the ESPnet recipe [31]. The other ASR
GPT sees all the candidates before deciding on the best one. model studied is the Whisper [5] [Link] model. In decoding
This is different from the rescoring process where language we suppress the probability of the most common punctuation
model scores are generated individually for each of the N-best marks as in [18]. Each ASR model is decoded with a beam
hypotheses without comparing the similarity and correlation size of 10 that generates a 10-best list as a by-product at infer-
between each other. ence. If not stated otherwise, the top five hypotheses are used
as input to ChatGPT, i.e. the size of the input N-best list is
5. The effect of adopting different N is studied in the ablation
3.2.2. Closest Mapping
experiment. We lowercase the ASR N-best list without per-
The closest mapping method (closest) is based on the assump- forming other text processing steps prior to input to ChatGPT.
tion that when ChatGPT performs unconstrained error correc- At the evaluation stage, we run the text normalisation scripts
tion, it first selects the best hypothesis from the given N-best from the Whisper project on both the ASR reference and the
list and then makes modifications based on this sentence to hypothesis text before calculating WER results.
yield the final output. We hope to find this “closest match” The proposed approaches are evaluated on three public
in a reverse process by finding the hypothesis within the ASR datasets, namely LibriSpeech [32], TED-LIUM3 [33], and the
N-best list that has the smallest Levenshtein distance to the Artie Bias Corpus [34]. LibriSpeech is an audiobook-based
ChatGPT unconstrained generation result. For instance, for English speech corpus, TED-LIUM3 is an audio dataset col-
 Dataset Subset # Utts # Words Hours LB TED Artie
System
CTr Whs CTr Whs CTr Whs
LibriSpeech test other 2,939 52K 5.3
TED-LIUM3 test 1,155 52K 4.6 Baseline 6.90 7.37 13.53 3.89 23.67 9.03
Artie Bias test 1,712 15K 2.4
Oracle 4.59 5.24 10.71 2.59 17.95 5.59
† † † †
Table 1. Statistics of test sets used in the experiments. T5 (uncon) 6.37 6.39 12.00 4.56 21.24 9.16
T5 (constr) 6.15† 6.69† 12.12† 3.64† 21.36† 8.14†
lected from TED talks, and the Artie Bias Corpus is a subset
of the Common Voice dataset [35] which is also read speech. ChatGPT Error Correction
The details of the datasets are presented in Table 1. We un- 0-shot uncon 6.64 7.71 11.35† 5.84 18.73† 8.30†
dertook a comparative analysis between ASR error correction 0-shot select 6.52† 7.24 12.61† 4.19 21.88† 8.47†
using the generative LLM ChatGPT and a standard error cor- 0-shot closest 6.29† 7.15 11.97† 4.56 20.64† 8.21†
rection model that adopts an E2E structure. Specifically, we
1-shot uncon 6.29† 7.18 10.13† 4.96 19.35† 8.45†
trained N-best T5 error correction models as described in Sec-
1-shot closest 6.24† 7.03† 11.96† 4.58 21.13† 8.53
tion 2. An error correction model was trained for each of the
two ASR systems mentioned above. Each N-best T5 model
Table 2. WER results for a Conformer-Transducer (CTr) sys-
was fine-tuned on the 10-best outputs of the associated ASR
tem and Whisper (Whs). Error correction results using a T5
model decoded on the 960 hours LibriSpeech training set.
model and ChatGPT are compared. † indicates the improve-
ment over baseline is statistically significant with p < 0.001.
4.2. Experimental Results
For the Conformer-Transducer model, TED-LIUM3 and
In Table 2, we study the behaviour of ChatGPT on ASR error Artie Bias can be considered as out-of-domain datasets. The
correction when a Transducer-based model or a pre-trained ASR system shows high error rates on these test sets while
Whisper model with an Attention-Encoder-Decoder (AED) the T5-based error correction gives 11.3% and 10.2% WERR.
structure is used as the base ASR system. Results from the Results from the ChatGPT-based methods show significant
fine-tuned N-best T5 error correction model are listed for performance improvement compared to the T5 model. On the
comparison. The best performance of the T5 model and TED-LIUM3 test set, the 1-shot uncon approach outperforms
ChatGPT based experiments are highlighted in bold. the ASR baseline by 25.1%. The result is even better than the
For the Conformer-Transducer model, LibriSpeech can be oracle WER of the 5-best list output by the ASR model. Chat-
considered as an in-domain dataset. In this case, the super- GPT leads to worse performance, however, when correcting
vised trained T5 model yields a relative gain of 10.9% (6.90% the Whisper model outputs. In particular, many more deletion
to 6.15%) over the ASR baseline. Unlike the T5 model, Chat- errors than in the baseline can be observed in the ChatGPT
GPT does not require any form of model training prior to er- outputs from all the proposed methods.
ror correction and is therefore more efficient. In the zero- On Artie Bias, 0-shot uncon yields the best performance
shot setting, both the selective approach and closest mapping for the Transducer ASR system, with 20.9% WERR (23.67%
perform better than the unconstrained generation. The 0-shot to 18.73%) over the baseline result. 1-shot uncon performs
closest which finds the closest match of the output corrected slightly worse than the zero-shot setting. Since we pick the
hypothesis in the given N-best list performs better than ask- example in the 1-shot prompt from the dev set of LibriSpeech,
ing ChatGPT to directly select the best one from the N-best this might lead to a mismatch on the Artie test set. Although
list. The unconstrained error correction results become much Whisper already demonstrates state-of-the-art performance,
better when we switch to the 1-shot uncon prompt (6.64% to both T5 and ChatGPT-based error correction approaches yield
6.29%), indicating that ChatGPT better understands the task gains of 9.8% and 9.0% WERR, respectively.
by referring to the given example. When we apply the closest
mapping in the 1-shot setting, the test set WER is reduced to
4.3. Discussion
6.24%, which is comparable to the T5 model performance.
When error correction is applied to Whisper on Lib- In Table 4, we calculate the WER breakdown of different
riSpeech, the unconstrained T5 model yields a 13.3% WERR types of errors for both ASR models. For the Conformer-
(7.37% to 6.39%). Less improvement is seen with the Chat- Transducer model, when using the zero-shot uncon prompt,
GPT approach. Although the 0-shot uncon method performs the error correction results from ChatGPT contain fewer sub-
worse than the ASR baseline, with N-best constrained er- stitution and insertion errors compared to the original ASR
ror correction such as closest mapping, ChatGPT improves baseline while causing much more deletions. With human
the performance over the original Whisper output. The best evaluation, we find out that in the ChatGPT output, error cor-
performance is also obtained with the 1-shot closest setting, rection results for 14 sentences are truncated (only the first
achieving a WER of 7.03% on the LibriSpeech test set. few words are in the ChatGPT output rather than the entire
 ASR Model Type Text
Reference Text now that that blew my mind and you know it took a lot of preparation we had to build cameras and lights ...
Hyp-1 now that that blue my mind and you know i took a lot of preparation we had to build cameras and lights ...
Hyp-2 now that that blue my mind and you know i took a lot of preparation we had to build camers and lights ...
Hyp-3 now that that blew my mind and you know i took a lot of preparation we had to build cameras and lights ...
Transducer
Hyp-4 now that that blow my mind and you know i took a lot of preparation we had to build cameras and lights ...
Hyp-5 now that that blew my mind and you know i took a lot of preparation we had to build camers and lights ...
ChatGPT now that that blew my mind and you know i took a lot of preparation we had to build cameras and lights ...
Hyp-1 now that **** blew my mind and you know it took a lot of preparation we had to build cameras and lights ...
Hyp-2 now that **** blew my mind and *** **** it took a lot of preparation we had to build cameras and lights ...
Hyp-3 now that **** blew my mind and you know it took a lot of preparation we had to build cameras and lights ...
Whisper
Hyp-4 now that **** blew my mind and *** **** it took a lot of preparation we had to build cameras and lights ...
Hyp-5 now that **** blew my mind and *** **** it took a lot of preparation and we had to build cameras and lights ...
ChatGPT now that **** blew my mind and *** **** it took a lot of preparation and we had to build cameras and lights ...

Table 3. Case analysis for ChatGPT 1-shot uncon error correction results with Conformer-Transducer and Whisper N-best lists.

sentence), contributing to 0.2% absolute WER. With 1-shot The Uniq metric refers to the average number of unique hy-
learning, the ChatGPT output is more stable and all the prob- potheses within one N-best list in the test set. For Transducer
lem cases are solved, yielding better overall performance. outputs, the result is close to 5 which is the size of the N-
best list, however, there are more repeated entries in Whisper
WER outputs. This is due to the fact that Whisper learns to gen-
Method erate sentences with inverse text normalisation (ITN) to im-
All Sub Del Ins
prove the readability, i.e. capitalisation added, punctuation in-
Transducer

ASR baseline 6.9 5.3 0.7 0.8 cluded, and disfluencies removed. As a result, in many cases
ChatGPT (0-shot uncon) 6.6 4.6 1.3 0.7 multiple hypotheses in an N-best list only differ in format, not
ChatGPT (1-shot uncon) 6.3 4.6 0.9 0.8 in actual content. Nevertheless, the diversity of the N-best list
ChatGPT (1-shot closest) 6.2 4.8 0.7 0.8 is important for our proposed methods to perform well.

ASR baseline 7.4 4.9 1.7 0.8 Cross WER

Whisper

Data Model Uniq

ChatGPT (0-shot uncon) 7.7 4.7 2.2 0.8 All Sub Del Ins
ChatGPT (1-shot uncon) 7.2 4.7 1.7 0.9 Transducer 4.9 9.1 7.1 1.0 1.0
ChatGPT (1-shot closest) 7.0 4.6 1.6 0.8 LB
Whisper 3.0 12.9 7.5 2.7 2.7

Table 4. Breakdown of WER for ChatGPT error correction Transducer 5.0 7.4 5.4 1.0 1.0
TED
results on Transducer and Whisper outputs on LibriSpeech. Whisper 2.6 9.9 3.9 3.0 3.0
Transducer 4.8 19.9 15.3 2.3 2.3
We observe that ChatGPT also has a tendency to remove Artie
Whisper 2.9 21.1 14.5 3.3 3.3
redundant spoken words from the given ASR hypothesis to
make the transcription more fluent. With 1-shot closest, we Table 5. Statistics of the ASR 5-best lists generated by
search from the given N-best list for the final output and there- the Conformer-Transducer and the Whisper model on Lib-
fore the introduced deletion errors can be reduced. There are riSpeech (LB), TED-LIUM3 (TED) and Artie test sets.
more deletions in the Whisper baseline output compared to
the Transducer’s. When we apply ChatGPT zero-shot uncon- Another observation is that for Whisper, even when the
strained error correction substitution errors are reduced while hypotheses in the N-best list are diverse, the difference may
more deletions are introduced. Again 1-shot closest reduces come from one hypothesis omitting or inserting some irrel-
substitution errors without increasing deletion errors. evant words in the output. This is illustrated with the Cross
To further study the possible reasons why ChatGPT is WER metric in Table 5. Here, we keep all the unique hypothe-
less effective on Whisper outputs in some cases, we anal- ses in an N-best list. Then for each pair of hypotheses in the
yse the N-best list of both ASR models, as shown in Table 5. remaining list, we calculate the WER result against each other
When computing the statistics, punctuation and special sym- and sum the result on the entire set. This metric can help us
bols are removed from the ASR hypotheses, leaving only En- measure the difference between hypotheses within one N-best
glish characters and numbers, to focus on meaningful content. list. The results show that the deletion and insertion rates of
Whisper on Cross WER are much higher than the Transducer only using the top one ASR hypothesis as input, ChatGPT
model, especially on TED-LIUM3. This suggests that Whis- makes many unnecessary changes to the input to make the
per may fail to faithfully transcribe the utterance in all N-best sentence more “reasonable” due to lack of information. With
hypotheses, resulting in sentences with varying lengths. Chat- the increased N-best list size, it learns to compare the differ-
GPT tends to choose more coherent ones, leading to many ences between the hypotheses and correct when the sentences
deletion errors in the output, which might explain why Chat- disagree with each other. For the selective approach, the size
GPT underperforms on the TED-LIUM3 data for Whisper. of the N-best list matters less as ChatGPT performs choice se-
In Table 3, we conduct case analysis for an error correc- lection rather than generating the entire corrected hypothesis.
tion example from the test set of TED-LIUM3. As the table
12
shows, for the Transducer ASR model, all the hypotheses are ASR Baseline 1-shot closest
10.22 10.14
of similar length containing all the information from the ut- 10 9.82 9.83 9.83
terance, and the Uniq metric is 5. ChatGPT helps to correct 8.85
8.40
“blue” into “blew” utilising the given N-best list and world 8 7.35

WER[%]
knowledge. Meanwhile, for Whisper 5-best hypotheses, the
Uniq metric is only 3 due to the repetition problem. In ad- 6
5.06 5.06
dition, disfluencies in the utterance (“that”, “you know”) and
4
the non-existent word (“and”) are incorrectly removed or in-
troduced in the output, resulting in more deletions and inser- 2
tions in Cross WER. The produced N-best list is hence less
informative and misleads ChatGPT into the wrong output. 0
Hyp-1 Hyp-2 Hyp-3 Hyp-4 Hyp-5
The selected closest hypothesis
4.4. Ablation and Analysis Fig. 3. Baseline WER of Transducer and error correction re-
sults with 1-shot closest. The LibriSpeech test set is split into
The results on the standard test sets indicate that ChatGPT is
5 parts according to the number of the closest hypothesis.
effective at detecting errors in the given ASR hypotheses and
generating the corrected transcription, especially for out-of- With 1-shot closest mapping, we select the ASR hypoth-
domain scenarios. To further study where the performance esis within the N-best list that is most similar to the Chat-
gain comes from, we built a ROVER-based system [36] to GPT output. Thus, for each utterance, the selected hypothesis
align and combine the hypotheses in an N-best list with falls in the range of Hyp-1 to Hyp-5, and we divide the Lib-
weighted voting, but it leads to worse results compared to the riSpeech test set into 5 splits accordingly. The proportions of
ASR baseline. It suggests that ChatGPT leverages its implic- each subset are 67%, 14%, 8%, 5%, and 6%. In Figure 3,
itly learned world knowledge to generate the corrected ASR we show the WER results of the ASR baseline and after error
transcription based on the given input information, instead of correction for each subset. When Hyp-1 is selected, WER re-
performing a simple voting process on the N-best list. mains the same as the ASR baseline. The figure shows that
the largest performance gain is achieved when the second best
Method Input LB TED Artie hypothesis from the N-best list is picked. The improvement
ASR baseline - 6.90 13.53 23.67 steadily declines when the number of the selected hypothe-
sis increases until the WER after error correction matches the
1-best 8.25 11.95 21.19 original ASR baseline when Hyp-5 is selected.
3-best 7.01 11.31 18.84
0-shot uncon
5-best 6.64 11.35 18.73
10-best 6.69 11.29 18.72 5. CONCLUSIONS

3-best 6.50 12.73 22.29 In this paper we investigate the use of a powerful generative
0-shot select 5-best 6.52 12.61 21.88 large language model, ChatGPT, to perform ASR error cor-
10-best 6.58 12.56 21.47 rection in zero-shot and 1-shot settings. Using only informa-
tion from the ASR N-best list, the system is able to correct
Table 6. Ablation of the size of the input N-best list using the errors either by selecting one of the N-best, constrained cor-
Conformer-Transducer outputs on three test sets. rection, or in an unconstrained fashion. The proposed meth-
ods show gains over the ASR baseline in general for two
In table 6, we take the Conformer-Transducer model as an state-of-the-art system architectures: Transducer and AED.
example to study the influence of the size of the N-best list. On the Transducer outputs, ChatGPT performs similarly or
Results show that using a large number of N is important for better than a bespoke ASR error correction system based on
ChatGPT to perform well with the zero-shot uncon prompt, the T5 foundation model. For the AED system, it underper-
especially for the in-domain setting. In the extreme case of forms T5 due to inherent problems with the AED N-best lists.
 6. REFERENCES [10] Horia Cucu, Andi Buzo, Laurent Besacier, and Corneliu
Burileanu, “Statistical error correction methods for
[1] Carl M Rebman Jr, Milam W Aiken, and Casey G domain-specific ASR systems,” in Statistical Language
Cegielski, “Speech recognition in the human–computer and Speech Processing: First International Conference,
interface,” Information & Management, vol. 40, no. 6, SLSP 2013, Tarragona, Spain, July 29-31, 2013. Pro-
pp. 509–519, 2003. ceedings 1. Springer, 2013, pp. 83–92.

[2] William Chan, Navdeep Jaitly, Quoc Le, and Oriol [11] Jinxi Guo, Tara N Sainath, and Ron J Weiss, “A spelling
Vinyals, “Listen, Attend and Spell: A neural network correction model for end-to-end speech recognition,” in
for large vocabulary conversational speech recognition,” Proc. 2019 IEEE International Conference on Acous-
in Proc. 2016 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE,
tics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5651–5655.
2016, pp. 4960–4964.
[12] Oleksii Hrinchuk, Mariya Popova, and Boris Gins-
[3] Alex Graves and Navdeep Jaitly, “Towards end-to-end burg, “Correction of automatic speech recognition
speech recognition with recurrent neural networks,” in with transformer sequence-to-sequence model,” in
Proc. International Conference on Machine Learning. Proc. 2020 IEEE International Conference on Acous-
PMLR, 2014, pp. 1764–1772. tics, Speech and Signal Processing (ICASSP). IEEE,
2020, pp. 7074–7078.
[4] Dario Amodei, Sundaram Ananthanarayanan, Rishita
Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, [13] Rao Ma, Hao Li, Qi Liu, Lu Chen, and Kai Yu,
Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang “Neural lattice search for speech recognition,” in
Chen, et al., “Deep Speech 2: End-to-end speech recog- Proc. 2020 IEEE International Conference on Acous-
nition in English and Mandarin,” in International con- tics, Speech and Signal Processing (ICASSP). IEEE,
ference on machine learning. PMLR, 2016, pp. 173– 2020, pp. 7794–7798.
182.
[14] Linchen Zhu, Wenjie Liu, Linquan Liu, and Edward Lin,
[5] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- “Improving ASR error correction using N-best hypothe-
man, Christine McLeavey, and Ilya Sutskever, “Robust ses,” in Proc. 2021 IEEE Automatic Speech Recognition
speech recognition via large-scale weak supervision,” in and Understanding Workshop (ASRU). IEEE, 2021, pp.
International Conference on Machine Learning. PMLR, 83–89.
2023, pp. 28492–28518.
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
[6] Zhong Meng, Sarangarajan Parthasarathy, Eric Sun, Kristina Toutanova, “BERT: Pre-training of Deep Bidi-
Yashesh Gaur, Naoyuki Kanda, Liang Lu, Xie Chen, rectional Transformers for Language Understanding,” in
Rui Zhao, Jinyu Li, and Yifan Gong, “Internal lan- Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
guage model estimation for domain-adaptive end-to-end
speech recognition,” in 2021 IEEE Spoken Language [16] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Technology Workshop (SLT). IEEE, 2021, pp. 243–250. Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei
Li, and Peter J Liu, “Exploring the limits of transfer
[7] Rahhal Errattahi, Asmaa El Hannani, and Hassan Ouah- learning with a unified text-to-text transformer,” The
mane, “Automatic speech recognition errors detection Journal of Machine Learning Research, vol. 21, no. 1,
and correction: A review,” Procedia Computer Science, pp. 5485–5551, 2020.
vol. 128, pp. 32–37, 2018.
[17] Rao Ma, Mark JF Gales, Kate Knill, and Mengjie
[8] Yichong Leng, Xu Tan, Rui Wang, Linchen Zhu, Jin Xu, Qian, “N-best T5: Robust ASR Error Correction using
Wenjie Liu, Linquan Liu, Xiang-Yang Li, Tao Qin, Ed- Multiple Input Hypotheses and Constrained Decoding
ward Lin, et al., “FastCorrect 2: Fast error correction on Space,” arXiv preprint arXiv:2303.00456 (Accepted by
multiple candidates for automatic speech recognition,” Proc. Interspeech 2023), 2023.
in Proc. Findings of the Association for Computational
Linguistics: EMNLP 2021, 2021, pp. 4328–4337. [18] Rao Ma, Mengjie Qian, Mark JF Gales, and Kate M
Knill, “Adapting an Unadaptable ASR System,” arXiv
[9] Haoyu Wang, Shuyan Dong, Yue Liu, James Logan, preprint arXiv:2306.01208 (Accepted by Proc. Inter-
Ashish Kumar Agrawal, and Yang Liu, “ASR error cor- speech 2023), 2023.
rection with augmented transformer for entity retrieval,”
in Proc. Interspeech 2020, 2020. [19] OpenAI, “GPT-4 technical report,” 2023.
[20] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier [29] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki
Martinet, Marie-Anne Lachaux, Timothée Lacroix, Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang,
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Zhengdong Zhang, Yonghui Wu, et al., “Conformer:
Azhar, et al., “Llama: Open and efficient foundation lan- Convolution-augmented transformer for speech recog-
guage models,” arXiv preprint arXiv:2302.13971, 2023. nition,” in Proc. Interspeech 2020, 2020, pp. 5036–
5040.
[21] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin
Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak [30] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng
Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,
et al., “Palm 2 technical report,” arXiv preprint “SpecAugment: A simple data augmentation method
arXiv:2305.10403, 2023. for automatic speech recognition,” in Proc. Interspeech
2019, 2019, pp. 2613–2617.
[22] Long Ouyang, Jeffrey Wu, Xu Jiang, Almeida, et al.,
“Training language models to follow instructions with [31] Shinji Watanabe, Takaaki Hori, Shigeki Karita,
human feedback,” in Advances in Neural Information Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson-
Processing Systems, S. Koyejo, S. Mohamed, A. Agar- Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner,
wal, D. Belgrave, K. Cho, and A. Oh, Eds. 2022, vol. 35, Nanxin Chen, et al., “ESPnet: End-to-End Speech Pro-
pp. 27730–27744, Curran Associates, Inc. cessing Toolkit,” in Proc. Interspeech 2018, 2018, pp.
2207–2211.
[23] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang,
Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, [32] Vassil Panayotov, Guoguo Chen, Daniel Povey, and San-
Mengshen He, Zhengliang Liu, et al., “Summary jeev Khudanpur, “LibriSpeech: an ASR corpus based
of ChatGPT/GPT-4 research and perspective towards on public domain audio books,” in Proc. 2015 IEEE In-
the future of large language models,” arXiv preprint ternational Conference on Acoustics, Speech and Signal
arXiv:2304.01852, 2023. Processing (ICASSP). IEEE, 2015, pp. 5206–5210.

[24] Zheheng Luo, Qianqian Xie, and Sophia Anani- [33] François Hernandez, Vincent Nguyen, Sahar Ghannay,
adou, “ChatGPT as a factual inconsistency evalua- Natalia Tomashenko, and Yannick Esteve, “TED-LIUM
tor for abstractive text summarization,” arXiv preprint 3: Twice as much data and corpus repartition for ex-
arXiv:2303.15621, 2023. periments on speaker adaptation,” in Speech and Com-
puter: 20th International Conference, SPECOM 2018,
[25] Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Leipzig, Germany, September 18–22, 2018, Proceedings
Jiao, and Michael Lyu, “ChatGPT or Grammarly? eval- 20. Springer, 2018, pp. 198–208.
uating ChatGPT on grammatical error correction bench-
mark,” arXiv preprint arXiv:2303.13648, 2023. [34] Josh Meyer, Lindy Rauchenstein, Joshua D Eisenberg,
and Nicholas Howell, “Artie Bias Corpus: An open
[26] Tao Fang, Shu Yang, Kaixin Lan, Derek F Wong, Jin- dataset for detecting demographic bias in speech appli-
peng Hu, Lidia S Chao, and Yue Zhang, “Is Chat- cations,” in Proceedings of the Twelfth Language Re-
GPT a highly fluent grammatical error correction sys- sources and Evaluation Conference, 2020, pp. 6462–
tem? a comprehensive evaluation,” arXiv preprint 6468.
arXiv:2304.01746, 2023.
[35] Rosana Ardila, Megan Branson, Kelly Davis, Michael
[27] Xinyue Liu, Mingda Li, Luoxin Chen, Prashan Wani- Kohler, Josh Meyer, Michael Henretty, Reuben Morais,
gasekara, Weitong Ruan, Haidar Khan, Wael Hamza, Lindsay Saunders, Francis Tyers, and Gregor Weber,
and Chengwei Su, “ASR N-best fusion nets,” in ICASSP “Common Voice: A Massively-Multilingual Speech
2021-2021 IEEE International Conference on Acous- Corpus,” in Proceedings of the Twelfth Language Re-
tics, Speech and Signal Processing (ICASSP). IEEE, sources and Evaluation Conference, 2020, pp. 4218–
2021, pp. 7618–7622. 4222.
[28] Karthik Ganesan, Pakhi Bamdev, B Jaivarsan, Amresh [36] Jonathan G Fiscus, “A post-processing system to yield
Venugopal, and Abhinav Tushar, “N-best ASR trans- reduced word error rates: Recognizer output voting er-
former: Enhancing SLU performance using multiple ror reduction (ROVER),” in 1997 IEEE Workshop on
ASR hypotheses,” in Proc. 59th Annual Meeting of Automatic Speech Recognition and Understanding Pro-
the Association for Computational Linguistics and the ceedings. IEEE, 1997, pp. 347–354.
11th International Joint Conference on Natural Lan-
guage Processing (Volume 2: Short Papers), 2021, pp.
93–98.