Literature Study: Effectiveness Of Automatic Speech Recognition In

Language Acquisition For Deaf Students

Toni Muhammad Rizal, Asri Widiatsih*, Partiwi Ngayuningtyas Adi, Syamsul Muarif
Corresponding Author: asriwidiatsih@[Link]
Universitas PGRI Argopuro Jember, Indonesia

ABSTRACT

This research aims to describe the effectiveness of Automatic Speech Recognition

as an assistive technology in language acquisition for deaf students. Deaf people
have lost their ability to hear, so they hinder the processing of language information
through their hearing, whether wearing or not hearing aids where their hearing
limit is sufficient to enable the successful processing of language information
through hearing. This study uses the literature study method by reviewing and
critically examining the existing knowledge, ideas and findings in academic-
oriented literature and formulating theoretical and methodological contributions
covering related topics. Data search was done by searching nationally and
internationally accredited journals on digital platforms such as Google Scholar,
SINTA and GARUDA. The data obtained is sorted based on the criteria for further
analysis using descriptive qualitative techniques. The use of ASR shows various
results, and it can be concluded that it is effective in language acquisition for deaf
students. Effectiveness occurs when paying attention to several aspects, including
the applications used, the level of room noise, the delivery of the teacher's
sentences, and the vocabulary understood by deaf students.

Keywords: Deaf, Automatic Speech Recognition

INTRODUCTION
Limited mastery of language is the main problem of communication for deaf children,
and this is because hearing barriers have a direct impact on language development
(Purbaningrum & Rofiah, 2018). Lederberg et al. (2013) argue that language has long been
considered a key area affected by deaf barriers. Language is an aspect that is at the
fundamental center because language is indispensable for daily communication. Through
language, individuals can express their ideas, thoughts and experiences to others.
Language is a means of communication used by all people in everyday life expressively and
receptively, namely understanding speech.
Deafness is a person who loses the hearing organ's function, inhibiting the hearing
instrument's language information process, either wearing or not wearing a hearing aid
where the hearing limit is sufficient to allow the success of the language information process
through the hearing organ. Deaf children are at risk of experiencing language barriers to
receptive language and expressive language. Children with receptive language barriers or
disorders have difficulty understanding the information they receive. In most cases,
children with receptive language disorders also experience expressive language disorders,
which means they struggle to use verbal or spoken language. According to Mustika (2017),
receptive language is a person's ability to recognize and react to someone's speech, to the

64
events of the surrounding environment, understand the meaning of mimicry and tone of
voice and finally, understand words. Due to hearing barriers, deaf children cannot
understand words through hearing, so deaf children use a variety of alternative
communication to understand speech, especially following learning. The success of deaf
students in following and actively participating in learning is highly dependent on the
receptive language ability of listening.
As a result of receptive language barriers, especially listening, it can have an impact
on other aspects of development, on the emotional aspect of children having a feeling of fear
because they do not understand the conversation, so that it also has an impact on the social
aspect where children are not easy to get along and only depend on people who understand
the language, in the cognitive domain the intellectual level of children is the same as hearing
children, communication problems result in the intelligence of deaf children lacking the
opportunity to develop like hearing children, (Wagino & Nehrulita, 2016).
In listening, deaf children often rely on the sense of sight. Therefore, deaf children are
known as visual children. It is important to find communication solutions for utilizing the
sensory organs that are still functioning optimally so that communication media that utilize
visuals become a separate solution for deaf children. Cawthon & Leppo (2013) conveyed
that children with hearing loss characteristics communicate through spoken language
(mouthing) by utilizing assistance from hearing aids, text, and reading the lips of the
interlocutor. Children with deaf characteristics use sign language as an alternative
communication.
Kawas et al. (2016) conveyed that hearing aids and lip reading are most effective in
face-to-face communication and are carried out with a small number of people or small
groups. Meanwhile, the use of hearing aids and lip reading will be less effective if there are
obstacles or distractions in the way, such as teaching and learning activities in the
classroom where the teacher's position is too far away to be seen clearly or when the high
level of room noise will hinder the successful use of hearing aids. Kawas et al. (2016) also
found that communication with sign language is difficult because not everyone can
understand and communicate with sign language. Another common option is to use the
services of a sign language interpreter or sign language interpreter for children's
communication in inclusive classrooms. Still, generally, the interpreter cannot fully
understand the material the teacher conveys and will have difficulty translating the
language conveyed into full sign language. Another obstacle to sign language interpreter
services is that the child's focus will be divided into many activities, such as paying
attention to the activities of the teacher who is teaching and paying attention to the sign
language of the interpreter so that activities such as recording material can be missed, in
this condition, writing becomes a necessary need to communicate for deaf children.
Automatic Speech Recognition (ASR) assistive technology can be an alternative to help deaf
students communicate (Glasser et al., 2017).
Automatic Speech Recognition is a technology that automatically recognizes sounds
and converts them into writing so children can get information or listen through visuals.
The advantage of this technology is that the costs required are relatively affordable
compared to sign language interpreter services (Kawas et al., 2016). Bell Labs first
developed automatic speech recognition (ASR) in the 1950s. Early speech recognition
applications targeted operating computers by voice, interacting with services over the phone
on voice menus, and spoken language that produced documents through speech rather
than using a keyboard. Users with deafblindness could have a phone with a text screen. A

65
third-party intermediary will read back the hearing party's conversation into speech
recognition trained for the intermediary. Automatic Speech Recognition (ASR) is starting to
be designed as a solution by researchers as a tool for speech recognition. The tool is a tool
to capture speech into written form that can be helpful for deaf children in listening to
classroom learning.
Seita & Huenerfauth (2020), utilizing Automatic Speech Recognition (ASR) placed on
the desk to help learners with hearing impairments communicate in class, concluded that
Google Live Transcribe, which has now changed its name to instant transcription, can be a
voice recognition assistive technology solution that improves receptive language skills in
listening. In research conducted by Loizides et al. (2020), it was found that Automatic
Speech Recognition (ASR) was very helpful for children with deaf barriers during the COVID-
19 pandemic, as WHO (2020) issued a social distancing policy and mandatory wearing of
masks. This makes students with deaf barriers look for ways to communicate; children with
deaf barriers who rely on lip-reading communication can be sure of difficulties due to a
friend's face communicating while covered by a mask. The same thing can happen if the
communicating friend keeps a distance of two meters, causing lip movements or speech to
be invisible. Automatic Speech Recognition (ASR) is also a visual aid recommended by
educational institutions during the pandemic because this assistive technology can be used
at a distance using a wireless microphone connected to a smartphone. Automatic Speech
Recognition (ASR) can be included in the Individualized Learning Program (IEP) as a special
technology service needed to meet the needs and educational goals of children in inclusive
classrooms (Wolfe et al., 2020). This aligns with Budiyanto's (2017) opinion, which states
that deafblind children can access services according to their conditions. The
communication media that deaf children use are not too demanding on the use of their
hearing devices; students with deaf barriers have the right to use communication media
that are considered appropriate based on the conditions experienced by them, such as
visual access to education (WFD, 2016).
Braun et al. (2018) have researched the use of ASR in college-level inclusion classes.
Automatic Speech Recognition is used within one semester and gets positive results;
students can record material independently and are more focused on paying attention to
the teacher during learning. In the first semester of 2017, students with hearing
impairments relied on lip reading in listening to the teaching and learning process.
Classmates assist them in writing down the explanation delivered by the teacher as well as
the point of the PowerPoint delivered; assistance from sign language interpreters is brought
in to help deliver material, and class design arrangements are conditioned so that deaf
students can maximally understand the explanation and writing of material on the board
during the learning process.
Entering the second semester of 2018, learners began using assistive technology
assistance called the Live Transcribe application; this application is now one type of
Automatic Speech Recognition (ASR) developed by a well-known company, Google and can
be used on smartphones; this assistive technology assistance is used because learners want
to understand the teacher's words independently and not always depend on their friends,
moreover reading lips is not too optimal. The lack of sign language interpreter services is
also the basis for deaf learners to take the initiative to use ASR technology. From the above
observations, an overview of the results of the use of ASR in college-level inclusion classes,
it is found that the positive effect of Automatic Speech Recognition (ASR) on the listening

66
skills of deaf students is quite helpful because students can focus more on the teacher and
students can be more independent.
However, deaf learners sometimes feel dissatisfied with Automatic Speech Recognition
(ASR) assistive technology; some of the things that cause this dissatisfaction include the
accuracy of sentences that are less organized or messy, and often there are word errors.
This usually happens because there is more than one voice source, so the teacher's voice is
less recognized and mixed. Another problem is that this application transcribes raw from
the captured sound. At the same time, the ability of people who are deaf or hard of hearing
to master the definition of vocabulary is very limited. By understanding the effectiveness of
Automatic Speech Recognition (ASR), those responsible for educating children with
deafblindness can provide appropriate educational services. These parties will be able to
provide an educational service tailored to the conditions experienced and the needs of
children with diverse deaf barriers so that the language problems of children with
deafblindness can be addressed and improved. By addressing problems closely related to
receptive language listening, the language quality of deaf children will be better. They can
communicate, improving the interpersonal relationships of children with deafblindness.
This, of course, can reduce the gap in social interaction between deaf children and children
in general.
Based on this condition, all stakeholders must provide optimal education services for
deafblind children. This optimal education service can be created when all parties
understand the use and effectiveness of Automatic Speech Recognition (ASR). Therefore, a
study on the effectiveness of Automatic Speech Recognition (ASR) for deaf children in
language acquisition is needed. Moreover, this related research is still rarely done,
especially in Indonesia. By studying the role of Automatic Speech Recognition (ASR) for deaf
individuals, people will realize the importance of using assistive technology. Various
assumptions about deafness and communication will be answered through a good and
relevant literature study. That way, home, school and community education services for
children with deafness and blindness can run optimally.
Referring to the problems that have been described, the purpose of this study is to
describe the effectiveness of using Automatic Speech Recognition (ASR) assistive technology
in language acquisition for deaf learners.

METHOD
The method used in this research is a literature study by reviewing and critically
examining ideas, knowledge and or findings in academically oriented literature and
formulating theoretical and methodological contributions covering related topics. Literature
study as an activity summarizes articles in journals and or proceedings, books, and other
documents relevant to certain selected topics. According to Efron and Ravid (2019), a
literature study is a systematic method for examining the literature on a particular topic by
analyzing, evaluating, and reviewing various theories, research, and practices from previous
researchers. In this literature study research, researchers collected several articles from
accredited journal proceedings relevant to the effectiveness of automatic speech recognition
(ASR) in the language acquisition of deaf students. The procedure used in this literature
study was carried out in several ways, namely:

67
1. Search Procedure
The search was conducted by looking for various articles from accredited journals
and proceedings regarding Automatic Speech Recognition (ASR) in language acquisition
in deaf students. The search was conducted online on digital platforms such as SINTA,
GARUDA, and Google Scholar. The article search was conducted from November 8, 2022,
to December 12, 2022.
2. Selection Procedure
After searching for articles, the collected literature was selected based on the topic's
relevance. After being assessed as having abstracts with relevant content, the next step
is to read the entire source content to find related points that can be used as material
for analysis. The following are some of the criteria for selecting literature in this literature
study research:
a. Articles published in 2013 - 2021
b. Deaf-related research articles
c. Literature on the effectiveness of Automatic Speech Recognition (ASR) and the
application of ASR to language skills in deaf children and students.
3. Analysis Procedure
After finding relevant reading sources, they are analyzed and compared by stating
the author's name and year of publication. After analysis and comparison, the data is
presented in a scientific article.

RESULT AND DISCUSSION

Deaf children experience obstacles in language acquisition or receptive language; this
condition is caused because children with deaf barriers cannot listen to spoken language
communication through the sense of hearing. Listening is listening to spoken words or
sentences with understanding, attention, appreciation, and interpretation in obtaining a
message, capturing the information conveyed and understanding the meaning of the
communication. By listening, people can understand the message and the contents of the
mind of the person who delivered it. The impact of difficulty understanding receptive
language includes difficulty following directions, understanding what body movements
mean, answering questions, recognizing objects and images, understanding reading, and
understanding a story. Deaf children try to understand things through the visual sense by
diverting their observations only to the eyes, so deaf children are commonly called visual
children. Through the sense of sight, children with deafblindness understand the language
of lip movements or mouthing. In addition to seeing the facial expressions and movements
of the interlocutor, the deaf child's eyes are also used to read the interlocutor's lip (Butler
et al., 2019).
The listening skills of children with deafblindness can be assisted through intervention
using Automatic Speech Recognition (ASR) assistive technology. ASR is a service solution
in the form of an application for deaf children with limited benefits from communication
using sign language gestures, lip reading and hearing aids. ASR automatically captures
sounds and transcribes them into a text display speech-to-text) to be read. This is in
contrast to hearing aids, which amplify or process the sound source to be heard through
residual hearing. ASR utilizes the visuals of the child with a deafblindness barrier to acquire
information. According to Shadiev et al. (2017), one of the functions of ASR is to help
students listen to the teacher's explanation in class because the teacher's spoken speech

68
can be read through the transcript of the ASR application. The advantages of ASR
applications are that they are very affordable and easy to use. Based on the analysis
conducted from several sources, the results obtained are related to the effectiveness of ASR
on the receptive language acquisition of deaf children. Elliot et al. (2016) suggest that using
Ausingning can effectively help children with deaf barriers listen to learning so that
children's learning abilities do not differ much from students in general.
In 2022, researchers surveyed deaf students at the PGRI Argopuro University Jember
College, and the survey data analysis revealed that the transcription in the lecture class
produced by ASR can help students access information delivered by lecturers so that
students with deaf barriers can understand the content of lecture material better.
Constantinou et al. (2020) evaluated the use of ASR in the classroom by implementing a
questionnaire completed by deaf students. Students' responses to the questionnaire were
analyzed, and the results showed that 70% of the students intended to continue using ASR
in the classroom learning process. According to the students in the study, ASR technology
is a quite effective tool to help take notes and record explanations from the teacher.
Behm et al. (2015) found that only college-age students with receptive language skills
benefited from ASR services. High school-aged children prefer to read the text in the
textbook rather than review the transcript, and this is because students feel confused about
understanding the meaning of sentences.
Some other benefits of using Automatic Speech Recognition (ASR) applications in
learning can improve students' independence, motivation, and productivity. Some of the
improvements include: (1) independence, students do not have a dependence on classmates
to interpret the discussion delivered by the teacher; students are also able to do homework
independently by looking at the results of transcript notes; (2) motivation, Automatic
Speech Recognition can encourage students to continue learning and enthusiasm for
understanding sentences, through ASR assistive technology students can get new things
that have never been obtained before, competition in academics between students with deaf
barriers and students in general also occurs because deaf students are confident in getting
the delivery of the same material content; (3) productivity, increased learning motivation
makes students more diligent in doing the assignments given, children can rely on
transcript notes obtained previously, (Blanchard et al., 2011)., 2016; Glasser et al., 2017).
ASR assistive technology can be combined with other technologies; for example, one
study found a classroom effect by combining an ASR program with video. Learning videos
that do not have subtitles can be automatically transcribed with ASR so that if there is
narration, the teacher does not need to narrate the speech in the video. It also helps
students take online classes through video conference, zoom and Skype that do not have
automatic transcription; Voice Note voice in the task grub can also be transcribed using
ASR (Braun et al., 2018; Kawas et al., 2016).
Several researchers conducted audiological testing on several existing applications;
this was done to determine the Word Error Rate (WER). Pragt et al. (2020) conducted tests
using four applications, each tested on the iOS and Android operating systems. The
applications tested consisted of Live Transcribe AVA, Earfy, and Speech. This trial was
conducted using Dutch and English, with room conditions also set to the noise level. The
results were that the number of words transcribed with the correct category was higher in
English than in Dutch. Halim & Satria (2020) also tested this assistive technology using
three applications tested on Android Smart: Voicebook, Speech Notes and Text to Speech.
This trial was conducted using the Indonesian language, and the results obtained from the

69
best application that children with deafblindness can use is Voice Notebook. Therefore, it
can be concluded that the Voice Notebook application has a higher and more effective level.
Seita & Huenerfauth (2020) researched using sentences and good speaking behavior
towards using ASR technology; the results aim to guide teachers in determining effective
speech characteristics. Speaking characteristics that need to be considered include (1)
Speed when speaking, the intensity of the teacher's speed when speaking affects the results
of the sentences that appear; (2) Voice volume, the size of the voice affects the capture of
sound in the application; (3) Gestures, use hand and body movements when speaking to
emphasize the meaning of the sentences conveyed; (4) Pause time, when speaking must be
ascertained when to stop after each word and or sentence; (5) Sentence structure, the series
of words need to be considered so that the words spoken are more effective.
Shadiev et al. (2014) suggest that teachers can adapt to the recognition capabilities of
Automatic Speech Recognition (ASR) while speaking with moderate speed and volume
intensity and not speaking spontaneously and appropriate word selection. Kawas et al.
(2016) urged teachers to speak with clear intonation and avoid non-lexical utterances (such
as "hm," "umm," or "uh..."). The teacher's voice should be recorded correctly. If responding
to a question from a student, the teacher should repeat the question and then be able to
respond (Ranchal et al., 2013). To improve accuracy during communication, the researcher
suggests that teachers shorten the pronunciation of words or sentences at a relatively
moderate speed intensity from one word to another to find and correct transcription errors.
Ranchal et al. (2013) suggested that teachers should stop talking periodically and ensure
classroom conditions so students can clearly understand the sentences displayed by ASR.
These efforts are made to increase the effectiveness of using ASR assistive applications for
children with deafblindness to achieve learning objectives.
Regular use of ASR technology programs must be utilized to train language acquisition
skills for students with deafblindness. Behm et al. (2015) found that students' language
absorption ability improved after using ASR for more than one semester at school. This
happened because teachers became accustomed to the characteristics of voice capture by
ASR, and students could understand the sentences conveyed by the teacher on the
application. This shows that consistently using the Automatic Speech Recognition (ASR)
service program as assistive technology is very helpful in improving children's language
acquisition skills.
Not all studies explain the positive results of the use of ASR; several considerations
make teachers and schools not implement the utilization of this technology, including: (1).
Word accuracy is one of the common considerations due to punctuation or missing words
and spelling errors in sentences, this can affect the cognitive side of students; (2) Students'
language skills, children's early language recognition skills are an important factor in the
success of the ASR program, because students' limitations in recognizing words make
students not easily understand the meaning of the teacher's words.
According to Seita & Huenerfauth (2020), using sentences is very important for
language comprehension; deaf children cannot understand the content of complex or long
sentences that contain figurative words. The better the language acquisition level of the
student when using Automatic Speech Recognition (ASR) assistive technology, the easier it
is for him to learn to understand the material's content.

70
CONCLUSION
Automatic Speech Recognition (ASR) is effective in language acquisition for deaf
students. This technology can be used in many languages around the world. The receptive
language of deaf children is inhibited due to the impact of damage to the sense of hearing,
so children cannot receive verbal information. Therefore, it is necessary to find solutions
that can help children develop receptive language skills, especially listening; this can help
children in language acquisition so they can communicate well.
This literature study shows the benefits of using Automatic Speech Recognition as an
assistive technology. With ASR technology, deafblind children can listen to teacher
explanations through transcribed text. Some things that need to be considered to ensure
effectiveness are paying attention to several aspects, including the application used, the
room's noise level, the use of teacher sentences and vocabulary that deaf children
understand.

REFERENCES
Android (2020). Live Transcribe Speech to Text AppAndroid.
[Link]
Halim, C., & Satria, F. (2020). Analisa Komparasi Perangkat Speech Recognizing dan
Potensinya dalam Membantu Proses Pembelajaan Difabel Rungu Guna Terciptanya
Kampus Inklusif Di Era 4.0. Risenologi: Jurnal Sains, Teknologi, Sosial, Pendidikan
dan Bahasa, 5(1), 56–63. [Link]
Kawas, s., Karalis. G., Wen, T., & Ladner, R. R. E. (2016). Improving real-time captioning
experiences for deaf and hard-of-hearing students. ASSETS 2016 – Proceedings of
the 18th International ACM SIGACCESS Conference on Computers and
Accessibility, 15-23. [Link]
Mustika, a. (2017). Pembelajaran Bahasa Reseptif Anak Tunarungu Pada Usia Dini di
Sekolah Prima Bhakti Mulya. Inclusive: Journal of Special Education, III (01), 117-
121.
Purbaningrum, E., & Rofiah, K. (2018). The Impact of Language Skills Guidance on Children
with Hearing Impairment Language Development.212, 503-506.
[Link]
Seita, M., & Huenerfauth, M. (2020). Deaf individual's views on speaking behaviors of
hearing peers when using an automatic captioning app. Conference on Human
Factors in Computing Systems – Proceeding, 1-8.
[Link]
Shadiev, R., Huang, Y. M., & Hwang. J. P. (2017). Investigating the effectiveness of speech-
to-text recognition applications on learning performance, attention meditation.
Educational Technology Research Development, 65(5), 1239-1261.
[Link]
Wagino & Nehrulita, H. (2016). The Influence of Total Physical Response Method Toward
Understanding Vocabulary of Hearing Impairment Children in Special Need
Kindergarten Dharma Wanita. Proceedings of International Research clinic &
Scientific Publications of Educational technology, 2010, 775-781
WFD. (2016). Know and Achieve Your Human Rights Toolkit. Helsinki: World Federation of
the Deaf

71
WHO. (2020). Coronacirus diseases (COVID-19)
[Link]
Wolfe, J., Smith, J., Neumann. S., Miller, S., Schafer. E. C., Birath, A. L., Childress, T.,
McNally, C., McNiece. C., madell. J., Spangler. C., Caraway. T. H., & Jones, C.
(2020). Optimizing Communication in Schools and Other Settings During COVID-
19. The Hearing Journal, 73(9), 40-45.
[Link]

72