2020 International Electronics Symposium (IES)

Developing English Conversation Chatbot Using

Dialogflow

Aliv Faizal Muhammad Dwi Susanto Akhmad Alimudin

Surabaya, Indonesia Surabaya, Indonesia Surabaya, Indonesia

aliv@[Link] dwi@[Link] alioke@[Link]

Farah Adila Moh. Hasbi Assidiqi Salim Nabhan

Surabaya, Indonesia Surabaya, Indonesia Surabaya, Indonesia

adila1198@[Link] hasbi@[Link] salimnabhan@[Link]

Abstract—Despite the fact that the use of English constantly enforced this method [3]. Besides, students also
language is inevitable nowadays, some people find it prefer to do "silent practice" in reading conversation texts
difficult to learn. Moreover, the advance of technology has while learning speak English skills and some just memorized
brought impacts on the way how people learn the language such them. In addition to this, as today’s learners are typically
as speech recognition technology that is used to develop learning digital natives, the nature of students’ learning languages
tools. However, some existing learning media have several have shifted from traditional way toward digital one through
shortcomings. Therefore, to bridge this gap, such media to learn
technology integration in their literacy practices [4].
and practice the language are needed. This paper focuses on
developing English conversation chatbot using speech Involving multimodalities such as audio and visual by
recognition and artificial intelligence technology with involving technology in teaching and learning English can be
Dialogflow platform as the artificial intelligence engine. beneficial. These new perspectives support the use of
Evaluations toward the chatbot were done by the expert review technology in teaching and learning English language.
to identify the achievement of indicators and the users to know
the level of responses accuracy. The result indicated that almost Concerning the using of web-based speech
all agents have reached all indicators and most of the responses recognition technology, Dio [5] developed learning media to
had accuracy rate of 100%. It is expected that the availability of practice English conversation skills. Web-based application
the chatbot helps students to improve their conversation skills.
using speech recognition was also developed as medium to
Keywords—Learning Media, Chatbot, Artificial Intelligence practice speaking [6]. Students can then learn their English
skills in particular digital environment which can be done
I. INTRODUCTION wherever and anytime. However, the learning media have
English is a language widely used to connect and some shortcomings such as limited vocabulary and
engage with citizens worldwide. English is used in almost all pronouncing the words that appear on the website. Therefore
fields in the modern age as it is today, including technology, learning language using media is needed such as using speech
schooling, tourism, industry, politics, trade, among others. recognition technology namely chatbot technology.
And the ability to talk English is very essential in the growth
of this age of globalization while confronted with intense Chatbot is a conversational agent which interacts
competition. However, some people get difficulty in learning with users using natural language [7]. The chatbot is made
English. Among the four English skills, students claim that with the Natural Language Processing (NLP) system which
the ability to speak English is more challenging than the other is a branch of artificial intelligence. Therefore, the authors
three English skills, writing, listening, and reading [1] develop English Conversation Chatbot by using the
Dialogflow platform as the artificial intelligence engine. It is
The ability to speak English has become the issue in hoped that this chatbot can enhance students’ English
the language education. Nevertheless, Heriansyah [2] noticed speaking skills and facilitate them to have digital experiences
that the problems that often existed during the teaching and in their conversation.
learning process when speaking English were 1) students II. THEORITICAL BASIS
often had no ideas of what to say, so they appeared to be
silent, 2) they were often nervous and awkward, not relaxed A. Dialogflow
when making mistakes, 3) students are afraid of making Dialogflow is a platform for natural language
mistakes, as their classmates will laugh at them. understanding that facilitates the design and integration of
conversational user interfaces into mobile applications, web
Besides, teachers often apply direct conversation applications, devices, bots, interactive voice response
between two or more people and are text-books oriented. The systems, etc. By using Dialogflow, it can provide users with
practices included doing the text conversations and reciting a new and interesting way to interact with the products they
them directly. As a result, students got bored since teachers

978-1-7281-9530-8/20/$31.00 ©2020 IEEE 468

Authorized licensed use limited to: Carleton University. Downloaded on November 03,2020 at 00:50:36 UTC from IEEE Xplore. Restrictions apply.
produce. Dialogflow can evaluate specific forms of consumer
data, including text or audio samples (e.g. from mobile or
voice recording). It can also respond to customers in a variety
of ways, either by text or by synthetic speech [8]
B. Branching Story

Fig.1. Example of story graph Fig.2. Modern English Conversation for Tourist Cover
A branching story structure is a story graph -a This book contains English conversations
directed graph of arc- connected nodes reflecting user containing situational conversations. The book can help
choices. Every potential route through the graph represents a improve English skills especially for tourists who want to
story to tell the user. The user's sense of control over the travel between countries. It presents problems in the field of
story's progress is constrained in the branching story graph by tourism, how to speak words, as well as interesting
the number of arcs along a specific direction. Figure 1 gives information and stories that use vocabulary that has been
an example of a story graph. The system commences non- adapted to the current conditions of society [11].
interactively with system-controlled characters performing F. Precision, Recall, and Accuracy
1 and 2 actions. The user then decides to perform either the
1 or the 2 operation. If 1 is selected, system-controlled
characters perform 3, 4, and 5 actions. In the absence of
any user action, a -transition is taken [9].
C. Chatbot
Chatbot is a computer program designed to
communicate with humans through text messages or speech.
Usually, chatbot is also equipped with artificial intelligence
and natural language processing which makes it an intelligent
computer program that can answer human-specific questions.
Chatbot is built on the basis of subjects that have been
modeled on the knowledge base. Many current chatbots are Fig.3. Machine Learning Evaluation Metrics
based on topics and issues that somebody would like to solve Classification specifies cases as either positive or
for personal or business purposes [7]. negative. Classification decisions may be reflected in a
system known as an uncertainty matrix or a contingency
D. Conversation Design table. The confusion matrix has four categories: True Positive
Conversation Design is a language-based design in (TP) is a correctly labeled positive example. False Positive
human conversation (similar to how object design on pen and (FP) refers to an example that is mistakenly classified as
paper is a language-based design). The more interfaces an positive. Real Negative (TN) is the negative which is
ordinary human interaction has to teach fewer users how to correctly defined as negative. Finally, False Negative (FN)
use it. This is a fusion of many disciplines of design including refers to positive, incorrectly labeled negative examples [12].
voice user interface design, interaction design, visual design,
motion design, audio design, and UX writing. Conversation The Confusion Matrix is shown in Figure 3(a).
Design's role as an architect is to map what users can do in Confusion matrix can be used to create points in ROC space
space while taking into account user needs and technological or in the PR space. The journal determines the metrics used
constraints. In detailed design specifications, they organize in each space given the uncertainty of the matrix, as shown in
conversations, decide the flow and underlying logic which Figure 3(b). One plot is False Positive Rate (FPR) on the x-
reflect complete user experience. They are collaborators with axis in the ROC space, and True Positive Rate ( TPR) on the
stakeholders and developers who can turn to develop and y-axis. FPR measures a small portion of negative examples
bringing to life experience [10]. that are mistakenly considered positive. TPR analyses a small
portion of the positive samples which are labeled correctly.
E. Designing Conversation Storyline
One recall plot in the PR space is on the x-axis, and the y-axis
In doing this research, it was found reference related is precision [12].
to the design of the conversation storyline in this chatbot. A definition for each metric is given in Figure 3(b).
This conversation storyline uses situational conversation, The journal deals with metrics as functions that act on the
which is a type of conversation that is based on certain underlying confusion matrix, which defines points in the
situations given. The following is a reference related to the ROC or PR space [12].
work of this research.

469

Authorized licensed use limited to: Carleton University. Downloaded on November 03,2020 at 00:50:36 UTC from IEEE Xplore. Restrictions apply.
 III. METHODOLOGY Scene Character &
Background Scene
A. Study Existing
Character: Waitress
Existing studies carried out by looking for similar Getting to know someone Background scene: In
application references. Then obtained the advantages and at Merlion Park front of Merlion Park
disadvantages of the application. As a result, we found Character : Other
English conversation training applications, Speechace, passenger
Duolingo, and Eviebot. Buy Souvenirs at Bugis Background scene :
B. Making a Conversation Storyline Street Bugis Street
In making this conversation storyline, first is Character: Seller
determining the overall theme of the conversation, then
determining each of its scenes and the setting of the place as C. Branching Storyline
well as character to illustrate each scene. After that it is made
After creating a conversation storyline, the next step
a sample conversation with the expected response taken
is sketching the storyline branching route. The goal is that to
based on the reference sources described earlier.
anticipate when the user enters input answers that are not in
TABLE I. CONVERSATION STORYLINE accordance with the expected answers that have been made.
This branching storyline was created with an open-source
Scene Character & tool, Twinery. The following is one sample branching
Background Scene storyline shown in the following image.
Making a passport at the Background scene :
Immigration Office Immigration office
Character : Officer
Exchange currency to Background scene :
Singapore Dollar Foreign Exchange
Bureau
Character : Officer
On the way to the airport Background scene : Taxi
by taxi Passenger seat
Character : Driver
Boarding Passes at The Background scene :
Airport Boarding pass gate
Character : Boarding
Pass Officer Fig.4. Branching Storyline Sample
Buy an EZ link card to use Background scene : D. Making an Agent
the MRT in Singapore Passenger service
Character : Passenger
Service Officer
Asking about the direction Background scene :
of the MRT line in Airport
Singapore Character : Other
passenger

Check in Hotel Background scene :

Receptionist
Character : Receptionist
Officer Fig.5. Dialogflow Console Agent
Getting Food Background scene :
The first step that must be done before creating a
Recommendations in Inside MRT
chatbot in Dialogflow is to create an agent first. Agent is a
Singapore Character : Other natural language understanding module that will later train
passenger and understand human language, so the conversation becomes
Exploring Lor Mee at Xin Background scene : Old more natural. This agent will be created as many as 12
Mei Xiang stall Airport Road Food according to the scene in the storyline that has been made.
Centre (in front of Xin
Mei Xiang stall)
Character: Waitress
Exploring Castella Cake at Background scene : Old
the Chong Pang Old-time Airport Road Food
Flavor stall Centre (in front of Chong
Pang Old-time Flavor
stall)

470

Authorized licensed use limited to: Carleton University. Downloaded on November 03,2020 at 00:50:36 UTC from IEEE Xplore. Restrictions apply.
E. Making an Intent The figure shows the creation of an entity in an agent
Dialogflow. The creation of this entity aims to identify and
extract data spoken by the user. The entities menu functions
to create your own entities that do not yet exist in system
entities. In system entities, there is already automatic data
extraction in the form of time, date, place, weather, country,
etc.
H. Making Context

Fig.6. Intents on Dialogflow Console

Fig.9. Context on Dialogflow Console
The figure shows the making intent of a console
dialogflow agent. In intent, there is a collection of several The figure shows the context in a console
questions from users that contain the same topic. If the user
Dialogflow agent. Context is a tool used to regulate
sends a question, the agent will match the question that entered
conversation flow so that conversations become structured
the data with the intents. Then, the agent will send the correct
answer. If a question is sent that does not match or does not and ordered. In this section, the developer determines his own
match the intent that was made, then the agent will send an name in the contexts section. There are 2 types of context,
answer from the fallback intent. namely: input context and output context.

F. Intent Parameter

Fig.7. Intent Parameter

The figure shows the intent parameters in a console Fig.10. Context Flow
dialogflow agent. Inside there are parameters, entities, and
values. These parameters must be checked in the required In the figure shows an example of the flow of
section. For example, if the user enters input data that does conversation that will occur. The user first says "Hello", then,
not contain dates or numbers as shown in the image this agent will match the response with the intent welcome.
(according to entities that have been checked), then this The intent has a make passport output context, so the context
system will ask again the time of departure according to the is active. Then the agent asks "How can I help you?". Users
conversation topic. Meanwhile, Entity is a tool to identify and respond with "I want to make a passport." Then the agent will
extract data spoken by users, such as time, date, place, match the response with the intent make passport. The Intent
weather, etc. In addition, Entity can also be customized has a make_passport input context. Then, the agent responds
according to the conversation topics needed in the entities as shown. The role of context here, connecting various
menu. intents. Thus, branched conversations will become structured
G. Making an Entity and ordered.

Fig.8. Entities on Dialogflow Console

471

Authorized licensed use limited to: Carleton University. Downloaded on November 03,2020 at 00:50:36 UTC from IEEE Xplore. Restrictions apply.
I. Test Console intent that has been made. Then, the analysis will be carried
out with a precision, recall, and accuracy approach. So we get
the accuracy or suitability of the agent in answering the
response from the user.

IV. TESTING
The test is carried out to the expert review and users.
The component is tested by expert review to know whether
there are indicators of the achievement of someone who can
speak English in the chatbot conversation content that has
been made. Meanwhile, the component tested by the user is
the level of accuracy or suitability of the agent in responding
Fig.11. Test Console on Dialogflow Console to sentences submitted by the user.
A. Testing of Expert Review
After compiling the conversation flow through
context, it is then simulated through the test console located In this part of the test, an expert review is carried out
on the side of the dialog flow console. The aim is to ensure through Google Form, which contains indicators of the
that the conversations in the chatbot are structured. achievement of someone in speaking English as follows:
1. Greetings
J. Web Demo 2. Welcoming
3. Introduction
4. Asking and Giving Information
5. Congratulations
6. Apologizing and Making Excuses
7. Giving Advice or Suggestion
8. Giving Order or Instruction
9. Expressing Requests & Prohibitions
10. Expressing Agreements & Disagreements
11. Thanking and Accepting Thanks
12. Partings

These indicators are classified for each agent,

Fig. 12. Web Demo whether the agent contains the indicators mentioned. Review
testing was conducted by the two experts in the field of
In the figure is a web demo located on the English education. Furthermore, the test results can be seen
integration menu in the Dialogflow console. This web demo in the Appendix. It can be concluded:
will be used as a simulator or platform to be tested on users
so they can interact with chatbots that have been made. The
Agent Immigration-Office
testing mechanism is the user will get as many as 12 web
The agent has reached almost all indicators except for
demo links. As previously explained, there will be 12 agents
indicators congratulations.
according to the number of scenes. So, each agent has a
Money-Changer Agent
different demo web menu.
For these agents, indicators that have not yet been reached are
K. Conversation History introduction, congratulations, and partings.
Go-to-the-airport agent
For these agents, indicators that have not yet been reached are
welcoming, introduction, congratulations, apologizing and
making excuses, and partings.
Agent Boarding-Passes-at-The-Airport
For these agents, indicators that have not yet been reached are
welcoming, introduction, congratulations, apologizing and
making excuses, and partings.
Agent Getting-EZ-Link-Card
Fig. 13. History Menu For these agents, indicators that have not yet been reached are
welcoming, congratulations, giving advice or suggestions,
The figure shows that after the chatbot is tested on and giving orders or instructions.
the user, the conversation will be recorded while the user Agent Asking-MRT-Line
interacts. On the console dialog flow history menu, there is a For these agents, indicators that have not yet been reached are
history of data conversations between users and agents such welcoming, introduction, and congratulations.
as date, time, platform type, number of interactions, to Agent Check-In-Hotel
display conversations that are not in accordance with the

472

Authorized licensed use limited to: Carleton University. Downloaded on November 03,2020 at 00:50:36 UTC from IEEE Xplore. Restrictions apply.
For these agents, almost all indicators are achieved except Agent Foreign-Exchange-Bureau Test Results
indicators congratulations.
Agent Getting-Food-Recommendations
For these agents, indicators that have not yet been achieved
are introduction and congratulations.
Agent Exploring-Food-at-Singapore
For these agents, indicators that have not yet been achieved
are introduction and congratulations.
Agent Exploring-Food-at-Singapore-Part-II
For these agents, almost all indicators are achieved except
indicators and congratulations.
Going-Vacation-to-Merlion-Park Agent Fig.15. Agent Foreign-Exchange-Bureau Accuracy
For these agents, indicators that have not yet been reached are
welcoming, congratulations, and giving advice or The figure explains that almost all responses to the
suggestions. Foreign-Exchange-Bureau Agent have 100% accuracy.
Agent Buy Souvenirs at Bugis Street Except for the "Yes" and "10 Million Rupiah" responses. In
For these agents, indicators that have not yet been achieved this response get an accuracy rate of 0%.
are indicators of congratulations, thanking and accepting
thanks, and partings. Agent Go-to-The-Airport Test Results
The results of testing by an expert review can be
concluded that almost all agents have reached all the
indicators someone can speak English. However, it is not
with congratulations indicators. In the future, it can be used
as a suggestion to create and develop new scenes with
indicators of congratulations.
B. Testing of Users
As explained earlier, in this part of the test,
approaches precision, recall, and accuracy are used. So that Fig.16. Agent Go-to-The-Airport Accuracy
the accuracy or suitability of the agent is obtained in response
to sentences submitted by users. The mechanism of this test The figure explains that there are 4 responses on Go-
is that the user try to interact with the agent through a demo to-The-Airport Agents that have an accuracy rate of 100%. In
or demo web platform located in the Dialogflow integration addition, the responses "Hello Taxi", "Please take me to the
console section. Then, during interaction, the conversation is airport", "I will go to Singapore", and "Here is my money"
be recorded in the history menu section. Next, the data in the get 94.4%, 79%, 58%, and 45.4%.
history menu is analyzed.
Agent Boarding-Passes-at-The-Airport Test Results
Agent Immigration-Office Test Results

Fig.17. Agent Boarding-Passes-at-The-Airport Accuracy

Fig.14. Agent Immigration-Office Accuracy The figure explains that almost all responses on
Agent Boarding-Passes-at-The-Airport have a 100%
The figure explains that almost all responses to accuracy rate. Except for the "Yes" and "Number of bags"
Agent Immigration-Office have an accuracy rate of 100%. responses. In this response, the accuracy rate was 50% and
Except for the "Greetings" and "I want to make a passport" 66.6%, respectively.
response. Each gets an accuracy rate of 96.9% and 94.7%.

473

Authorized licensed use limited to: Carleton University. Downloaded on November 03,2020 at 00:50:36 UTC from IEEE Xplore. Restrictions apply.
 Agent Getting-EZ-Link-Card Test Results The figure explains that almost all responses to the
Getting-Food-Recommendations Agent have an accuracy
rate of 0%. Except for the responses "Greetings" and "Please
Recommend me delicious food" which gets an accuracy rate
of 90%.

Agent Exploring-Food-at-Singapore Test Results

Fig.18. Agent Getting-EZ-Link-Card Accuracy

The figure explains that almost all responses on

Agent Getting-EZ-Link-Card have a 100% accuracy rate.
Except for the responses "I would like to buy EZ Link Cards"
and "Sure". In this response, the accuracy rate was 89% and
80%, respectively. Fig. 22. Agent Exploring-Food-at-Singapore Accuracy

Agent Asking-about-MRT-Line Test Results The figure explains that all responses to Agent
Exploring-Food-at-Singapore have a 100% accuracy rate.

Agent Exploring-Food-at-Singapore-Part-II Test

Results

Fig.19. Agent Asking-about-MRT-Line Accuracy

The figure explains that almost all responses to the

Asking-about-MRT-Line Agent have an accuracy rate of
100%. Except for the "Greetings" response, which gets an Fig. 23. Agent Exploring-Food-at-Singapore-Part-II
accuracy rate of 87.50%. Accuracy

Agent Check-In Hotel Test Results The figure explains that all responses to Agent
Exploring-Food-at-Singapore have a 100% accuracy rate.

Agent Going-Vacation-to-Merlion-Park Test

Results

Fig. 20. Agent Check-In Hotel Accuracy

The figure explains that all responses to Agent

Check-In-Hotel have a 100% accuracy rate.

Agent Getting-Food-Recommendations Test Fig. 24. Agent Going-Vaction-to-Merlion-Park Accuracy

Results
The figure explains that almost all responses in
Agent Going-Vacation-to-Merlion-Park have an accuracy
rate of 100%. Except for the response "I'm from Indonesia"
which gets an accuracy rate of 17%.

Fig. 21. Agent Getting-Food-Recommendations Accuracy

474

Authorized licensed use limited to: Carleton University. Downloaded on November 03,2020 at 00:50:36 UTC from IEEE Xplore. Restrictions apply.
 Agent Buy-Souvenirs-at-Bugis-Street Test Results [6] A. F. Muhammad, D. E. Pratama, and A. Alimudin,
“Development of Web Based Application with
Speech Recognition As English Learning
Conversation Training Media,” in IES 2019 -
International Electronics Symposium: The Role of
Techno-Intelligence in Creating an Open Energy
System Towards Energy Democracy, Proceedings,
2019, pp. 571–576, doi:
10.1109/ELECSYM.2019.8901594.
[7] Accenture Interactive, “Chatbots in customer
Fig.25. Agent Buy-Souvenirs-at-Bugis-Street Accuracy
service,” 2016.
[8] “Dialogflow basics,” Google Claud. [Online].
The figure explains that all responses to Agent Buy-
Available:
Souvenirs-at-Bugis-Street have a 100% accuracy rate.
[Link]
[Accessed: 02-Jul-2020].
V. CONCLUSION [9] M. O. Riedl and R. M. Young, “From linear story
generation to branching story graphs,” in
The English conversation chatbot using speech Proceedings of the 1st Artificial Intelligence and
recognition and artificial intelligence technology with Interactive Digital Entertainment Conference, AIIDE
Dialogflow platform as the artificial intelligence engine has 2005, 2005, pp. 111–116.
been developed and evaluated by the expert review and the [10] “Conversation design–What is conversation
users. From the results of this research, it can be identified design?,” [Link].
that through storyline design and branching conversations, [Online]. Available:
making chatbots, and testing, the level of accuracy of the [Link]
response of an agent increase if the variation of the n/conversation-design/what-is-conversation-
vocabulary data set is increased, and the conversation [Link]#. [Accessed: 02-Jul-2020].
branching path is expanded more. This media can be utilized [11] G. Maba, Modern English Conversation for Tourist.
by language learners to practice English conversation. Jombang: Lintas Media, 2000.
Besides, there is still a space to develop further this [12] L. A. Staeheli and D. Mitchell, “The relationship
application. between precision-recall and ROC Curves,” in
Proceedings of the 23rd International Con- tion 2
REFERENCES defines Precision and Recall for the reader un-
ference on Machine Learning, 2006, pp. 546–559,
[1] Y. Bani Ahmad, “Student Perceptions on English doi: 10.4135/9780857021113.n29.
pronunciation after course phonetics and
phonology,” Judika (Jurnal Pendidik. Unsika), vol.
6, no. 1, pp. 12–16, 2018.
[2] G. Amaral et al., “Speaking problems faced by the
English department students of Syiah Kuala
University,” Ling. Didakt., vol. 6, no. 1, pp. 37–44,
2012, doi: 10.1017/CBO9781107415324.004.
[3] E. A. Amalo, I. D. Agusalim, and C. D.
Murdaningtyas, “Developing visual novel game with
speech-recognition interactivity to enhance students’
mastery on English expressions,” J. Sos. Hum., vol.
10, no. 2, pp. 129–130, 2017, doi:
10.12962/j24433527.v10i2.2865.
[4] S. Nabhan and R. Hidayat, “Investigating literacy
practices in a university EFL context from
multiliteracies and multimodal perspective : A case
study,” Adv. Lang. Lit. Stud., vol. 9, no. 6, pp. 192–
199, 2018, doi:
[Link]
[5] D. E. Pratama, “Pengembangan aplikasi berbasis web
dengan speech recognition sebagai media
pembelajaran latihan conversation bahasa Inggris,”
Politeknik Elektronika Negeri Surabaya, 2019.
[Online]. Available:
[Link]
of-web-based-application-with-speech-recognition-
as-english-learning-conversation-training-media/.
[Accessed: 01-Jul-2020].

475

Authorized licensed use limited to: Carleton University. Downloaded on November 03,2020 at 00:50:36 UTC from IEEE Xplore. Restrictions apply.