GMR Institute of Technology, Rajam

GMR Institute of Technology

Conversational Image Recognition Chatbot

Dec 5, 2024
1
emb
Dec
201

1
er
7

Dec 5, 2024 1
 Conversational Image Recognition Chatbot

[Link] (22341A4264)
[Link] kumar (22341A4263)
GMR Institute of Technology

[Link] (22341A4261)
[Link] prasad (22341A4262)
[Link] kumar (23345A4201)
[Link] (22341A4260)s

Under the guidance of

[Link]
Assistant Professor
GMR Institute of Technology
 Abstract

The Conversational Image Recognition Chatbot, developed for the Smart India Hackathon
(SIH), is an innovative tool designed to streamline user interaction with visual content through
natural language processing. This chatbot integrates advanced image recognition technologies
with a conversational interface, enabling users to query and obtain detailed information about
GMR Institute of Technology

images in real-time. The system leverages machine learning algorithms to accurately interpret and
analyze various image attributes, such as objects, text, and context, providing concise and relevant
descriptions. By combining image processing with a user-friendly dialogue system, this chatbot
enhances accessibility and efficiency in domains like education, e-commerce, and customer
service. The project aims to demonstrate the potential of AI-driven conversational systems in
simplifying complex tasks, promoting user engagement, and fostering intuitive interactions
between humans and machines. The chatbot represents a significant advancement in the field of
artificial intelligence, offering practical applications across multiple industries while pushing the
boundaries of human-computer interaction.

Keywords:
Conversational Image Recognition, Natural Language Processing (NLP),Machine Learning
Algorithms,Human-Computer Interaction,Artificial Intelligence (AI)
 Base Paper
Ex:Mahesh Babu, A., Jawarneh, M., Arias‐Gonzáles, J. L., Meenakshi, Kasat, K., & Yuvaraj, K.
P. (2024). Conversational Chatbot With Object Recognition Using Deep Learning and Machine
Learning. Conversational Artificial Intelligence, 335-352.
GMR Institute of Technology

Objectives
1. Real-Time Image Recognition

2. Accurate Image Analysis

3. Enhance Accessibility and Efficiency

4. Demonstrate AI Potential

5. Expand AI Applications Across Industries

 Introduction

Conversational image recognition is revolutionizing chatbots by integrating computer vision

techniques, enabling more dynamic and meaningful interactions. These chatbots go beyond the
limitations of text-based communication by processing and analyzing images through tasks like
GMR Institute of Technology

object detection, image classification, and scene understanding. By leveraging advanced

algorithms and machine learning models, chatbots can accurately identify objects, classify scenes,
and extract relevant information, creating a more intuitive and engaging user experience that
supports visual communication and delivers personalized responses.

This integration significantly broadens chatbot capabilities, allowing for applications such as
product identification, visual search, and image-based question answering. The technology’s
impact is seen across various industries: in customer service, it helps with product identification
and troubleshooting, while in e-commerce, it enables visual search for finding products based on
images. In education, conversational image recognition supports interactive learning by responding
to image-based queries, enriching the learning experience. However, challenges such as image
quality, computational complexity, and ethical concerns must be addressed to ensure the
responsible and efficient deployment of this transformative technology.
 Literature survey
Reference 1 :Mahesh Babu, A., Jawarneh, M., Arias‐Gonzáles, J. L., Meenakshi, Kasat,
K., & Yuvaraj, K. P. (2024). Conversational Chatbot With Object Recognition Using Deep
Learning and Machine Learning. Conversational Artificial Intelligence, 335-352.
The report outlines the development of a chatbot that integrates image recognition and natural
language processing using an encoder-decoder model. Trained on the COCO dataset, it employs
GMR Institute of Technology

Mask R-CNN for object detection and instance segmentation to answer questions about images.
The report also covers the model's architecture, experiments, and suggests potential improvements.

Reference 2 :Levy, M., Ben-Ari, R., Darshan, N., & Lischinski, D. (2024).
Chatting makes perfect: Chat-based image retrieval. Advances in Neural
Information Processing Systems, 36.
The paper discusses "ASK IMAGE," a chatbot that generates image captions and answers related
questions by combining computer vision and natural language processing. It leverages convolutional
and recurrent neural networks trained on the COCO and SQuAD datasets. The study covers the
system's architecture, implementation, and challenges. Additionally, it suggests potential
improvements for creating more accurate and contextually relevant image descriptions
 Literature survey
Reference 3 :Sidlauskiene, J., Joye, Y., & Auruskeviciene, V. (2023). AI-based chatbots
in conversational commerce and their effects on product and price perceptions. Electronic
Markets, 33(1), 24.
The research paper explores how AI-based chatbots with anthropomorphic design cues impact
product personalization and consumers' willingness to pay higher prices in conversational
GMR Institute of Technology

commerce. Through experiments, it finds that anthropomorphic chatbots enhance perceived

personalization, particularly for lonely consumers. The study offers valuable insights for improving
consumer engagement through chatbot design, with implications for e-commerce marketers.

Reference 4 :Kaushik, A., Jacob, B., & Velavan, P. (2022). An exploratory study on a
reinforcement learning prototype for multimodal image retrieval using a conversational search
interface. Knowledge, 2(1), 116-138.
The study developed a conversational image search system that helps users find relevant images
through dialogue, incorporating reinforcement learning to improve the search experience. Usability
testing showed positive results, indicating the system was effective and user-friendly for complex
image search tasks.
 Literature survey
Reference 5 :Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M., ... &
Batra, D. (2022). Visual dialog. In Proceedings of the IEEE conference on computer vision and
pattern recognition (pp. 326-335).
Visual Dialog involves an AI agent engaging in conversation about images, grounding questions in
GMR Institute of Technology

visual content and using dialog history for context. The VisDial v0.9 dataset includes 1.2M question-
answer pairs on 120k images. Advanced encoder-decoder models and a retrieval-based evaluation
protocol measure AI performance against humans.

Reference 6 :Pearce, K., Alghowinem, S., & Breazeal, C. (2023, June). Build-a-bot:
teaching conversational ai using a transformer-based intent recognition and question
answering architecture. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol.
37, No. 13, pp. 16025-16032).
Build-A-Bot" is an open-source educational tool designed to help students and teachers create
transformer-based chatbots using course material. It offers hands-on experience with AI
fundamentals, including data collection and intent recognition. Future efforts will focus on
deploying it in schools and collecting user feedback.
 Literature survey
Reference 7 :Ismail, A. H., Al Ghaili, M. K. A., Yusof, M. A. H. M., Mastoi, S. A., Rosle,
M. H., Ilias, B., & Azmi, M. S. M. (2024). A Deep Learning Approach for Face Detection and
Recognition to Initiate Human-Robot Conversation. Applied Mathematics and Computational
Intelligence (AMCI), 13(2), 1-10.
This paper explores integrating AI into mobile robots to enhance conversational intelligence. It
GMR Institute of Technology

details the design of a robot with a screen for a face, and systems for person recognition and
tracking, all powered by an NVIDIA® JetsonTM Nano. The goal is to enable natural interactions
between the robot and human users.

Reference 8 : Adyapady, R. R., & Annappa, B. (2023). A comprehensive review of

facial expression recognition techniques. Multimedia Systems, 29(1), 73-103.
Facial expression recognition is a crucial area in computer vision and AI. It involves developing
algorithms to accurately analyze emotions from visual cues. The field is shifting towards real-world
applications, while addressing challenges and identifying future opportunities for more robust
systems.
 Literature survey
Reference 9 :Bhutada, S., Madabhushi, M., Shivani, S., & Choolakal, S. (2023, June).
Facial Emotion Recognition Using Chatbot and Raspberry Pi. In International Conference on
Soft Computing and Signal Processing (pp. 53-66). Singapore: Springer Nature Singapore.
This chatbot leverages NLP and ML to simulate conversations, incorporating emotion recognition
GMR Institute of Technology

via facial expressions. Built with Python, OpenCV, NumPy, pandas, Docker, and TensorFlow, it
seeks to improve chatbot effectiveness by responding to detected emotions.

Reference 10:Nie, L., Jiao, F., Wang, W., Wang, Y., & Tian, Q. (2022). Conversational
image search. IEEE Transactions on Image Processing, 30, 7732-7743.
Conversational image search that tackles understanding complex user intents through a multimodal
neural network. It also enhances image representation with knowledge from a memory network,
making searches more precise. The model significantly outperforms benchmarks and provides
useful resources for researchers.
 Literature survey
Reference 11:Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot
technology. In IFIP international conference on artificial intelligence applications and
innovations (pp. 373-383). Springer, Cham.
The paragraph outlines the evolution of chatbots in diverse sectors like marketing and healthcare,
highlighting their growing utility and classification by application. It discusses how social
GMR Institute of Technology

stereotypes impact chatbot design and provides an overview of their architecture and development
platforms, emphasizing the need for in-depth study.

Reference 12:Rodriguez, C., & Oppenheimer, D. M. (2024). Creating a Bot-tleneck

for malicious AI: Psychological methods for bot detection. Behavior Research Methods, 1-
18.
The paragraph discusses the limitations of CAPTCHAs in identifying bots, as recent AI
advancements enable bots to bypass them easily. A more effective but labor-intensive method
uses image-processing and causal-reasoning questions, though this approach may soon be
obsolete with the rise of AI like GPT.
 Literature survey
Reference 13:Vu, T., Zhang, L., & Cho, E. (2022). Improving multimodal
conversational systems with visual grounding. Journal of Artificial Intelligence
Research, 74, 32-48.
It explores visual grounding techniques, which involve associating dialogue with
GMR Institute of Technology

relevant images to improve the chatbot’s ability to understand and respond to queries
more accurately. This is particularly useful in multimodal systems where both text and
visual inputs are necessary.
Reference 14:Chen, Y., Zhang, Z., & Li, X. (2023). Cross-modal attention networks for
conversational image [Link] of the 2023 Conference on Neural
Information Processing Systems (NeurIPS), 1450-1462.
Cross-modal attention network to enhance conversational image understanding. By leveraging
attention mechanisms to fuse visual and textual modalities, the model improves chatbots' ability to
comprehend and discuss image content. It demonstrates significant advancements in generating
contextually appropriate responses and managing complex multimodal interactions.
 Literature survey
Reference 15:Wang, R., Yang, T., & Xu, J. (2024). Enhancing chatbot responses with
visual context through hierarchical [Link] of the 2024 International Joint
Conference on Artificial Intelligence (IJCAI), 3120-3132.

A hierarchical attention model to enhance chatbot responses by integrating both local and global
GMR Institute of Technology

visual features. This approach allows for more nuanced and contextually accurate responses in
image-based conversations. The results show significant improvements in chatbots' ability to
engage in meaningful and context-aware dialogue.
 Comparison Table
Sl. No. Title Year Objectives Limitations Advantages Performance Gaps
Metrics

1. Conversationa 2024 Integrate Limited to Advanced MRR:0.6041 Potential

l Chatbot With image COCO object R@1 :0.4617 improvement
Object recognition dataset; detection and R@5:0.7780 s in dataset
GMR Institute of Technology

Recognition and NLP; use potential for instance R@10:0.873 diversity and
Using Deep Mask R-CNN model bias segmentation; model
Learning and for object detailed generalizatio
Machine detection architecture n
Learning coverage

2. Chatting 2024 Combine image Challenges in Integration of 78% success Need for more
makes perfect: captioning with generating convolutional rate after 5 diverse datasets
Chat-based NLP; improve accurate image and recurrent dialogue for better
image retrieval contextually descriptions neural rounds. generalization
relevant image networks; 75% success
descriptions detailed system rate when
questions are
implementation
asked by
humans.

3. AI-based 2023 Study impact of Focus on Enhanced MRR:0.253 Broader

chatbots in anthropomorphi anthropomorph perceived R@1 :0.309 application
conversational c chatbots on ic design; may personalization; R@5:0.229 beyond
commerce and personalization not apply insights into R@10:0.753 anthropomorph
their effects on and willingness universally consumer ic design
product and to pay engagement in
price e-commerce
perceptions
 4. An exploratory study 2022 Develop a Limited to Effective and MRR:67.94 Potential to
on a reinforcement conversational usability user-friendly scale and refine
learning prototype for image search testing; may not interface for R@1:55.05 the
multimodal image system using address all complex image R@5:8398 reinforcement
retrieval using a reinforcement search searches; R@10:91.58 learning
conversational search learning challenges positive approach
interface usability
feedback
GMR Institute of Technology

5. Visual Dialog 2022 Engage in Dependence Advanced MRR value is Improvements

conversation on VisDial dialog 0.606. in handling
about images; dataset; grounding; Recall@1 of more diverse
use dialog limitations in extensive 0.376 and and complex
history for handling question- Recall@5 of dialogs
context complex answer pairs 0.764.
dialogs for training

6. Build-a-bot: 2023 Educate Focused on Hands-on BERT Base Future

teaching students on educational experience scoring 79.6 deployment
conversational ai creating tool; may not with AI BERT Large and real-
using a transformer- address real- fundamentals; scoring 82.1 world
transformer-based based chatbots world open-source feedback
intent recognition deployment tool for collection
and question challenges education
answering
architecture
emb
Dec
201
er
7

15
 7 A Deep 2024 Integrate AI Limited to Enhanced 0.5 meters: Broader
Learning for face specific natural Face detection application
Approach for detection and hardware interactions is successful, beyond
Face recognition in (NVIDIA® between but recognition current
Detection and mobile robots JetsonTM robots and fails. hardware and
Recognition to Nano); users; 2.0 meters face
Initiate potential innovative and beyond: recognition
Human-Robot challenges in use of face Face methods
GMR Institute of Technology

Conversation face detection recognition

recognition fails even
though face
detection is
possible.
8 A 2023 Review facial General Summary of Accuracies Further
comprehensiv expression review; may existing tend to range exploration of
e review of recognition lack depth in techniques; between 65% integration
and 75%. techniques
facial algorithms specific application
expression and their techniques potential
recognition applications
techniques

9 Facial 2023 Develop a Limited to Integration of Accuracy Enhancement

Emotion chatbot with specific facial values ranging in emotion
Recognition emotion hardware emotion from 56% to recognition
Using Chatbot recognition (Raspberry recognition 73%. and chatbot
and Raspberry using facial Pi); may not with chatbot; adaptability
Pi expressions handle all use of various
emotional software tools
nuances
emb
Dec
201
er
7

16
 10 Conversation 2022 Improve Focus on Advanced R@1:77.29 Further
al image image search multimodal multimodal R@5:79.98 optimization
search through neural search of search
R@10:76.11
multimodal networks; capabilities; algorithms
neural may require significant and data
networks extensive data improvement requirements
over
benchmarks
GMR Institute of Technology

11 An overview 2022 Outline General Broad Classification In-depth

of chatbot evolution, overview; understanding of chatbots; exploration of
technology architecture, may not of chatbot architectural recent
and cover recent technology insights advancements
applications advancements and its and specific
of chatbots in depth applications applications

12 Creating a 2024 Develop new Labor- Innovative Detection Exploration

Bot-tleneck methods for intensive approach to accuracy; of more
for malicious detecting approach; bot detection method scalable and
AI: malicious AI may become using effectiveness less labor-
Psychological bots obsolete with psychological intensive
methods for advancing AI methods detection
bot detection technologies methods
emb
Dec
201
er
7

17
 13 Improving 2022 Enhance Limited to Improved MRR:0.5965 Further
multimodal chatbot visual understanding R@1:45.55 exploration of
conversational understanding grounding and response R@5:76.22 integration
systems with by associating techniques; accuracy in R@10:85.37 techniques
visual dialogue with may not multimodal and context
grounding relevant address all systems handling
images conversational
challenges
GMR Institute of Technology

14 Cross-modal 2023 Use cross- May rely Enhanced R@1: 63.3% More diverse
attention modal heavily on ability to and complex
R@5: 87.4%
networks for attention to attention generate content
conversational improve mechanisms; contextually R@10: 93.6% handling
image image challenges in appropriate
understanding understanding handling responses;
in diverse advanced
conversations content attention
mechanisms

15 Enhancing 2024 Integrate local Hierarchical More nuanced Hierarchical Further

chatbot and global attention may and attention refinement in
responses visual features be complex; contextually effectiveness; hierarchical
with visual to improve performance accurate response attention and
context chatbot may vary responses; nuance generalization
through responses across improved to various
hierarchical different engagement in scenarios
attention scenarios image-based
conversations
emb
Dec
201
er
7

18
 Graphical representation with literature Survey
GMR Institute of Technology
 GMR Institute of Technology

7
201
er
emb
Dec
4

0.5
1.5
2.5
3.5

0
1
2
3

20
 References
1)Mahesh Babu, A., Jawarneh, M., Arias‐Gonzáles, J. L., Meenakshi, Kasat, K., & Yuvaraj, K. P.
(2024). Conversational Chatbot With Object Recognition Using Deep Learning and Machine
Learning. Conversational Artificial Intelligence, 335-352.
2) Levy, M., Ben-Ari, R., Darshan, N., & Lischinski, D. (2024). Chatting makes perfect: Chat-
GMR Institute of Technology

based image retrieval. Advances in Neural Information Processing Systems, 36.

3) Sidlauskiene, J., Joye, Y., & Auruskeviciene, V. (2023). AI-based chatbots in conversational
commerce and their effects on product and price perceptions. Electronic Markets, 33(1), 24.
4)Kaushik, A., Jacob, B., & Velavan, P. (2022). An exploratory study on a reinforcement learning
prototype for multimodal image retrieval using a conversational search
interface. Knowledge, 2(1), 116-138.

5)Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M., ... & Batra, D. (2022). Visual
dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp.
326-335).
 6) Pearce, K., Alghowinem, S., & Breazeal, C. (2023, June). Build-a-bot: teaching conversational ai
using a transformer-based intent recognition and question answering architecture. In Proceedings of
the AAAI Conference on Artificial Intelligence (Vol. 37, No. 13, pp. 16025-16032).

7) Ismail, A. H., Al Ghaili, M. K. A., Yusof, M. A. H. M., Mastoi, S. A., Rosle, M. H., Ilias, B., &
Azmi, M. S. M. (2024). A Deep Learning Approach for Face Detection and Recognition to Initiate
GMR Institute of Technology

Human-Robot Conversation. Applied Mathematics and Computational Intelligence (AMCI), 13(2),

1-10.

8) Adyapady, R. R., & Annappa, B. (2023). A comprehensive review of facial expression

recognition techniques. Multimedia Systems, 29(1), 73-103.

9)Bhutada, S., Madabhushi, M., Shivani, S., & Choolakal, S. (2023, June). Facial Emotion
Recognition Using Chatbot and Raspberry Pi. In International Conference on Soft Computing and
Signal Processing (pp. 53-66). Singapore: Springer Nature Singapore.
10) Nie, L., Jiao, F., Wang, W., Wang, Y., & Tian, Q. (2022). Conversational image search. IEEE
Transactions on Image Processing, 30, 7732-7743.
emb
Dec
201
er
7

22
 11)Adamopoulou, E., & Moussiades, L. (2022). An overview of chatbot technology. In IFIP
international conference on artificial intelligence applications and innovations (pp. 373-383).
Springer, Cham.

12)Rodriguez, C., & Oppenheimer, D. M. (2024). Creating a Bot-tleneck for malicious AI:
Psychological methods for bot detection. Behavior Research Methods, 1-18.
GMR Institute of Technology

13)Vu, T., Zhang, L., & Cho, E. (2022). Improving multimodal conversational systems with
visual grounding. Journal of Artificial Intelligence Research, 74, 32-48.

14) Chen, Y., Zhang, Z., & Li, X. (2023). Cross-modal attention networks for conversational
image [Link] of the 2023 Conference on Neural Information Processing
Systems (NeurIPS), 1450-1462.
15) Wang, R., Yang, T., & Xu, J. (2024). Enhancing chatbot responses with visual context
through hierarchical [Link] of the 2024 International Joint Conference on
Artificial Intelligence (IJCAI), 3120-3132.
emb
Dec
201
er
7

23
 METHODOLOGY

[Link] of the Research Problem

The primary aim of this research is to develop a conversational chatbot system with
integrated object recognition capabilities, leveraging deep learning and machine
GMR Institute of Technology

learning techniques. The chatbot is designed to recognize objects in real-time through

a camera input and engage in a meaningful dialogue about the identified objects.

[Link] Design
This study adopts an experimental research design that integrates deep learning
models for object recognition with a machine learning-based conversational chatbot.
The system is tested in a simulated environment to evaluate its performance and
accuracy in both object recognition and natural language processing (NLP).

24
 3. Data Collection Methods

3.1. Object Recognition Model

• Dataset: The object recognition model uses a pre-trained deep learning model such as
YOLO (You Only Look Once) or ResNet, which is trained on a large-scale dataset like
COCO (Common Objects in Context) for detecting and classifying various objects.
GMR Institute of Technology

3.2. Conversational Chatbot

• Chatbot Data: For the conversational component, the chatbot was trained on a dataset
of dialogue pairs extracted from publicly available corpora such as Cornell Movie
Dialogues or a custom-built dataset, tailored to conversations about objects.
• Preprocessing: Textual data was preprocessed to clean, tokenize, and filter irrelevant
information. This included removing stop words, punctuation, and applying
lemmatization.
emb
Dec
201
er
7

25
 4. Model Architecture

4.1. Deep Learning for Object Recognition

• Model Choice: A deep learning model like YOLOv5 was used for object detection and
classification. This model was chosen due to its high speed and accuracy in real-time
applications.
• Training: The model was fine-tuned using the collected dataset. Transfer learning was
GMR Institute of Technology

applied to reduce the training time by using pre-trained weights on the COCO dataset.

4.2. Conversational Chatbot

• NLP Model: The chatbot’s natural language processing functionality was built using a
sequence-to-sequence model (Seq2Seq) with LSTM (Long Short-Term Memory) layers
for handling dialogue generation. For improved performance, a Transformer-based
model like GPT-2 or BERT was used to understand and generate human-like responses.
• Training: The conversational model was trained on the chatbot dialogue dataset.
Techniques like beam search were used during inference to generate more coherent and
contextually appropriate responses.
emb
Dec
201
er
7

26
 Input Image

Object Detection
GMR Institute of Technology

Feature Extraction

Natural Language Processing (NLP)

Contextual Understanding

Response Genertion

output

27
 GMR Institute of Technology

Figure:1
Figure:2

28
 Results:

Data splitting:I split the 83k dataset into 80k for training, 3k for validation, and 40k for
testing using a stratified approach to ensure that tag patterns are well-represented across
all sets. This ensures balanced training and evaluation of the model.
GMR Institute of Technology

Preprocessing:I extracted VGG16-relu7 features for each image and paired them with
corresponding questions, simplifying dataset processing. Additionally, I corrected
spelling, converted digits to words, removed contractions, tokenized the text with NLTK,
and built a dictionary of words appearing at least five times in the training set.

Tuning Hyperparameters: The model, implemented in Torch, uses early stopping based
on MRR and has 2-layer LSTMs with 512-dimensional hidden states and 300-
dimensional embeddings for words and images. Adam optimizer is used with a
dynamically adjusted learning rate and gradient clipping to prevent explosion.

29
 Evaluation and Performance:For evaluation, I used MRR and Recall@k metrics,
where higher values are better for MRR and Recall@k, and lower for mean rank.
Discriminative models outperformed generative ones, and models with detailed history
performed better across different datasets
GMR Institute of Technology

Results Analysis: When the model is asked more complex questions, its responses
become less accurate, likely due to the image context history or insufficient training
data. To improve this, ensuring that each model is trained with enough data is essential
for better performance.

Model R@1 R@5 R@10 MeanR MRR

lf-disc-faster-rcnn-x101 0.4617 0.7780 0.8730 4.7545 0.6041
lf-gen-faster-rcnn-x101 0.3620 0.5640 0.6340 19.4458 0.4657

30
 GMR Institute of Technology

7
201
er
emb
Dec
4

31
 Conclusion:
In conclusion, the Conversational Image Recognition Chatbot represents a
significant advancement in the integration of artificial intelligence and user
interaction. By combining sophisticated image recognition technologies with natural
GMR Institute of Technology

language processing, this chatbot enhances the way users interact with visual
content. Its applications in fields such as education, e-commerce, and customer
service demonstrate its potential to improve accessibility, streamline tasks, and
promote intuitive communication between humans and machines. This innovative
tool not only showcases the capabilities of AI-driven systems but also opens the door
to future advancements in human-computer interaction, offering practical solutions
to complex challenges across various industries.

32
 GMR Institute of Technology

33