AI DOCTOR MEDICAL CHATBOT INTRODUCTION

CHAPTER 1
INTRODUCTION

1.1 Introduction
Access to timely and reliable healthcare remains a major challenge for many indi-
viduals, particularly in remote, rural, and economically disadvantaged regions. Pa-
tients often face difficulties in consulting qualified medical professionals due to factors
such as long waiting times, high consultation costs, limited hospital infrastructure,
and geographical barriers. Even in urban areas, hospitals and clinics are frequently
overcrowded, placing heavy workloads on doctors and reducing the quality of person-
alized patient care. As a result, many individuals delay seeking medical attention or
rely on unreliable online information, which can lead to misdiagnosis and worsening
health conditions.
Traditional digital healthcare solutions, such as basic symptom checker websites
and telemedicine platforms, provide limited support. Most existing systems rely pri-
marily on text-based interaction, requiring users to manually type symptoms. This
approach is not suitable for all patients, especially the elderly, visually impaired, or
those with low literacy levels. Furthermore, these systems generally lack the abil-
ity to analyze medical images such as skin conditions, wounds, or prescriptions, and
they often fail to support regional or local languages. Consequently, patients struggle
to clearly explain their symptoms, and the guidance provided may be incomplete or
inaccurate.
Recent advancements in Artificial Intelligence, particularly in Large Language
Models (LLMs), Computer Vision, and Speech Processing, have opened new pos-
sibilities for intelligent healthcare assistance. Modern multimodal AI systems can un-
derstand and process text, voice, and images simultaneously. Speech-to-Text models
can accurately transcribe patient speech, vision models can analyze medical images,
and Text-to-Speech systems can generate natural human-like voice responses. When
combined, these technologies can simulate a virtual medical consultation experience,
making healthcare more interactive, accessible, and user-friendly.
A Multimodal AI Doctor system can act as a “digital healthcare assistant,” capa-
ble of understanding spoken symptoms, interpreting uploaded medical images, and
responding with clear medical guidance in both text and voice formats. Such a system
can provide preliminary diagnosis, suggest possible medical conditions, assess severity
levels, and recommend next steps such as home care, specialist consultation, or emer-
gency attention. Additionally, multilingual support allows patients to interact in their
preferred language, reducing communication barriers and improving understanding.

Department of AI ML,Vemana IT 1 2025-26

AI DOCTOR MEDICAL CHATBOT INTRODUCTION

Despite the potential of AI-driven healthcare systems, many existing solutions are
either expensive, complex, or dependent on proprietary hardware and closed plat-
forms. There is a strong need for a scalable, cost-effective, and intelligent medical
chatbot that can run on commonly available devices such as smartphones, laptops, or
web browsers without requiring specialized medical equipment.
The objective of this project is to develop an AI Doctor Medical Chatbot with
Multimodal LLM that integrates Speech-to-Text, Computer Vision, Large Language
Models, and Text-to-Speech technologies into a single unified platform. By enabling
voice-based interaction, medical image analysis, prescription scanning, and multi-
lingual communication, the system aims to provide accessible, efficient, and reliable
preliminary healthcare assistance. This project seeks to enhance healthcare accessibil-
ity, reduce the burden on medical professionals, and empower patients with informed
decision-making, ultimately improving the overall quality and reach of healthcare ser-
vices.

1.2 Scope
This system is not intended to replace professional medical practitioners or emer-
gency healthcare services but is designed to assist and complement traditional health-
care systems by providing preliminary medical guidance and health awareness. The
AI Doctor Medical Chatbot works alongside existing healthcare facilities by offering
real-time symptom analysis, medical image interpretation, and voice-based consul-
tation. It assumes that the user interacts with the system through a camera- and
microphone-enabled device such as a smartphone or laptop and receives responses
through text and synthesized speech.
The system primarily focuses on early-stage medical assistance, symptom triage,
and health information delivery, rather than definitive diagnosis or treatment. It is
suitable for both home-based use and remote healthcare access, including scenarios
where immediate access to doctors is limited. By combining Large Language Models,
Computer Vision, Speech-to-Text, and Text-to-Speech technologies, the system aims
to enhance patient understanding, reduce anxiety, and improve decision-making re-
garding medical care.
The scope of this project includes:
• Multimodal Patient Interaction: Supporting text, voice, and image-based
inputs to allow users to describe symptoms naturally and upload medical
images such as skin conditions, wounds, or prescriptions.

• Symptom Analysis and Triage: Providing AI-driven analysis of patient symp-

toms, severity estimation.

Department of AI ML,Vemana IT 2 2025-26

AI DOCTOR MEDICAL CHATBOT INTRODUCTION

• Medical Image Interpretation: Analyzing uploaded medical images using

computer vision models to assist in identifying visible conditions and ab-
normalities.

• Speech-to-Text Processing: Accurately converting patient speech into text

for further AI-based medical reasoning.

• Text-to-Speech Response: Delivering AI-generated medical guidance through

natural voice output for better accessibility and ease of use.

• Multilingual Support: Supporting interaction in multiple languages such as

English, Kannada, Hindi, Tamil, and Telugu to reduce language barriers in
healthcare.

• User History Management: Maintaining secure records of previous consulta-

tions to enable continuity and reference in future interactions.

• Affordability and Accessibility: Designed to operate on standard consumer

devices without the need for specialized medical hardware, making it a cost-
effective and scalable solution.

1.3 Objectives
The main objectives of the AI Doctor Medical Chatbot with Multimodel LLM
project are:

• Multimodal Medical Interaction: Design and develop a medical chatbot ca-

pable of understanding and processing text, voice, and image inputs using a
multimodal Large Language Model (LLM).

• Speech-Based Consultation: Integrate an efficient Speech-to-Text (STT) model

(such as OpenAI Whisper) to accurately transcribe patient speech into text
for symptom analysis.

• Medical Image Analysis: Implement computer vision techniques to analyze

uploaded medical images (such as skin conditions, wounds, eye images, and
prescriptions) to assist in preliminary diagnosis.

• AI-Driven Symptom Analysis: Use Large Language Models to analyze patient

symptoms, identify possible medical conditions, estimate severity levels, and
generate appropriate healthcare recommendations.

Department of AI ML,Vemana IT 3 2025-26

AI DOCTOR MEDICAL CHATBOT INTRODUCTION

• Multilingual Communication: Enable the system to support multiple lan-

guages (English, Kannada, Hindi, Tamil, and Telugu) to improve healthcare
accessibility for diverse users.

• Text-to-Speech Response System: Integrate a Text-to-Speech (TTS) module

to convert AI-generated medical advice into natural and human-like voice
output for better user engagement.

• User-Friendly Interface: Develop a simple and intuitive web-based interface

(using Gradio or React) that allows easy interaction for users of all age groups
and technical backgrounds.

• Secure User Data Management: Implement secure user authentication and

consultation history storage to ensure privacy and continuity of medical guid-
ance.

1.4 Organization of the Project

The project report is organized into eight chapters, each focusing on a specific
aspect of the AI Doctor Medical Chatbot with Multimodel LLM development. The
structure of the report is as follows:

• Chapter 1– Introduction: Provides an overview of the project, including

the background of AI-based healthcare systems, problem definition, scope,
objectives, and the overall organization of the report.

• Chapter 2– Literature Survey: Presents a detailed review of existing

research related to medical chatbots, Large Language Models in healthcare,
multimodal AI systems, speech-based medical assistants, and medical image
analysis. A comparative analysis highlighting the strengths and limitations
of existing healthcare solutions is also included.

• Chapter 3– System Analysis: Describes the existing digital healthcare

systems and their limitations. It also explains the proposed AI Doctor system,
its advantages, and the feasibility study covering technical, operational, and
economic aspects.

• Chapter 4– System Specification: Details the hardware and software

requirements of the AI Doctor Medical Chatbot. This chapter also includes
the functional and non-functional requirements necessary to ensure system
performance, reliability, security, and user accessibility.

Department of AI ML,Vemana IT 4 2025-26

AI DOCTOR MEDICAL CHATBOT INTRODUCTION

• Chapter 5– Project Description: Explains the complete working of the

system, including the problem definition, overall project overview, system
architecture, module descriptions, and design diagrams such as data flow
diagrams, use case diagrams, and sequence diagrams.

• Chapter 6– System Implementation: Discusses the implementation phase

in detail, including backend development using Python and FastAPI, fron-
tend development using [Link] or Gradio, integration of multimodal LLMs,
Speech-to-Text and Text-to-Speech modules, medical image analysis, pre-
scription OCR setup, and screenshots of the working system.

• Chapter 7 – System Testing: Describes the testing methodologies adopted,

the test cases executed, and the overall system performance evaluation, in-
cluding speech recognition accuracy, medical image analysis accuracy, re-
sponse time, and system reliability.

• Chapter 8 – Conclusions and Future Enhancements: Summarizes the

major outcomes of the project, highlights the key achievements, and proposes
future enhancements such as advanced diagnostic models, wearable health
integration, offline AI processing, and improved multilingual support.

• References: Lists all research papers, technical articles, and online resources
referred to during the project.

• Appendix: Includes supporting materials such as frontend and backend

source code, API integration details, installation procedures, configuration
files, and system screenshots.

Department of AI ML,Vemana IT 5 2025-26

AI DOCTOR MEDICAL CHATBOT INTRODUCTION

Gantt Chart – AI DOCTOR MEDICAL CHATBOT WITH MULTIMODEL LLM

(Feb-Nov 2025)

Figure 1.1: Timeline Diagram

Department of AI ML,Vemana IT 6 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

CHAPTER 2
LITERATURE SURVEY

2.1 “Multimodal Medical AI: Integrating Text, Image, and

Speech for Clinical Decision Support”

J. Li1, X. Zhang2, and F. Wang3 Journal of Biomedical Informatics Received: 15

March 2022, Accepted: 10 June 2022, Date of Publication: 20 June 2022, Current
Version: August 2022.
Overview:
The paper titled “Multimodal Medical AI: Integrating Text, Image, and Speech for
Clinical Decision Support” presents a comprehensive analysis of multimodal artificial
intelligence systems applied to healthcare decision-making. The study emphasizes the
importance of combining textual patient records, medical images, and speech-based
symptom descriptions to enhance diagnostic accuracy and contextual understanding.
Conventional medical AI systems often depend on single-modality inputs, which re-
strict their ability to capture complex clinical information.
The authors explore the use of deep learning models and multimodal fusion tech-
niques to integrate data from different sources. Medical images are analyzed using
convolutional neural networks, textual data is processed through natural language
processing models, and speech inputs are converted into text using speech-to-text
systems. The study also discusses challenges such as data heterogeneity, privacy
protection, computational complexity, and the need for explainable AI in clinical en-
[Link] study also discusses challenges such as data heterogeneity, privacy
protection, computational complexity, and the need for explainable AI in clinical en-
vironments.
Medical images are processed using convolutional neural networks (CNNs) for vi-
sual feature extraction, textual data such as clinical notes and patient histories are
analyzed using natural language processing (NLP) models, and speech inputs are con-
verted into structured text using speech-to-text systems. By fusing these modalities,
the system achieves a more holistic representation of patient health conditions. The
study also highlights key challenges, including data heterogeneity, patient privacy and
security concerns, increased computational complexity, and the critical requirement
for explainable AI to ensure trust and transparency in clinical environments.

Department of AI ML,Vemana IT 7 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

Advantages:

• Multimodal Medical Analysis: The system integrates text, voice, and med-
ical image inputs to provide a more comprehensive preliminary medical assess-
ment.

• Improved Accessibility: Voice-based symptom input and text-to-speech out-

put enable hands-free interaction, benefiting elderly users and patients with dis-
abilities.

• Early Health Assessment: Provides quick preliminary observations and rec-

ommendations before visiting a medical professional, reducing delays in care.

• Scalability with LLM Integration: The AI Doctor system can be continu-

ously improved through model updates without significant changes to the system
architecture.

• User-Friendly Interaction: Natural language interaction allows users to com-

municate medical concerns without requiring technical or medical expertise.

Disadvantages:

• Dependency on AI Accuracy: Incorrect interpretation of symptoms or im-

ages may lead to inaccurate preliminary suggestions.

• Data Privacy and Security Concerns: Handling sensitive medical data re-
quires strict compliance with healthcare data protection standards.

• Computational Resource Requirements: Multimodal AI models require

higher processing power and optimized deployment for real-time performance.

• Limited Clinical Authority: The system cannot replace professional medical

diagnosis or treatment and must be used only for preliminary guidance.

2.2 “Transforming Healthcare with AI Chatbots: Uses and

Applications — A Scoping Review”
Marina Barreda, David Cantarero-Prieto, Daniel Coca, Abraham Delgado, Paloma
Lanza-León, Javier Lera, Rocío Montalbán, and Flora Pérez. Received: 18 September
2024, Accepted: 23 January 2025, Published: 2025.

Department of AI ML,Vemana IT 8 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

Overview:
The paper titled “Transforming Healthcare with Chatbots: Uses and Applications
— A Scoping Review” presents a comprehensive review of the growing role of ar-
tificial intelligence–based chatbots in modern healthcare systems. The study sys-
tematically analyzes recent research published over the last five years, focusing on
chatbot applications in medical diagnosis, mental health support, patient monitoring,
health education, appointment management, and public health interventions. The
review highlights that AI chatbots, powered by Natural Language Processing (NLP)
and machine learning techniques, enable efficient, scalable, and continuous healthcare
support. These systems are capable of symptom assessment, preliminary diagno-
sis, medical information delivery, medication reminders, and psychological assistance.
The authors emphasize that chatbots significantly improve healthcare accessibility,
particularly in underserved and remote regions, while reducing the workload on medi-
cal professionals. The study also discusses the increasing adoption of Large Language
Models (LLMs), which enhance conversational quality, contextual understanding, and
diagnostic accuracy.
In relation to your project, “AI Doctor”, this paper strongly supports the feasibility
and relevance of chatbot-based virtual medical consultation systems. Your AI Doctor
extends these concepts by integrating multimodal inputs such as voice-based symptom
description, image-based diagnosis, and prescription scanning. By combining speech
recognition, computer vision, and LLM reasoning, your system provides a more inter-
active, patient-centric, and intelligent healthcare solution. While the reviewed paper
focuses on text-based chatbot systems, your project advances this domain by enabling
real-time voice interaction and visual medical analysis, making it more suitable for
practical clinical assistance and telemedicine use cases.

Advantages:
• Improved Healthcare Accessibility: AI chatbots provide 24/7 medical as-
sistance, especially beneficial for rural and underserved populations.

• Efficient Symptom Assessment: Automated symptom analysis enables early

identification of potential health conditions before clinical visits.

• Reduced Workload on Doctors: Routine queries, appointment scheduling,

and basic medical guidance are handled automatically.

• Scalable and Cost-Effective: Chatbot systems can serve a large number of

users simultaneously without proportional increases in cost.

• Enhanced Patient Engagement: Conversational interfaces improve user in-

teraction, compliance, and health awareness.

Department of AI ML,Vemana IT 9 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

Disadvantages:

• Limited Diagnostic Accuracy: Chatbots provide preliminary assessments

and cannot replace professional medical diagnosis.

• Data Privacy Concerns: Handling sensitive health data raises ethical and
security challenges.

• Dependence on Data Quality: Incorrect or incomplete user inputs may lead

to inaccurate recommendations.

• High Initial Development Cost: Advanced AI models and infrastructure

may require significant investment.

• User Trust and Acceptance: Some users may hesitate to rely on AI-driven
medical advice.

2.3 “Enhancing Clinical Accuracy of Medical Chatbots With

Large Language Models”
Zhonghua Liu, Yu Quan, Xiaohong Lyu, and Mohammed J. F. Alenazi. Received:
2 June 2024, Revised: 5 August 11 September 2024, Accepted: 23 September 2024,
Published: September 2025.
Overview:
The paper titled “Enhancing Clinical Accuracy of Medical Chatbots With Large
Language Models” presents an advanced framework to improve the reliability, con-
textual understanding, and clinical correctness of AI-powered medical chatbots. The
study addresses a major limitation of existing chatbot systems—inconsistent responses
and lack of medical accuracy in multi-turn conversations.
The authors propose a Multi-Turn Medical Dialogue (MC-MTD) model that in-
tegrates Large Language Models (LLMs) with novel architectural enhancements to
maintain conversational context and prioritize clinically relevant information. The
approach focuses on improving dialogue coherence, symptom understanding, and med-
ical entity recognition across extended patient–chatbot interactions.
Key technical contributions include:

• Contextual Layer Normalization (CLN) to stabilize training of deep transformer

models

• Contextual Sliding Window Reply Prediction (CSWRP) to capture fine-grained

local dialogue context

Department of AI ML,Vemana IT 10 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

• Local Critical Information Distillation (LCID) to emphasize medically important

details

• A dedicated Encoder–Decoder–Context Fusion Module (EDC) architecture for

multi-turn dialogue handling
The system is evaluated using MIMIC-III and n2c2 clinical datasets, where it sig-
nificantly outperforms existing state-of-the-art chatbot models in terms of perplexity,
BLEU-2 score, recall@K, medical entity extraction, and response coherence. The re-
sults demonstrate that incorporating structured context management and information
distillation mechanisms substantially improves chatbot clinical accuracy and reliabil-
ity
Advantages:
• High Clinical Accuracy: Advanced context modeling reduces misleading, con-
tradictory, or unsafe medical responses.

• Improved Context Retention: Multi-turn dialogue handling ensures that

patient history and symptoms are preserved across conversations.

• Reliable Medical Entity Recognition: Accurate extraction of diseases, symp-

toms, and treatments improves diagnostic quality.

• Scalable Healthcare Support: Capable of assisting a large number of patients

simultaneously with consistent performance.

• Doctor Support Tool: Acts as a clinical decision-support system for prelimi-

nary consultations and patient triage.
Disadvantages:
• Cannot Replace Doctors: The system provides guidance but cannot deliver
final clinical diagnoses.

• High Computational Cost: Training and deploying LLM-based models re-

quire powerful computing infrastructure.

• Data Dependency: Model performance depends heavily on high-quality and

representative medical datasets.

• Privacy and Security Risks: Handling sensitive health data requires strict
compliance with healthcare regulations.

• Limited Dataset Generalization: Predominantly evaluated on U.S.-centric

datasets; global applicability requires further validation.

Department of AI ML,Vemana IT 11 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

2.4 “Toward Inclusive Healthcare: An LLM-Based Multimodal

Chatbot for Preliminary Diagnosis”
Ishita Agarwal, V. Sakthivel, and P. Prakash. Received: 27 June 2025, Accepted:
22 July 2025, Published: July 2025.
Overview:
The paper titled “Toward Inclusive Healthcare: An LLM-Based Multimodal Chat-
bot for Preliminary Diagnosis” presents the design and implementation of an advanced
multimodal medical chatbot aimed at improving healthcare accessibility, especially
for underserved and low-resource populations. The study addresses global healthcare
challenges such as limited access to medical professionals, high consultation costs, and
lack of infrastructure by proposing an AI-driven preliminary diagnosis system.
The proposed system leverages Large Language Models (LLMs) in combination
with a Retrieval-Augmented Generation (RAG) framework to provide accurate, context-
aware, and reliable medical guidance. Unlike traditional chatbot systems that rely
solely on pre-trained model knowledge, the RAG-based approach grounds responses in
a curated medical knowledge base (sourced from WebMD) using FAISS-based vector
retrieval, significantly reducing hallucinations and improving response credibility.
A key contribution of this work is its multimodal capability, allowing users to pro-
vide both textual symptom descriptions and medical images (such as skin conditions
or eye disorders). Medical images are first converted into structured textual descrip-
tions using the Gemini Flash 2.0 model, which are then used for semantic retrieval
and diagnosis reasoning. The system also maintains conversation context through
summarized chat history, enabling coherent multi-turn interactions.
Experimental evaluation across multiple medical queries demonstrates that the
proposed multimodal RAG-based chatbot produces more specific, relevant, and safer
diagnostic suggestions compared to standalone LLMs, while maintaining real-time
response performance. The study concludes that combining multimodal reasoning
with retrieval grounding is an effective strategy for building scalable, inclusive, and
trustworthy AI healthcare assistants

Advantages:
• Inclusive Healthcare Access: Enables preliminary medical guidance for users
in rural, remote, and underserved regions.

• Reduced Hallucination: RAG-based grounding ensures responses are based

on verified medical knowledge.

• Multimodal Diagnosis: Supports both symptom-based text input and

Department of AI ML,Vemana IT 12 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

• Context-Aware Conversations: Maintains chat history, enabling coherent

and reliable multi-turn interactions.

• Cost-Effective and Scalable: Reduces dependence on immediate doctor avail-

ability and supports large user bases.

Disadvantages:

• Preliminary Diagnosis Only: The system cannot replace professional medical

consultation or clinical decision-making.

• Knowledge Base Dependency: Diagnostic accuracy depends on the com-

pleteness and quality of the medical dataset.

• Limited Image Interpretation: Complex or rare medical conditions may re-

quire additional fine-tuning and validation.

• Privacy and Ethical Concerns: Handling sensitive medical data demands

strict security and regulatory compliance.

• Lack of Clinical Validation: The system is not extensively validated by cer-

tified healthcare professionals.

2.5 “AI-Enabled Medical Chatbots: Advancements in Patient

Query Handling and Automated Healthcare Delivery”
Sandeep Singh, Tripti Rathore, and Nipun Singhal. Academic Editors: Zhigang
Zhu, John Ross Rizzo, and Hao Tang. Received: 27 November 2024 Revised: 28
December 2024 Accepted: 2 January 2025 Published: 4 January 2025.
Overview:
The paper titled “AI-Enabled Medical Chatbots: Advancements in Patient Query
Handling and Automated Healthcare Delivery” provides an in-depth analysis of how
artificial intelligence is reshaping modern healthcare services through intelligent chat-
bot systems. The study explores the evolution of medical chatbots from traditional
rule-based systems to advanced AI-driven conversational agents powered by Natural
Language Processing (NLP), Deep Learning, and Large Language Models (LLMs).
The authors highlight that AI-enabled medical chatbots are increasingly capable
of handling complex patient queries, understanding contextual medical conversations,
and delivering preliminary diagnostic insights. These systems assist users by analyz-
ing symptoms, answering medical questions, recommending treatments, and guiding
patients toward appropriate healthcare services. The paper emphasizes that such

Department of AI ML,Vemana IT 13 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

chatbots improve healthcare efficiency by reducing response time, minimizing human

workload, and providing continuous support to patients.
A significant focus of the study is on the integration of LLMs and multimodal AI
techniques, enabling chatbots to process not only text-based queries but also medical
images and voice inputs. Vision-based models are employed for analyzing X-rays, CT
scans, and dermatological images, while speech recognition systems enhance accessibil-
ity for elderly and visually impaired users. The paper demonstrates that multimodal
chatbots significantly outperform traditional text-only systems in terms of diagnostic
accuracy, user engagement, and clinical relevance.
The research also presents a structured chatbot architecture consisting of text
processing modules, image analysis components, and response generation units. Per-
formance evaluations reveal that AI-enabled chatbots achieve high accuracy (above
90), improved conversational coherence, faster inference time, and better ethical com-
pliance when compared to earlier machine-learning-based healthcare systems. The
authors further discuss how these systems support automated healthcare delivery, in-
cluding symptom triage, appointment scheduling, medication reminders, and follow-
up care.
In addition to technical advancements, the paper addresses critical challenges such
as data privacy, ethical responsibility, bias mitigation, and regulatory compliance. It
stresses the importance of designing chatbot systems that adhere to healthcare regula-
tions like HIPAA and GDPR, ensuring patient safety and trust. The study concludes
that AI-enabled medical chatbots represent a transformative solution for scalable,
efficient, and patient-centric healthcare delivery, especially in resource-constrained
environments.

Advantages:

• Improved Patient Query Handling: Accurately understands and responds

to complex medical questions using advanced NLP and LLM reasoning.

• Multimodal Healthcare Support: Integrates text, voice, and medical image

analysis for richer and more reliable diagnostic assistance.

• Automated Healthcare Delivery: Enables symptom triage, preliminary di-

agnosis, appointment scheduling, and follow-up guidance.

• Reduced Doctor Workload: Automates routine medical inquiries and patient

education, allowing doctors to focus on critical cases.

Department of AI ML,Vemana IT 14 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

• Scalable and Efficient: Supports large patient populations with minimal ad-
ditional operational cost.

Disadvantages:

• Limited Clinical Authority: Chatbots provide guidance and cannot replace

professional medical diagnosis or treatment.

• Ethical and Privacy Concerns: Handling sensitive medical data requires

strict security, encryption, and regulatory compliance.

• High Computational Requirements: Advanced LLMs and vision models

demand powerful hardware and cloud infrastructure.

• Dependence on Training Data: System performance depends heavily on the

quality, diversity, and completeness of medical datasets.

• User Trust Challenges: Some patients may hesitate to rely on AI-driven med-
ical advice without human verification.

2.6 “A Review of Applying Large Language Models in

Healthcare”
Qiming Liu, Ruirong Yang, Qin Gao, Tengxiao Liang, Xiuyuan Wang, Shiju Li,
Bingyin Lei, and Kaiye Gao. Received: 1 December 2024 Accepted: 20 December
2024 Published: 31 December 2024 Current Version: 13 January 2025.
Overview:
The paper titled “A Review of Applying Large Language Models in Healthcare”
presents a comprehensive and systematic review of the rapid advancements and grow-
ing applications of Large Language Models (LLMs) in the healthcare domain. Pub-
lished in IEEE Access, this study addresses the increasing demand for efficient health-
care services amid limited medical resources and highlights how LLMs can play a
transformative role in modern medical systems.
The authors begin by discussing the fundamental architecture and working princi-
ples of LLMs, emphasizing the role of Transformer-based models, self-attention mech-
anisms, and large-scale pretraining. The paper systematically explains how LLMs
are trained for healthcare applications through data collection, preprocessing, pre-
training, fine-tuning, and reinforcement learning from human feedback (RLHF). Spe-
cial attention is given to handling sensitive medical data, ensuring anonymization.
A major contribution of this review is the detailed analysis of six key
application areas of LLMs in healthcare:

Department of AI ML,Vemana IT 15 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

• Disease Diagnosis and Clinical Decision Support: LLMs assist in inter-

preting patient symptoms, clinical notes, and medical histories to provide pre-
liminary diagnostic suggestions and treatment recommendations. Models such
as HuatuoGPT and Med-PaLM demonstrate improved diagnostic accuracy and
multi-turn consultation capabilities.

• Medical Knowledge Dissemination: LLMs enhance patient education by

answering health-related questions, explaining diseases, and improving health
literacy. The paper highlights that models like ChatGPT and MedGPT perform
comparably to human medical assistants in knowledge delivery and even achieve
passing scores in exams such as USMLE.

• Medical Assistants and Chatbots: AI-powered chatbots reduce physician

workload by handling routine patient queries, summarizing electronic health
records, generating discharge summaries, and supporting administrative tasks.
Multimodal chatbots also enable continuous patient interaction and follow-up
care.

• Medical Image Analysis: The review explains how LLMs, when integrated
with vision models, assist in interpreting X-rays, CT scans, dermatological im-
ages, and handwritten prescriptions. Multimodal systems such as Visual Med-
Alpaca and XrayGPT demonstrate promising results in medical image under-
standing.

• Biomedicine and Drug Discovery: LLMs contribute to precision medicine by

analyzing genomic data, predicting drug–drug interactions, assisting in vaccine
development, and supporting protein structure analysis. Models like AlphaFold
and ProGen highlight the potential of AI in biomedical research.

• Medical Education: LLMs support medical training by generating clinical

case scenarios, quizzes, simulations, and personalized feedback. Their strong
performance in medical examinations indicates their usefulness as educational
tools for medical students

The paper also presents a bibliometric analysis of LLM-related healthcare research

from 2018 to 2024, revealing a sharp increase in publications and global research
interest, particularly in the United States and the United Kingdom. Keyword trend
analysis shows a recent focus on validation, reliability, and ethical deployment of
LLMs.
Finally, the authors discuss major challenges, including model interpretability, data
privacy, ethical concerns, bias, and reliability, and propose future research directions

Department of AI ML,Vemana IT 16 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

such as multimodal diagnostics, integration with wearable devices, telemedicine plat-

forms, and virtual reality–based healthcare systems. The study concludes that while
LLMs have shown remarkable progress, further optimization and clinical validation
are essential for safe and large-scale deployment in healthcare
Advantages:

• Comprehensive Healthcare Coverage: Supports diagnosis, medical educa-

tion, patient assistance, imaging analysis, and biomedical research within a uni-
fied AI framework.

• Multimodal Capability: Effectively processes and reasons over text, medical

images, and structured clinical data.

• Improved Healthcare Efficiency: Reduces doctor workload and patient re-

sponse time through automated preliminary consultation.

• Scalable and Cost-Effective: Suitable for large populations and resource-

limited healthcare environments.

• Strong Research Foundation: Backed by extensive literature surveys and

bibliometric analysis, ensuring scientific credibility.

Disadvantages:

• Interpretability Issues: The black-box nature of LLMs limits transparency

and clinical explainability.

• Data Privacy Risks: Handling sensitive medical data requires strict compli-
ance with healthcare regulations.

• Bias and Reliability Concerns: System performance depends heavily on the

quality and diversity of training data.

• High Computational Cost: Training and deployment of multimodal LLMs

demand significant computational resources.

• Limited Clinical Validation: The system cannot replace professional medical

judgment and requires further clinical evaluation.

Department of AI ML,Vemana IT 17 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

2.7 “Multimodal Large Language Models for Medicine: A

Comprehensive Survey”
Jiarui Ye and Hao Tang. Published in IEEE Transactions on Pattern Analysis and
Machine Intelligence, 2025.
Overview:
The paper titled “Multimodal Large Language Models for Medicine: A Compre-
hensive Survey” presents an extensive and authoritative review of the recent progress,
applications, challenges, and future directions of Multimodal Large Language Models
(MLLMs) in the medical and healthcare domain. The study systematically analyzes
over 330 recent research papers, making it one of the most comprehensive surveys on
medical MLLMs to date.
The authors begin by explaining the evolution of language models, starting from
Statistical Language Models (SLMs) and Neural Language Models (NLMs) to Pre-
trained Language Models (PLMs) and modern Large Language Models (LLMs). With
the emergence of Transformer-based architectures and the release of models such as
GPT-4, the focus has shifted toward multimodal learning, where models can jointly
process text, images, audio, video, and biomedical data. The paper highlights that
healthcare naturally involves multiple data modalities, making MLLMs particularly
suitable for medical applications.

A major contribution of this survey is the detailed explanation of MLLM

architecture, which typically consists of:
• A core LLM for reasoning and language generation.

• Modality-specific encoders (vision, audio, text).

• An alignment module to map heterogeneous modalities into a shared semantic

space.
A major contribution of this review is the detailed analysis of six key
application areas of LLMs in healthcare:
• Medical Report Generation: MLLMs generate structured clinical reports
from medical images such as X-rays, CT scans, MRIs, and ultrasound images.
The survey discusses models like XrayGPT, Med-Flamingo, LLaVA-Med, MAIRA,
and ChatCAD, which significantly reduce radiologist workload while maintaining
professional medical terminology and report structure.

• Professional and Compassionate Medical Communication: The survey

highlights how MLLMs improve doctor–patient interaction by supporting med-

Department of AI ML,Vemana IT 18 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

ical chatbots capable of symptom analysis, follow-up care, mental health coun-
seling, and empathetic conversation. By integrating visual, textual, and audio
cues, MLLMs address emotional context, tone, and behavioral signals, which are
critical in psychological and primary healthcare support.

• Clinical Surgery Assistance: Advanced MLLMs assist surgeons by analyz-

ing surgical videos, endoscopic images, and patient history, providing real-time
guidance and post-operative report generation. Models such as SurgicalGPT and
LLaVA-Surg demonstrate strong performance in surgical visual question answer-
ing (VQA) and procedural understanding.

Advantages:

• Comprehensive Multimodal Understanding: Processes text, images, au-

dio, and medical records jointly for holistic medical reasoning.

• Improved Clinical Efficiency: Reduces doctor workload by automating symp-

tom analysis, report interpretation, and preliminary consultations.

• Enhanced Patient Interaction: Supports empathetic, context-aware, and

multi-turn medical conversations.

• Scalable Healthcare Solutions: Suitable for telemedicine, remote diagnosis,

and underserved or rural regions.

• Strong Research Validation: Backed by extensive literature, benchmarks,

and peer-reviewed studies.

Disadvantages:

• Hallucination Risk: Incorrect or ungrounded medical outputs can lead to

serious consequences if not properly constrained.

• High Computational Cost: Training and deployment of large multimodal

models require powerful hardware and infrastructure.

• Data Scarcity: Limited availability of high-quality, diverse, and annotated

medical datasets.

• Bias and Fairness Issues: Model performance may vary across different pop-
ulations and demographics.

• Limited Clinical Deployment: Requires extensive validation, regulatory ap-

proval, and ethical compliance before real-world use.

Department of AI ML,Vemana IT 19 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

2.8 “Leveraging Large Language Models: Implementing an

Advanced AI Chatbot for Healthcare”
Ajinkya Mhatre, Sandeep R. Warhade, Omkar Pawar, Sayali Kokate, Samyak Jain,
and Dr. Emmanuel M. Published in International Journal of Innovative Science and
Research Technology (IJISRT), Volume 9, Issue 5, May 2024.
Overview:
The paper titled “Leveraging Large Language Models: Implementing an Advanced
AI Chatbot for Healthcare” presents the design, implementation, and evaluation of an
LLM-powered medical chatbot aimed at addressing general illness-related queries and
improving healthcare accessibility. The study emphasizes the growing role of Large
Language Models (LLMs) in delivering automated, context-aware, and conversational
healthcare assistance.
The authors begin by highlighting the challenges faced by modern healthcare sys-
tems, including rising medical costs, limited availability of healthcare professionals,
and delayed access to medical advice. To address these issues, the paper proposes an
AI-driven healthcare chatbot that can provide preliminary medical guidance before a
patient consults a doctor. The system is designed to assist users by answering com-
mon health-related questions, offering symptom explanations, and delivering basic
disease-related information.
A key contribution of this work is its retrieval-based chatbot architecture, which
integrates LangChain, vector databases, cosine similarity, and Retrieval-Augmented
Generation (RAG). Medical data is sourced from approved textbooks and guidelines
by the Indian Council of Medical Research (ICMR) and AIIMS, ensuring domain
reliability. Unstructured medical documents are processed using text extraction tech-
niques, chunked into smaller segments, and converted into vector embeddings. These
embeddings are stored in a vector database to enable semantic similarity search dur-
ing user interaction.
The chatbot uses cosine similarity to retrieve the most relevant document chunks
related to a user’s query, which are then combined with prompt templates and passed
to the LLM for response generation. This approach improves contextual relevance
and reduces hallucination compared to standalone LLM responses. The system also
maintains a user interaction history using MongoDB, storing previous questions and
answers to enhance personalization and continuity.
Experimental evaluation is conducted using the MedMCQA dataset, which con-
tains over 194,000 medical entrance examination questions covering 21 medical sub-
jects. The results show that the chatbot performs well on simple factual queries, but
exhibits reduced accuracy on complex reasoning-based questions, particularly those

Department of AI ML,Vemana IT 20 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

involving relationships between multiple medical concepts. The paper reports an

overall accuracy of approximately 61, with performance improving significantly when
deployed on GPU-based systems compared to CPU-only environments.
The authors also compare their proposed model with popular LLMs such as Chat-
GPT, Bard, and Gemini, demonstrating competitive performance while highlighting
the trade-off between accuracy and caution in medical response generation. The study
concludes that LLM-powered chatbots have strong potential for preliminary health-
care assistance but must be carefully validated to ensure safety, reliability, and ethical
compliance
Advantages:

• Improved Healthcare Accessibility: Provides medical guidance and prelim-

inary consultation support before visiting a doctor.

• Context-Aware Responses: Uses retrieval-based grounding to generate reli-

able and medically relevant answers.

• Cost-Effective Solution: Reduces unnecessary clinical visits for minor and

non-critical health concerns.

• Scalable Architecture: Capable of serving a large number of users simultane-

ously without performance degradation.

• Dataset-Driven Evaluation: Validated using the MedMCQA medical bench-

mark dataset, ensuring research-backed performance.

Disadvantages:

• Limited Diagnostic Accuracy: Suitable only for preliminary medical guid-

ance and not for definitive diagnosis.

• Lower Performance on Complex Queries: Struggles with multi-concept or

rare medical conditions.

• Hardware Dependency: Performance and response accuracy improve signifi-

cantly on GPU-based systems.

• Privacy and Security Risks: Handling sensitive user health data requires
strict safeguards and regulatory compliance.

• Cannot Replace Doctors: Final diagnosis and treatment decisions must al-
ways involve medical professionals.

Department of AI ML,Vemana IT 21 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

2.9 A Chatbot-Based Question and Answer System for the

Auxiliary Diagnosis of Chronic Diseases Based on Large
Language Model”
Sainan Zhang and Jisung Song. Scientific Reports (Nature Portfolio), Volume 14,
Article 17118, 2024. Received: 27 November 2023 Accepted: 11 July 2024 Published:
2024.
Overview:
The paper titled “A Chatbot-Based Question and Answer System for the Auxiliary
Diagnosis of Chronic Diseases Based on Large Language Model” presents the design,
development, and evaluation of an AI-powered medical chatbot system, named Chat
Ella, aimed at assisting in the auxiliary diagnosis of chronic diseases. The study
focuses on addressing the growing global burden of chronic diseases and the increasing
demand for remote, cost-effective, and accessible healthcare solutions.
The authors begin by highlighting that nearly 25 of adults worldwide suffer from
one or more chronic conditions, which significantly impacts healthcare systems in
terms of cost, workload, and resource allocation. Long waiting times, limited avail-
ability of doctors, and the increasing preference for online medical consultation have
motivated the development of intelligent telemedicine systems. However, reliable and
accurate online diagnostic tools for chronic diseases remain limited, motivating this
research.
The proposed system leverages a fine-tuned GPT-2 Large Language Model to in-
terpret patient-provided symptom descriptions and predict possible chronic diseases.
Unlike conventional clinical decision support systems (CDSS) that rely on rule-based
or classical machine-learning approaches, Chat Ella utilizes deep learning and natural
language understanding to enable conversational, context-aware diagnosis. The model
was trained and fine-tuned using a Kaggle-sourced medical dataset containing symp-
tom descriptions for 24 common chronic diseases, including diabetes, hypertension,
asthma, arthritis, and gastrointestinal disorders.
The system architecture consists of a backend diagnostic engine and a frontend
conversational interface. The backend performs symptom classification and disease
probability ranking using the GPT-2 model, while the frontend—implemented using
[Link]—provides an intuitive chat-based interface for user interaction. Upon receiv-
ing symptom input, the system compares user symptoms with disease patterns stored
in the database, ranks potential diseases based on probability, and returns the most
likely diagnostic suggestions in real time.
Extensive evaluation was conducted using multiple metrics such as accuracy, preci-
sion, recall, F1-score, and Area Under the Curve (AUC). Experimental results demon-

Department of AI ML,Vemana IT 22 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

strate 97.5 accuracy and an AUC of 0.999, indicating excellent discriminative capa-
bility. In addition, a Chatbot Usability Questionnaire (CUQ) study involving 64 par-
ticipants was conducted to assess user experience. The system achieved an average
CUQ score of 68.31, surpassing standard usability benchmarks, with users reporting
high satisfaction, ease of use, and convenience for daily medical consultation.
The authors also discuss key limitations, including reliance on English-only in-
teraction, limited dataset size, lack of explainability due to the black-box nature of
LLMs, and the need for clinical validation before real-world deployment. Despite
these limitations, the study concludes that Chat Ella demonstrates strong potential
as a household-level auxiliary medical diagnostic tool for chronic disease management
and telemedicine support
Advantages:

• High Diagnostic Accuracy: Achieves strong performance metrics for chronic

disease prediction.

• Conversational Interface: Enables natural and user-friendly interaction.

• Reduced Doctor Workload: Handles preliminary symptom analysis and

guidance.

• Cost-Effective Telemedicine Support: Suitable for home-based and remote

consultation.

• Validated Usability: Supported by CUQ-based user satisfaction evaluation.

Disadvantages:

• Text-Only Interaction: Does not support voice or image inputs in its current
form.

• Limited Dataset Size: Requires larger and more diverse medical datasets for
generalization.

• Black-Box Model: GPT-2 lacks explainability, limiting clinical transparency.

• Language Limitation: Primarily supports English-language symptom de-

scriptions.

• Clinical Deployment Constraints: Requires extensive validation before real-

world medical use.

Department of AI ML,Vemana IT 23 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

2.10 “PharmaLLM: A Medicine Prescriber Chatbot Exploiting

Open-Source Large Language Models”
Ayesha Azam, Zubaira Naz, and Muhammad Usman Ghani Khan. Published in
Human-Centric Intelligent Systems, 2024. Received: 2 August 2024 Accepted: 17
October 2024 Published Online: 19 November 2024.
Overview:
The paper titled “PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-
Source Large Language Models” presents the development and evaluation of an LLM-
powered medical chatbot designed to provide accurate medicine prescription informa-
tion using open-source large language models. The study addresses growing concerns
regarding the reliability and safety of generic LLMs when applied directly to health-
care without domain-specific fine-tuning.
The authors highlight that while large language models such as GPT-3, GPT-4,
and LLaMA have shown impressive language understanding capabilities, their direct
use in medical applications can lead to inaccurate or potentially harmful advice. This
risk is especially high when models are not trained on curated medical datasets. To
overcome this limitation, the study proposes PharmaLLM, a fine-tuned version of the
Tiny LLaMA (LLaMA-2 variant) model, optimized specifically for medicine-related
queries.
A key contribution of this work is the use of Parameter-Efficient Fine-Tuning
(PEFT) through Low-Rank Adaptation (LoRA). The authors employ LoRA with
a rank of 16, a learning rate of 2e-4, batch size of 12, and three training epochs to
efficiently adapt the model while minimizing computational cost. This approach en-
ables the chatbot to achieve strong performance even on resource-constrained systems,
making it suitable for deployment in low-resource healthcare environments.
The system is trained on the “EDA | 11,000 Medicines” Kaggle dataset, which
contains detailed information about medicine names, compositions, uses, side effects,
manufacturers, and user satisfaction ratings. During preprocessing, medicines with
poor reviews (below 70) were removed to enhance the reliability of recommendations.
The dataset was converted into structured sentence formats using GPT-4-assisted
preprocessing, enabling effective fine-tuning of the LLM.

Advantages:

• Improved Healthcare Accessibility: Provides medical guidance and prelim-

inary consultation support before visiting a doctor.

• Context-Aware Responses: Uses retrieval-based grounding to generate.

Department of AI ML,Vemana IT 24 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

• Cost-Effective Solution: Reduces unnecessary clinical visits for minor and

non-critical health concerns.

• Scalable Architecture: Capable of serving a large number of users simultane-

ously without performance degradation.

• Dataset-Driven Evaluation: Validated using the MedMCQA medical bench-

mark dataset, ensuring research-backed performance.

Disadvantages:

• Limited Diagnostic Accuracy: Suitable only for preliminary medical guid-

ance and not for definitive diagnosis.

• Lower Performance on Complex Queries: Struggles with multi-concept or

rare medical conditions.

• Hardware Dependency: Performance and response accuracy improve signifi-

cantly on GPU-based systems.

• Privacy and Security Risks: Handling sensitive user health data requires
strict safeguards and regulatory compliance.

• Cannot Replace Doctors: Final diagnosis and treatment decisions must al-
ways involve medical professionals.

2.11 Comparative Analysis

When evaluating digital healthcare solutions, it is essential to understand how dif-

ferent systems address challenges such as medical accessibility, diagnostic accuracy,
patient interaction, and scalability. Traditional healthcare models like in-person doc-
tor consultations have long been the backbone of medical practice. However, with
the rapid growth of artificial intelligence and telemedicine, newer solutions such as
rule-based medical chatbots and AI-powered diagnostic systems have emerged. Each
approach offers distinct advantages and limitations, which the proposed AI Doctor
system aims to overcome.
Traditional hospital-based consultations provide highly reliable medical diagnosis
through trained professionals and clinical examination. Doctors can interpret complex
symptoms, medical history, and emotional cues, ensuring accurate decision-making.
However, this approach is often constrained by limited availability, high consultation
costs, long waiting times, and poor accessibility in rural or underserved regions. Addi-
tionally, healthcare systems are frequently overburdened, leading to delayed diagnosis
and treatment.

Department of AI ML,Vemana IT 25 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

Table 2.1 Comparative Analysis

[Link]. FEATURE/ TRADITIONAL RULE-BASED TEXT-BASED PROPOSED

CRITERIA DOCTOR MEDICAL MEDICAL AI DOCTOR
CONSULTA- SYSTEMS CHATBOTS SYSTEM
TION
1 DIAGNOSIS CLINICAL EX- RULE-BASED TEXT-BASED MULTIMODAL
METHOD AMINATION LOGIC AI REASONING AI(VOICE+
IMAGE+LLM)
2 ACCESSIBILITY LIMITED 24/7 BASIC 24/7 TEXT 24/7 MULTI-
AVAILABIL- ONLY MODAL AC-
ITY CESS
3 SYMPTOM IN- IN-PERSON PREDEFINED TEXT INPUT VOICE + TEXT
PUT MODE VERBAL QUESTIONS INPUT
4 IMAGE-BASED MANUAL IN- NOT AVAIL- NOT SUP- SKIN / REPORT
DIAGNOSIS SPECTION ABLE PORTED / PRESCRIP-
TION ANALY-
SIS
5 CONTEXT HIGH (HUMAN) NONE LIMITED MULTI-TURN
AWARENESS SINGLE-TURN LLM CONTEXT
RETENTION
6 TECHNOLOGY LOW (HUMAN- LOW MODERATE HIGH (CLOUD
DEPENDENCE BASED) AI)
7 USER TRAIN- MINIMAL MINIMAL MINIMAL MINIMAL
ING RE- (VOICE-
QUIRED DRIVEN)
8 COST HIGH LOW MEDIUM MEDIUM
9 SCALABILITY LIMITED HIGH HIGH VERY HIGH
(CLOUD-
BASED)
10 ACCESSIBILITY LOCATION DE- GENERIC LITERATE SMARTPHONE-
TO USERS PENDENT USERS USERS BASED, VOICE
CONTROLLED

Rule-based medical chatbots were introduced to address some of these challenges

by providing basic medical information and symptom checking. These systems op-
erate on predefined decision trees and static rules, offering limited interaction and
medical guidance. While they are easy to deploy and cost-effective, they lack con-
textual understanding, adaptability, and the ability to handle complex or ambiguous
medical queries. Their responses are often generic and may fail in multi-turn conver-
sations or uncommon medical scenarios.
Text-based AI chatbots powered by Natural Language Processing (NLP) represent
a significant improvement over rule-based systems. They can understand free-text
symptom descriptions, generate more natural responses, and support basic diagnostic
reasoning. However, most text-only chatbots are limited to a single modality and rely
entirely on user-provided textual input.
In contrast, the proposed AI Doctor system introduces a comprehensive, multi-
modal healthcare solution by integrating voice input, image-based diagnosis, prescrip-
tion scanning, and Large Language Model (LLM) reasoning into a unified platform.
Unlike traditional systems that depend solely on text or human availability, the AI

Department of AI ML,Vemana IT 26 2025-26

AI DOCTOR MEDICAL CHATBOT LITERATURE SURVEY

Doctor enables users to describe symptoms using voice, upload medical images, and
receive intelligent, context-aware medical assessments in real time. This makes the
system particularly useful for elderly users, visually impaired individuals, and patients
with limited literacy.
The AI Doctor also surpasses existing chatbot systems in terms of clinical relevance
and interaction quality. By combining speech recognition, computer vision, and LLM-
based medical reasoning, the system can analyze diverse data sources simultaneously,
leading to more informed and personalized responses. Unlike static rule-based tools,
it adapts dynamically to user input and maintains conversational context across mul-
tiple interactions. Moreover, by operating as a web-based or mobile application, the
AI Doctor achieves scalability while keeping deployment costs moderate.
Although the AI Doctor relies heavily on advanced computational models and
requires careful handling of medical data privacy, its design prioritizes accessibility,
efficiency, and user-centric interaction. By leveraging widely available smartphone
hardware and cloud-based AI services, the system balances functionality with afford-
ability.
In summary, while traditional consultations and earlier chatbot systems have played
an important role in healthcare delivery, their limitations in accessibility, scalability,
and multimodal understanding necessitate more intelligent solutions. The proposed
AI Doctor effectively bridges these gaps by providing a scalable, interactive, and mul-
timodal virtual healthcare assistant. This positions it as a superior and future-ready
alternative for preliminary medical consultation, especially in telemedicine and remote
healthcare environments.

Department of AI ML,Vemana IT 27 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM ANALYSIS

CHAPTER 3
SYSTEM ANALYSIS

3.1 Existing System

Existing digital healthcare systems mainly include online symptom checkers, telemed
icine platforms, and basic medical chatbots. Most of these systems rely on text-based
interaction, requiring users to manually type their symptoms, which is not conve-
nient for elderly patients or users with low literacy. They do not support natural
voice-based communication, making patient interaction less intuitive.
Current medical chatbots generally cannot analyze medical images such as skin
conditions, wounds, or scanned prescriptions. Patients must depend on verbal descrip-
tions alone, which may lead to inaccurate or incomplete medical guidance. Addition-
ally, many systems lack multilingual support, limiting accessibility for non-English-
speaking users.
Telemedicine applications connect patients to doctors through calls or video con-
sultations, but these services are often costly, time-limited, and dependent on doctor
availability. During emergencies or in remote areas, immediate access to medical
professionals may not be possible.
Some AI-based healthcare systems exist, but they often require expensive infras-
tructure or subscriptions and provide limited personalization. Most do not maintain
detailed consultation history or provide automated prescription analysis.
These limitations highlight the need for a simple, affordable, and intelligent med-
ical assistant that supports voice, image, and text inputs, offers real-time medical
guidance, and improves healthcare accessibility through a single unified platform.

3.2 Proposed System

The proposed AI Doctor Medical Chatbot with Multimodal LLM is designed to
overcome the limitations of existing digital healthcare systems by integrating speech
processing, computer vision, and large language models into a single intelligent med-
ical assistant. Instead of relying only on text-based interaction, the system enables
users to communicate naturally through voice, images, and text, acting as a “digital
medical assistant.”
The system accepts spoken symptoms through a microphone and converts them
into text using a Speech-to-Text model such as OpenAI Whisper. It also allows
users to upload medical images such as skin conditions, wounds, eye images, and
prescriptions. These multimodal inputs are processed using computer vision and
OCR techniques to extract meaningful medical information.

Department of AI ML,Vemana IT 28 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM ANALYSIS

A multimodal Large Language Model serves as the core reasoning engine of the sys-
tem. It analyzes patient symptoms, image-based findings, and extracted text to gen-
erate possible medical conditions, assess severity, and provide appropriate healthcare
recommendations. The system responds in both text and natural speech, improving
accessibility for elderly and low-literacy users.
To enhance accessibility, the chatbot supports multilingual interaction, allowing
users to communicate in languages such as English, Kannada, Hindi, Tamil, and
Telugu. A Text-to-Speech module converts the AI-generated response into clear audio
output for hands-free consultation.
The system also maintains secure user profiles and consultation history, enabling
continuity of care and reference for future interactions. All components are designed
to run on standard smartphones or laptops without the need for specialized medical
hardware, making the solution affordable and scalable.
Figure 3.1 (placeholder) illustrates the overall system architecture, showing the
flow from multimodal user input to AI processing and final voice/text output.
Key Features of the Proposed System:

• Multimodal interaction using text, voice, and medical images.

• AI-driven symptom analysis using Large Language Models.

• Medical image and prescription analysis using computer vision and OCR.

• Multilingual support for improved healthcare accessibility.

• Text-to-Speech output for voice-based medical guidance.

• Secure user authentication and consultation history storage.

• Runs on standard camera- and microphone-enabled devices.

3.3 Feasibility Study

The proposed system is technically feasible as it relies on widely available and
well-established technologies. The core components include Large Language Models
(LLMs), Speech-to-Text, Computer Vision, OCR, and Text-to-Speech, all of which
are accessible through open-source libraries or cloud-based APIs.
The system requires only standard hardware such as a camera, microphone, and
internet-enabled device (smartphone or laptop). No specialized medical hardware or
sensors are required. Backend development using Python and FastAPI and frontend
development using React or Gradio ensures smooth integration and scalability.

Department of AI ML,Vemana IT 29 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM ANALYSIS

3.3.1 Technical Feasibility

The proposed system is technically feasible as it relies on widely available and
well-established technologies. The core components include Large Language Models
(LLMs), Speech-to-Text, Computer Vision, OCR, and Text-to-Speech, all of which
are accessible through open-source libraries or cloud-based APIs.
The system requires only standard hardware such as a camera, microphone, and
internet-enabled device (smartphone or laptop). No specialized medical hardware or
sensors are required. Backend development using Python and FastAPI and frontend
development using React or Gradio ensures smooth integration and scalability. The
modular architecture allows easy testing, maintenance, and future upgrades, making
the system technically reliable.
3.3.2 Operational Feasibility
The system is designed to be simple, intuitive, and accessible for users of all age
groups and technical backgrounds. Voice-based interaction enables users to describe
symptoms naturally without typing, which is especially beneficial for elderly users
and individuals with disabilities.
The chatbot provides real-time responses through both text and voice, ensuring
clear and timely medical guidance. Multilingual support further enhances usability by
allowing users to interact in their preferred language. Secure user authentication and
consultation history improve continuity of care without complicating user interaction.
Since the system runs as a web-based application, users can access it anytime with-
out special installation or training, making it operationally feasible for daily use.

Department of AI ML,Vemana IT 30 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM SPECIFICATION

CHAPTER 4

SYSTEM SPECIFICATION

4.1 Hardware Requirements

The AI Doctor Medical Chatbot with Multimodal LLM is designed using stan-
dard consumer-grade hardware to ensure affordability, scalability, and ease of access
for users. The system does not require specialized medical equipment or high-end
hardware. It relies on commonly available devices such as a camera, microphone, and
a computing device capable of handling AI-based processing. This hardware setup
ensures smooth interaction, real-time response, and reliable performance for medical
consultation.

4.1.1 Processor
A mid-range processor such as Intel Core i5 / i7 or AMD Ryzen 5 / Ryzen 7 (3.0
GHz or higher) is sufficient for handling backend processing, AI inference requests,
image analysis, and audio processing. Since most heavy AI computation is performed
using cloud-based APIs, extreme processing power is not mandatory.

4.1.2 Memory
minimum of 8 GB RAM is required for running the backend server, frontend inter-
face, and handling concurrent API calls. For smoother performance and development
environments, 16 GB RAM is recommended.

4.1.3 Camera
A built-in or external webcam (minimum 720p resolution) is required to cap-
ture medical images such as skin conditions, wounds, or prescription images. Higher
resolution cameras improve image clarity and diagnostic accuracy.

4.1.4 Audio Components

A microphone is required to capture user voice input for speech-based medical
consultation. Speakers or headphones are used to deliver clear Text-to-Speech audio
responses, enabling hands-free interaction and better accessibility.

Department of AI ML,Vemana IT 31 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM SPECIFICATION

4.2 Software Requirements

The software forms the core intelligence of the AI Doctor Medical Chatbot with
Multimodal LLM. It is responsible for handling speech recognition, medical image
analysis, symptom reasoning, multilingual interaction, and voice-based responses.
The system follows a web-based architecture, where the backend performs all AI
processing and decision-making, while the frontend provides an interactive interface
for users to communicate through text, voice, and images.

4.2.1 Operating System

The system is designed to run on commonly used operating systems such as
Windows 10/11, macOS, and Ubuntu Linux. These platforms fully support Python,
JavaScript, and modern web frameworks required for AI integration. No specialized
operating system is required, making the system easily deployable on standard devices.

4.2.2 Programming Languages

The system uses two primary programming languages:

(a) Python 3.8+ (Backend)

Python is used for:

• Implementing the backend server using FastAPI.
• Integrating Large Language Models (LLMs).
• Processing medical images and prescriptions.
• Managing Speech-to-Text and Text-to-Speech services.
• Handling OCR and AI-based medical reasoning
Python is chosen for its strong AI ecosystem, simplicity, and compatibility with ma-
chine learning libraries.

(b) JavaScript / TypeScript (Frontend)

JavaScript (with [Link] or Gradio) is used for:

• Building the web-based user interface.
• Capturing voice input and image uploads.
• Displaying medical responses and consultation history.
• Playing Text-to-Speech audio output.
JavaScript ensures a smooth, responsive, and browser-accessible interface.

Department of AI ML,Vemana IT 32 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM SPECIFICATION

4.2.3 Frameworks and Libraries

(a)FastAPI

• Python-based backend framework.

• Handles API requests for symptom analysis, image upload, voice input, and user
authentication.

• Enables fast and scalable server-side processing.

(b) [Link] / Gradio

• Used to develop the frontend interface

• Provides components for chat interaction, voice recording, and image upload.

• Ensures accessibility and ease of use.

(c) Large Language Models (LLMs)

• Used for symptom analysis and medical response generation.

• Supports multimodal input (text + image context).

• Enables natural and conversational medical guidance.

(d)Computer Vision Libraries

• Used for analyzing medical images such as skin conditions and prescriptions.

• Supports preprocessing and feature extraction.

These frameworks work together to create an intelligent, interactive, and scalable

medical assistant.

4.2.4 Speech and OCR Services

(a) Speech-to-Text (STT)

• Converts user speech into text (e.g., OpenAI Whisper).

• Enables voice-based medical consultation

TIt allows the system to communicate with the user naturally, without needing extra
apps or devices.

Department of AI ML,Vemana IT 33 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM SPECIFICATION

(b) Text-to-Speech (TTS)

• Converts AI-generated medical responses into natural voice output.

• Improves accessibility and hands-free interaction.

(c)OCR Services

• Extracts text from prescription images and medical documents.

• Supports multiple languages such as English, Kannada, and Hindi.

These services allow the system to listen, read, understand, and speak naturally.

4.2.5 External APIs

Medical AI and Language APIs

• Used for Large Language Model inference and multimodal processing

• Provide accurate and real-time medical reasoning.

These APIs enhance system intelligence, while the core application logic remains
independent and modular.

4.2.6 Other Tools

(a) Backend Serve

• FastAPI development server or production server (e.g., Uvicorn/Nginx).

• Ensures smooth communication between frontend and backend.

(b) Package Managers

Package managers used to install:

• pip for Python dependencies

• npm for JavaScript and frontend libraries

These tools simplify installation, updates, and long-term maintenance of the project.

Department of AI ML,Vemana IT 34 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM SPECIFICATION

4.3 Functional Requirements

Functional requirements describe what the AI Doctor Medical Chatbot with Mul-
timodal LLM must perform to meet its objectives. The system continuously processes
user inputs in the form of text, voice, and images, analyzes medical information us-
ing AI models, and provides healthcare guidance through intelligent text and voice
responses.
The key functional requirements of the system are described below:

• Symptom Analysis: The system accepts user-described symptoms through

text or voice input and analyzes them using a Large Language Model. It identifies
possible medical conditions, assesses severity levels, and provides preliminary
healthcare guidance.

• Speech-to-Text Processing (STT Module): The system converts patient

voice input into text using a Speech-to-Text model. This enables natural, hands-
free medical consultation for users who prefer speaking over typing.

• Medical Image Analysis: The system allows users to upload medical images
such as skin conditions, wounds, eye images, or tongue images. These images are
analyzed using computer vision techniques to assist in identifying visible medical
issues.

• Prescription Text Recognition (OCR Module): The system processes pre-

scription images and medical documents using OCR services. Extracted text such
as medicine names and dosages is analyzed and read aloud for user understand-
ing.

• AI-Generated Medical Response: Based on symptom input, image analysis,

and extracted text, the system generates clear and structured medical responses
using a multimodal LLM. The response includes possible conditions, precautions,
and recommended next steps.

• Text-to-Speech Output (TTS Module): The system converts AI- generated

medical responses into natural spoken audio. Voice output must be clear and
immediate, enabling hands-free interaction and improved accessibility.

• User Interaction and Accessibility Modes: The system supports:

– Voice Mode: Fully voice-driven interaction for hands-free consultation

– Text Mode: Text-based chat interface for users who prefer typing Users can
switch between modes easily based on preference.

Department of AI ML,Vemana IT 35 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM SPECIFICATION

• Multilingual Support: The system supports interaction in multiple languages

such as English, Kannada, Hindi, Tamil, and Telugu. Both text and voice re-
sponses are delivered in the user’s selected language.

• Authentication and History Management: User The system allows users

to register and log in securely. It stores previous consultation history, including
symptoms and responses, for future reference.

• Performance Requirements: The system must provide responses in near real

time. The delay between user input and AI-generated response should not exceed
2–3 seconds under normal conditions.

• Error Handling: The system must handle errors such as unclear voice input,
unsupported image formats, OCR failures, or API issues gracefully. In such cases,
it should provide user-friendly messages such as: “Please repeat your symptoms,”
“Image not clear,” or “Unable to process request.”

4.4 Non-Functional Requirements

Non-functional requirements describe the quality attributes and operational char-
acteristics of the AI Doctor Medical Chatbot with Multimodal LLM. These require-
ments ensure that the system performs efficiently, securely, and reliably in real-world
healthcare scenarios, while providing a smooth and accessible experience for users.
The following non-functional requirements define the overall performance, usability,
reliability, and maintainability of the system:

• Performance: The system must process user requests and generate AI responses
in near real time. The response time between user input and medical output
should not exceed 2–3 seconds under normal network conditions. Speech output
should be delivered immediately after response generation.

• Reliability: The system should operate consistently during continuous usage

without crashes or unexpected failures. In case of API errors, unclear inputs, or
processing failures, the system must retry or notify the user appropriately.

• Accuracy: Symptom analysis, medical image interpretation, and OCR-based

prescription reading should achieve high accuracy. The system aims for above
90accuracy in speech recognition, image processing, and text extraction to ensure
reliable medical guidance.

• Usability: The user interface must be simple, intuitive, and easy to navigate
for users of all age groups. Voice responses should be clear, natural.

Department of AI ML,Vemana IT 36 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM SPECIFICATION

• Accessibility: The system must support voice-based interaction, adjustable

speech rate, volume control, and multilingual output. These features ensure
accessibility for elderly users, visually impaired individuals, and users with low
literacy.

• Maintainability: The system follows a modular software architecture, where

components such as symptom analysis, image processing, OCR, and speech mod-
ules can be updated or replaced independently without affecting the entire sys-
tem.

• Portability: The application should run smoothly on Windows, macOS, and

Linux systems and be accessible through standard web browsers. It should also
support deployment on smartphones and laptops without major changes.

• Scalability: The system should support future expansion, such as adding ad-
vanced diagnostic models, wearable device integration, electronic health record
(EHR) connectivity, and additional language support.

• Security: All external API communications must use secure HTTPS connec-
tions. User authentication data and consultation history should be protected,
and sensitive medical information must not be exposed to unauthorized users.

• Privacy: The system should minimize storage of sensitive data such as audio
recordings and medical images. Any stored data must be user-consented and
handled securely.

• Availability: The system should be available for use at all times, subject to in-
ternet connectivity and third-party API availability. Graceful degradation must
be provided if external services are temporarily unavailable.

• Data Integrity: Medical data, OCR results, and consultation history must
remain accurate and consistent without corruption during processing or storage.

• Safety: The system must clearly state that it provides preliminary medical guid-
ance and is not a replacement for professional healthcare. Critical or uncertain
cases should prompt users to consult a qualified doctor.

• Efficiency: The system should optimize resource usage, minimizing CPU and
memory consumption while maintaining fast response times, especially on portable
devices.

Department of AI ML,Vemana IT 37 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

CHAPTER 5

PROJECT DESCRIPTION

5.1 Problem Definition

Healthcare accessibility remains a significant challenge, especially in rural and re-
mote areas where the availability of qualified medical professionals is limited. Patients
often face long waiting times, high consultation costs, and difficulty in accessing timely
medical advice. Even in urban regions, overcrowded hospitals and clinics reduce the
quality of personalized care, forcing patients to rely on unreliable online information
or delay treatment.
Traditional healthcare support systems such as symptom-checker websites and
telemed icine platforms provide limited assistance. Most of these systems rely on
text-based interaction, which is inconvenient for elderly users, visually impaired in-
dividuals, and patients with low literacy levels. Additionally, they lack the ability
to analyze medical images such as skin conditions, wounds, or scanned prescriptions,
which are critical for accurate preliminary diagnosis. Language barriers further limit
their effectiveness, as many systems support only English.
While modern AI chatbots exist, many are either expensive, cloud-restricted, or
not specifically designed for healthcare use. They often fail to integrate speech input,
image analysis, and contextual medical reasoning into a single platform. This creates
a gap between basic digital health tools and intelligent, interactive medical assistance.
To address these limitations, this project proposes the AI Doctor Medical Chat-
bot with Multimodal LLM, an intelligent healthcare assistant that integrates speech
processing, computer vision, and Large Language Models. The system allows users
to describe symptoms through voice or text, upload medical images, and receive AI-
generated medical guidance in both text and speech formats.
The goal of this project is to provide an accessible, cost-effective, and intelligent
medical assistance system that enhances early-stage diagnosis, reduces dependency on
immediate physical consultations, and empowers users to make informed healthcare
decisions. By using commonly available devices such as smartphones and laptops,
the AI Doctor acts as a “digital healthcare assistant” that bridges the gap between
traditional healthcare services and modern AI-driven solutions.

5.2 Overview of the Project

The AI Doctor Medical Chatbot with Multimodal LLM is an AI-based healthcare
assistant designed to provide preliminary medical guidance through text, voice, and
image-based interaction. Acting as a “digital doctor,” the system interprets user in-
puts and delivers real-time medical advice using advanced AI technologies.

Department of AI ML,Vemana IT 38 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

Users can interact with the system by speaking their symptoms, typing medical
queries, or uploading images such as skin conditions, wounds, eye images, or pre-
scriptions. Speech input is converted into text using a Speech-to-Text model, while
uploaded images are processed using computer vision and OCR techniques. All ex-
tracted information is analyzed by a multimodal Large Language Model, which serves
as the core reasoning engine.
The backend of the system is developed using Python and FastAPI, handling AI
model integration, data processing, and response generation. The frontend, built us-
ing [Link] or Gradio, provides an intuitive web-based interface for user interaction.
The system delivers responses in both text and natural voice output using Text-to-
Speech technology, enabling hands-free consultation.
A multilingual support module allows users to communicate in languages such as
English, Kannada, Hindi, Tamil, and Telugu, improving accessibility across diverse
user groups. Secure user authentication and consultation history storage ensure pri-
vacy and continuity of medical guidance.
The modular architecture of the AI Doctor system supports future enhancements
such as advanced diagnostic models, wearable health integration, offline AI process-
ing, and electronic health record connectivity. Overall, the system offers a scalable,
affordable, and intelligent healthcare solution that improves accessibility, efficiency,
and user confidence in everyday medical consultation scenarios.

5.3 System Architecture

The system architecture of the AI Doctor Medical Chatbot with Multimodal LLM
is divided into layered components (see Fig. 5.1) to ensure modularity, scalability, and
efficient processing.(see Fig. 5.1).

• Input Layer: Captures raw user inputs. A microphone is used to capture pa-
tient voice input for symptom description, while a camera or file upload module
captures medical images such as skin conditions, wounds, eye images, or pre-
scriptions. Text input is also accepted through the web interface.

• Processing Layer: Speech input is processed by the Speech-to-Text (STT)

module to convert voice into text. Uploaded images are processed by computer
vision and OCR modules to extract visual and textual medical information. All
extracted data is prepared and formatted for AI-based reasoning.

• Intelligence Layer: A Multimodal Large Language Model (LLM) serves as

the core reasoning engine. It analyzes patient symptoms, image-based findings,
and extracted prescription text to generate medical insights, possible conditions,
severity assessment, and healthcare recommendations.

Department of AI ML,Vemana IT 39 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

• Control and Management Layer: This layer manages user authentication,

consultation history, language selection, and request routing. It also coordinates
interactions between AI modules and prioritizes responses when multiple inputs
are processed simultaneously.

• Output Layer: The system delivers responses through text output and Text-to-
Speech (TTS) audio. The web interface displays structured medical advice, while
the TTS module provides natural spoken responses for hands-free consultation.

Figure 5.1: System Architecture

5.4 Module Description

The AI Doctor Medical Chatbot with Multimodal LLM is organized into three
major modules that work together to provide intelligent medical assistance through
symptom analysis, medical image interpretation, and voice-based interaction. Each
module plays a crucial role in ensuring that the system operates accurately, securely,
and efficiently in real-world healthcare scenarios.

Department of AI ML,Vemana IT 40 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

5.4.1 Multimodal Input Processing Module

The Multimodal Input Processing Module is responsible for capturing and process-
ing user inputs in the form of text, voice, and medical images. It prepares all medical
data for AI-based reasoning.

• Speech Input Processing: The system captures patient voice input using a
microphone and converts it into text using a Speech-to-Text (STT) model. This
enables natural, hands-free symptom description.

• Medical Image Processing: Users can upload medical images such as skin
conditions, wounds, eye images, or tongue images. Computer vision techniques
are applied to extract relevant visual features.

• Prescription Text Recognition (OCR): Prescription images and medical

documents are processed using OCR services to extract medicine names, dosage
details, and instructions.

• Real-Time Data Preparation: All extracted text and image-based informa-

tion is cleaned and formatted before being forwarded to the AI reasoning module.

This module forms the foundation of the system by accurately capturing and
preparing multimodal medical inputs.

5.4.2 AI Reasoning & Medical Response Module

The AI Reasoning and Medical Response Module acts as the core intelligence of
the system. It uses a Multimodal Large Language Model (LLM) to analyze medical
inputs and generate appropriate healthcare guidance.

• Symptom Analysis: The LLM analyzes patient symptoms provided through

text or voice to identify possible medical conditions and assess severity levels.

• Image-Based Medical Reasoning: Visual features extracted from medical

images are combined with symptom data to improve diagnostic understanding.

• Medical Recommendation Generation: The system generates structured

medical responses including possible conditions, precautions, self-care advice,
and recommendations for consulting a doctor.

• Multilingual Response Generation: Medical responses are generated in the

user’s selected language such as English, Kannada, Hindi, Tamil, or Telugu.

This module enables intelligent, contextual, and personalized medical assistance.

Department of AI ML,Vemana IT 41 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

5.4.3 Voice Assistant & User Interaction Module

The Voice Assistant and User Interaction Module manages all system outputs and
ensures smooth, accessible communication between the user and the AI Doctor.

• Text-to-Speech Output: AI-generated medical responses are converted into

natural spoken audio using a Text-to-Speech (TTS) engine. Speech rate, pitch,
and volume can be adjusted as per user preference.

• Voice Command Recognition: The system listens for commands such as

“describe symptoms,” “scan prescription,” “repeat advice,” or “change language”
and routes them to the appropriate module.

• User Interface Management: The web-based interface displays chat responses,

uploaded images, and consultation history, supporting both voice-based and text-
based interaction.

• Alert and Error Handling: In case of unclear input, unsupported images, or

processing errors, the system provides clear voice and text notifications to guide
the user.

5.5 Data Flow Diagrams (DFD)

The Data Flow Diagram (DFD) illustrates how data flows through the AI Doctor
Medical Chatbot with Multimodal LLM system. It shows the interaction between the
user, microphone, image input module, backend processing units, Large Language
Model, OCR engine, speech modules, and the user interface. The DFD helps in
understanding how patient symptoms, medical images, voice commands, and AI-
generated responses move through the system to provide intelligent medical assistance.
It also represents the coordination between the backend server (FastAPI), frontend
interface ([Link] / Gradio), and AI services.

5.5.1 Level 0 – Context Diagram

The Level 0 DFD (Figure 5.2) provides a high-level overview of the AI Doctor Medical
Chatbot as a single unified system interacting with various external entities.
Description:
• The User (Patient) interacts with the system by providing symptoms through
text or voice input and by uploading medical images. The user receives medical
guidance in text and audio form.

• A Microphone captures the user’s voice input, which is processed by the Speech-
to-Text (STT) module.

Department of AI ML,Vemana IT 42 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

• A Camera / Image Upload Module captures or uploads medical images such

as skin conditions, wounds, or prescriptions and sends them to the backend for
analysis.

• The Backend Server (FastAPI) processes symptom data, medical images,

OCR requests, and manages communication with AI models.

• The Large Language Model (LLM) analyzes multimodal inputs and gener-
ates medical insights, recommendations, and responses.

• The OCR Service receives medical document or prescription images from the
backend and returns extracted text.

• The Frontend Interface ([Link] / Gradio) displays chatbot responses,

medical insights, and plays audio output.

• The Text-to-Speech (TTS) Engine converts AI-generated medical responses

into spoken audio and delivers them to the user.

Figure 5.2: Level 0 DFD – Context Diagra

5.5.2 Level 1 – Detailed Data Flow Diagram

The Level 1 DFD (Figure 5.3) provides a detailed view of how data flows inter-
nally through the AI Doctor Medical Chatbot with Multimodal LLM. It breaks down
the system into multiple processes representing data acquisition, medical analysis, AI
reasoning, and response generation.

Department of AI ML,Vemana IT 43 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

Process 1: Data Acquisition

Inputs:

• Voice input from the user.

• Medical image uploads.

Source:

• Microphone.

• Web Interface (Image Upload / Text Input).

Process Description:

• The microphone captures real-time voice input.

• Voice input is converted into text using the STT engine.

• Medical images are uploaded and preprocessed.

• Text input is directly collected from the chat interface.

Outputs:

• Text-based symptoms.

• Preprocessed medical images.

Process 2: Symptom & Image Analysis

Inputs:

• Text-based symptoms.

• Medical image data.

Source:

• STT Module.

• Image Processing Module.

Process Description:

• The system analyzes symptoms provided by the user.

• Medical images are processed to extract relevant visual features.

• Extracted information is prepared for AI reasoning.

Department of AI ML,Vemana IT 44 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

Outputs:

• Structured symptom data.

• Image-based medical insights.

Process 3: AI Reasoning (LLM Processing)

Inputs:

• Symptom data.

• Image analysis results.

• OCR-extracted text.

Source:

• Multimodal Large Language Model (LLM).

Process Description:

• The LLM analyzes multimodal medical inputs.

• Possible medical conditions and severity levels are identified.

• Medical recommendations are generated.

Outputs:

• AI-generated medical responses.

• Diagnosis suggestions and advice.

Process 4: OCR & Prescription Text Analysis

Inputs:

• Prescription images.

• Medical documents.

Source:

• OCR Service API.

Process Description:

• The backend sends images to the OCR service.

• Extracted text such as medicine names and dosage is returned.

• OCR data is forwarded for AI analysis and explanation.

Department of AI ML,Vemana IT 45 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

Outputs:

• Recognized prescription text.

Process 5: Response Generation & Audio Output

Inputs:

• AI-generated medical response.

• OCR results.

Source:

• Text-to-Speech Engine.

• Frontend Interface.

Process Description:

• Medical responses are displayed on the UI.

• Text responses are converted into spoken audio.

• Voice output is delivered to the user.

Outputs:

• Spoken medical advice.

• Displayed chatbot responses.

Process 6: System Control & Feedback Loop

Inputs:

• User follow-up queries.

• System responses.

Source:

• Frontend UI.

Process Description:

• User interacts continuously with the chatbot.

• Context is maintained for follow-up consultation.

Outputs:

• Continuous medical consultation.

• Updated AI interaction cycle.

Department of AI ML,Vemana IT 46 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

Figure 5.3: Level 1 Data Flow Diagram

5.6 Use Case Diagram

The Use Case Diagram (Figure 5.4) illustrates the functional relationship between
the User (Patient) and the AI Doctor Medical Chatbot with Multimodal LLM Sys-
tem. It shows the various interactions the user can perform through text, voice, and
image inputs and how the system responds through symptom analysis, medical im-
age processing, prescription reading, and voice-based medical guidance. The system
boundary defines all internal operations performed by the AI Doctor Medical Chat-
bot, while the actor represents an external user interacting with the system using
natural language and multimedia inputs.

Department of AI ML,Vemana IT 47 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

Actors:
• User (Patient) — Primary Actor:
The main user who interacts with the system by describing symptoms through
voice or text and by uploading medical images. The user receives AI-generated
medical guidance, recommendations, and alerts through text and audio output.

Figure 5.4: Use Case Diagram

Use Cases:
• Describe Symptoms: The user provides health-related symptoms through
voice or text input. The system captures the input and prepares it for AI-based
medical analysis.

• Voice-Based Consultation: The user interacts with the system using voice
commands such as “Describe my symptoms,” “Repeat advice,” or “Change lan-
guage.” The Speech-to-Text module processes the voice input and triggers the
corresponding medical analysis.

• Medical Image Analysis: The user uploads medical images such as skin con-
ditions, wounds, eye images, or tongue images. The system analyzes the image
using computer vision techniques to assist in identifying visible medical condi-
tions.

• Prescription Reading (OCR): The user requests the system to read a pre-
scription or medical document. The image is sent to the OCR module, extracted
text is processed, and the system explains the prescription details in an under-
standable manner.

• Symptom Analysis & Diagnosis Assistance: The system analyzes the user’s
symptoms and image-based inputs.

Department of AI ML,Vemana IT 48 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

• Receive Medical Advice: Based on AI analysis, the system provides medical

advice, precautions, self-care suggestions, and recommendations on whether to
consult a healthcare professional.

• Text-to-Speech Medical Response: The system converts AI-generated med-

ical responses into spoken audio, allowing the user to receive guidance hands-free.

• Multilingual Interaction: The user can choose a preferred language such as

English, Kannada, Hindi, Tamil, or Telugu, and the system delivers medical
responses in the selected language.

• View Consultation History: The user can view or listen to previous consul-
tation records and AI responses for reference and continuity of care.

5.7 Sequence Diagram

The Sequence Diagram (Figure 5.5) illustrates the dynamic behavior of the AI
Doctor Medical Chatbot with Multimodal LLM by showing the sequence of inter-
actions between the system components over time. It demonstrates how data flows
between the user, microphone, image input module, backend server, OCR engine,
Large Language Model, and voice assistant during a medical consultation. This dia-
gram captures both the operational flow and decision-making sequence required for
intelligent and continuous healthcare assistance.

Objects Involved:

• User (Patient): Issues symptoms through voice or text input and receives
medical advice in spoken and text form.

• Microphone: Captures the user’s voice input for speech-based consultation.

• Camera / Image Upload Module: Captures or uploads medical images such

as skin conditions, wounds, or prescriptions.

• Backend Server (FastAPI): Central processing unit that manages speech-

to-text conversion, medical image analysis, OCR requests, AI reasoning, and
response generation.

• Speech-to-Text (STT) Engine: Converts user voice input into text for further
medical analysis.

• Multimodal Large Language Model (LLM): Analyzes symptoms, image-

based inputs, and extracted text to generate medical insights and recommenda-
tions.

Department of AI ML,Vemana IT 49 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

• OCR Service: Extracts text from prescription images and medical documents
when requested by the user.

• Voice Assistant (Text-to-Speech Engine): Delivers AI-generated medical

responses as spoken audio and handles voice-based interaction.

Sequence of Operations:

• System Initialization: The user starts the AI Doctor Medical Chatbot. The
frontend interface is initialized, and the backend server becomes ready to receive
user inputs.

• User Input Capture: The user provides symptoms through voice or text and
may upload a medical image. Voice input is captured through the microphone,
and images are captured through the image upload module.

• Speech Processing: The Speech-to-Text engine converts spoken symptoms

into text and forwards them to the backend server.

• Medical Image and OCR Processing: Uploaded medical images are ana-
lyzed. If the image contains text such as a prescription, it is sent to the OCR
service, which extracts readable text and returns it to the backend.

• AI Reasoning: The backend forwards symptom data, image analysis results,

and OCR text to the multimodal LLM. The LLM identifies possible medical
conditions, severity levels, and generates healthcare recommendations.

• Response Generation: The AI-generated medical response is structured into

clear guidance, precautions, and suggested next steps.

• Audio Output: The Voice Assistant converts the AI-generated response into
spoken audio using Text-to-Speech technology and delivers it to the user.

• User Feedback Loop: The user may ask follow-up questions or request clari-
fication. The system maintains conversational context and continues the consul-
tation process.

Department of AI ML,Vemana IT 50 2025-26

AI DOCTOR MEDICAL CHATBOT PROJECT DESCRIPTION

Figure 5.5: Sequence Diagram

Department of AI ML,Vemana IT 51 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM IMPLEMENTATION

CHAPTER 6

SYSTEM IMPLEMENTATION

6.1 Introduction

System implementation is the phase where the designed architecture and functional
modules are developed into a working prototype. This stage involves integrating in-
put devices such as the microphone and camera with software modules including
speech recognition, medical image analysis, OCR processing, Large Language Model
reasoning, and voice-based response generation. For the AI Doctor Medical Chatbot
with Multimodal LLM, the implementation combines artificial intelligence, computer
vision, natural language processing, and a web-based interface to deliver intelligent
medical consultation to users.

6.1.1 Hardware Implementation

The hardware layer of the system consists of input devices, audio interfaces, and
computing resources. These components work together to capture user inputs, pro-
cess medical information, and deliver AI-generated responses.

• Microphone Integration

– A standard microphone is used to capture user voice input.

– Speech-to-Text (STT) engine converts spoken symptoms into text.

• Camera / Image Input Integration

– A webcam or smartphone camera captures medical images such as skin

conditions, wounds, or prescriptions.

– Images are sent to the backend server for analysis and OCR processing.

• Audio Output System

– Speakers or headphones deliver real-time text-to-speech medical responses.

– Output includes diagnosis explanations, precautions, and medical advice.

• Computing Unit

– A laptop or smartphone with a minimum 3.0 GHz CPU performs AI infer-

ence and backend processing.

Department of AI ML,Vemana IT 52 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM IMPLEMENTATION

• Device Setup

– The microphone and camera are either built-in or externally connected.

– USB or wireless connections ensure stable data transmission.

– Proper device handling ensures safe and continuous operation.

The hardware subsystem provides the necessary input and output interfaces for
real-time medical consultation.

6.1.2 Software Implementation

The software implementation includes backend development, frontend design, AI

model integration, OCR services, and speech processing. It is divided into three
primary layers: AI Processing Backend, User Interface, and Speech/OCR Modules.
• Backend (FastAPI Server & AI Processing)

– Receives user symptoms through text or voice input.

– Processes medical images using computer vision techniques.

– Integrates a Multimodal Large Language Model for medical reasoning.

– Handles OCR requests for prescription and document reading.

– Manages API communication with the frontend using JSON responses.

• Frontend ([Link] / Gradio Interface)

The web interface acts as the primary dashboard for users and healthcare inter-
action.

– Displays chatbot-based medical consultation responses.

– Shows uploaded medical images and extracted text.

– Supports both text-based and voice-based interaction.

– Updates responses dynamically using API calls.

This interface ensures smooth and accessible interaction for users.

• Medical OCR Reader (OCR/TTS Module)

– Captures prescription or medical document images on user request.

– Sends images to the OCR service for text extraction.

Department of AI ML,Vemana IT 53 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM IMPLEMENTATION

– Extracted text is explained using the AI model.

– Supports multilingual output through text-to-speech.

• Voice Assistant (Text-to-Speech Module)

– Converts AI-generated medical responses into spoken audio.

– Supports adjustable speech speed and volume.

– Enables hands-free medical consultation.

6.1.3 Integration and Testing Setup

All modules were integrated to function as a unified AI-powered medical chatbot:

• User inputs are captured through the frontend interface.

• Speech-to-Text converts voice symptoms into text.

• Medical images and OCR text are processed by AI modules.

• The Multimodal LLM generates medical insights and advice.

• Text and audio responses are delivered with low latency.

• The system was tested with different symptoms and image samples.
Testing confirmed stable performance, accurate medical explanations, and clear voice
output suitable for real-time medical consultation.

Department of AI ML,Vemana IT 54 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM IMPLEMENTATION

6.2 Screenshots

Figure 6.1: AI Doctor Medical Chatbot User Interface

Figure 6.2: Medical Image Upload and Analysis Interface

Department of AI ML,Vemana IT 55 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM IMPLEMENTATION

Figure 6.3: Virtual AI Doctor UI

Figure 6.4: AI Doctor Generated Medical Assessment and Recommendations

Department of AI ML,Vemana IT 56 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM TESTING

CHAPTER 7

SYSTEM TESTING

System testing is performed to verify that the AI Doctor Medical Chatbot with
Multimodal LLM operates accurately, reliably, and safely under real-time usage con-
ditions. The goal of this phase is to ensure that all integrated components — including
the microphone, camera/image input module, backend AI processing, OCR engine,
Large Language Model, voice assistant, and user interface — work together seam-
lessly to meet the system’s objectives. Testing also validates system behavior across
different user inputs, image qualities, languages, and interaction patterns, ensuring
robustness, stability, and overall effectiveness in medical consultation scenarios.

7.1 Tests Conducted

The following tests were carried out to evaluate the functional and non-functional
aspects of the AI Doctor Medical Chatbot:
• Voice Input Capture Test: Ensures the microphone captures user speech
clearly for medical consultation.
• Speech-to-Text Accuracy Test: Validates accurate conversion of spoken
symptoms into text using the STT engine.
• Text-Based Symptom Input Test: Checks correct handling of typed symp-
tom descriptions in the chatbot interface.
• Medical Image Upload Test: Verifies that uploaded medical images (skin,
wound, prescription) are captured and processed correctly.
• Medical Image Analysis Test: Evaluates the system’s ability to analyze visual
medical data and extract relevant features.
• OCR Text Recognition Test: Evaluates OCR performance in extracting En-
glish, Kannada, and Hindi text from prescription images.
• AI Response Accuracy Test: Ensures the AI-generated medical advice is
relevant, coherent, and clinically meaningful.
• Voice Output (TTS) Test: Ensures spoken medical responses are clear, au-
dible, and delivered at the correct time.
• Voice Command Test: Checks whether user commands such as “Repeat ad-
vice,” “Read prescription,” or “Explain again” are correctly interpreted and ex-
ecuted.

Department of AI ML,Vemana IT 57 2025-26

AI DOCTOR MEDICAL CHATBOT SYSTEM TESTING

• Latency Test: Measures the time from user input to AI response, verifying that
it remains under 2 seconds (observed: 1–1.5 seconds).

• Full System Integration Test: Ensures that voice input, image analysis,
OCR, AI reasoning, and audio output operate together without performance
degradation.

7.2 Test Cases

Table 2: Test Case Results

ID Test Case De- Input / Con- Expected Actual Out- Result

scription dition Output put
TC-01 Voice Input Speak symp- Voice captured Clear audio Pass
Capture Test toms clearly captured
TC-02 Speech-to-Text Spoken symp- Correct text Accurate con- Pass
Accuracy Test toms conversion version
TC-03 Text Input Test Type symp- Symptoms pro- Processed cor- Pass
toms cessed correctly rectly
TC-04 Medical Image Upload Image accepted Image pro- Pass
Upload Test skin/wound cessed
image
TC-05 Medical Image Provide medi- Relevant fea- Extracted cor- Pass
Analysis Test cal image tures extracted rectly
TC-06 OCR English Prescription Correct text ex- Accurate OCR Pass
Test (English) traction output
TC-07 OCR Kan- Regional lan- Text extracted Extracted accu- Pass
nada/Hindi guage text and explained rately
Test
TC-08 AI Response Symptoms + Relevant medi- Relevant out- Pass
Accuracy Test image cal advice put
TC-09 Voice Output AI response Clear spoken Clear audio Pass
Test output
TC-10 Latency Test Full interaction <2 sec response 1.2 sec average Pass
time

Department of AI ML,Vemana IT 58 2025-26

AI DOCTOR MEDICAL CHATBOT CONCLUSION AND FUTURE ENHANCEMENTS

CHAPTER 8

CONCLUSION AND FUTURE

ENHANCEMENTS

8.1 Conclusion

The AI Doctor Medical Chatbot with Multimodal LLM has been successfully de-
signed, implemented, and tested as an intelligent healthcare assistance system that
provides real-time medical consultation through voice, text, and image-based inter-
action. The project integrates natural language processing, medical image analysis,
OCR, and speech technologies to deliver accurate and accessible healthcare guidance.

The system effectively analyzes user-provided symptoms, interprets medical im-

ages, extracts prescription text using OCR, and generates meaningful medical advice
using a Large Language Model. The integration of Text-to-Speech enables hands-free
interaction, improving accessibility for elderly users and individuals with limited lit-
eracy.
A user-friendly web-based interface developed using [Link] allows users to in-
teract with the chatbot seamlessly, while the backend server efficiently manages AI
processing, OCR services, speech recognition, and response generation. The system
operates smoothly on standard consumer hardware without requiring specialized med-
ical devices, making it cost-effective and widely deployable.

Extensive testing demonstrated accurate speech recognition, reliable OCR per-

formance across multiple languages, relevant AI-generated medical responses, and
low-latency voice output. The AI Doctor Medical Chatbot therefore serves as a prac-
tical, scalable, and intelligent solution for preliminary medical assistance and health
awareness in real-world scenarios.

8.2 Future Enhancements

While the current system performs effectively, several enhancements can further
improve its capabilities, accuracy, and real-world usability. Potential future improve-
ments include:

• Clinical Decision Support Integration:

– Integrate verified medical knowledge bases and clinical guidelines to improve

diagnostic reliability.

Department of AI ML,Vemana IT 59 2025-26

AI DOCTOR MEDICAL CHATBOT CONCLUSION AND FUTURE ENHANCEMENTS

• Wearable and Mobile Deployment:

– Deploy the system as a mobile application for Android and iOS.

– Enable wearable integration for continuous health monitoring.

• Enhanced Medical Image Analysis:

– Integrate specialized medical vision models for dermatology, ophthalmology,

and wound analysis.

• Advanced Multimodal Reasoning:

– Improve LLM reasoning to combine symptoms, images, and patient history

for more accurate medical insights.

– Support follow-up questioning for better diagnosis refinement.

• Offline AI and OCR Support:

– Implement on-device OCR and speech synthesis to reduce dependency on

cloud services.

– Enable limited offline consultation in low-connectivity areas.

• Multilingual Expansion:

– Add support for more Indian regional languages and dialects to improve
accessibility.

• Doctor Connectivity and Telemedicine:

– Enable direct connection with certified doctors for emergency or advanced

consultation.

– Support secure report sharing and follow-up appointments.

Department of AI ML,Vemana IT 60 2025-26

AI DOCTOR MEDICAL CHATBOT REFERENCES

REFERENCES
[1] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed.,
Pearson, 2021.

[2] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

[3] A. Esteva, B. Kuprel, R. A. Novoa, et al., “Dermatologist-level classification of

skin cancer with deep neural networks,” Nature, vol. 542, pp. 115–118, 2017.

[4] P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol, “AI in healthcare: Trans-

forming the practice of medicine,” Nature Medicine, vol. 28, pp. 31–38, 2022.

[5] J. Li, X. Zhang, and F. Wang, “Multimodal medical AI: Integrating text, image,
and speech for clinical decision support,” Journal of Biomedical Informatics, vol.
131, Article 104091, 2022.

[6] Z. Zhang, L. Yang, and Y. Xia, “Medical image analysis with multimodal deep
learning: Recent advances and challenges,” IEEE Transactions on Medical Imag-
ing, vol. 41, no. 9, pp. 2453–2468, 2022.

[7] A. Huang, M. Ramesh, and R. Gupta, “Vision-language models in medical diag-

nosis: A survey,” IEEE Access, vol. 11, pp. 45612–45625, 2023.

[8] OpenAI, “GPT API Documentation,” 2023. [Online]. Available:

[Link]

[9] Hugging Face, “Transformers: State-of-the-art Natural Language Processing,”

2023. [Online]. Available: [Link]

[10] Google Cloud, “Speech-to-Text API Documentation,” 2023. [Online]. Available:

[Link]

[11] HIPAA, “Health Insurance Portability and Accountability Act Compliance,”

U.S. Department of Health and Human Services, 2023. [Online]. Available:
[Link]

[12] NDHM, “National Digital Health Mission Guidelines,” Government of India,

2023. [Online]. Available: [Link]

Department of AI ML,Vemana IT 61 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

APPENDIX
A.1 Backend Code (FastAPI / Flask – Multimodal AI Processing)

The backend of the AI Doctor Medical Chatbot with Multimodal LLM is imple-
mented using Python (FastAPI / Flask). It handles symptom processing, medical
image analysis, OCR requests, Large Language Model inference, and communication
with the frontend interface.
The complete backend implementation (API endpoints, LLM integration, OCR
handler, speech processing, and response generation logic) is provided below.

A.1.1 Source Code

This section presents the major program codes used in the project.

BACKEND SERVER – SYMPTOM ANALYSIS + IMAGE PROCESSING + LLM

INFERENCE

from dotenv import loadd otenv

loadd otenv()

importos
importtime
importuuid
importshutil
f romf astapiimportF astAP I, U ploadF ile, F ile
f romf [Link] iddleware
f romf [Link] ileResponse
f rombraino ft hed octorimportencodei mage, analyzei magew ithq uery
f romvoiceo fp atientimportrecorda udio, transcribew ithg roq
f romvoiceo ft hed octorimporttextt os peechw ithg tts, textt os peechw ithe levenlabs

InitializeF astAP Iapp

app = F astAP I()

[Link] iddleware(
CORSM iddleware,
allowo rigins = [” ∗ ”],
allowc redentials = T rue,
allowm ethods = [” ∗ ”],

Department of AI ML,Vemana IT 62 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

allowh eaders = [” ∗ ”],

)

Systemprompt
systemp rompt = ”””Y [Link], aprof essional
medical consultant with extensive clinical experience.
Your task is to analyze the provided medical image along with the patient’s descrip-
tion.

Analysis Guidelines:
1. First describe what you see in the image in clinical terms
2. Identify any abnormalities, lesions, or concerning features
3. Formulate a differential diagnosis (2-3 most likely conditions)
4. Suggest appropriate treatments or remedies for each possible diagnosis
5. Provide dosage guidelines for recommended medications
6. Recommend if the patient should seek in-person consultation

Begin your response directly with "Based on what I can see..."

Always include disclaimers about AI consultation and medical safety.

Patient’s description: """

Function to process inputs

def processi nputs(audiof ilepath, imagef ilepath) :
results = ”speecht ot ext” : ””, ”doctorr esponse” : ””, ”voicef ilepath” : ””

outputf ilepath = f ”outputa udio/uuid.uuid4().hex.mp3”

[Link](”outputa udio”, existo k = T rue)

try :
if audiof ilepath :
results[”speecht ot ext”] = transcribew ithg roq(
GROQA P IK EY = [Link](”GROQA P IK EY ”),
audiof ilepath = audiof ilepath,
sttm odel = ”whisper − large − v3”
)
else :
results[”speecht ot ext”] = ”N oaudioprovided.”

Department of AI ML,Vemana IT 63 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

if imagef ilepath :
f ullq uery = systemp rompt + results[”speecht ot ext”]
results[”doctorr esponse”] = analyzei magew ithq uery(
query = f ullq uery,
encodedi mage = encodei mage(imagef ilepath),
model = ”llama − 3.2 − 11b − vision − preview”
)
else :
results[”doctorr esponse”] = ”N oimageprovidedf oranalysis.”

textt os peechw ithe levenlabs(

inputt ext = results[”doctorr esponse”],
outputf ilepath = outputf ilepath
)
results[”voicef ilepath”] = outputf ilepath

exceptExceptionase :
results[”doctorr esponse”] = f ”Erroroccurred : str(e)”
textt os peechw ithg tts(
inputt ext = ”Anerroroccurred.P leasetryagain.”,
outputf ilepath = outputf ilepath
)
results[”voicef ilepath”] = outputf ilepath

returnresults[”speecht ot ext”], results[”doctorr esponse”], results[”voicef ilepath”]

AP I : Analyze
@[Link](”/analyze/”)
asyncdef analyze(audio : U ploadF ile = F ile(...), image : U ploadF ile = F ile(...)) :
tempd ir = ”tempu ploads”
[Link](tempd ir, existo k = T rue)

audiop ath = [Link](tempd ir, f ”uuid.uuid4().hexaudio.f ilename ”)

imagep ath = [Link](tempd ir, f ”uuid.uuid4().heximage.f ilename ”)

withopen(audiop ath, ”wb”)asf :

[Link] ileobj(audio.f ile, f )

Department of AI ML,Vemana IT 64 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

withopen(imagep ath, ”wb”)asf :

[Link] ileobj(image.f ile, f )

transcription, diagnosis, voicef ile = processi nputs(audiop ath, imagep ath)

return”transcription” : transcription, ”analysis” : diagnosis,

"voicef ile” : [Link](voicef ile)

AP I : Getaudiof ile
@[Link](”/audio/f ilename”)
asyncdef geta udio(f ilename : str) :
f ilep ath = [Link](”outputa udio”, f ilename)
if [Link](f ilep ath) :
returnF ileResponse(f ilep ath, mediat ype = ”audio/mpeg”)
return”error” : ”F ilenotf ound”

Root
@[Link](”/”)
def root() :
return”message” : ”M ediScanAIBackendisrunning”

A.2 Frontend Code ([Link] / Gradio Interface + Web Speech API)

The frontend is developed using [Link] (or Gradio). It provides an interactive

chat-based interface through which users communicate with the AI Doctor using text,
voice, and medical image inputs.
This section contains the frontend components for:

• Chat-based medical consultation UI

• Medical image upload interface

• Voice command input

• Text-to-Speech output

• Accessibility controls (language selection, speech rate)

Department of AI ML,Vemana IT 65 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

A.2.1 Source Code

This section presents the major program codes used in the project.

REACT FRONTEND – CHAT UI, IMAGE UPLOAD, VOICE INPUT, TTS

—–html—-

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AI Doctor - Your Intelligent Medical Assistant</title>
<link rel="stylesheet" href="[Link]
awesome/6.4.0/css/[Link]">
<link href="[Link]
[Link]/css2? family=Poppins:
wght@300;400;500;600;700display=swap" rel="stylesheet">
<link rel="stylesheet" href="css/[Link]">
</head>
<body>
<div class="app-container">
<header class="app-header">
<div class="logo-container">
<img src="images/[Link]" alt="AI Doctor Logo" class="logo">
<h1>AI Doctor</h1>
</div>
<nav class="main-nav">
<ul>
<li><a href="features">Features</a></li>
<li><a href="how-it-works">How It Works</a></li>
<li><a href="about">About</a></li>
<li><a href="contact">Contact</a></li>
</ul>
</nav>
<div class="auth-buttons">
<button class="btn btn-outline">Login</button>
<button class="btn btn-primary">Sign Up</button>

Department of AI ML,Vemana IT 66 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

</div>
</header>

/* Base Styles */
:root
–primary-color: 4a6fa5;
–secondary-color: 166088;
–accent-color: 4fc3f7;
–dark-color: 2b2d42;
–light-color: f8f9fa;
–success-color: 4caf50;
–warning-color: ff9800;
–danger-color: f44336;
–text-color: 333;
–text-light: 777;
–border-radius: 8px;
–box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
–transition: all 0.3s ease;

margin: 0;
padding: 0;
box-sizing: border-box;

body
font-family: ’Poppins’, sans-serif;
color: var(–text-color);
background-color: var(–light-color);
line-height: 1.6;

/* Typography */
h1, h2, h3, h4, h5, h6
font-weight: 600;
margin-bottom: 1rem;

Department of AI ML,Vemana IT 67 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

p
margin-bottom: 1rem;

a
text-decoration: none;
color: var(–primary-color);
transition: var(–transition);

a:hover
color: var(–secondary-color);

/* Layout */
.app-container
display: flex;
flex-direction: column;
min-height: 100vh;

.app-main
flex: 1;
padding: 2rem 0;

.container
width: 100%;
max-width: 1200px;
margin: 0 auto;
padding: 0 1.5rem;

.section-title
text-align: center;
margin-bottom: 3rem;
font-size: 2rem;

Department of AI ML,Vemana IT 68 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

color: var(–dark-color);
position: relative;

.section-title::after
content: ”;
display: block;
width: 80px;
height: 4px;
background: var(–primary-color);
margin: 0.5rem auto 0;
border-radius: 2px;

A.3 OCR + AI Integration Code

The AI Image Analysis module is a core component of the AI Doctor system and
is responsible for analyzing uploaded medical images such as skin conditions, wounds,
rashes, and other visible symptoms. Unlike traditional systems that rely on Optical
Character Recognition (OCR), this project does not use OCR for text extraction.
Instead, it employs a vision-enabled Large Language Model (LLM) to perform direct
visual and contextual medical analysis.

When a user uploads a medical image along with a voice or text description, the
system encodes the image into a Base64 format and forwards it to the AI model along
with a structured medical prompt. The model interprets the visual features of the
image and correlates them with the patient’s description to identify possible abnor-
malities and conditions.

The AI model generates a professional medical response that includes clinical obser-
vations, possible diagnoses, treatment suggestions, and recommendations for further
medical consultation. This response is then forwarded to the Voice Assistant module
for audio playback, ensuring accessibility for visually impaired users. By eliminating
OCR dependency, the system achieves more accurate medical reasoning and faster
image-based diagnosis.
2mm

Department of AI ML,Vemana IT 69 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

A.3.1 Source Code

This section presents the major program code used for AI-based medical image
analysis in the AI Doctor system.

AI Image Analysis Module (braino ft hed [Link])

from dotenv import loadd otenv

loadd otenv()

importos
importbase64
importtime
f romgroqimportGroq
importlogging

Conf igurelogging
[Link] ig(
level = [Link] F O,
f ormat =′ )
logger = [Link](”BrainOf Doctor”)

AP IKey
GROQA P IK EY = [Link](”GROQA P IK EY ”)

def encodei mage(imagep ath) :

”””
Convertimagetobase64encodingf orAP Isubmission.
Handlesdif f erentimagef ormatsandincludeserrorhandling.
”””
try :
if [Link](imagep ath) :
raiseF ileN otF oundError(f ”Imagef ilenotf ound : imagep ath”)

withopen(imagep ath, ”rb”)asimagef ile :

returnbase64.b64encode(imagef [Link]()).decode(′ utf − 8′ )
exceptExceptionase :
[Link](f ”Errorencodingimage : str(e)”)
raise

Department of AI ML,Vemana IT 70 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

def analyzei magew ithq uery(

query,
encodedi mage,
model = ”llama − 3.2 − 11b − vision − preview”,
maxr etries = 3
):
”””
Analyzemedicalimagewithenhancederrorhandlingandretrymechanism.

Args :
query(str) : T hepromptsentwiththeimage
encodedi mage(str) : Base64encodedimage
model(str) : V ision − languagemodel
maxr etries(int) : Retryattempts

Returns :
str : AI − generatedmedicalanalysis
”””

client = Groq(apik ey = GROQA P IK EY )

messages = [
”role” : ”user”, ”content” : [”type” : ”text”, ”text” : query,
"type": "imageu rl”,
”imageu rl” : ”url” : f ”data : image/jpeg; base64, encodedi mage”, ,
,
],

Retrylogic
f orattemptinrange(maxr etries) :
try :
[Link] o(
f ”Analyzingimagewithmodel(attemptattempt + 1/maxr etries)”
)

chatc ompletion = [Link](

Department of AI ML,Vemana IT 71 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

messages = messages,
model = model,
temperature = 0.2,
maxt okens = 1024
)

response = chatc [Link][0].[Link]

[Link] o(”Analysiscompletedsuccessf ully”)
returnresponse

exceptExceptionase :
[Link](f ”Attemptattempt + 1f ailed : str(e)”)
if attempt < maxr etries − 1 :
waitt ime = 2 ∗ ∗attempt
[Link] o(f ”Retryinginwaitt imeseconds...”)
[Link](waitt ime)
else :
[Link](”Allretryattemptsf ailed”)
raiseException(
f ”F ailedtoanalyzeimageaf termaxr etriesattempts : str(e)”
)

Exampleusage(commentedf ordeployment)
”””
ifn ame= =” ain
m
” :imagep ath=”testm edicali [Link]”
encodedi mg = encodei mage(imagep ath)

testq uery = ”W hatmedicalconditionisshowninthisimage?”

response = analyzei magew ithq uery(
query = testq uery,
encodedi mage = encodedi mg
)

print(”AnalysisResult : ”)
print(response)
”””

Department of AI ML,Vemana IT 72 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

A.4 Voice of the Patient Module (Speech-to-Text Input)

The Voice of the Patient Module is responsible for capturing the patient’s spoken
symptoms and converting them into accurate textual input for further medical anal-
ysis by the AI Doctor system. Unlike a conversational voice assistant, this module
focuses exclusively on voice-based symptom input, enabling patients to describe their
medical condition naturally without typing. The module records audio input using
a microphone, applies ambient noise adjustment for clarity, and securely stores the
recorded speech in audio format. The captured audio is then transcribed into text
using a state-of-the-art speech-to-text model (Whisper Large v3 via Groq API). This
transcribed text is forwarded to the multimodal AI reasoning pipeline, where it is
combined with image inputs and textual context to generate medical observations
and recommendations.
By converting patient speech into structured text, this module improves accessi-
bility for elderly users, individuals with disabilities, and users who prefer voice-based
interaction. The module does not perform response generation or voice output, en-
suring a clean separation between voice input processing and AI medical reasoning.

A.4.1 Source Code

This section presents the major program code used for capturing and transcribing
patient voice input.

Voice of the Patient – Speech Recording and Transcription Module

voiceo ft hep [Link]

f romdotenvimportloadd otenv
loadd otenv()

importlogging
importspeechr ecognitionassr
f rompydubimportAudioSegment
f romioimportBytesIO
importos
importtempf ile
f romgroqimportGroq

Conf igurelogging
[Link] ig(

Department of AI ML,Vemana IT 73 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

level = [Link] F O,
f ormat =′ )
logger = [Link](”V oiceOf P atient”)

def recorda udio(f ilep ath, timeout = 20, phraset imel imit = N one) :
”””
Enhancedf unctiontorecordaudiof romthemicrophone
withbetteruserf eedback.

Args :
f ilep ath(str) : P athtosavetheaudiof ile
timeout(int) : M axtimetowaitf orspeech
phraset imel imit(int) : M axrecordingduration
”””

recognizer = [Link]()

try :
withsr.M icrophone()assource :
[Link] o(”Adjustingf orambientnoise...”)
[Link] ora mbientn oise(source, duration = 1)
[Link] o(”Startspeakingnow...”)

audiod ata = [Link](

source,
timeout = timeout,
phraset imel imit = phraset imel imit
)
[Link] o(”Recordingcomplete.”)

wavd ata = audiod [Link] avd ata()

audios egment = AudioSegment.f romw av(BytesIO(wavd ata))
audios [Link](
f ilep ath,
f ormat = ”mp3”,
bitrate = ”128k”
)

Department of AI ML,Vemana IT 74 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

[Link] o(f ”Audiosavedtof ilep ath”)

returnT rue

exceptsr.W aitT imeoutError :

[Link](”N ospeechdetectedwithintimeoutperiod”)
returnF alse

[Link] :
[Link](f ”AP Iunavailable : e”)
returnF alse

exceptExceptionase :
[Link](f ”Anerroroccurred : e”)
returnF alse

def transcribew ithg roq(

GROQA P IK EY,
audiof ilepath,
sttm odel = ”whisper − large − v3”
):
”””
T ranscribepatientaudiousingGroqW hispermodel.

Args :
GROQA P IK EY (str) : GroqAP Ikey
audiof ilepath(str) : P athtoaudiof ile
sttm odel(str) : Speech − to − textmodel

Returns :
str : T ranscribedtext
”””

client = Groq(apik ey = GROQA P IK EY )

try :
if [Link](audiof ilepath) :
raiseF ileN otF oundError(
f ”Audiof ilenotf ound : audiof ilepath”

Department of AI ML,Vemana IT 75 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

audio = AudioSegment.f romf ile(audiof ilepath)

if audiof [Link](′ .mp3′ ) :

tempf ile = tempf ile.N amedT emporaryF ile(
suf f ix =′ .wav ′ ,
delete = F alse
)
tempf ilepath = tempf [Link]
tempf [Link]()

[Link](tempf ilepath, f ormat = ”wav”)

processf ilepath = tempf ilepath
else :
processf ilepath = audiof ilepath

[Link] o(f ”T ranscribingaudiowithsttm odel...”)

withopen(processf ilepath, ”rb”)asaudiof ile :
transcription = [Link](
model = sttm odel,
f ile = audiof ile,
language = ”en”
)

if ′ tempf ilepath′ inlocals() :

[Link](tempf ilepath)

[Link] o(”T ranscriptioncomplete”)

[Link]

exceptExceptionase :
[Link](f ”T ranscriptionerror : str(e)”)
raise

Exampleusage(commentedf ordeployment)
”””
ifn ame= =” ain
m ”:

Department of AI ML,Vemana IT 76 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

GROQA P IK EY = [Link](”GROQA P IK EY ”)

audiof ilepath = ”patientr ecording.mp3”

recorda udio(f ilep ath = audiof ilepath)

transcribedt ext = transcribew ithg roq(

GROQA P IK EY = GROQA P IK EY,
audiof ilepath = audiof ilepath
)

print(”T ranscribedT ext : ”)

print(transcribedt ext)
”””

A.5 Screenshots

Figure A.1: AI Doctor Prescription Scanning Module

Department of AI ML,Vemana IT 77 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

Figure A.2: Medical Image OCR Analysis Window

Figure A.3: Home Interface of AI Doctor System

A.6 Installation Procedure

This section explains the complete installation process required to deploy the AI
Doctor Medical Chatbot with Multimodal LLM, including hardware setup, backend
configuration, frontend setup, OCR integration, and voice services.

A.6.1 Hardware Setup

The system uses standard, low-cost consumer hardware.
• Required Components:

– Laptop / PC with Intel i5 / Ryzen 5 or higher.

– Microphone or headset for voice interaction.
– Webcam or smartphone camera for medical image capture.

Department of AI ML,Vemana IT 78 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

• Hardware Setup:

– Ensure microphone and camera are properly connected.

– Enable camera and microphone permissions in the browser.
– Verify audio output using speakers or headphones.

A.6.2 Backend Setup (Python + FastAPI / Flask)

• Step 1 – Install Python Dependencies

– pip install fastapi flask uvicorn requests opencv-python pillow

– pip install torch transformers speechrecognition pyaudio

• Step 2 – Configure AI Models

– Download or configure the Multimodal LLM model.

– Ensure required model weights are available locally or via API.

• Step 3 – Start Backend Server

– python [Link] (for Flask)

– or uvicorn main:app –reload (for FastAPI)
– Backend runs on [Link] or [Link]

A.6.3 Frontend Setup ([Link])

• Step 1 – Install Node Modules

– npm install

• Step 2 – Start React Development Server

– npm start
– Application runs on [Link]

A.6.4 Connecting Frontend and Backend

Update the backend API URL inside the React frontend:
• const API_URL = "[Link] (or 5000 based on backend)

A.6.5 OCR API Configuration

• Create an account with an OCR service ([Link] or equivalent).
• Generate a free API key.
• Add the API key to the backend configuration:

– OCR_API_KEY = "ENTER-YOUR-API-KEY"

Department of AI ML,Vemana IT 79 2025-26

AI DOCTOR MEDICAL CHATBOT APPENDIX

A.6.6 Speech and Browser Permission Setup

Enable the following permissions in the browser:
• Microphone access for voice input.
• Camera access for medical image capture.
• Auto-play audio permission for Text-to-Speech output.

Department of AI & ML, Vemana IT 80 2025-26

AI DOCTOR MEDICAL CHATBOT CERTIFICATES

CERTIFICATES

Department of AI & ML, Vemana IT 81 2025-26