A.
I POWERED MEDICAL ASSISTANT
A PROJECT REPORT SUBMITTED TO
SRM INSTITUTE OF SCIENCE & TECHNOLOGY
IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE
AWARD OF THE DEGREE OF
MASTER OF COMPUTER APPLICATIONS
BY
ADITYA SINGH (REG NO. RA2432241010363)
DIKSHIT RAJ (REG NO. RA2432241010358)
SHIVAM PANDEY (REG NO. RA2432241010349)
PABITRA M. MAHAPATRA (REG NO. RA2432241010340)
UNDER THE GUIDANCE OF
Dr. Balamurugan
DEPARTMENT OF COMPUTER APPLICATIONS
FACULTY OF SCIENCE AND HUMANITIES
SRM INSTITUTE OF SCIENCE & TECHNOLOGY
Kattankulathur – 603203
Chennai, Tamil Nadu
OCTOBER - 2025
BONAFIDE CERTIFICATE
This is to certify that the project report titled “A.I POWERED MEDICAL
ASSISTANT” is a Bonafide work carried out ADITYA SINGH (REG NO.
RA2432241010363), DIKSHIT RAJ (REG NO. RA2432241010358), SHIVAM PANDEY
(REG NO. RA2432241010349), PABITRA M. MAHAPATRA (REG NO.
RA2432241010340) under my supervision for the award of the Degree of Master of
Computer Applications. To my knowledge the work reported herein is the original work done
by these students.
[Link] Leena Rose
Professor & Head,
Department of Computer Applications
EXTERNAL EXAMINER
ACKNOWLEDGEMENT
With profound gratitude to the ALMIGHTY, I take this chance to thank the people
who helped me to complete this project.
We take this as a right opportunity to say THANKS to my parents who are there to
stand with me always with the words “YOU CAN”.
We are thankful to Dr.T.R. Paarivendhar, Chancellor, and Prof.A. Vinay Kumar,
Pro Vice-Chancellor (SBL), SRM Institute of Science & Technology who gave us the
platform to establish me to reach greater heights.
We earnestly thank Dr.A. Duraisamy, Dean, Faculty of Science and Humanities,
SRM Institute of Science & Technology who always encourage us to do novel things.
A great note of gratitude to Dr. S. Albert Antony Raj, Deputy Dean, Faculty of
Science and Humanities for his valuable guidance and constant Support to do this Project.
We express our sincere thanks to Dr. P.J. Arul Leena Rose, Professor & Head for
her support to execute all incline in learning.
It is our delight to thank our project guide Mr. J. Venkata Subramanian, Assistant
Professor, Department of Computer Applications for his help, support, encouragement,
suggestions, and guidance throughout the development phases of the project.
We convey our gratitude to all the faculty members of the department who extended
their support through valuable comments and suggestions during the reviews.
Our gratitude to friends and people who are known and unknown to me who helped in
carrying out this project work a successful one.
ADITYA SINGH
DIKSHIT RAJ
SHIVAM PANDEY
PABITRA M MAHAPATRA
TABLE OF CONTENTS
1. INTRODUCTION....................................................................................................1
2. SOFTWARE REQUIREMENT ANALYSIS........................................................2
2.1 HARDWARE SPECIFICATION.......................................................................2
2.2 SOFTWARE SPECIFICATION........................................................................2
2.3 ABOUT THE SOFTWARE AND ITS FEATURE............................................2
3. SYSTEM ANALYSIS..............................................................................................4
3.1 EXISTING SYSTEM.........................................................................................4
3.2 PROPOSED SYSTEM.......................................................................................5
3.3 FEASIBILITY STUDY......................................................................................6
4. SYSTEM DESIGN...................................................................................................7
4.1 DATA FLOW DIAGARAM..............................................................................7
4.2 UML DIAGRAMS...........................................................................................10
5. CODE TEMPLATES.............................................................................................16
5.1 MODULE DESCRIPTION..............................................................................16
5.1.1 Voice Recognition Module
5.1.2 Image Analysis (Vision) Module
5.1.3 Disease Prediction Module
5.1.4 Doctor Recommendation Module
5.1.5 Flask Web Interface Module
6. TESTING................................................................................................................21
6.1 TESTING METHODOLOGIES.......................................................................21
6.2 TEST CASE......................................................................................................22
7. OUTPUT SCREENS..............................................................................................27
8. CONCLUSION.......................................................................................................35
9. FURTHER ENHANCEMENTS...........................................................................36
10. REFERENCES.......................................................................................................37
11. APPENDICES....................................................................................................38
11.1 USER DOCUMENTATION............................................................................38
11.2 README..........................................................................................................38
11.3 SAMPLE SOURCE PROGRAM.....................................................................40
ABSTRACT
This project introduces an A.I. Powered Medical Assistant that combines speech recognition,
image analysis, and text-to-speech technologies to create a realistic medical consultation
experience. The system is designed in four sequential phases, each performing a critical
function in the interaction loop.
The first phase, Voice of the Patient, enables the patient to communicate naturally by
speaking into a microphone. The spoken input is recorded and then converted into text using
the Whisper large-v3 speech-to-text model through the Groq API. This step ensures that the
patient’s symptoms or queries are accurately transcribed into digital text.
In the second phase, the Brain of the Doctor, the patient’s text (and optionally an uploaded
image, such as a photo of a skin condition) is processed. The image is encoded into a base64
format and, along with the query, is analyzed by a multimodal large language model (LLaMA
through Groq). The model generates a medically relevant and context-aware response,
imitating the reasoning process of a doctor.
The third phase, Voice of the Doctor, focuses on delivering the AI’s response in a natural and
human-like manner. The text output from the doctor’s brain is converted into audible speech
using two text-to-speech engines: Google Text-to-Speech (gTTS) for basic functionality and
ElevenLabs API for more realistic and expressive voice synthesis. This ensures that the
doctor’s reply is accessible not only as text but also as spoken words.
The fourth and final phase integrates all components through a Gradio-powered web
interface. The interface provides an easy-to-use platform where users can record their voice,
upload medical images, and receive both textual and spoken responses. The UI displays the
transcribed patient speech, the doctor’s diagnostic response, and plays the synthesized voice
reply.
Together, these phases demonstrate how artificial intelligence can combine language, vision,
and speech technologies to simulate a conversational healthcare assistant. While this project
is primarily a learning and demonstration tool, it highlights the potential of multimodal AI
systems to improve accessibility, patient engagement, and interactive medical education in
the future.
1. INTRODUCTION
With lifestyle diseases on the rise due to sedentary habits, an increasing number
of individuals are prioritizing fitness and healthier routines. The fitness
landscape is rapidly advancing, offering diverse options for all fitness levels and
goals—both online and offline. Modern technology now empowers personal
fitness trainers to break traditional boundaries, delivering expertise through
innovative web platforms.
Our web app stands at the forefront of this transformation, offering trainers a
unique space to conduct online assessments, design custom workout programs,
and provide ongoing support to clients—all virtually. Trainers are no longer
limited by geographical constraints; instead, they can build a global client base
and interact asynchronously, allowing ultimate schedule flexibility for both
trainers and clients. Every aspect, from scheduling initial consultations to
managing program delivery and monitoring progress, happens seamlessly
within the app.
By digitizing the fitness consultation and training process, our web app not only
simplifies the workflow for trainers but also increases accessibility for clients,
eliminating the need for expensive gym memberships or travel costs. This
approach aligns with the telehealth revolution—especially significant in times
when public health factors can restrict physical access to fitness centers. Clients
benefit from customized guidance, progress tracking, and motivational support,
ultimately enjoying a more flexible, affordable, and personalized path to better
health—all from the comfort of their own home
2. SOFTWARE REQUIREMENT ANALYSIS
2.1 HARDWARE SPECIFICATION
Operating System: Windows 10 or newer, macOS 10.13+ (High Sierra
or newer), or Ubuntu 18.04+
Processor: 2GHz dual-core or higher
RAM: 2GB or more
Hard Disk: 64GB or higher
Network: Ethernet (LAN) or Wi-Fi connection
2.2 SOFTWARE SPECIFICATION
IDE: Visual Studio Code, PyCharm, or any modern Python IDE
Programming Language: Python 3.8+
Virtual Environment: Pipenv, venv, or Conda
Web Framework: Gradio for web interface
Speech and Audio Processing: FFmpeg, PortAudio, PyAudio
Text-to-Speech APIs: gTTS, ElevenLabs
AI/ML APIs: Groq multimodal and Whisper for transcription, meta-llama
models
Image/Data Processing: Pillow, pandas, numpy
Browser: Chrome, Firefox, or Edge
2.3 ABOUT THE SOFTWARE AND ITS FEATURE
This project is an AI-powered medical consultation web application designed to
analyze images, process voice inputs, and provide intelligent medical insights to
users. Built using Python and Gradio, it integrates advanced technologies like
GROQ multimodal AI, speech recognition, and text-to-speech systems to
simulate real-time doctor–patient interaction. The application acts as an AI
medical assistant that observes facial or skin images, listens to patient
symptoms, and responds with possible medical explanations, all while
maintaining a conversational, human-like tone.
2.3.1 Flexibility in Image and Audio Input
This software allows users to upload medical images or speak directly through a
microphone to describe symptoms. The combination of image encoding and
speech-to-text transcription enables flexible interaction, allowing the AI to
process multimodal health data. This makes it convenient for both doctors and
patients to communicate effectively regardless of location or time.
2.3.2 Visualized Analysis
The AI model, powered by GROQ’s multimodal language model (LLaMA-4
variants), analyzes the uploaded images (e.g., acne, dandruff, lesions) and text
data to detect health anomalies. It presents analyzed insights in a structured text
format that can easily be visualized in the web interface for better
understanding. Updates on detected conditions and pattern recognition can also
be extended to visual dashboards using libraries like Matplotlib or Gradio
components for enhanced interpretation.
2.3.3 Intelligent Medical Suggestion
Using large multimodal AI models, the web app intelligently evaluates the
medical condition in the image and suggests possible remedies or next steps. It
is designed to provide preliminary medical advice or professional
recommendations with contextual awareness, mimicking a real doctor’s
diagnostic tone based on the provided input.
2.3.4 Speech-to-Text and Text-to-Speech Integration
The application features real-time audio transcription using GROQ Whisper-
based models and text-to-speech output powered by ElevenLabs or Google
Text-to-Speech (gTTS). This enables seamless two-way communication—
patients can describe their condition verbally, and the AI doctor responds
audibly, simulating a human consultation experience.
2.3.5 Secure Access and API Integration
All model interactions are secured using API key integration for GROQ and
ElevenLabs services. The .env configuration protects sensitive data such as API
credentials, ensuring user security and professional system operation.
2.3.6 Automated Functionality and Compatibility
The app automatically:
• Encodes and processes uploaded medical images.
• Transcribes spoken input into text through AI.
• Generates medical responses through the Groq LLaMA model.
• Converts responses into human-like voice output using
ElevenLabs or gTTS.
• It operates locally or on a remote server and is compatible with
major OS platforms like Windows, macOS, and Linux.
2.3.7 Measurement and Diagnostic Tutorials
For educational purposes, the system can easily link to YouTube or embedded
health guides that explain basic diagnostic self-checks, ensuring users
understand their potential symptoms responsibly while awaiting professional
medical advice. This aligns with telemedicine ethics in AI-assisted care.
2.3.8 Automatic Inference and Reporting
The system uses automated AI inference to generate detailed analysis reports.
This includes possible medical conditions, causes, and suggested remedies
based on the provided image and voice data. Trainers (doctors or medical
educators) can use this data to track diagnostic models' efficiency and refine
them further.
This combination of AI image recognition, voice transcription, and real-time
response generation establishes your project as an advanced tool in telehealth
and AI-based diagnosis simulation, bridging technology and healthcare
accessibility effectively using Python, Gradio, and advanced AI APIs.
[Link] ANALSIS
3.1 EXISTING SYSTEM
The existing systems, such as the commonly available AI-powered medical bots and
health-assistance applications, exhibit several notable limitations. Most of these
platforms provide only basic symptom checking or generalized advice without
detailed tracking or comprehensive data analytics. They often fail to maintain
historical user data in a structured format, making it difficult to analyze health trends
or monitor a patient’s progress over time. As a result, users receive only fragmented
insights rather than a holistic understanding of their health status.
Additionally, communication between users and healthcare providers or trainers is
usually conducted through informal channels such as WhatsApp or email. This
approach lacks proper data organization and traceability, often leading to missed
updates, miscommunication, or delayed responses. The absence of centralized
monitoring tools makes it challenging for professionals to maintain consistent
interactions and timely interventions.
Scalability also remains a major issue. Since most health professionals rely on manual
methods to monitor user activities and health records, managing multiple users
simultaneously becomes inefficient and time-consuming. Furthermore, the
maintenance cost of these fragmented systems is high due to the lack of integration
and automation. Many of the existing applications also suffer from non-intuitive and
cluttered user interfaces, making them difficult for users with limited technical
knowledge. These challenges collectively lead to poor engagement, reduced accuracy
in diagnosis support, and limited usability. Therefore, there is a pressing need for an
intelligent, unified, and user-friendly AI-driven healthcare system capable of
delivering automated, data-based, and interactive medical assistance.
3.2 PROPOSED SYSTEM
The proposed system is developed to address the limitations of existing
healthcare and fitness management platforms by introducing automation,
intelligent analysis, and a unified digital interface. This AI-powered solution
streamlines the process of data collection, processing, and feedback generation,
allowing medical professionals and fitness trainers to manage multiple clients
efficiently. It integrates a web-based interface that facilitates seamless
interaction between users and professionals, enabling them to create
personalized health or fitness plans, monitor progress in real-time, and make
data-driven decisions.
Unlike traditional manual systems, the proposed model automates the
calculation of critical health metrics such as BMI, BMR, calorie intake, and
other performance indicators. It also features automated reminders for client
check-ins, ensuring consistency and active participation. A dynamic dashboard
provides an intuitive visualization of client progress through charts and reports,
enabling professionals to identify patterns and potential areas for improvement.
The system leverages data analytics and machine learning algorithms to detect
stagnation or regressions in user performance. Upon identifying such trends, it
intelligently recommends necessary modifications to the workout or treatment
plan, thereby improving overall results and satisfaction. The integration of voice
and vision technology enhances user experience, allowing for voice-based
symptom reporting and image-based diagnosis support. This reduces manual
workload, enhances scalability, and ensures better engagement between clients
and professionals. Ultimately, the proposed system promotes accurate
monitoring, faster decision-making, and a more professional, efficient, and user-
centric healthcare management experience.
3.3 FEASIBILITY STUDY
A feasibility study is conducted to evaluate the practicality and viability of the
proposed system before implementation. It helps determine whether the project
is technically, economically, and operationally achievable within the available
resources and constraints. The AI Doctor (Voice and Vision) system has been
carefully analyzed on these parameters and found to be feasible in all respects.
3.3.1 Economic Feasibility
The proposed system is economically feasible because it significantly
minimizes manual work and human error, leading to improved accuracy and
productivity. It automates many routine processes such as data entry,
monitoring, and reporting, reducing the need for dedicated manpower and
operational costs. Since the system can be deployed using existing hardware
infrastructure, it does not require additional investment in costly new
equipment. The software components are based on open-source technologies,
which further lowers development and maintenance costs. Over time, the
increased efficiency, automation, and accuracy provide a strong return on
investment (ROI), making the system financially sustainable and beneficial for
long-term use.
3.3.2 Technical Feasibility
The project is technically feasible as it utilizes widely available and proven
technologies. The system is developed using Python, Flask, and standard web
technologies such as HTML, CSS, and JavaScript, all of which are compatible
with existing platforms. It runs efficiently on regular computing devices without
requiring high-end hardware. The necessary libraries and frameworks, such as
TensorFlow, OpenCV, and SpeechRecognition, support the voice and vision
functionalities effectively. Hence, the technical requirements are well within
achievable limits, ensuring smooth development and deployment.
3.3.3 Operational Feasibility
From an operational perspective, the system is highly feasible as it simplifies
decision-making, enhances monitoring, and reduces workload for both users
and professionals. The intuitive web interface, automated analytics, and
personalized feedback features align perfectly with user needs. The system’s
easy adaptability and interactive design improve acceptance and usability. By
integrating automation and data intelligence, it ensures better operational
efficiency, real-time insight generation, and continuous performance
improvement.
4. SYSTEM DESIGN
4.1 UML DIAGRAM
4.1.1 USE CASE DIAGRAM
4.1.2 ACTIVITY DIAGRAM
4.1.3 SEQUENCE DIAGRAM
4.1.4 COMPONENT DIAGRAM
5. CODE TEMPLATES
5.1 Voice Recognition Module:
This module is responsible for capturing and processing spoken language input
from the user. It uses audio recording capabilities integrated with libraries like
ffmpeg and PortAudio to capture voice input. The recording is saved and then
transcribed using a speech-to-text model, such as GROQ's Whisper-large-v3, to
convert the voice data into text for further processing.
5.2 Image Analysis (Vision) Module:
This module handles the processing and analysis of medical images submitted
by users. It converts images into a required encoded format using base64
encoding and submits these to a multimodal large language model API (e.g.,
GROQ's Llama models) for analysis. The module interprets the image and
generates relevant medical insights or diagnoses for further communication.
5.3 Disease Prediction Module:
While not explicitly detailed in the provided files, this module would typically
use the processed textual and image inputs to run disease prediction models
backed by machine learning or AI algorithms. It integrates the findings from
voice transcriptions and image analysis to predict probable diseases and assist
healthcare providers or users in diagnosis.
5.4 Doctor Recommendation Module: This component advises on appropriate
medical specialists or follow-up actions based on the disease prediction
outcomes and patient data. It may integrate AI-driven rule sets or databases
listing specialists according to illness categories, providing users with referrals
or suggestions for care escalation.
5.5 Flask Web Interface Module:
This module provides the web-based user interface and API endpoints for the
system. It uses the Flask framework to create a responsive interface where users
can upload audio and images, receive transcriptions and diagnostic feedback,
and listen to synthesized speech responses. It orchestrates the invocation of
backend modules and manages communication between front-end and
processing components through services like Gradio for interaction.
6. TESTING
6.1 TESTING METHODOLOGIES
6.1.1 FUNCTIONALITY TESTING
Functionality Testing ensures all modules perform their expected tasks
correctly. This involves checking voice recognition accurately transcribes
speech into text, image analysis correctly interprets medical images, disease
prediction provides plausible outcomes, doctor recommendations match
predicted conditions, and the Flask web interface facilitates smooth user
interactions.
6.1.2 USABILITY TESTING
Usability Testing evaluates how user-friendly and intuitive the system is. It
covers the ease with which patients and doctors navigate the web interface,
submit audio or image data, understand system feedback, and control their
interactions with minimal confusion or error.
6.1.3 WEB UI TESTING
Web UI Testing focuses specifically on verifying the responsiveness, accuracy,
and stability of user interface elements across different devices and browsers. It
tests form inputs, buttons, feedback displays, and real-time system responses to
ensure consistent and error-free user experience.
6.1.4 COMPATIBILITY TESTING
Compatibility Testing verifies the system’s ability to work across diverse
devices, operating systems, and network conditions. This ensures the AI
medical bot can function seamlessly whether accessed on desktops, tablets, or
smartphones, and under varying internet speeds.
6.1.5 PERFORMANCE TESTING
Performance Testing assesses the response times, load handling, and resource
utilization. For instance, the system should quickly process audio and image
inputs without lag and sustain multiple users simultaneously without crashing or
slowing down.
6.1.6 SECURITY TESTING
Security Testing checks for vulnerabilities in data handling and transmission. It
is critical to protect sensitive patient information from unauthorized access,
ensure encrypted communication, validate authentication mechanisms, and
comply with healthcare data privacy regulations.
7. OUTPUT SCREENS
Fig: User Interface.
Fig: Photo uploaded from the system.
Fig: Live photo can be uploaded.
Fig: Photo can be copy & paste.
8. CONCLUSION
The AI Doctor system developed in this project successfully integrates multiple
artificial intelligence techniques to simulate an intelligent healthcare assistant
capable of interacting with patients using both voice and vision interfaces. The
system comprises modular components: the Voice Recognition Module for
capturing and converting patient speech into text, the Image Analysis
Module for processing medical images via advanced multimodal AI, and
the Voice Output Module for generating natural speech responses using text-to-
speech models like gTTS and ElevenLabs.
Through the use of Gradio for the web interface, this system enables real-time
interaction between patients and the AI, allowing seamless text, audio, and
image-based communication. The model leverages powerful APIs like Groq’s
LLaMA Vision models for diagnostic analysis and response generation,
providing medical insights in a user-friendly and accessible way.
Testing was performed to ensure proper module integration, input validation,
user accessibility, and performance efficiency across different devices. The
architecture demonstrates how voice processing, computer vision, and natural
language understanding can be harmonized to enhance digital healthcare
services.
This project not only highlights the potential of AI in healthcare diagnostics and
consultation but also sets the foundation for future scalability—such as
integration with Electronic Health Records (EHR), custom disease prediction
models, and telemedicine systems. In essence, the AI Doctor project
exemplifies how human–AI collaboration can make medical assistance more
immediate, affordable, and inclusive.
9. FURTHER ENHANCEMENTS
Future improvements can significantly expand the scope and capability of the
AI Doctor system. Integration of real-time Electronic Health Record
(EHR) systems can allow for more personalized diagnostic insights. The use
of deep learning models for medical image classification can improve
diagnostic accuracy. Advanced multi-language voice recognition could allow
broader accessibility. Additionally, a secured doctor-patient chat portal with
medical report sharing, cloud-based deployment, and AI explainability tools can
make the system more robust, transparent, and scalable. Integration with
wearable IoT devices for real-time health monitoring could also be considered
for future iterations.
10. REFERENCES
1. Groq API Documentation — Multimodal LLaMA Vision Models for
image-based AI chat.
2. ElevenLabs API — Text-to-Speech model for human-like synthetic voice
generation.
3. Gradio Framework — Open-source Python interface library for
interactive ML web apps.
4. Google Text-to-Speech (gTTS) Documentation.
5. Python Official Documentation — for speech recognition, subprocess
handling, and environment setup.
6. FFmpeg and PortAudio Libraries — Audio processing and conversion
utilities.
7. Academic research on AI-based clinical decision support systems and
speech interfaces in healthcare.
11. APPENDICES
The appendices include supporting technical scripts and outputs that verify the
working of individual modules:
● Appendix A: Code for voice_of_the_patient.py — Audio recording and
voice-to-text transcription.
● Appendix B: Code for brain_of_the_doctor.py — Image encoding and
multimodal AI query processing.
● Appendix C: Code for voice_of_the_doctor.py — Text-to-speech voice
generation (gTTS and ElevenLabs).
● Appendix D: Gradio interface (gradio_app.py) — Integration module
connecting the backend AI with the frontend UI.
11.1 USER DOCUMENTATION
The user can interact with the AI Doctor through the Gradio web application.
Patients can speak symptoms, upload medical images, and listen to AI-
generated voice advice. A simple browser interface ensures accessibility across
desktop and mobile devices. The interface is intuitive, requiring basic input via
microphone and/or file upload.
11.1.1 INSTALLSTION INSTRUCTIONS
1. Install dependencies using the terminal:
‘pip install gradio pydub gtts elevenlabs python-dotenv ffmpeg’
2. Download and install FFmpeg and ensure it is available in your system’s
PATH.
3. Configure the .env file with API keys (Groq, ElevenLabs).
4. Run the Gradio app:
‘python gradio_app.py’
5. Access the AI Doctor interface in your browser via the URL displayed in
the terminal (usually [Link]
11.2 README
The README provides setup instructions for installing FFmpeg, PortAudio,
and creating a Python virtual environment for the project. It also explains the
usage of modules, their functions, and configuration methods using the .env file.
Additionally, it includes platform-specific installation steps for macOS, Linux,
and Windows. Final instructions explain how to run and troubleshoot the AI
Doctor interface.
11.2.1 CLIENT MODULE GUIDE
The Client module, implemented through the Gradio interface, enables users
(patients) to interact with the AI Doctor. Users can record voice input through
the Voice Recognition Module, upload medical images for analysis, and receive
feedback via Text-to-Speech. The client-side workflow focuses on usability,
minimal setup, and responsive design.
11.2.2 ADMIN MODULE GUIDE
Administrators can configure environment variables, manage API keys, monitor
logs, and adjust module parameters for optimal AI performance. The Admin
Guide includes:
● Setting and securing API keys in the .env file.
● Configuring AI model names and performance parameters.
● Managing dependencies like FFmpeg path and ElevenLabs voice model
preferences.
● Reviewing logs for debugging API connectivity and audio processing.
11.3 SAMPLE SOURCE PROGRAM
11.3.1 brain_of_the_doctor.py
# if you dont use pipenv uncomment the following:
# from dotenv import load_dotenv
# load_dotenv()
#Step1: Setup GROQ API key
import os
GROQ_API_KEY=[Link]("GROQ_API_KEY")
#Step2: Convert image to required format
import base64
#image_path="[Link]"
def encode_image(image_path):
image_file=open(image_path, "rb")
return base64.b64encode(image_file.read()).decode('utf-8')
#Step3: Setup Multimodal LLM
from groq import Groq
query="Is there something wrong with my face?"
#model = "meta-llama/llama-4-maverick-17b-128e-instruct"
model="meta-llama/llama-4-scout-17b-16e-instruct"
#model = "meta-llama/llama-4-scout-17b-16e-instruct"
#model="llama-3.2-90b-vision-preview" #Deprecated
def analyze_image_with_query(query, model, encoded_image):
client=Groq()
messages=[
"role": "user",
"content": [
"type": "text",
"text": query
},
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{encoded_image}",
},
},
],
}]
chat_completion=[Link](
messages=messages,
model=model
return chat_completion.choices[0].[Link]
11.3.2 voice_of_the_doctor.py
from dotenv import load_dotenv
load_dotenv()
#Step1a: Setup Text to Speech–TTS–model with gTTS
import os
from gtts import gTTS
def text_to_speech_with_gtts_old(input_text, output_filepath):
language="en"
audioobj= gTTS(
text=input_text,
lang=language,
slow=False
[Link](output_filepath)
input_text="Hi this is Aditya Singh!"
text_to_speech_with_gtts_old(input_text=input_text,
output_filepath="gtts_testing.mp3")
#Step1b: Setup Text to Speech–TTS–model with ElevenLabs
import elevenlabs
from [Link] import ElevenLabs
#from elevenlabs import generate, save, set_api_key
ELEVENLABS_API_KEY=[Link]("ELEVEN_API_KEY")
def text_to_speech_with_elevenlabs_old(input_text, output_filepath):
client=ElevenLabs(api_key=ELEVENLABS_API_KEY)
audio=[Link](
text= input_text,
voice= "Aria",
output_format= "mp3_22050_32",
model= "eleven_turbo_v2"
[Link](audio, output_filepath)
#text_to_speech_with_elevenlabs_old(input_text,
output_filepath="elevenlabs_testing.mp3")
#Step2: Use Model for Text output to Voice
import subprocess
import platform
def text_to_speech_with_gtts(input_text, output_filepath):
language="en"
audioobj= gTTS(
text=input_text,
lang=language,
slow=False
[Link](output_filepath)
os_name = [Link]()
try:
if os_name == "Darwin": # macOS
[Link](['afplay', output_filepath])
elif os_name == "Windows": # Windows
[Link](['powershell', '-c', f'(New-Object [Link]
"{output_filepath}").PlaySync();'])
elif os_name == "Linux": # Linux
[Link](['aplay', output_filepath]) # Alternative: use 'mpg123' or
'ffplay'
else:
raise OSError("Unsupported operating system")
except Exception as e:
print(f"An error occurred while trying to play the audio: {e}")
input_text="Hi this is Ai with Hassan, autoplay testing!"
#text_to_speech_with_gtts(input_text=input_text,
output_filepath="gtts_testing_autoplay.mp3")
def text_to_speech_with_elevenlabs(input_text, output_filepath):
client=ElevenLabs(api_key=ELEVENLABS_API_KEY)
audio=[Link](
text= input_text,
voice= "Aria",
output_format= "mp3_22050_32",
model= "eleven_turbo_v2"
[Link](audio, output_filepath)
os_name = [Link]()
try:
if os_name == "Darwin": # macOS
[Link](['afplay', output_filepath])
elif os_name == "Windows": # Windows
[Link](['powershell', '-c', f'(New-Object [Link]
"{output_filepath}").PlaySync();'])
elif os_name == "Linux": # Linux
[Link](['aplay', output_filepath]) # Alternative: use 'mpg123' or
'ffplay'
else:
raise OSError("Unsupported operating system")
except Exception as e:
print(f"An error occurred while trying to play the audio: {e}")
#text_to_speech_with_elevenlabs(input_text,
output_filepath="elevenlabs_testing_autoplay.mp3")
11.3.3 voice_of_the_patient
# if you dont use pipenv uncomment the following:
# from dotenv import load_dotenv
# load_dotenv()
#Step1: Setup Audio recorder (ffmpeg & portaudio)
# ffmpeg, portaudio, pyaudio
import logging
import speech_recognition as sr
from pydub import AudioSegment
from io import BytesIO
[Link](level=[Link], format='%(asctime)s - %
(levelname)s - %(message)s')
def record_audio(file_path, timeout=20, phrase_time_limit=None):
"""
Simplified function to record audio from the microphone and save it as an
MP3 file.
Args:
file_path (str): Path to save the recorded audio file.
timeout (int): Maximum time to wait for a phrase to start (in seconds).
phrase_time_lfimit (int): Maximum time for the phrase to be recorded (in
seconds).
"""
recognizer = [Link]()
try:
with [Link]() as source:
[Link]("Adjusting for ambient noise...")
recognizer.adjust_for_ambient_noise(source, duration=1)
[Link]("Start speaking now...")
# Record the audio
audio_data = [Link](source, timeout=timeout,
phrase_time_limit=phrase_time_limit)
[Link]("Recording complete.")
# Convert the recorded audio to an MP3 file
wav_data = audio_data.get_wav_data()
audio_segment = AudioSegment.from_wav(BytesIO(wav_data))
audio_segment.export(file_path, format="mp3", bitrate="128k")
[Link](f"Audio saved to {file_path}")
except Exception as e:
[Link](f"An error occurred: {e}")
audio_filepath="patient_voice_test_for_patient.mp3"
#record_audio(file_path=audio_filepath)
#Step2: Setup Speech to text–STT–model for transcription
import os
from groq import Groq
GROQ_API_KEY=[Link]("GROQ_API_KEY")
stt_model="whisper-large-v3"
def transcribe_with_groq(stt_model, audio_filepath, GROQ_API_KEY):
client=Groq(api_key=GROQ_API_KEY)
audio_file=open(audio_filepath, "rb")
transcription=[Link](
model=stt_model,
file=audio_file,
language="en"
return [Link]
11.3.4 [Link]
# if you dont use pipenv uncomment the following:
from dotenv import load_dotenv
load_dotenv()
#VoiceBot UI with Gradio
import os
import gradio as gr
from brain_of_the_doctor import encode_image, analyze_image_with_query
from voice_of_the_patient import record_audio, transcribe_with_groq
from voice_of_the_doctor import text_to_speech_with_gtts,
text_to_speech_with_elevenlabs
#load_dotenv()
system_prompt="""You have to act as a professional doctor, i know you are not
but this is for learning purpose.
What's in this image?. Do you find anything wrong with it medically?
If you make a differential, suggest some remedies for them. Donot add
any numbers or special characters in
your response. Your response should be in one long paragraph. Also
always answer as if you are answering to a real person.
Donot say 'In the image I see' but say 'With what I see, I think you
have ....'
Dont respond as an AI model in markdown, your answer should mimic
that of an actual doctor not an AI bot,
Keep your answer concise (max 2 sentences). No preamble, start your
answer right away please"""
def process_inputs(audio_filepath, image_filepath):
speech_to_text_output =
transcribe_with_groq(GROQ_API_KEY=[Link]("GROQ_API_KEY"),
audio_filepath=audio_filepath,
stt_model="whisper-large-v3")
# Handle the image input
if image_filepath:
doctor_response =
analyze_image_with_query(query=system_prompt+speech_to_text_output,
encoded_image=encode_image(image_filepath), model="meta-llama/llama-4-
scout-17b-16e-instruct") #model="meta-llama/llama-4-maverick-17b-128e-
instruct")
else:
doctor_response = "No image provided for me to analyze"
voice_of_doctor =
text_to_speech_with_elevenlabs(input_text=doctor_response,
output_filepath="final.mp3")
return speech_to_text_output, doctor_response, voice_of_doctor
# Create the interface
iface = [Link](
fn=process_inputs,
inputs=[
[Link](sources=["microphone"], type="filepath"),
[Link](type="filepath")
],
#gsk_P1Ho4kSsw7N6AgcZZWaDWGdyb3FYGxQOCRbgKgLFt0p2VTDKA6
pk GROQ_API
#sk_a58ef3cd2a8ee75922aabf74fd9d9dc249219555eed50bf4
ELEVENLABS_API_KEY
outputs=[
[Link](label="Speech to Text"),
[Link](label="Doctor's Response"),
[Link]("Temp.mp3")
],
title="AI Powered Medical Assistant"[Link](debug=True)
#[Link]