Assignment: Multilingual Vernacular Voicebot with RAG
Context
You are building a voicebot for rural/semi-urban India where users ask queries about
government schemes, banking products, and agriculture advisory in their local language
(Hindi, Tamil, Bengali, etc.).
The system must:
1. Ingest documents (provided corpus: scheme FAQs, agriculture advisory notes).
2. Allow voice-based queries in a vernacular language.
3. Transcribe speech to text using an open-source STT model (e.g., AI4Bharat STT,
Whisper fine-tuned on Indic languages).
4. Use RAG pipeline with LangChain (or equivalent) to query the knowledge base.
Create knowledge base architecture so that we it can be fed regularly for new
schemes/products etc. This includes scripts, FAQs and other documentations.
5. Generate an LLM-powered response in the same language as the query.
○ Use AI4Bharat LLMs (IndicGPT) etc. for translation/generation in vernacular.
○ Use another LLM (e.g., Llama-2, Mistral, Falcon, etc.) for reasoning/summary in
English if needed → then translate back.
6. Convert the final response to speech using Indic TTS (e.g., Indic-TTS, Coqui,
AI4Bharat).
Tasks
1. Corpus Preparation
○ Load the provided multilingual dataset (JSON/CSV with govt schemes in
English + Hindi + one regional language).
○ Build a vector store index (FAISS/Chroma/Weaviate).
2. Speech-to-Text (STT)
○ Integrate an open-source STT model (AI4Bharat STT / Whisper fine-tuned on
Indic).
○ Handle at least one vernacular language + code-mixed inputs (e.g., “PM Kisan ka
benefit batao please”).
3. RAG Pipeline
○ Implement with LangChain (or equivalent custom orchestration).
○ Must support:
■ Language detection → query processing.
■ Translation (if needed): vernacular → English (use AI4Bharat
translation).
■ Context retrieval + generation with an LLM (open-source model of
candidate’s choice).
■ Response re-translation into the user’s vernacular.
4. Conversational Layer
○ Maintain short-term memory (at least last 3 turns).
○ Handle follow-ups like:
■ User: “PM Kisan Yojana ke baare mein batao.”
■ User: “Isme loan waiver milta hai kya?” (should link to previous query).
5. Text-to-Speech (TTS)
○ Generate vernacular voice response using AI4Bharat TTS or other Indic TTS
models.
6. Confidence & Hallucination Handling
○ Implement confidence scoring (retriever + generator).
○ If confidence < threshold → bot should reply “Mujhe is vishay mein jaankari nahi
hai.”
Deliverables
1. GitHub repo with:
○ Code (STT + RAG + TTS pipeline).
○ Configurable settings for language selection.
○ Dockerfile (bonus).
2. Demo CLI / Streamlit app:
○ Input: audio file in vernacular language.
○ Output: transcript, retrieved context, generated text response, and audio reply.
3. Technical write-up (max 3 pages):
○ Model choices (why AI4Bharat, which LLMs).
○ Orchestration flow.
○ Handling of language consistency.
○ Anti-hallucination strategies.
Evaluation Rubric
Criterion Weight What’s Assessed
End-to-end 25% Working STT → RAG → TTS pipeline
functionality
RAG implementation 25% LangChain (or custom), retrieval accuracy, grounded
answers
Vernacular handling 20% AI4Bharat usage, translation quality, code-mixed input
handling
Conversational flow 15% Context memory, follow-up handling
Code quality 10% Structure, modularity, documentation
Bonus features 5% Docker, GPU optimization, multi-turn demo