Urdu Advanced RAG — Implementation Overview

What It Does
Architecture at a Glance
Module Breakdown
Ingestion (ingestion/)
Retrieval (retrieval/)
Generation (generation/)
API ([Link])
Key Design Decisions
Storage Layout
Environment Variables (.env)

Urdu Advanced RAG — Implementation

Overview

What It Does
A Retrieval-Augmented Generation (RAG) pipeline built specifically for Urdu
books/OCR text. You upload a .txt Urdu file, it gets indexed, then you ask questions
in Urdu and get cited answers via Google Gemini.

Architecture at a Glance

Upload .txt
│
▼
┌─────────────┐ ┌──────────────┐ ┌──────────────────────────┐
│ Cleaner │────▶│ Chunker │────▶│ Embedder │
│ (normalize │ │ (sentence- │ │ multilingual-e5-large │
│ Urdu OCR) │ │ boundary │ │ → FAISS index (dense) │
│ │ │ overlap) │ │ → BM25 index (sparse) │
└─────────────┘ └──────────────┘ └──────────────────────────┘

Query (Urdu)
│
▼
┌────────────┐ ┌────────────┐
 │ FAISS │ │ BM25 │ ← Stage 1: Hybrid retrieval (top-20 each)
│ (dense) │ │ (sparse) │
└─────┬──────┘ └─────┬──────┘
└────────┬────────┘
▼
┌─────────────┐
│ RRF Fusion │ ← Stage 2: Reciprocal Rank Fusion → top-30
└──────┬──────┘
▼
┌─────────────┐
│ Reranker │ ← Stage 3: Cross-encoder ms-marco-MiniLM → top-k
└──────┬──────┘
▼
┌─────────────┐
│ Gemini │ ← Stage 4: Answer generation with citations
└─────────────┘

Module Breakdown

Ingestion (ingestion/)

File Role

NFC normalization, fixes Urdu char variants (‫ ك→ک‬,‫)ي→ی‬, strips

[Link]
diacritics, OCR noise, page numbers

Splits on Urdu sentence endings (‫)! ۔ ؟‬, produces overlapping

[Link]
chunks with token-count control

Embeds chunks with intfloat/multilingual-e5-large, saves

[Link]
FAISS + BM25 + metadata JSON to storage/

Chunk metadata stored per chunk: chunk_id, text, token_count, book_title,

author, chapter, page_start, page_end, position

Retrieval (retrieval/)

File Role

Dense cosine similarity search using FAISS IndexFlatIP.

faiss_retriever.py
Query gets "query: " prefix for e5 models.
File Role

Sparse keyword search using BM25Okapi (whitespace

bm25_retriever.py
tokenized for Urdu).

RRF fusion: score = Σ 1/(60 + rank) — rewards

[Link]
chunks appearing high in both lists.

Cross-encoder ms-marco-MiniLM-L-6-v2 re-scores query-

[Link]
chunk pairs. Falls back to RRF scores if unavailable.

Generation (generation/)

File Role

Builds the Urdu system prompt + numbered excerpts block. System

[Link] instruction is fully in Urdu — LLM is told to only answer from provided
excerpts.

Google Gemini wrapper (google-genai SDK). Supports both

[Link]
streaming (SSE tokens) and non-streaming responses.

API ([Link])

Endpoint Method Purpose

/ingest POST Upload .txt file + metadata → background indexing

Poll ingestion state

/ingest/status GET
(starting/cleaning/chunking/indexing/done/error)

Ask a question, returns answer + citations (streaming or

/query POST
JSON)

/chunks GET Browse/search all indexed chunks with pagination

/health GET Index stats + current ingestion state

/index DELETE Wipe all indexes and start fresh

Key Design Decisions

Hybrid retrieval — dense alone misses exact keyword matches in Urdu; sparse
alone misses semantic similarity. RRF combines both without needing score
normalization.
Sentence-boundary chunking — chunks never cut mid-sentence, preserving
meaning. Overlap ensures context at chunk boundaries is not lost.
e5 prefix convention — intfloat/multilingual-e5-large requires
"passage: " prefix at index time and "query: " at query time for correct cosine
alignment.
Cross-encoder reranking — bi-encoder retrieval is fast but approximate; the
cross-encoder does full query-passage attention for precise final ranking.
Urdu-only LLM instruction — the system prompt is written in Urdu and
constrains the model to only use retrieved excerpts, reducing hallucination.

Storage Layout

storage/
[Link] ← FAISS binary index (vectors)
[Link] ← BM25Okapi pickled object
[Link] ← All chunk metadata (text + source info)

All three are rebuilt atomically on each /ingest call and wiped by DELETE /index.

Environment Variables (.env)

Variable Default Purpose

Required. Google AI
GEMINI_API_KEY —
Studio key.

Gemini model for

LLM_MODEL gemini-2.0-flash
answer generation
Variable Default Purpose

intfloat/multilingual-e5- HuggingFace
EMBEDDING_MODEL
large embedding model

cross-encoder/ms-marco- HuggingFace cross-

RERANKER_MODEL
MiniLM-L-6-v2 encoder

FAISS candidates per

TOP_K_DENSE 20
query

BM25 candidates per

TOP_K_SPARSE 20
query

Set to 0 on first run to

TRANSFORMERS_OFFLINE 1
download models

Set to 0 on first run to

HF_HUB_OFFLINE 1
download models