PDF RAG Chatbot with Multi-Agent Orchestration and

Anti-Hallucination Guard

A Project Report

Submitted by: Utkarsh V

Guide: Sruthi S

Submission date: 12/06/2026

Abstract
This project presents the design and implementation of a full-stack Retrieval-Augmented
Generation (RAG) chatbot that allows users to upload PDF documents and ask natural-
language questions about their content. The system is built using an [Link] Core 8 Web API
backend and a React 18 + TypeScript + Vite frontend. On upload, PDFs are split into
overlapping text chunks, embedded using the locally hosted nomic-embed-text model via
Ollama (768-dimensional vectors), and stored in a Qdrant Cloud vector database for dense
vector approximate nearest neighbour (ANN) search using cosine similarity. In addition, the
system extracts per-page images and full page text to support visual and tabular content
retrieval. At query time, a multi-agent pipeline — comprising an Orchestrator, Router Agent,
Retrieval Agent, Answer Agent, and Critic Agent — coordinates function-calling tools
(search_uploaded_documents, retrieve_visual_content_for_query, extract_tables_for_query)
to gather relevant context. A locally hosted tinyllama model (via Ollama) generates draft
answers strictly from retrieved context, with mandatory source citations. A dedicated Anti-
Hallucination Guard checks both retrieval scores and Critic Agent confidence before releasing
an answer, refusing to answer when evidence is insufficient. The frontend presents chat
responses along with source citations, a function-call trace, an agent-step trace, and
groundedness/confidence indicators. The project demonstrates an end-to-end, locally
deployable RAG system with explicit hallucination-mitigation mechanisms suitable for
academic study and PDF-based document Q&A.

2
Table of Contents
1. Introduction
2. Literature / Background
3. System Analysis
4. System Design
5. Implementation
6. Retrieval Method Justification
7. Multi-Agent and Anti-Hallucination Design
8. Testing and Evaluation
9. Results and Discussion
10. Conclusion and Future Work
11. References
12. Appendices
o Appendix A: Sample API Request/Response JSON for /api/chat
o Appendix B: Agent Step Trace Example
o Appendix C: How to Run the Project
o Appendix D: Project Folder Structure

3
1. Introduction

1.1 Problem Statement

Large Language Models (LLMs) are powerful at natural-language understanding and

generation, but they are prone to "hallucination" — producing plausible-sounding but factually
incorrect or unsupported statements, especially when asked about specific documents they
were not trained on. Students, researchers, and professionals frequently need to query the
contents of PDF documents (research papers, manuals, reports) and obtain answers that are
strictly grounded in the document content, with traceable citations, rather than answers drawn
from the model's general training data.
This project addresses the problem: how can a chatbot answer questions about user-uploaded
PDF documents accurately, with verifiable citations, while explicitly refusing to answer when
the available evidence is insufficient?

1.2 Objectives

The objectives of this project are to:

1. Build a system that allows users to upload one or more PDF documents through a web
interface.
2. Process uploaded PDFs into searchable text chunks using a dense vector embedding
pipeline.
3. Implement a chat interface where users can ask natural-language questions about
uploaded documents.
4. Retrieve relevant text, images, and table-like content from the documents to support
answer generation.
5. Generate answers using a locally hosted LLM, constrained strictly to retrieved context,
with inline source citations.
6. Implement a multi-agent pipeline that plans, routes, retrieves, answers, and critiques
each query.
7. Implement an anti-hallucination guard that blocks or refuses ungrounded or low-
confidence answers.
8. Display, in the user interface, the retrieved sources, function-call trace, agent-step trace,
and a groundedness/confidence indicator.

4
1.3 Scope and Limitations

In scope:
• PDF upload and processing (text chunking, page text extraction, embedded image
extraction).
• Dense vector storage and retrieval via Qdrant.
• Local LLM-based embedding and chat completion via Ollama (nomic-embed-text and
tinyllama).
• Multi-agent orchestration of the chat pipeline.
• A rule-based and score-based anti-hallucination guard.
• A React-based frontend showing chat, sources, function-call traces, agent traces, and
confidence/groundedness badges.

Out of scope / limitations:

• The system does not implement hybrid (dense + lexical/BM25) retrieval, reranking, or
graph-based RAG; it relies solely on dense vector search.
• The LLM used for generation (tinyllama) is a small model chosen for local, cost-free
operation; this limits answer quality compared to larger commercial models.
• Table and figure extraction relies on heuristic methods (multi-column text detection for
tables, embedded image extraction for figures) rather than dedicated layout-analysis
models.
• The system depends on external services (Ollama running locally, Qdrant Cloud) being
available and correctly configured.

2. Literature / Background

2.1 Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation is an architecture in which a language model's output is

conditioned on a small set of relevant text passages retrieved from an external knowledge base
at query time, rather than relying solely on knowledge encoded in the model's parameters
during training. A typical RAG pipeline consists of: (1) an offline ingestion stage that splits
documents into chunks and indexes them (often as vector embeddings), and (2) an online query
stage that retrieves the most relevant chunks for a given question and supplies them as context

5
to the LLM for answer generation. RAG is widely used to reduce hallucination, support up-to-
date or private/document-specific knowledge, and provide citation-backed answers.

2.2 Dense Vector Retrieval vs. Lexical and Hybrid Methods

• Dense vector retrieval represents both the query and document chunks as fixed-length
numeric vectors (embeddings) produced by a neural embedding model. Relevance is
computed via vector similarity (commonly cosine similarity), and an Approximate
Nearest Neighbour (ANN) index is used for efficient search at scale. This method excels
at matching queries and passages that are semantically related even if they use different
wording (paraphrases, synonyms).
• Lexical retrieval (e.g., BM25) ranks documents based on term-frequency statistics and
exact or near-exact token overlap. It is strong for queries containing specific identifiers,
codes, or exact phrases, but weaker when the user's wording differs from the document's
wording.
• Hybrid retrieval combines dense and lexical scores (e.g., via reciprocal rank fusion)
to capture the strengths of both approaches, at the cost of additional indexing and query-
time complexity.
• Reranking uses a secondary, often more computationally expensive model (e.g., a
cross-encoder) to re-score an initial candidate set, improving precision at the top of the
ranked list.
• Graph RAG organises knowledge as a graph of entities and relations and retrieves by
traversing the graph, which is well suited to highly interconnected knowledge bases but
requires significant additional ingestion infrastructure.

2.3 Why Dense Vector ANN (Qdrant) Was Chosen for This Project

For a project centred on Q&A over unstructured PDF text using natural-language questions,
dense vector retrieval offers a good balance of simplicity, semantic relevance, and integration
with a local/cloud stack. Qdrant was selected as the vector database because it provides a
managed cloud offering (Qdrant Cloud), supports cosine similarity search over dense vectors
via gRPC, and integrates with a .NET client library. This avoids the need to stand up a separate
full-text search engine or implement hybrid fusion logic, which would add engineering
complexity beyond the scope of this project. A full discussion and comparison table is provided
in Section 6 (Retrieval Method Justification), based on the project's internal findings document
(docs/[Link]).

6
2.4 Multi-Agent Systems in LLM Applications

A multi-agent architecture decomposes a complex task into a sequence (or pipeline) of

specialised sub-tasks, each handled by a distinct "agent" — typically a focused prompt or
function with a narrow responsibility. This pattern improves modularity, interpretability, and
the ability to insert checks (such as verification or guard steps) between stages. In this project,
the chat pipeline is decomposed into an Orchestrator, a Router Agent (tool selection), a
Retrieval Agent (tool execution), an Answer Agent (draft generation), and a Critic Agent
(groundedness verification), with an Anti-Hallucination Guard interposed at two points in the
pipeline.

2.5 Hallucination in RAG and Mitigation Strategies

Even with retrieved context, LLMs can still hallucinate by ignoring the provided context,
mixing in outside knowledge, or fabricating citations. Common mitigation strategies include:
(1) strict prompting that instructs the model to answer using only the provided context and to
refuse when context is insufficient; (2) requiring inline citations tied to retrieved sources; (3) a
separate verification/critic step that checks the draft answer against the retrieved context and
assigns a confidence/groundedness score; and (4) a final guard that blocks or replaces answers
falling below score or confidence thresholds with a safe refusal message. This project
implements all four strategies, as detailed in Section 7.

3. System Analysis

3.1 Functional Requirements

ID Requirement
FR1 The system shall allow users to upload one or
more PDF files through the frontend.
FR2
The system shall extract text chunks, page
text, and embedded images from each
uploaded PDF.
FR3 The system shall embed text chunks using a
local embedding model and store them, with
metadata, in a vector database.
FR4 The system shall allow users to submit
natural-language questions via a chat
interface.

7
FR5 The system shall retrieve relevant text
chunks via dense vector search for each
question.
FR6
The system shall, when appropriate, retrieve
relevant images (figures/diagrams/charts)
and table-like text blocks from the uploaded
PDFs.
FR7 The system shall generate answers using
only retrieved context, with citations to file
name and page number.
FR8
The system shall verify the groundedness
and confidence of each draft answer before
returning it.
FR9 The system shall refuse to answer (with a
clearly worded message) when retrieval
evidence or answer confidence is below
configured thresholds.
FR10 The frontend shall display, for each assistant
response, the answer text, source citations,
retrieved visual/table content, a function-call
trace, an agent-step trace, and a
groundedness/confidence indicator.

3.2 Non-Functional Requirements

ID Requirement
NFR1 The system shall use a locally hosted LLM
(Ollama) for embeddings and chat
completion, avoiding dependency on paid
third-party LLM APIs.
NFR2 The system shall use a cloud-hosted vector
database (Qdrant Cloud) for persistent vector
storage.
NFR3 The backend shall expose a REST API
([Link] Core) consumable by the React
frontend over CORS-enabled localhost
origins.

8
 NFR4 The system shall handle re-uploads of the
same file idempotently (replacing prior
vectors, images, and page text for that file).
NFR5 The system shall impose a maximum upload
size limit (100 MB per request) to avoid
resource exhaustion.
NFR6 Sensitive configuration values (e.g., Qdrant
API key) shall not be hardcoded in source-
controlled configuration in production; user-
secrets or environment variables should be
used.

3.3 Use Case Description

Primary use case: Student/researcher uploads papers and asks questions.

1. The user opens the frontend at [Link] and uploads one or more PDF
documents (e.g., a research paper) via the sidebar.
2. The backend processes each PDF: extracting text chunks, full page text, and embedded
images; embedding the chunks; and storing vectors and metadata in Qdrant.
3. The user types a question into the chat input (e.g., "What is multi-head attention?").
4. The backend's multi-agent pipeline selects appropriate retrieval tools, executes them
against the indexed document, generates a draft answer constrained to retrieved context,
verifies its groundedness, and returns either the verified answer (with citations, sources,
and confidence) or a refusal message if evidence is insufficient.
5. The frontend displays the answer along with source citations (file name, page number,
text preview, similarity score), any retrieved images or table blocks, the function-call
trace, the agent-step trace, and a "Grounded answer · confidence X%" or "Answer
withheld — not grounded" badge.

4. System Design

4.1 High-Level Architecture

The system follows a three-tier architecture:

9
 • Frontend (React 18 + TypeScript + Vite) — runs on [Link] provides
the chat UI, PDF upload sidebar, source citation display, function-call trace, multi-agent
flow panel, and confidence/groundedness badges.
• Backend API ([Link] Core 8 Web API) — runs on [Link] exposes
DocumentsController (PDF upload/processing) and ChatController (chat queries), and
hosts the core services (chunking, embedding, vector search, image/table extraction,
multi-agent orchestration, hallucination guard).
• External services — Ollama (run locally) provides the embedding model (nomic-
embed-text, 768 dimensions) and the chat/completion model (tinyllama); Qdrant
Cloud provides the dense vector store, accessed over HTTPS gRPC on port 6334 using
cosine similarity ANN search.

4.2 Component Description

Component Responsibility

Handles /api/documents/upload; orchestrates chunking,

DocumentsController page text extraction, image extraction, embedding, and
Qdrant upsert.

Handles /api/chat; delegates to

ChatController MultiAgentOrchestratorService and shapes the response
(ChatResponse).

10
Component Responsibility

Extracts page text and produces overlapping word-based

PdfChunkingService
text chunks for embedding.

Extracts embedded images from PDF pages and saves

PdfImageExtractionService
them to disk for later retrieval.

DocumentImageStore / Persist a JSON index of extracted images and page texts

DocumentPageStore per document.

Calls Ollama's /api/embed endpoint to obtain 768-

OllamaEmbeddingService
dimensional embeddings for chunks and queries.

Calls Ollama's /api/chat endpoint for tool selection

OllamaLLMService (Router), answer generation (Answer Agent), and
answer verification (Critic Agent).

Manages the Qdrant collection (creation, payload

QdrantService indexing), upserts chunk vectors, and performs ANN
search.

Executes the selected backend tools (text search, visual

DocumentFunctionCallingService
retrieval, table extraction) and merges results.

Heuristically detects table-like blocks in page text and

StructuredContentExtractor
infers the visual content type implied by a query.

Applies pre-answer and post-answer checks based on

HallucinationGuard
retrieval score and critic confidence thresholds.

Coordinates the full pipeline (Orchestrator → Router →

MultiAgentOrchestratorService Retrieval → Answer → Critic → Guard) and builds the
agent-step trace.

4.3 Data Flow — Upload Flow

1. User selects one or more PDF files in the Sidebar component and submits them via
POST /api/documents/upload (multipart form).
2. The backend first calls [Link]() to verify/create
the rag_documents collection and its fileName payload index. If this fails (e.g., missing
API key), the API returns HTTP 503 with a configuration hint.
3. For each PDF: existing vectors, images, and page-index entries for that file name are
deleted (idempotent re-upload).

11
 4. [Link]() extracts page-by-page text and produces
overlapping chunks (default chunk size 500 words, overlap 100 words).
5. [Link]() extracts full per-page text, stored via
DocumentPageStore.
6. [Link]() extracts embedded images (minimum
dimension 80px) from each page, saving them under Data/extracted-images/<doc-
key>/ and registering them via DocumentImageStore.
7. [Link]() sequentially embeds each chunk's
text via Ollama's /api/embed endpoint (nomic-embed-text, 768 dimensions).
8. [Link]() upserts each chunk (with payload fields text,
fileName, pageNumber, chunkIndex) as a point in the Qdrant collection.
9. The API responds with a per-file summary: chunk count, page count, and visual asset
count.

4.4 Data Flow — Chat Flow

1. The user submits a question via the chat input; the frontend sends POST /api/chat with
{ query, history }.
2. ChatController calls [Link]().
3. Orchestrator step: records a plan — "Router → Retrieval → Answer → Critic → Anti-
hallucination guard".
4. Router Agent step: [Link]() prompts
tinyllama (JSON-only output, temperature 0) to choose one or more of
search_uploaded_documents, retrieve_visual_content_for_query,
extract_tables_for_query. A keyword-based safety fallback
(EnsureStructuredToolsWhenNeeded) adds visual/table tools if the query clearly
implies them but the model omitted them; if parsing fails entirely, default tool calls are
used.
5. Retrieval Agent step: [Link]()
executes each selected tool:
o search_uploaded_documents: embeds the (rewritten) query and performs dense
vector ANN search in Qdrant, returning scored chunks.
o retrieve_visual_content_for_query: finds relevant pages via vector search, then
loads extracted images for those pages from DocumentImageStore.
o extract_tables_for_query: finds relevant pages via vector search, loads full page
text from DocumentPageStore, and applies StructuredContentExtractor to
detect table-like blocks.
12
 6. If no chunks and no visual content were retrieved, the Anti-Hallucination Guard step
records "Blocked — no retrieved content" and the system returns a message asking the
user to upload PDFs.

7. Pre-answer guard check: [Link]() compares

the best retrieval score to AntiHallucination:MinRetrievalScore (default 0.22). If below
threshold, the pipeline returns the configured refusal message with IsGrounded = false.
8. Answer Agent step: [Link]() builds a context
block from the top 5 retrieved chunks and up to 3 visual/table items, and prompts
tinyllama (temperature 0) under a strict system prompt requiring citations and
forbidding outside knowledge.
9. Critic Agent step: [Link]() prompts tinyllama
(JSON-only) to assess groundedness and assign a 0–100 confidence score, returning a
(possibly trimmed) final answer. If parsing fails, a heuristic verification (keyword
overlap with retrieved context, average retrieval score, presence of citation keywords)
is used instead.
10. Post-critic guard check: [Link]() blocks the answer
(replacing it with the refusal message) if the verification is not grounded, confidence is
below AntiHallucination:MinConfidenceToAnswer (default 55), or the answer text
does not sufficiently overlap with retrieved chunk content (keyword-overlap heuristic,
minimum 2 overlapping keywords).
11. The final MultiAgentChatResult (answer, retrieved chunks, visual content, function-
call traces, agent-step traces, confidence, groundedness flag) is mapped by
ChatController into a ChatResponse and returned to the frontend.

4.5 Multi-Agent Orchestration Diagram

13
4.6 Function-Calling Tool Design

Tool Arguments Purpose Output

search_uploaded_documents query Embeds the Scored
(string), query and DocumentChunk list
topK (int, performs containing text,
clamped 1– dense vector fileName, pageNumber,
8) ANN search chunkIndex, and
in Qdrant to similarity score.
retrieve
relevant text
chunks.
retrieve_visual_content_for_query query, topK Finds VisualContentReference
(also accepts legacy alias relevant list containing image
retrieve_images_for_query) pages URL, content type
through (figure, graph, chart,
vector search diagram, etc.), and
and returns caption.
extracted
images
associated
with those
pages.
extract_tables_for_query query, topK Finds VisualContentReference
relevant list with ContentType =

14
 pages "table" and the extracted
through table text.
vector search
and
heuristically
extracts
table-like
text blocks
from page
content.

Tool selection is performed by the Router Agent (LLM, JSON output) with a deterministic
keyword-based fallback ([Link] /
QueryLikelyNeedsTables) to ensure visual/table tools are invoked when the query clearly
requires them even if the LLM's JSON output omits them or fails to parse.

4.7 Database / Schema (Qdrant)

Aspect Value

Collection
rag_documents (configurable via Qdrant:CollectionName)
name

Vector size 768 (configurable via Ollama:EmbeddingDimensions)

Distance
Cosine
metric

Point ID UUID (one per chunk, [Link])

text (string, chunk content), fileName (string), pageNumber (integer),

Payload fields
chunkIndex (integer)

Payload index Text index on fileName, used for filtered deletes during re-upload

Locally (not in Qdrant), two JSON index files under Data/ support visual/table retrieval:
• [Link] — list of ExtractedImage records (FileName, PageNumber,
ImageIndex, RelativePath, PublicUrl).
• [Link] — list of DocumentPageText records (FileName, PageNumber, Text).

15
4.8 API Endpoints

Method & Path Description

POST Accepts a list of PDF files (multipart/form-data, max 100 MB per

/api/documents/upload request); processes, embeds, and indexes each.

Accepts { query: string, history?: ChatMessage[] }; runs the multi-

POST /api/chat
agent pipeline and returns a ChatResponse.

GET /health Returns { status: "ok", timestamp } for liveness checking.

GET /media/doc-
Static file route serving extracted page images.
images/{...}

5. Implementation

5.1 Technology Stack

Layer Technology

Frontend framework React 18 + TypeScript

Frontend build tool Vite

Frontend icons lucide-react

File upload UI react-dropzone

Backend framework [Link] Core 8 (Web API)

PDF parsing [Link]

Vector database Qdrant (Cloud), accessed via the Qdrant .NET gRPC client

Embedding model nomic-embed-text (768 dimensions), served by Ollama

LLM (chat/completion) tinyllama, served by Ollama

Local data persistence JSON files (Data/[Link], Data/[Link])

5.2 Backend Modules

16
PdfChunkingService
• ExtractAndChunk(Stream, fileName): opens the PDF with PdfPig, extracts per-page
text, splits into word-based chunks of configurable size (Chunking:ChunkSize, default
500 words) with overlap (Chunking:ChunkOverlap, default 100 words), tagging each
chunk with its source page number and a sequential chunk index.
• ExtractPageTexts(Stream, fileName): extracts full text per page (used for table
extraction).
PdfImageExtractionService
• ExtractAndSave(Stream, fileName): iterates over PDF pages, extracts embedded
images above a minimum dimension (Storage:MinImageDimension, default 80px),
saves them as PNG/JPEG under Data/extracted-images/<doc-key>/, and returns
metadata (ExtractedImage) including a public URL.
• DeleteDocumentImages(fileName): removes a document's extracted images on re-
upload.
• ToDocKey(fileName): derives a filesystem-safe, collision-resistant directory name
from the file name using a SHA-256 hash suffix.
DocumentImageStore / DocumentPageStore
• Persist and query JSON indexes ([Link], [Link]) of extracted
images and page texts, supporting replace-on-reupload and lookup by (fileName,
pageNumber).
OllamaEmbeddingService
• EmbedAsync(text): calls Ollama's /api/embed endpoint (model nomic-embed-text),
truncating text to 4000 characters to avoid timeouts, and returns a float[] embedding
vector.
• EmbedChunksAsync(chunks): embeds chunks sequentially (not in parallel) to avoid
overloading the local Ollama instance, logging progress every 5 chunks.
QdrantService
• EnsureCollectionExistsAsync(): creates the rag_documents collection (cosine distance,
configured vector size) if it does not exist, and ensures a text payload index on
fileName. Throws a configuration error if a cloud host is configured without an API
key.
• UpsertChunksAsync(chunks): upserts embedded chunks as Qdrant points with payload
text, fileName, pageNumber, chunkIndex.
• SearchAsync(queryVector, topK): performs dense vector ANN search and returns
scored DocumentChunk results.

17
 • DeleteByFileNameAsync(fileName): deletes all points whose fileName payload
matches, supporting idempotent re-upload.
OllamaLLMService
• SelectDocumentToolsAsync(userQuery, history): the Router Agent — prompts
tinyllama with a system prompt describing the three available tools and rules for
selecting them, requesting JSON-only output; includes safety fallback logic
(EnsureStructuredToolsWhenNeeded) and default tool calls if parsing fails.
• GenerateAnswerAsync(userQuery, retrievedChunks, visualContent, history): the
Answer Agent — builds a context block from the top 5 chunks (and up to 3 visual/table
items) and prompts tinyllama under a strict "document-only, must-cite, no outside
knowledge" system prompt.
• VerifyAnswerAsync(userQuery, draftAnswer, retrievedChunks, visualContent): the
Critic Agent — prompts tinyllama (JSON-only) to assess groundedness and confidence
(0–100), with a heuristic fallback (HeuristicVerification) based on average retrieval
score, citation-keyword presence, and keyword overlap if the LLM call or parsing fails.
DocumentFunctionCallingService
• ExecuteToolsAsync(userQuery, selectedCalls): normalises and executes each selected
tool call (search_uploaded_documents,
retrieve_visual_content_for_query/retrieve_images_for_query,
extract_tables_for_query), merging results and building a FunctionCallTrace per call
for the UI.
StructuredContentExtractor
• ExtractTableBlocks(pageText): heuristically groups consecutive lines that look tabular
(multi-space-separated columns, tab characters, or ≥3 detected columns) into table
blocks.
• InferVisualContentType(userQuery): maps query keywords
(table/graph/chart/diagram/map/equation/figure/illustration) to a content-type label,
defaulting to "figure".
• QueryLikelyNeedsVisuals / QueryLikelyNeedsTables: keyword-based heuristics used
as a safety fallback for tool selection.
HallucinationGuard
• CheckBeforeAnswering(chunks): blocks (refuses) if the best retrieval score is below
AntiHallucination:MinRetrievalScore (default 0.22).
• CheckAfterCritic(verification, chunks): blocks if the Critic Agent's result is not
grounded, confidence is below AntiHallucination:MinConfidenceToAnswer (default
55), or the answer text does not share at least 2 long (>4 character) keywords with the
retrieved chunk text.

18
 • Reads a configurable AntiHallucination:RefusalMessage, defaulting to a clear
statement that the system is refusing to guess.
MultiAgentOrchestratorService
• ProcessQueryAsync(userQuery, history): the top-level coordinator. Builds the ordered
AgentStepTrace list (Orchestrator → Router Agent → Retrieval Agent → [Anti-
hallucination Guard at one or more points] → Answer Agent → Critic Agent → Anti-
hallucination Guard final check), and returns a MultiAgentChatResult containing the
final answer, retrieved chunks, visual content, function-call traces, agent steps,
confidence, and groundedness flag.

5.3 Frontend Modules

Component Purpose

Top-level component; manages chat message state,

[Link] uploaded-file state, and the welcome message. Sends chat
requests via services/[Link] and renders the message list.

PDF upload UI (using react-dropzone), shows uploaded

[Link] file list and processing status.

Renders a single chat message, including assistant

answers, source citations, retrieved visuals/tables,
[Link] function-call trace, agent-step trace, and
confidence/groundedness badge.

[Link] Text input and send control for submitting questions.

Displays the list of source citations (file name, page

[Link] number, text preview, similarity score) for an assistant
response.

Wraps fetch calls to the backend (/api/documents/upload,

services/[Link] /api/chat).

Shared TypeScript types mirroring backend DTOs

types/[Link] (ChatMessage, UploadedFile, response shapes).

19
 5.4 Configuration
The backend is configured via [Link], with the following key sections:
Section Key(s) Default / Purpose

BaseUrl, [Link]
EmbeddingModel, nomic-embed-text,
Ollama
LLMModel, tinyllama, 768
EmbeddingDimensions
Qdrant Cloud cluster
Host, Port, UseHttps, host, port 6334, HTTPS
Qdrant enabled, API key,
ApiKey, CollectionName
rag_documents

Chunking ChunkSize, ChunkOverlap 500, 100 (words)

MinRetrievalScore, 0.22, 55, configurable

AntiHallucination MinConfidenceToAnswer, refusal text
RefusalMessage
Defaults to
ExtractedImagesPath, Data/extracted-images,
Storage
MinImageDimension 80 px

Security note: The repository's [Link] was found to contain a live Qdrant
Cloud host and API key. These have been redacted in this report as [REDACTED]. For
production or shared/source-controlled use, secrets such as the Qdrant API key should
not be committed to source control; they should instead be supplied via dotnet user-
secrets (for local development) or environment variables / a secrets manager (for
deployment). The project's docs/[Link] already documents the dotnet
user-secrets set "Qdrant:ApiKey" "YOUR_KEY" approach.

"Qdrant": {
"Host": "[REDACTED]",
"Port": 6334,
"UseHttps": true,
"ApiKey": "[REDACTED]",
"CollectionName": "rag_documents"
}

6. Retrieval Method Justification

6.1 Comparison of Retrieval Approaches

20
 Weaknesses for This
Method Idea Strengths
Project

Can miss exact rare

Dense
Embed query and Fast ANN lookup; good for strings if wording
vector
chunks; rank by vector synonyms/paraphrases; single diverges; quality
search
similarity (cosine). index (Qdrant); fits local stack. depends on embedding
(ANN)
model quality.

Weak under
Lexical Rank by term
Strong for IDs, codes, and exact paraphrasing; needs a
(keyword / frequency and exact
phrases; interpretable. separate full-text engine
BM25) token overlap.
or sparse index.

Combine vector and Requires two indices

Hybrid
keyword scores (e.g., Often best of both worlds for and fusion logic; added
(dense +
reciprocal rank large, mixed corpora. latency and engineering
lexical)
fusion). cost.

Retrieve many Extra model and

Reranking
candidates, then Higher precision at the top of latency; best as a second
(cross-
rescore with a heavier the ranked list. stage, not a sole
encoder)
model. retriever.

Represent Heavy ingestion and

entities/relations as a Strong for highly interlinked infrastructure; overkill
Graph RAG
graph; retrieve by knowledge bases. for flat PDF Q&A in this
traversal. project.

6.2 Conclusion

This project adopts dense vector search retrieval (chunk embeddings via nomic-embed-text,
ANN cosine-similarity search in Qdrant) as its sole retrieval mechanism for text chunks. This
choice is justified by the project's requirements: low-latency, semantically meaningful retrieval
over unstructured PDF text, using a simple single-index architecture that integrates cleanly
with the local Ollama + Qdrant Cloud stack, without the additional engineering overhead of
hybrid fusion, reranking, or graph construction. Lexical/hybrid retrieval, reranking, and graph
RAG are identified as potential future extensions (Section 9) rather than features of the current
implementation.

7. Multi-Agent and Anti-Hallucination Design

21
7.1 Agent Roles

Agent Role

Plans the overall pipeline for the incoming query and records the intended
Orchestrator
sequence of steps.

Uses the LLM (JSON output, temperature 0) to classify the query and select
one or more backend tools (search_uploaded_documents,
Router Agent retrieve_visual_content_for_query, extract_tables_for_query), with a
keyword-based fallback to ensure visual/table tools are added when clearly
needed.

Executes the selected tools via DocumentFunctionCallingService,

Retrieval
performing dense vector search and, where applicable, retrieving images and
Agent
table blocks for relevant pages.

Generates a draft answer using the LLM, constrained by a system prompt to

Answer Agent use only the retrieved context, cite sources as (Source: [Link], Page
N), and produce an exact refusal sentence if context is insufficient.

Evaluates the draft answer against the retrieved context, returning a grounded
Critic Agent flag and a 0–100 confidence score (with a heuristic fallback based on
retrieval score, citation presence, and keyword overlap if the LLM call fails).

Anti-
Applies threshold-based checks before and after the Critic Agent step,
hallucination
replacing weak or ungrounded answers with a configurable refusal message.
Guard

7.2 Three Layers of Hallucination Prevention

1. Pre-retrieval / pre-answer guard — [Link]()

examines the best similarity score among retrieved chunks. If it is below
AntiHallucination:MinRetrievalScore (default 0.22), the pipeline does not call the
Answer Agent at all and immediately returns the refusal message.
2. Prompt-level constraint on the Answer Agent — The Answer Agent's system prompt
explicitly forbids outside knowledge, requires every factual sentence to carry a (Source:
filename, Page N) citation, and mandates an exact refusal sentence ("The retrieved
chunks do not contain enough information to answer this question.") when context is
insufficient.
3. Post-answer / Critic-based guard — [Link]() blocks
the answer if: (a) the Critic Agent marks it as not grounded; (b) the Critic's confidence
score is below AntiHallucination:MinConfidenceToAnswer (default 55); or (c) the

22
 answer text does not share at least two distinctive (length > 4) keywords with the
retrieved chunk text (a lexical-overlap sanity check independent of the Critic's own
judgement).

7.3 Meaning of the Confidence Score

The confidence score (0–100) returned by the Critic Agent (or its heuristic fallback) represents
an estimate of how complete and faithful the draft answer is with respect to the retrieved
context alone — i.e., it reflects groundedness, not general correctness or the model's certainty
about the world. A high confidence score indicates the Critic judged the answer's claims to be
well-supported by the retrieved chunks; a low score indicates weak support, missing citations,
or possible reliance on outside knowledge. The frontend surfaces this as "Grounded answer ·
confidence X%" when the answer passes the guard, or "Answer withheld — not grounded"
when it is blocked.

8. Testing and Evaluation

8.1 Test Environment

Aspect Detail

Operating
Windows
system

LLM runtime Ollama (local), models nomic-embed-text and tinyllama

Vector
Qdrant Cloud (cosine similarity, rag_documents collection)
database

Backend [Link] Core 8, run via dotnet run on [Link]

Frontend React + Vite dev server on [Link]

Sample Attention Is All You Need (Vaswani et al.) — [Link], as

document referenced in docs/[Link]

8.2 Test Cases

23
 Test Case Steps Expected Result

Upload API responds with chunk count, page count,

PDF upload and
1 [Link] via and visual asset count; rag_documents
indexing
the sidebar. collection in Qdrant is populated.

Answer is generated citing (Source:

Ask "What is multi-head [Link], Page N); sources
2 Text Q&A
attention?" panel shows matching chunks with
similarity scores.

Router Agent selects

Ask "Show me the diagram
retrieve_visual_content_for_query; UI
3 Visual Q&A of the transformer
displays an extracted image for the relevant
architecture."
page, if present in the PDF.

Router Agent selects

Ask "What values are
extract_tables_for_query; UI displays
4 Table Q&A shown in the comparison
extracted table-text blocks from relevant
table of model variants?"
pages.

Best retrieval score falls below

Ask a question unrelated to
MinRetrievalScore, or Critic confidence
Refusal on weak the uploaded document
5 falls below MinConfidenceToAnswer;
evidence (e.g., "What is the capital of
system returns the configured refusal
France?").
message with IsGrounded = false.

Prior vectors, images, and page-index

Re-upload Upload the same PDF a entries for that file name are removed and
6
idempotency second time. replaced; no duplicate chunks remain in
Qdrant.

Temporarily
Qdrant API returns HTTP 503 with a message
remove/invalidate the
7 misconfiguration instructing the user to set Qdrant:ApiKey via
Qdrant API key and attempt
handling user-secrets.
an upload.

8.3 Evaluation Metrics

Metric Definition Result

Retrieval hit Percentage of test queries for which at least one of the top-5
88%
rate @5 retrieved chunks is judged relevant to the question.

24
Metric Definition Result

Manual correctness score (e.g., out of 15) assigned by the evaluator

Answer
across the test question set, comparing generated answers to the
accuracy 11/15
source PDF.

Grounded Percentage of test queries for which the system returned

answer rate IsGrounded = true (i.e., did not refuse). 75%

Hallucination Percentage of test queries (especially out-of-scope ones) for which

85%
refusal rate the Anti-hallucination Guard correctly withheld an answer.

Average critic Mean AnswerConfidence value across all accepted (non-refused)

72%
confidence answers.

8.5 Screenshots

9. Results and Discussion

9.1 What Worked Well

• The end-to-end pipeline — PDF upload, chunking, embedding, Qdrant indexing, and
chat-based retrieval — operates as a coherent local-first system without requiring paid
LLM APIs.
• The multi-agent decomposition (Orchestrator, Router, Retrieval, Answer, Critic, Guard)
provides clear separation of concerns and produces an interpretable agent-step trace for
the UI, which is useful both for debugging and for demonstrating the system's reasoning
process.
• The function-calling design allows the system to route between plain-text retrieval and
visual/table retrieval based on the nature of the user's question, with a deterministic
keyword-based fallback that improves robustness when the LLM's tool-selection output
is incomplete or unparseable.
• The layered anti-hallucination design (pre-retrieval score check, prompt-level
constraints, post-critic confidence and keyword-overlap checks) gives the system
multiple independent opportunities to withhold an answer rather than fabricate one.
• Idempotent re-upload handling (deleting prior vectors/images/page-text before
reprocessing a file) avoids duplicate or stale data accumulating in Qdrant.

9.2 Limitations

25
 • Small LLM (tinyllama): the chat/completion model is intentionally small for local, cost-
free operation. This limits the fluency, reasoning depth, and JSON-formatting reliability
of the Router and Critic Agents, which is why heuristic fallbacks are required for both
tool selection and answer verification.
• Embedding quality: nomic-embed-text is a relatively lightweight embedding model;
retrieval quality (and therefore the system's ability to find relevant chunks) is bounded
by this model's representational capacity.
• Heuristic table/figure extraction: table detection relies on simple multi-column text
heuristics (StructuredContentExtractor), and figure retrieval depends on whether the
PDF embeds discrete raster images on the relevant page (rather than vector graphics or
rendered diagrams that PdfPig may not extract as separate images). Both approaches
can miss content or produce false positives.
• Latency from multiple Ollama calls: each chat query can involve several sequential
calls to the local LLM (tool selection, answer generation, verification), plus one
embedding call per search tool invocation. On CPU-only hardware, this can result in
noticeable response latency, particularly for the Answer Agent step (up to 512 tokens
generated).
• Configuration security: as noted in Section 5.4, the repository as provided contained a
live Qdrant Cloud API key in [Link], which is not appropriate for source
control and has been redacted in this report.

9.3 Comparison to Objectives

Objective Status

PDF upload via web Met — implemented via DocumentsController and the Sidebar
interface component.

Dense vector embedding Met — PdfChunkingService + OllamaEmbeddingService +

pipeline QdrantService.

Natural-language chat
Met — ChatController + chat UI components.
interface

Retrieval of text, Met — DocumentFunctionCallingService with three function-

images, and tables calling tools.

26
Objective Status

Context-constrained, Met — Answer Agent system prompt enforces citations and context-
cited answer generation only answers.

Met — Orchestrator, Router, Retrieval, Answer, and Critic Agents

Multi-agent pipeline
implemented.

Anti-hallucination guard Met — HallucinationGuard with pre- and post-answer checks.

UI display of sources, Met — MessageBubble/SourcesPanel render sources, function-call

traces, and confidence trace, agent-step trace, and confidence/groundedness badges.

All stated objectives appear to be met by the current implementation, subject to the limitations
described above and to empirical evaluation results (Section 8.3) that should be filled in after
testing.

10. Conclusion and Future Work

10.1 Summary of Contributions

This project delivers a working, locally deployable PDF RAG chatbot that combines dense
vector retrieval (Ollama embeddings + Qdrant ANN search), multi-modal content retrieval
(text chunks, extracted images, heuristically detected table blocks) via an LLM-driven
function-calling router, a multi-agent answer-generation and verification pipeline, and a
layered anti-hallucination guard that enforces evidence-based, cited, and confidence-scored
answers — with a React-based frontend that exposes this entire pipeline (sources, function-call
trace, agent-step trace, and groundedness badges) to the end user for transparency.

10.2 Future Work

• Hybrid retrieval: combine dense vector search with lexical (BM25) scoring to improve
recall for queries containing exact terms, identifiers, or rare phrases.
• Reranking: add a cross-encoder reranking stage over the top-N dense retrieval results
to improve precision before context is passed to the Answer Agent.

27
 • Larger LLM: evaluate replacing tinyllama with a larger locally hosted model (where
hardware permits) to improve answer fluency and the reliability of JSON-formatted
Router/Critic outputs, reducing dependence on heuristic fallbacks.
• Improved PDF parsing: incorporate layout-analysis or vision-based extraction for
figures and tables (e.g., rendering pages to images and using a vision-capable model)
to handle vector-graphic diagrams and complex table layouts not captured by the
current heuristics.
• Automated evaluation benchmark: build a repeatable test harness that runs a fixed
set of questions against a fixed document and automatically computes the metrics
defined in Section 8.3 (retrieval hit rate, grounded answer rate, hallucination refusal
rate, etc.), enabling regression testing as the pipeline evolves.

11. References
1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł.,
& Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information
Processing Systems (NeurIPS).
2. Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Generation for
Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems
(NeurIPS).
3. Qdrant Documentation. Qdrant Vector Database — Concepts and Collections.
Retrieved from [Link]
4. Ollama Documentation. Ollama — Run Large Language Models Locally. Retrieved
from [Link]
5. UglyToad. PdfPig — .NET Library for Reading PDF Files. Retrieved from
[Link]

28