Technic l By S b ri B l ji
GOOOOOD MORNING!
Think Big, Execute Bigger!
a
a
a
a
a
SABARI BALAJI
Enterprise & Cloud Solutions Architect |
19+ Ye rs Experience
@SabariBalaji
learnings@[Link]
a
Pro le Insight
E
nterprise & Cloud Solutions Architect with 19 years of industry experience,
currently working at Tech Mahindra. I design and modernize real-world
enterprise applications using Java & Spring ecosystem, I am helping systems
move from legacy setups to cloud-native platforms. My work also includes integrating
Generative AI, building agent-based AI systems, and applying AI-driven development
practices (AI-DLC) for App Modernization.
fi
Technic l By @S b riB l ji
GenAI: Integrating LLM
& RAG in Practice
Technical Workshop
a
a
a
a
a
Tinder AIDLC
?
Ghosting Qu ntiz tion
G.O.A.T
Context Embeddings
Sn ck
Chunking
Cringey
Lock In FAISS
C.E.O Feedb ck Loop
a
a
a
a
Let’s st rt with Why?
Why most GenAI Apps F il in Production?
a
a
“
Before we understand about LLM Models,
Prompts, or tools. We need to x on how
we think about Developing a GenAI
System.
fi
T
he biggest misconception in developing
GenAI system tod y is people h ve st rted to
think th t LLM is lone contributing for
developing GenAI System.
e ANSWER: LLM is not the only component
required for developing an Enterprise graded
GenAI System
Th
a
a
a
a
a
An LLM is just one component in developing re l
GenAI Engineering system. Let’s Explore e ch &
every component in det il.
GenAI = Intelligence + Control + Govern nce + Oper tions
LLMs sit in the middle, not t the top.
a
a
a
a
a
a
a
User Inter ction L yer (Entry Point)
User Inter ction
STAGE-1
Ch tbots WebApp API Access
is is where the system is experienced by users chatbots, web applications, internal tools, or API
consumers. A user may ask a question like “Summarize our Q3 nancial risks” via a web UI or a
Slack bot. is layer does no intelligence by itself; it only captures intent, authentication
context, and request metadata (user ID, tenant, role). ink of this as the front door, not the
brain
Th
a
a
Th
a
a
Th
fi
D t Sources (Enterprise Knowledge)
D t Sources
STAGE-2
Documents D t b ses Websites
ese are the raw knowledge assets the GenAI system reasons over. Examples include PDFs in
SharePoint, rows in PostgreSQL, Con uence pages, Jira tickets, CRM records, or even public
websites. e key architectural point: LLMs do not train on this data directly. is data
remains in your control and is accessed dynamically via retrieval mechanisms.
Th
a
a
a
a
a
a
a
Th
fl
Th
D t Ingestion & Processing (Knowledge Prep r tion)
D t Ingestion & Processing
STAGE-3
D t Cr wling ETL Text P rsing D t Cle nsing
Raw data is unusable for LLMs. is layer handles crawling, ETL, text extraction, cleaning,
chunking, and normalization. For example, a 200-page policy document is split into 500–1,000
token chunks, metadata is added (document type, department, version), and sensitive elds may
be masked. NOTE: is is a Batch or Near-Real-Time Pipeline, not part of the request path.
a
a
a
a
a
a
a
a
a
a
a
Th
Th
a
a
fi
Vector D t b se & Embedding Index (Sem ntic Memory)
Vector D t b se
[0.0134, 0.8723, -0.4532, .., 0.5892] STAGE-4
Vector
Each processed chunk is converted into a numerical
Output embedding using an embedding model. ese vectors are
Embedding
Embedding stored in a vector database (Pinecone, Weaviate,
Model OpenSearch, FAISS). Unlike keyword search, this enables
semantic similarity. For example, a query like “employee
exit rules” can retrieve content titled “Termination &
O boarding Policy” even if the words don’t match exactly.
Embeddings = Numeric l Represent tion
Source Subject
of Text, Im ge, Audio, etc…
You're a Genie, Harry!
ff
a
a
a
a
a
a
a
a
Th
a
a
Extern l APIs & Tools (Augmented Intelligence)
RAG STAGE-5
Some questions require live or computed data, not static knowledge. is layer integrates
external APIs such as weather services, nancial APIs, ERP systems, calculators, or internal
microservices. Example: “What is today’s inventory shortfall?” triggers a real-time call to an
inventory API. is is what makes GenAI actionable, not just conversational.
a
Th
fi
Th
LLM Integr tion L yer (Model Control Pl ne)
LLM Integr tion
STAGE-6
Prompt
Pre-Tr ined LLM Fine-Tuning Model Engineering
is is where you integrate foundation models (OpenAI, Anthropic, open-source LLMs). It
includes prompt templates, system instructions, ne-tuning (optional), guardrails, and
token controls. For example, prompts enforce “Answer only from retrieved context. If unsure, say I
don’t know.” is layer is critical for controllability, safety, & consistency.
Th
a
a
Th
a
a
fi
a
RAG Pipeline (Core Intelligence Loop)
RAG Pipeline
Knowledge Retrieval
Query Processing Retrieval
Semantic Document
STAGE-7
Search Retrieval
Vector Storage
Embedding Index
K Nearest Neighbor Search Context Augmentation
is is the heart of the system. When a user query arrives, it is embedded and matched against the
vector database. e top-K relevant chunks are retrieved and injected into the prompt as context.
is prevents hallucination and ensures answers are grounded in enterprise data. Example:
answering “What is our leave carry-forward limit?” using the latest HR policy, not generic internet
knowledge.
Th
Th
Th
Orchestr tion L yer (System Br instem)
Orchestr tion L yer
STAGE-8
Work ow Monitoring &
API Coordin tion
M n gement Logging
is layer coordinates everything query routing, tool invocation, retries, fallbacks, parallel calls,
and state management. Frameworks like LangChain, Semantic Kernel, or custom orchestrators
live here. Example: the system may rst retrieve documents, then call an API, then summarize
both before sending to the LLM. Without orchestration, GenAI systems collapse under
complexity.
Th
a
a
fl
a
a
a
a
a
fi
a
Gener tive AI Model (Re soning & Synthesis)
Gener tive AI Model
STAGE-9
Text Code
Gener tion Im ge Gener tion Gener tion
Here the LLM performs reasoning, summarization, code generation, or multimodal outputs.
Importantly, the model is not trusted alone it reasons over supplied context. Example outputs
include executive summaries, SQL queries, test cases, or architectural explanations. e model
synthesizes, not invents.
a
a
a
a
a
a
a
Th
Knowledge B se Feedb ck Loop (Le rning Without Tr ining)
Feedb ck Loop
Hum n in the Loop [Feedb ck]
STAGE-10
User feedback, corrections, and new documents ow back into the knowledge base. is improves
retrieval quality over time without retraining the LLM. Example: if users consistently reject an
answer, the underlying document chunking or metadata can be improved. is is continuous
knowledge evolution, not model retraining.
a
a
a
a
a
fl
a
Th
Th
a
Ev lu tion, Monitoring & Security (Production Re diness)
Ev lu tion, Monitoring & Security
Perform nce A/B Testing Feedb ck Security
Metrics Loop Compli nce
STAGE-11
is layer measures accuracy, latency, cost, hallucination rate, and user satisfaction. It includes A/B
testing of prompts, audit logs, PII controls, role-based access, and compliance checks. In
enterprises, is layer determines whether GenAI is allowed in production.
Th
a
a
a
a
a
a
a
Th
a
Deployment & Sc l bility (Enterprise Sc le)
Deployment & Sc l bility
STAGE-12
Cloud On-Prem
Auto Sc ling
Deployment Deployment
Finally, the system runs on cloud, on-prem, or hybrid infrastructure with autoscaling, isolation,
and cost controls. Stateless services scale horizontally; vector DBs scale independently; models
may be hosted or consumed via API. is ensures the system can handle 10 users or 10 million
users with the same architecture.
a
a
a
a
a
Th
a
LLM Twin
D t Collection Pipeline Fe ture Pipeline
Stre ming Ingestion Pipeline
Medium Medium ETL No SQL DB
ARTICLES
Substack Substack ETL
ARTICLES
CDC
Linkedin Linkedin ETL
QUEUE POSTS
Github Github ETL
CODE
Tr ining Pipeline Met d t
& Text
ARTICLES
D t to Qu dr nt Vector
LLM Fine- Met d t DB ARTICLES
tuning
Prompt
& Text Retriev l
L yers
Clients Retriev l POSTS
Experiment Clients
CODE
Vector DB
Tr cker
LLM
C ndid te
Model Registry Use “Vector
Approved DB Retriev l
LLM Clients”
Ev lu te
LLM Inference Pipeline
C ndid te Write post bout LLM?
Lo d &
REST API
Qu ntize LLM
Prompt Monitoring
Qu dr nt
Twin LLM
Query to Met d t Retriev l
Prompt & Text
Clients Gener ted Output
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
OpenAI CFO Sarah Friar: e business case
for AI is simple: more compute, more
revenue
OpenAI scaled from 200 megawatts to 2
gigawatts, and ARR climbed from $2 billion
to over $20 billion, yet we remain
absolutely constrained on compute. " e
signal from everywhere is real... we are in a
paradigm shift”.
Th
Th
Boston Dynamics has
just released a new
video of its upgraded
next-generation
humanoid robot
called Atlas
Th nk You!
a