LLM Inference - Architect Notes
LLM Inference Complete Notes
Complete Step-by-Step Notes for an AI System Architect
Prepared for Betopia AI Ecosystem Planning and
Learning
LLM Inference - Architect Notes
Table of Contents
1. Introduction
2. What is LLM inference?
3. How LLM inference works
4. Context capture during inference
5. Performance bottlenecks
6. Optimization techniques
7. Infrastructure and hardware
8. Measuring inference performance
9. Benchmarks
10. End-to-end LLM inference workflows
11. Real-world use cases
12. Future of LLM inference
13. Conclusion
14. Author and source resources
15. AI System Architect checklist
LLM Inference - Architect Notes
1. Executive Summary
LLM inference is the production process of using a trained large language model to generate
responses for new prompts. It is not training; the model weights are not updated. Inference is a
forward-only process where the model reads input tokens, builds context, and generates output
tokens step by step.
For an AI System Architect, inference is not only a model problem. It is a full system design
problem involving GPU memory, token throughput, request latency, batching, KV cache, model
routing, security guardrails, observability, cost, and user experience.
Production requirement Architect meaning
Users should see responses quickly in chatbots, copilots and
Low latency
interactive tools.
High throughput The platform must serve many users or requests at the same time.
The system must support large models, long contexts and higher
Efficient GPU memory usage
concurrency without exhausting memory.
Real-time responsiveness Streaming and chat experiences should feel smooth and consistent.
Architect view: LLM inference is not only a model problem. It is a system design problem
that connects model size, GPU memory, batching, KV cache, latency, throughput, routing,
monitoring and cost.
2. What is LLM Inference?
LLM inference means running a pre-trained model to predict and generate output tokens for a new
input prompt. The model receives tokens, processes them through transformer layers, predicts the
next token, appends it to context, and repeats until the response is complete.
Inference = Tokenize input -> Prefill context -> Decode output tokens -> Stream final
response
Fig: High-level workflow added to make the original inference stages easier to understand.
Training vs Inference
Training is different from inference. Training updates model weights through backpropagation.
Inference is read-only and forward-only. This is why inference systems focus on serving speed,
memory efficiency, concurrency and cost control.
Area Training Inference
LLM Inference - Architect Notes
Purpose Teach the model by updating weights Use learned weights to generate responses
Yes, backpropagation updates
Weight update No, forward-only read process
parameters
Compute pattern Very compute-heavy over large datasets Continuous real-time serving workload
Main objective Accuracy and learning Latency, throughput, reliability and cost
Typical output Updated model checkpoint Generated response tokens
Whenever a user interacts with a chatbot, summarization tool, code assistant or AI-powered
search tool, the application performs LLM inference to generate the answer.
Efficiency impact area Why it matters
End-to-end latency Controls total user waiting time.
Controls model size, context length and concurrency
GPU memory consumption
capacity.
Controls how many requests the platform can serve per
Request throughput
second.
Infrastructure cost Controls monthly cloud/GPU/server spending.
Overall scalability Controls whether the system can grow with usage.
3. How Does LLM Inference Work?
A production inference pipeline is designed for speed, accuracy and scalability. The source PDF
explains three core stages: tokenization, prefill and decoding. Architecturally, these stages sit
inside a larger API and serving system.
Prefill and decoding workflow with KV cache.
3.1 Tokenization
The raw user input text is converted into tokens. Tokens are numerical representations that the
model can process. Depending on the tokenizer, text may be split into words, subwords,
characters or token pieces.
Example Explanation
Raw text "Machine learning is useful"
Tokens [Machine] [learning] [is] [useful]
Token IDs Numerical IDs such as 3901, 6231, ...
LLM Inference - Architect Notes
More tokens increase cost, context usage, prefill work and KV
Why important
cache size
Total input tokens = system prompt tokens + user prompt tokens + retrieved context tokens +
chat history tokens
For architecture planning, token count is the unit that connects user behavior to GPU memory,
latency and cost. Long prompts and long retrieved context increase prefill latency and KV cache
memory.
3.2 Prefill Phase
The prefill phase processes the entire input sequence in parallel. It builds the initial internal model
state and initializes the KV cache. Prefill is usually compute-intensive, especially when prompts are
long or when RAG injects many context chunks.
Prefill workload roughly increases with input length and model size
The prefill phase also initializes the KV cache. KV cache stores attention information for tokens
that have already been processed. Long prompts and large models make prefill more compute-
intensive and memory-intensive.
Prefill factor Effect
Large model More layers and parameters increase compute
Long input context More tokens must be processed
RAG context More retrieved chunks increase input tokens
Can improve GPU utilization but may increase waiting
Batching
time
Architect note: Long prompts increase prefill cost. For enterprise systems, prompt
templates, RAG context size and conversation history must be controlled to avoid slow first-
token latency.
3.3 Decoding Phase
After prefill, the model starts decoding. The decoding phase generates output tokens one by one
in an autoregressive manner. Each new token depends on previous input and output tokens. This
is why generation is sequential and why Time Per Output Token (TPOT) is an important metric.
Decode loop = predict next token -> append token -> update KV cache -> stream token ->
repeat
KV cache is critical during decoding. Without KV cache, the model would repeatedly recompute
attention states for older tokens. With KV cache, previous key-value tensors are reused, which
improves generation speed but consumes GPU memory.
LLM Inference - Architect Notes
Parallel or
Stage Primary pressure Architect focus
sequential
Fast tokenizer and efficient
Tokenization Pre-processing CPU/application overhead
request handling
Reduce prompt length,
Parallel over input Compute and memory
Prefill optimize attention and
tokens bandwidth
batching
Sequential token by Optimize TPOT, batching and
Decoding Latency and KV-cache growth
token KV-cache memory
3.3 KV Cache
KV cache stores previously computed key and value tensors from attention. Without KV cache, the
model would repeatedly recompute attention for earlier tokens during every decoding step. KV
cache improves speed but consumes GPU memory.
KV cache memory increases with: users x context length x layers x KV heads x head dimension
x precision
Figure: Example KV cache growth with longer context
Variable Meaning Architecture impact
Concurrent users Active sequences being served More users require more KV memory
Context length Prompt + history + generated tokens Long context can exhaust VRAM
Precision Bytes per KV value FP8 KV uses less memory than FP16/BF16
Layers and heads Model architecture parameters Larger architectures increase KV memory
4. How Does an LLM Capture Context During Inference?
LLMs use transformer architecture and self-attention to capture relationships between tokens. The
attention mechanism helps the model decide how strongly each token should focus on every other
token in the sequence.
LLM Inference - Architect Notes
Context capture through query, key and value tensors.
During inference, query, key and value tensors are created. Tensors are multi-dimensional
numerical data structures used by neural networks. In attention, these tensors represent different
views of the same token sequence and help calculate token relationships.
Component Meaning in inference
Query (Q) Represents what a token is looking for in context.
Key (K) Represents information other tokens can be matched against.
Represents the content that can be passed forward after attention
Value (V)
scoring.
Stores key and value tensors from previous tokens to avoid repeated
KV cache
computation.
Tensor operations Large matrix and tensor computations that dominate GPU workload.
Optimized kernels Low-level GPU operations that reduce execution time and cost.
Optimized implementations such as FlashAttention reduce memory access overhead during
attention computation. This makes attention faster and more memory-efficient, especially on
NVIDIA GPUs and other accelerators.
4. Performance Metrics
Inference performance must be measured with engineering metrics. A model can have high token
speed but still feel slow if TTFT is poor. A system can fit in memory but still fail under concurrency
if throughput is insufficient.
LLM Inference - Architect Notes
5. Performance Bottlenecks of LLM Inference
Several bottlenecks limit production inference performance. These bottlenecks directly affect
latency, throughput and scalable deployment.
Fig: Major performance bottlenecks that an AI system architect must control.
Bottleneck Cause Symptom Possible fix
Model weights + KV OOM, low max Quantization, shorter
GPU memory limit
cache exceed VRAM concurrency context, more GPUs
More parameters require High latency, low Smaller model,
Large model size
more compute throughput distillation, routing
Long prompt/history/RAG Slow prefill, high Context compression,
Long sequence length
context memory chunk ranking
GPU underused or queue Low throughput or high Dynamic batching, tune
Poor batching
too long waiting time batch tokens
Unoptimized tensor FlashAttention,
Slow kernels High TPOT
operations optimized runtime
External model or Good model TPS but high Connection pooling,
Network/API gateway
gateway overhead total latency regional placement
Important relationship: Model size, context length and concurrency increase GPU memory
LLM Inference - Architect Notes
pressure. Batch size, kernel efficiency and hardware utilization control throughput. Latency
is affected by both prefill and decoding stages.
6. Optimization Techniques for LLM Inference
Inference optimization improves speed, memory efficiency, concurrency and cost. Each
optimization has trade-offs between accuracy, speed, memory usage and infrastructure cost.
Figure 7: Main optimization techniques and their primary benefit
Table: Optimization technique map with the main trade-off.
Technique What it does Trade-off
Quantization Stores weights/KV in lower precision such Possible quality loss if aggressive
LLM Inference - Architect Notes
as FP8 or INT8
Improves attention memory access and Requires compatible
Flash Attention
speed hardware/software
Combines requests to improve GPU
Dynamic batching May increase waiting time
utilization
Tensor parallelism Splits model across multiple GPUs Adds communication overhead
Uses memory more efficiently during
KV cache optimization Requires runtime tuning
decode
Uses smaller model trained from larger
Distillation May lose some capability
model behavior
Small draft model proposes tokens, large
Speculative decoding More complex serving setup
model verifies
Architect decision rule: Do not apply optimization blindly. Benchmark before and after
each change using TTFT, TPOT, throughput, GPU memory, GPU utilization and output quality.
7. Infrastructure and Hardware for LLM Inference
Modern inference depends on specialized hardware and optimized software working together. The
infrastructure must meet latency, throughput and scalability requirements.
Fig: Hardware and serving software stack for production inference.
Hardware/software element Role in inference
Execute large-scale tensor and matrix operations with massive
High-performance accelerators
parallelism.
High-bandwidth accelerator Moves large tensors and KV cache data between memory and
memory compute units efficiently.
Reduce memory overhead and improve hardware utilization,
Optimized compute kernels
latency and throughput.
vLLM, Hugging Face and similar frameworks optimize API serving,
LLM serving frameworks
batching, memory management and execution paths.
Efficient serving layer Handles concurrent user requests reliably in production.
Serving frameworks such as vLLM and Hugging Face optimize model execution by managing
scheduling, batching, KV cache and API workflows. These optimizations help deliver reliable, real-
time and cost-effective generative AI services.
Infrastructure layer Key design question
GPU / accelerator Can it fit the model and deliver target TPS?
VRAM / HBM Can it hold weights, KV cache and overhead?
Can GPUs communicate efficiently for tensor
Interconnect
parallelism?
LLM Inference - Architect Notes
Does it support batching, KV cache and optimized
Serving runtime
kernels?
API layer Can it enforce auth, rate limits and routing?
Can we see latency, TPS, GPU memory, errors and
Observability
cost?
8. Measuring Inference Performance
AI system architects must track performance with clear metrics. Metrics show whether the system
is ready for production and where optimization is needed.
Core inference metrics and benchmark relationship.
Metric Meaning Architect use
Measure user experience and SLA
Latency Time taken to generate a response.
compliance.
Tokens per second or requests per Measure serving capacity and scaling
Throughput
second. needs.
GPU usage Hardware efficiency during inference. Detect underutilization or overload.
Cost per Infrastructure cost allocated to each Compare local deployment vs commercial
request request. API.
Useful production metrics: Track TTFT, TPOT/ITL, p50/p90/p95/p99 latency, output
tokens/sec, requests/sec, failure rate, GPU memory used, GPU utilization, queue time and
cost per request.
9. GPU Capacity and Memory Calculation
Capacity planning starts with memory feasibility. This answers: can the model, fixed overhead and
KV cache fit in available GPU VRAM? It does not prove speed; speed must be benchmarked.
LLM Inference - Architect Notes
Figure 8: Capacity planning workflow
9.1 Step 1: Calculate usable GPU memory
Usable VRAM = Number of GPUs x VRAM per GPU x GPU memory utilization
Example: If a server has 8 GPUs and each GPU has 96 GB VRAM, total physical VRAM is 8 x 96 =
768 GB. If planning utilization is 90%, usable planning VRAM is 768 x 0.90 = 691.2 GB.
9.2 Step 2: Calculate model weight memory
Model weight memory = Number of parameters x bytes per parameter
Precision Bytes per parameter Example for 38B model
FP32 4 bytes 38B x 4 = 152 GB
BF16 / FP16 2 bytes 38B x 2 = 76 GB
FP8 / INT8 1 byte 38B x 1 = 38 GB
INT4 0.5 byte 38B x 0.5 = 19 GB
Precision is used because every model parameter must be stored in GPU memory. Lower precision
reduces memory and often improves speed, but it may reduce quality if not validated.
9.3 Step 3: Add fixed overhead
Fixed cost = Model weights + runtime overhead + CUDA/NCCL overhead + serving metadata
This memory is used before request-specific KV cache is considered. It covers the model and
serving system baseline.
9.4 Step 4: Estimate KV cache memory
KV cache bytes = 2 x users x sequence length x layers x KV heads x head dimension x bytes
per KV element
The factor 2 exists because attention stores both Key and Value tensors. KV cache is often the
reason long-context, high-concurrency serving becomes expensive.
10. Latency, TPS and Cost Calculation
10.1 Throughput demand
Required output TPS = Concurrent users x target output TPS per user
Example: 50 concurrent users x 40 tokens/sec/user = 2,000 output tokens/sec. With a 20% safety
margin, target system throughput becomes 2,000 x 1.20 = 2,400 output tokens/sec.
LLM Inference - Architect Notes
10.2 TPOT calculation
TPOT = 1000 ms / TPS
If a user target is 40 tokens/sec, TPOT = 1000 / 40 = 25 ms/token. Lower TPOT means smoother
streaming.
10.3 Cost per request
Cost per request = infrastructure cost per request + API cost +
storage/bandwidth/monitoring cost
Local monthly GPU cost = GPU hourly price x number of GPUs x 24 x 30
API cost = (input tokens / 1,000,000 x input price) + (output tokens / 1,000,000 x output
price)
For Betopia AI Ecosystem planning, compare local GPU cost against commercial API cost. Local
models are often cheaper at high steady volume, while commercial models may be better for
variable or quality-sensitive traffic.
11. Production Deployment Architecture
A production architecture should route requests intelligently. Not every request needs the largest
model. Simple requests can use fast/cheap models; complex reasoning can use stronger models;
internal coding or batch workloads can use local models if latency is acceptable.
Request type Recommended route Reason
Simple chat / short Q&A Fast small model Low latency and low cost
Complex reasoning High-quality model Better quality and reasoning
Grounded answer from enterprise
Document Q&A RAG + selected model
knowledge
Code generation Coding model Specialized capability
High-risk prompt Guardrail + human review if needed Security and compliance
LLM Inference - Architect Notes
Figure 9: Secure production architecture with routing, cache, guardrails and monitoring
12. Benchmarking and Monitoring
Benchmarks compare workflows, batch sizes, optimization techniques and hardware setups.
Benchmarking helps identify bottlenecks, improve performance and optimize resource allocation
for real production use cases.
Benchmarking proves whether a deployment is actually ready. Memory calculation alone is not
enough. A model can fit in VRAM but fail due to poor batching, long TTFT, low throughput or high
failure rate.
Benchmark dimension What to test
Hardware setup GPU type, memory, number of GPUs, interconnect and CPU support.
Model setup Model size, precision, quantization level and context length.
vLLM/Hugging Face settings, batching, tensor parallelism and KV-cache
Serving setup
behavior.
Traffic setup Concurrency, prompt length, output length and request distribution.
Production SLO TTFT, throughput, p95 latency, failure rate and cost.
Figure 10: Benchmarking workflow
13. Step-by-Step Deployment Roadmap
Step Action Output
1 Define product use cases Chat, RAG, coding, summarization, agents
2 Define SLO targets TTFT, p95 latency, TPS/user, failure rate
3 Choose candidate models Fast model, quality model, local model
4 Estimate memory Weights + overhead + KV cache
5 Deploy test server vLLM or equivalent runtime
6 Run synthetic benchmarks Measure TTFT, TPOT, TPS, failure rate
7 Run workload-specific tests Real prompts from Betopia users
8 Tune runtime Batching, context length, precision, KV cache
9 Add gateway and guardrails Security, rate limit, policy
10 Release gradually Internal beta -> wider rollout
11 Monitor continuously Dashboard, alerts, cost reports
12 Re-benchmark after changes Model update, infra change, traffic growth
LLM Inference - Architect Notes
14. End-to-End LLM Inference Workflow
The source PDF describes a typical end-to-end workflow for LLM inference. The architect version
below keeps the same sequence and adds the production perspective.
End-to-end workflow from data preparation to monitoring.
1. Prepare curated datasets.
2. Fine-tune or distill the base model when needed.
3. Export optimized model weights.
4. Deploy the model through an API.
5. Serve real-time inference using batching and KV cache.
6. Monitor performance using latency, throughput, GPU and cost metrics.
Most production systems use Python for orchestration, monitoring and LLM serving integration.
The serving system usually sits behind an API gateway and includes logging, routing, security
controls and observability.
How architect-level capacity planning connects model, precision, context, concurrency and benchmark evidence.
LLM Inference - Architect Notes
15. The Future of LLM Inference
As generative AI adoption grows, organizations must improve inference efficiency, reduce latency,
improve throughput, reduce infrastructure cost and scale workloads efficiently.
Emerging trend Meaning
Maximizes hardware usage while trying to
Dynamic batching
preserve low latency.
Reduces redundant computation and supports
Memory-aware KV cache management
long-context or high-concurrency workloads.
Supports real-time and enterprise-grade
Scalable API deployments
applications.
Reduces memory overhead and improves
Efficient compute kernels
execution speed.
Improves long-context and high-concurrency
Smarter KV cache management
serving.
Lowers memory and compute requirements with
Advanced quantization
minimal quality loss.
Hardware designed for transformer-based
Specialized AI accelerators
workloads.
Enhanced scheduling, batching and decoding
Improves inference optimization at serving time.
algorithms
With the right combination of hardware, software and optimization strategy, LLMs can support
real-time, scalable and cost-effective AI solutions.
16. Conclusion
LLM inference is the engine that turns trained language models into practical user-facing AI
systems. A production platform must address bottlenecks, apply optimization techniques and use
high-performance GPU infrastructure.
Target outcome Architect interpretation
Low latency Users see the first and final response quickly.
High throughput The platform serves many requests or tokens per second.
Scalable deployments The system can grow with traffic and workload needs.
The system delivers value without uncontrolled infrastructure
Cost-efficient operations
spending.
Mastering LLM inference optimization is an important skill for AI engineers, platform teams and
enterprise architects.
17. AI System Architect Checklist
Use this checklist when designing or reviewing an LLM inference system.
Area Questions to answer
Model What is the model size, precision, context window and quality requirement?
Do model weights, KV cache, activations and runtime overhead fit in GPU
Memory
memory?
Latency What are TTFT, TPOT and end-to-end p95 latency targets?
Throughput How many tokens/sec and requests/sec are required at peak concurrency?
Batching Does batching improve throughput without hurting user experience?
KV cache Is cache memory controlled for long context and high concurrency?
Which techniques are enabled: quantization, FlashAttention, tensor parallelism,
Optimization
distillation?
Infrastructure Are GPUs, memory bandwidth, interconnect and serving framework suitable?
Benchmarking Have benchmark results validated the target workload?
Cost What is cost per request and monthly infrastructure cost?
Monitoring Are latency, throughput, GPU usage, memory, errors and cost tracked
LLM Inference - Architect Notes
continuously?
Source reference: Created from the uploaded PDF titled "What is LLM inference?". The
document follows the source order and preserves all technical points while adding architect-
oriented figures and workflows.