ML INFERENCE ENGINEERING

Master Study Notes

5-Week Technical Deep Dive
Tokenization → Quantization → Inference Optimization → Fine-Tuning → Production

Covers: BPE WordPiece KV Cache PagedAttention vLLM / TGI Quantization LoRA / QLoRA RLHF / DPO

WEEK 1
Foundations of LLM Inference
Attention · Tokenization · KV Cache · Speculative Decoding · Quantization

1.1 Attention Mechanism (Transformer Core)

Attention is the backbone of every modern LLM. It lets every token in a sequence look at every other token and
decide how much to 'pay attention' to it.

Scaled Dot-Product Attention — Formula

Attention(Q, K, V) = softmax( QKᵀ / √dₖ ) · V

Q (Query) What the current token is asking / looking for

K (Key) What each token advertises / represents

V (Value) The actual content each token contributes

√dₖ Scaling factor — prevents softmax saturation in high-dim spaces

softmax Converts raw scores to a probability distribution (sum=1)

Multi-Head Attention (MHA)

Instead of one attention head, run h parallel attention heads, each with their own Wᵢᴼ, Wᵢᴷ, Wᵢᵛ projections, then
concatenate and project back.
Why: Each head learns to attend to different relationship types (syntax, coreference, semantics). More
representation power with same parameter budget.
MHA Variants — Inference Cost Reducers
Full separate KV heads per query head. Most expressive, highest memory cost
MHA (standard)
at inference.

All query heads share ONE key/value head. ~8× KV cache reduction. Used in
MQA (Multi-Query)
Falcon, early Gemini.

GQA (Grouped- Query heads grouped; each group shares KV heads. Balance between MHA
Query) quality and MQA speed. Used in LLaMA-3, Mistral.

Exact attention, tiled SRAM computation. No approximation. Avoids O(N²)

FlashAttention
memory by chunking. Default in vLLM.

Improved parallelism on sequence length dimension + better work partitioning.

FlashAttention-2
~2× faster than FA-1.

1.2 Tokenization Algorithms

Tokenization converts raw text → integer IDs. The choice of algorithm controls vocab size, out-of-vocabulary
handling, and language coverage.

Byte Pair Encoding (BPE)

Start with character-level vocab. Iteratively merge the most-frequent adjacent pair. Each merge adds one new
token.

BPE Merge Example

Corpus: 'lo lo lo low low lower' Step 1: 'l' 'o' → merge into 'lo' (freq 5) Step 2: 'lo' 'w' → merge into 'low' (freq
2) Result: vocab grows token by token from bottom up

BPE Strengths BPE Weaknesses

Deterministic — same corpus → same vocab Greedy — not globally optimal merges

Handles new words via sub-word fallback Splits can differ language to language

Used by GPT-2, GPT-4, Llama, RoBERTa No probabilistic alternatives at inference

WordPiece
Used by: BERT, DistilBERT, mBERT, Electra
Same iterative merging intuition as BPE but selects merges that maximize the likelihood of the corpus under a
language model, not just raw frequency.
score(A,B) = freq(AB) / ( freq(A) × freq(B) )
Prefer merges where the pair appears far more often than chance → more linguistically coherent subwords. Prefix
'##' marks continuation tokens.

SentencePiece / Unigram LM
Used by: LLaMA-1/2, Mistral, T5, ALBERT, Gemma
Trains directly from raw text without pre-tokenization (no whitespace splitting). Uses a Unigram Language Model:
starts with large vocab, prunes tokens that minimally hurt log-likelihood. Whitespace is treated as a character (▁).
 Unigram LM Pros Unigram LM Cons

Works on any language incl. Japanese/Chinese Slower training than BPE

Probabilistic — can sample multiple tokenizations Harder to control vocab size precisely

Language-agnostic (space-free) baseline Requires dedicated SentencePiece library

Tokenization — Key Metrics to Memorize

GPT-2: 50K. LLaMA-2: 32K. LLaMA-3: 128K. Larger vocab = fewer tokens per
Vocab Size
sentence but larger embedding matrix.

Avg tokens per word. Lower = more efficient. English ~1.3, rare languages can
Fertility
be 5+.

Long tokens take same time as short in LLMs (one forward pass). Critical for cost
Tokenization Bias
estimation.

<|bos|> <|eos|> <|pad|> <|unk|> — model-specific. Wrong special tokens = silent

Special Tokens
bugs.

1.3 KV Cache
During autoregressive decoding, every forward pass recomputes K and V for ALL previous tokens — wasteful. KV
cache stores past K and V tensors so they are computed once and reused.

KV Cache — Memory Formula

KV Cache Bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element
Example — LLaMA-2 7B (FP16): 32 layers × 32 KV heads × 128 dim × 2048 seq × 2 bytes ≈ 8 GB (This is
why long-context inference is expensive even for small models)

Process entire input prompt. All K/V computed and stored. This is compute-
Prefill phase
bound.

Generate one token at a time. Read cached K/V. This is memory-bandwidth-

Decode phase
bound.

TTFT Time To First Token — measures prefill latency. Dominated by FLOPs.

Time Per Output Token — measures decode latency. Dominated by memory

TPOT
bandwidth.

Model FLOPs Utilization — actual FLOPs / theoretical peak FLOPs. Target >50%
MFU
for efficiency.

Memory Bandwidth Utilization — actual BW used / peak BW. Decode is MBU-

MBU
bound.

1.4 PagedAttention
Standard KV cache pre-allocates contiguous memory for max_seq_len — massive waste when actual sequences
are short. PagedAttention (vLLM) borrows from OS virtual memory paging.
 Fixed-size contiguous chunk of GPU memory (e.g. 16 tokens × head_dim).
Block
Analogous to OS memory page.

Per-sequence map: logical token position → physical block ID. Sequences can
Block Table
be non-contiguous in physical memory.

For beam search / parallel sampling, blocks are shared until a branch writes to
Copy-on-Write
them — then copied.

External fragmentation eliminated. Internal fragmentation bounded by block_size

Fragmentation
- 1 tokens.

2–4× more requests served simultaneously vs contiguous KV cache. Enables far

Throughput gain
higher GPU utilization.

1.5 Memory Optimization Strategies

Activation Don't save intermediate activations during forward pass. Recompute during
Checkpointing backward. Trades compute for memory. Key for training.

Gradient Subset of checkpointing — only save at segment boundaries. Used in HF

Checkpointing Transformers via gradient_checkpointing_enable().

Mixed Precision Store weights in BF16 (2 bytes), master copy in FP32. ~2× memory saving vs
(BF16) pure FP32 training.

Move optimizer states or frozen layers to CPU RAM. Slower but enables larger
CPU Offloading
models on fewer GPUs. Used in DeepSpeed ZeRO-3.

Parallelize KV cache reads across sequence dimension during decode. Reduces

Flash Decoding
memory bandwidth pressure.

1.6 Speculative Decoding

Core insight: the large target model spends most time waiting (memory-bandwidth bottleneck). Use a small cheap
draft model to speculatively generate k tokens, then verify all k tokens in parallel with one target model forward
pass.

Small fast model (e.g. LLaMA 68M) generates k candidate tokens

Draft model
autoregressively.

Target model runs ONE forward pass over draft tokens. Accept tokens where
Verification
target agrees; reject remainder.

Probability that a draft token is accepted. Higher α = more speedup. Task/model-

Acceptance rate α
pair dependent.

If α=0.7 and draft generates k=4: ~1/(1-α^k) ≈ 2–3× decode speedup with zero
Theoretical gain
quality loss.

Adds multiple decoding heads to the target model itself. No separate draft model;
Medusa
each head predicts token+k.

Self-Speculative Use early exit layers of the same model as draft. Avoids KV cache duplication.
1.7 Draft Models
The draft model must share the same vocabulary as the target (identical tokenizer). Common pairs: LLaMA-3 70B
+ LLaMA-3 1B, Qwen-72B + Qwen-0.5B, Vicuna-13B + T5-small (suboptimal — different vocab).

Critical Constraint
Draft model must have IDENTICAL vocab/tokenizer as target model. Mismatched tokenization =
verification failure. This is why most speculative decoding pairs are same model family.

1.8 Quantization
Quantization reduces numerical precision of weights (and optionally activations) to save memory and accelerate
inference. The central challenge: minimize accuracy loss from the information compression.

Data Types Reference

FP32 32-bit float. 1 sign + 8 exp + 23 mantissa bits. Training standard. 4 bytes/param.

16-bit float. 1+5+10 bits. 2× memory saving. Inference default on most GPU
FP16
frameworks.

Brain Float 16. 1+8+7 bits. Same exponent range as FP32. Preferred for training
BF16
(no overflow).

8-bit integer. 4× vs FP32. Requires calibration. ~1% accuracy drop on most

INT8
models.

4-bit integer. 8× vs FP32. Aggressive. Requires advanced methods

INT4
(GPTQ/AWQ) to be viable.

8-bit float (E4M3 or E5M2). Better dynamic range than INT8. H100/H200 native
FP8
support.

Normal Float 4-bit. Used in QLoRA. Quantile-based levels optimal for normally
NF4
distributed weights.

Quantization Methods — Technical Comparison

PTQ — Post-Training Quantize after training. No re-training. Uses calibration dataset to find optimal
Quantization scale/zero-point.

QAT — Quantization- Simulate low-precision during training. Fake quantize forward pass, real-valued
Aware Training backward pass. Better quality but expensive.

PTQ. Solves per-layer weight quantization as a second-order optimization (uses

GPTQ
inverse Hessian). Column-by-column with error correction. INT4/INT3.

AWQ (Activation- Observes activation magnitudes per-channel. Protects salient weights (high
aware Weight Quant) activation = high importance) from aggressive quant. INT4. Faster than GPTQ.

Migrates quantization difficulty from activations to weights via per-channel

SmoothQuant
scaling. Activations are hard to quantize (outliers); weights are easy.

Mixed-precision per-layer. Can mix INT4 + INT8 + FP16. CPU-friendly. Used for
GGUF / [Link]
on-device inference.
 Sparse + Quantized. High-magnitude outlier weights kept at FP16 in sparse
SpQR
format; rest at INT4. Best quality at INT4.

Quantization — Calibration
PTQ methods need a calibration dataset (~128-512 samples) to compute optimal scaling factors. Quality depends
on: dataset representative-ness (use task-specific data), number of calibration steps, and whether activations are
also quantized.

INT8 vs FP8 vs INT4 — Rule of Thumb

INT8 (W8A8): Safe default. ~1% degradation. Use for production when memory matters. FP8 (H100 only):
Preferred over INT8 — better numeric range. Use bitsandbytes / TensorRT-LLM. INT4 (GPTQ/AWQ): Use
only when memory is critical. Need eval to confirm task-specific quality.

WEEK 2
Tokenization Depth + Inference Optimization +
Serving
vLLM · TGI · TensorRT-LLM · Model Parallelism · MoE · GPU Memory

2.1 Tokenization — Vocab Size vs Sequence Length Tradeoff

More tokens per sentence (longer sequences). Handles rare words via sub-word
Small Vocab (32K)
fallback. Lower embedding memory. E.g. LLaMA-2.

Fewer tokens per sentence (shorter). Better multilingual coverage. Larger

Large Vocab (128K+)
embedding matrix (128K × 4096 × 2B = 1GB). E.g. LLaMA-3.

Vocab Size vs It's a direct tradeoff: doubling vocab roughly halves sequence length for same
Sequence Length text. Shorter sequences = faster attention (O(N²)).

Why LLaMA-3 Jumped to 128K Vocab

LLaMA-2 at 32K vocab tokenizes non-English text very inefficiently (e.g., Chinese chars → many tokens).
LLaMA-3 upgraded to 128K for multilingual parity. Cost: ~1B extra parameters just for embedding table.
Trade-off: worth it for global models.

2.2 Inference Optimization — Batching Strategies

Fixed batch size. Pad shorter sequences to max_len. Wasteful — idle compute
Static Batching
for padded tokens. Simple to implement.

Batch requests arriving in a time window. Better GPU utilization than static. Still
Dynamic Batching
waits for full batch.

Immediately insert new requests into available GPU slots as old ones finish.
Continuous Batching
Operates at token granularity. Core of vLLM.
 Iteration-Level AKA continuous batching. Each decode step can have different batch members.
Scheduling No padding waste.

Split long prefill into fixed-size chunks. Mix prefill chunks with decode steps.
Chunked Prefill
Prevents prefill from starving decode requests.

2.3 Serving Frameworks — Technical Comparison

vLLM TGI (Text Generation Inference)

Python-native, PagedAttention core Rust frontend + Python backend

Best for research + production Python HuggingFace ecosystem native

Continuous batching, prefix caching Flash Attention, continuous batching

Supports tensor parallelism natively Supports tensor + pipeline parallelism

OpenAI-compatible API OpenAI-compatible API

Best for: high-throughput OpenAI-like API Best for: HF model deployment, quick prototyping

NVIDIA's C++ inference engine. Compiles model to CUDA kernels via TensorRT.
TensorRT-LLM Highest raw throughput on NVIDIA GPUs. FP8/INT8 native. Complex
deployment.

Microsoft. Scales model parallelism. Strong for multi-GPU deployment. Integrates

DeepSpeed-MII
with Azure.

Triton Inference NVIDIA's model serving platform. Supports multiple backends (TRT, ONNX,
Server PyTorch). Production SLA features.

Local inference focused. Easy setup. Uses GGUF/[Link] under the hood. Not
Ollama
production-grade.

2.4 Model Parallelism

Split individual weight matrices across GPUs column/row-wise. Each GPU holds
Tensor Parallelism
a shard. All-reduce after matmul. Reduces per-GPU memory. Used for single-
(TP)
node multi-GPU. Megatron-LM style.

Pipeline Parallelism Split model layers across GPUs sequentially. GPU 0 has layers 0–7, GPU 1 has
(PP) 8–15, etc. Micro-batching needed to hide pipeline bubbles. Good for multi-node.

Sequence Parallelism Split the sequence dimension across GPUs for LayerNorm and Dropout.
(SP) Complement to TP. Used in Megatron.

Each GPU holds full model copy, different batch shards. Gradient sync after
Data Parallelism (DP)
backward. Scales training throughput linearly.

Expert Parallelism For MoE — distribute expert FFNs across GPUs. Each GPU runs a subset of
(EP) experts. Combine with TP/DP.

ZeRO (Optimizer DeepSpeed. Stage 1: partition optimizer states. Stage 2: + gradients. Stage 3: +
State Partitioning) parameters. ZeRO-3 reduces per-GPU memory ~Nx for N GPUs.
2.5 GPU Memory Management
GPU memory is the #1 constraint in LLM inference engineering. Know this breakdown cold.

weights bytes = params × bytes_per_param (FP16=2, INT8=1, INT4=0.5)

Model Weights
LLaMA-3 70B FP16 = 70B × 2 = 140 GB

2 × layers × kv_heads × head_dim × seq_len × batch_size × dtype Dominates at

KV Cache
long context / large batch

Batch-size dependent. Forward pass intermediate tensors. ~seq_len × d_model

Activations
× layers × dtype

Fragmentation GPU allocator overhead. Typically reserve 10-15% headroom.

CUDA Kernels Static overhead ~1-2 GB for CUDA context, libraries (cuBLAS, cuDNN).

The 80 GB A100 Rule

LLaMA-3 70B in FP16 = 140 GB → needs 2× A100 80GB with tensor parallelism. With AWQ INT4: 70B ×
0.5 = 35 GB → fits 1× A100 80GB (with room for KV cache). This is why quantization matters operationally
— it changes your hardware tier.

2.6 KV Cache Optimization Techniques

Cache KV of shared system prompts. Amortize prefill cost across all requests
Prefix Caching
using the same system prompt. vLLM supports this natively.

Sliding Window Only attend to last W tokens instead of full context. O(N×W) memory vs O(N²).
Attention Used in Mistral. Loses long-range coherence.

Multi-Query Attention Drastically shrinks KV cache by sharing K/V heads (see Week 1). ~8× reduction.
(MQA)

KV Cache Store KV cache in INT8/INT4 instead of FP16. ~2-4× KV memory reduction.

Quantization Minor quality loss on long contexts.

H2O (Heavy-Hitter Evict KV cache entries for low-attention tokens. Keep 'heavy hitters' — tokens
Oracle) that consistently receive high attention scores.

2.7 Cost Optimization

Throughput vs Higher batch size = better GPU utilization = lower cost/token. But increases per-
Latency request latency. Tune based on SLA.

INT8: ~2× throughput gain, ~1% quality loss. INT4: ~4× but needs eval. FP8 on
Quantization ROI
H100: ~2× with almost no quality loss.

AWS/GCP allow ~60-70% cost reduction for batch/async workloads. Risk:

Spot Instances
preemption. Use for offline eval, batch generation.

Don't over-provision. A100 80GB vs A100 40GB: 2× cost. If model fits 40GB —
Right-sizing
use it. Monitor actual GPU memory utilization.

Train smaller model to mimic larger one. 1/5 the parameters, ~90% quality on
Distillation
specific tasks. One-time training cost.
2.8 Mixture of Experts (MoE)
MoE replaces the dense FFN layer with N expert FFNs. A router decides which k experts process each token
(sparse activation). Only ~k/N experts activate per token → same quality as dense at fraction of compute.

Learned linear layer + softmax over experts. Top-k routing: select top-k experts
Router / Gating
per token. k=2 is most common.

Critical: prevent all tokens routing to same expert. Auxiliary loss penalizes
Load Balancing
imbalance. Expert capacity buffers overflow.

Mixtral 8×7B: 47B total params but only 13B active per token (2 of 8 experts).
Active Parameters
Inference cost ~= 13B dense model.

Place different experts on different GPUs. All-to-all communication for token

Expert Parallelism
routing. Communication overhead is key challenge.

DeepSeekMoE: some experts are always active (shared), rest are sparse.
Shared Experts
Reduces specialization failure.

MoE Inference Challenge

Load balancing degrades at low batch sizes — with batch=1, routing is highly variable, experts may be
under-utilized. MoE shines at high throughput. At low latency/small batch, dense models can actually win.

WEEK 3
Retrieval-Augmented Generation (RAG)
Dense · Sparse · Hybrid · Embeddings · Re-ranking · Evaluation

3.1 Retrieval Strategies

Query → embedding vector → ANN search in vector DB. Semantic similarity.
Dense Retrieval
Handles paraphrase and synonymy. Requires embedding model.

BM25 / TF-IDF. Term matching. Fast, interpretable, no GPU needed. Fails on

Sparse Retrieval
synonym/paraphrase. Strong on exact keyword queries.

Combine dense + sparse scores. Reciprocal Rank Fusion (RRF) or weighted

Hybrid Retrieval
sum. Best of both worlds. Used in production by most RAG systems.

Hypothetical Document Embeddings. Generate a fake answer with LLM, embed

HyDE
it, retrieve similar real docs. Bridges query-document gap.

Recursive document summarization + clustering. Multi-level tree of summaries.

RAPTOR
Better for complex multi-hop queries.

LLM generates multiple paraphrase queries. Retrieve for each. Union results.
Query Expansion
Improves recall at cost of latency.
BM25 — The Sparse Standard
BM25(q, d) = Σ IDF(qᵢ) × [ f(qᵢ,d) × (k₁+1) / (f(qᵢ,d) + k₁ × (1 - b + b × |d|/avgdl)) ]

Inverse Document Frequency — rare terms get higher weight.

IDF(qᵢ)
log((N-n+0.5)/(n+0.5))

f(qᵢ,d) Term frequency in document d

Term frequency saturation parameter. Typical: 1.2–2.0. Higher = more TF

k₁
weighting.

b Document length normalization. 0=no norm, 1=full norm. Typical: 0.75.

3.2 Embedding Models + Optimization

text-embedding-ada- OpenAI. 1536-dim. General-purpose. No fine-tuning. Black box.
002

MS MSMARCO-trained. Excellent MTEB scores. Open source. Prefix prompts:

E5 / E5-mistral
'query:' / 'passage:'.

Best open-source embeddings (BGE-M3). Multi-lingual, multi-granularity.

BGE (BAAI)
Supports dense + sparse + ColBERT.

GTE Alibaba DAMO. Strong multilingual. Available in 7B variant (GTE-Qwen2).

Matryoshka Train embeddings that work at multiple dimensions (1024, 512, 256). Truncate to
Embeddings (MRL) smaller dim for speed/cost without full re-training.

Embedding Optimization
Fixed-size vs sentence vs semantic chunking. Chunk size is the most impactful
Chunking Strategy
RAG hyperparameter. 512 tokens typical.

20-50 token overlap between chunks prevents context splitting at chunk

Overlap
boundaries.

Embedding Store embeddings in INT8 instead of FP32. ~4× storage savings with ~1%
Quantization retrieval degradation.

Dimensionality PCA on embedding vectors. Compress 1536→256 dims. Reduces ANN index
Reduction size. Minor quality tradeoff.

3.3 Re-ranking
Two-stage retrieval: fast ANN recall (top-100), slow cross-encoder re-rank (top-5). Cross-encoders jointly encode
query+document → much more accurate but O(k×N) cost.

Embed query and doc independently. Score = cosine(q, d). Fast: ANN search.
Bi-encoder
Used for first-stage retrieval.

Concatenate [CLS] query [SEP] doc. BERT-style binary classification. Sees full
Cross-encoder
interaction. Slow but accurate. Re-ranking only.

ColBERT Late interaction: embed query and doc separately to token-level. Score = sum of
 max-similarity per query token. Balance between both.

Prompt LLM to score relevance. Highest quality but expensive. Use only for final
LLM-as-Reranker
refinement.

FlashRank / BGE- Production re-rankers. BGE-reranker-large commonly used. ~20ms latency for
Reranker top-100 re-rank.

3.4 RAG Evaluation Metrics

Retrieval Metrics Generation Metrics

Recall@K — fraction of relevant docs in top-K Faithfulness — does answer contradict retrieved
context?

Precision@K — fraction of top-K that are relevant Answer Relevance — does answer address the
question?

MRR (Mean Reciprocal Rank) — rank of first Context Relevance — is retrieved context relevant
correct doc to question?

NDCG (Normalized Discounted Cumulative Gain) RAGAS Score — combines faithfulness +

— rank-weighted precision relevance automatically

Hit Rate — does correct doc appear anywhere in EM (Exact Match) / F1 — for extractive QA tasks
top-K?

RAGAS Framework
RAGAS auto-evaluates RAG pipelines using 4 metrics: Faithfulness (LLM-judged), Answer Relevance
(embedding similarity), Context Recall (LLM-judged), Context Precision (LLM-judged). Requires ground-
truth QA pairs. Use for systematic pipeline comparison.

WEEK 4
Fine-Tuning & Alignment
LoRA · QLoRA · SFT · RLHF · DPO · LLM-as-Judge · DeepEval

4.1 LoRA — Low-Rank Adaptation

Instead of updating all W (d×d matrix, d²parameters), decompose the update ΔW into two low-rank matrices: A
(d×r) and B (r×d), where r << d. Total params = 2×d×r instead of d².

W' = W + ΔW = W + BA [r << d, e.g. r=8 vs d=4096 → 8192 vs 16M params per layer]

Higher r = more expressive update = more params. r=4,8,16,64 are common. r=1
Rank r
for minimal adaptation.

Scaling factor for ΔW. Effective learning rate = α/r. Commonly α=2r or α=r.
Alpha (α)
Prevents instability.
 Which weight matrices to adapt. Common: q_proj, v_proj (attention). Adding
Target Modules
k_proj, o_proj, gate/up/down improves quality.

Merge at Inference W' = W + BA. After training, merge for zero-overhead inference. No extra latency.

LLaMA-3 8B: 8B params. LoRA r=16 targeting all attention: ~40M trainable
Parameter Count
params (0.5% of total).

4.2 QLoRA
LoRA applied to a quantized base model (NF4). The base model weights are frozen at 4-bit. LoRA adapters train
in BF16. Two innovations enable this without quality loss:

4-bit quantization with quantile-based levels. Optimal for normally distributed

NF4 (Normal Float 4)
(Gaussian) weights. LLM weights are approximately Gaussian.

Quantize the quantization constants themselves. Saves ~0.37 bits/param

Double Quantization
additionally.

NVIDIA unified memory to page optimizer states CPU↔GPU. Prevents OOM

Paged Optimizers
during large gradient accumulation.

QLoRA fine-tunes LLaMA-2 65B on single 48GB GPU (impossible with full
Memory Reduction
LoRA). Democratizes fine-tuning.

QLoRA matches LoRA quality on most tasks. The 4-bit base model introduces
Quality
negligible degradation vs BF16.

LoRA vs QLoRA Decision Rule

If you have the GPU memory → use LoRA (BF16 base). Cleaner gradients, slightly better results. If
memory constrained (single GPU < 24GB for 7B) → use QLoRA. Almost same quality. For production:
merge LoRA adapters into base for zero-latency inference.

4.3 Adapter Methods Overview

Low-rank weight update injection at linear layers. Most popular. Low param
LoRA
count, good quality.

Decomposes weight into magnitude + direction. Updates both separately. Slight

DoRA
quality improvement over LoRA.

Inject Inhibit & Amplify Activations. Only 3 learned vectors per layer. Fewer
IA³
params than LoRA but lower quality.

Prepend learnable soft tokens to each layer's K/V. Frozen model. No weight
Prefix Tuning
injection. Weaker than LoRA.

Only prefix to input embedding. Lightest but weakest adaptation. Good for very
Prompt Tuning
large models (GPT-3 scale).

Update all parameters. Best quality. Requires full model VRAM. Catastrophic
Full Fine-Tuning
forgetting risk without careful LR schedule.
4.4 Instruction Tuning & SFT (Supervised Fine-Tuning)
SFT trains the model on (instruction, response) pairs using standard next-token prediction loss. The goal: teach
the model to follow instructions rather than predict natural text distribution.

Chat templates vary by model: LLaMA uses [INST] tags, ChatML uses <|
Data Format
im_start|>. Must match exactly.

Only compute loss on response tokens, not instruction tokens. Prevents model
Loss Masking
from optimizing instruction memorization.

Quality >> quantity. 1K high-quality examples often beats 100K noisy ones
Dataset Size
(Alpaca vs LIMA study).

Concatenate multiple (instruction, response) pairs with EOS separators into

Packing
max_seq_len chunks. Maximizes GPU utilization.

1e-4 to 5e-5 for LoRA SFT. Lower than pretraining (2e-4 to 3e-4). Warm up for 3-
Learning Rate
10% of steps.

4.5 RLHF — Reinforcement Learning from Human Feedback

PPO-based RLHF (Standard Pipeline)
Step 1: SFT Supervised fine-tune on demonstrations. Creates SFT model M_sft.

Collect preference pairs (chosen, rejected). Train reward model RM to score

Step 2: RM Training
responses. Binary Bradley-Terry model.

Optimize M_sft via PPO against RM reward. KL penalty vs reference model

Step 3: RL/PPO
prevents reward hacking.

β × KL(π_θ || π_ref). Prevents policy from deviating too far from SFT model
KL Penalty
(mode collapse / reward hacking).

Model finds ways to maximize RM score without being actually better. E.g. longer
Reward Hacking
outputs trick RM. KL penalty mitigates.

DPO — Direct Preference Optimization

DPO eliminates the reward model entirely. Derives the optimal policy directly from preference data using a
reparameterized objective. More stable and simpler than PPO.

L_DPO = -E[ log σ( β × log(π_θ(y_w|x)/π_ref(y_w|x)) - β × log(π_θ(y_l|x)/π_ref(y_l|x)) ) ]

y_w Chosen/winning response

y_l Rejected/losing response

Temperature parameter. Controls how much to deviate from reference. Higher β

β
= stays closer to SFT base.

No RM training. No PPO instability. Single training loop. ~Same quality as PPO

Why DPO wins
on most tasks. Used in LLaMA-3, Mistral.

PPO Advantages DPO Advantages

 Can improve beyond SFT significantly Much simpler implementation

Online learning — generates new samples Stable training — no RL instability

Better for complex reasoning tasks No separate reward model needed

Used in GPT-4, Claude original Lower compute requirements

4.6 LLM-as-Judge & Automated Evaluation

Use strong LLM (GPT-4, Claude) to score model outputs. Prompt with rubric.
LLM-as-Judge
Correlates well with human judgment (~80-90%).

Multi-turn benchmark. 80 questions across 8 categories. GPT-4 judges on 1-10

MT-Bench
scale. Standard for chat model comparison.

Win rate vs reference model (text-davinci-003 or GPT-4). Biased toward length.

Alpaca Eval
Good for instruction following.

LMSYS Chatbot Arena. Human pairwise preferences → Elo ranking. Gold

Arena Elo
standard but slow/expensive.

LLM judges prefer responses in first position. Mitigate: swap order and average.
Positional Bias
Critical to control for.

LLM judges prefer longer responses. Explicitly penalize verbosity in rubric, or use
Verbosity Bias
length-normalized scoring.

DeepEval Framework
Checks if LLM output contradicts source context. Uses LLM to decompose
Faithfulness Metric
claims and verify each against context.

Embedding similarity between question and generated statements extracted from

Answer Relevancy
answer.

Hallucination Metric Checks if output contains facts not present in context. Similar to faithfulness.

DeepEval's LLM-based evaluator. Uses chain-of-thought scoring. Customizable

G-Eval
criteria.

RAGAS Integration DeepEval wraps RAGAS metrics. Unified API for RAG evaluation.

WEEK 5
Production Inference Systems
Scalable Design · Rate Limiting · Load Balancing · Multi-Tenant LLM Serving
5.1 Designing Scalable Inference
Add more inference nodes behind a load balancer. Stateless inference servers
Horizontal Scaling
— any node can handle any request. Scale to demand.

Larger GPUs (H100 vs A100). More memory per node. Reduces need for tensor
Vertical Scaling
parallelism. Higher single-server throughput.

Tensor/pipeline parallelism across GPUs within one node. Necessary for 70B+
Model Sharding
models on A100s.

Redis or Kafka as request queue. Decouple API tier from inference tier. Buffer
Request Queuing
traffic spikes. Enable priority queuing.

Return request_id immediately. Poll or webhook for result. Enables fire-and-

Async Inference
forget for batch workloads.

Scale inference pods based on queue depth, GPU utilization, or p99 latency. K8s
Autoscaling
HPA + custom metrics via KEDA.

Latency SLA Targets (Reference)

Use Case Typical SLA

Real-time chat (interactive) TTFT < 500ms, TPOT < 50ms

Copilot / code completion TTFT < 200ms, TPOT < 30ms

Document summarization (batch) Total latency < 10s acceptable

Offline batch generation Throughput > latency

Multi-modal (image + text) TTFT < 1s (preprocessing adds latency)

5.2 Rate Limiting

Tokens refill at rate r. Max capacity b. Allows bursts up to b. Most common
Token Bucket
algorithm for API rate limiting.

Requests drain at constant rate regardless of arrival. Smooths traffic but

Leaky Bucket
drops/queues bursts. No burst allowance.

Track exact timestamps of requests. O(window_size) memory. Accurate but

Sliding Window Log
memory intensive.

Sliding Window Hybrid: weighted sum of previous + current window. O(1) memory. ~1% error vs
Counter exact.

LLM-specific: rate limit on tokens per minute (TPM) not just requests. Reflects
Token-based limiting
actual compute load.

Atomic rate limit check-and-decrement in single Redis call. No race conditions.

Redis + Lua script
Used in production API gateways.

5.3 Load Balancing for LLM Serving

Round Robin Distribute requests sequentially. Simple. Ignores server load. Bad for variable-
 length LLM requests.

Route to server with fewest active requests. Better for LLM but ignores token
Least Connections
count.

Least Outstanding vLLM's recommended strategy. Route to server with fewest pending tokens in KV
Requests (LOR) cache. Accounts for actual GPU load.

KV Cache-aware Route requests with same prefix to same server (prefix cache hit). Requires
Routing prefix hashing + sticky routing.

If running multiple model variants, route by model name + version. Prevents cold
Model-aware LB
start on every request.

Poll /health endpoint. Remove unhealthy nodes. LLM inference nodes can
Health Checks
silently degrade — monitor GPU memory + error rate.

5.4 Rating / Quality Guardrails (Inline)

Classify input before inference: PII detection, prompt injection detection, topic
Input Guards
filtering. Reject or sanitize before model.

Post-process model output: toxicity scoring, hallucination detection, PII scrubbing

Output Guards
before returning to user.

Open-source framework for input/output validation. Validators: regex, LLM-

Guardrails AI
judged, embedding similarity, custom Python.

NVIDIA. Colang language for declarative guardrails. Defines allowed/blocked

NeMo Guardrails
topics + dialog flows.

Every guardrail adds latency. Budget: input guard <50ms, output guard <100ms.
Latency tradeoff
Use fast classifiers (distilBERT, not GPT-4).

5.5 Multi-Tenant LLM Serving

Serve multiple LoRA adapters on ONE base model GPU. Swap adapters per
LoRA Multiplexing
request. Used in Punica/S-LoRA. Huge cost reduction for multi-tenant.

System for serving thousands of LoRA adapters. Stores adapters in CPU RAM,
S-LoRA
fetches to GPU per request. Batches adapters for GPU efficiency.

Separate KV cache pools per tenant. Prevents cross-tenant context leakage.

Tenant Isolation
Resource quotas per tenant (max_tokens, concurrent_requests).

Premium tenants get lower queue priority numbers. Preemption: lower-priority

Priority Queuing
requests pause for high-priority. Requires checkpoint/resume.

Track tokens consumed per tenant per model. Enable chargeback. vLLM
Cost Attribution
exposes per-request token counts in response.

Route each tenant to their fine-tuned LoRA adapter without knowing about other
Model Routing
tenants. Gateway layer handles routing logic.

S-LoRA — The Multi-Tenant Unlock

Without S-LoRA: serving 100 tenants = 100 separate GPU deployments (100× cost). With S-LoRA: 100
 LoRA adapters share ONE base model. ~1× cost + per-request adapter switching. The trick: adapter
params are tiny (0.1-1% of base). Fit 1000s in CPU RAM. Fetch on demand.

5.6 Observability & Monitoring Stack

Scrape metrics endpoints (/metrics). Store time-series. Metrics: req/sec,
Prometheus
tokens/sec, TTFT p50/p95/p99, GPU utilization, KV cache utilization.

Dashboards on Prometheus data. Set alerts. Standard LLM serving dashboard

Grafana
includes: throughput, latency percentiles, error rates, GPU memory.

Distributed tracing. Trace request through: API gateway → queue → inference

OpenTelemetry
server → guard → response. Critical for debugging latency spikes.

NVIDIA Data Center GPU Manager. Exports GPU telemetry to Prometheus: SM

DCGM Exporter
utilization, memory bandwidth utilization, temperature, power draw.

vLLM exposes: num_requests_running, num_requests_waiting,

vLLM Metrics gpu_cache_usage_perc, tokens_per_sec. Monitor cache utilization to tune batch
size.

MASTER CHEAT SHEET — Numbers to Memorize

A100 80GB peak 2 TB/s HBM2e. H100: 3.35 TB/s HBM3.

bandwidth

A100 80GB peak 312 TFLOPS. H100: 989 TFLOPS (FP16 w/ sparsity).
FP16 FLOPS

LLaMA-3 8B weights 8B × 2 bytes = 16 GB

(FP16)

LLaMA-3 70B weights 70B × 2 bytes = 140 GB → needs 2× A100 80GB

(FP16)

LLaMA-3 70B weights 70B × 0.5 bytes = 35 GB → fits 1× A100 80GB

(INT4 AWQ)

Arithmetic Intensity FLOPs / bytes_moved. Decode: low AI (memory-bound). Prefill: high AI

(AI) (compute-bound).

Roofline boundary ~550 FLOPs/byte. Operations below this line are memory-bound.
(A100)

Typical speculative 2–3× decode throughput. Acceptance rate α ~0.6–0.8.

decoding speedup

LoRA trainable % for ~0.1–1% of total params with r=8–16.

7B

RLHF → DPO quality DPO ≈ PPO on most benchmarks. PPO wins on complex reasoning.
gap
BM25 parameters k1=1.2–2.0, b=0.75. These are standard Elasticsearch defaults.
(defaults)

vLLM PagedAttention Typically 16 tokens per block. Configurable.

block size

Embedding dim GPT- 1536 dimensions

4 ada-002

Continuous batching 2–4× throughput vs static batching at same GPU utilization.

advantage