Quantization for LLMs — Lecture Series

Part 2: Algorithms
From Distribution Mismatch to Production-Ready Quantization
AnhND + Claude
Prerequisite: Part 1 — Hardware Constraints

What This Lecture Is — and Is Not

Part 1 answered: given a target device and deployment scenario, what kind of quantiza-
tion does the hardware require?
Part 2 answers: given that requirement, how do we minimise the accuracy loss?

What this lecture covers:

• The mathematical foundations of uniform and non-uniform quantization

• How to choose a clip range and build a calibration set
• The two paradigms: Quantization-Aware Training vs. Post-Training Quantization
• Why outliers are the central enemy — and how each algorithm fights them
• Weight-only PTQ: GPTQ, AWQ
• Weight-activation PTQ: SmoothQuant, QuaRot, FlatQuant
• KV cache quantization: PolarQuant, QJL, TurboQuant
• Post-quantization correction: AdaRound, BRECQ, EGBC

What this lecture does not cover:

• Hardware constraints, bandwidth arithmetic, kernel fusion ← Part 1, Part 1

• Serving infrastructure, batching strategies, speculative decoding
• Structured pruning, knowledge distillation

Lecture Roadmap
1. Uniform Quantization — the scalar view; XW when weights and activations are
quantized; INT matmul pipeline; granularity
2. Non-Uniform Quantization — codebooks, LUT dequantization, lattice methods,
NVFP4
3. Clip Range & Calibration — rounding vs. clipping error; how the calibration set is
built; Min-Max, Percentile, MSE, KL
4. QAT vs. PTQ — two paradigms; STE; BitNet; who can run what
5. The Outlier Problem — why outliers break quantization; activation vs. weight outliers
6. Weight-Only PTQ — GPTQ (Hessian correction); AWQ (activation-guided scaling)
7. Weight-Activation PTQ — SmoothQuant; rotation-based methods (QuaRot, FlatQuant)
8. KV Cache Quantization — why KV is different; PolarQuant; QJL; TurboQuant
9. Post-Quantization Correction — residual bias; AdaRound; EGBC
10. Synthesis — Algorithm × Hardware matrix; decision flowchart

1
The Narrative Thread
Every section answers the question raised by the previous one:

How the story unfolds

S1–2 establish what quantization is: the grid, the error, the two families (uniform / codebook).
↓ so where do we put the grid?
S3 answers with clip range and calibration.
↓ and when do we run this — during training or after?
S4 maps the two paradigms and their trade-offs.
↓ what goes wrong in practice, even with a good grid?
S5 names the enemy: outliers.
↓ so how do we fight them — weights first?
S6 covers weight-only methods (GPTQ, AWQ).
↓ what if we must also quantize activations?
S7 adds rotation and migration (SmoothQuant, QuaRot).
↓ what about the KV cache at long context?
S8 treats KV as a distinct sub-problem.
↓ can we fix what the rounding left behind?
S9 applies post-quantization correction (AdaRound, EGBC).
↓ so which algorithm do I use for my hardware?
S10 closes the loop with the Algorithm × Hardware matrix.

The One Unifying Idea

Every PTQ algorithm is a specific answer to the same question:[0.5em] how do

we make the weight (or activation) distribution easier to represent on a
uniform integer grid?

Three levers — used alone or in combination by every method in this lecture:

1. Flatten the distribution before rounding (AWQ, SmoothQuant, rotation methods)

2. Correct the error after rounding (GPTQ, AdaRound, EGBC)
3. Match quantization granularity to the hardware datapath (per-group on GPU,
per-channel on NPU)

Connection to Part 1
Part 1 derived which type of quantization the hardware requires (W4A16 vs W8A8,
per-group vs per-channel). Part 2 derives which algorithm minimises accuracy loss within
that type. The two parts are orthogonal axes — hardware and algorithm — and both
must be right for a deployment to succeed.

2
Notation Used in This Lecture
Symbol Meaning
W ∈ Rd×C Full-precision weight matrix (d: input dim, C: output chan-
nels)
X ∈ Rd×m Input activations from m calibration samples
Wq , X q Quantized weight / activation (integer values)
Ŵ , X̂ Dequantized weight / activation (float approximation)
s Scale factor (step size)
z Zero-point (offset for asymmetric quantization)
b Bit-width (e.g. b = 4 for INT4)
µ = E[X] Mean activation vector
Q(·) Quantization operator
WxAy Shorthand: weights at x bits, activations at y bits (e.g.
W4A16, W8A8)
⌊·⌉ Round to nearest integer
⌊·⌋, ⌈·⌉ Floor / ceiling (the two rounding candidates)
H Hessian of the layer reconstruction loss
R Orthogonal rotation matrix

Convention. Unless explicitly stated, uniform quantization is assumed throughout. Non-uniform

(codebook) methods are always labelled as such.

1. Uniform Quantization

From a single float to the integer grid — and what happens to XW

1.1 The Scalar View

Start with the simplest possible case: one floating-point number x.

Quantize — map x to an integer xq :

j x m 
xq = clip + z, 0, 2b − 1
s

Dequantize — recover an approximation x̂:

x̂ = s · (xq − z)

Three knobs — and their effects:

Knob Symbol What it controls
Scale (step size) s>0 Grid spacing; resolution vs. range
Zero-point (offset) z∈Z Asymmetry; z = 0 gives symmetric
Bit-width b Number of levels = 2b

3
 The fundamental tension
Increasing s widens the representable range but coarsens the grid (larger rounding error
per value). Decreasing s gives finer resolution but clips more values to the boundary.
You cannot improve both simultaneously at fixed b. This tension is the root
cause of every difficulty in Sections 5–9.

1.2 Geometric Picture — Rounding vs. Clipping Error

clip min clip max
rounding clipping
x xout
x
x̂ x̂out
−4s −3s −2s −1s 0s 1s 2s 3s 4s

Two distinct error sources:

• Rounding error — every value that falls inside the clip range is rounded to the nearest
grid point. Error bounded by s/2 per value. Decreasing s reduces this.
• Clipping error — values outside the clip range are saturated to the boundary. Error
can be arbitrarily large. Increasing s (widening the range) reduces this.

Key insight. The optimal clip range is not [min(x), max(x)]. Deliberately clipping a
few outliers — accepting some clipping error — often dramatically reduces the rounding
error for the bulk of the distribution. This is the entire motivation for Section 3 (clip
range calibration).

1.3 Symmetric vs. Asymmetric Quantization

Symmetric (z = 0)
max |W |
s= , z=0
2b−1 − 1
• Clip range: [−α, +α]
• Dequantize: ŵ = s · wq (no zero-point subtraction)
• Simpler kernel math
• Good for weights (approximately symmetric distributions)
• Required by some hardware (INT8 Tensor Core accumulation)

Asymmetric (z ̸= 0)
 
max(X) − min(X) min(X)
s= , z=−
2b − 1 s

• Clip range: [α, β], α ̸= −β

• Uses all 2b levels optimally
• Extra zero-point term at dequantize time
• Good for activations (ReLU outputs are non-negative; asymmetric uses the full range)
• AWQ uses asymmetric per-group

4
 Hardware note
The zero-point subtraction (wq − z) must be evaluated at dequantization. For W4A16 this
is absorbed into the register-level unpack — negligible cost. For W8A8 INT matmul the
zero-point induces correction terms in the INT32 accumulator output, requiring extra
additions per output element. ← Part 1, S6 — Kernels & Fusion

1.4 From Scalar to Matrix — Weight-Only (W4A16)

Consider a linear layer Y = X ⊤ W , with W ∈ Rd×C , X ∈ Rd×m .

Quantize only the weights. Store Wq (integers); keep X in FP16.

  
W
Wq = Q(W ) = clip + z, 0, 2b − 1 Ŵ = s (Wq − z)
s
Y ≈ X ⊤ Ŵ = s · X ⊤ (Wq − z)

What the hardware sees (← Part 1, S6 — Kernels & Fusion ):

load + unpack in registers FP16 Tensor Core MMA
Wq −−−−−−−−−−−−−−−−→ Ŵ
|{z} −−−−−−−−−−−−−−−→ Y
|{z}
|{z}
INT4 in HBM FP16 in registers FP16 output

Why this works. The activation X stays in FP16 throughout — no quantization error
is introduced into the activations. The only error source is the weight approximation
Ŵ ≈ W . At 4-bit per-group, this error is small enough that model quality is largely
preserved. This is the dominant production scheme for LLMs.

Bandwidth connection
Storing W in INT4 instead of FP16 reduces weight bytes by 4×. Every weight byte
saved is a direct reduction in HBM transfer time. This — not arithmetic speedup — is
why W4A16 works. ← Part 1, S3 — The Memory Bottleneck

1.5 Both Quantized — W8A8 and the INT Matmul Pipeline

Now quantize both weights and activations:

Xq = Q(X), Wq = Q(W )
X̂ Ŵ = (sX (Xq − zX )) (sW (Wq − zW )) = sX sW · (Xq − zX )(Wq − zW )

The inner product (Xq − zX )(Wq − zW ) is now an integer matmul:

single fused kernel

INT8 × INT8 INT8 Tensor Core INT32 Rescale FP16

inputs MMA accumulator × sX sW output

5
Where the arithmetic speedup comes from: INT8 Tensor Cores deliver 2× the through-
put of FP16 Tensor Cores on A100 (624 TOPS vs. 312 TFLOPS). ← Part 1, S6 — Compute
Architecture

Critical condition
This arithmetic speedup only materialises when the workload is compute-bound —
i.e. batch size B ≳ B ∗ ≈ 156 on A100. At small batch sizes (online serving), the chip is
memory-bound and W8A8 is strictly worse than W4A16 in tokens/second while introducing
more accuracy risk. ← Part 1, S5 — Roofline Model

1.6 Why Not Always Use INT Matmul?

For CNNs — it works well.

• Activations are ReLU-clipped: non-negative, bounded, well-behaved.

• Distributions are approximately uniform within the clip range.
• INT8 matmul + dequantize + INT8 re-quantize is the standard pipeline (Jacob et al.
2018 — the canonical mobile quantization recipe).
• Error introduced at each layer is small and does not compound rapidly.

For LLM transformers — it is much harder.

• Softmax requires high dynamic range in its logit inputs: attention scores span a wide
range; coarse quantization shifts the softmax distribution and destroys ranking.
• LayerNorm is sensitive to the mean and variance of its inputs; quantization bias shifts
these statistics across layers.
• Activation outliers (Section 5): a small number of input dimensions carry extremely
large magnitudes — up to 100× the typical value. These blow up the quantization range
for the entire activation tensor.
• Residual connections accumulate quantization error across all layers; the error com-
pounds in a way that CNN skip connections do not exhibit at the same severity.

This is why W4A16 dominates production LLM deployment, not W8A8. Keeping
activations in FP16 sidesteps all four problems above. The price: no arithmetic speedup
from INT8 Tensor Cores. The benefit: orders-of-magnitude simpler quantization with
substantially better accuracy at 4-bit.

1.7 Granularity — How Many Scale Factors?

The scale s (and zero-point z) need not be a single global value. Granularity controls how
many independent quantization grids are used.

Granularity Scales per layer Accuracy Overhead

Per-tensor 1 Worst None

Per-channel C (one per output row) Good Minimal
Per-group C × ⌈d/g⌉ Best Modest (g=128 typical)

Per-group intuition. Every g consecutive input-dimension weights within a channel share

one (s, z) pair. Finer groups ⇒ smaller effective range per group ⇒ smaller step size ⇒ lower

6
rounding error. Group size g = 128 is the standard: AWQ and GPTQ both use it.

GPU (Tensor Cores)

Per-group scales applied during register-level unpack. Cost: ∼2 instructions per group.
Essentially free relative to HBM load time.

NPU (Systolic Array)

Per-group scales disrupt the pipelined data flow. Often unsupported natively; may
require a dequantize–requantize round-trip at the operator boundary. Use per-channel
instead. ← Part 1, S6 — Compute Architecture

1.8 Granularity — Storage Overhead in Practice

Example: weight matrix W ∈ R4096×4096 , INT4, group size g = 128, FP16 scales.

d 4096
Scale count = C × = 4096 × = 4096 × 32 = 131 072
g 128
Scale storage = 131 072 × 2 bytes (FP16) = 256 KB per layer

Where do the scales live?

• 256 KB fits comfortably in the A100’s 40 MB L2 cache.

• A well-written kernel loads scales into shared memory (192 KB per SM) once per tile and
reuses them — no extra HBM transactions per thread.
• A poorly written kernel loads scales from HBM per warp, multiplying the effective
bandwidth cost by the warp count. This alone can cause a 2× performance gap.

Effective bit-width with scales. Adding FP16 scales to INT4 weights increases the
average bit-width slightly:
16 16
Effective bits = 4 + =4+ = 4.125 bits per weight
g 128

At g = 128, the overhead is only 3% above pure INT4. At g = 32, it rises to 4.5 bits — still
acceptable. At g = 8, it reaches 6 bits, largely defeating the purpose of INT4.

1.9 Putting It Together — The W4A16 Data Flow

unpack cast +
INT4 weightscooperative INT4 tile FP16 × FP16 FP32 acc
tile load + dequant FP16 weights MMA epilogue
+ FP16 scales + scales Tensor Core → FP16
in registers
in HBM in shared mem MMA output

HBM boundary register boundary

• HBM → shared memory: the only expensive step — this is where the 4× bandwidth
saving of INT4 is realised.
• Unpack in registers: ∼3 instructions per 8 weights; negligible vs. HBM latency.

7
 • Tensor Core MMA: runs in FP16 — identical to the FP16 baseline kernel. INT4
storage provides no arithmetic speedup here; the benefit is purely bandwidth.
• FP16 output: activations were never quantized; downstream layers see unmodified FP16
values.

Section 1 in one sentence. Uniform quantization maps floats to integers via a scale
and zero-point; at W4A16, the only change relative to FP16 inference is that weights
travel through HBM in a 4× smaller format and are unpacked to FP16 in registers —
everything else is identical.

Hook into Section 2

The question Section 1 leaves open
Uniform quantization places an equal-width grid over the value range. This is optimal
only if values are uniformly distributed within that range.
Weight distributions in LLMs are not uniform — they tend to be roughly Gaussian
or Laplacian, with most values near zero and a long tail of larger magnitudes. A non-
uniform grid that concentrates bins where the data actually lives can achieve the same
quantization error with fewer bits per weight.
Section 2 introduces codebook (non-uniform) quantization: what it is, why it can be
more accurate, and why it comes with a hardware cost that makes uniform quantization
the default in production.

2. Non-Uniform Quantization

Codebooks, lookup tables, and floating-point formats — when equal-width bins are not the right answer

2.1 Why Uniform Bins Are Often Wrong

Recall from Section 1: a uniform grid places 2b equally-spaced levels across the clip range.

This is optimal only if the data is uniformly distributed. LLM weight distributions
are not uniform — they are typically bell-shaped (Gaussian or Laplacian), with the vast
majority of values concentrated near zero and a long, thin tail.

density
bins wasted[-2pt]on sparse tail

w
coarse where[-2pt]data is dense

The mismatch. With b = 4 (16 levels) and a Gaussian weight distribution, roughly
half the levels land in the sparse tails where almost no weights live. Only ∼8 levels
cover the dense central region. A non-uniform grid that concentrates levels near zero
can achieve lower expected quantization error with the same number of bits.

8
2.2 What Is Non-Uniform Quantization?
General definition. Replace the uniform grid {0, s, 2s, . . . , (2b −1)s} with an arbitrary set
of 2b centroids:

C = {c1 , c2 , . . . , c2b }, ck ∈ R
Quantize: wq = arg mink |w − ck | (assign to nearest centroid)
Dequantize: ŵ = cwq (lookup by index)

The quantized representation stores only the index wq ∈ {1, . . . , 2b } — b bits per weight,
same as uniform. The centroids C are stored once per layer (or per group) as a small lookup
table (LUT).

Voronoi boundaries (decision boundaries)

w
c1 c2 c3 c4 c5 c6 c7 c8 c9 c10

Convention reminder
From here on: uniform quantization is the default. Any method using a codebook
will say so explicitly.

2.3 LUT Dequantization — The Hardware Cost

Uniform dequantize (one multiply-add):
ŵ = s · (wq − z) ⇒ 2–3 arithmetic instructions per weight

Codebook dequantize (one table lookup):

ŵ = C[wq ] ⇒ 1 LUT read — but from where?

The cost of the LUT read depends critically on where C lives in the memory hierarchy
← Part 1, S4 — Memory Hierarchy :

Location Latency Throughput Fits?

Registers 0 cycles ∼100 TB/s Only if 2b ≤ 16 (INT4)

Shared memory ∼30× faster than L2 20–50 TB/s Yes (2b × 2 B ≪ 192 KB)
L2 cache variable 10–12 TB/s Yes, but bank conflicts possible
HBM ∼200 ns 2–3.35 TB/s Defeats the purpose

Why codebook kernels are harder to write

For INT4 (24 = 16 centroids × 2 bytes = 32 bytes), the entire LUT fits in registers —
fast. For INT8 (28 = 256 centroids × 2 bytes = 512 bytes), it must live in shared memory,
requiring careful bank-conflict-free layout. The standard fused dequant-GEMM pattern
from Part 1 cannot be reused without modification; a new kernel must be written. This
is the main reason uniform quantization dominates production: the fused
kernel ecosystem already exists.

9
2.4 Codebook Type I — k-Means Clustering
The simplest non-uniform codebook: run k-means on the weight distribution with k = 2b
clusters.

Procedure:

1. Run k-means on all weights in a layer (or per-group):

X
min min(wi − c)2
C c∈C
i

2. Store the cluster indices wq (b bits per weight) and the centroid table C (FP16, 2b entries).
3. At inference: dequantize via ŵ = C[wq ].

Strengths
• Centroids adapt to the actual weight distribution
• Significant accuracy improvement vs. uniform at the same bit-width
• Simple to implement and understand

Weaknesses
• k-means is sensitive to initialisation
• Codebook is layer-specific: different LUT per layer, complicates batched kernel
launches
• Dequantization is a LUT op, not multiply-add — non-trivial kernel change

Historical note
k-means codebook quantization has a long history in signal processing (vector quantiza-
tion, JPEG, MP3). Its application to neural network weights dates to Han et al. (2016)
— “Deep Compression” — which combined weight pruning, k-means quantization, and
Huffman coding.

2.5 Codebook Type II — Lattice-Based Codebooks (QuIP#)

The key insight behind lattice codebooks. Rather than learning centroids independently
(k-means), use a mathematically structured lattice as the codebook. Lattices have provably
optimal packing density in high dimensions and admit fast nearest-neighbour search without
a full scan.

The E8 lattice (used in QuIP#):

• An 8-dimensional lattice with the densest known packing in 8 dimensions.

• |E8 | = 240 shortest vectors; a scaled and shifted version yields a codebook of 28 = 256
points in R8 .
• Key property: quantizing a vector of 8 weights jointly — rather than each weight
independently — exploits inter-weight correlations and achieves lower distortion per bit.

10
 Vector quantization view. Instead of scalar quantization wi → ck ∈ R, use vector
quantization:
(w1 , . . . , w8 ) → ck ∈ E8 ⊂ R8
Store one 8-bit index per 8 weights: effectively 1 bit per weight in terms of index
overhead, with the E8 lattice providing much better coverage than any scalar 1-bit
scheme.

Why lattice beats k-means. k-means centroids are data-dependent and must be
stored per layer. Lattice centroids are fixed by the mathematical definition of E8 —
they are the same for every layer, eliminating the per-layer LUT storage and allowing
the decode logic to be hardcoded in the kernel.

2.6 Other Lattice and Codebook Structures

Structure Dim Used in Notes

E8 lattice 8 QuIP# Densest 8D packing; 256-entry

codebook; fast decode via Gosset
group symmetry
D4 lattice 4 variants Checkerboard lattice in 4D; sim-
pler decode than E8 ; slightly lower
accuracy
Product quantization d general Split d-dim vector into m sub-
vectors; each sub-vector has its
own small codebook; total m ×
2b/m centroids
Residual quantization d RVQ Quantize residual of previous
round iteratively; used in audio
codecs and some LLM methods

Product quantization is particularly relevant because it scales to high dimensions without

exponential codebook growth: a 128-dim vector quantized with m=16 sub-vectors of size
8 each needs only 16 × 256 = 4096 centroids total, regardless of how many weights are
quantized.

TurboQuant — forward reference

TurboQuant applies a codebook approach specifically to the KV cache, where the access
pattern of the attention kernel makes LUT dequantization less disruptive than it is in a
weight GEMM. We revisit this in Section 8.

2.7 Floating-Point Formats — Hardware-Native Non-Uniform Quantization

Floating-point formats are non-uniform quantization in disguise.
A b-bit float with e exponent bits and m mantissa bits (b = 1 + e + m) represents values as:

x = (−1)sign × 2exp−bias × (1 + mantissa)

The spacing between adjacent representable values is not constant — it is proportional to

the value’s magnitude. Small values are represented densely; large values are represented

11
sparsely. This is exactly the non-uniform coverage that matches a Gaussian/Laplacian weight
distribution.

Format Bits Exp Mantissa Hardware Throughput

BF16 16 8 7 A100, H100 312 TFLOPS (A100)

FP16 16 5 10 A100, H100 312 TFLOPS (A100)
FP8 E4M3 8 4 3 H100+ 1979 TFLOPS (H100)
FP8 E5M2 8 5 2 H100+ 1979 TFLOPS (H100)
NVFP4 4 2 1 Blackwell TBD (native INT4 TC)

FP8 vs INT8. E4M3 has more representable values near zero (where weights cluster)
and fewer at large magnitudes. This means FP8 often requires no explicit clip range
calibration for weights — the floating-point format naturally matches the distribution.
For activations with outliers, FP8 E5M2’s wider dynamic range (5 exponent bits) is
preferred over E4M3.

2.8 NVFP4 — Native Floating-Point at 4 Bits

NVFP4 (NVIDIA FP4, introduced in the Blackwell architecture) is a 4-bit floating-point
format: 1 sign bit, 2 exponent bits, 1 mantissa bit.

Why this matters:

• On current GPUs (A100, H100), INT4 is software-emulated : weights are stored in INT4
and unpacked to FP16 in registers. The matmul still runs on FP16 Tensor Cores. The
bandwidth saving is real; the arithmetic speedup is not. ← Part 1, S6 — Compute Architecture
• Blackwell’s Tensor Cores natively support FP4 multiply-accumulate. FP4 storage therefore
provides both bandwidth savings (4× vs FP16) and arithmetic speedup vs FP8/INT8, in
any regime — not just compute-bound.
• The floating-point non-uniformity means FP4 can represent a wider range of weight
magnitudes than INT4 at the same 4 bits, with natural concentration of levels near zero.

Design implication
The roofline framework from Part 1 still applies to FP4 on Blackwell — but the ridge
point analysis changes because FP4 Tensor Core throughput is higher than FP16. The
hardware co-design argument from Part 1 Section 1 holds: the algorithm (choose FP4
vs INT4 vs FP8) must match what the specific chip can execute natively. ← Part 1, S1
— Why Hardware First?

12
2.9 Accuracy vs. Hardware Cost — Where Each Method Sits

Quantization accuracy
Lattice E8
k-means(QuIP#)
Pareto frontier
codebook
FP8
(H100 native)
Uniform INT4
per-group
(AWQ/GPTQ)
Uniform
INT4
per-tensor production
sweet spot

Hardware cost / kernel complexity

Why the sweet spot is per-group uniform. The Pareto frontier shows that moving
from per-tensor to per-group INT4 yields a large accuracy gain at very low hardware
cost. Moving further to codebook methods yields incremental accuracy improvement at
substantially higher kernel complexity. For most deployments, per-group uniform INT4
(AWQ/GPTQ style) sits at the right trade-off point.

2.10 Section Summary and Convention

Section 2 in four points
1. Non-uniform quantization places bins where data actually lives, reducing expected
error relative to uniform at the same bit-width.
2. Codebook methods (k-means, lattice E8 , product quantization) achieve the best
accuracy per bit, but dequantization is a LUT lookup — requiring new kernel designs
and incurring shared-memory access overhead.
3. Floating-point formats (FP8, NVFP4) provide hardware-native non-uniform
quantization: the exponent field naturally concentrates representable values near
zero, matching weight distributions, without any explicit codebook.
4. Uniform quantization dominates production because its fused dequant-GEMM
kernel ecosystem is mature and its per-group variant already sits near the Pareto
frontier of accuracy vs. hardware cost.

Convention — active from here

Uniform quantization is assumed throughout the remainder of this lecture
unless a method is explicitly described as using a codebook.
Non-uniform methods (QuIP#, TurboQuant, NVFP4) are labelled as such each time
they appear.

13
Hook into Section 3
The question Section 2 leaves open
Both uniform and non-uniform quantization share the same prerequisite: before any grid
can be placed, someone must decide what range to cover.
The clip range [α, β] determines the step size s = (β − α)/(2b − 1), which sets both the
resolution and the saturation point. A poorly chosen clip range makes every algorithm
in Sections 6–9 irrelevant, because the rounding grid is already badly positioned.
Section 3 answers: how is the clip range chosen, how is the calibration set that informs
this choice constructed, and what estimation methods exist — from the naive (min-max)
to the principled (MSE minimisation, KL divergence)?

3. Clip Range & Calibration

The decision every algorithm silently depends on — and how to make it well

3.1 The Silent Prerequisite

Every PTQ algorithm in Sections 6–9 takes quantized weights as input. But quantization
requires a clip range [α, β] first:

β−α jαm
s= , z = − , ŵ = s (wq − z)
2b − 1 s
Every downstream quantity — step size s, rounding decisions, Hessian-based corrections,
bias estimates — is a function of the chosen [α, β].

The silent failure mode

A poorly chosen clip range makes every subsequent algorithm irrelevant.
• Range too wide: step size s is large, rounding error dominates — GPTQ’s Hessian
corrections are trying to fix a coarse grid.
• Range too narrow: outliers saturate to the boundary, clipping error dominates —
AWQ’s scaling cannot recover values that have been destroyed by saturation.
Clip range calibration is structurally prior to all of S6–S9.

Weights vs. activations

Weight tensors are static — their statistics can be computed directly from the model
checkpoint, no calibration data needed. Activation tensors are dynamic — they depend
on the input; their statistics must be estimated from a representative calibration set.

3.2 The Rounding–Clipping Trade-Off

Total quantization error decomposes into two independent components:

14
 E[(w − ŵ)2 ] = E[(w − ŵ)2 1w∈[α,β] ] + E[(w − ŵ)2 1w∈[α,β]
/ ]
| {z } | {z }
rounding error clipping error

error

total error rounding error ∝ s2 ∝ α2

α∗

optimal clip
α∗ < max |w|

min-max
(too wide)

clip range half-width α

clipping error

Key insight. The optimal clip range α∗ is strictly smaller than max |w|. Deliberately
clipping the most extreme values — accepting a small clipping error — allows a finer
grid that reduces rounding error for the bulk of the distribution. Min-max is almost
never optimal.

3.3 Building the Calibration Set

What the calibration set is. A small collection of representative inputs passed through
the full-precision model in a single forward pass (no gradients) to collect per-layer activation
statistics.

Choice Practical guidance

Dataset General-purpose corpus (Pile, C4, WikiText-2) for language

models. Domain-matched data improves results when the
deployment domain is specialised (code, medical, legal).
Sample count 128–512 sequences is the standard pragmatic choice. GPTQ
and AWQ both use 128 samples. More samples give dimin-
ishing accuracy returns while increasing calibration time.
Sequence length Longer sequences expose more of the activation distribution,
especially for attention layers. Very short sequences miss
certain position-dependent patterns.
Diversity Samples should cover the range of topics and styles expected
at inference. A calibration set drawn entirely from one
domain can cause the clip range to be badly tuned for other
domains.

What is collected
For each linear layer, collect the input activation matrix X ∈ Rd×m (where m aggregates
all token positions across all calibration sequences). From X: compute per-channel
min, max, percentiles, mean µ, and variance σ 2 — stored once, used by all clip range
estimators below.

15
3.4 Clip Range Estimators — Overview
Given the collected statistics, five standard estimators:

Method Criterion Cost Typical use

Min-Max No clipping error O(1) Baseline / weights only

Percentile Fixed tail fraction O(m) Fast activation quant
MSE Min squared error O(m · G) Production PTQ
KL divergence Min distribution shift O(m · G) TensorRT pipeline
SQNR Max signal-to-noise O(m · G) Signal-processing tradition
m: calibration samples; G: number of grid search candidates.

All five estimators require the same calibration data — they differ only in the objective
used to score candidate clip ranges. In practice, MSE minimisation is the most widely
used in PTQ pipelines (GPTQ, AWQ, AutoGPTQ default settings) because it directly
minimises the quantity that matters: the expected squared deviation between w and ŵ.

3.5 Estimator I — Min-Max

α = min(W ), β = max(W )
max(W ) − min(W )
s=
2b − 1

Strengths
• Zero clipping error by construction
• No calibration data needed for weights
• O(1) computation (one pass over weights)
• Deterministic: no hyperparameter

Weaknesses
• Dominated by outliers: one value at 50× the typical magnitude forces s to be 50×
too large
• Rounding error becomes catastrophic for the bulk of the distribution
• Almost never the right choice for LLMs where activation outliers are structural

Worked example. Channel with values mostly in [−1, 1] but one outlier at 50. With b = 4
(16 levels):
50 − (−50)
s= ≈ 6.67
15
Every value in [−1, 1] maps to the same quantization level — essentially 1-bit resolution in
the region that contains 99%+ of the data.

16
3.6 Estimator II — Percentile Clipping

α = percentile(W, p), β = percentile(W, 1 − p)

Typical value: p = 0.001 (clip the bottom and top 0.1%).

Intuition. The extreme outliers that dominate min-max are rare by definition. Clipping the
bottom and top p-fraction removes them, dramatically tightening the clip range and reducing
s, at the cost of a small clipping error for the clipped fraction.

percentile range (clips rare tails)

outlier outlier
w
min-max range (dominated by outliers)

Sensitivity warning

The percentile p is a hyperparameter. At small calibration set sizes (m = 128), the

99.9th percentile estimate is noisy — a single unusual sample can shift it substantially.
Using p = 0.01 (clip 1%) is more stable but clips more aggressively. Always validate
the clip range visually on a few representative layers.

3.7 Estimator III — MSE Minimisation

Directly minimise the quantity we care about: the mean squared error between the
original and quantized values.

α∗ , β ∗ = arg min Ew (w − Qα,β (w))2

 
α≤β

where Qα,β is the quantization operator with clip range [α, β].

In practice: grid search over candidate ranges.

1. Generate G candidate clip ranges (e.g. β ∈ {0.5, 0.55, . . . , 1.0} × max |w|, symmetric or
asymmetric).
2. For each candidate, quantize all calibration values and compute the average squared error.
3. Select the candidate with the lowest error.

Cost: O(m × G) — fast in practice with G ≈ 100 candidates and m = 128 samples.

Why MSE is the standard choice. It balances rounding error and clipping error
optimally under a squared-loss criterion — the same criterion used by GPTQ’s recon-
struction objective and EGBC’s bias correction. The estimate is also less sensitive to
the calibration set size than percentile-based methods, because the MSE averages over
all values rather than relying on tail statistics.

17
3.8 Estimator IV — KL Divergence
Preserve the shape of the distribution, not just the magnitude of error.

α∗ , β ∗ = arg min DKL p(w) ∥ p(Qα,β (w))

α,β

Intuition. MSE penalises large pointwise errors. KL divergence penalises changes to the
probability distribution of values — it cares that the quantized distribution has the same
shape as the original, not just the same average squared deviation.

This is particularly useful when:

• The quantized values feed into a softmax (attention scores, output logits): softmax is
sensitive to the relative magnitudes of its inputs, so preserving the distribution shape
matters more than minimising individual errors.
• The deployment metric is perplexity or calibration rather than a regression loss: these
metrics depend on the probability model, which is determined by the distribution of logits.

TensorRT default
NVIDIA’s TensorRT uses KL-divergence calibration as its default for INT8 activation
quantization. The implementation discretises both distributions into 2048-bin histograms
and minimises KL over a set of candidate saturation thresholds.

3.9 Estimator V — Signal-to-Quantization-Noise Ratio

E[w2 ]
SQNR(α, β) = α∗ , β ∗ = arg max SQNR
E[(w − Qα,β (w))2 ] α,β

Connection to MSE. Maximising SQNR is equivalent to minimising MSE when the signal
power E[w2 ] is constant (i.e. not a function of [α, β]). In practice, the two criteria give nearly
identical clip ranges.

Analytical solutions for standard distributions.

• Gaussian N (0, σ 2 ): optimal clip at α∗ ≈ 2.83 σ for 8-bit quantization (decreases with b).
• Laplacian Lap(0, b): optimal clip at α∗ ≈ 5.0 b for 8-bit quantization.

These analytical solutions are useful for rapid initialisation of the grid search in MSE or KL
optimisation — start near the analytical optimum and search a narrow range.

SQNR comes from the signal processing tradition (audio coding, image compression) and is
less commonly seen in recent PTQ papers, which tend to use MSE or KL directly. It is worth
knowing because some hardware vendor documentation (particularly for DSP-based edge chips)
still expresses quantization specifications in terms of SQNR targets.

18
3.10 Per-Channel and Per-Group Clip Ranges
The clip range estimators above apply to a single tensor. In practice, clip ranges are computed
independently per channel or per group, matching the granularity of the quantization scheme.

Per-channel clip range (one [αj , βj ] per output channel j):

• Each output channel has its own weight distribution; a shared clip range would be
dominated by the widest channel.
• Computing per-channel min/max (or percentile, or MSE optimum) adds negligible cost —
it is a single pass over each column of W .
• Per-channel is strictly better than per-tensor for LLMs and should be considered
the minimum granularity.

Per-group clip range (one [αj,g , βj,g ] per group of 128 weights):

• Each group gets its own MSE-optimal range.

• The scale sj,g and zero-point zj,g must be stored alongside the quantized weights — the
256 KB overhead computed in Section 1.8.
• Yields the best accuracy at INT4 and is the default for AWQ and GPTQ.

Activation clip ranges are harder

Activation statistics vary per input token — a clip range computed on the calibration
set may not generalise to all inference inputs. Two responses: (i) use a conservative
(wider) range with percentile calibration; (ii) use dynamic quantization (compute the
clip range per-token at inference time), at the cost of additional compute per layer.

3.11 Calibration Sensitivity — What Can Go Wrong

Even with a good estimator, the calibration set can mislead:

Failure mode Symptom Mitigation

Too few samples Tail percentile estimates are Use ≥ 128 samples; prefer
noisy; outlier-sensitive layers MSE over percentile
get wrong clip range

Domain mismatch Calibration set has different Use domain-matched data; or

activation magnitude profile widen the clip range conserva-
than inference data tively

Short sequences Positional-dependent activa- Use sequences of at least 512–

tion patterns not observed; 1024 tokens
KV cache statistics incom-
plete

Single-domain bias Clip range tuned for one task Mix data from multiple
fails on another sources (C4, WikiText, task-
specific)

19
 Connection to EGBC (Section 9)

EGBC (Section 9) uses a James-Stein shrinkage estimator to obtain a more robust

estimate of the activation mean µ = E[X] from a small calibration set. This is a principled
statistical response to the “too few samples” failure mode: shrink noisy per-dimension
estimates toward a common centre, reducing the variance of the estimator at the cost of
a small bias.

3.12 Section Summary — The Clip Range Checklist

Before running any PTQ algorithm

1. Granularity: use at least per-channel clip ranges; per-group (g = 128) if hardware

supports it.
2. Estimator: use MSE minimisation as the default. Min-max is almost always
suboptimal for LLMs. KL divergence is preferred when the quantized values feed
directly into softmax.
3. Calibration data: 128–512 samples, domain-representative, sequences of 512+
tokens. Mix multiple sources if the deployment domain is broad.
4. Symmetric vs. asymmetric: symmetric for weights (simpler kernel, minimal
accuracy loss); asymmetric for non-negative activations (uses full 2b range).
5. Validate: inspect the clip range and step size for a few representative layers. A step
size orders of magnitude larger than the typical weight value is a red flag.

Hook into Section 4

The question Section 3 leaves open

We now know what to quantize (uniform integer grid, per-group clip range, MSE-
calibrated) and how to choose the range.
The remaining question is when to quantize — and this is not just a scheduling decision.
It fundamentally changes what information is available, what the model can adapt to,
and what computational resources are required.
Quantization-Aware Training (QAT) simulates the quantization during training,
allowing the model to adapt its weights to minimise quantization error end-to-end.
Post-Training Quantization (PTQ) applies quantization after training is complete,
using only a small calibration set and no gradient computation.
Section 4 maps both paradigms in detail: their inputs, outputs, costs, and the regimes
where each is the right choice.

4. QAT vs. PTQ — Two Paradigms

When to simulate quantization during training, and when to apply it after

4.1 The Fundamental Split

Two fundamentally different answers to the question: when does the model learn to live with
quantization error?

20
 QAT

Pretrained or full dataset Training with backprop Quantized model

random-init model simulated quant (high accuracy)

PTQ

Pretrained 128 samples Calibration no backprop Quantized model

model only + quantization (good accuracy)

The key asymmetry. QAT has access to gradients and the full training signal — the
model actively adapts its weights to minimise task loss under quantization. PTQ has no
gradients — it can only rearrange the discrete levels of an already-trained model. This
asymmetry explains almost every difference in cost, accuracy, and applicability between
the two.

4.2 Structured Comparison

QAT PTQ

Starting model Random init or pretrained Pretrained only

Training data Full dataset (billions of to- Calibration set (128–512
kens) samples)
Compute Full training run (weeks to Hours to days (layer-wise)
months at scale)
Gradients Yes — backprop through No — forward pass only
simulated quantization
Accuracy at 4-bit Best Competitive (within 1–2
PPL)
Accuracy at ≤3-bit Best — often the only viable Degrades significantly
option
Who can run it Large models: big tech only. Anyone, regardless of model
Small models: anyone. size
Primary target Ultra-low bit (≤3-bit), Large LLMs, rapid deploy-
CNNs, small transformer ment, 4-bit and above
models

4.3 QAT — Simulated Quantization in the Forward Pass

Core mechanism. During training, apply the quantization operator in the forward pass so
the loss function “sees” the quantization error:

Ŵ = s Q(W/s) − z (fake-quantize)
backprop ∂L update
Ltask (Ŵ ) −−−−−→ −−−−→ W ← W − η ∇W L
∂W

The weights W remain in full precision throughout training. Only the forward pass uses the

21
fake-quantized version Ŵ . The backward pass updates W (full precision) using the gradient
of the task loss computed through Ŵ .

Why this works. With access to the full training objective, the model can learn weight
configurations that are both task-effective and robust to quantization error. Weights
gradually migrate away from rounding boundaries; the distribution adapts to be more
quantization-friendly. This adaptation is impossible in PTQ, which has no gradient
signal.

4.4 The Gradient Problem — round() Has No Gradient

The obstacle. The rounding operation ⌊·⌉ is piecewise constant — its derivative is zero
almost everywhere and undefined at integers. Standard backpropagation cannot flow a
gradient through it.
⌊x⌉
d
dx ⌊x⌉ = 0 a.e.

Three solutions:

1. Straight-Through Estimator (STE) — the standard approach; pretend the gradient

of round is 1.
2. Learned step-size (LSQ) — make s a learnable parameter; its gradient is tractable.
3. Soft / relaxed quantization — replace ⌊·⌉ with a smooth approximation during
training.

4.5 The Straight-Through Estimator (STE)

Proposed by Hinton (2012), formalised by Bengio et al. (2013).

Forward pass:  
ŵ = s · w/s (actual rounding)
Backward pass (STE):

∂ ŵ STE
= 1|ŵ|≤β (pass gradient through if not clipped)
∂w

What STE does. In the backward pass, treat the rounding function as if it were the identity
(gradient = 1) for values inside the clip range, and as zero for clipped values. The gradient of
the task loss flows through the fake-quantize node as if quantization did not happen.

Why it works in practice

• Biased but consistent: gradients point in approximately the right direction
• Widely validated across thousands of QAT experiments
• Simple to implement in any autograd framework

22
 Known limitations
• Gradient is biased — can cause oscillation near rounding boundaries
• Does not work well at very low bit-widths (b = 1) without additional tricks
• Scale s is treated as a fixed hyperparameter (not learned) in basic STE

4.6 Beyond STE — LSQ and Soft Quantization

Learned Step-Size Quantization (LSQ, Esser et al. 2020).
Make the scale s a learnable parameter with its own gradient:

∂ ŵ ∂ 
= s · ⌊w/s⌉ ≈ ⌊w/s⌉ − (w/s) · 1|w/s|≤2b−1
∂s ∂s

The gradient w.r.t. s tells the network how to adjust the step size to minimise the task
loss — the scale is no longer a fixed calibration choice but part of the learned model. LSQ
achieves state-of-the-art QAT accuracy at 4-bit and above, particularly for CNNs and small
transformer models.

Soft / Relaxed Quantization. Replace the hard rounding function with a smooth
approximation:
1
⌊x⌉soft = x − sin(2πx) (periodic approximation)
2π
or a sigmoid-based relaxation that interpolates between floor and ceiling. Provides true
gradients everywhere. Used in research settings; less common in production QAT pipelines.

Connection to AdaRound (Section 9)

AdaRound can be viewed as a PTQ analogue of soft quantization: it learns a per-

weight binary rounding decision (floor or ceiling) via a continuous relaxation, but uses a
calibration-set reconstruction loss rather than the full task loss.

4.7 Ultra-Low Bit — BitNet and Ternary Quantization

At 1–2 bits, PTQ fails. The quantization error is so large that calibration-set correction
cannot recover model quality. QAT — specifically, training from scratch under quantization
— is the only viable approach.

BitNet 1.58 (Ma et al. 2024).

• Weights constrained to {−1, 0, +1} — “1.58 bits” per weight (log2 3 ≈ 1.58).
• Trained from scratch on very large token budgets (same order as full-precision pretraining).
• Key result: with enough data, ternary models approach the performance of full-precision
models at the same parameter count. The model learns to compensate for the extreme
quantization through weight magnitude and structure.

W ∈ {−1, 0, +1}d×C Y = X ⊤W (additions only — no multiplies)

23
 Hardware implication. A ternary matmul requires only additions and subtractions —
no multiply instructions at all. This motivates custom silicon (or very efficient FPGA
designs) where multiply units are replaced with adder trees, dramatically reducing power
and area. On current GPUs, ternary weights must still be unpacked to FP16 for the
Tensor Core MMA, so the arithmetic benefit is not yet realised without specialised
hardware.

4.8 QAT in Practice — Who Can Run It?

Large LLMs (≥7B)

• Full QAT requires retraining on hundreds of billions of tokens

• At 7B parameters, a single training run takes weeks on hundreds of GPUs
• Only organisations with datacenter-scale compute (Meta, Google, Microsoft, Mistral,
. . . ) can afford this
• PTQ is the only practical option for most teams deploying large models

Small models (≤1B)

• QAT is accessible: a 350M model can be fine-tuned with QAT on a single A100 in
hours
• CNNs (ResNet, EfficientNet) are natural QAT targets — well-behaved activations,
mature tooling (PyTorch, TensorFlow Lite)
• Mobile / edge deployment often requires ≤4-bit; QAT is the right tool
• If you can afford QAT for your model size, it is almost always worth it

The middle ground: QAT fine-tuning

A compromise used in practice for medium-sized models (1–7B): start from a pretrained
checkpoint, then run a short QAT fine-tuning pass on a domain-specific dataset (rather
than full pretraining from scratch). This requires far less compute than full QAT while
recovering some of the accuracy gap left by PTQ.

4.9 PTQ in Detail — The Practical Workhorse

Inputs:

• A pretrained full-precision model (any size)

• A calibration set of 128–512 samples (no labels, no task objective)

Output:

• A quantized model: integer weights Wq , per-group scales s, zero-points z

• Unchanged architecture — no new parameters, no changes to the inference graph

Process (per layer, no gradient):

1. Collect input activations X from the calibration forward pass

2. Choose clip range via MSE or percentile (Section 3)
3. Apply the quantization algorithm (NTR, GPTQ correction, AWQ scaling, . . . )
4. Optionally apply post-quantization correction (AdaRound, EGBC — Section 9)

24
 The core PTQ insight
Flatten the weight distribution as much as possible before applying the
uniform grid.
All PTQ algorithms in Sections 6–9 are specific instantiations of this single idea:
• AWQ: scale channels to equalise magnitudes
• GPTQ: redistribute error to already-quantized weights
• SmoothQuant: migrate difficulty from activations to weights
• EGBC: correct the first-moment shift after rounding

4.10 Why PTQ Works at 4-Bit but Struggles Below 3-Bit

At 4-bit (16 levels per group of 128 weights):

• Step size s ≈ σ/2 for a typical weight channel (where σ is the channel standard deviation).
• Rounding error per weight ≲ s/2 ≈ σ/4.
• Output error per channel: ∥∆W x∥ is small relative to ∥W x∥ — residual connections
absorb the perturbation.
• PTQ correction methods (Section 9) can recover the small residual bias. ✓

At 3-bit (8 levels per group) and below:

• Step size s ≈ σ — each level covers one standard deviation of weight variation.
• Rounding error per weight ≲ s/2 ≈ σ/2.
• Output error per channel grows proportionally; residual connections no longer absorb the
perturbation.
• The calibration set (128 samples) provides insufficient signal to correct errors of this
magnitude — there are more degrees of freedom in the error than in the calibration data.

Below 3-bit, PTQ accuracy degrades significantly. QAT (or training-from-scratch

as in BitNet) is required to achieve competitive accuracy at these extreme bit-widths.

4.11 Notation — WxAy and What It Implies

The shorthand WxAy encodes two quantization decisions:

25
Notation Meaning Typical regime

W4A16 Weights in INT4,

activations in FP16 Online serving (memory-
bound);
dominant production scheme
W8A8 Weights in INT8,
activations in INT8 Offline / large-batch
(compute-bound);
enables INT8 Tensor Cores
W4A8 Weights in INT4,
activations in INT8 Middle ground; upcoming
with
native INT4 Tensor Cores (Blackwell)
W8A16 Weights in INT8,
activations in FP16 Rarely used; bandwidth
saving
only 2× vs FP16
W2A16 Weights in 2-bit,
activations in FP16 Research / ultra-low bit;
typically requires QAT
W1A8 Ternary weights,
INT8 activations BitNet-style; requires
training from scratch

The hardware connection

The Ax part determines whether INT Tensor Cores can be used (requires x ≤ 8). The Wx
part determines the weight bandwidth saving. Both must match the hardware’s native
datapath. ← Part 1, S5 – Roofline; S6 – Compute Architecture

26
4.12 Section Summary
Section 4 in five points
1. QAT simulates quantization during training, allowing the model to adapt end-to-end.
Produces the best accuracy at any bit-width, but requires full training compute.
2. PTQ applies quantization after training using only a small calibration set and no
gradients. Practical for any model size; the standard approach for large LLMs.
3. STE is the canonical gradient estimator for QAT — a biased but effective approxi-
mation that treats rounding as identity in the backward pass.
4. BitNet / ternary QAT shows that training-from-scratch at extreme bit-widths
(≤2-bit) can match full-precision quality, at the cost of full pretraining compute and
specialised hardware.
5. PTQ works at 4-bit, struggles below 3-bit. The crossover is determined by
whether the rounding step size is small relative to the weight distribution’s standard
deviation.

Hook into Section 5

The question Section 4 leaves open
We now know the two paradigms, how gradients flow through quantization, and when
each approach is appropriate.
But we have not yet addressed the single biggest obstacle to achieving good quantization
accuracy in practice — for both QAT and PTQ, but especially for PTQ:
Outliers.
A small number of weight channels and, more severely, a small number of activation
dimensions carry values that are orders of magnitude larger than the rest. These outliers
dominate the clip range, force a coarse step size, and cause essentially all the quantization
error that PTQ algorithms in Sections 6–9 are trying to fix.
Section 5 names the enemy precisely: what outliers are, why they arise in large
transformers, why they are structurally different in activations vs. weights, and exactly
how they break uniform quantization.

5. The Outlier Problem

The single biggest obstacle to LLM quantization — and why it gets worse with scale

5.1 Naming the Enemy

Definition
An outlier is a weight value or activation value whose magnitude is dramatically larger
than the typical value in the same tensor — often 10×–100× larger. In LLM quantization,
outliers are not random noise. They are structural: persistent, reproducible, and
concentrated in specific dimensions.

Why this section is here. Every algorithm in Sections 6–9 exists primarily to deal with
outliers. Understanding them precisely — what they are, where they live, and exactly how

27
they break quantization — is the prerequisite for understanding why any of those algorithms
make the design choices they do.

The narrative so far. Sections 1–3 established the quantization grid and clip range.
Section 4 established when to apply quantization. Section 5 establishes what goes wrong
— the enemy that all subsequent algorithms are designed to fight.

5.2 Where Outliers Come From

Weight outliers are relatively mild and well-understood:

• Arise from the interaction of Adam optimiser, weight decay, and layer normalisation
during training.
• Some output channels learn much larger weights than others because they encode more
“salient” features.
• Magnitude: typically 3×–10× the median channel norm.
• Per-channel scaling (Section 3) largely handles this — each channel gets its own clip
range.

Activation outliers are severe and were largely unexpected:

• First documented at scale by Dettmers et al. (2022) in LLM.int8() — the paper that
made the problem famous.
• Appear in models with ≥ 6.7B parameters trained on large corpora.
• A small number of fixed input feature dimensions (typically 1–5 out of 4096+) develop
persistent large magnitudes.
• Magnitude: up to 100× the typical activation value.
• Structural: the same dimensions are large across all tokens, all layers, all forward passes.
They are a property of the trained model, not the input.

Why scale matters

Below ∼1B parameters, activation outliers are rare and mild. Above ∼6.7B parameters,
they are ubiquitous and severe. The transition is relatively sharp — a phase transition
in the representational structure of large transformers.

5.3 Visualising Activation Outliers

outlier[-2pt]dim outlier[-2pt]dim

outlier (∼100× typical)

normal value
tokens m

hidden dimension d (16 of 4096 shown)

28
 Key structural property. The outlier dimensions (red columns) are identical across
all tokens. Every row of the activation matrix has a large value in dimension 5 and
dimension 12, regardless of the input text. This means the outlier pattern is a property
of the model weights, not the input — and it cannot be addressed by per-token scaling
alone.

5.4 How Outliers Break Uniform Quantization

Recall (Section 1): the clip range determines the step size:

β−α
s=
2b − 1
One large outlier forces β (or |α|) to be large, which forces s to be large, which coarsens the
grid for all other values.

Worked example. INT8 quantization (b = 8, 256 levels). A hidden dimension with values
in [−1, 1] but one outlier at 100:

100 − (−100)
smin-max = ≈ 0.784
255
Every value in [−1, 1] is assigned one of only
 
2
+ 1 = 3 distinct levels
0.784

out of 256 available. 99%+ of values use ≈ 1% of the grid.

The range-resolution trap

• Keep the outlier in range ⇒ step size ≫ typical value spacing ⇒ catastrophic rounding
error for all non-outlier values.
• Clip the outlier ⇒ large clipping error on the outlier, which then propagates through
softmax and LayerNorm.
• There is no good choice under a single global scale. The outlier and the bulk
of the distribution are fundamentally incompatible with a shared uniform grid.

29
5.5 Weight Outliers vs. Activation Outliers

Weight outliers Activation outliers

Magnitude 3×–10× typical Up to 100× typical

Location Some output channels of Fixed input dimensions of
W X
Persistence Static — fixed at inference Static across tokens (struc-
tural), but value varies per
input
Affects W4A16 and W8A8 W8A8 only (W4A16 keeps
X in FP16)
First fix Per-channel scaling (Sec- Per-channel migration
tion 3) (SmoothQuant, Section 7)
Deeper fix GPTQ error redistribu- Rotation (QuaRot,
tion, AWQ scaling (Sec- FlatQuant, Section 7)
tion 6)

Why W4A16 sidesteps activation outliers entirely. In W4A16, activations are

never quantized — they remain in FP16. Only weights are quantized, and weight outliers
are manageable with per-channel or per-group scaling. This is a major practical reason
why W4A16 dominates production LLM deployment over W8A8.

5.6 Per-Channel Scaling — The First Partial Fix

Observation. Weight outliers are concentrated in specific output channels of W . If each
channel gets its own scale sj , the outlier in channel j only affects channel j’s grid — not the
other C − 1 channels.

Per-channel quantization:
  
max |w:,j | wij
sj = , wq,ij = clip , −(2b−1 ), 2b−1 − 1
2b−1 − 1 sj

ŵij = sj · wq,ij

What this achieves and what it does not:

Achieves
• Isolates each channel’s clip range — one large channel cannot contaminate others
• Dramatically reduces per-channel quantization error vs. per-tensor
• Necessary minimum for LLMs — per-tensor is almost never acceptable

30
 Does not achieve
• Does not fix outliers within a channel — the outlier still dominates that channel’s
step size
• Does not address activation outliers (fixed in X, not in W )
• Does not correct the residual bias E[(Wq − W )X] ̸= 0

Per-group scaling (group size 128) goes further: within a channel, each group of 128 weights gets
its own scale, so even intra-channel outliers are isolated. This is the default for AWQ and GPTQ.

5.7 The Intra-Channel Outlier Problem

Per-channel scaling isolates channels from each other. But within a channel, outliers at
specific input dimensions still dominate the group step size.

|wmost
ij | values use only lowest 1–2 levels
outlier
step size s ∝ max |wij |
outlier

input dimension i within a group of 128

This is exactly the problem AWQ addresses. AWQ identifies that the input
dimensions with large |µi | (large mean activation magnitude) correspond to the weight
dimensions that carry the most output influence. By rescaling those weight dimensions
before quantization, AWQ effectively reduces the intra-group outlier without changing
the mathematical output. We examine the mechanism in Section 6.

5.8 Activation Outliers in W8A8 — The Deeper Problem

When activations must also be quantized (W8A8), the structural outlier dimensions of X
create a severe problem that per-channel weight scaling cannot fix.

Why activation outliers are harder than weight outliers:

1. Dynamic values. Unlike weights, activation values change with each input. The
outlier dimension is fixed, but its magnitude varies per token and per input sequence. A
per-channel scale calibrated on 128 samples may be wrong for a specific inference input.
2. No offline fix. Weight outliers can be handled offline (before deployment) by analysing
the weight matrix. Activation outlier scales must either be calibrated offline (and may
generalise poorly) or computed online (dynamic quantization — expensive per token).
3. Propagation through softmax. A large quantization error in an attention Key shifts
the softmax distribution catastrophically — wrong attention weights corrupt the entire
layer output, not just one dimension.

31
 The SmoothQuant insight (preview)

The key observation: if the outlier is in the activation dimension i, we can divide Xi by a
constant si (making it smaller, easier to quantize) and multiply the corresponding weight
row Wi,: by si (absorbing the scale). The product XW is unchanged. This “migrates”
the quantization difficulty from activations to weights. Details in Section 7.

5.9 Alternative Response — SpQR

Sparse-Quantized Representation (SpQR, Dettmers et al. 2023).
Rather than trying to quantize outlier weights accurately with a modified grid, SpQR simply
stores them in higher precision.

W = Wsparse + Wdense
| {z } | {z }
outlier weights in FP16 remaining weights in INT4

Procedure:

1. Identify outlier weights (top ∼1% by magnitude or sensitivity).

2. Store outliers in a sparse FP16 format alongside their indices.
3. Quantize all remaining weights to INT4.
4. At inference: Y = X ⊤ Wdense + X ⊤ Wsparse (two matmuls — one INT4, one sparse FP16).

Strengths
• Outlier weights are exact — zero quantization error for the most influential weights
• Principled: protects weights by their actual impact on output

Weaknesses
• Two matmuls per layer — kernel complexity increases
• Sparse FP16 storage overhead reduces effective compression ratio
• Less hardware-friendly than pure INT4 weight storage

32
5.10 The Outlier Landscape — How Each Algorithm Responds

Algorithm Outlier type ad- Mechanism

dressed

Per-channel scaling Weight channel out- Independent clip range per chan-
liers nel
Per-group scaling Intra-channel weight Independent clip range per 128
outliers weights
AWQ Intra-channel weight Scale salient dims before quan-
outliers tization
GPTQ Intra-channel weight Redistribute error via inverse
outliers Hessian
SmoothQuant Activation outliers Migrate magnitude to weights
(W8A8)
QuaRot/FlatQuant Mixed-dim activation Rotation spreads energy uni-
outliers formly
SpQR Most influential Store outliers in FP16, rest in
weight outliers INT4
EGBC Residual bias from all Correct first-moment shift after
sources rounding

The hierarchy of responses

1. Isolate (per-channel/group) — prevent outliers from contaminating other values.

2. Flatten (AWQ, SmoothQuant, rotation) — reduce the outlier magnitude before
quantization.
3. Correct (GPTQ, EGBC) — fix the residual error that flattening leaves behind.

5.11 Section Summary

Section 5 in five points
1. Outliers are structural, not random: the same weight channels and activation
dimensions are persistently large across all inputs and layers.
2. Activation outliers are worse than weight outliers: they appear above ∼6.7B
parameters, reach 100× typical magnitude, and are incompatible with a shared
uniform grid when activations must be quantized.
3. The range-resolution trap: a single outlier forces the step size s to be large,
destroying resolution for the bulk of the distribution. There is no good choice under
a single global scale.
4. W4A16 avoids activation outliers entirely by keeping activations in FP16. This
is a primary reason it dominates production LLM deployment.
5. Three levels of response — isolate (per-channel/group), flatten (AWQ,
SmoothQuant, rotation), correct (GPTQ, EGBC) — correspond to the three algo-
rithm families in Sections 6–9.

33
Hook into Section 6
The question Section 5 leaves open
We have named the enemy and established the three levels of response.
The first battle is weight-only quantization: target W4A16, GPU deployment, online
serving (the memory-bound regime from Part 1). Activation outliers are sidestepped by
keeping X in FP16. The remaining challenge is weight outliers — intra-channel values
that dominate the per-group step size and cause excessive rounding error.
Section 6 covers the two dominant weight-only PTQ algorithms:
• GPTQ — correct the error after rounding by redistributing it to unquantized weights
via the inverse Hessian.
• AWQ — flatten the distribution before rounding by scaling salient weight dimensions
guided by activation magnitudes.
These are not competing methods — they address the same problem from opposite
directions, and they are composable: AWQ can be applied first, then GPTQ, then
EGBC.

6. Weight-Only PTQ — GPTQ & AWQ

Correcting after rounding vs. flattening before rounding — two complementary answers to the same
problem

6.1 Context — What W4A16 Requires

Target regime (from Part 1):

• Online serving, batch size B = 1–16

• Memory-bound: the binding constraint is HBM bandwidth
• Goal: reduce weight bytes 4× (FP16 → INT4) to approach the 4× token-rate ceiling
• Activations stay in FP16 — activation outliers are irrelevant

What the algorithm must solve:

• Given pretrained W ∈ Rd×C and calibration activations X ∈ Rd×m

• Find quantized Wq (INT4, per-group g = 128) such that the layer output error ∥X ⊤ (W −
Ŵ )∥ is minimised
• No gradients, no architectural changes, no modification to the inference graph

The baseline: Nearest-to-Round (NTR). Simply quantize each weight indepen-

dently to its nearest integer level. Fast, zero calibration cost, but ignores all structure:
the interaction between weights, the activation distribution, and the cumulative effect of
rounding on the layer output. Both GPTQ and AWQ are improvements over NTR —
from different directions.

34
6.2 GPTQ — Motivation
The NTR blind spot. Nearest-to-Round minimises each weight’s individual rounding
error, but ignores correlations: rounding wij introduces an error that propagates through the
output X ⊤ W , affecting the error contribution of all remaining unquantized weights.

Optimal Brain Quantization (OBQ) — the foundation. Quantize weights one at a

time. After quantizing weight wij , update all remaining unquantized weights to compensate
for the error introduced.

Layer reconstruction objective:

C
X
min ∥X ⊤ (W − Ŵ )∥2F = min ∥X ⊤ (wj − ŵj )∥22
Wq Wq
j=1

Each output channel j is independent — the loss decomposes per column.

OBQ per-channel problem. For column j, quantize weights wj,1 , wj,2 , . . . , wj,d one at a
time. After quantizing wj,i , update remaining weights via:

(wj,i − ŵj,i )
δwj,i′ = − · [Hj−1 ]ii′ ∀ i′ > i
[Hj−1 ]ii

where Hj = 2XX ⊤ is the layer Hessian for column j.

6.3 GPTQ — The Algorithm

GPTQ (Frantar et al. 2022) makes OBQ practical for large models via two key observations:

Observation 1: All columns share the same Hessian. H = 2XX ⊤ ∈ Rd×d does not
depend on j. Compute and invert H once; reuse for all C output channels.

Observation 2: Quantize in column order, not arbitrary order. Processing weights

column-by-column (all channels simultaneously) allows the Cholesky decomposition of H −1
to be updated incrementally — turning an O(d3 ) per-weight operation into an amortised
O(d2 ) per-column operation.

GPTQ — Core Loop (per layer)

Require: W ∈ Rd×C , X ∈ Rd×m , group size g

1: Compute H = 2XX ⊤ ; obtain H −1 via Cholesky
2: for i = 1 to d do
3: ŵ:,i ← Q(w:,i ) {quantize column i across all output channels}
4: ei ← w:,i − ŵ:,i {rounding error for column i}
[H −1 ]i, i+1:d
5: W:, i+1:d −= ei {propagate error to remaining columns}
[H −1 ]ii
6: end for
Ensure: Wq (INT4), per-group scales s, zero-points z

35
6.4 GPTQ — Intuition and Error Flow
quantized
current
unquantized (updated)

output channels input dimension i

error redistributed to remaining weights

Key intuition. When weight i is rounded, the introduced error is “pushed forward” onto
weights i + 1, . . . , d in proportion to their Hessian coupling. Weights that are highly correlated
with i (large [H −1 ]ii′ ) receive a larger compensating update. The result: the overall layer
output error is minimised even though individual rounding errors are non-zero.

6.5 GPTQ — Strengths and Weaknesses

Strengths
• Captures weight correlations — the only weight-only method that explicitly models
how quantizing one weight affects others
• Strong accuracy at 4-bit, competitive at 3-bit
• Hardware-friendly output: standard per-group INT4, compatible with all fused
dequant-GEMM kernels
• Composable: can be applied after AWQ or other preprocessing

Weaknesses
• Calibration overfitting: the Hessian correction is fitted to the calibration set.
With only 128 samples, the correction can overfit to calibration distribution and
degrade on out-of-distribution inputs
• Heavier compute than AWQ: O(d2 ) per layer for the Hessian inversion (hours for
70B models)
• Does not address the first-moment bias E[(Wq − W )X] — EGBC can fix this residual

Practical note. GPTQ is implemented in the AutoGPTQ and ExLlamaV2 libraries. For a 70B
model, GPTQ quantization takes 4–8 hours on 4×A100. The resulting model is indistinguishable
from the FP16 baseline on most benchmarks at 4-bit per-group-128.

6.6 AWQ — Motivation: Weight Saliency

The AWQ (Lin et al. 2023) starting observation.
Not all weights are equally important. A weight wij contributes to the output by the product
wij · xi , where xi is the i-th input activation. If |xi | is large (an activation-outlier dimension),
then even a small rounding error in wij produces a large output error.

36
 Saliency of weight wij :

saliency(wij ) ∝ |µi | · |wij | where µi = E[xi ]

Weights at input dimensions with large mean activation magnitude |µi | are salient —
their quantization error has a disproportionate effect on the output.

AWQ’s insight. Rather than protecting salient weights by storing them in higher
precision (as SpQR does), AWQ makes them easier to quantize accurately by rescaling the
corresponding input dimension before the quantization grid is applied. No architecture
change, no extra parameters, no sparse storage.

6.7 AWQ — The Scaling Mechanism

The key mathematical equivalence. For any positive diagonal scale vector s ∈ Rd>0 :

X ⊤ W = (X · diag(s)−1 ) · (diag(s) · W )
| {z } | {z }
X̃ W̃

The product is exactly preserved. No approximation.

How AWQ uses this. Choose si > 1 for salient dimensions (those with large |µi |):

• W̃i,: = si · Wi,: — the salient weight row is scaled up, compressing its dynamic range
relative to the (now larger) per-group maximum. Equivalently: the salient row occupies
more of the quantization grid — finer effective resolution.
• X̃:,i = X:,i /si — the activation dimension is scaled down, absorbed into the modified
weights. At inference, X̃ is the actual input to the quantized layer; the scale si is absorbed
offline into a preceding LayerNorm or stored as a per-channel correction.

What “absorbed into LayerNorm” means

Most transformer layers have a LayerNorm immediately before the linear layer. Layer-
Norm computes γ · x̂ + β with learned γ, β. AWQ absorbs s−1 into γ offline: γ̃ = γ/s.
This changes no parameters at inference — it is a free fold-in that requires no new ops
and no inference-time overhead.

6.8 AWQ — Choosing the Scale

How is s determined?
AWQ minimises the per-channel layer reconstruction error over a search space of scale
candidates:

2
s∗ = arg min X ⊤ W − X̃ ⊤ Q(W̃ ) F
s

where W̃ = diag(s)W and X̃ = X diag(s)−1 .

37
In practice: grid search over a simple parameterisation. AWQ parameterises si = |µi |α
for α ∈ [0, 1]:

• α = 0: no scaling (s = 1) — equivalent to NTR

• α = 1: full activation-proportional scaling
• Search over ∼20 values of α; pick the one minimising reconstruction error on the calibration
set

Why this is fast. No Hessian inversion. No gradient computation. The search over α
requires only ∼20 forward passes through the quantization operator — a few minutes
per layer even for the largest models. AWQ is typically 10×–20× faster than GPTQ.

6.9 AWQ — Geometric View of the Scaling Effect

Before AWQ scaling
density After
density AWQ scaling (si > 1)
wide range tighter spread

scale

wi (salient dim) w̃i = si wi

coarse grid (s large) finer effective resolution

What the scaling achieves. By scaling the salient weight dimension up (and the
corresponding activation down by the inverse), the effective per-group step size for that
dimension becomes finer — more quantization levels cover the region where the salient
weights actually live. The output product XW = X̃ W̃ is unchanged.

6.10 GPTQ vs. AWQ — Comparison

GPTQ AWQ

Core mechanism Error redistribution via Activation-guided rescal-

inverse Hessian ing before rounding
When it acts After rounding (corrects Before rounding (flattens
error) distribution)
Calibration sensitivity Higher — Hessian fitted Lower — scale search is
to calibration set more robust
Speed Slower (O(d2 ) per layer; Fast (grid search; min-
hours for 70B) utes for 70B)
Accuracy at 4-bit Slightly better on average Competitive; occasion-
ally better
Accuracy at 3-bit Better Degrades more
Composability Can follow AWQ prepro- Applied first; GPTQ can
cessing follow
Output format Per-group INT4 Per-group INT4
Bias correction Does not address Does not address
E[(Wq − W )X] E[(Wq − W )X]

38
 They are composable, not competing

AWQ scales the weight matrix first (flattening outliers), then GPTQ applies error
redistribution on the scaled matrix, then EGBC corrects the residual first-moment bias.
Each method addresses a different layer of the problem. In practice, AWQ + GPTQ
together outperform either alone at 3-bit.

6.11 Hardware Fit — Kernel Compatibility

Both GPTQ and AWQ produce the same output format:

• INT4 quantized weights Wq , packed two-per-byte

• FP16 per-group scale sj,g and zero-point zj,g (one pair per 128 weights per output channel)
• Stored in a pre-permuted layout for Tensor Core alignment

This format is directly consumed by the standard W4A16 fused dequant-GEMM kernel
described in Part 1, Section 7:

INT4 Wq
tile in unpack FP16 Ŵ FP16 MMA FP16
+ FP16 scales
shared mem registers Tensor Core output
HBM

Zero inference overhead from the quantization algorithm. The entire

GPTQ/AWQ correction happens offline during quantization. At inference, the quantized
model runs with exactly the same kernel as any other INT4 model — no extra ops, no
algorithm-specific code.

6.12 What Neither GPTQ nor AWQ Addresses

Both methods minimise the second-order reconstruction error ∥X ⊤ (W − Ŵ )∥2F . Neither
explicitly controls the first-order moment:
The residual bias

E (Wq − W )X = E[X]⊤ (Wq − W ) = µ⊤ ∆W ̸= 0

 

Even after GPTQ or AWQ, the expected output of the quantized layer differs from the
expected output of the full-precision layer by this systematic bias.
• Softmax inputs are shifted — attention calibration is distorted.
• LayerNorm statistics change — normalisation is off.
• The bias compounds across layers.

This is the gap that Section 9 (EGBC) fills. EGBC treats bias correction as a
discrete selection problem: selectively flip quantized weights from ⌊·⌋ to ⌈·⌉ to drive
µ⊤ (Wq − W ) toward zero, per output channel, with negligible computational overhead.
EGBC is designed to be applied after GPTQ or AWQ as a lightweight post-processing
step.

39
6.13 Section Summary
Section 6 in five points
1. GPTQ quantizes weights column-by-column, redistributing each rounding error to
unquantized weights via the inverse Hessian. Strong accuracy, especially at 3-bit;
heavier compute; prone to calibration overfitting.
2. AWQ scales salient weight dimensions (those corresponding to large-magnitude acti-
vation inputs) before quantization, compressing their dynamic range and improving
per-group resolution. Fast, robust, and composable.
3. Both target the same format: per-group INT4 (W4A16), fully compatible with
the fused dequant-GEMM kernel ecosystem.
4. They are composable: AWQ scales first, GPTQ corrects second, EGBC removes
the residual bias third.
5. Neither addresses the first-moment bias µ⊤ (Wq − W ), which is the residual
left for Section 9 (EGBC) to correct.

Hook into Section 7

The question Section 6 leaves open
GPTQ and AWQ solve the weight-only problem well at 4-bit. But they both rely on a
critical assumption: activations stay in FP16.
This assumption fails when the deployment requires W8A8 (offline, large-batch, compute-
bound regime from Part 1). In W8A8, activations must also be quantized to INT8 to
unlock INT8 Tensor Core arithmetic. And quantizing activations means facing the
outlier problem head-on — the activation outlier dimensions that W4A16 simply ignored.
Section 7 covers the two dominant approaches to weight-activation quantization:
• SmoothQuant — migrate quantization difficulty from activations to weights via
per-channel scaling.
• QuaRot / FlatQuant — apply an orthogonal rotation to spread outlier energy
uniformly across all dimensions, making both weights and activations easier to
quantize.

7. Weight-Activation PTQ

When activations must also be quantized — migration, rotation, and the cost of going beyond W4A16

7.1 Why W8A8 — Recap from Part 1

The regime shift. ← Part 1, S5 – Roofline Model
At small batch sizes (B ≲ 16), LLM inference is memory-bound: the binding constraint is
HBM bandwidth, and W4A16 is optimal.
At large batch sizes (B ≳ 156 on A100), the workload crosses the ridge point and becomes
compute-bound. In this regime:

• Weights are loaded once but reused B times — bandwidth cost per token shrinks with B
• The compute units (Tensor Cores) become the bottleneck

40
• INT8 Tensor Cores deliver 2× the throughput of FP16
• To use INT8 Tensor Cores, both weights and activations must be in INT8

W8A8 pipeline:
INT8 TC MMA ×s s
acc −−−X−−
Xq × Wq −−−−−−−−−−→ |{z} W
→ |{z}
Y
|{z} |{z}
INT8 INT8 INT32 FP16

The new obstacle

Quantizing activations means confronting activation outliers directly. The 100×-
magnitude dimensions that W4A16 could ignore now destroy the INT8 activation clip
range — and there is no “keep activations in FP16” escape hatch.

7.2 The Activation Quantization Challenge

Why activations are harder to quantize than weights:

Weights Activations

When known Offline (static) Online (changes per input)

Outlier location Some output channels Fixed input dimensions,
but value varies per token
Clip range Calibrate once offline Must generalise to all in-
puts, or recompute per to-
ken
Per-token scaling Not needed Needed but expensive: one
scale per token per layer
Distribution shape Roughly Gaussian per Heavy-tailed with persis-
channel tent high-magnitude di-
mensions

The core insight shared by all W8A8 methods: rather than trying to quantize
the outlier activation dimensions accurately (which requires a very large clip range and
coarse grid), reduce their magnitude before quantization. The two approaches
differ in how they reduce it: migration (SmoothQuant) or rotation (QuaRot, FlatQuant).

7.3 SmoothQuant — Motivation

Key observation (Xiao et al. 2022): activation outliers live in fixed input dimensions,
and the corresponding weight rows are often not outliers.

41
 X (activations) W (weights)

×
difficulty in X,normal weights
outlier dim not in W

The SmoothQuant idea. If the difficulty is in the activations but not the weights,
move it from activations to weights via a mathematically equivalent per-channel scaling.
The resulting activations are smooth (easy to quantize); the resulting weights are slightly
harder but still quantizable.

7.4 SmoothQuant — The Scaling Mechanism

The exact equivalence:

Y = XW = (X diag(s)−1 ) · (diag(s)W )
| {z } | {z }
X̃, smooth W̃ , absorbs outlier

Choose si > 1 for outlier dimension i: divides X:,i by si (reduces outlier magnitude)
and multiplies Wi,: by si (weight row absorbs the scale).

Choosing the migration strength α:

max |X:,i |α
si = , α ∈ [0, 1]
max |Wi,: |1−α
• α = 0: no migration — s = 1, original problem
• α = 0.5: split difficulty equally between X and W
• α = 1: full migration — all difficulty moves to weights

Optimal α is found by grid search over the calibration set, minimising the combined
quantization error of X̃ and W̃ . In practice α = 0.5 works well across most layers.

7.5 SmoothQuant — Zero Runtime Cost

Where does s live at inference?
The scale vector s can be absorbed offline into the preceding LayerNorm:
LayerNorm Quantize Linear
before[-2pt]smooth
γ, β X → Xq Xq Wq

fold s−1 into γ

LayerNorm Quantize Linear

after[-2pt]smooth
γ̃ = γ/s X̃ → Xq Xq W̃q

42
 Inference graph is unchanged. The LayerNorm weight γ is modified offline (γ̃i =
γi /si ). The linear layer weight is modified offline (W̃i,: = si Wi,: , then quantized). At
inference: standard LayerNorm + standard INT8 linear — no extra ops, no per-token
scaling, no new parameters.

7.6 SmoothQuant — What It Does Not Solve

SmoothQuant handles outliers in fixed, per-channel dimensions by migrating them to weights.
Two structural limitations remain:

Limitation 1: Mixed-dimension outliers. In some layers — particularly attention output

projections and deeper FFN layers — the outlier pattern is not cleanly per-channel. Outlier
magnitude is spread across many dimensions with no single dominant dimension to migrate.
Per-channel scaling s cannot flatten a mixed pattern.

Limitation 2: The weight side gets harder. Migration moves difficulty from X to W .
If α is too large, the weight rows absorb too much outlier magnitude and become harder to
quantize than the original weights. The optimal α trades off these two difficulties — it never
eliminates both simultaneously.

The rotation solution. Both limitations can be addressed by applying an orthogonal

rotation R to the weights and activations. Rotation does not rescale individual dimensions
— it spreads energy across all dimensions, reducing both the per-dimension maximum
and the outlier concentration. This is the core idea behind QuaRot and FlatQuant.

7.7 Rotation — The Core Idea

Why does rotation help?
An outlier in dimension i means |xi | ≫ |xj | for most j. After rotation by R ∈ Rd×d
(orthogonal: R⊤ R = I):
Xd
x̃ = Rx, x̃k = Rki xi
i=1
Each rotated dimension x̃k is a weighted combination of all original dimensions. The outlier
in xi is spread across all d rotated dimensions.

Key norms. Rotation preserves the ℓ2 norm (total energy is unchanged):

∥x̃∥2 = ∥Rx∥2 = ∥x∥2

But it reduces the ℓ∞ norm (maximum absolute value):

∥x̃∥∞ ≤ ∥x∥∞ (often much smaller in practice)

Since the clip range is determined by ∥ · ∥∞ , rotation directly tightens the quantization
clip range.

The product is preserved:

XW = (XR⊤ )(RW ) = X̃ W̃
Rotate the activations and the weights by inverse rotations — the output is unchanged.

43
7.8 Hadamard Rotation — Fast and Parameter-Free
The specific rotation used in QuaRot and FlatQuant.
The normalised Hadamard matrix Hd = √1 Hd where Hd is the d × d Walsh-Hadamard
d
matrix (entries ±1, orthogonal rows).

Why Hadamard?

Advantages
• Fast: O(d log d) via Fast Walsh-Hadamard Transform (vs O(d2 ) for general rotation)
• Fixed: no learned parameters, no calibration needed for R itself
• Uniform spreading: each output dimension receives equal contribution from all
inputs — theoretically optimal flattening
• Fused: can be computed as a single kernel fused with preceding LayerNorm

Limitations
• Requires d to be a power of 2 (padding needed otherwise)
• Fixed structure may leave some residual outlier patterns in pathological weight
matrices
• The inverse rotation R−1 = R⊤ must be applied to activations online — adds
O(d log d) per layer at inference

Random rotation is an alternative: draw R uniformly from the orthogonal group O(d). In
expectation this achieves the same flattening as Hadamard, and avoids pathological cases where
Hadamard leaves structure intact. Used in some ablation studies; Hadamard is preferred in
practice for its speed and reproducibility.

7.9 Runtime Cost of Rotation — What Is Free and What Is Not

Two types of rotation application:

Absorbed offline (free)

Apply R to the weight matrix:
W̃ = RW
This is done once before deployment. W̃ is quantized and stored. Zero runtime cost.
Applies to: Q, K, V, O projections; FFN up/gate/down weights.

Applied online (non-free)

Apply R−1 = R⊤ to activations:
x̃ = R⊤ x
This must happen at inference time, once per layer per token. Cost: O(d log d) per layer
(Hadamard transform).
In practice: fused with preceding LayerNorm — near-free relative to the linear layer
GEMM.

44
 Why full d × d rotation is acceptable here

For weight-activation quantization of linear layers, the online cost is O(d log d) per token.
At d = 4096 and Hadamard, this is ∼ 49 000 operations vs ∼ 33 000 000 for the linear
layer itself — about 0.15% overhead. Fusing with LayerNorm eliminates even this.
Compare with KV cache rotation in Section 8, where the situation is different.

7.10 QuaRot and FlatQuant

Both methods apply Hadamard-based rotation to enable W4A8 or W8A8 quantization of
transformer layers.

QuaRot (Ashkboos et al. FlatQuant (Sun et al.

2024) 2024)

Rotation type Random Hadamard (fixed) Learned Kronecker decom-

position
Applied to All linear projections All linear projections
Calibration for R None (fixed R) Yes — optimise R on cali-
bration set
Rotation overhead O(d log d) online, fused O(d log d) online, fused
with LN with LN
Target W4A16 and W4A8 W4A8 and W8A8
Key strength Simple, no optimisation Learns layer-specific rota-
needed tion, better accuracy
Key weakness Fixed R may miss layer- Calibration adds complex-
specific outlier structure ity; Kronecker factor stor-
age

FlatQuant’s Kronecker decomposition

FlatQuant parameterises the rotation as R = A ⊗ B (Kronecker product of two smaller

matrices). This reduces the parameter count of the learned rotation from O(d2 ) to O(d),
making calibration practical while still allowing layer-specific adaptation.

7.11 The Full W8A8 Pipeline with Rotation

Rotate W SmoothQuant Quantize W̃

offline
W̃ = RW scale s to INT8

LayerNorm Hadamard Quantize X̃ INT8 TC Rescale

online γ̃ = γ/s x̃ = R⊤ x to INT8 MMA to FP16

• Offline: rotate weights, apply SmoothQuant scaling, quantize to INT8 — all before
deployment, zero inference cost.
• Online: LayerNorm (with absorbed scale), Hadamard rotation (fused with LN), quantize
activations to INT8, INT8 Tensor Core MMA, rescale INT32 accumulator to FP16.

45
• Net effect: activations arrive at the quantization step with no dominant outlier dimensions
— INT8 uses its full 256 levels productively.

7.12 SmoothQuant vs. Rotation — When to Use Which

SmoothQuant QuaRot / FlatQuant

Outlier type handled Per-channel (fixed dimen- Any pattern (rotation

sions) spreads all)
Mechanism Scale: X̃i = Xi /si , Rotate: X̃ = R⊤ X, W̃ =
W̃i,: = si Wi,: RW
Calibration Calibrate s per layer QuaRot: none.
FlatQuant: calibrate R
per layer
Online cost None (absorbed into LN) O(d log d) Hadamard per
layer (fused, <1% over-
head)
Composable? Yes — apply before rota- Yes — apply after
tion SmoothQuant
Accuracy Good for standard outlier Better for complex /
pattern mixed patterns
Implementation Simple, widely supported More complex kernel;
needs Hadamard-fused
LN

Best practice: combine them

SmoothQuant and rotation are complementary. SmoothQuant handles the dominant
per-channel outlier efficiently (free); rotation handles the residual mixed-pattern outlier.
Applying both in sequence — SmoothQuant first, then QuaRot — gives better accuracy
than either alone, at the cost of adding the Hadamard online step.

7.13 Section Summary

Section 7 in five points

1. W8A8 is the correct strategy in the compute-bound regime (B ≳ B ∗ ), unlocking

INT8 Tensor Core arithmetic — but requires quantizing activations, which forces
confronting activation outliers.
2. SmoothQuant migrates outlier magnitude from activations to weights via a per-
channel scale s, absorbed offline into LayerNorm — zero runtime cost, handles
fixed-dimension outliers.
3. Rotation (QuaRot, FlatQuant) applies an orthogonal transform to spread outlier
energy uniformly across all dimensions, reducing both weights’ and activations’ ℓ∞
norm. Handles mixed-pattern outliers that SmoothQuant cannot.
4. Hadamard rotation is the practical choice: O(d log d) fast transform, parameter-
free, fuseable with LayerNorm for negligible online overhead.
5. SmoothQuant and rotation are composable: apply SmoothQuant first (free),
then rotation for residual patterns. The combined pipeline enables accurate W8A8 or
W4A8 quantization for large-batch, high-throughput serving.

46
Hook into Section 8
The question Section 7 leaves open
Sections 6 and 7 have covered the quantization of model weights and layer activations —
the two components of every linear layer.
But at long context lengths (S ≳ 8192 tokens), a third memory bottleneck emerges that
neither section addresses: the KV cache.
Recall from Part 1, Section 3: at S = 100 000 tokens, the KV cache requires ∼210 GB —
six times the 35 GB of INT4 model weights. Quantizing only the model weights is no
longer sufficient; the KV cache has become the dominant bandwidth sink.
Section 8 treats KV cache quantization as a distinct sub-problem: why it is harder than
weight quantization, how rotation applies differently here, and the specific algorithms —
PolarQuant, QJL, TurboQuant — designed for this setting.

8. KV Cache Quantization

At long context, the cache overtakes the weights — a distinct sub-problem with its own algorithms

8.1 The KV Cache Bandwidth Problem

Recall from Part 1, Section 3. ← Part 1, S3 – The Memory Bottleneck
The KV cache stores Keys and Values for all past tokens so attention can be computed
without re-running the full sequence:

KV cache size = 2 · L · H · dh · S · B · w
L = layers, H = heads, dh = head dim, S = sequence length, B = batch, w = bytes per
value.

Llama-3 70B (L=80, H=64, dh =128, BF16, batch 1):

Seq. length S KV cache INT4 weights Dominant term

2 048 4 GB 35 GB Weights
8 192 17 GB 35 GB Weights
32 768 67 GB 35 GB KV cache
100 000 210 GB 35 GB KV cache (6× weights)

The long-context regime

Beyond S ≈ 32 000 tokens the KV cache dominates memory. INT4 weight quantization
is no longer sufficient – the cache that was not quantized has become the bottleneck.
KV cache quantization is a distinct, necessary sub-problem for long-context
deployment.

47
8.2 Why KV Cache Quantization Is Harder

Weight quantization KV cache quantiza-

tion

When known Static – offline before de- Dynamic – new K/V com-
ployment puted every decode step
Clip range Calibrate once, stable Must generalise to all se-
quence lengths and all in-
put domains
Outlier pattern Per-channel, fixed Per-head, varies across
layers and positions
Sensitivity Rounding error in W af- Rounding error in K en-
fects output linearly ters softmax – exponen-
tial sensitivity
Scale storage Stored alongside weights Must be stored alongside
(offline) each KV entry (per-token
overhead)
FlashAttention Unaffected FlashAttention removed
attention-score HBM
cost; KV cache is remain-
ing bottleneck

√
The softmax sensitivity issue. Attention scores softmax(QK ⊤ / dh ) are exponen-
tially sensitive to Key values: a small shift in K can dramatically alter which tokens are
attended to. Key quantization is therefore harder than Value quantization – the two
often need different bit-widths or methods.

8.3 How FlashAttention Changes the Quantization Priority

Before FlashAttention. Naive attention wrote the full S × S score matrix to HBM
(quadratic in S). At S = 32 768, this was ∼4 GB per layer per head.

After FlashAttention. FlashAttention tiles the attention computation to keep the score
matrix in SRAM – it is never written to HBM. ← Part 1, S7 – Kernels & Fusion

FlashAttention eliminates the attention-score HBM cost. The remaining HBM

costs for attention are:
1. Reading the KV cache: every decode step reads all S past Keys and Values from
HBM. Cost grows linearly with S.
2. Writing new K/V entries: one new pair per layer per step, appended to the
cache.
Quantizing the KV cache directly reduces item 1 – the dominant long-context
bandwidth cost.

KV cache quantization became an active research area after FlashAttention was widely adopted:
once the score-matrix bottleneck was eliminated, the cache itself became the next target.

48
8.4 Granularity – Per-Head and Per-Token Scales
Why per-tensor is insufficient for KV. Key and Value distributions vary significantly
across heads, layers, and sequence positions. A single global scale gives too coarse a grid.

Granularity Scales stored per decode step Overhead

Per-tensor 1 per layer Negligible; accuracy poor

Per-head H per layer Small; good accuracy
Per-token H × L per step Moderate; best accuracy
Per-channel H × dh (offline) Moderate; stable

Scale storage overhead at long context

At per-token per-head granularity, each new token appends H × L FP16 scales to the
cache. For Llama-3 70B: 64×80 = 5 120 FP16 values = 10 KB per token. At S = 100 000:
∼1 GB of scales vs ∼210 GB of quantized KV data – less than 0.5% overhead.

8.5 Rotation for KV Cache – Why Full-d Rotation Is Acceptable

Recall from Section 7: for linear-layer weight-activation quantization, the online rotation
cost is O(d log d) per token – small but real.

For KV cache, the cost structure is different.

KV cache memory scales with the number of tokens S, not with the head dimension dh . The
rotation matrix R ∈ Rdh ×dh is a fixed overhead – stored once per head, not per token:

Rotation storage = H × d2h × 4 B = 64 × 1282 × 4 ≈ 4 MB

vs. KV cache at S = 100 000 : 210 GB

Rotation matrix storage is ∼0.002% of the KV cache at long context.

Consequence. For KV cache quantization, a full dh × dh learned rotation (even non-

Hadamard) incurs negligible overhead because the rotation matrix does not grow with
sequence length. KV-specific rotation methods can afford more flexible, accurate rotation
designs than the linear-layer methods in Section 7.

8.6 PolarQuant – Rotation-Based KV Quantization

PolarQuant applies a per-head orthogonal rotation to Key and Value tensors before quanti-
zation, designed to flatten the KV distribution.

Advantages in KV setting
• Full dh × dh rotation – more flexible than Hadamard
• Learned per-head R adapts to each head’s specific K/V distribution
• K and V can use independent rotations
• Storage overhead: negligible vs. cache at long context

49
 Challenges
• Online cost: O(d2h ) per token per head when computing new K/V entries
• Query Q must also be rotated to preserve QK ⊤ : (QR)(RK)⊤ = QK ⊤
• Learned R requires calibration optimisation
• Rotation before quantization adds a kernel step to the attention prefill path

The Q–K consistency constraint

Rotating Keys by R requires rotating Queries by R as well: (QR)(RK)⊤ = QRR⊤ K ⊤ =

QK ⊤ . The rotation must be applied before the attention score computation – fuseable
with the Q and K projection at negligible cost.

8.7 QJL – Quantized Johnson-Lindenstrauss Transform

QJL (Zandieh et al. 2024) uses a randomised sketch that approximately preserves the
inner products needed for attention, enabling aggressive 1-bit compression of Keys.

Johnson-Lindenstrauss property. For random Φ ∈ Rk×dh with i.i.d. N (0, 1/k)

entries, for any q, k ∈ Rdh :

Pr |⟨Φq, Φk⟩ − ⟨q, k⟩| ≥ ε∥q∥2 ∥k∥2 ≤ 2 exp(−kε2 /4)

 

With k ≪ dh , inner products are preserved with high probability.

QJL procedure:

1. Project Keys: K̃t = ΦKt ∈ Rk (compress from dh to k, e.g. k = dh /4)

2. Quantize projected Keys to 1 bit per dimension
3. At attention: compute approximate scores from the 1-bit sketch
4. Use full INT8 Values for the weighted sum

Compression. k dimensions at 1 bit ≈ k/dh × 1/16 of BF16 storage. With k = dh /4:

roughly 0.25 bits per original dimension – 64× compression of Keys relative to BF16.
Inner product error is bounded by the JL guarantee.

8.8 TurboQuant – Codebook Quantization for KV

TurboQuant applies a learned codebook (non-uniform quantization, Section 2) to KV cache
values, exploiting the structure of the attention access pattern.

Why codebook is more acceptable for KV than for weights.

Recall Section 2: codebook dequantization uses a LUT lookup, which disrupts the fused
dequant-GEMM pipeline in weight kernels.
For KV attention, memory access is already irregular: each decode step reads the entire cache
with different patterns per head. The LUT overhead is relatively less disruptive in this
irregular-access regime.

50
 Procedure. Learn codebook C = {c1 , . . . , c2b } per head from calibration data. At each step:

kq = arg min |k − cj |2 , k̂ = C[kq ]

j

Store the b-bit index; dequantize via LUT at attention time.

Accuracy benefit. The codebook concentrates quantization levels where the K/V
distribution is dense (near zero), reducing expected quantization error vs. uniform INT4
at the same 4 bits. Same benefit as Section 2, now applied where the LUT overhead is
tolerable.

8.9 KVQuant and KIVI – Production-Oriented Methods

KVQuant (Hooper et al. 2024).

• Per-channel non-uniform quantization (NF4-style levels) fitted to the KV distribution,

plus a small set of full-precision residuals for outlier dimensions.
• Achieves INT2–INT4 with near-lossless perplexity at context lengths up to 100K tokens.
• Compatible with FlashAttention via a quantization-aware attention kernel.

KIVI (Liu et al. 2024).

• Keys and Values have different outlier structures: Key outliers are primarily along the
channel dimension; Value outliers are primarily along the token dimension.
• Applies per-channel INT2 to Keys, per-token INT2 to Values.
• A small sliding-window “residual cache” keeps recent tokens in full precision; older tokens
are quantized.
• Achieves 2× memory reduction at INT2 with less than 1% degradation on most bench-
marks.
Key vs. Value asymmetry

Keys enter softmax (exponential sensitivity to channel-wise errors); Values are multiplied
by attention weights (linear sensitivity, tolerates per-token quantization). KIVI exploits
this asymmetry explicitly.

51
8.10 The KV Quantization Landscape

Method Mechanism Bit-width Best for

Uniform INT8 per-head Per-head scale 8-bit K+V S ≤ 8K; simple,

fast
KIVI Per-ch K / per- INT2 K+V S ≤ 32K; mini-
tok V mal loss
KVQuant Non-uniform + INT2–4 S ≤ 128K; high
residuals accuracy
PolarQuant Learned rota- INT4 Medium-long;
tion per head mixed outliers
TurboQuant Codebook per INT4 High accuracy
head where LUT OK
QJL 1-bit JL sketch ∼0.25b K + Extreme context
of Keys INT8 V S ≥ 100K

Choosing by context length

1. S ≤ 8 000: per-head INT8; KV cache is not dominant.
2. 8 000 < S ≤ 32 000: KIVI (INT2) or per-head INT4.
3. 32 000 < S ≤ 128 000: KVQuant or PolarQuant.
4. S > 128 000: QJL for Keys; INT4–8 for Values.

8.11 Section Summary

Section 8 in five points

1. At long context (S ≳ 32 000), the KV cache overtakes model weights as the

dominant memory and bandwidth sink; weight quantization alone is insufficient.
2. KV quantization is harder than weight quantization: dynamic, exponentially
sensitive through softmax (Keys), requires per-token scale storage.
3. Rotation is more flexible for KV than for linear layers: the rotation matrix
is a fixed dh × dh overhead, negligible at long context where the cache scales with S.
4. Key/Value asymmetry matters: Keys are channel-sensitive (softmax), Values
are token-sensitive (linear weighted sum). KIVI and KVQuant exploit this for INT2
quantization.
5. Method choice depends on context length: INT8 for short, INT4/KIVI for
medium, KVQuant/QJL for extreme. No single method dominates all regimes.

52
Hook into Section 9
The question Section 8 leaves open

Sections 6, 7, and 8 cover the three quantization targets: model weights (W4A16), layer
activations (W8A8), and the KV cache.
Each method minimises reconstruction error ∥X ⊤ (W − Ŵ )∥2F or distribution mismatch.
But none explicitly controls the first-order moment:

E (Wq − W )X = µ⊤ (Wq − W ) ̸= 0
 

This residual systematic bias shifts the expected output of every quantized layer –
distorting softmax calibration, LayerNorm statistics, and compounding across layers.
Section 9 introduces post-quantization correction: methods that treat the residual bias
as a discrete selection problem and fix it with minimal overhead. The centrepiece is
EGBC – positioned alongside AdaRound and BRECQ in the correction landscape.

9. Post-Quantization Correction

Fixing what rounding leaves behind — from bias absorption to discrete smart flipping

9.1 The Residual Bias Problem

What all previous methods leave behind.
GPTQ minimises ∥X ⊤ (W −Ŵ )∥2F . AWQ flattens the distribution before rounding. SmoothQuant
and rotation reduce activation outliers. Yet after any of these methods, a systematic error
remains:

E yq − y = E (Wq − W )X = E[X]⊤ (Wq − W ) = µ⊤ ∆W ̸= 0

   

where µ = E[X] ∈ Rd and ∆W = Wq − W is the quantization error matrix.

Why this matters.

• Softmax calibration: the attention logit distribution shifts by µ⊤ ∆W — attention

weights change even before any token-specific variation.
• LayerNorm drift: the mean of the LayerNorm input changes, shifting the normalised
distribution and the effective scale γ.
• Compounding: each layer’s bias adds to the next. At 32–80 layers, even a small per-layer
shift accumulates into a significant output distribution mismatch.

Key insight. GPTQ controls the variance of the output error (second-order moment).
Correction methods control the mean of the output error (first-order moment). Both
are needed for a well-calibrated quantized model.

9.2 Why µ = E[X] ̸= 0 in LLMs

The bias term µ⊤ ∆W is non-trivial only when µ =
̸ 0. In LLMs, this is almost always the
case.

53
Sources of non-zero activation mean:

• LayerNorm with learned bias: γ x̂ + β — the additive offset β shifts the distribution
away from zero.
• Residual stream accumulation: the residual connection adds each layer’s output to a
running sum; the sum develops a non-zero mean over depth.
• Activation functions: SiLU and GeLU are not zero-centred — their outputs have a
positive mean for typical inputs.
• Outlier dimensions: the structural outlier dimensions identified in Section 5 have
persistently large |µi |, contributing the most to µ⊤ ∆W .

Implication for bias correction

The dominant contribution to µ⊤ ∆W comes from the small number of dimensions i

where |µi | is large — the same outlier dimensions that AWQ and SmoothQuant target.
Correction methods that identify and selectively adjust weights at these dimensions get
the most benefit per operation. This is precisely what EGBC’s smart flipping exploits.

9.3 Bias Absorption — Nagel et al. (2019)

The first bias correction approach. Compute the mean quantization error per output
channel and absorb it into the layer’s explicit bias term.

bcorrected
j = bj + µ⊤ (wj − ŵj ) = bj − µ⊤ ∆wj
where bj is the bias for output channel j and ∆wj = ŵj − wj is the quantization error
in column j. After correction: E[yq,j ] = E[yj ] for each channel.

Strengths
• Closed-form — no optimisation
• Negligible compute cost
• Exact first-moment alignment per output channel
• Works for any quantization scheme as a post-step

Critical limitation
• Requires an explicit bias parameter bj in the layer
• Modern LLMs (LLaMA, Mistral, Phi, Falcon) are bias-less by design — no bias in
MLP or attention projections
• Adding a new bias parameter breaks optimised kernels and architectural symmetry
• “Hides” the error rather than correcting the weights

9.4 AdaRound — Learned Rounding

AdaRound (Nagel et al. 2020) reframes rounding as a learning problem: for each weight,
should we use ⌊w/s⌋ (floor) or ⌈w/s⌉ (ceiling)? NTR always picks the closer one — but the
globally better choice may sometimes be the farther one.

54
 Optimisation objective (per layer):
2
X
min X ⊤ (W − Ŵδ ) F
+λ h(δi )
δ∈[0,1]n
i

where Ŵδ = s (⌊W/s⌋ + δ), δi ∈ [0, 1] is a continuous relaxation of the binary rounding
decision (δi = 0: floor, δi = 1: ceil), and h(δi ) is a regulariser that drives δi toward 0 or
1 (binary).

Training procedure. Initialise δi = ⌈wi /s⌉ − wi /s (distance to ceiling). Minimise via SGD
with STE through the binary projection. Anneal λ to force binary decisions at convergence.

What AdaRound achieves. By optimising the rounding decisions jointly over the
calibration set, AdaRound can choose a pattern of floors and ceilings that minimises the
layer output error rather than the individual weight errors. This is a stronger objective
than NTR and produces significantly better accuracy at 4-bit and 3-bit.

9.5 AdaRound — Strengths and Weaknesses

Strengths
• Directly optimises layer output error — the right objective
• Works for bias-less architectures: no new parameters, just different rounding decisions
• Strong accuracy at 4-bit and especially 3-bit
• Generalises beyond NTR: any rounding pattern is reachable

Weaknesses
• SGD per layer: thousands of gradient steps for each linear layer
• For a 70B model with ∼800 linear layers: days of compute on 8×A100
• Impractical for rapid deployment cycles
• Optimises reconstruction error (second moment) — does not explicitly target the
bias (first moment)

Practical note. AdaRound is widely used for small models (CNNs, 1–7B transformers) where
per-layer SGD takes minutes not days. For large LLMs (≥13B), it is typically replaced by GPTQ
(which achieves similar quality via the Hessian approach without SGD) or EGBC (which corrects
the bias at negligible cost).

9.6 BRECQ — Block-Wise Reconstruction

BRECQ (Li et al. 2021) extends AdaRound by optimising rounding decisions across
multiple layers jointly.

55
 Block reconstruction objective. Instead of minimising per-layer output error:
2
min fblock (X; W ) − fblock (X; Ŵδ ) F
δℓ

where fblock is the output of a contiguous block of k layers (e.g. one transformer block:
attention + FFN).

Why block-wise matters. Per-layer correction (AdaRound) ignores how quantization error
in layer ℓ propagates to layer ℓ + 1. Block-wise correction allows rounding decisions in earlier
layers to partially compensate for errors they introduce in later layers.

Advantage over AdaRound

• Captures cross-layer interactions
• Better accuracy at 2–3-bit where layer-local correction is insufficient

Disadvantage
• Even heavier compute than AdaRound: larger optimisation problem per block
• Completely impractical for 70B+ models without massive compute

9.7 EGBC — Motivation

The gap in the correction landscape.
AdaRound and BRECQ achieve excellent accuracy by optimising rounding via SGD — but
require thousands of gradient steps per layer. Bias absorption is free — but inapplicable to
bias-less architectures.

The open question

Can we correct the systematic bias µ⊤ (Wq − W ) in bias-less LLM architectures without
gradient computation, without new parameters, and with negligible computational cost?

The EGBC insight. For any quantized weight ŵij , there are exactly two candidate discrete
levels: ⌊wij /s⌋ (floor) and ⌈wij /s⌉ (ceiling). NTR always picks the closer one. But a strategic
flip to the adjacent level — guided by the activation mean µ — can reduce the per-channel
bias µ⊤ (Wq − W ) without solving any continuous optimisation problem.

EGBC treats bias correction as a discrete selection problem, not a continuous

optimisation task. This is the key design choice that makes it compatible with bias-less
architectures and achieves negligible runtime cost.

9.8 EGBC — The Bias-Oriented Objective

Per-channel decomposition.
The first-moment alignment objective decomposes per output channel j:

56
 C
2
2
X
⊤
Lbias (Wq ) = µ (Wq − W ) 2
= µ⊤ (wq,j − wj )
j=1
| {z }
b2j

where bj := µ⊤ (wq,j − wj ) is the per-channel bias for output channel j.

Each channel is independent — the loss decomposes into C independent per-channel scalar
problems.

Effect of a single flip. Flipping weight wq,ij from its current level to the adjacent level
changes wq,ij by ±sj (the per-channel step size). This changes bj by:

bj −→ bj ± µi sj
∆Lbias,j = (bj ± µi sj )2 − b2j = ±2bj µi sj + µ2i s2j

Choosing the sign that reduces |bj | gives ∆Lbias,j = −2|bj ||µi |sj + µ2i s2j , which is negative
(beneficial) precisely when 2|bj | > |µi |sj .

9.9 EGBC — Which Weights Should Be Flipped?

The flip condition from Section 9.8: flipping wq,ij reduces the bias if and only if
2|bj | > |µi |sj .
This reveals a non-monotone relationship between |µi | and flip benefit:

Regime Flip effect Decision

|µi | ≈ 0 (near-zero mean) ∆bj ≈ 0 — negligible correction Skip: no benefit

|µi | moderate Discrete step µi sj meaningfully reduces |bj | Flip if condition holds
|µi | very large (outlier) Step µi sj overshoots zero, increases b2j Skip: destructive

EGBC implements two filtering mechanisms:

1. Knee-point masking: sort dimensions by |µi | descending; detect the “elbow” in the
magnitude profile via a maximum-distance-to-chord criterion. Dimensions above the knee
(outliers) are excluded from flipping.
2. Budget constraint: at most p% of eligible weights per output channel are flipped
(default p = 1%), limiting local quantization distortion.

Why masking outlier dimensions

An outlier dimension |µi | ≫ 2|bj |/sj produces a flip step that overshoots the target,
worsening the bias. The knee-point detects where “high magnitude” ends and “moderate
magnitude” begins — the effective flipping regime.

9.10 EGBC — The Greedy Subset-Sum Algorithm

Per-channel problem formulation.
For output channel j, let Ij be the eligible index set (after knee-point masking and sign
filtering). Each eligible weight i ∈ Ij contributes a signed correction vj,i = µi ∆wq,j,i when
flipped. We want to select a subset S ⊆ Ij such that:

57
 X
min bj − vj,i s.t. |S| ≤ Bj
S⊆Ij
i∈S

This is an instance of the approximate Subset-Sum problem.

EGBC’s greedy solution. Rather than searching all subsets (exponential), restrict to prefix
subsets ordered by distance to the rounding boundary (weights closer to the midpoint
between floor and ceiling are cheaper to flip — smaller local distortion):

EGBC greedy flip (per channel j)

Require: eligible set Ij , bias bj , corrections {vj,i }, budget Bj

1: Sort Ij by rounding deviation (descending)
2: best ← |bj |; k ∗ ← 0; running ← 0
3: for k = 1 to min(|Ij |, Bj ) do
4: running ← running + vj,ik
5: if |bj − running| < best then
6: best ← |bj − running|; k ∗ ← k
7: end if
8: end for
9: Flip weights {wq,j,i1 , . . . , wq,j,ik∗ }
Ensure: Updated Wq with k ∗ flips in channel j

Cost: O(|Ij | log |Ij |) sort + O(|Ij |) scan per channel. Total: negligible relative to the
preceding quantization step.

9.11 EGBC — Robust Mean Estimation via James-Stein Shrinkage

The estimation problem.
EGBC’s bias objective and knee-point masking both depend on µ = E[X] ∈ Rd . In practice,
µ is estimated from a small calibration set (m = 128 samples) using the sample mean x̄.

Why the sample mean is problematic. When d is large (4096, 8192) and m is small
(128), x̄ is noisy:

• Noisy extreme values incorrectly label many dimensions as “outliers” in the knee-point
detection, causing the masking to exclude good flipping candidates.
• Noisy µi values give wrong flip signs, worsening rather than improving bj .

EGBC uses James-Stein shrinkage:

µ̂JS = x̄g 1 + (1 − c)(x̄ − x̄g 1)

(d − 2)σ̂ 2
c= , c ← clip(c, 0, 1)
∥x̄ − x̄g 1∥22
1 P
where x̄g = d i x̄i is the grand mean.

For d ≥ 3, James-Stein shrinkage is known to reduce mean-squared estimation error vs. the
sample mean — shrinking noisy extreme coordinates toward a common centre, giving more
reliable flip decisions under limited calibration data.

58
9.12 EGBC — Position in the Correction Landscape

Method Objective Solve Cost Bias-less

method arch

Nagel 2019 (absorption) First moment Closed form Negligible No

AdaRound Second mo- SGD per High Yes
ment (recon.) layer
BRECQ Second mo- SGD per Very high Yes
ment (block) block
EGBC First mo- Greedy Negligible Yes
ment

EGBC’s unique position

EGBC is the only method that:
1. Targets the first-order moment (bias) explicitly
2. Requires no gradient computation
3. Is compatible with bias-less architectures (LLaMA, Mistral, Phi, Falcon)
4. Has negligible computational cost (O(d log d) per channel)
It is designed as a post-processing step after GPTQ or AWQ: those methods
minimise the second moment; EGBC then corrects the residual first-moment bias they
leave behind.

9.13 EGBC — The Complete Correction Pipeline

EGBC post-processing

Calibration Clip range AWQ or GPTQ Knee-point

James-Stein
forward pass calibration quantization masking
estimate µ̂
collect X MSE / KL get Wq get I

Greedy flip
Deploy
per channel
INT4 model
update Wq

• Blue stages: standard PTQ pipeline (calibration, clip range, AWQ/GPTQ)

• Red stages: EGBC post-processing (estimate µ with shrinkage, mask outlier dims, greedy
flip to cancel bj )
• Output: same INT4 format as before EGBC — no new parameters, no inference graph
change, no runtime overhead

Composability. EGBC is a drop-in post-step. The same INT4 quantized weights, same
per-group scales — only selected integer values are changed by ±1. All existing fused
dequant-GEMM kernels work unchanged.

59
9.14 Section Summary
Section 9 in five points

1. The residual bias µ⊤ (Wq −W ) ̸= 0 is left by all reconstruction-based PTQ methods

(GPTQ, AWQ, SmoothQuant). It shifts softmax calibration, LayerNorm statistics,
and compounds across layers.
2. Bias absorption (Nagel 2019) fixes this in closed form but requires explicit bias
parameters — inapplicable to modern bias-less LLMs.
3. AdaRound learns optimal rounding decisions via SGD, achieving strong accuracy
at 3–4-bit, but is computationally prohibitive for 70B+ models.
4. BRECQ extends AdaRound to block-wise optimisation, capturing cross-layer inter-
actions at even higher compute cost.
5. EGBC corrects the first-moment bias by selectively flipping quantized weights via a
greedy subset-sum algorithm, with negligible cost, no gradients, and full compatibility
with bias-less architectures. It is designed as a complementary post-step to GPTQ
or AWQ.

Hook into Section 10

The question Section 9 leaves open
We have now covered the complete algorithm toolkit:
• S6: GPTQ and AWQ for weight-only W4A16
• S7: SmoothQuant and rotation for W8A8
• S8: PolarQuant, QJL, TurboQuant for KV cache
• S9: AdaRound, BRECQ, EGBC for bias correction

The practical challenge: given a specific hardware target and deployment scenario, which
combination of these methods should be used? The answer requires mapping hardware
constraints (from Part 1) to algorithm choices (from Part 2).
Section 10 closes the loop: an Algorithm × Hardware matrix, a decision flowchart
companion to Part 1’s synthesis, and the three algorithm-design principles that mirror
Part 1’s three hardware constraints.

10. Synthesis — Algorithm × Hardware

Closing the loop: every algorithm choice is a hardware consequence

60
10.1 What Part 2 Has Built
The journey so far

S1–2 What quantization is: uniform grid, codebook, floating-

point non-uniformity
S3 How to place the grid: clip range, calibration, MSE vs.
KL vs. percentile
S4 When to quantize: QAT (gradients, full data) vs. PTQ
(no gradients, 128 samples)
S5 The enemy: outliers – structural, persistent, 100× typical
in activations
S6 Weight-only (W4A16): GPTQ (correct after), AWQ (flat-
ten before)
S7 Weight-activation (W8A8): SmoothQuant (migrate),
QuaRot/FlatQuant (rotate)
S8 KV cache: KIVI, KVQuant, PolarQuant, QJL, Turbo-
Quant
S9 Post-correction: AdaRound, BRECQ, EGBC (first-
moment bias)

The unifying question of Section 10: given a target device, deployment scenario,
and context length – which combination of algorithms should you use? The answer
comes directly from the hardware constraints established in Part 1.

10.2 Three Algorithm Design Principles

Every PTQ algorithm in Sections 6–9 instantiates one or more of three principles. These
mirror the three hardware constraints from Part 1 Section 8.

The three levers

1. Flatten the distribution before rounding.
Reduce outlier magnitude so the uniform grid fits better.
Methods: AWQ, SmoothQuant, QuaRot, FlatQuant, PolarQuant.

2. Correct the error after rounding.

Fix what rounding got wrong – second-order or first-order.
Methods: GPTQ (Hessian), AdaRound (learned floor/ceil), EGBC (first-moment
greedy flip).

3. Match quantization granularity to the hardware datapath.

Per-group on GPU (free in registers); per-channel on NPU (systolic array constraint);
per-head for KV (attention kernel access pattern).
This is not an algorithm – it is a deployment constraint that every algorithm must
respect.

Most production pipelines combine all three: flatten first (AWQ or SmoothQuant), correct second
(GPTQ or EGBC), and match granularity to the target device throughout.

61
10.3 Algorithm × Hardware Matrix

Online Offline batch Edge NPU Long context

serving
W4A16, GPU W8A8, GPU per-ch KV cache
INT8/INT4

Flatten AWQ SmoothQuant per-channel PolarQuant

scaling
Flatten (rotation) optional QuaRot/FlatQuant — PolarQuant
FlatQuant
Correct GPTQ + GPTQ + AdaRound —
EGBC EGBC
Codebook QuIP# — — TurboQuant
(accuracy cost)
Ultra-low bit — — BitNet (custom QJL (∼0.25b
HW) K)
KV-specific — — — KIVI,
KVQuant

Read each column top-to-bottom as the recommended pipeline for that regime. Rows are not
mutually exclusive – flatten + correct is the standard combination. Colour coding: blue =
GPTQ family, green = AWQ family, purple = rotation/migration, orange = KV methods,
red = correction.

10.4 Decision Flowchart

Model fits No Reduce w until

in memory? N · w ≤ device mem

Yes

B < B∗?
Yes (mem-bound) No (comp-bound)

W4A16 W8A8 / FP8

AWQ + GPTQ + EGBC SmoothQuant + QuaRot

GPU or NPU?
GPU NPU

Per-group INT4 Per-channel INT8

(AWQ/GPTQ style) (or INT4 if native)

S > 32K?
No Yes

KV cache: INT8 Add KV quant:

per-head sufficient KIVI / KVQuant / QJL

62
10.5 Reading the Flowchart
Gate 1 – Does the model fit? If not, quantization is mandatory and the minimum
bit-width that achieves the required footprint must be chosen first. No accuracy argument
overrides a hard memory constraint.

Gate 2 – Is batch size below B ∗ ? B ∗ = Ppeak /bs . Below B ∗ : memory-bound, W4A16.

Above B ∗ : compute-bound, W8A8 or FP8. ← Part 1, S5 – Roofline Model

Gate 3 – GPU or NPU? GPU: per-group INT4 (scales applied in registers, free). NPU:
per-channel INT8 or INT4 if natively supported; per-group scales disrupt the systolic array
pipeline. ← Part 1, S6 – Compute Architecture

Gate 4 – Is context length > 32K? Yes: add a KV cache quantization method from
Section 8. No: per-head INT8 for KV is sufficient at short context.

After the flowchart. The flowchart identifies the type of quantization required. Within
that type, the algorithm choice (GPTQ vs. AWQ vs. SmoothQuant) is determined by
accuracy requirements, calibration budget, and composability preferences – all covered
in Sections 6–9.

10.6 The Standard Production Pipeline (GPU, W4A16)

For most LLM deployments today (online serving, GPU, B ≤ 16, S ≤ 8K):

scales Wq Wq
AWQ s GPTQ corrected EGBC debiased Deploy
flatten saliency Hessian correction bias correction INT4 kernel

128 calibration
samples

Step What it fixes Cost Output

AWQ Intra-group weight outliers Minutes Scaled W

GPTQ Reconstruction error (2nd order) Hours Corrected Wq
EGBC First-moment output bias Seconds Debiased Wq
Deploy — — INT4 W4A16 kernel

10.7 The Standard Pipeline for W8A8 (Offline / Large Batch)

For high-throughput offline processing (GPU, B ≥ 256, compute-bound regime):

s absorbed rotated INT8

SmoothQuant into LN Wq
QuaRot W̃ GPTQ Deploy
migrate outliers Hadamard rotate correct W INT8 TC kernel

63
 Why EGBC is less critical in W8A8

EGBC corrects the first-moment bias µ⊤ (Wq − W ). In W8A8 at INT8 (256 levels), the
step size s is much finer than at INT4 (16 levels), so the absolute magnitude of the
residual bias is smaller. EGBC still helps but the benefit is smaller than at INT4.
SmoothQuant + rotation already do most of the work.

10.8 Connecting Back to Part 1

Part 1 derived three hardware constraints. Part 2 algorithms are the responses:

Part 1 constraint Part 2 algorithm response

Bandwidth determines INT4 weights (W4A16) via AWQ

bit-width. Reduce bytes + GPTQ. INT2 KV via KIVI /
transferred to hit the token- KVQuant.
rate ceiling.

Batch size determines W4A16 (memory) vs W8A8 (com-

which roof. Memory- pute). The crossover at B ∗ =
bound at small B; compute- P/bs determines the algorithm
bound at large B. family.

Native datapath deter- Per-group INT4 on GPU (AWQ,

mines granularity. GPU: GPTQ). Per-channel INT8 on
per-group free. NPU: per- NPU. Per-head scales for KV
group costly. cache.

The two parts are orthogonal axes. Part 1 identifies the type of quantization the
hardware requires. Part 2 identifies the algorithms that minimise accuracy loss within
that type. Both must be right for a deployment to succeed. Getting the hardware axis
wrong (wrong bit-width, wrong granularity) wastes all the accuracy work done by the
algorithm axis.

64
10.9 Common Mistakes — and How to Avoid Them

Mistake Symptom Fix

Using W8A8 for online Slower than FP16 Switch to W4A16;

serving (memory-bound, despite quantiza- check roofline
B ≤ 16) tion; no arithmetic regime
speedup

Per-tensor clip range for One outlier destroys Always use per-
LLMs resolution for 99% channel minimum;
of weights per-group pre-
ferred

Min-max clip range Coarse step size; ex- Use MSE minimi-
cessive rounding er- sation; allow delib-
ror for bulk values erate clipping

Applying AWQ with- Per-group scales Use per-channel

out checking group-scale incur dequantize on NPU; verify
NPU support round-trip on NPU runtime support

Ignoring KV cache at Memory OOM or Add KIVI /

long context bandwidth ceiling KVQuant for
hit despite INT4 S > 32 000
weights

Calibration set too small GPTQ Hessian Use ≥128 diverse

/ mismatched overfits; clip range samples; domain-
wrong for inference matched data

10.10 Forward Pointer — What Part 3 Would Cover

Parts 1 and 2 together answer: how to produce a quantized model that is accurate and fast.

Open questions for a Part 3:

• Quantization-aware serving. Speculative decoding with a quantized draft model +

full-precision verifier. Mixed-precision routing: which layers tolerate INT4 vs. which need
INT8 per input?
• Dynamic quantization at inference. Per-token activation clip ranges computed online;
adaptive bit-width allocation based on input difficulty.
• Quantization + sparsity. Combining weight pruning (structured or unstructured) with
INT4 quantization – SparseGPT + GPTQ interactions.
• Emerging hardware. Native FP4 on Blackwell; INT4 on next-generation NPUs; how
the roofline analysis and algorithm choices change.
• Evaluation beyond perplexity. Downstream task accuracy, calibration under quanti-
zation, robustness to domain shift.

These topics build directly on the hardware model (Part 1) and algorithm landscape (Part 2)
established here. The mental model you have built – roofline, three levers, flatten/correct/match
– transfers directly.

65
10.11 Part 2 in Ten Sentences
The complete picture
1. Uniform quantization maps floats to integers via a scale and zero-point; per-group granularity
gives the best accuracy-efficiency trade-off on GPU.
2. Non-uniform (codebook) quantization concentrates bins where data is dense; better accuracy
per bit but requires LUT dequantization.
3. The clip range must be calibrated before any algorithm runs; MSE minimisation on 128
calibration samples is the standard.
4. QAT (with STE) trains through quantization for best accuracy; PTQ applies quantization
post-hoc with no gradients.
5. Outliers – structural, persistent, up to 100× typical in activations – are the central obstacle
to accurate quantization.
6. W4A16: AWQ flattens weight saliency; GPTQ corrects reconstruction error; EGBC removes
residual bias.
7. W8A8: SmoothQuant migrates activation outliers to weights; QuaRot/FlatQuant spreads
them via Hadamard rotation.
8. At long context (S > 32K), the KV cache dominates; KIVI, KVQuant, PolarQuant, and
QJL address this distinct sub-problem.
9. Post-quantization correction (AdaRound, EGBC) fixes the first-moment bias µ⊤ (Wq − W )
left by all rounding methods.
10. Hardware determines the design space; algorithms populate it. Both axes must be right.

Thank You

Quantization for LLMs[0.4em] Part 2: Algorithms[1.5em]

AnhND + Claude[2.5em]

The three levers.[0.5em] Flatten the distribution before rounding. Correct the error
after rounding. Match granularity to the hardware datapath.

Part 1: Hardware Constraints • Part 2: Algorithms • Part 3: (forthcoming)

66