Efficient Inference for Large Language Models

Algorithm, Model, and System

Xuefei Ning1 Guohao Dai2,4 Haoli Bai3 Lu Hou3 Yu Wang1 Qun Liu3

1TsinghuaUniversity 2Shanghai Jiao Tong University 3Huawei 4Infinigence-AI

foxdoraame@[Link], daiguohao@[Link], baihaoli@[Link],
houlu3@[Link], yu-wang@[Link], [Link]@[Link]

Tutorial Website: [Link]

 Contents
Model-Level
1 Background 5 Optimization

2 Preliminary System-Level
≈
6 Optimization

3 Problem Definition & Algo-Level

Conceptual Analysis 7 Optimization

4 Practical Pipeline 8 Conclusion

 Contents
Model-Level
1 Background 5
(Xuefei Ning)
Optimization
(Lu Hou)

2 Preliminary System-Level
(Xuefei Ning)
≈
6 Optimization
(Guohao Dai)
Problem Definition &
3 Algo-Level
Conceptual Analysis
(Xuefei Ning)
7 Optimization
(Haoli Bai)

4 Practical Pipeline
(Xuefei Ning)
8 Conclusion
(Haoli Bai)
 Contents
Model-Level
1 Background 5 Optimization

2 Preliminary System-Level
≈
6 Optimization

3 Problem Definition & Algo-Level

Conceptual Analysis 7 Optimization

4 Practical Pipeline 8 Conclusion

 Background
Towards general and generative intelligence, scaling up model / data /
computation based on Transformer is a mainstream and effective pathway.

Focus Shift 1: Discriminative => Generative Current mainstream path

Based on the Transformer
architecture, scaling model size ,
training data / computation, and
test computation.

Human AI World

Model
Performance

Focus Shift 2: Specialized => General

Data / Model size / Computation
 Background
The model scale and input/output length in generative intelligence
research and applications have increased significantly.

2018 - 2025 2025

1e13 5 orders of magnitude increase
in parameter size [1]

Supported Input Length

Parameter Size

2018-2024
2018 3 orders of
1e8 magnitude increase in the
supported input length

Example:
DeepSeek-R1 [2] (year 2025): 671B params, open source.
1e2
Release Time

[1] Villalobos et al. “Machine Learning Model Sizes and the Parameter Gap.” arXiv 2022. [4] Achiam, Josh, et al. "Gpt-4 technical report." arXiv 2023.
[2] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement [Link] 2025. [5] Reid, Machel, et al. "Gemini 1.5: Unlocking multimodal
[3] Esser, Patrick et al., Scaling rectified flow transformers for high-resolution image synthesis, ICML 2024. understanding across millions of tokens of context." arXiv 2024.
 Background
Application demands e.g., multimodal input, advanced tasks, may
continually drive increases in model scale and input/output length.

Multimodal Input Advanced Tasks

Modeling multimodal data may push the scaling Agentic pipelines for broad applications require
saturation point of model size higher. stronger models and longer input/output contexts.
Multi-Agent Collaboration
~100B ~1000B ? Memory

texts texts, images, videos, audios, … Single- Planning

Agent
Supporting high-resolution images and longer videos Action
requires extended input/output contexts.
Test-time compute scaling (especially CoT) for
reasoning require longer input/output contexts.
 Application Challenge
This scaling of the model size and input/output poses challenges for
efficient inference across platforms and application scenarios.

Model size, input/output length Inference cost (time, storage, energy)

Human AI World

Deployment

Cloud Edge
Requirements or constraints of application scenarios and platforms

High Low Small Low Energy

Throughput Latency Storage Consumption
 Application Challenge
Time to first token (TTFT) and time per output token (TPOT) w.r.t. model size
and input/output lengths, estimated with three costs (compute, memory
access and memory footprint) of the model spec and device spec.
Different #Prefill Tokens using Llama-3.1-8B (BS=1) Different Models with 4k Input Tokens (BS=1)
Compute(GFLOPS) MemoryAccess(GB) MemoryNeed(GB) Compute(GFLOPS) MemoryAccess(GB) MemoryNeed(GB)
37465.4x,
100000 596PFLOPS 1000 281.7x,
Normalized to 1k Input

Normalized to 1B Model
10000 533.5x, 3.5PFLOPS 196.7x,
1000 15.6TB 1.6TB
100
100 84.4x,
10 769GB
26.2x, 10
1 401GB
1k 2k 4k 8k 16k 32k 64k 128k 256k 512k 1024k
1
Input Token Length Llama-3.2 Llama-3.2 Llama-3.1 Llama-3.1 Llama-3.1
-1B -3B -8B -70B -405B

Compute Memory VRAM Compute Memory VRAM

8B Model GPU TTFT* TPOT* 4K Tokens GPU TTFT* TPOT*
Amount Access Consume Amount Access Consume

1K token 15.9 TFLOPs 29.2 GB 15.3 GB 1xA100 73~170 ms 10~24 ms 1B Model 12.4 TFLOPs 19.2 GB 15.3 GB 1xA100 57~132 ms 1.8~4.3 ms

1M token 597 PFLOPs 15.6 TB 401 GB 6xA100 7.6~17.7 min 32~75 ms 405B Model 3.5 PFLOPs 1.6 TB 769 GB 10xA100 1.6~3.7 s 55~128 ms
*TTFT is estimated using Compute Amount / (Peak OPS x compute-util), as prefill is computation-bounded. We assume a compute utilization range 30%~70% to report the estimation.
TPOT is estimated using Memory Access / (Bandwidth x bandwidth-util), as decoding is memory-bounded. We assume a bandwidth utilization range 30%~70% to report the estimation.
A100 Peak Compute Performance (FP16) = 312 TFLOPS; Peak Bandwidth = 2 TB/s
 Application Challenge
Per-request energy consumption w.r.t. model size and input/output lengths,
estimated with actual latency measurement, device & schedule
assumptions, and device spec.
Compa Estimated energy Estimated energy
ny; consumption consumption
Model Date
Host (100in-300out) (10kin-1.5kout)
(Device) (Wh) (Wh)

GPT-4.1
0.10±0.04 0.45±0.21
nano OpenAI;
Azure Apr,
GPT-4.1
(H200& 2025 0.42±0.20 1.59±0.80
mini
H100)
GPT-4.1 0.92±0.50 4.23±1.97

LLaMA-
0.10±0.02 0.60±0.09
3.1-8B
Meta; Jegham et al. made an attempt to estimate the
LLaMA- AWS Jul,
3.1-70B (H200& 2024
1.10±0.13 11.63±1.39 environmental footprint of LLM inference at per-
LLaMA-
H100) prompt level of commercial AI providers, based on
1.99±0.32 20.76±1.80
3.1-405B assumptions on the infrastructure and scheduling.

[1] Jegham, Nidhal, et al. "How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference." arXiv preprint arXiv:2505.09598 (2025).
 Application Challenge
Overall energy consumption, estimated with reported usage, device &
schedule assumptions, and device spec.

A rough estimation result: Assuming a total of 772 billion

queries (estimated with OpenAI 2024 report and the
usage growth pattern, assume 80% short queries)
annually in 2025, GPT-4o inference require
approximately 𝟒×𝟏𝟎𝟏𝟏 Wh, exceeding the total electricity
Jegham et al. made an attempt to estimate the
consumption of 35,000 U.S. residential households.
environmental footprint of LLM inference at per-
prompt level of commercial AI providers, based on
assumptions on the infrastructure and scheduling.

[1] Jegham, Nidhal, et al. "How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference." arXiv preprint arXiv:2505.09598 (2025).
 Contents
Model-Level
1 Background 5 Optimization

2 Preliminary System-Level
≈
6 Optimization

3 Problem Definition & Algo-Level

Conceptual Analysis 7 Optimization

4 Practical Pipeline 8 Conclusion

 Contents
2 Preliminary

• 2.1: LLMs’ Model & Algorithm

• 2.2: Basics of Software
≈ and Hardware
• 2.3: NVIDIA GPU’s Software and Hardware
• 2.4: Deployment
 Contents
2 Preliminary

• 2.1: LLMs’ Model & Algorithm

• 2.2: Basics of Software
≈ and Hardware
• 2.3: NVIDIA GPU’s Software and Hardware
• 2.4: Deployment
 Model: Architecture of LLMs
• Most LLMs are based on Transformer architecture[1], consisting of an
input embedding layer, Transformer blocks, and a decoding layer.
+
• A Transformer block consists of: FC2

• Attention-Linear (transform for Q, K, V, O) Activation

Feed Forward
Network (FFN)
• Multi-Head Self-Attention FC1
layer

• Feed Forward Network (FFN) LayerNorm

Residual
• Layer Norm Stream
+
WO
𝑸𝑲𝑻
softmax 𝑽
Multi-head Self-Attention 𝒅𝒌
Q K V where 𝑄, 𝐾, 𝑉 ∈ 𝑅 #×%

WQ WK WV Attention
layer
LayerNorm

…
I Ġlove ĠEM N LP 𝑁 tokens
[1] Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).
 Attention Layer

Attention Layer
Wk WO

K
The idea of attention module is to
establish token-to-token X
WQ
Q A
Wv
V X’
“attention” relationships within
a sequence.

This relationship is modeled Calculation of each attention head:

by an attention matrix, • 𝑄 = 𝑋𝑊" ; 𝐾 = 𝑋𝑊# ; 𝑉 = 𝑋𝑊$ to map 𝑋 ∈
where each row represents
ℝ%×' to 𝑄, 𝐾, 𝑉 ∈ ℝ%×(
one token’s attention
distribution to previous • Calculate inner product S = 𝑄𝐾 ) , apply mask
tokens (sum up to 1). and softmax to get the attention matrix 𝐴 =
Causal attention mask: softmax(𝑆 + 𝑀) ∈ ℝ%×%
Each token only has positive • 𝑋 * = 𝐴𝑉 to get the output
attention to previous tokens
 Generative Modeling Algo: Autoregressive Model
• Contemporary LLMs are generative models that uses the “autoregressive”
generative modeling method.

• The core task of generative modeling is to learn a parametrized model 𝑝! (𝒙)

%
from observed data 𝒙 " "#$ , which in some sense capture the unknown real
distribution of data 𝑝&'(' (𝒙) , and can do stochastic sampling (i.e., sample
generation) & probabilistic inference (e.g., likelihood estimation).

• Autoregressive models are a family of generative modeling methods that

models the joint probability of a token sequence 𝒙 = [𝑥$ , 𝑥) , … , 𝑥* ] as a product of
conditional probability distributions, each conditioned on the preceding tokens:
*

𝑝 𝑥$ , 𝑥) , … , 𝑥* = 𝑝 𝑥$ + 𝑝(𝑋" = 𝑥" |𝑥$ , … , 𝑥"+$ )

"#)
 Sampling Process of LLMs
LLMs are autoregressive models that uses one transformer to model the
conditional distributions.
• Denoting the vocabulary set as |𝑉|, the transformer maps the token sequence 𝑥!" ∈ 𝑉 "#$ to
a sequence of logits 𝑙 ∈ ℝ("#$)×|)| , where the logits 𝑙 𝑖 − 1 ∈ ℝ|)| corresponding to 𝑥"#$ is
regarded as the categorical distribution’s parameter of 𝑥" .

The KV cache design for sampling from causal LLMs:

• At first glance, in each step of autoregressive sampling: 𝑥" ∼ 𝑝* (𝑥!" ), the model forward
process has 𝑂((𝑖 − 1)+) computation complexity. Thus the overall sampling process
(assume 𝑁 steps) has 𝑂(𝑁 ,) complexity.
• Most contemporary LLMs choose to use causal attention: The calculation of features of 𝑥"
only attends to 𝑥!" . In this way, newly sampled tokens 𝑥-" don’t influence the features
corresponding to 𝑥" .
• This enables us to “cache” already calculated features (specifically, the Key and Value) of
preceding tokens to avoid recalculating their features, thus reduce overall computation
complexity of sampling to 𝑂(𝑁 +).
 Sampling Process of LLMs
After applying the KV cache technique, a typical LLM inference/sampling*
process can be divided into two stages:
• Prefilling Stage: The LLM calculates and saves the KV cache of the initial
input tokens, and samples the first output token.
• Decoding Stage: The LLM samples the output tokens one by one with the KV
cache, and in the meantime updates the KV cache.

*In this tutorial, the terms “inference” and “sampling” will be used interchangeably. Although they differ in the context of probabilistic modeling, here they both refer to
either a single sampling step—i.e., a forward pass of the model—or the overall sampling process, depending on context.
 Attention Computation in Prefill/Decode Stages
Prefill Decode
Attention Layer Attention Layer
Wk K WO Wk K-Cache K WO

Who Who

V-Cache
WQ Wv WQ Wv
won Q A V won V
? ?
Jack Q A V
Jack
Won

The LLM calculates and saves the KV The LLM samples the output tokens
cache of the initial input tokens, and one by one with the KV cache, and in
samples the first output token. the meantime updates the KV cache.

Calculate 𝑁 query, key, value; Calculate 1 query, key, value; read N

calculate 𝑁×𝑁-sized attention matrix A. key, value from KV cache; calculate
1×(𝑁 + 1)-sized attention matrix A.
 Contents
2 Preliminary

• 2.1: LLMs’ Model & Algorithm

• 2.2: Basics of Software
≈ and Hardware
• 2.3: NVIDIA GPU’s Software and Hardware
• 2.4: Deployment
 Software, Hardware, Instruction
Let’s first have an overview of the most basic terms of interest:

• What is a computing hardware? Hardware consists of Instruction Set Architecture (ISA)

microelectronic components that transfer and transform Define the instruction interface of a device

electrical signals. Through these physical operations,

hardware realizes high-level abstractions, that is Example 1: CPU ISA defines instruction format and meaning. [1]
executing functionalities described by instructions, Typical ISA for CPU includes MIPS, RISC-V, x86, x86-64 (amd64), etc.

such as storing, transferring, and transforming data.

• What is software? All software -- no matter it is an OS, a

game, or an AI model’s inference engine -- is a structured
pack of instructions and data.
Example 2: NVIDIA Ampere GPU architecture has compute
• Instructions are the interface between software and capability sm80, which indicates its supported certain
instructions (ISA) and some microarchitectural features
hardware. Instruction Set Architecture (ISA) defines this
instruction interface: what instructions are available, how
they are encoded and executed.

[1] [Link]
 A Conceptual Layered Overview
Runtime Lib Toolchain OS / Driver • Toolchain produces hardware-specific instructions.
Software Interact with Compile instructions Orchestrate resources or provide • Runtime library manages program execution, send
device at runtime for host and device low-level resource-access API instruction to device, transfer data to/from device, and
optionally call toolchain dynamically.
Hardware System • A hardware system consists of host, accelerator
devices, and their interconnects
Accelerator Device(s)
Host Machine
(CPU & Memory & …) AI Chip Off-Chip Memory
Accelerator’s Instruction Set
Architecture (ISA)
Accelerator Chip Define the instruction interface of
the AI accelerator device
Microarchitecture
Define functional modules (controller, computation,
memory) and how they “connect”: (1) control path: Physical Design /
• A microarchitecture design organizes modules to
how the controller parse the instruction and distribute Implementation implement the accelerator’s ISA
control information; (2) data path: how modules
parse and pass data based on control information

Functional Modules • A functional module combines circuits to perform

certain functions
Arithmetic units, controller, cache, …

Circuits • Basic logic and memory elements

Logic circuits, memory cells constructed by transistors, interconnects, capacitors, …
 A Composition Overview
• A system primarily consists of the following components:
PCIe Bandwidth: ~100GB/s Data cache
during
Global control Host AI Chip computation
CPU On-chip cache SRAM
Capacity: The total amount of data Capacity: tens of MB
that a memory component can hold. Bandwidth: 10–
Weight cache Input/Output cache 100 TB/s
Large-capacity Off-chip
storage memory
DRAM Processing unit
(Use HBM as an example)

register
register

Output
Capacity: tens of GB Vector/Scalar Computation

Input
Bandwidth: ~TB/s Matrix/Tenso processing core
HBM, DDR, r processing
LPDDR, … unit
unit
Bandwidth: The rate at which data can be
transferred between two components of a system. Computation
Connecting with other AI chips
Interconnection Controller flow control of
interface AI chips

PPT credit: Prof. Zhenhua Zhu@Tsinghua University

 Common Hardware Types
• Common hardware types: CPU, GPU, FPGA, ASIC
• Chip-level metrics/characteristics: peak compute performance (OPS), energy efficiency
(OPS/W), power (W), and area (mm²).
• Also need to consider: generality and suitableness to algorithms; comprehensiveness of the
software ecosystem.

CPU GPU FPGA ASIC

Central Processing Unit Graphics Processing Unit Field-Programmable Gate Array Application Specific Accelerator
High generality Strong parallel computing Hardware programmability High specialization
Low AI computing capability Flexibility and High performance and low
performance High-bandwidth memory reconfigurability power consumption

General More general and fine-grained ISA. Rely on More specialized and coarse-grained Specialized
software to implement coarse-grained ISA or even template-based design that
operators, algorithms. hard code an algorithm.
 Chip-level Evaluation Metrics
• Chip-level metrics/characteristics
Chip-level evaluation metrics
Computation per unit time
(OPS)

Performance
Computation per
unit of energy Energy Area (mm2) → Tape-
(OPS/W, OPs/J) Efficiency out fabrication and
PPA packaging costs

Power Area
Power (W) → Energy
consumption level

PPT credit: Prof. Zhenhua Zhu@Tsinghua University

AI1.0 Accelerators in Different Scenarios
From cloud center to tiny edge device

Sensor, Wearable Device Mobile / IoT Device Smart City / Auto-driving Car Cloud Center

Webpage
 From AI 1.0 to AI 2.0: Energy Efficiency Metric
Hardware energy efficiency → Inference-system energy efficiency
TOPs/J → Tokens/J

Metric!Tokens/J

ASIC, WaferLLM,
University of Edinburgh, C.
He, et al., 6.2 Tokens/J WebPage
PIM/NDP, Towards, Tsinghua,
L. Guo et al., 47 Tokens/J

FPGA, TerEffic, PKU/NUS,

GPU, AWQ, MIT, J. Lin, et. al., C. Li, et al., 6.3 Tokens/J
0.8 Tokens/J

Paper
 Basic Knowledge : Development of Chips
Stanford：
EIE sparse NVIDIA：
accelerator A100
HUAWEI:
600mW 1.5TOPS/W FP16（Tensor Core）： AMD：MI100
2014 Institute Ascend 310 312 TFLOPS FP 32：95.7 TFLOPS NVIDIA
Computing THU：FPGA16 16 TOPS（INT8）
8 TFLOPS（FP16） B200/B100
2006 Technology, FPGA accelerator FP16（Tensor
NVIDIA： CAS： 187.8 GOPS NVIDIA H100 Core）：2250
Wafer-level TFLOPS
Introduced Introduced Cambricon：
Chips FP32：60 TFLOPS
CUDA DianNao NVIDIA：P100 MLU100 400,000 computing FP16（Tensor Core）：
452GOPS，485mW 10.6 TFLOPS 32 TOPS（INT8） units 1,000 TFLOPS
932 GOPS/W 16 TFLOPs（FP16）

2009 2015 2017 2019 2021 2023

2006 2014 2016 2018 2020 2022 2024

2017 Google： 2018-19 Google： 2021 Google: 2023 Google: TPU v5

Unified training Large-scale TPU v4 393 TOPS（BF16/INT8）
2015 Google： and inference training TPU v3 275 TOPS（BF16/INT8）
ICML 2009
Began deploying TPU v2 90 TFLOPS
NVIDIA：First Wafer-level chip
TPU v1 45 TFLOPS，200W PIM chips
GPU- WSE-2
accelerated 92TOPS (INT4)，40W NVIDIA： NVIDIA: HBM-PIM
850,000 AI-optimized
70% energy efficiency
machine ~2.5TOPS/W V100 with Edge-side improvement
cores

learning paper Tensor Core Jetson Nano 2× performance

125 TFLOPS 472 GFLOPS，5-10W
PPT credit: Zhenhua Zhu
improvement AMD MI300
 Contents
2 Preliminary

• 2.1: LLMs’ Model & Algorithm

• 2.2: Basics of Software
≈ and Hardware
• 2.3: NVIDIA GPU’s Software and Hardware
• 2.4: Deployment
 Hardware System & GPU Device
NVIDIA GPU
• GPU device:
• GPUs use a Single Instruction Multiple Threads (SIMT) architecture.
• Compared with CPUs, GPUs are better suited for programs featuring simple
control logic and large-scale parallel computation.
• Hardware system: A GPU is not an independent computing platform, but rather a
co-processor to the CPU.

[1] C. John, et al. ”Professional CUDA C Programming."

 GPU Microarchitecture
• Design Concept of GPUs
• Single Instruction Multiple Threads (SIMT): A single instruction is executed in
parallel by multiple threads.

SM (Streaming
Multiprocessor)
serves as the
basic hardware
unit for parallel
instructions.

SIMT
GPU Architecture
[1] E. Lindholm, et al. ”NVIDIA Tesla: A Unified Graphics and Computing Architecture.”, in IEEE Mirco, 2008.
 GPU Software Stack
• NVIDIA GPU Software Stack
• CUDA (Compute Unified Device Architecture) is the NVIDIA’s GPU parallel
programming platform and programming model, featuring a rich software ecosystem.
• AI & LLM frameworks is built on them.
Analysis
Compiler Debugger
Tools
LLM Training
nvcc nsight cuda-gdb
Framework
Megatron-LM

CUDA Toolkit Base AI Framework

Toolchain & Runtime Lib PyTorch, LLM Inference
Tensorflow, Framework
Jax
TensorRT-LLM
Communication Operator
Library Library
NCCL, cuBLAS,
NVSHMEM cuSPARSE
[1] M. Shoeybi, et al. ” Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.”, arXiv, 2019.
 GPU Programming Model
• NVIDIA GPU Software Stack
• CUDA allows writing GPU code in high-level languages such as C/C++, reducing
programming complexity.
AI algorithms

Python (PyTorch)
（pybind） User
interface
CUDA C

PTX
Processed by
compilers
SASS Assemble

GPU
 GPU Programming Model
• Programming Organization: How to Parallelize?

SIMT: Single Instruction Multiple Threads

Thread
Minimal parallel Kernel<<<grid, block>>>
unit

Block
A block contains multiple
threads. Number of
threads per block

Grid Number of
A grid contains multiple blocks, blocks per grid
encompassing all threads of a
single kernel.

Why define the above hierarchical organization?

Thread It corresponds to the GPU memory hierarchy!
[1] C. John, et al. ”Professional CUDA C Programming."
 GPU Programming Model
• GPU Memory Hierarchy
• Thread
SIMT: Single Instruction Multiple Threads
• Each thread has its own
registers (RG).
• Register contents are not
shared between threads.
• Block
• All threads within a block
share Shared Memory. Can
cooperate & communicate
RG RG RG
through it.
• Grid
• All blocks access data from
Global Memory (High
Bandwidth Memory, HBM).

Memory Hierarchy
[1] C. John, et al. ”Professional CUDA C Programming."
 GPU Programming Model
• GPU Memory Hierarchy
• From the perspective of memory access efficiency: how threads are organized
significantly impacts kernel performance.

Thread （256KB/SM）
RG

Shared ~19TB/s
Block （192KB/SM）
Memory RG RG RG

Global ~1.6TB/s
Grid Memory （40GB/GPU）

Bandwidth and capacity of each memory

hierarchy (A100 40GB GPU)
Memory Hierarchy
[1] C. John, et al. ”Professional CUDA C Programming."
 Contents
2 Preliminary

• 2.1: LLMs’ Model & Algorithm

• 2.2: Basics of Software
≈ and Hardware
• 2.3: NVIDIA GPU’s Software and Hardware
• 2.4: Deployment
 Algorithm Deployment Process
• How AI algorithms are deployed on hardware for actual computation
• AI Inference can be seen as forwarding data on a computational graph, where each
node represents a single operator, edge represents dependency.
• During deployment, operators are translated into hardware instructions. In the runtime,
hardware executes instructions.
Add & LayerNorm
𝑌 = 𝑊)𝑋
FC2

Activation N
FC1
FC1
K 𝑊
Add & LayerNorm Taking FC1 as an
K
WO example
Multi-head Self-Attention
Q K V M 𝑋 𝑌
K Cache V Cache

WQ WK WV

Abstracted as a single GEMM

Transformer layer in LLMs operator
 Algorithm Deployment Process
• How AI algorithms are deployed on hardware for actual computation
• AI Inference can be seen as forwarding data on a computational graph, where each
node represents a single operator, edge represents dependency (software level).
• During deployment, operators are translated into hardware instructions. In the runtime,
hardware executes instructions.
N

K 𝑊

K Key question: Given the hardware, how can we evaluate and

improve the inference efficiency of an LLM?
M 𝑋 𝑌

Abstracted as a GEMM Hardware Instructions (CUDA) The hardware executes

operator Include computation, memory, and instructions to perform
control. computations.
[1] Bastian Hagedorn, et al. “Graphene: An IR for Optimized Tensor Computations on GPUs.” ASPLOS 2023.
 !" Contents

Model-Level
1 Background 5 Optimization

2 Preliminary System-Level
≈
6 Optimization

3 Problem Definition & Algo-Level

Conceptual Analysis 7 Optimization

4 Practical Pipeline 8 Conclusion

 Review: Application Challenge
The scale of the model size and input/output pose challenges for efficient
inference across platforms and application scenarios.

Model size, input/output length Inference cost (time, storage, energy)

Human AI World

Deployment

Cloud Edge
Requirements or constraints of application scenarios and platforms

High Low Small Low Energy

Throughput Latency Storage Consumption
 Review: Application Challenge
The scale of the model size and input/output pose challenges for efficient
inference across platforms and application scenarios.

Model size, input/output length Inference cost (time, storage, energy)

To meetHuman
requirements or constraint AI
of application scenarios World
and
platforms, we need to optimize the resource consumption of AI inference.
Deployment

Cloud Edge
Requirements or constraints of application scenarios and platforms

High Low Small Low Energy

Throughput Latency Storage Consumption
 Problem Definition: Objectives & Constraints
Optimization objective or constraint: Usually, latency, memory, energy
consumption or throughput will be the ultimate objective or constraint on
“efficiency”. In the meantime, the intelligence level of AI needs to be retained.

Measured metrics
Measured by testing model on platform, platform-related
Directly correspond to objectives / constraints,
related to final user experience, resource consumption, etc.
1. KV Cache
[Link]
Activation
3. Other

Time Throughput Latency Overheads

Peak
memory
footprint
Time To First
Token (TTFT)
Memory Model
Memory footprint
Params Time Per Output Token
(TPOT)

Time
End-to-end/Request/Generation Latency
Energy Energy Prefill Stage Decode Stage
*Note this is only a conceptual illustration. In actual serving framework, the KV cache pool is usually pre-allocated.
 Problem Definition: Objectives & Constraints
Optimization objective or constraint: Usually, latency, memory, energy
consumption or throughput will be the ultimate objective or constraint on
“efficiency”. In the meantime, the intelligence level of AI needs to be retained.

Measured metrics Proxy metrics

Measured by testing model on platform, platform-related Estimated with only model specification, platform-agnostic
Directly correspond to objectives / constraints, Widely used as the objective in some academic work or early
related to final user experience, resource consumption, etc. stage of model-level optimization.
In practice, they are also useful in diagnosis of the bottleneck
and quick estimation of the measured metrics.
Time Throughput Latency
Compute Amount
• FLoating Point OPerations (FLOPs)
• Multiply–ACcumulate operations (MACs)
Memory
Memory footprint
Param Size

Energy Energy (Estimated) (Estimated) Memory

Memory footprint Access Amount
 How Proxy Metrics Relate with Measured Metrics
• We can use three important proxy metrics to analyze the efficiency
• Compute amount: the amount of operations
• Memory access amount: the amount of data that read or written between off-chip
DRAM and GPU chip
• Memory footprint: the occupied off-chip DRAM size to store parameters/KV
cache/activation

Higher Latency
Higher Compute Amount

Lower Throughput Then, let’s bring hardware

Higher Memory specification into the picture.
Access Amount Higher Energy
Consumption
Higher Memory Footprint Higher (Actual)
Memory Footprint
 How Proxy Metrics Relate with Measured Metrics
./01234 50/263 =40/:> 5??4@@ 50/263
• Intuitively, 7489 ./01234 74:;., A86BCDB3E
are two lower bounds of latency.

• But the compute units might not be fully utilized, the bandwidth might not be fully utilized:
+,-./0/1 234567/ 8/9:. +,-./0/1 ><?1@.17-
Compute_utilization = 8/<= 234567/ 8/9:.
Bandwidth_utilization = 8/<= ><?1@.17-
• Review our previous estimation example:

Llama-3.1-8B Compute Memory VRAM

GPU TTFT* Why do we use compute amount
#Prefill tokens Amount Access Consume
instead of memory access amount to
1K token 15.9 TFLOPs 29.2 GB 15.3 GB 1xA100 73~170 ms estimate latency for the prefilling stage?
=> It’s because prefiling stage is
The compute amount of prefilling 1K token with Llama-3.1-8B is 15.9 TFLOPs. usually more “computation-bounded”

NVIDIA A100 80G’s FP16 peak compute performance is 312 TFLOPS.

𝑪𝒐𝒎𝒑𝒖𝒕𝒆 𝑨𝒎𝒐𝒖𝒏𝒕 >?.@ ABCDEF ?> LF
𝐋𝐚𝐭𝐞𝐧𝐜𝐲 = 𝑷𝒆𝒂𝒌 𝑪𝒐𝒎𝒑𝒖𝒕𝒆 𝑷𝒆𝒓𝒇. × 𝐜𝐨𝐦𝐩𝐮𝐭𝐞_𝐮𝐭𝐢𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 = G>H ABCDEI × JKLMNOP_NOQRQSTOQKU = JKLMNOP_NOQRQSTOQKU

If we assume 30%~70% compute utilization, we can get 73ms~170ms.

 Performance Analysis Model: Roofline Model
• Proposed by David Patterson in 2009
• Purpose: Uses an algorithm’s compute and memory access characteristics (operational
intensity) along with the chip’s peak performance and memory bandwidth to roughly assess
computational bottlenecks and guide subsequent optimization directions.

Peak performance
The maximum
Performance

performance when
Compute

compute units are fully

utilized.

Operational Intensity
#compute amount
Number of operations per byte of
memory accessed #memory access amount
 Performance Analysis Model: Roofline Model
• Proposed by David Patterson in 2009
• Purpose: Uses an algorithm’s compute and memory access characteristics (operational
intensity) along with the chip’s peak performance and memory bandwidth to roughly assess
computational bottlenecks and guide subsequent optimization directions.
Performance
Compute

When bandwidth is fully utilized,

performance depends on memory bandwidth.
Slope: memory bandwidth (Bytes/s)

Operational Intensity
Number of operations per byte of
memory accessed
 Performance Analysis Model: Roofline Model
• Proposed by David Patterson in 2009
• Purpose: Uses an algorithm’s compute and memory access characteristics (operational
intensity) along with the chip’s peak performance and memory bandwidth to roughly assess
computational bottlenecks and guide subsequent optimization directions.

Compute–memory balance point

Peak performance
Performance

Memory Bound
Compute

Compute Bound

Slope: memory bandwidth (Bytes/s)

Operational Intensity
Number of operations per byte of
memory accessed
 Performance Analysis Model: Roofline Model
• Take LLM as an example:
• Prefill stage: Compute bound; Decode stage: memory bound

Roofline Model of NVIDIA A100

Prefill Decode
𝒍𝒅 𝒅
𝑸/𝑲/𝑽/𝑶
𝒍+𝒅 𝒅+𝟏
𝒍
𝑸𝑲𝑻 𝒍
𝒍+𝟏
≫𝟏 ~𝟏
𝒍 Compute
𝑨𝒕𝒕𝒆𝒏𝑽 𝒍 bound
𝒍+𝟏
Memory
𝒍𝒅𝑭𝑭𝑵 𝒅𝑭𝑭𝑵
𝑭𝑭𝑵 bound
𝒍 + 𝒅𝑭𝑭𝑵 𝒅𝑭𝑭𝑵 + 𝟏

Operational intensity at different stages

(FLOPs/Byte)
 What is the Optimization Space
For designing Sampling
Autoregressive the optimization
Process space, we need to know the “bottleneck”
modules or properties of the current algorithm, model, and software that
hinders the efficiency of running them on the given hardware.

Root causes of LLM inference inefficiency

• Application: The input / output token length can be very long.

• Algorithm: Autoregressive model samples tokens one by one.

• Model: (i) The transformer model has a large number of weights

and computations. (ii) Attention modules have quadratic
complexity w.r.t. the input token length.
 What is the Optimization Space
Cause Cause what? Optimization Ideas
Idea 1: Parallelize the sequential
sampling of existing model?
Algori-
Application Idea 2: Compress the input
• Low operational intensity cause low
thm context to shorter one?
Long Input/Output modify the
compute utilization algorithm Idea 3: Don’t use autoregressive
• High end-to-end latency
model?
• Dynamically increasing KV cache
might cause memory fragmentation Idea 1: Remove redundant
Algorithm with a naïve system implementation, params/acts/computation?
Autoregressive increasing both memory footprint and Model Static (model compression) or
Sampling Method access cost dynamic (dynamic inference)
modify the
model Idea 2: Design novel
lightweight structure (e.g.,
efficient FFN & attention)
Model • Large computation
Focus 1: compiler/runtime
• Large Model Scale • Large memory access
lib/hardware for efficient NN
• Attention Operation • Large memory footprint System
modify the execution
software & Focus 2: request scheduling /
hardware resource management for
service-level objectives
 What is the Optimization Space
Optimization Ideas Techniques Optimization Space
Idea 1: Parallelize the sequential Efficient Output Decoding
sampling of existing model? parallel generation, verification, or
Algori- refinement strategies There is no representation/space or
optimization formalization general
thm Idea 2: Compress the input Input Compression to many work. The design of these
modify the context to shorter one? prompt compression, RAG methods directly change a core application
algorithm or algorithm property to improve efficiency.
Idea 3: Don’t use autoregressive Alternative Generative
model? Paradigms

Idea 1: Remove redundant Model Compression

params/acts/computation? reduce model redundancy in a
static manner
Static (model compression) or • Model Structure (e.g., #layer, #channel)
Model dynamic (dynamic inference) Dynamic Inference • Value Representation (e.g., low-bit
modify the reduce model redundancy in a representation)
model Idea 2: Design novel dynamic manner
lightweight structure (e.g., Structure Design
efficient FFN & attention) design novel structure, which often
require training
Focus 1: compiler/runtime
lib/hardware for efficient NN • Computational Graph (e.g., fusion)
System Operator-Level Opt. • Kernel Implementation
execution
modify the • Request scheduling, resource
software & Focus 2: request scheduling / Framework-Level Opt. management, model placement
resource management for • Framework Implementation
hardware Hardware-Level Opt.
service-level objectives • Hardware Implementation
 Contents
Model-Level
1 Background 5 Optimization

2 Preliminary System-Level
≈
6 Optimization

3 Problem Definition & Algo-Level

Conceptual Analysis 7 Optimization

4 Practical Pipeline 8 Conclusion

 Practical Pipeline of Model/System-Level Method Design
Estimation According to Specifications include Application & Model & Hardware
Application Specification Model Specification Hardware Specification Can estimate theoretically:
• Objectives, e.g., memory • Num hidden layers • Peak Compute Performance • Judge compute or memory
footprint, latency, throughput • Num key-value heads • Memory Capacity bound of each module by
• Context lengths • Hidden size • Memory Bandwidth roofline model
• Batch size • Intermediate size • Estimate bottleneck module
• … • Num attention heads • Estimate overall objectives
• Head dim

Profile/Diagnosis Tool: Nsight System & Nsight Compute

Performance
Operational Intensity

Model-Level Design System-Level Design (Operator-Level)

• Analyze compressing which dimension / how to redesigning the module
• Operation fusion to reduce memory access and kernel launch
might help with the efficiency most
overhead
• Analyze algorithmic redundancy & property, how can we retain/restore
• Reimplement some operations
performance
 GPU Software Stack
• NVIDIA GPU Diagnosis Tool
• Nsight System
• A system-level analysis tool used for diagnosing performance bottlenecks from a
global perspective and identifying key operators that require optimization.
nsys nvprof -o {output-file} python3 [Link]
• Taking LLaMA2-7B as an example, Nsight Systems can visualize all operators
and their calls across all inference stages. It enables researchers to quickly
pinpoint efficiency bottlenecks from a global perspective.
• Input length: 2048 tokens; decoding: 2 tokens

Prefill 1-step Decode

 GPU Software Stack
• NVIDIA GPU Diagnosis Tool
• Nsight System
• A system-level analysis tool used for diagnosing performance bottlenecks from a
global perspective and identifying key operators that require optimization.
nsys nvprof -o {output-file} python3 [Link]
• Taking LLaMA2-7B as an example, Nsight Systems can visualize all operators
and their calls across all inference stages. It enables researchers to quickly
pinpoint efficiency bottlenecks from a global perspective.
• Nsight Systems analysis reveals that, during the decode stage, the General Matrix-Vector
Multiplication (GEMV) operator in the linear layer is the primary performance bottleneck.
 GPU Software Stack
• NVIDIA GPU Diagnosis Tool
• Nsight Compute
• A kernel-level performance analysis tool focused on deep optimization of
individual CUDA kernels, used in conjunction with the Roofline Model for
performance analysis.
ncu –set full -o {output-file} python3 [Link]
• Taking LLaMA2-7B as an example, using Nsight Compute to analyze the GEMV
operator in the linear layer shows that the operator is severely memory-bound.
Performance（TFLOPS）

• Solution approach: Apply INT4 quantization to the weights to reduce weight’s memory
access cost.
Input Output
FP16 (us) INT4 (us)
channel channel
4096 11008 159.3 52.0
11008 4096 45.6 37.6
4096 4096 43.5 23.0
Operational Intensity (FLOP/Byte) RTX 3090 GPU
 GPU Software Stack
• NVIDIA GPU Diagnosis Tool
• Nsight Compute
• Taking LLaMA2-7B as an example
• Implementation: To achieve acceleration, the dequantization operator and the GEMV
computation operator need to be fused.

Dequantization Operator
Fusion
GEMV
 Example: Design Thought of Model Compression Method

• How to design a sparsification method for a given model and scenario

• Example: MoA[1]

Hardware Data distribution Algorithm Capability

efficiency visualization performance requirements for
For long
evaluation evaluation sparse models
context, the
attention
operator
Sparse patterns differ Needs to be
incurs the
Dynamic/Static across different extensible/scalable
highest heads for long context
computational
cost.

Dynamic: Static:
Design extension
Sparse low high Homogeneous Heterogeneous
rules of the attention
Attention hardware hardware sparse pattern attention pattern
span
efficiency efficiency

[1] Fu, Tianyu*, Huang, Haofeng*, Ning, Xuefei*, et al. “MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression.” CoLM 2025.
 Contents
Model-Level
1 Background 5 Optimization

2 Preliminary System-Level
≈
6 Optimization

3 Problem Definition & Algo-Level

Conceptual Analysis 7 Optimization

4 Practical Pipeline 8 Conclusion

 What is the Optimization Space
Optimization Ideas Techniques Optimization Space
Idea 1: Parallelize the sequential Efficient Output Decoding
sampling of existing model? parallel generation, verification, or
Algori- refinement strategies There is no representation/space or
optimization formalization general
thm Idea 2: Compress the input Input Compression to many work. The design of these
modify the context to shorter one? prompt compression, RAG methods directly change a core application
algorithm or algorithm property to improve efficiency.
Idea 3: Don’t use autoregressive Alternative Generative
model? Paradigms

Idea 1: Remove redundant Model Compression

params/acts/computation? reduce model redundancy in a
static manner
Static (model compression) or • Model Structure (e.g., #layer, #channel)
Model dynamic (dynamic inference) Dynamic Inference • Value Representation (e.g., low-bit
modify the reduce model redundancy in a representation)
model Idea 2: Design novel dynamic manner
lightweight structure (e.g., Structure Design
efficient FFN & attention) design novel structure, which often
require training
Focus 1: compiler/runtime
lib/hardware for efficient NN • Computational Graph (e.g., fusion)
System Operator-Level Opt. • Kernel Implementation
execution
modify the • Request scheduling, resource
software Focus 2: request scheduling / Framework-Level Opt. management, model placement
resource management for • Framework Implementation
Hardware-Level Opt. • Hardware Implementation
service-level objectives
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm- Model Compression
Input Compression
level prompt compression, RAG • Quantization
Alternative Generative • Sparse Attention
Paradigms • Weight Pruning
• Sharing
Model Compression • Knowledge Distillation
reduce model redundancy in a
static manner

Model- Dynamic Inference Dynamic Inference

reduce model redundancy in a
level dynamic manner
• Module-granularity
Structure Design • Model-granularity
design novel structure, which often
require training
Structure Design
Operator-Level Opt.
• Mixture-of-Experts (Efficient FFN)
System- • Efficient Attention
Framework-Level Opt.
level
Hardware-Level Opt.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm- Model Compression
Input Compression
level prompt compression, RAG • Quantization
Alternative Generative • Sparse Attention
Paradigms • Weight Pruning
• Sharing
Model Compression • Knowledge Distillation
reduce model redundancy in a
static manner

Model- Dynamic Inference Dynamic Inference

reduce model redundancy in a
level dynamic manner
• Module-granularity
Structure Design • Model-granularity
design novel structure, which often
require training
Structure Design
Operator-Level Opt.
• Mixture-of-Experts (Efficient FFN)
System- • Efficient Attention
Framework-Level Opt.
level
Hardware-Level Opt.
 Definition
• Motivation
Neural network weights and activations are typically represented using high-precision floating-point
formats. However, there exists numerical redundancy in neural network computations, and using
lower-precision arithmetic does not significantly affect the accuracy of the network [1].
• Definition
Quantization: Represent weights & activations with low-bit numbers, thereby storing them or
computing them with reduced numerical precision.

Sign 1 bit Mantissa 7 bit

[-127, 128]

IEEE 754 standard 32-bit floating-point data 8-bit fixed-point data

* 图片来源: MIT EfficientML Course
[1] Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, Han et al., ICLR 2016.
 Two Procedures
• Offline stage: Model quantization procedure
• Convert the FP weights into low-bit-width weights
• Determine the quantization parameters for activations (optional, if using low-precision activation and not using
online quant) Quantize floating-point weights (FP32/FP16) into low-bit weights (INT8/INT4).

Quantization
Tools

FP32 INT8
Weights Weights

• Online stage: Quantized inference procedure

• Low-Precision computation: LP arithmetic -> Requantization
Weight Bias

int8 int16
Input int32 int32 Output
Linear Accumulator Activation Quant
(int8) (int8)

• High-Precision computation: Dequant -> HP arithmetic -> (optional) Quant

[1] [Link]
 Model Quantization Procedure

• Example: Uniform Fixed-Point Quantization

• A quantization method needs to answer: “How to
convert floating-point numbers into fixed-point
number representation? ”

Fixed-point Floating-point Rounding function

• Quantization parameters: scaling factor (S); zero
point (Z)
• How to decide quantization parameters? Take
asymmetric quantization (Z ≠ 0) as an example, if
we want the float-point range to cover [𝑟T"U , 𝑟TVW ]:

* 图片来源: MIT EfficientML Course

[1] Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, Han et al., ICLR 2016.
 Quantization
• Quantization Granularity
• For actual storage decrease and speed-up, a group of values needs to share the same quantization
parameters (the group size is called quantization granularity, e.g., tensor-wisely, channel-wisely).

S Z INT Tensor-wise quantization

S Z INT
S Z INT
FP16 Channel-wise quantization
S Z INT
S Z INT

S Z INT INT S Z
S Z INT INT S Z
Group-wise quantization
S Z INT INT S Z
S Z INT INT S Z
* 图片来源: MIT EfficientML Course
[1] Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, Han et al., ICLR 2016.
 Quantization
• Core focus of quantization parameter decision: Appropriately balance representational
range and precision (i.e., balancing truncation error and rounding error).

Large rounding error Large truncation error Well-balanced

(large scale factor) (small scale factor) (appropriate scale factor)

[1] Zhao, Ritchie, et al. "Improving neural network quantization without retraining using outlier channel splitting." International conference on machine learning. PMLR, 2019.
 Quantization
• Two types of quantization process
• PTQ v.s. QAT
PTQ Workflow QAT Workflow
Pretrain
Repara- Quantizer Repara- Quantizer Quant.
meterize Selection meterize Selection configuration

Weight
Act. quant. Adjust PTQ Training
quant. QAT
parameters quant. value techniques
parameters

Pros Cons Cons

Pros
No weight optimization, Lack of recovery,
Quantization with High computational
No training cost, Difficult to quantize
Fast quantization models to ultra-low
performance recovery, cost and data
Enabling lower bitwidth demand
process bitwidth
 Quantization
• Representative studies
Quantized Tensor Type Quantized Quantized Quantized Value
Weight Activation KV Cache Format Criterion Update
GPTQ √ Uniform Statistic-based √
AWQ √ Uniform Search-based √
SqueezeLLM √ Non-uniform Statistic-based
GPT3.int8() √ √ Uniform Statistic-based
SmoothQuant √ √ Uniform Statistic-based √
RPTQ √ √ Uniform Statistic-based
OminiQuant √ √ Uniform Search-based
FlexGen √ √ Uniform Statistic-based
Atom √ √ √ Uniform Statistic-based
KVQuant √ Non-uniform Statistic-based
KIVI √ Uniform Statistic-based
[1] Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers." arXiv preprint arXiv:2210.17323 (2022).
[2] Lin, Ji, et al. "Awq: Activation-aware weight quantization for on-device llm compression and acceleration." Proceedings of machine learning and systems 6 (2024): 87-100.
[3] Kim, Sehoon, et al. "Squeezellm: Dense-and-sparse quantization." arXiv preprint arXiv:2306.07629 (2023).
[4] Dettmers, Tim, et al. "Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale." Advances in neural information processing systems 35 (2022): 30318-30332.
[5] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International conference on machine learning. PMLR, 2023.
[6] Yuan, Zhihang, et al. "Rptq: Reorder-based post-training quantization for large language models." arXiv preprint arXiv:2304.01089 (2023).
[7] Shao, Wenqi, et al. "OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models." ICLR. 2024.
[8] Sheng, Ying, et al. "Flexgen: High-throughput generative inference of large language models with a single gpu." International Conference on Machine Learning. PMLR, 2023.
[9] Zhao, Yilong, et al. "Atom: Low-bit quantization for efficient and accurate llm serving." Proceedings of Machine Learning and Systems 6 (2024): 196-209.
[10] Hooper, Coleman, et al. "Kvquant: Towards 10 million context length llm inference with kv cache quantization." Advances in Neural Information Processing Systems 37 (2024): 1270-1303.
[11] Liu, Zirui, et al. "KIVI: a tuning-free asymmetric 2bit quantization for KV cache." Proceedings of the 41st International Conference on Machine Learning. 2024.
 SmoothQuant
• Typical PTQ Method: SmoothQuant[1]

• In large models, activations often contain

Motiv extremely large outliers, leading to Insight Analysis of weight and
ation significant quantization errors during W8A8 activation data distribution
quantization.
• This study focuses on analyzing the data • Outliers in large-model activations appear in
distribution patterns of the model and specific channels. This property can be
exploring methods to reduce outliers,
leveraged to balance the data distribution of
aiming to achieve nearly lossless W8A8
quantization. weights and activations.
Freque
ncy

Outliers

Val
ue
[1] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." ICML, 2023.
 SmoothQuant

Method

[1] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." ICML, 2023.
 AWQ
• Typical PTQ Method: AWQ[1]

• Not all weights in an LLM are equally Methods To consider bothsalient and non
Motiv important. salient weights, AWQ searches for
ation • Protecting only 1% salient weights an optimal scaling factor that
can greatly reduce quantization minimizes the reconstruction error
error. for a certain layer.

[1] Lin, Ji, et al. "Awq: Activation-aware weight quantization for on-device llm compression and acceleration." Proceedings of machine learning and systems 6 (2024): 87-100.
 FlatQuant: Flatness Matters for LLM
Quantization
• Motivation: Affine transformations are more powerful to suppress outliers
• Methodology: Learning affine transformations for each linear layer
• Reducing transformation overhead: Kronecker product & kernel fusion

[1] Sun, Y., Liu, R., Bai, H., et al. "FlatQuant: Flatness Matters for LLM Quantization". ICML 2025.
 FlatQuant: Flatness Matters for LLM
Quantization
How to Integrate FlatQuant with the Transformer architecture?

Integration with
Self-attention

Integration with feed-

forward network
 FlatQuant: Flatness Matters for LLM
Quantization
The mean square error of quantization along channels & tokens can be effectively reduced
 IntactKV: Keeping Pivot Tokens Intact
Observation: the pivot tokens exhibit attention sinks together with massive activation outliers.

[1] Ruikang Liu, Haoli Bai†, [Link]. IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact. Findings of ACL, 2024.
 IntactKV: Keeping Pivot Tokens Intact
Avoid the quantization error accumulated on pivot tokens that are critical to the performance.

• The system prompt

contains most pivot
tokens with massive • The rest KV cache
outliers are generated by
quantized model

• System KV cache
are generated by the
BF16 model
• They can be further
trained like LLM
parameters

[1] Ruikang Liu, Haoli Bai†, [Link]. IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact. Findings of ACL, 2024.
 Evaluating Quantized Models
[1] Li, S., et al. "Evaluating Quantized
Large Language Models." ICML 2024.
• Evaluation Dimensions
• Effects of quantization on 5 major categories of
tasks
• Effects of quantization on 11 model families
• Effects of quantizing 3 tensor types on model
performance
• Application scope of SOTA quantization
methods

Paper link Open source

 Evaluating Quantized Models
• Effects of Quantization on Tensor Types
• The larger the model size, the higher the tolerance for Weight and KVcache Quantization,
and the lower the tolerance for Activation Quantization.
• The larger the model size, the fewer outliers in the Weight and KV Cache tensors, and
the more outliers in the Activation tensors.

[1] Li, Shiyao, Ning, Xuefei, et al. "Evaluating Quantized Large Language Models." ICML2024.
 Evaluating Quantized Models
• Effects of Quantization on Emergent Abilities
• The tolerance to quantization varies across the four abilities, listed in descending order of
tolerance: In-context Learning ∼ Instruction Following > Multi-Step Reasoning ~ Self-
calibration.

[1] Li, Shiyao, Ning, Xuefei, et al. "Evaluating Quantized Large Language Models." ICML2024.
 Evaluating Quantized Reasoning Models
• Reasoning LLMs: Qwen 1.5B - 32B distilled from DeepSeek-R1
• Hard tasks (e.g., AIME) suffer more than easier ones (e.g., GSM8K)
• W8A8 and W4A16 is safe to use (<1% acc drop)
• W4A4 and KV4 can be still risky in practice

[1] Liu, R., Sun Y., Zhang M., Bai H., et al. "Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models". COLM 2025.
 Evaluating Quantized Reasoning LLMs
The scaling effect of quantized reasoning LLMs
• (a) & (b): Large quantized LLMs are preferred to small BF16 LLMs w.r.t. size and latency
• (c) Test-time scaling: higher accuracy with more reasoning tokens, but at a slower rate
when compared to BF16 models

(a) Model Size (GB) (b) Latency (s) (c) Test-time Scaling
[1] Liu, R., Sun Y., Zhang M., Bai H., et al. "Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models". COLM 2025.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm- Model Compression
Input Compression
level prompt compression, RAG • Quantization
Alternative Generative • Sparse Attention
Paradigms • Weight Pruning
• Sharing
Model Compression • Knowledge Distillation
reduce model redundancy in a
static manner

Model- Dynamic Inference Dynamic Inference

reduce model redundancy in a
level dynamic manner
• Module-granularity
Structure Design • Model-granularity
design novel structure, which often
require training
Structure Design
Operator-Level Opt.
• Mixture-of-Experts (Efficient FFN)
System- • Efficient Attention
Framework-Level Opt.
level
Hardware-Level Opt.
 Model-level: Sparse Attention
• Sparse Attention
• Omit certain attention calculations
• to enhance computational efficiency: saving computation on S and O

• Static vs Dynamic Mask

• Static Mask: the attention mask is predefined and remains fixed.
• Dynamic Mask: the attention mask is determined online based on the input.
 Model-level: Sparse Attention
• Sparse Attention
• Sparse Pattern: local, global, random, dilated
block size
• Granularity: blockwise

• Token pruning vs clustering vs merging

retained pruned cluster 0 cluster 1

token

Token pruning Token clustering Token merging

 Sparse Attention
• Representative studies
Mask Generation Sparse Pattern
Static Dynamic Local Global Dilated Random Clustering Pruning Merging
Sparse
√ √ √ √
Transformers
StreamingLLM √ √ √
BigBird √ √ √ √
Spatten √ √
Reformer √ √
H2O √ √ √
MoA √ √ √
NSA √ √ √ √

[1] Child, Rewon, et al. "Generating long sequences with sparse transformers." arXiv preprint arXiv:1904.10509 (2019).
[2] Xiao, Guangxuan, et al. "Efficient Streaming Language Models with Attention Sinks." The Twelfth International Conference on Learning Representations.
[3] Zaheer, Manzil, et al. "Big bird: Transformers for longer sequences." Advances in neural information processing systems 33 (2020): 17283-17297.
[4] Wang, Hanrui, Zhekai Zhang, and Song Han. "Spatten: Efficient sparse attention architecture with cascade token and head pruning." 2021 IEEE International Symposium on High-Performance
Computer Architecture (HPCA). IEEE, 2021.
[5] Kitaev, Nikita, Lukasz Kaiser, and Anselm Levskaya. "Reformer: The Efficient Transformer." International Conference on Learning Representations.
[6] Zhang, Zhenyu, et al. "H2o: Heavy-hitter oracle for efficient generative inference of large language models." Advances in Neural Information Processing Systems 36 (2023): 34661-34710.
[7] Fu, Tianyu, et al. "Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths." Second Conference on Language Modeling.
[8] Yuan, Jingyang, et al. "Native sparse attention: Hardware-aligned and natively trainable sparse attention." arXiv preprint arXiv:2502.11089 (2025).
 StreamingLLM
• Sparse Attention: StreamingLLM

Motiv • Sliding-window attention Metho • Key finding: LLMs tend to assign

ation mechanism discards part of the d attention scores to the initial
long-term historical information tokens (attention sink).
that is rarely needed, but this • Therefore, StreamingLLM not only
often results in severe retains sliding-window attention but
performance degradation. also preserves the attention scores
of the initial tokens.
Wikitext PPL
Wikitext PPL
5158
5.40

[1] Xiao, Guangxuan, et al. “Efficient Streaming Language Models with Attention Sinks.” ICLR 2024.
 MoA
• Sparse Attention: MoA

Motiv • Existing methods apply a

Insigh 🔍 Different attention heads have
homogeneous sparse mask to each different attention spans, requiring
ation attention head, which fails to capture t heterogeneous sparse attention.
the diverse attention patterns in
LLMs and consequently leads to a
significant drop in model
performance.

StreamingLLM [2] with Long Input

🔍 Different input lengths
fixed-length local attention and
Attention

have different attention

Span

global attention on the initial tokens

Short Input span growth patterns,
homogeneous
sparse mask
? requiring appropriate length
growth rules.
[1] Fu, Tianyu, et al. "Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths." Second Conference on Language Modeling.
 MoA
Metho • Different attention heads: require searching for different sparse attention span.
• Different input lengths: require searching for suitable attention span growth rules for
d different attention heads.
Heterogeneous search space Analyze attention value importance Optimization
Construct a search space of sparse Based on gradients, analyze the impact of Formulate an optimization problem to
schemes that includes different sparse different attention values on the prediction select sparse patterns under sparsity
attention patterns and their variation results, and obtain the accuracy–sparsity constraints, minimizing the impact of
rules with sequence length. trade-off curve for different sparse schemes. sparse attention on accuracy.

[1] Fu, Tianyu, et al. "Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths." Second Conference on Language Modeling.
 MoA
Resul Increase the inference throughput by about 7×.
ts Expand the effective context length by 3.9×.
Accuracy-Throughput Needle-in-a-haystack Task Efficiency

3.9x 1.7x-1.9x
Effective context length
Throughput Improvement
Compared with VLLM on
7B and 13B LLMs using
256k
Extrapolatable input length
50% attention sparsity on
A100-80GB GPUs

Vicuna-7B, 8K input length

Long Context Understanding Task
Accuracy
Improvement
Code 7-70B models
50% sparsity

[1] Fu, Tianyu, et al. "Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths." Second Conference on Language Modeling.
 NSA
• Sparse Attention: NSA

• Use different sparse patterns and enable

Motivation Method end-to-end training.
• Applying sparsity post-hoc forces
models to deviate from their
pretrained optimization trajectory and
causes performance degradation.

1. Token Merging 2. Token Selection 3. Local

Merge multiple Retain only the most Attends to nearby
Training: Inference: tokens into coarse important tokens to tokens within a
representations. apply on fine-grained sliding window.
Dense attention Sparse attention attention.

[1] Yuan, Jingyang, et al. "Native sparse attention: Hardware-aligned and natively trainable sparse attention." arXiv preprint arXiv:2502.11089 (2025).
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm- Model Compression
Input Compression
level prompt compression, RAG • Quantization
Alternative Generative • Sparse Attention
Paradigms • Weight Pruning
• Sharing
Model Compression • Knowledge Distillation
reduce model redundancy in a
static manner

Model- Dynamic Inference Dynamic Inference

reduce model redundancy in a
level dynamic manner
• Module-granularity
Structure Design • Model-granularity
design novel structure, which often
require training
Structure Design
Operator-Level Opt.
• Mixture-of-Experts (Efficient FFN)
System- • Efficient Attention
Framework-Level Opt.
level
Hardware-Level Opt.
 Model-level: Weight Pruning
• Weight Pruning
• Remove less critical weights and structures from models
Unstructured Structured
Pruning Pruning
Granularity Individual weight values Structural units, e.g., channels,
layers, experts

Performance Low High Unstructured Pruning

loss Granularity: Weight
Actual speed-up No Yes
on hardware
Research (1) accelerate pruning process (1) decide structured pattern
directions (2) design effective pruning (2) design effective pruning
strategies (e.g., pruning metrics, metrics
pruning ratios) Structured Pruning
Granularity: Channel/Group/Layer
Representative SparseGPT, Prune and Tune, ISC, LLM-Pruner, LLaMA-Sheard,
Studies BESA ZipLM, LoRAPrune, EEP
 SparseGPT

Idea Method
• Removing weights and updating the • Incrementally prune weights in each column of the
remaining ones to compensate for the weight, using a sequence of Hessian inverses,
and updating the remainder of the weights.
error.

Category

• Type: Unstructured Pruning

• Granularity: Individual weight values

[1] Elias Frantar, et al. “SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot” ICML 2023.
 SparseGPT

Results
• High Sparsity, Low Accuracy Loss: Prunes OPT-175B to 60% sparsity in
one shot with a negligible increase in perplexity
• High Efficiency: Process the 175-billion-parameter models in under 4.5
hours, removing more than 100 billion weights.

[1] Elias Frantar, et al. “SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot” ICML 2023.
 LLM-Pruner

Idea Methods
• Identify and remove non-critical, coupled • LLM-Pruner automatically identifies and
structures. removes non-critical, coupled structures
based on gradient information, and recovers
performance using a LoRA with a small
dataset.

Category
• Type: Structured Pruning
• Granularity: Head, Channel

[1] Xinyin Ma, et al. “LLM-Pruner: On the Structural Pruning of Large Language Models” Neurips 2023.
 Channel Permutation for Better N:M Sparsity
Find a permutation strategy that preserves more important parameters under N:M sparsity

2:4 Sparsity
2 zeros out of every 4
contiguous elements

[1] Zhang, Y., Bai, H., et al. "Plug-and-play: An Efficient Post-training Pruning Method for Large Language Models". ICLR 2024.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm- Model Compression
Input Compression
level prompt compression, RAG • Quantization
Alternative Generative • Sparse Attention
Paradigms • Weight Pruning
• Sharing
Model Compression • Knowledge Distillation
reduce model redundancy in a
static manner

Model- Dynamic Inference Dynamic Inference

reduce model redundancy in a
level dynamic manner
• Module-granularity
Structure Design • Model-granularity
design novel structure, which often
require training
Structure Design
Operator-Level Opt.
• Mixture-of-Experts (Efficient FFN)
System- • Efficient Attention
Framework-Level Opt.
level
Hardware-Level Opt.
 Sharing
• Definition
Reuse parameters, states, or intermediate results across different parts of the model.
• Shared Tensor Type
Shared Tensor Type
KV Cache Weight KV Cache
Block Sharing
Weight
Sharing Subformer √
Block MobileLLM √
Dynamic layer tying √
…

LCKV √
CLA √
Block
[1] Reid, Machel, Edison Marrese-Taylor, and Yutaka Matsuo. "Subformer: Exploring weight sharing for parameter efficiency in generative transformers." arXiv preprint arXiv:2101.00234 (2021).
[2] Liu, Zechun, et al. "Mobilellm: Optimizing sub-billion parameter language models for on-device use cases." Forty-first International Conference on Machine Learning. 2024.
[3] Hay, Tamir David, and Lior Wolf. "Dynamic Layer Tying for Parameter-Efficient Transformers." The Twelfth International Conference on Learning Representations.
[4] Wu, Haoyi, and Kewei Tu. "Layer-Condensed KV Cache for Efficient Inference of Large Language Models." Proceedings of the 62nd Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers). 2024.
[5] Brandon, William, et al. "Reducing transformer key-value cache size with cross-layer attention." Advances in Neural Information Processing Systems 37 (2024): 86927-86957.
 MobileLLM
• MobileLLM

Idea Methods
• Weight sharing between two • Design three different weight-sharing strategies:
adjacent blocks avoids weight
movement, requiring only
computing the block twice
and incurring minimal latency
overhead.

Category
Immediate block- Repeat-all-over Reverse
• Type: Weight Sharing wise sharing sharing sharing

[1] Zechun Liu, et al. “MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases” ICML 2024.
 LCKV
• Layer-Condensed KV Cache

Motivation Methods
• In LCKV, all layers attend to only the top
• Existing methods focus on compressing the layer's KVs.
KV cache sequence length. • A few "warmup" layers with standard
attention are kept to maintain performance.
• This approach reduces the number of
cached layers, not just the sequence length.

[1] Haoyi Wu, et al. “Layer-Condensed KV Cache for Efficient Inference of Large Language Models” ACL 2024.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm- Model Compression
Input Compression
level prompt compression, RAG • Quantization
Alternative Generative • Sparse Attention
Paradigms • Weight Pruning
• Sharing
Model Compression • Knowledge Distillation
reduce model redundancy in a
static manner

Model- Dynamic Inference Dynamic Inference

reduce model redundancy in a
level dynamic manner
• Module-granularity
Structure Design • Model-granularity
design novel structure, which often
require training
Structure Design
Operator-Level Opt.
• Mixture-of-Experts (Efficient FFN)
System- • Efficient Attention
Framework-Level Opt.
level
Hardware-Level Opt.
 Knowledge Distillation
• Motivation
Although the compressed lightweight model achieves better hardware efficiency, its accuracy is
lower under conventional training methods. It is necessary to design training approaches to
achieve better accuracy recovery.
• Definition
Knowledge Distillation: Use a teacher model to guide the training of a student model, enabling
the student model to learn the “knowledge” of the teacher model to help improve its accuracy.

Teacher Student

The teacher model (large) helps the student

model (small) recover accuracy.
 Knowledge Distillation
• Aligning Objective
Teacher Student
Objective
Block 1. Feature Block
Feature Logits Data
Block Block TED √
MiniLLM √

…
…

GKD √
Block Block DISCO √
2. Logits MCKD √
Logits Logits
DeepSeek-R1 √
3. Data
Data Data
[1] Liang, Chen, et al. "Less is more: Task-aware layer-wise distillation for language model compression." International Conference on Machine Learning. PMLR, 2023.
[2] Gu, Yuxian, et al. "MiniLLM: Knowledge Distillation of Large Language Models." The Twelfth International Conference on Learning Representations.
[3] Agarwal, Rishabh, et al. "Gkd: Generalized knowledge distillation for auto-regressive sequence models." CoRR (2023).
[4] Chen, Zeming, et al. "DISCO: Distilling Counterfactuals with Large Language Models." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
[5] Zhao, Jiachen, et al. "Multistage collaborative knowledge distillation from large language models." arXiv preprint arXiv:2311.08640 (2023).
[6] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm- Model Compression
Input Compression
level prompt compression, RAG • Quantization
Alternative Generative • Sparse Attention
Paradigms • Weight Pruning
• Sharing
Model Compression • Knowledge Distillation
reduce model redundancy in a
static manner

Model- Dynamic Inference Dynamic Inference

reduce model redundancy in a
level dynamic manner
• Module-dimension
Structure Design • Model-dimension
design novel structure, which often
require training
Structure Design
Operator-Level Opt.
• Mixture-of-Experts (Efficient FFN)
System- • Efficient Attention
Framework-Level Opt.
level
Hardware-Level Opt.
 Model-level: Dynamic Inference
• Motivation & Definition
• During neural network inference, not all data needs to go through the same
computation flow. The core design idea of dynamic inference algorithms is to
determine the required computations based on the runtime input data.
• What is the data granularity for dynamic inference (e.g., query-level, token-level, etc.)
• Which dimensions are dynamically adjusted (e.g., layer, model)
• How to dynamically adjust the corresponding dimensions based on input data (e.g., training
a router)

MoE: Token-level dynamic module routing R2R: Token-level dynamic model routing
[1] Dynamic Neural Networks: A Survey, Han et al., IEEE Transactions on Pattern Analysis and Machine Intelligence.
 Mixture-of-Depths
• Mixture-of-Depths

Idea Methods
• In language modeling, not all tokens • At specific layers, a learned router selects the
and sequences require the same time top-k most important tokens to be processed
or effort to accurately make a by the self-attention and MLP blocks, while
prediction. other tokens bypass these computations
through a simple residual connection.

Category

• Data Granularity: token-level

• Dimension: layer
• Method: training a router

[1] David Raposo, et al. “Mixture-of-Depths: Dynamically allocating compute in transformer-based language models.” ArXiv 2024.
 RouteLLM
• RouteLLM

Motivation Methods
• Direct simpler queries to smaller • RouteLLM trains a router model on human
models and more complex ones to preference data to intelligently direct queries to
larger models to balance response either a strong, expensive LLM or a weak, cheap
quality with cost efficiency. one.

Category
• routers outperform
• Granularity:query-level random baselines
• Dimension: model
• Method: similarity-based
retrieval / training a router
[1] David Raposo, et al. “RouteLLM: Learning to Route LLMs with Preference Data.” ArXiv 2024.
 Recent Work: SLM-LLM Mix Inference (R2R)
Use small language model (SLM) and LLM for different reasoning steps
Fast but weak SLM Given same context, SLM and LLM
Motivation Insight
slow but strong LLM predictions are often identical

Tested results on AIME’24-25

Type Model Accuracy Latency

(s / question)

SLM R1-1.5B ☹ 9% ☺ 199

• 89% identical predictions
LLM R1-32B ☺ 45% ☹ 498 • 11% different predictions
5% • some are neutral, like
89 11 divergent
alternative expressions
We should selectively use SLM and LLM for %
identical %
different 6% • only few diverge the
neutral
different generation steps, constructing a fast and meaning, logic, or
strong mix-inference method conclusion of reasoning

[1] Tianyu, Fu, et al. "Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing" Submitted to NeurIPS’25. [Under Review]
 Recent Work: SLM-LLM Mix Inference (R2R)
Label divergent token, then train a neural token-router,
utilizing LLMs only for path-divergent tokens during SLM generation
Label divergent tokens Train neural router,
Method Method route to LLM for divergent SLM tokens
generate model preference training data

input: It’s
SLM: It’s 99 It’s hard, It’s hard, re It’s hard, rewrite
✔ ✔ ✔
LLM: It’s hard
output: hard , re write

Key idea: Routing scheme:

Step1. find all predictions where SLM-LLM differ We train a 56M neural router
Step2. from the difference, let LLM continue generation until the end Given SLM output token & its last-layer hidden states,
of current sentence, to understand difference’s impact it classifies whether this token is divergent,
Step3. ask another LLM to verify if difference causes divergence Immediately route to LLM if predicted as divergent

[1] Tianyu, Fu, et al. "Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing" Submitted to NeurIPS’25. [Under Review]
 Experimental Results
Mixing R1-1.5B & 32B, uing R2R with 5.6B avg. activated param. per token
achieve performance exceeding R1-14B
Performance-Efficiency Reaching 84.3 token/s
Result Demo
Pareto Frontier on two A800-80GB GPUs

same avg. parameter, better accuracy

source
code

For R2R-5.6B (mix R1-1.5B & 32B)

• Comparing R1-14B, 1.50x speedup, 1.07x accuracy
• Comparing R1-32B, 2.76x speedup, achieving 92% of
its accuracy with only 11%-15% LLM usage R1-32B R2R
Finished in: 1min 12s Finished in: 32s

[1] Tianyu, Fu, et al. "Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing" Submitted to NeurIPS’25. [Under Review]
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm- Model Compression
Input Compression
level prompt compression, RAG • Quantization
Alternative Generative • Sparse Attention
Paradigms • Weight Pruning
• Sharing
Model Compression • Knowledge Distillation
reduce model redundancy in a
static manner

Model- Dynamic Inference Dynamic Inference

reduce model redundancy in a
level dynamic manner
• Module-granularity
Structure Design • Model-granularity
design novel structure, which often
require training
Structure Design
Operator-Level Opt.
• Mixture-of-Experts (Efficient FFN)
System- • Efficient Attention
Framework-Level Opt.
level
Hardware-Level Opt.
 Model-level: Efficient Structure Design
• Efficient structure design
• Efficient FFN design: Mixture-of-Expert (MoE, one common architecture
follow the dynamic inference idea) • Expert: module to process different inputs
• Shared expert: an expert module that is always
active for every input
• Expert granularity: the FFN intermediate
hidden dimension

• Router: direct the input to the appropriate

expert networks
• Basically, each token is routed to a fixed
number of experts based on scores
produced by the router.
• In training or multi-batch inference scenarios,
load balancing among experts also needs to be
considered.[1]

[1] Mu, Siyuan, and Sen Lin. "A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications." arXiv preprint arXiv:2503.07137 (2025).
 MoE
• DeepSeekMoE

Motivation Methods
• The designated expert will intend • Segment the experts into a finer grain by
to assemble different types of splitting the FFN intermediate hidden dimension.
knowledge in its parameters, • Isolate certain experts to serve as shared
which are hard to utilize experts that are always activated.
simultaneously.
• Multiple experts may converge in
acquiring shared knowledge in
their respective parameters,
leading to redundancy in expert
parameters.

[1] Dai, Damai, et al. "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models." Proceedings of the 62nd Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers). 2024.
 Efficient Attention mechanism
• Efficient structure design
• Efficient Attention mechanism

Different heads Low-rank compression for keys

share a single set and values to reduce KV
of keys and cache.
values.
[1] Liu, Aixin, et al. "Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model." arXiv preprint arXiv:2405.04434 (2024).
 Model-level: Efficient Structure Design
• Efficient structure design
• Non-Transformer architecture design
• State Space Model (SSM)
• core idea: compress token information into hidden state

Research directions:
1. Design better parametrization or initialization strategy.
2. Design better model architecture based on SSM.
 Mamba
• Mamba
• Let the SSM parameters be
Motivation • Linear-time-invariant (LTI) Methods
SSMs cannot efficiently functions of the input,
select data in an input- allowing the model to
dependent manner. selectively propagate or
forget information
depending on the current
token.

[1] The router can direct simpler queries to smaller models and more complex ones to larger models, thereby balancing response quality with cost efficiency.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm-
Input Compression
level prompt compression, RAG
Alternative Generative
Paradigms Operator-Level Opt.
Model Compression
reduce model redundancy in a Framework-Level Opt.
static manner

Model- Dynamic Inference Hardware-Level Opt.

reduce model redundancy in a
level dynamic manner
Structure Design
design novel structure, which often
require training

Operator-Level Opt.
System-
Framework-Level Opt.
level
Hardware-Level Opt.
 Design Space
• System Design: Operator-level Optimization

AI algorithm max. 𝑢𝑡𝑖𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛,

s. t. 𝑢𝑠𝑎𝑔𝑒 ≤ 𝑟𝑒𝑠𝑜𝑢𝑟𝑐𝑒.

Model Operator-level Optimization

Software
Level

Computation graph and operators Computation

Operator
Graph

Intermediate representation
and hardware instructions
Hardware
Level Operator Computation Memory
Hardware Fusion Opt. Opt.
 Operator-level Optimization
Optimizes the hardware utilization by tailoring
Method
workload mapping to hardware specifications

FlashAttention FlashDecoding++
Add & LayerNorm
Matmul Operator （NeurIPS 22） （MLSys 24）
FC2 FC2
Main Operator of NN
Reduces the Reduces the update overhead of
Act. Act.
with Highly Optimized memory footprint attention operator and optimizes
FC1 FC1 Linear Algebra Libraries by fusing attention GEMM in decoding, achieving up
into one kernel to 4x end-to-end speedup
Add & LayerNorm

WO
cuBLAS FlashDecoding FlashInfer
Multi-head Self-Attention
Q K V （~2007-present） （arXiv 23） （MLSys 25 Best Paper）
K Cache V Cache
Attention Operator NVIDIA GPU’s Uses sequence Dynamically aware
WQ WK WV
official library partitioning for workload allocation and
Core of Transformer Achieves over 90% decoding, using unified mask
peak utilization of achieving up to 50x representation
A transformer block Tensor Cores speedup for long 13%-69% e2e speedup
texts
*Matmul: Matrix Multiplication
 Matmul Operator

• Implementation of Matmul on GPU

𝐶 0×0 = 𝐴0×0 ×𝐵 0×0
• Observation: No dependencies in C,
B
→ Parallel computation
• Method: Each thread loads a row of A and a
column of B → computes → writes to C
Computational-to-Memory Ratio A C
An important metric for measuring GPU utilization
Higher value → Higher compute unit utilization

𝑐𝑜𝑚𝑝𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝑎𝑚𝑜𝑢𝑛𝑡
Comp−to−Mem Ratio =
𝑚𝑒𝑚𝑜𝑟𝑦 𝑎𝑐𝑐𝑒𝑠𝑠 𝑎𝑚𝑜𝑢𝑛𝑡

Comp-to-Mem Ratio is only 0.25FLOP/Byte in this example

*Total computation: 2*4*4*4=128FLOP. Total memory access: 4*(2*4*4*4)=512Byte
 Matmul Operator

• Implementation of Matmul on GPU

Operator Implementation: 𝐶 0×0 = 𝐴0×0 ×𝐵 0×0
__global__ void MatMul(int *A, int *B, int *C, int width) {
//eg. width=4 B
int row = threadIdx.y;
int col = threadIdx.x;

if (row < width && col < width) {

int sum = 0;
for (int k = 0; k < width; k++) { A C
sum += A[row * width + k] * B[k * width + col];
} //Each thread processed one row of A, one column of B
C[row * width + col] = sum;
//Write result to the corresponding position of C
}
}
 Matmul Operator
• Computation Optimization: Tiling
• Motivation: Maximize GPU compute utilization 𝐶 0×0 = 𝐴0×0 ×𝐵 0×0
B
• Method: Increasing tile size.
Example: 1 thread → 2 rows of A & 2 cols of B
→ 4 elements in C

Tiling raises Comp-to-Mem Ratio to A C

0.5FLOP/Byte

⚠ Oversized tiles reduce parallelism. The

key is balance.

*Total compute: 128FLOP!Total access reduces to4*(2*4*4*4)/2=256Byte

 Matmul Operator
• Computation Optimization: Tiling
Operator Implementation: 𝐶 0×0 = 𝐴0×0 ×𝐵 0×0
__global__ void MatMulTiling(int *A, int *B, int *C, int width) {
//eg. TILE_WIDTH=2
int tx = threadIdx.x; int ty = threadIdy.y;
B
for (int i = 0; i < TILE_WIDTH; i++) {
for (int j = 0; j < TILE_WIDTH; j++) {
int row = ty * TILE_WIDTH + i;
int col = tx * TILE_WIDTH + j;
A C
int sum = 0;
for (int k = 0; k < width; k++) {
sum += A[row * width + k] * B[k * width + col];
} //Read A and B from global memory and compute
C[row * width + col] = sum;
}
} //Each thread processes 2 rows of A and 2 columns of B
}
 Matmul Operator
• Memory Optimization: Using Shared Memory
• Motivation: Enable data sharing and fast communication 𝐶 0×0 = 𝐴0×0 ×𝐵 0×0
between threads
B
• Method: Load A and B into shared memory for reuse
→ reduce global memory access B$

A C

Comp-to-Mem Ratio reaches 1FLOP/Byte A$ C2,2

Store in SMEM
(Shared memory)

*store A and 𝐵 into shared memory, total memory access reduces to 128Byte
 Matmul Operator
• Memory Optimization: Using Shared Memory
Operator Implementation: 𝐶 0×0 = 𝐴0×0 ×𝐵 0×0
__global__ void MatMul_shared(int *A, int *B, int *C, int width) {
__shared__ int sharedA[width][width]; // Declare shared memory
__shared__ int sharedB[width][width]; B
int tx = threadIdx.x; int ty = threadIdx.y;

sharedA[ty][tx] = A[ty * width + tx]; // Write to SMEM B$

sharedB[ty][tx] = B[ty * width + tx];
__syncthreads(); // Sync all threads
for (int i = 0; i < TILE_WIDTH; i++) {
for (int j = 0; j < TILE_WIDTH; j++) { A C
// Read from SMEM & compute (row/col calculation are omitted)
for (int k = 0; k < width; k++) {
sum += sharedA[row][k] * sharedB[k][col]; A$ C2,2
}
C[row * width + col] = sum;
} Store in SMEM
} (Shared memory)
}
 Matmul Operator
• Special Hardware: using Tensor Core
• Why: CUDA Cores bottleneck at large-scale matrix ops in deep learning
• Tensor Core:
• NVIDIA's dedicated DL cores. The V100 GPU uses
Tensor Cores achieves 2×
• First introduced in Volta (2017) speedup over the P100[1]

• Input: 16bit → Multiply-accumulate → Stored in 32-bit registers

32bit

32bit

32bit

32bit

32bit
+ + + +

× × × ×
A!,# "$,! A%,# "$,% A&,# "$,& A',# "$,'
16bit 16bit 16bit 16bit 16bit 16bit 16bit 16bit

[1] [Link]
 Matmul Operator
• Special Hardware: using Tensor Core
•CUDA provides mma.h and wmma API for Tensor Core operations
Pesudo code：
#include <mma.h>
__global__ void MatMul_mma(half *a, half *b, float *c, int M, int N, int K) {
// Declare fragments (16x16x16 tile size)
wmma::fragment<wmma::matrix_a, 16, 16, 16, half, wmma::row_major> a_frag;
wmma::fragment<wmma::matrix_b, 16, 16, 16, half, wmma::col_major> b_frag;
wmma::fragment<wmma::accumulator, 16, 16, 16, float> c_frag;

wmma::fill_fragment(c_frag, 0.0f); // Init accumulator

wmma::load_matrix_sync(a_frag, a, K);
wmma::load_matrix_sync(b_frag, b, N);
wmma::mma_sync(c_frag, a_frag, b_frag, c_frag); // MM on Tensor Core

wmma::store_matrix_sync(c, c_frag, N, wmma::mem_row_major); // Store result

}
 Decoder-only LLM Inference
• Decoder-only LLM Inference!Two distinct phases
• Prefill phase: Handling input prompts, saving generated K and V in KV Cache
• Decode phase: Update and use KV cache for computing Attention to generate a new token
Output: [‘Processing’] (1*dim) Output: [‘!’] (1*dim)
Add & LayerNorm Add & LayerNorm

FC2 FC2

Activation Activation

FC1 FC1

Add & LayerNorm Add & LayerNorm

WO WO

Multi-head Self-Attention Multi-head Self-Attention KV cache

Q K V Q K V
K V expands with
Cache Cache
each token
WQ WK WV WQ WK WV

Prompt: ['I', 'like', ‘natural', ‘language’] (4*dim) Prompt: [‘I’, ‘like’, ‘natural’, ‘language’, ‘Processing] (1*dim)
 Attention Operator

• Prefill Optimization: FlashAttention[1,2,3]

• FlashAttention optimizes attention computation in prefill phase
• One of the most widely adopted acceleration methods with 16.2k GitHub stars[4]

[1] Dao, Tri, et al. "Flashattention: Fast and memory-efficient exact attention with io-awareness." Advances in neural information processing systems 35 (2022): 16344-16359.
[2] Dao, Tri. "Flashattention-2: Faster attention with better parallelism and work partitioning." arXiv preprint arXiv:2307.08691 (2023).
[3] Shah, Jay, et al. "Flashattention-3: Fast and accurate attention with asynchrony and low-precision." Advances in Neural Information Processing Systems 37 (2024): 68658-68685.
[4] [Link]
 Attention Operator
• Prefill Optimization: FlashAttention[1,2,3]
• Why：Complex Attention I/O; Large activation memory
• How：Operator fusion, including fwd and bwd
• Results：2-4x speedup; memory: 𝑂 𝑁 : → 𝑂(𝑁)

Saves I/O & memory via operator fusion Tiling strategies differ between fwd/bwd passes

[1] Dao, Tri, et al. "Flashattention: Fast and memory-efficient exact attention with io-awareness." Advances in neural information processing systems 35 (2022): 16344-16359.
[2] Dao, Tri. "Flashattention-2: Faster attention with better parallelism and work partitioning." arXiv preprint arXiv:2307.08691 (2023).
[3] Shah, Jay, et al. "Flashattention-3: Fast and accurate attention with asynchrony and low-precision." Advances in Neural Information Processing Systems 37 (2024): 68658-68685.
 Attention Operator

• Decode Optimization: FlashDecoding[1]

• Why: only 1 token/step in decode → using prefill tiling strategy directly causes low GPU
utilization
SM1

tiling SM2
Prefill tiling Fully utilizes
GPU SMs
SM3

SM4
Attention map

Using prefill tiling SM1

tiling directly SM3!SM4 idle!
in decode SM2
Attention map
[1] [Link]
 Attention Operator
• Decode Optimization: FlashDecoding[1]
• Method: Increases matrix tile count to boost SM utilization, achieving higher GPU
efficiency

Using prefill SM1

tiling
tiling directly SM3!SM4 idle!
in decode SM2

SM1
tiling
SM2
FlashDecoding
SM3

Expand tiling SM4 Achieves 5-10x compute speedup

dimensions
[1] [Link]
 FlashDecoding++
Motivation Attention & Matmul operators still optimizable in LLM inference

Attention"FlashAttention[1]# Matmul"cuBLAS[2], CUTLASS[3]#

Online Softmax → Global reduction causes Inefficient tiling for “short-wide” matrices
significant update overhead in decode phase
Low Parallelism Update
Overhead K * = N
K
GEMM Shape: M, K, N
N
BN
Decode BK B1 B’1
Update phase B2 B’2
Overhead B3 B’3

…
…
Original Softmax Online Softmax B
All elements wait for global M dimension = batch
Global reduction via
reduction incremental updates
size M A1 A2 A3 … A C1 C2 C
K N
[1] Dao T, Fu D, Ermon S, et al. Flashattention: Fast and memory-efficient exact attention with io-awareness[J]. Advances in neural information processing systems, 2022, 35: 16344-16359.
[2] [Link]
[3] [Link]
 FlashDecoding++
Motivation Suboptimal implementation against various Matmul shapes

Wrong low-level implementation choice → Significant performance loss

M=32 M=3
Various
Matmul
K=4096 K=5120 … M=1024
K=1024 Shape 1: M=1, K=4096, N=4096]
N=11008 N=5120 N=1024
Shapes FastGEMV is 20% faster
than cuBLAS
Model Input
specs dynamics
…
Manual Hardware Shape 2: M=4, K=4096, N=4096]
implementatio capability
n Flat GEMM is 50% faster
than FastGEMV

Different FastGEMV Flat GEMM cuBLAS

Vector unit-based Shape-optimized General-purpose
Implementations
[1] [Link]
[2] [Link]
[3] [Link]
 FlashDecoding++
Leverage value distribution in LLM inference to
Method
optimize Attention operator[1]

Mathematical Equivalence & Remove Update Overhead via Preset Max

Numerical Feasibility
Update
Mathematical Overhead
Equivalence
Arbitrary max value Preset Max =
Precision loss only from 6
data type overflow
Llama2-7B OPT-6.7B ChatGLM2-6B

Update
99.99% 99.99% 99.99% Numerical Overhead
Feasibility
Data shows no FP32 Online softmax Async softmax
[-16.8] [6.5] [-440.2] [71.6] [-16.8] [6.5]
overflow in 99.99% Parallel but large update Parallel with zero update
cases overhead overhead
-70 -20 -10 0 10 40 -440 -40 -20 0 20 60 80 -70 -20 -10 0 10 40

[1] Ke, Hong, et al. “FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics.” Proceedings of Machine Learning and Systems. 2024.
 FlashDecoding++
Synchronized partia
Leverage value distribution in LLM inference to
Method
optimize Attention operator[1] Attention
N-1
mul1 ma
synchronized u

Async Pipeline for Acceleration Recomputation for Correctness

Remove update overhead via preset max value Fallback to online softmax on overflow detection
Attention
N-1
mul1
Overflow check: Set threshold
N- N+ unified max valu
partial softmax via preset max à threshold detection
1
"N-1#
Q#K max exp sum A#V
1"N+1# Asynchronized softm
$% Matrix/vector $%
Extra overhead
computations are sequential Overflow Under-utilized comp
*+(,-./ Fallback padding
A
012345
Async Pipeline:
67Tensor Overlap matrixCore49:;<=
Core8CUDA & vector unit usage Preset Max =
flat-shape zero
6 GEMM Update
"N-1# "N+1#
N-
$%
Q#K exp A#V N+
$% Overhea
A ! B
1
sum
1 Async softmax Online softmaxd
"#$%&'(max) （Overflow）
Preset max value load A
double
[1] Ke, Hong, et al. “FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics.” Proceedings of Machine Learning and Systems. 2024.
buffering
Flat GEMM optimiza
 FlashDecoding++
Leverage typical shapes in LLM inference to
Method
optimize Matmul operator[1]

Short-wide Matmul Bottleneck Analysis Adaptive Tiling + Double Buffering

speedup
1
Adaptive tiling & double buffering for different shapes
BN BBN BN
B B
BN NBN N
N

BK BK BKB1BKBB11 BB1 2BB

2 2B2
BBKK BBBKK1BBB21BB
B32 B
BB4233 BB4B34 B4
1 B
Tiling B’ B’ B’ B’2 B’21 B’
B 1 B’
B’
12
B’32 B’
B’43 B’4
B’1 B’B1N1 B’1 2 B’
2
N B’12 B’B’2 3 B’
B’1 B’ B’34 B’4
strategy 2
B BB BB
BK B1
B’1
BB
B2
B’2
K B1 B2 B3 B4
B’1 B’2 B’3 B’4
BB
B
idle B
GPU Boosts
idle GPU
0 utilization
utilizatio
time
Optimal BN selection for given N n

N is large: Low Parallelism N is large: Memory-bound N is large: Low Parallelism N is large: Memory-bound
Need more tiles Low Comp-to-mem ratio Fine-grained tiling on N-dim for Double buffering to
sufficient parallelism hide access latency

[1] Ke, Hong, et al. “FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics.” Proceedings of Machine Learning and Systems. 2024.
 FlashDecoding++
Leverage Matmul shape patterns in LLM inference for
Method
dynamic implementation selection[1]

Shape Variation Patterns Offline Table building &

Online heuristic selection
4 shape categories in inference with only M
dimension varies 1:GEMV 2: Flat GEMM 3: cuBLAS
Vector-unit based Optimized for small General purpose
Observation 1: M
Fixed [N, K] → only
4 combinations per Rows: varying M dim
model

Observation 2:
Only M dimension For each [N, K]
varies with input Offline: fine 2 critical
Prefill: M=total token points per [N, K]
count
Decode: M=batch size Only 4 shape Online: select based
categories! on M value
Columns: [N, K] combinations

[1] Ke, Hong, et al. “FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics.” Proceedings of Machine Learning and Systems. 2024.
 FlashDecoding++
Throughput surpasses SOTA by over 10%
Method
1.88x faster v.s. HuggingFace on average

Throughput Comparison on NV A100 GPU Comparsion on AMD MI210 GPU

1.24x/1.13x faster v.s. vLLM/ TensorRT-LLM 1.86x faster vs vLLM on average
throughput token/s
3x
2.9x
120
2.3x 2.5x
2.2x 2.2x 115.56
2.0x 98.19
2x 90
89.25 93.63
88.45
1.0x 78.69
1x 60

40.07

FlashDecoding++
HuggingFace LightLLM DeepSpeed vLLM OpenPPL TRT-LLM

*Test setup: L Llama2-7B, bs=1, 128 I/O tokens, single A100 GPU

[1] Ke, Hong, et al. “FlashDecoding++: Faster Large Language Model Inference with Asynchronization, Flat GEMM Optimization, and Heuristics.” Proceedings of Machine Learning and Systems. 2024.
 Design Space
• System Design: Framework-level Optimization

AI algorithm max. 𝑡ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡,

s. t. 𝑙𝑎𝑡𝑒𝑛𝑐𝑦 ≤ 𝑆𝐿𝑂.

Software Model Framework-level

Level Optimization

Computation graph and operators

Edge Cloud

Intermediate representation
and hardware instructions
Hardware Request Memory
Offload
Level Scheduling Optimization
Hardware
 Framework-level Optimization
Optimizing the system throughput
Method
adhering to the service-level objective (SLO)
vLLM Sarathi-Serve SGLang
batching
How to schedule? （SOSP 23） （OSDI 24） （NeurIPS 24）
Paged KV cache Mixed P/D request Prefix caching technique
Basic memory batching Reuses repeated KV
or management for LLM Basic batching method for cache across requests
Inference Engine: Parallel Strategy serving co-located systems 6x throughput
2-4x throughput 2x throughput improvement
Card0 Card0 improvement improvement
or
Request Queue Card1 Card1 Orca DistServe Mooncake
（OSDI 22） （OSDI 24） （FAST 25 Best
Decode (D) Request Completed Continuous batching P/D disaggregated Paper）
Requests Basic scheduling system
Prefill (P) Request KV cache-centric
method for LLM Specialized optimization scheduling and storage
Send inference serving for P/D instances strategy based on a
Request ~10x throughput Improves effective
disaggregated system
User 0 improvement throughput by 4.48x
design
User 1
 Offloading Techniques
• Offloading Technique: Definition and Motivation
• Definition: Offloading parts of the model (model weights, KV cache, etc.) from GPU to
other devices (e.g., CPU) for storage or even computation, to save space and improve
computational efficiency
• Motivation: The large volume of model parameters and KV cache data exceeds the
storage capacity of GPU memory

LLaMA2-13B: when the context length reaches

100k, the required KV cache reaches nearly 82GB,
exceeding the memory capacity of a single GPU
DeepSeek-V3: parameter size reaches 1250GB

[1] Memory Analysis on the Training Course of DeepSeek Models, Zhang et al. Arxiv Preprint 2502.07486.
 Offloading Techniques
• Offloading Techniques: Categories
Offloading Model Weights Offloading KV Cache Offloading Experts

Representative Work: FlexGen[1] Representative Work: FastDecode[2] Representative Work: KTransformers[3]

Adopt the Zig-zag computation order to Offload KV Cache to CPUs to relieve KTransformers offloads experts to
hide the communication time of weight the storage pressure on GPUs CPUs (detailed in the next page)
transmission during the decode phase

[1] FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, Sheng et al. Arxiv Preprint 2303.06865.
[2] FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines, He et al. Arxiv Preprint 2403.11421.
[3] [Link]
 Request Scheduling
• Request Scheduling
• Definition: Scheduling requests in different phases to GPU instances for batched
computation
• Motivation: Requests for LLM inference have varying lengths and distinct phases (Prefill
and Decode phases), and scheduling greatly impacts the performance of inference

Decode：
memory-bound
bottleneck

Prefill:
Statistical distribution of request lengths compute-bound
bottleneck
across different datasets
How to perform batch processing for
requests with varying lengths?
 Request Scheduling
• Request Scheduling: Batching
• Orca (G. Yu et al, OSDI’22) proposes the continuous batching technique, which batches
requests with varying lengths at the TOKEN granularity. Compared to request-level
batching, it improves throughput by 36.9x

Request Request
1 1
Request Request
2 2
Request Request
3 3
Request Request
4 4

time axis time axis

Request-level batching Token-level batching (Continuous batching)
The processing time depends on the longest Batching requests of different lengths by concatenating
request, leading to low utilization them in the Token dimension

[1] G. Yu, et al. ”ORCA: A Distributed Serving System for Transformer-Based Generative Models.”, OSDI, 2022.
 Memory Optimization
• Memory Optimization: PagedAttention
• The growth of KV cache and memory fragmentation limit concurrency, resulting in
suboptimal system throughput.
• vLLM (W. Kwon et al, SOSP’23) proposes PagedAttention,which stores KV cache in a
paged manner. This approach effectively eliminates memory fragmentation, and improves
throughput by 2–4 times.

KV cache accounts for a large KV cache in LLM inference grows

proportion in distributed systems. with the generation process
[1] W. Kwon, et al. ” Efficient Memory Management for Large Language Model Serving with PagedAttention.”, SOSP, 2023.
 Memory Optimization
• Memory Optimization: PagedAttention
• Memory: KV cache is stored in blocks along the sequence dimension at different
physical addresses.
• Computation: During attention computation, the physical addresses of the corresponding
KV cache are retrieved by looking up a table.

Computation: Index the KV cache addresses

Memory: Logically continuous sequences are stored
required for computation by looking up a
in blocks at actual physical addresses.
table.
[1] W. Kwon, et al. ” Efficient Memory Management for Large Language Model Serving with PagedAttention.”, SOSP, 2023.
 semi-PD
Key Co-located and disaggregated systems for Prefill/Decode have their own
Problem strengths and weaknesses in computation and storage.
Co-located Systems"vLLM, SGLang# Disaggregated Systems"Mooncake, DistServe#

Single Fused Instance Prefill Instance

Prefill Decode Both the Prefill and Decode stages
Prefill Decode +Decode Instance
Requests Requests of a request are computed and Requests Requests
After a request is completed in
stored within the same instance.
the Prefill instance, the KV
Representative cache is transmitted to the
Decode instance for computation.
Frameworks
Computatio
Computation
GPU GPU Representative
n Fusion GPU SMs
Separation SMs SMs Frameworks

Storage
Storage GPU KV GPU
Fusion GPU HBM cache
Separation HBM HBM

[1] W. Kwon, et al. ” Efficient Memory Management for Large Language Model Serving with PagedAttention.”, SOSP, 2023.
[2] L. Zheng, et al. “SGLang: Efficient Execution of Structured Language Model Programs”, NeurIPS, 2024.
[3] R. Qin, et al. “Mooncake: Trading More Storage for Less Computation”, FAST, 2025.
[4] DeepSeek Team. “DeepSeek-V3 Technical Report”. arXiv, 2024.
 semi-PD
Key Co-located and disaggregated systems for Prefill/Decode have their own
Problem strengths and weaknesses in computation and storage.

Co-located Systems"vLLM, SGLang#

Single Instance Prefill Requests Decode Mixed batching of requests
Requests
Prefill Decode request 1/2/4 request request 1/2/3/4/5
Both the Prefill and Decode phases 3/5
of a request are computed and Requests Requests
Requests at different phases wait Computational resources
stored within the same instance.
for each other scheduled by the compiler

Representativ
e Frameworks
Mixed Computational Disadvantages
computation
GPU SMs Resource contention and latency
interference between P and D

Storage Advantages
Shared No need to transfer KV cache
Storage
GPU HBM between P and D → High HBM
utilization

[1] W. Kwon, et al. ” Efficient Memory Management for Large Language Model Serving with PagedAttention.”, SOSP, 2023.
[2] L. Zheng, et al. “SGLang: Efficient Execution of Structured Language Model Programs”, NeurIPS, 2024.
 semi-PD
Key Co-located and disaggregated systems for Prefill/Decode have their own
Problem strengths and weaknesses in computation and storage.

Disaggregated Systems"Mooncake, DistServe#

Prefill Instance + Decode Computational

Instance (P/D Instances) Prefill Prefill Advantage
Requests Requests Isolated computation between P
After a request is completed in and D, with no latency
the Prefill instance, the KV interference.
cache is transmitted to the Storage Disadvantage
Transmission overhead, storage
Decode instance for
imbalance, and more...
computation.
Representativ Isolated GPU GPU
e Frameworks Computation SMs SMs Instance switching requires
transferring KV cache
Storage Imbalance D Instance
switc
P Instance
h 1
The P The D 1
KV
Isolated GPU KV GPU instance instance ca
cache ch
Storage HBM HBM sends away stores KV e D Instance
KV cache. cache for long.
2

[1] R. Qin, et al. “Mooncake: Trading More Storage for Less Computation”, FAST, 2025.
[2] DeepSeek Team. “DeepSeek-V3 Technical Report”. arXiv, 2024.
 semi-PD
Combine the computational advantage of disaggregated systems and
Method
the storage advantage of co-located systems[1]
Computation Disaggregation & Storage Unification: P/D computing resources are
isolated, but storage resources are shared

Prefill Decode
Requests Requests P/D computing resources are isolated
and divided into different processes
Computational Advantage
P/D isolated computation with no
latency interference.
Isolated
Computation GPU SMs

but share the same storage resources Storage Advantage

No need to transfer KV cache
Shared
GPU HBM between P/D; High storage
Storage utilization of HBM.

[1] Ke, Hong, et al. ”semi-PD: Towards Efficient LLM Serving via Phase-wise Computation Disaggregation and Unified Storage." arXiv preprint arXiv:2504.19867. 2025.
 semi-PD
Combine the computational advantage of disaggregated systems and the
Method
storage advantage of co-located systems[1]
Isolated Computation and Shared Storage via IPC Low-Overhead Resource Adjustment Mechanism

Resident processes manage weights and KV cache storage,

request 0/1/4 eliminating the need for reloading and copying.
Isolated GPU Prefill Process Computation Resource
Computation SMs Allocation Adjustment
request 2/3 request 1/2/4 Reload + Copy
request
Decode Process 1/2/4
Prefill Process
P/D uses asynchronous processes to achieve Before
request 3/5 Reload + Copy request 3/5
computation resource isolation at the SM level.
Decode Process

Prefill Resident process (broadcasting storage address for new process)

Shared Atomic Requests
Storage Storag Memory request
Decode After request 1/2/4
e Allocation 1/2/4
Requests Prefill Process
Atomic memory allocation is used to avoid request 3/5 request 3/5
Write-After-Read (WAR) conflicts. Decode Process
[1] Ke, Hong, et al. ”semi-PD: Towards Efficient LLM Serving via Phase-wise Computation Disaggregation and Unified Storage." arXiv preprint arXiv:2504.19867. 2025.
 semi-PD
Llama3 series models: 1.55-1.72x improvement in request service rate under
Result
given SLOs DeepSeek-V3 model: 1.49-2.58x reduction in latency

Instance-Level Inference Cluster-Level Inference

Lower Time To First Token (TTFT) and Time Per Output Token (TPOT). semi-PD instances collectively participate in request routing.

vLLM-S: vLLM + SplitFuse

vLLM-D: Default

Llama3-70B, ShareGPT, FP16, 4xA100 (vLLM, semi-PD), 8xA100 (DistServe)

xPyDzS: x Prefill instances, y Decode instances, z semi-PD instances,

implemented based on NVIDIA Dynamo!baseline", DeepSeek-V2-
Lite model, single A100 per instance

Significant reduction in TTFT/TPOT achieved after replacing 1

Prefill instance and 3 Decode instances with 4 semi-PD instances.
DeepSeek-V3, FP16, MATH-500, 8xH200

[1] Ke, Hong, et al. ”semi-PD: Towards Efficient LLM Serving via Phase-wise Computation Disaggregation and Unified Storage." arXiv preprint arXiv:2504.19867. 2025.
 Design Space
• System Design: Hardware-level Optimization
Dense computing + High-end hardware
AI algorithm

Model Duration!
Software
Level 4 cycles

High-end hardware (4 Units)

Computation graph and operators Principle of Sparse Computing
1.3×
Sparse computing + Low-end hardware Performance
Gain
Intermediate representation
and hardware instructions
Duration!
Hardware
Level 3 cycles
Hardware
Low-end hardware(2 Units)
 Hardware for Video Generation Models
Video generation is one of the important modalities,
and it is a promising path towards as the physical world’s simulators

OpenAI’s Sora model Follow-ups by major technology firms

Google

Videos generated Kuaishou

by Sora are almost
indistinguishable
from real ones.[1]
ByteDance
Source![1] OpenAI" [Link]
 Hardware for Video Generation Models
VGMs are mainly based on diffusion transformers,
composed of noising and denoising process

The structure of VGMs Diffusion

Forward diffusion (backward propagation):
Gradually add Gaussian noise of different amplitudes

Reverse diffusion (inference): Denoise gradually

The principle is to endow the model with the ability to
At present, most of the mainstream video generation generate videos through multiple rounds of noise
models adopt the Diffusion Transformer (DiT) architecture[1] superposition and denoising training.
Source![1] Scalable diffusion models with transformers"ICCV 2023.
 Hardware for Video Generation Models
Different from LLMs, the inference bottleneck of video generation models
(VGMs) has shifted from memory-bound to compute-bound

LLMs VGMs
(mainly considering the (Temporal, spatial,
decode stage) and FFN structure)

: ≈ 16𝐹𝑁𝑑 :
Computation* ≈ 12𝑑 + 2𝑁𝑑
+ 2𝐹𝑁(𝐹 + 𝑁)𝑑

Memory ≈ 16𝑑 :
≈ 12𝑑 : + 2𝑁𝑑
access* + 15𝐹𝑁𝑑
Operational
≈𝟏 ≈ 𝑭𝑵 LLMs VGMs
intensity (OI)
F: Video frames. N: Tokens. d: Hidden dimension.
*Calculation of a single block/layer.
 Hardware for Video Generation Models
H.262/MPEG-2 [1995] H.264/MPEG-4 [1996]
Developed for TV The most used video
H.261 [1988]
broadcasts and DVD- codec today, max
First modern and Video playback, max
useful codec, max resolution
resolution 1920*1080 4096*2048
resolution 352*288

H.265/HEVC [2013]
Further reduce the
MPEG-1 [1993] H.263 [1996]
video size, enables
Compressed video Low-bandwidth
technologies such as
making video CDs standard used for the
AR and VR [1]
possible internet

1000x size compression and overhead reduction!

Source![1] [Link]
 Hardware for Video Generation Models
Video Compression Important

Video with
small size
DCT
(Discrete Cosine
Unimportant
Transform)
Unimportant
Video Generation Important (INT8)
(FP16)

Generated
1. Activation sparsification 2. Hybrid precision quantization video
 Hardware for Video Generation Models
For linear-heavy video generation models, we propose FlightVGM, a HW-SW
co-design with temporal-spatial & floating-fixed strategies

Similarity Frame
Spatial-temporal
To Token
Token compression 2 Only
skip skip skip
1
Token 2 4 !
Token T
2 Token 2 1 skip
3 4

e
am
1 1 2 3 4 Token

Fr
Fixed-point DSP-Expansion Floating-point
computation (DSP-E) computation
(e.g., attention) (e.g., linear)
Max
DSP-E DSP-E
(INT8 mode) DSP (scalar) (FP16 mode)
Ø Attention dominated! resolution ↑ or dim.↓
Ø Linear dominated!resolution↓ or dim.↑ Sparse Computing + Configurable Design
→ Higher Performance
 Hardware for Video Generation Models
Main To address the computational redundancy problem in VGMs, a
Method temporal-spatial joint sparse method is proposed

Utilize Inter/Intra-frame Similarity Mining Similarities by Grouping

Step-1: Inter-frame sparsification
Similarity Table-1
F.1 F.2 F.3
T. 1 0.03 0.52 T. 1
T. 2 Ref. 0.98 0.62 T. 2
T. 3 F. 3
frame 0.96 0.36 T. 3
T. 4 F. 2
F. 1 0.63 0.99 T. 4
Step-2: Intra-frame sparsification
Similarity Table-2
T.0 Ref. token
T. 1 T.1 0.99 0 0.79
T. 2 T.2 0.22 0 0.97
T. 3 F. 3
F. 2 T.3 0.57 0.98 0
T. 4 F. 1 F. 1 F. 2 F. 3
Additional cost can be ignored
(6𝑇𝑁𝒅 vs. 𝑇𝑁𝒅𝟐 ), 𝑑=1152
[1] Liu J, Zeng S, Ding L, et al. Flightvgm: Efficient video generation model inference with online sparsification and hybrid precision on fpgas[C]//Proceedings of the 2025 ACM/SIGDA International Symposium
on Field Programmable Gate Arrays. 2025: 2-13.
 Hardware for Video Generation Models
Main A DSP58 extension architecture is proposed to enhance
Method utilization under mixed-precision computing

Floating and Fixed Hybrid Architecture Expanded Architecture based on DSP IP

A A
*
_.frac INT8 B DSP58 P INT8 B DSP58 P
INT8 INT8
C C
MUL
MUL * MUL
MUL

Max
Max Alignment
Normalization Max
Max Alignment
Normalization
X, Y
_.exp INT6
EXP INT
INT INT6
EXP INT
INT
ADD
Align ADD
ADD ADD
Align ADD
ADD x, y
FP16 mode INT8 mode - -!"
! = #×%! + '! , ) = #×%" + '" *, + = ,! , ," × !! + .! , ."
-"! -""

FP16 mode INT8 mode

(2 floating-point MACs) (4 floating-point MACs)

[1] Liu J, Zeng S, Ding L, et al. Flightvgm: Efficient video generation model inference with online sparsification and hybrid precision on fpgas[C]//Proceedings of the 2025 ACM/SIGDA International Symposium
on Field Programmable Gate Arrays. 2025: 2-13.
 Hardware for Video Generation Models
n Models and datasets
n Models!Latte-1 and Open-Sora 1.2
n Datasets!UCF-101
n Metrics
n CLIPSIM: Text-Video Alignment
n VBench: Video quality
n Baseline
n Generic hardware: NVIDIA 3090 GPU
n FPGA-based accelerator for Transformer: HiSpMV[FPGA’24] and
FlightLLM[FPGA’24]
n ASIC-based accelerator for DiT: InterArch[DAC’24] and CMC[ASPLOS’24]
 Hardware for Video Generation Models
With 21× lower peak compute than the NVIDIA 3090, FlightVGM achieves
1.3× higher speedup and 4.5× better energy efficiency

Video generated by original model

Video generated by efficient model

[1] Liu J, Zeng S, Ding L, et al. Flightvgm: Efficient video generation model inference with online sparsification and hybrid precision on fpgas[C]//Proceedings of the 2025 ACM/SIGDA International Symposium
on Field Programmable Gate Arrays. 2025: 2-13.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm-
Input Compression
level prompt compression, RAG
Efficient Output Decoding
Alternative Generative • Speculative Decoding
Paradigms • Jacobi Decoding
• Agentic Generation
Model Compression
reduce model redundancy in a
static manner Input Compression
Model- Dynamic Inference • Input Compression
reduce model redundancy in a
level dynamic manner
Alternative Generative Paradigms
Structure Design
design novel structure, which often • Diffusion for Text
require training

Operator-Level Opt.
System-
Framework-Level Opt.
level
Hardware-Level Opt.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm-
Input Compression
level prompt compression, RAG
Efficient Output Decoding
Alternative Generative • Speculative Decoding
Paradigms • Jacobi Decoding
• Agentic Generation
Model Compression
reduce model redundancy in a
static manner Input Compression
Model- Dynamic Inference • Input Compression
reduce model redundancy in a
level dynamic manner
Alternative Generative Paradigms
Structure Design
design novel structure, which often • Diffusion for Text
require training

Operator-Level Opt.
System-
Framework-Level Opt.
level
Hardware-Level Opt.
 Speculative Decoding: Concepts
● Recall: autoregressive decoding of LLMs
- Decoding stage: memory bound (data transfer of model weights & KV cache)
- Redundant computation is left unused!

Decoding stage

T1 T2 T3 …

Auto-regressive decoding
LLM

User Prompt T1 T2 T3
 Speculative Decoding: Concepts
● Core idea of speculative decoding
- Use a small draft model to generate multiple token for verification
- The LLM conducts parallel verification (memory bound allows more computation)
- Key elements: 1) the acceptance rate of generated tokens; 2) the cost of draft model

Context T1 T2 T3 T4 Context T1 T2 T3 T4

Auto-regressive decoding Parallel verification

Draft Model LLM

Context T1 T2 T3 Context T1 T2 T3 T4

[1] Leviathan, Yaniv, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. ICML 2023.
 Speculative Decoding: Demo
● Demo from [1]
- Green: accepted tokens
- Red: rejected tokens
- Blue: corrected tokens

[1] Leviathan, Yaniv, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. ICML 2023.
 Speculative Decoding: Speed-up Estimation
● The speed-up rate of speculative decoding (SD) can be estimated!
- S: the total number of tokens
- R: the number of SD rounds
- γ: the number of generated tokens in each SD round

① ② ③

① The relative latency of draft model to the target model

② The cost of multi-token verification. A large batch size B is harmful to speed-up rate

③ The negligible cost of token sampling for the rejected tokens

[1] Sadhukhan, Ranajoy, et al. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. ICLR 2025.
[2] Huang, Zongle, et al. MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE. arXiv preprint arXiv:2505.19645 (2025).
 Speculative Decoding: Representative Works
● How to find an good draft model?
- consistent with the target model
- efficient in decoding

23-9: Medusa [1] 24-1: Eagle [2] 24-10: LayerSkip [3] 24-12: DeepSeek-V3 [4]

[Draft model] Independently [Draft model] A single transformer [Draft model] The first a few [Draft model] Independent multi-layer
trained multi-layer decoders layer taking the output from LLM layers of the LLM itself perception pre-trained together with DS-V3

[1] Tianle Cai, et. al. Medusa: Simple llm inference acceleration framework with multiple decoding heads. ICML 2024.
[2] Yuhui Li, et. al, EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, ICML 2024
[3] Mostafa Elhoushi, et. al. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding, arXiv preprint, 2024
[4] Deepseek Team. DeepSeek-V3 Technical Report.
 Speculative Decoding: Eagle
Drat model: a single transformer layer
- # params: 0.25B ~ 1B
- # training data: 1B tokens
- Acceptance rate: 75%

Tree attention: more tokens per pass

- Verify more tokens per pass
- E.g., 12 tokens (4 paths)

[1] Yuhui Li, et. al, EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, ICML 2024
[2] Yuhui Li, et, al, Eagle-2: Faster inference of language models with dynamic draft trees. EMNLP 2024
[3] Yuhui Li, et. al, EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, arXiv preprint 2503.01840.
 Speculative Decoding: Multi-Token Prediction
● Multi-Token Prediction (MTP): trained from scratch with the LLM backbone

[1] Deepseek Team. DeepSeek-V3 Technical Report.

 Speculative Decoding for MoE Architectures
● SD is hard to tackle MoE:
● MoE does not favor small batch: Additional memory costs to load experts
● SD does not favor large batch: SD is beneficial for memory-bound systems
● SD + MoE is helpful under medium batch size
● The number of activated experts saturates, but does not reach compute-bound

[1] Huang, Zongle, et al. MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE. NeurIPS 2025 spotlight.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm-
Input Compression
level prompt compression, RAG
Efficient Output Decoding
Alternative Generative • Speculative Decoding
Paradigms • Jacobi Decoding
• Agentic Generation
Model Compression
reduce model redundancy in a
static manner Input Compression
Model- Dynamic Inference • Input Compression
reduce model redundancy in a
level dynamic manner
Alternative Generative Paradigms
Structure Design
design novel structure, which often • Diffusion for Text
require training

Operator-Level Opt.
System-
Framework-Level Opt.
level
Hardware-Level Opt.
 Jacobian Decoding

Drawbacks of speculative decoding

• Low acceptance rate, decoding time etc.
• A draft model needs to be separately trained and paired with the target LLM
Jacobian decoding:
A single LLM without
the draft model

[1] [Link]
 Lookahead Decoding
“Lookahead”: reuse the promising draft from past N-gram trajectories

[1] [Link]
 Lookahead Decoding

Lookahead branch
maintains a fixed-sized, 2D Verification branch selects
window to generate n-grams and verifies promising n-gram
from the Jacobi iteration candidates.
trajectory.

[1] [Link]
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm-
Input Compression
level prompt compression, RAG
Efficient Output Decoding
Alternative Generative • Speculative Decoding
Paradigms • Jacobi Decoding
• Agentic Generation
Model Compression
reduce model redundancy in a
static manner Input Compression
Model- Dynamic Inference • Input Compression
reduce model redundancy in a
level dynamic manner
Alternative Generative Paradigms
Structure Design
design novel structure, which often • Diffusion for Text
require training

Operator-Level Opt.
System-
Framework-Level Opt.
level
Hardware-Level Opt.
 Skeleton-of-Thought (SoT)
SoT: LLM generates the skeleton autoregressively, and then each points in parallel
(an attempt in agentic generation for efficiency)
• Skeleton Stage: Guide the LLM to output a
concise skeleton of the answer
• Point-expanding Stage: Guide the LLM to
expand on each point from the skeleton in
parallel
• Achieve up to 2.39x end-to-end speed-up

SoT in practice: A router to classify queries (SoT-R)

[1] Ning, Xuefei*, Zinan Lin*, et. al., ”Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation." ICLR 2024.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm-
Input Compression
level prompt compression, RAG
Efficient Output Decoding
Alternative Generative • Speculative Decoding
Paradigms • Jacobi Decoding
• Agentic Generation
Model Compression
reduce model redundancy in a
static manner Input Compression
Model- Dynamic Inference • Input Compression
reduce model redundancy in a
level dynamic manner
Alternative Generative Paradigms
Structure Design
design novel structure, which often • Diffusion for Text
require training

Operator-Level Opt.
System-
Framework-Level Opt.
level
Hardware-Level Opt.
 Prompt Compression
● Prompt compression: eliminate redundant tokens in the prompt
Selective Context: filter out redundant tokens to shorten the input prompt

1. Identify the token importance

2. Group tokens to units

3. Sort units in the descending order

4. Keep units above the threshold

[1] Li Y, Dong B, Guerin F, et al. Compressing Context to Enhance Inference Efficiency of Large Language Models. EMNLP, 2023.
 LLMLingua
Compress the prompt with an small language model, with an reduction rate up to 20x.

0. Distribution alignment
Instruction tuning of small LLM

1. Budget controller
Calculate the token-wise perplexity and
sort in the descending order

2. Iterative prompt compression

Group tokens into segments to ensure
their dependency, and compute
segment-wise perplexities

[1] Huiqiang Jiang, et al. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation. EMNLP, 2023.
 LLMLingua-2
A better way to construct training data for the compressor: instructed by GPT-4

[1] Pan Z, Wu Q, Jiang H, et al. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression[J]. arXiv preprint arXiv:2403.12968, 2024.
 AutoCompressor
The LLM learns to summarize history context given the instruction <summary_token>

The context only consists of summary tokens

The LLM will generate summarized history

<summary_token>: A new token to tell the
LLM to summarize the context

[1] Chevalier A, Wettig A, Ajith A, et al. Adapting Language Models to Compress Contexts. EMNLP, 2023.
 RECOMP
● Prompt compression for RAG systems
● Otherwise, the retrieved documents can be extremely long

Ultra-long context!

Abstractive Compressor: summarize text from the documents

Extractive Compressor: extract text from the documents
 Context Folding for Agentic AI
● Context Folding: prompt compression for Agentic AI
● Deep research, coding agent, etc.

Key idea: A LLM to summarize the history

whenever the context reaches some threshold

[1] [Link]
 Context Folding for Agentic AI
● Context Folding: prompt compression for Agentic AI

1. Learn to create, solve and

summarize sub-tasks

2. Reduce the output context

from sub-task result

3. SFT/RL training to enable

the ability to fold context

4. Context reduction 10x

without accuracy drop on
SWE & BrowseComp

[1] Weiwei Sun, et al. Scaling Long-Horizon LLM Agent via Context-Folding. arXiv,2510.11967.
 Menu of Techniques
Efficient Output Decoding
parallel generation, verification, or
refinement strategies
Algorithm-
Input Compression
level prompt compression, RAG
Efficient Output Decoding
Alternative Generative • Speculative Decoding
Paradigms • Jacobi Decoding
• Agentic Generation
Model Compression
reduce model redundancy in a
static manner Input Compression
Model- Dynamic Inference • Input Compression
reduce model redundancy in a
level dynamic manner
Alternative Generative Paradigms
Structure Design
design novel structure, which often • Diffusion for Text
require training

Operator-Level Opt.
System-
Framework-Level Opt.
level
Hardware-Level Opt.
 Diffusion Language Models
Any other solutions? —— Don’t use the autoregressive model?
—— To fully parallelized input & output, diffusion language models

Completion Infiliing

[1] Jiacheng Ye, et al. Dream 7B: Diffusion Large Language Models. arXiv preprint, 2508.15487.
 Diffusion Language Models
● Autoregressive modeling v.s. Diffusion modeling

[1] Jiacheng Ye, et al. Dream 7B: Diffusion Large Language Models. arXiv preprint, 2508.15487.
[2] Shen Nie, et al. Large Language Diffusion Models. NeurIPS 2025.
 Diffusion Language Models
● Inference latency of diffusion models
 Contents
Model-Level
1 Background 5 Optimization

2 Preliminary System-Level
≈
6 Optimization

3 Problem Definition & Algo-Level

Conceptual Analysis 7 Optimization

4 Practical Pipeline 8 Conclusion

 Tutorial Review
● Motivation: Scaling up model / data / computation based on Transformer is the
mainstream and effective pathway for stronger generative AI till now. The
scaling up of the model size and input & output cause efficiency issues.

● Preliminary:
● Most LLMs use autoregressive model as the generative modeling method, the
transformer architecture, in which the attention operation is a core mechanism.
● We introduce basic concepts of software, hardware system, device, chip,
microarchitecture, and the interface between software & hardware – instruction.
● AI inference is seen as forwarding data on a computational graph, where each node
represents a single operator, edge represents dependency. Operators are translated
to instructions. Hardware execute instructions.
 Tutorial Review
● Problem Definition: Usually, latency, memory, energy consumption and
throughput will be the ultimate objective or constraint on “efficiency”. In the
meantime, the intelligence level of AI needs to be retained.
● Measured metrics are actually tested on platform (thus is platform-related), and
directly correspond to the objectives / constraints.
● Proxy metrics are estimated with only model specification. In practice, they are
useful in diagnosis of the bottleneck and estimation of measured metrics.

● Practical Pipeline: We can estimate bottleneck modules and overall objectives,

whether each module is compute or memory bound using some simple method
(e.g., roofline model), then we can actually profile them (NVIDIA GPU: Nsight
system & compute). Finally, we design method accordingly.
 Tutorial Review
Optimization Ideas Techniques Optimization Space
Idea 1: Parallelize the sequential Efficient Output Decoding
sampling of existing model? parallel generation, verification, or
Algori- refinement strategies There is no representation/space or
optimization formalization general
thm Idea 2: Compress the input Input Compression to many work. The design of these
modify the context to shorter one? prompt compression, RAG methods directly change a core application
algorithm or algorithm property to improve efficiency.
Idea 3: Don’t use autoregressive Alternative Generative
model? Paradigms

Idea 1: Remove redundant Model Compression

params/acts/computation? reduce model redundancy in a
static manner
Static (model compression) or • Model Structure (e.g., #layer, #channel)
Model dynamic (dynamic inference) Dynamic Inference • Value Representation (e.g., low-bit
modify the reduce model redundancy in a representation)
model Idea 2: Design novel dynamic manner
lightweight structure (e.g., Structure Design
efficient FFN & attention) design novel structure, which often
require training
Focus 1: compiler/runtime
lib/hardware for efficient NN • Computational Graph (e.g., fusion)
System Operator-Level Opt. • Kernel Implementation
execution
modify the • Request scheduling, resource
software Focus 2: request scheduling / Framework-Level Opt. management, model placement
resource management for • Framework Implementation
Hardware-Level Opt. • Hardware Implementation
service-level objectives
 Future Directions
● Application Requirements
● Long context (e.g., for complex reasoning)
● Multi-modality input/output
● Multi-model agentic application

● Recent Active Directions

● Algorithm-level
● Agentic generation (multi-agent collaboration)
● Agent context engineering
● Model-level
● Latent reasoning
● Efficient architecture design
● System-level
● Agent infrastructure
 Future Directions
Higher precision
Higher compression rate: lower bits
Hardware-friendly algorithms
…

Data-mixture of QAT
Compression of reasoning models
Training objectives v.s. LLM origins
RL with quantization

New format: MXFP4 or NVFP4?

The off-policy effect by
quantization
Efficient agentic LLMs Efficient context engineering
 Thank You !

Xuefei Ning1, Guohao Dai2,4, Haoli Bai3, Lu Hou3, Yu Wang1, Qun Liu3
1Tsinghua University 2Shanghai Jiao Tong University 3Huawei 4Infinigence-AI

Tutorial Website
[Link]