Knowledge Distillation

Model Compression

Thanh Huy (MSc) – TA

Year 2026 Code-Data Quiz-Feedback
Thien Nguyen – STA
 Getting Started
❖ Content

Our objectives:
• Discuss about the motivation and definition of
Knowledge Distillation technique.
• Some current applications of Knowledge
Distillation.
• Basic approaches for Knowledge Distillation.
• Investigate how to implement the Response-
based Knowledge Distillation method using
PyTorch.

2
 Outline
❖ Introduction
❖ Knowledge Distillation
❖ Response-based Knowledge
❖ Question

3
Introduction

4
 Introduction
❖ Getting Started

DL Model
DL Model

Small Deep Learning (DL) model. Large Deep Learning (DL) model.

In Deep Learning, there are models differ in size. In theory, the larger the model, the better the performance.

5
 Introduction
❖ Challenges & Motivation in Deploying Large Language Models (LLMs)
● LLMs are becoming increasingly popular
○ For example: GPT-3 has 175B parameters
and large ensemble models.
• Computationally heavy.
• resource-intensive.
• hard to deploy in real-world applications. Table: GPT-3 Model Sizes and Parameters. Source: link.

Subset Model 1

Subset Model 2 predictions

Dataset

Subset Model 3

6
Figure: Ensemble Models Illustration.
 Introduction
❖ Challenges & Motivation in Deploying Large Language Models (LLMs)
● Conflict between training and deployment stages
? How can we maintain high performance
○ Training: Requires large resources (RAM, GPU), no real-time constraints. in large models while ensuring practical
deployment?
○ Deployment: Demands high-speed processing and resource efficiency.

Figure: Deployment requirements. Source: link.

7
 Introduction
❖ Model Compression – A Solution to Simplify Neural Networks
● Objective: Reduce the complexity of neural networks while maintaining high accuracy.
● Main Techniques:
○ Quantization.
○ Weights Pruning.
○ Knowledge Distillation.
○ High-performance libraries.

Figure: Model compression. Source: link.

8
 Introduction
❖ Learning from Nature: A Simple Machine Learning Approach
● Biological Example: ● Knowledge Distillation Method:

Knowledge
Transfer

Teacher Model Student Model

Data

Figure: Butterfly Life Cycle. Source: link. Figure: Simple Visualization of Knowledge
Distillation Method.
9
 Introduction
❖ Introduction to Knowledge Distillation
Knowledge Distillation is a technique that allows transferring knowledge from large, computationally expensive
models to smaller models without losing effectiveness.

soft labels
predictions

Teacher Model
pre-trained - deep/large neural network Knowledge
Training Transfer
Data

hard labels
predictions true labels

Student Model

to be trained - light/small neural network

10
 Introduction
❖Survey: Knowledge Distillation Applications

Figure: The applications of knowledge distillation. Source: link.

11
 Introduction
❖Knowledge Distillation: A Survey

Figure: The schematic structure of knowledge distillation. Source: link.

12
 Introduction
❖ Survey: Knowledge Distillation Research (2015–2024)

2015 Hinton et al. - Distilling the Knowledge in a Neural Network.

2016 Kim et al. - Sequence-Level Knowledge Distillation.

2019 Jiao et al. - TinyBERT: Distilling BERT for Natural Language Understanding.

2021 Anderson et al. - Compressing Visual-linguistic Model via Knowledge Distillation.

2023 Gu et al. - MiniLLM: Knowledge Distillation of Large Language Models.

2024 Ko et al. - DistiLLM: Towards Streamlined Distillation for Large Language Models.

13
 Introduction
❖ Survey: Knowledge Distillation Research (2015–2024)

2015 Hinton et al. - Distilling the Knowledge in a Neural Network. Established KD as a method to transfer
First introduced the concept of Knowledge Distillation. knowledge from a large "teacher" model to a
smaller "student" model via soft logits.
2016

2019

Table: Frame classification accuracy and WER showing that the distilled
2021 single model performs about as well as the averaged predictions of 10 models
that were used to create the soft targets. Source: link.

2023

2024

14
 Introduction
❖Survey: Knowledge Distillation Research (2015–2024)

Extended KD to sequence-to-sequence learning,

2016 Kim et al. - Sequence-Level Knowledge Distillation. improving tasks like neural machine translation
Pioneered sequence-level KD for natural language processing tasks. through token-level and sequence-level distillation.

2019

2021

2023

2024

Figure: Overview of the different knowledge distillation approaches. Source: link.

15
 Introduction
❖Survey: Knowledge Distillation Research (2015–2024)

2019 Jiao et al. - TinyBERT: Distilling BERT for Natural First to use layer-wise distillation to create a compact
Language Understanding. version of BERT, enabling NLP applications on low-
Contribution to NLP applications applied KD. resource devices.
2021

2023

2024

Figure: The illustration of TinyBERT learning. Source: link. Figure: TinyBERT application visualization.

16
 Introduction
❖Survey: Knowledge Distillation Research (2015–2024)

2021 Anderson et al. - Compressing Visual-linguistic Model via Adapted KD to multimodal tasks such as Visual
Question Answering (VQA) and image captioning,
Knowledge Distillation.
improving efficiency and scalability.
First to apply KD to visual-linguistic models.
2023

2024

Figure: Overview of proposed VL distillation schema. Source: link.

17
 Introduction
❖Survey: Knowledge Distillation Research (2015–2024)

2023 Gu et al. - MiniLLM: Knowledge Distillation of Large Applied KD to compress large language models
(LLMs) while maintaining high performance,
Language Models.
facilitating deployment on edge devices.
MiniLLM enhances SeqKD by improving efficiency and
addressing data sparsity for LLMs.
2024

Figure: The comparison of MINILLM with the sequence-level KD (SeqKD).

Source: link.
18
 Introduction
❖Survey: Knowledge Distillation Research (2015–2024)

2024 Ko et al. - DistiLLM: Towards Streamlined Distillation Introduced adaptive techniques to optimize
inference and memory efficiency, enabling virtual
for Large Language Models.
assistants and real-time NLP systems.
First to refine KD pipelines for real-time LLM applications.

Figure: Adaptive Off-Policy Approaches. Source: link. Figure: DistiLLM applications visualization.

19
Knowledge Distillation

20
 Knowledge Distillation
❖ Introduction

Figure: The generic teacher-student framework for knowledge distillation. Source: link.
21
 Knowledge Distillation
❖ Introduction
KD uses different types of knowledge:

● Response-based: Model outputs.

● Feature-based: Intermediate layers.

● Relation-based: Relationships between data points.

Figure: The schematic illustrations of sources of response-based

knowledge, feature-based knowledge and relation-based knowledge in a
deep teacher network.. Source: link.

22
 Knowledge Distillation
❖ Introduction
KD uses different types of knowledge:

● Response-based: Model outputs.

Teacher Model Logits

Input Data Distillation Loss

Student Model Logits

Figure: Response-based Knowledge.

23
 Knowledge Distillation
❖ Introduction to Knowledge Distillation

soft labels
Teacher’s
Training predictions
data

Teacher Model

pre-trained - deep neural network Knowledge

Transfer

hard labels
Student’s predictions true labels
Training
data
Student Model

to be trained - light neural network

24
 Knowledge Distillation
❖ Introduction
KD uses different types of knowledge:

● Feature-based: Intermediate layers.

Teacher Model

Layer 1 Layer 2 ..… Layer n Logits

Input Data Distillation Loss

Layer 1 Layer 2 ..… Layer n Logits

Student Model

Figure: Feature-based Knowledge. 25

 Knowledge Distillation
❖ Introduction
KD uses different types of knowledge:

● Relation-based: Relationships between data points.

Instance Relations
Teacher …

Input Data Distillation Loss

Student …
Instance Relations

Figure: Relation-based Knowledge.

26
Response-based Knowledge

27
 Response-based Knowledge
❖ Introduction
Response-based Knowledge Distillation (KD): Utilizes the outputs (logits) of the Teacher model to train the Student
model. This is the most common and straightforward method in KD.

Teacher Model Logits

Input Data Distillation Loss

Student Model Logits

Figure: Response-based Knowledge.

28
 Response-based Knowledge
❖ Problem: How to distill the knowledge

Teacher Output. Student Output.

Given knowledge from teacher, how can we
transfer it into the student network?

29
 Response-based Knowledge
❖ Distillation Loss

where

Teacher Model Logits

formula

Input
Distillation Loss
Data
Idea: Let the student try to align
Student Model Logits its logits with the teacher’s logits
through loss function.
Figure: The generic response-based knowledge distillation. Source: link.
30
 Response-based Knowledge
❖ Distillation loss: KL-Divergence

Kullback-Leibler Divergence (KL-Divergence):

A tool from information theory that measures how
>< one probability distribution differs from another. In
the case of this knowledge distillation approach, it
helps the student model align its predictions with
the teacher's knowledge.

How to mearsure the difference between two distrubutions?

31
 Response-based Knowledge
❖ Distillation loss: KL-Divergence

• Following wikipedia, the KL-Divergence is define as:

• When applying to the context of Knowledge Distillation (KD), this formula can be written as:

32
 Response-based Knowledge
❖ How to distill the knowledge: Generalization

When training a machine learning model, it is often

exptected to have a well-generalized model that can work Given the original output of the teacher with sharp
well on unseen data. probabilities, the student (in theory) can mostly likely
be able to learn only which class is correct1.

33
1 [Link]
 Response-based Knowledge
❖ How to distill the knowledge: Generalization

Teacher Original Output. Teacher Scaled Output.

Idea1: If somehow we can scale (soften) the teacher output, the relationship between classes is revealed.
34
1 [Link]
 Response-based Knowledge
❖ How to distill the knowledge: Temperature Scaling

Temperature Scaling: A technique used in machine learning to

adjust the sharpness of a model's output probabilities by dividing
the logits by a temperature parameter 𝑇 > 1 before applying the
Softmax function.

This process softens the probability distribution, making it less

confident and highlighting relationships between different
outcomes, which can be useful for tasks requiring probabilistic
interpretation or better generalization.

35
 Response-based Knowledge
❖ Softmax with Temperature • T = 1: Standard Softmax.
• T>1: Softmax reduces the gap between the top-1
class probability and others, decreasing top-1 and
increasing the rest.

With T = 1: With T = 2 (T>1):

increase

Top-1 decrease

increase

36
 Response-based Knowledge
❖ Softmax with Temperature

Figure: Illustration of Temperature Scaling technique in different temperature value. 37

 Response-based Knowledge
❖ Distillation loss: Apply temperature scaling • T = 1: Standard Softmax.
• T>1: Softmax reduces the gap between the top-1
class probability and others, decreasing top-1 and
increasing the rest.

N = Number of samples in a batch

(batch size).
38
 Response-based Knowledge
❖ Response-based Knowledge Distillation Loss
• Soft Targets Loss:

• Hard Targets Loss:

• Combined Loss (Final Loss):

39
 Response-based Knowledge
❖ Distillation Loss

where

Teacher Model Logits extended formula

Input
Distillation Loss
Data

Student Model Logits

Figure: The generic response-based knowledge distillation. Source: link. 40

 Response-based Knowledge
❖ Coding Introduction
Description: Given an image dataset about weather, build a weather classification model using ResNet and then
enhance with the Knowledge Distillation technique.

41
 Response-based Knowledge
❖ Weather Dataset

11 Classes & 6862 images

hail: 591
glaze: 639
rime: 1160
lightning: 377
rainbow: 232
fogsmog: 851
rain: 526
sandstorm: 692
dew: 698
frost: 475
snow: 621

42
 Response-based Knowledge
❖ Weather Dataset

• Train: 4,802 samples.

• Val: 1,373 samples.
• Test: 687 samples.
• Size: 224 x 224.
• Image type: RGB.

43
 Response-based Knowledge
❖ Knowledge Distillation Scenarios

Knowledge Knowledge
Transfer Transfer

Teacher Model Student Model Teacher Model Student Model

(already pre-
trained on
ImageNet)

Data Data

Case 1: Teacher and Student are trained on the same data. Case 2: Teacher and Student are trained on different data.

44
 Response-based Knowledge
❖ Pipeline Case 1
Dataset

Train Student Train Teacher

Train new Student

with KD Methods

Evaluation
45
 Response-based Knowledge
❖ Pipeline Case 2
Dataset

Train Student Trained Teacher

Train new Student

with KD Methods

Evaluation
46
 Response-based Knowledge
❖ ResNet Models

47
 Response-based Knowledge
❖ Student

n
cl
as
se
CE Loss

s
Figure: Example with ResNet18.

48
 Response-based Knowledge
❖ Teacher

Example for ResNet

n
cl
as
se
s
CE Loss

Figure: Example with ResNet34.

49
 Response-based Knowledge
❖ Pipeline Case 1
Dataset

Train Student Train Teacher

Train new Student

with KD Methods

Evaluation
50
 Response-based Knowledge
❖ Coding Step 1: Import libraries and set random seed

51
 Response-based Knowledge
❖ Coding Step 2: Read dataset

Each folder name is also a class name, thus

we could get list of classnames by reading
over them.

52
 Response-based Knowledge
❖ Coding Step 2: Read dataset

We get all image paths as well as

their corresponding label.

53
 Response-based Knowledge
❖ Coding Step 3: Train, val, test split

54
 Response-based Knowledge
❖ Coding Step 4: Create PyTorch Dataset

Declare train, val, test

datasets object.

55
 Response-based Knowledge
❖ Coding Step 5: Create DataLoader
We must also declare the
value of batch size for
training and testing.

DataLoader is an iterator,
so we can get a sample
(image_path, label) and
visualize it.

56
 Response-based Knowledge
❖ Coding Step 6: ResNet Architecture

57
 Response-based Knowledge
❖ Coding Step 6: Implement Residual Block

weight layer

relu
identity
weight layer

+
relu

Figure: Residual Block.

58
 Response-based Knowledge
❖ Coding Step 6: Implement Residual Block

weight layer

relu
identity
weight layer

+
relu

Figure: Residual Block.

59
 Response-based Knowledge
❖ Coding Step 7: Implement ResNet

60
 Response-based Knowledge
❖ Coding Step 7: Implement ResNet

61
 Response-based Knowledge
❖ Coding Step 7: Implement ResNet

62
 Response-based Knowledge
❖ Coding Step 7: Implement ResNet

After initialize the model, we

should whether it works
correctly or not.

63
 Response-based Knowledge
❖ Coding Step 7: ResNet for Teacher and Student

Student Model: ResNet18. Teacher Model: ResNet18.

64
 Response-based Knowledge
❖ Coding Step 8: Implement training function

65
 Response-based Knowledge
❖ Coding Step 9: Training

66
 Response-based Knowledge
❖ Coding Step 10: Evaluation

67
 Response-based Knowledge
❖ Results

Student
Val accuracy: 0.7283
Test accuracy: 0.7278

Teacher
Val accuracy: 0.7655
Test accuracy: 0.7540

68
 Response-based Knowledge
❖ Pipeline Case 1
Dataset

Train Student Train Teacher

Train new Student

with KD Methods

Evaluation
69
 Response-based Knowledge
❖ Train KD Student

n
cl
as
se
s
CE Loss

70
 Response-based Knowledge
❖ Train KD Student

n
cl
as
se
s
CE Loss

Soft Target
Loss

n
cl
as
se
s
71
 Response-based Knowledge
❖ Train KD Student

n
cl
as
se
s
CE Loss

Soft Target
Loss

Total Loss

n
cl
as
se
s
72
 Response-based Knowledge
❖ Coding Step 1: Initialize Teacher and New Student

Knowledge
Transfer

Teacher Model New Student

(trained in Model
Weather Dataset)

73
 Response-based Knowledge
❖ Coding Step 2: Implement training function

74
 Response-based Knowledge
❖ Results

Student
Val accuracy: 0.7691
Test accuracy: 0.7467

KD with Teacher
on the same dataset
Val accuracy: 0.7865
Test accuracy: 0.7758

75
 Response-based Knowledge
❖ Pipeline Case 2
Dataset

Train Student Train Teacher

Train new Student

with KD Methods

Evaluation
76
 Response-based Knowledge
❖ Initialize Teacher (pre-trained on ImageNet) and New Student

Knowledge
Transfer

Teacher Model New Student

(already pre- Model
trained on
ImageNet)

Loading pre-trained ResNet34 on ImageNet using timm library.

77
 Response-based Knowledge
❖ Results

Student KD with Teacher

Val accuracy: 0.7691 pretrained on ImageNet
Test accuracy: 0.7467 Val accuracy: 0.7552
Test accuracy: 0.7234

KD with Teacher
Teacher on the same dataset
Val accuracy: 0.7669
Val accuracy: 0.7865
Test accuracy: 0.7583
Test accuracy: 0.7758

78
 Response-based Knowledge
❖ Results

Method Config Val Acc Test Acc Memory FLOPS

Student only [2, 2, 2, 2] 0.7691 0.7467 11.19 M 3.66 GFLOPs
Teacher only (1) [3, 4, 6, 3] 0.7669 0.7583 21.3 M 7.36 GFLOPs
Student KD [2, 2, 2, 2] 0.7552 0.7234 11.19 M 3.66 GFLOPs
Teacher (ImageNet)
Student KD [2, 2, 2, 2] 0.7865 0.7758 11.19 M 3.66 GFLOPs
Teacher (1)

79
QUIZ

80
Summarization and Q&A

81
 Summarization and Q&A
❖ Summarization

In this section, we have discussed about:

• Discuss about the motivation and definition of
Knowledge Distillation technique.
• Some current challenges and applications of
Knowledge Distillation.
• Basic approaches for Knowledge Distillation.
• Investigate how to implement the Response-
based Knowledge Distillation method using
PyTorch.

82
Question

?
83
84
 Quantization
Model Compression Series

MSc Nguyen Quoc Thai

Code-Data Quiz-Feedback STA To Phat Dat
 Objectives

Introduction to Quantization Quantization in Neural Network

v Motivation v Neural Network Quantization Methods
v Quantization v Post Training Quantization
v Floating-point Representation v Quantization Aware Training
v Uniform vs Non-Uniform Quantization v Experimentation

2
 Outline
FP16 S 5 bits 10 bits
SECTION 1

Introduction to Quantization Bfloat 16 S 8 bits 7 bits

SECTION 3

Quantization in NN

3
 Motivation
! Problem
v The growth of deep learning model size

The growth of deep learning model size: NLP vs. Computer vision (2014 - 2021)
4
 Motivation
! Problem
But how about memory?

Correlation between model scale and performance: Increasing the number of parameters leads to
superior problem-solving abilities in Math, Coding, and Science.
5
 Motivation
! Why do we need Quantization?
v The growth of deep learning model size
Model Size VRAM Needed (FP32)
v During model inference, we need to load all
GPT-2-0.1B 0.1B 0.5GB
parameters into memory
Tiny Llama-1.1B-Chat 1.1B 4.6GB
v Hardware Constraints: Standard PCs and Mistral-7B-v0.2 7B 33GB
smartphones cannot accommodate the high Llama-3-70B-Instruct 70B 311GB
memory footprint of modern LLMs Mistral-8x22B-v0.1 141B 631GB

Bloom-176B 176B 787GB

LLM model size and VRAM needed

We need to compress the model

6
 Motivation
! Why do we need Quantization?

Accessibility: Compressed models can run on smaller devices.

Efficiency: Smaller models require less computational power.

Compressed model

Faster Inference: Compressed models improve real-time

applications.

7
 Motivation
! Why do we need Quantization?

Design Systems

Our focus today!

Improve the efficiency of

Quantization
Neural Network

Distillation

Pruning

Compression Methods 8
 Quantization
! Definition
v Quantization aims to reduce the total amount of bits required to represent each parameter, usually by converting
floating-point numbers into integers
v Quantization can also speed up computation, as working with smaller data types is faster.

Floating Point Integer

-0.2 1 0.3 1 3 2
Quantization
0.1 -0.6 -0.7 1 0 0

1.2 0.4 0 3 2 1

32 bit 8 bit
9
 Quantization
! Quantized Model Size

VRAM Needed
Model Size
FP32 FP16 INT8

GPT-2-0.1B 0.1B 0.56 GB 0.28 GB 0.14 GB

Tiny Llama-1.1B-Chat 1.1B 4.6 GB 2.3 GB 1.1 GB

Mistral-7B-v0.2 7B 33 GB 16.5 GB 8.25 GB

Llama-3-70B-Instruct 70B 311 GB 155 GB 77 GB

Mistral-8x22B-v0.1 141B 631 GB 316 GB 158 GB

Bloom-176B 176B 787 GB 394 GB 200 GB

10
 Quantization
! Number representation: Decimal
v Decimal numbers are just numbers that also include negative powers of the base.

567. 34510 = 5 × 102 + 6 × 101 + 7 × 100 + 3 × 10−1 + 4 × 10−2 + 5 × 10−3

10 m 10 m - 1 … 102 101 100 10 -1 10 -2 10 -3 … 10 - (n - 1) 10 - n

× × × × × × × × × ×
dm dm - d2 d1 d0 . d-1 d-2 d-3 dn - 1 dn
1 … …

11
 Quantization
! Number representation: Binary
But where do we put the point?
v In binary:

2m 2 m-1 … 22 21 20 2 -1 2 -2 2 -3 … 2 - (n - 1) 2-n
× × × × × × × × × ×
bm bm - b2 b1 b0 . b-1 b-2 b-3 b- n - bn
1 … … 1

v Example: 1001.10112 = 1 × 23 + 1 × 20 + 1 × 2 -1 + 1 × 2 -3 + 1 ×2 -4 = 8 + 1 + 0.5 + 0.125 + 0.0625 = 9.6875

Higher values of n yield greater precision

12
 Quantization
! Fixed-point representation

1 1 0 0 1 0 1 1 0 1 1 0 1 1 1 1 1

Greater Precision Wider Range

● Example: 00000000.00001012
We must store a lot of meaningless bits
11100000.00000002

Any way to make the point move?

13
 Quantization
! Floating-point representation

31 30 23 22 0
0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Sign Exponent (8 bit) Fraction/ Mantissa (23 bit)

Sign = 0
(positive)
The exponent field stores it

Example: 101.112 can be written as 1.0111 × 2 2

The mantissa field stores it

14
 Quantization
! Floating-point representation

31 30 23 22 0
0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Sign Exponent (8 bit) Fraction/ Mantissa (23 bit)

Sign = 0 It can be negative! How to represent it?

(positive)

Real Value = (−1)S × M × 2Exponent

Any problem with exponent?

15
 Quantization
! Floating-point representation

31 30 23 22 0
0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Sign Exponent (8 bit) Fraction/ Mantissa (23 bit)

Sign = 0 Exponent now save the value

(positive)

Plus a bias = 127

b = 8 bit can represent a range [-127; 127] The range shifts to [0; 254]

Solution: Add a bias to the exponent! Bias = 2b - 1 - 1

16
 Quantization
! Floating-point representation

31 30 23 22 0
0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Sign Exponent (8 bit) Fraction/ Mantissa (23 bit)

Sign = 0
(positive)

Real Value = (−1)S × M × 2E

E = exponent − bias

17
 Quantization
! Floating-point representation

FP32 S 8 bits 23 bits

FP16 S 5 bits 10 bits

Bfloat 16 S 8 bits 7 bits

Data Type Range Precision

FP32 1.18 x 10−38 to 3.40 x 1038 6 – 9 decimal digits

FP16 4.88 x 10−4 to 6.55 x 104 5 – 6 decimal digits

Bfloat16 1.18 x 10−38 to 3.40 x 1038 3 decimal digits

18
 Quantization
! Idea of Quantization
α Real value (Floating-point) β

quantize

Integer

Asymmetric Quantization

Uniform Quantization
Symmetric Quantization
Quantization

Non-uniform Quantization
19
 Quantization
! Uniform Quantization vs. Non-Uniform Quantization

Uniform quantization Non-uniform quantization

20
 Quantization
! Types of Uniform Quantization : Asymmetric and Symmetric

Asymmetric Symmetric
β 0 α -α 0 α

Qmin -z 0 Qmax Qmin z=0 Qmax

[-128 , 127] (signed int8)
(-2b-1, 2b-1 - 1) = [-128, 127]
[0, 255] (unsigned int8)

z acts as an anchor point mapping the original range into the discrete integer range.
21
 Quantization
! Asymmetric Formula

Asymmetric
β 0 α

Scaling Factor

Qmin -z 0 Qmax
Zero-point

Clipping range
22
 Quantization
! Understand Asymmetric Formula

Asymmetric
Scaling Factor β 0 α

“Each step in integer range contains S values of original range.”

Qmin -z 0 Qmax

How do we map zero to integer range?

23
 Quantization
! Understand Asymmetric Formula

Asymmetric
Zero-point β 0 α

“To go from β to 0, we need z steps in integer range”

Qmin -z 0 Qmax

This is zero-point position!

24
 Quantization
! Understand Asymmetric Formula

Asymmetric
β 0 α

Distance from xf to 0 Position of z in integer

Qmin -z 0 Qmax
Position of quantized value in integer range!

25
 Quantization
! Asymmetric Dequantization

Asymmetric
β 0 α

Qmin -z 0 Qmax

26
 Quantization
! Asymmetric Dequantization

27
 Quantization
! Asymmetric Dequantization

-1.3521 -1.8095 0.2968 -1.3974 0.9779 0.7762 2.8330 0.7332 -0.8949 -0.8428

Quantize

-103 -128 -13 -106 25 14 127 11 -78 - 75

Dequantize

-1.3472 -1.8024 0.2913 -1.4019 0.9831 0.7829 2.8401 0.7282 -0.8921 -0.8375
 Quantization
! Lossy Compression

What factors can cause big errors?

Outlier Data distribution shift

Quantization aims to minimize the lossy

compression

29
 Quantization
! Symmetric Quantization

Symmetric
-α 0 α

Qmin z=0 Qmax

30
 Quantization
! Symmetric Quantization

31
 Quantization
! Symmetric Quantization

0.7339 0.4977 0.9103 -0.4479 -0.3806 -0.0671 0.1176 1.9213 0.6556 0.0380

Quantize

49 33 60 -30 -25 -4 8 127 43 3

Dequantize

0.7413 0.4992 0.9077 -0.4539 -0.3782 -0.0605 0.1210 1.9213 0.6505 0.0454

32
 Quantization
! Asymmetric vs. Symmetric Comparison

33
 Quantization
! Asymmetric vs. Symmetric Comparison

Feature Asymmetric Quantization Symmetric Quantization

Implementation More Complex Simple

Computational Cost Slightly higher Simple calculation

Efficiency for non-uniform distributions More efficient Less efficient

34
 Quantization
! Calibration

● Calibration is the process of choosing the clipping range [α, β]

● Simple approach: Choosing the min/max of the signal

What problem in the figure?

35
 Quantization
! Calibration

Choose max absolute value

vulnerable to outliers!

Percentile: Set a range to a percentile of the distribution.

Calibration
Strategy Entropy: KL divergence to minimize information loss.

36
 Quantization
! Calibration
Outlier!!!

0.3 0.6 -0.3 0.5 0 0.7 -99 0.2

Quantize

0 1 0 -1 0 1 -127 0

Dequantize

0.0 0.77 0.0 -0.77 0.0 0.77 -99 0.00

37
 Quantization
! Calibration Strategy: Percentile

Percentile (Vietnamese: “Bách phân vị”) is a statistical measure that indicates the value below which a given
percentage of observations in a dataset fall.

Ignore outlier

Only use this range (99.99%)

38
 Quantization
! Calibration Strategy: Percentile
Do not select it!

0.3 0.6 -0.3 0.5 0 0.7 -99 0.2

Quantize

65 127 -65 -108 0 127 -128 43

Dequantize

0.3 0.59 -0.3 -0.5 0.0 0.59 -99 0.19

39
 Quantization
! Non-Uniform Quantization

Step increase linearly

Lots of data in this area

Many data points get

quantized into 1 bin

40
 Quantization
! Non-Uniform Quantization

Fewer values

Still the same

number of bins

41
 Outline
FP16 S 5 bits 10 bits
SECTION 1

Introduction to Quantization Bfloat 16 S 8 bits 7 bits

SECTION 3

Quantization in NN

42
 Quantization In NN
! Neural Network

Weight

Bias
Y = XW + B Usually quantized as int32
Perform all operations using
integer arithmetic

Input
sometimes called “activation”
43
 Quantization In NN
! Neural Network Quantization Methods

Static PTQ

Post Training Quantization

(PTQ)

Methods Dynamic PTQ

Quantization Aware Training

(QAT)

44
 Quantization In NN
! Post Training Quantization: Static

Pre-trained Model

Attach Observers

Unlabeled Data Calibrate Calculate the s and z

Quantized Model

45
 Quantization In NN
! Post Training Quantization: Static

Input Output
FP32 FP32

Quantize Stub INT8 Dequantize Stub

● All weights and activations are quantized to INT8 before inferences

● Uses unlabeled data to determine scale (s) and zero-point (z).
● Add quantization layers to convert input to INT8 and map output back to FP32.

46
 Quantization In NN
! Post Training Quantization: Static

Pre-trained Model

Attach Observers

Unlabeled Data Calibrate

Quantized Model

47
 Quantization In NN
! Post Training Quantization: Static

Pre-trained Model

Attach Observers

Unlabeled Data Calibrate

Quantized Model

48
 Quantization In NN
! Dynamic PTQ

Calibrate Quantize Dequantize

Input Layer Layer Output

FP32 1 2 FP32

Calibrate Quantize Dequantize

In this method, the model performs calibration and quantization on-the-fly before each layer during inference,
converting the data back to FP32 (dequantize) after each operation is completed

49
 Quantization In NN
! Dynamic PTQ

Calibrate Quantize Dequantize

Input Layer Layer Output

FP32 1 2 FP32

Calibrate Quantize Dequantize

One line only!

50
 Quantization In NN
! Quantization Aware Training Sequence of Quantize and Dequantize operations

● Insert some fake modules in the computational graph of the model to simulate the effect of the
quantization during training.
● The loss function gets used to update weights that constantly suffer from the effect of quantization,
usually leads to a more robust model.

51
 Quantization In NN
! Quantization Aware Training

Weight r Quantized Weight Q

(FP) (INT)

1.1 2.2 1 2
Forward Pass
-1.7 3.6 -2 2

0.1 -0.1 0.1 -0.1

Backward Pass
-0.2 0.2 -0.2 0.2

Gradient dL/dr Gradient dL/dQ

(FP) (FP)

52
 Quantization In NN
! Quantization Aware Training Non differentiable

Weight r Quantized Weight Q

(FP) (INT)

1.1 2.2 1 2
Forward Pass
-1.7 3.6 -2 2

53
 Quantization In NN
! Quantization Aware Training Non differentiable

Weight r Quantized Weight Q

(FP) (INT)

1.1 2.2 1 2
Forward Pass
-1.7 3.6 -2 2

0.1 -0.1 0.1 -0.1

Backward Pass
-0.2 0.2 -0.2 0.2

Gradient dL/dr Gradient dL/dQ

(FP) (FP)

54
 Quantization In NN
! Quantization Aware Training

55
56
 Summary
Introduction to Quantization Quantization in Neural Network
v Motivation v Neural Network Quantization Methods
v Quantization v Post Training Quantization
v Floating-point Representation v Quantization Aware Training
v Uniform vs Non-Uniform Quantization v Experimentation

57
Thanks!
Any questions?

58
 ONNX and TensorRT
Class: Model Compression Series

Thuan Duong – TA
Phat Dat – sTA

Year 2025 Study Materials Quiz&Feedback

 Objective
AI VIETNAM
All-in-One Course
(TA Session)

ONNX

24/1/26 2
 Objective
AI VIETNAM
All-in-One Course
(TA Session)

TensorRT

24/1/26 3
 AI VIETNAM
All-in-One Course
(TA Session)

Outline
Ø Motivation
Ø ONNX
Ø TensorRT
Ø Questions

24/1/26 4
 AI VIETNAM
All-in-One Course
(TA Session)

Motivation

24/1/26 5
 Motivation
AI VIETNAM
All-in-One Course
(TA Session)

v Problems
• There is no single tool for all tasks. Data
scientists use various frameworks based
on their specific needs.
Microsoft

Research flexibility
Deep Learning
Frameworks

Production efficiency

24/1/26 6
 Motivation
AI VIETNAM
All-in-One Course
(TA Session)

v Framework lock-in
• To move a model from research to production, engineers often have to rewrite the model from scratch in a
different language or framework, which is time-consuming and prone to errors.

Research Team Production Team

rewrite code

Time consuming Errors

24/1/26 7
 Motivation
AI VIETNAM
All-in-One Course
(TA Session)

v Low inference speed

• Models need to run on various hardware backends, such as CPUs, GPUs or NPUs.

PC, Laptop Server

Model built in GPU
GPU
framework

CPU GPU
Mobile devices

CPU TPU

NPU Frameworks must support to optimize on different

devices, which is complex.

24/1/26 8
 Motivation
AI VIETNAM
All-in-One Course
(TA Session)

v Production requires

Ø Maximize Accuracy / State-of-the-art

Ø Unlimited Resources
Ø Flexible Environment

• Latency: Must be real-time

• Throughput: Handle thousands of requests/second.
• Hardware: Specific constraints
• Cost: Optimization directly impacts the cloud bill.

24/1/26 9
 Motivation
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX

24/1/26 [Link] 10
 Motivation
AI VIETNAM
All-in-One Course
(TA Session)

v TensorRT

24/1/26 [Link] 11
 AI VIETNAM
All-in-One Course
(TA Session)

ONNX

24/1/26 12
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v What is ONNX?
• Open Neural Network Exchange (ONNX) is an open standard for exchanging AI models.
• ONNX defines a common set of operators and a common file format to enable AI developers to use models
with a variety of frameworks, tools, runtimes, and compilers.

Frameworks Devices

Converters Deploy

24/1/26 13
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX in AI stack

Flexibility for model

design & training

CPU-EPs CPU-EPs CPU-EPs

Frameworks
.pth, .pb, .caffemodel, .tflite, ... The bridge

Devices
.onnx

Hardware-optimized execution
(ONNX Runtime, TensorRT).

24/1/26 14
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Concepts
Metadata: contains information
about the model

Version/ Opset

Graph: the core logic, defines

the structure of neural network

24/1/26 Ref: [Link] 15

 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX File Format

Model Graph
• Version info • Inputs and Outputs
• Metadata • List of computation nodes
• Acrylic computation dataflow graph • Graph name

Computation node
• Number of inputs of defined types
• Operator
• Operator parameters

24/1/26 Ref: [Link] 16

 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Graph Components
Nodes (Operators) Edges
a x
• Represent computational operations (e.g., Conv, Relu, MatMul).
• Act as function calls: they consume Inputs and produce Outputs. Mul
• Defined by the Opset version to ensure compatibility. b

Edge Nodes
Add
• Represent the data flow between Nodes.
• Establish the topology of the DAG (Directed Acyclic Graph).
• Carry data (Tensors) from one operation to the next.
output

24/1/26 17
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Graph Components
Attributes
a x
• Static parameters specific to a Node (e.g., kernel_size, strides, padding).
• Fixed configurations that do not change during inference. Mul
b
Inputs & Outputs
• Define the graph's interfaces
Add
• Strongly typed: Must specify shape (dimensions) and
data type (e.g., FP32, INT64).

output

24/1/26 18
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX Operators
• ONNX defines a specifics for every operation (Math, NN Layers, Activation).
Ø Ensures consistent math behavior across all platforms.

24/1/26 Ref: [Link] 19

 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX Operators
Versioning Strategy:
• Each operator has versions
• All operators are managed via Opset versions.

Newer Opset = More supported

operators & updated.

Must match the Opset version

supported by your Target Runtime.

24/1/26 20
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Opset
The Opset (Operator Set) in ONNX defines a collection of
operators that a model can use

Update operations

Opset 14 Opset 18

ReLU ver. 13 Add ver. 6 ArgMax Ver. 11 ReLU ver. 14 Add ver. 14 ArgMax Ver. 13

24/1/26 21
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Netron visualization

• An open-source viewer for neural networks, deep learning,

and machine learning models.

• Natively supports ONNX, TensorFlow Lite, Keras, Caffe,

and many others.

[Link]

24/1/26 22
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Export model to ONNX

Tracing
Source model Input Dummy

CONVERTER Opset version

[Link]

Validate/ Check

24/1/26 23
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Converters

tensorflow-onnx
sklearn-onnx

???

LibSVM onnxmltools

24/1/26 24
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Tensorflow to ONNX

Dummy input for tracing

Converter

24/1/26 25
 ONNX
AI VIETNAM
All-in-One Course
(TA Session) Ref: [01]_Tensorflow2ONNX_VGG16.ipynb

v Demo notebook [01] 4

24/1/26 26
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Pytorch to ONNX

Dummy input
for tracing

Converter

24/1/26 27
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Demo notebook [00]

4
1

2
Ref: [00]_Pytorch2ONNX_Resnet50_CPU.ipynb

24/1/26 28
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX Runtime
ONNX Runtime: A high-performance, cross-platform inference
engine specifically designed to run ONNX models.

Ø Supports models trained in PyTorch, TensorFlow, Keras, etc.

Ø Optimized for diverse hardware backends (CPU, GPU, NPU)
Ø Runs on Linux, Windows, macOS, Android, and IOS. CPU-EPs CPU-EPs CPU-EPs

Devices

24/1/26 29
Ref [Link]
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX Runtime architecture

Splits the graph into subgraphs based on available
Parses .onnx file into an optimized Execution Providers (e.g., GPU vs. CPU).
internal graph representation.

24/1/26 Ref: [Link] 30

 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Graph Optimization (3 levels)

Simplification: Removes redundant nodes

Fusion: Merges multiple ops into a single

kernel to reduce memory access.

Layout: Optimizes memory layout for

target hardware.

24/1/26 Ref: [Link] 31

 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Execution Providers (EPs).

An abstraction layer that connects the runtime engine to
specific hardware accelerators (drivers).

Write once,
run anywhere!

24/1/26 Ref: [Link] 32

 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Why ONNX Runtime?

Performance Boost

Hybrid Execution

Production Readiness

Lightweight

24/1/26 Ref: [Link] 33

 ONNX
AI VIETNAM
All-in-One Course [02]_ONNX_Pytorch_vs_Tensorflow.ipynb
(TA Session)

v PyTorch, Tensorflow
to ONNX Runtime

Ø ONNX Runtime and GPU

acceleration delivers
substantial performance gains,
especially for lightweight
architectures.

24/1/26 34
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Benchmark PyTorch vs ONNX Runtime

24/1/26 35
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Demo 1: multi-platform

NVIDIA T4 RTX 3060

24/1/26
Ubuntu 22.04.4 LTS Ubuntu 24.04.4 LTS 36
 ONNX
AI VIETNAM
All-in-One Course
(TA Session)

v Limitations
Limited Operator Support
Not all operators (especially newer/dynamic ones) map 1-1 from frameworks to ONNX, causing export failures.

Complex Custom Ops

Implementing custom layers is difficult, often requiring manual C++/CUDA coding.

Hard to Debug
As a static graph, tracing errors is much harder compared to native PyTorch/TensorFlow code.

24/1/26 37
 AI VIETNAM
All-in-One Course
(TA Session)

Processor

24/1/26 38
 Processor
AI VIETNAM
All-in-One Course
(TA Session)

v CPU vs GPU

Link Link GPU (Graphics Processing Unit)

CPU (Central Processing Unit)

24/1/26
39
 Processor
AI VIETNAM
All-in-One Course
(TA Session)

v CPU vs GPU

24/1/26 Ref: [Link] 40

 GPU
AI VIETNAM
All-in-One Course
(TA Session)

v CUDA
NVIDIA CUDA (Compute Unified Device Architecture) is NVIDIA’s parallel computing platform
that allows developers to use GPUs to accelerate general-purpose computations beyond graphics

24/1/26 Ref: [Link] 41

 NVIDIA GA100
AI VIETNAM
All-in-One Course
(TA Session)

24/1/26 42
 NVIDIA GPU
AI VIETNAM
All-in-One Course
(TA Session)

v Streaming Multiprocessor (SM)

24/1/26 43
 NVIDIA GA100
AI VIETNAM
All-in-One Course
(TA Session)

v Streaming Multiprocessor (SM)

The full implementation of the GA100 GPU:
• 8 GPCs, 8 TPCs/GPC, 2 SMs/TPC, 16 SMs/GPC, 128 SMs per full
GPU
• 64 FP32 CUDA Cores/SM, 8192 FP32 CUDA Cores per full GPU
• 4 Third-generation Tensor Cores/SM, 512 Third-generation Tensor
Cores per full GPU
• 6 HBM2 stacks, 12 512-bit Memory Controllers

NVIDIA A100 Tensor Core GPU implementation

• 7 GPCs, 7 or 8 TPCs/GPC, 2 SMs/TPC, up to 16 SMs/GPC, 108
SMs
• 64 FP32 CUDA Cores/SM, 6912 FP32 CUDA Cores per GPU
• 4 Third-generation Tensor Cores/SM, 432 Third-generation Tensor
Cores per GPU
• 5 HBM2 stacks, 10 512-bit Memory Controllers

24/1/26 Ref: [Link] 44

 AI VIETNAM
All-in-One Course
(TA Session)

TensorRT

24/1/26 45
 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v What is TensorRT
Ø A high performance SDK for optimizing trained deep-learning models and running on NVIDIA hardwares.
Ø Perform optimization such as layer fusion, kernel autotuning, quantization and more.
Ø TensorRT contains a deep learning inference optimizer and a runtime for execution.
Ø Speedup inference time and reduced latency.

24/1/26 46
 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v TensorRT Ecosystem

TensorRT-LLM

Inference for LLMs

TensorRT TensorRT Model Optimizer

Inference for non-LLMs, automotive Model optimizations like Quantization,

embedded, robotics and edge applications Distillation, Sparsity, etc.

TensorRT for RTX TensorRT Cloud.

Inferencing on RTX PCs Compile in the Cloud

24/1/26 47
 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v Use TensorRT
ONNX Conversion Before TensorRT

Trained ONNX TensorRT TensorRT

DNN Conversion Optimizer Runtime

TensoRT Framework Integration

TensorRT Framework
Integration APIs
torch-tensorrt
tensorflow-trt
TensorRT TensorRT
Optimizer Runtime
Trained Model In Framework Inference

24/1/26 48
 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v Optimization techniques

Layer & Tensor Fusion

Mixed Precision Kernel Auto-tuning

Trained TensorRT Optimizer Optimized

DNN Inference Engine
.engine
Dynamic Tensor Memory Multi-Stream Execution

24/1/26 49
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Mixed
• DNNs often trained with full precision (FP32). α Real value (FP32) β

• Inference allows lower precision via a calibration step. quantize

INT8

Static PTQ Pre-trained Model

Attach Observers Smaller Model Size

Lower Memory Utilization and Latency
Unlabeled Data Calibrate Calculate the s and z
Higher Throughput
Quantized Model INT8

24/1/26 50
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Precision modes
Pros Cons

Consumes the most

Highest accuracy, huge dynamic
FP32 range
memory and bandwidth, Priority
slowest inference speed.

Reduces memory footprint by 50%.

Speedup on GPUs with Tensor
FP16 Smaller dynamic range.
Cores.
Accuracy very close to FP32.

Risk of accuracy
Fastest throughput
INT8 Smallest model size
degradation if the model
is sensitive.
Exponent
Sign Mantissa

FP32 S 8 bits 23 bits

FP16 S 5 bits 10 bits

INT8 8 bits

24/1/26 51
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Kernel Auto-Tuning
• Kernel: A low-level function written to be executed on the GPU.
• The actual "worker" performing operations (MatMul, Conv2d, Element-wise ops).
• There are multiples kernel for common operations.

Kernel 1: Optimized for using cache.

X
Kernel 2: Optimized for Bignum.
Matrix multiplication
Kernel 3: Optimized for Ampere chip

24/1/26 52
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Kernel Auto-Tuning
• TensorRT selects the optimal kernels based on user’s parameters: batch-size, input data size…
• TensorRT selects the optimal kernel based on user’s target flatform.

Try each kernel

0.01 ns

0.02 ns write to
X

TensorRT Optimizer
Matrix multiplication 0.001 ns .engine

0.015 ns

24/1/26 53
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Layer & Tensor Fusion

Load Read
Conv
Write
Add
Read
GPU
VRAM
Bias
Write
Compute
Read
ReLU
Write

24/1/26 54
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Layer Fusion

Vertical Layer Fusion

Conv

Fuse CBR Read only 1 time GPU

Bias (Conv-Bias-ReLU) VRAM

ReLU
Load and compute
within cache

24/1/26 55
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Layer Fusion

Horizontal Layer Fusion

CBR CBR CBR CBR CBR CBR

CBR CBR CBR CBR

Load
Kernel Load One
Load Load kernel kernel
Input Input
Kernel Kernel for all

24/1/26 56
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Layer Fusion

Vertical & Horizontal Layer Fusion

Output Output

Concat Concat

CBR CBR CBR CBR CBR CBR CBR CBR

CBR CBR Max Pool CBR CBR Max Pool

Input Input

24/1/26 57
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Tensor Fusion

Output Tensor Copy

A
Concat
Tensor
Copy
B
CBR CBR CBR CBR
Concat
CBR CBR Max Pool

Compute
Solutio A
Compute
Input n
B
Allocat
e Concat

24/1/26 58
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Tensor Fusion

Output Output
Delete
Concat

CBR CBR CBR CBR CBR CBR CBR CBR

CBR CBR Max Pool CBR CBR Max Pool

Input Input

24/1/26 59
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Dynamic Tensor Memory

• Allocate just the memory required for each tensor and only for the duration its usage
• Reduces memory footprint and improves memory re-use

Compute 1Mb 1Mb Do not allocate

Layer 1 Layer 2 Layer 3

1Mb 1Mb 1Mb

Layer 1 Layer 2 Layer 3
Allocate memory
Deallocate 1Mb 1Mb
GPU totally Layer 1 Layer 2 Layer 3

allocates 3MB Re-use

before computing
GPU only allocates
2MB
24/1/26 60
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Multi-Stream Execution
• Allow processing multiple input streams in parallel
• Increase throughput

Single Stream Multi-Stream

No need GPU execution

Task 1 Task 2 … Task n

GPU must wait

(no execute) Parallel

Memory is loading

24/1/26 61
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v trtexec
• A command-line tool designed for TensorRT performance benchmarking
• The tool can build tensorrt engine and measure .onnx files performance directly

trtexec

Throughput: 507.399 qps

Latency: min = 1.96301 ms,
max = 1.97534
Trained ONNX ... TensorRT TensorRT
DNN Conversion Optimizer Runtime
.onnx benchmarking

24/1/26 62
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v trtexec
trtexec --onnx=[Link] --shapes=data:4x3x224x224 --fp16 --noDataTransfers --useCudaGraph --
useSpinWait

After running the trtexec command, trtexec will parse ONNX file, build a TensorRT plan file, measure the performance of
this plan file, and then print a performance summary as follows:

[04/25/2024-23:57:45] [I] === Performance summary ===

[04/25/2024-23:57:45] [I] Throughput: 507.399 qps
[04/25/2024-23:57:45] [I] Latency: min = 1.96301 ms, max = 1.97534 ms, mean = 1.96921 ms, median = 1.96917 ms,
percentile(90%) = 1.97122 ms, percentile(95%) = 1.97229 ms, percentile(99%) = 1.97424 ms
[04/25/2024-23:57:45] [I] Enqueue Time: min = 0.0032959 ms, max = 0.0340576 ms, mean = 0.00421173 ms, median = 0.00415039
ms, percentile(90%) = 0.00463867 ms, percentile(95%) = 0.00476074 ms, percentile(99%) = 0.0057373 ms
[04/25/2024-23:57:45] [I] H2D Latency: min = 0 ms, max = 0 ms, mean = 0 ms, median = 0 ms, percentile(90%) = 0 ms,
percentile(95%) = 0 ms, percentile(99%) = 0 ms
[04/25/2024-23:57:45] [I] GPU Compute Time: min = 1.96301 ms, max = 1.97534 ms, mean = 1.96921 ms, median = 1.96917 ms,
percentile(90%) = 1.97122 ms, percentile(95%) = 1.97229 ms, percentile(99%) = 1.97424 ms
[04/25/2024-23:57:45] [I] D2H Latency: min = 0 ms, max = 0 ms, mean = 0 ms, median = 0 ms, percentile(90%) = 0 ms,
percentile(95%) = 0 ms, percentile(99%) = 0 ms
[04/25/2024-23:57:45] [I] Total Host Walltime: 3.00355 s
[04/25/2024-23:57:45] [I] Total GPU Compute Time: 3.00108 s
[04/25/2024-23:57:45] [I] Explanations of the performance metrics are printed in the verbose logs.

24/1/26 Ref: [Link] 63

 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Dynamic Shape & Optimization Profile

• Dynamic shapes in TensorRT allow deferring the specification of certain tensor dimensions
until runtime, enabling flexibility for models that handle inputs of varying sizes.

Static Shape: [1, 3, 224, 224] Dynamic: [-1, 3, -1, -1]

Flexible
.engine
224 x 224

.engine
224 x 224

224 x 224 512 x 512

TensorRT need to know the range of all possible input sizes so it can prepare memory

24/1/26 64
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v Dynamic Shape & Optimization Profile

A Profile containing 3 states for each Input Tensor:
● Min (Minimum): The smallest size you allow (e.g., 1 x 3 x 224 x 224).
● Opt (Optimal - Most Important): The most common size you expect to run. TensorRT will select the kernels that
are fastest specifically for this size (e.g., 8 x 3 x 512 x 512)
● Max (Maximum): The largest possible size. TensorRT uses this to allocate the maximum necessary VRAM to
prevent memory overflows (e.g., 16 x 3 x 1024 x 1024)

1. trtexec 2. Using Python API

trtexec --onnx=[Link] \
--minShapes=input:1x3x224x224 \
--optShapes=input:8x3x512x512 \
--maxShapes=input:16x3x1024x1024 \
--saveEngine=model_dynamic.engine

24/1/26 65
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX + TensorRT workflow

1 Use trtexec (no code)
.engine

Build .engine

TensorRT TensorRT
Trained ONNX Check .onnx Optimizer builder
DNN Conversion by trtexec

2 Python code APIs .engine

● Engine file only run on device it is built in (e. g. If you build .engine in RTX 3060, the .engine file only run on GTX 3060)
Send .onnx file along with .engine so user can build engine file on new device.

● TensorRT does not fallback to CPU. If an operator is not supported by GPU, it will fails.

24/1/26 66
 Optimization techniques
AI VIETNAM
All-in-One Course
(TA Session)

v ONNX + TensorRT workflow

● ONNX Runtime has Graph Partitioning mechanism which automatically fallback
to CPU if operator is not supported by CPU.
● Use TensorRT as a provider of ONNX Runtime

TensorrtExecutionProvider

build + inference

Trained ONNX
DNN Conversion
Else: automatically fallback to CPU
The model always run
Don’t need to modify TensorRT for new operator

24/1/26 67
 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v Installation (Pip Python)

Prerequisites

• CUDA 12.x or 13.x

• NVIDIA CUDA Toolkit

Related repositories:
• [Link]
• [Link]
• [Link]

24/1/26 Ref: [Link] 68

 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v Installation (Ubuntu)
1. Install CUDA ([Link]
2. Download TensorRT package (.deb) that matches the Ubuntu version and CPU architecture.
3. Install TensoRT via .deb package

CUDA version

24/1/26 Ref: [Link] 69

 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v Installation (Ubuntu)
1. Install CUDA Ref: [Link]
2. Download TensorRT package (.deb) that matches the Ubuntu version and CPU architecture.
3. Install TensoRT via .deb package

Install

24/1/26 Ref: [Link]

Ref: [Link] 70
 TensorRT Engine
AI VIETNAM
All-in-One Course
(TA Session)

v Build Pipeline
Initialize TensorRT Builder & Logger

Create network with EXPLICIT_BATCH to

support dynamic shapes

Parse the ONNX model into TensorRT network

Configure builder:
• Set WORKSPACE memory (1 GB) for
layer optimization
• Define optimization profile for dynamic
batch size (1 → 8)

Build a serialized TensorRT engine

24/1/26 Ref: [Link] 71

 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v Inference using engine

Deserialize TensorRT engine and create execution
context

Set dynamic input shape for the current batch

Allocate host (CPU) and device (GPU) memory

Bind input/output buffers to TensorRT tensors

Run asynchronous inference on CUDA stream

Ref: [Link]
24/1/26 72
library/[Link]#deserializing-a-plan
24/1/26 73
 TensorRT
AI VIETNAM
All-in-One Course
(TA Session)

v Demo 2: multi-platform

NVIDIA T4 RTX 3060

24/1/26
Ubuntu 22.04.4 LTS Ubuntu 24.04.4 LTS 74
 AI VIETNAM
All-in-One Course
(TA Session) Question

?
24/1/26 75
24/1/26 76