VIETNAM GENERAL CONFEDERATION OF LABOUR
TON DUC THANG UNIVERSITY
FACULTY OF INFORMATION TECHNOLOGY
FINAL PROJECT
FINAl PROJECT NLP
Lecturer: [Link]. LE ANH CUONG
Student: NGUYEN KHAC HUY – 521H0502
GIANG HOANG DAT – 521H0498
TRAN DINH QUANG VINH – 521H0333
Class : 21H50301
Group : 9
Academic year : 2024-2025
HO CHI MINH CITY, 2025
1
VIETNAM GENERAL CONFEDERATION OF LABOUR
TON DUC THANG UNIVERSITY
FACULTY OF INFORMATION TECHNOLOGY
FINAL PROJECT
FINAl PROJECT NLP
Lecturer: [Link]. LE ANH CUONG
Student: NGUYEN KHAC HUY – 521H0502
GIANG HOANG DAT – 521H0498
TRAN DINH QUANG VINH – 521H0333
Class : 21H50301
Group : 9
Academic year : 2024-2025
HO CHI MINH CITY, 2025
1
ACKNOWLEDGEMENT
To complete this final project, we would like to express our gratitude to Ton
Duc Thang University for providing the necessary facilities and to the teachers who
have supported us throughout our studies at the TDT University. Moreover, we love to
give
thanks to our parents who gave us many chances to approach knowledge at TDTU and
motivated, supported us a lot.
Especially, we would like to thank Assoc. Prof. Dr. Le Anh Cuong for teaching
us with great dedication and detail, so that we have enough knowledge that we used for
this essay. Due to our limited experience and acknowledgement, we make sure that
there
are some mistakes in our work. we sincerely hope to receive feedbacks and
constructive
criticism from teacher who instructed us. So that we can complete this essay more
effectively.
We would like to express our heartfelt thanks and wishes teacher good health.
1
THE PROJECT IS COMPLETED AT TON DUC THANG
UNIVERSITY
I hereby declare that this is the product of our project and is guided by Pham Van Huy.
The research contents and results in this topic are honest and have not been published
in any forms before. The data in the tables for analysis, comments, and evaluation were
collected by the author from different sources and clearly stated in the reference
section.
In addition, the project also uses a number of comments, assessments as well as data
from other authors and other organizations, all with citations and source notes.
If any fraud is discovered, I will take full responsibility for the content of my project.
Ton Duc Thang University is not involved in copyright violations caused by me during
the implementation process (if any).
Ho Chi Minh City, 09th April 2025
Authors
(Signatures and full names)
Giang Hoằng Đạt
Nguyễn Khắc Huy
Trần Đình Quang Vinh
2
SUMMARY
Machine Translation using Deep Learning is the process of automatically
translating text from one language to another using neural network models.
In this project, we developed an English-to-Vietnamese translation system based
on a sequence-to-sequence (Seq2Seq) architecture with LSTM (Long Short-Term
Memory) layers. We began by collecting and preprocessing bilingual sentence pairs,
tokenizing the text, and adding special tokens to mark sentence boundaries.
The model includes an encoder to process input English sentences and a decoder
to generate Vietnamese translations.
After training with categorical crossentropy loss and the Adam optimizer, the
system was able to produce basic translations. While the current model shows
encouraging results, its accuracy can be further improved by incorporating attention
mechanisms, using larger datasets, or transitioning to Transformer-based architectures.
3
TABLE OF CONTENTS
ACKNOWLEDGEMENT 1
THE PROJECT IS COMPLETED AT TON DUC THANG UNIVERSITY 2
SUMMARY 3
TABLE OF CONTENTS 4
1.1. Overview of Large Language Model (LLMS) 5
1.2. Policy Gradient Methods (PPO, TRPO, REINFORCE) 6
1.2.1. Proximal Policy Optimization (PPO) 6
1.2.2. Trust Region Policy Optimization (TRPO) 7
1.2.3. REINFORCE 7
1.3. Direct Preference Optimization (DPO) 8
1.4. Actor-Critic Methods (A2C/A3C) 9
1.5. Comparison of Algorithms 11
1.6. 11
CHAPTER II –: 11
CONCLUSION 12
REFERENCES 13
4
CHAPTER I – Introduction to Reinforcement Learning in Large
Language Model (LLMS)
Reinforcement Learning (RL) trains machine learning models to make
sequential decisions in a game-like environment using trial and error. An AI
agent receives rewards or penalties based on its actions, aiming to maximize
total rewards. By running multiple experiments in parallel on powerful
computing infrastructure, RL effectively fosters machine creativity, leveraging
extensive simulated experiences unlike human learning.
[Link] of Large Language Model (LLMS)
Large Language Models (LLMs) like GPT, LLaMA, or BERT are often
fine-tuned using Reinforcement Learning (RL) algorithms to enhance
performance, particularly for tasks like aligning responses with user intent,
reducing biases, or improving safety.
LLMs are one of the most successful applications of Transformer Models.
They are not only used to teach human language to AI, but also to understand
proteins, write software code, and many other possibilities.
In addition to accelerating natural language processing applications like
translation, chatbots, and AI assistants – large language models are also used in
healthcare, software development, and many other applications.
The most prominent RL method used is Reinforcement Learning from Human
Feedback (RLHF), alongside variants and related algorithms.
5
1.2. Policy Gradient Methods (PPO, TRPO, REINFORCE)
1.2.1. Proximal Policy Optimization (PPO)
a. Formula
[
Lclip ( θ )=Et min (r t (θ ) ^
A t , clip ( r t ( θ ) , 1−ϵ , 1+ϵ ) ^
At ) ]
Where:
π θ ( a t ∣s t )
r t ( θ )= : Probability ratio between new and old policies.
πθ (at ∣ st )
old
^At : Advantage function, measuring action quality.
ϵ : Clipping parameter (e.g., 0.1–0.2).
b. Algorithm:
Initialize policy π_θ.
Collect trajectories (states, actions, rewards) using the current
policy.
Compute advantages A ̂ _t using a value function (e.g.,
Generalized Advantage Estimation).
Update policy by maximizing the clipped objective using
gradient ascent.
Repeat until convergence.
c. Significance
PPO balances stability and efficiency with a clipping mechanism
to prevent large policy updates.
Widely used in RLHF for LLMs (e.g., ChatGPT) to align
responses with human feedback, ensuring safety and relevance.
1.2.2. Trust Region Policy Optimization (TRPO)
6
a. Formula:
max E s ,a ∼ π
θ
θ old
[ πθ ( a ∣ s)
π θ ( a ∣s )
old
^
A ( s , a)
]
Subject to:
E s [ KL ( π θ ∣∣ π θ ) ] ≤ δ
old
Where:
KL: Kullback-Leibler divergence.
δ : Trust region size.
b. Algorithm:
Initialize policy π_θ.
Collect trajectories.
Compute advantages A ̂(s,a).
Maximize the surrogate objective using conjugate gradient,
enforcing the KL constraint via line search.
Update policy and repeat.
c. Significance:
TRPO ensures stable updates with a trust region, offering
theoretical guarantees of monotonic improvement.
Used in early RLHF experiments for LLMs but less common
due to computational complexity.
1.2.3. REINFORCE
a. Formula:
∇ θ J ( θ )=E π [ ∇ θ log π θ ( a ∣ s ) ⋅Gt ]
θ
Where:
7
π θ ( a ∣s ) : Policy probability.
Gt : Cumulative reward (return) from time t .
b. Algorithm:
Initialize policy π_θ.
Sample trajectories using the policy.
Compute returns G_t for each time step.
Update policy using the gradient ∇_θ logπ_θ (a∣s)⋅G_t.
Repeat until convergence.
c. Significance:
REINFORCE is simple and foundational but suffers from high
variance, making it inefficient for LLMs.
Rarely used in modern LLMs, mainly in research or simple
tasks.
1.3. Direct Preference Optimization (DPO)
a. Formula
LDPO ( θ ) =−E( x , y w
, y l)
[ (
log σ β log
πθ ( yw ∣ x)
π ref ( y w ∣ x )
−β log
πθ ( yl ∣ x )
π ref ( y l ∣ x ) )]
Where:
y w : Preferred response.
y l: Less preferred response.
π θ: Policy to optimize.
π ref : Reference policy.
8
β : Regularization coefficient.
b. Algorithm:
Collect preference pairs ( x , y w , y l ) from human feedback.
Optimize policy π θby minimizing the DPO loss using gradient descent.
Repeat until convergence.
c. Significance
DPO simplifies RLHF by directly optimizing the policy without a reward
model, reducing computational overhead.
Effective for LLM fine-tuning (e.g., Zephyr) with high efficiency and
comparable performance to PPO.
[Link]-Critic Methods (A2C/A3C)
a. Formula:
Policy gradient (Actor):
∇ θ J ( θ )=E π [ ∇ θ log π θ ( a ∣ s ) ⋅ ^
θ
A ( s , a) ]
Value loss (Critic):
[
L ( ϕ )=E ( V ϕ ( s )−Gt )
2
]
Where:
^A ( s , a ): Advantage, often computed as Gt −V ϕ ( s ) .
V ϕ ( s ) : Value function estimating expected returns.
9
b. Algorithm:
Initialize policy (actor) π θ and value function (critic) V ϕ .
Collect trajectories using the policy.
Compute advantages ^A ( s , a ) using the critic.
Update the actor (policy) and critic (value function) using their respective
gradients.
(For A3C) Use asynchronous updates across multiple agents.
Repeat until convergence.
c. Significance:
A2C/A3C reduces variance compared to REINFORCE by using a critic to
estimate value functions.
Rarely used in LLMs due to PPO’s superior stability and efficiency, but
tested in dialogue systems.
[Link] of Algorithms
10
Sumary:
PPO: Best balance of stability, efficiency, and scalability for LLMs.
DPO: Simplest and highly efficient, ideal for resource-constrained
settings.
TRPO: Stable but too complex for large LLMs.
REINFORCE: Inefficient and unstable, unsuitable for LLMs.
A2C/A3C: Moderately effective but overshadowed by PPO.
1.6.
CHAPTER II –:
11
CONCLUSION
12
REFERENCES
ENGLISH
• [1] Proximal Policy Optimization (PPO) in Reinforcement Learning
• [2] Reinforcement Learning
• [3]
13