Encoder Decoder Models:

Encoder-decoder model is a type of neural network that is mainly used for tasks
where both the input and output are sequences. This architecture is used when
the input and output sequences are not the same length for example translating a
sentence from one language to another, summarizing a paragraph, describing an
image with a caption or convert speech into text. It works in two stages:
● Encoder: The encoder takes the input data like a sentence and processes
each word one by one then creates a single, fixed-size summary of the
entire input called a context vector or latent space.
● Decoder: The decoder takes the context vector and begins to produce the
output one step at a time.

Encoder-Decoder Model Architecture

In an encoder-decoder model both the encoder and decoder are separate
networks each one has its own specific task. These networks can be different
types such as Recurrent Neural Networks (RNNs), Long Short-Term Memory
networks (LSTMs), Gated Recurrent Units (GRUs), Convolutional Neural
Networks (CNNs) or even more advanced models like Transformers.
Working of Encoder Decoder Model
The actual working of the encoder decoder model is shown in below diagram.
Now we will understand it stepwise:

Step 1: Tokenizing the Input Sentence

● The sentence "I am learning AI" is first broken into tokens: ["I", "am",
"learning", "AI"].
● Each word (token) is converted into a vector that a machine can
understand. This process is called embedding.

Step 2: Encoding the Input

● The Encoder processes these embeddings using self-attention.
● Self-attention helps the encoder to focus on important words. For
example while encoding "learning", it understands its relation with "I"
and "AI."
● After processing the encoder generates a Context Vector which captures
the meaning of the entire sentence. For example in the image The arrows
show how each word relates to the others during encoding. The final
output from the encoder is the context representation
Step 3: Passing the Context to the Decoder
● The Context Vector is passed to the Decoder as shown in image.
● It acts like a summary of the full input sentence.
Step 4: Decoder Generates Output Step-by-Step
● The Decoder uses the context and starts creating the output one word at a
time.
 ● First it predicts the first word then uses that to predict the second word
and so on
Step 5: Decoder Attention
● While generating each word the decoder attends to different parts of the
input sentence to make better predictions.
● For example when translating "learning," it might pay more attention to
the word "learning" in the input.
Step 6: Producing the Final Output
The decoder continues generating until the full translated sentence is produced.
Each output token depends on the previous ones and the input context. You
finally see the output tokens generated on the right side of the diagram
completing the translation.

Decoder only:
Overview:
The input and output of a transformer:
● The input is a prompt (often referred to as context) fed into the
transformer as a whole. There is no recurrence.
● The output depends on the goal of the model. For GPT models, the
output is a probability distribution of the next token/word that
comes after the prompt. It outputs one prediction for the complete
input.

Key components:
1. The embedding: the input of the transformer model is a prompt.
This prompt needs to be embedded into something that the model
can use.
2. The block(s): This is the main source of complexity. Each block
contains a masked multi-head attention submodule, a feedforward
network, and several layer normalization operations. Blocks are
put in sequence to make the model deeper.
3. The output: the output of the last block is fed through one more
linear layer to obtain the final output of the model (a classification,
a next word/token etc.)
The following visualization gives an overview of the transformer
architecture.
Theory for Task 1 and Task 2
I. Task 2 – Vietnamese Abstractive Summarization with ViT5 and
Synthetic Data
1. Transformer architecture and the T5 “text-to-text” idea

The Transformer architecture relies on self-attention to model long-range dependencies

in text without recurrent connections. This makes it highly parallelizable and very
effective for modern NLP tasks.

T5 (Text-to-Text Transfer Transformer) proposes a unified view of NLP where every

task (classification, translation, summarization, QA, etc.) is cast as “text in → text out”.
The model is first self-supervised pre-trained on large corpora with a span-corruption
objective, then fine-tuned on downstream tasks by changing only the textual instruction
and dataset. This framework motivates our setup in Task 1: summarization is naturally a
text-to-text problem.

2. ViT5 – a T5-style model for Vietnamese

ViT5 is a Transformer-based encoder–decoder model specialized for Vietnamese. It follows

the T5 pre-training recipe but uses a large, high-quality Vietnamese corpus and is later fine-
tuned on Vietnamese downstream tasks. Experiments show that ViT5 achieves state-of-the-
art performance on Vietnamese abstractive text summarization and competitive results on
named-entity recognition.

In this assignment, we start from a pre-trained ViT5 (e.g., VietAI/vit5-base) and fine-tune it
for Vietnamese summarization.

3. Abstractive summarization

There are two main paradigms in summarization:

- Extractive summarization selects important sentences from the source text without rewriting
them.
- Abstractive summarization generates new sentences that may use different wording while
preserving the main meaning.

ViT5 works in the abstractive setting: the encoder reads the source document and produces
contextual representations, while the decoder generates the summary token by token,
conditioned on the encoder output and previously generated tokens. Training minimizes
cross-entropy (negative log-likelihood) between predicted tokens and the reference summary.

4. Fine-tuning a pre-trained encoder–decoder

The motivation for using a pre-trained model is transfer learning: the model has already
learned general linguistic knowledge and can adapt to downstream tasks with relatively little
task-specific data.

In Task 1, the fine-tuning pipeline is:

1) Pre-processing & tokenization: map each (document, summary) pair into input IDs with
fixed maximum lengths, add special tokens, and create decoder labels.
2) Training objective: minimize cross-entropy over the target summary tokens with teacher
forcing.
3) Decoding at inference time: use beam search or sampling (top-k / nucleus) to generate
summaries.

5. Traditional automatic metrics: ROUGE, BLEU, BERTScore

Because summarization output is open-ended, we need automatic metrics comparing model

outputs with human references.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a family of metrics that

measure n-gram overlap between system summaries and reference summaries. ROUGE-N
counts overlapping n-grams (e.g., ROUGE-1 for unigrams, ROUGE-2 for bigrams);
ROUGE-L uses the longest common subsequence. Higher ROUGE values indicate that more
content from the reference has been captured, though they do not fully reflect semantics or
fluency.

BLEU was originally introduced for machine translation and measures n-gram precision
between system and reference texts, with a brevity penalty to discourage overly short outputs.
It is commonly reused in summarization as a complementary measure of n-gram overlap.

BERTScore is a more recent metric that moves beyond surface n-grams. It computes token-
level similarity using contextual embeddings from pre-trained models (e.g., BERT) and then
aggregates them into precision, recall, and F1 scores. Because it operates in embedding
space, BERTScore is more sensitive to semantic similarity and paraphrasing than ROUGE or
BLEU.

In Task 1, these metrics (ROUGE, BLEU, BERTScore) are used as “traditional” evaluation
to compare the ViT5 baseline and the ViT5 model trained with additional synthetic data.

6. Synthetic data augmentation with LLMs

Synthetic data refers to data points generated by another model rather than collected from
real-world annotations. With strong large language models (LLMs), we can generate
alternative paraphrased summaries for the same document, or entirely new (document,
summary) pairs in the same style.

Recent work shows that synthetic data can significantly improve smaller or mid-sized
models, especially when high-quality human-labeled data are limited, although noisy
synthetic data may hurt performance.

In this task we: (1) train a baseline ViT5 summarizer on the original Vietnamese dataset; (2)
use a strong LLM to generate synthetic summaries (e.g., paraphrases of reference
summaries); (3) fine-tune ViT5 a second time on the mixture of original and synthetic data;
and (4) compare traditional metrics before and after augmentation.

II. Task 3 – Free-form Text Generation with Qwen3-0.6B

1. Autoregressive (decoder-only) language models

Task 2 uses a decoder-only language model, similar in spirit to GPT-style models.

These models are trained with a simple objective:

Predict the next token given all previous tokens. During training, this is done on large
text corpora with teacher forcing; at inference time, the model generates sequences by
repeatedly sampling the next token from its predicted distribution.
Fine-tuning such a model on a task-specific dataset (e.g., Vietnamese stories,
explanations, or instruction-style data) teaches it to produce outputs better aligned
with that domain.

2. The Qwen and Qwen3 family

Qwen is a family of open-weight large language models developed by Alibaba. The

latest generation, Qwen3, includes a range of sizes and both dense and Mixture-of-
Experts variants, and supports many languages and dialects. In this assignment we use
a relatively small model, Qwen3-0.6B, as the base model for Vietnamese free-form
text generation. The model is fine-tuned on a Vietnamese dataset so that its style and
content better match the desired task (for example, answering prompts or writing short
paragraphs).

3. Perplexity as a traditional metric

For language models, perplexity (PPL) is one of the classic evaluation metrics.
Intuitively, it measures how “surprised” the model is by the test data:

Lower perplexity means the model assigns higher probability to the observed text, i.e.,
it predicts tokens more confidently.

However, perplexity has important limitations: it is computed on a fixed reference

corpus, not on model-generated responses to prompts; a model can have low
perplexity but still produce unhelpful, irrelevant, or hallucinated answers; and it
focuses on token-level likelihood, not on task-level qualities like coherence,
informativeness, or faithfulness.

In Task 2, perplexity is used as the traditional quantitative metric to compare different

fine-tuned versions of Qwen3-0.6B.

4. LLM-as-a-Judge for generation quality

For open-ended generation, automatic n-gram metrics (BLEU, ROUGE) and

perplexity often correlate poorly with human preferences. To address this, a growing
line of work uses LLMs themselves as evaluators (“LLM-as-a-Judge”). A strong LLM
receives the prompt and the candidate output (and optionally a reference), and we
design an evaluation prompt that asks it to rate or rank the output according to criteria
such as coherence, relevance to the prompt, fluency/grammar, and factual correctness.

Recent empirical studies show that LLM-as-a-Judge can approximate human ratings
reasonably well and is widely used in practice for evaluating LLM-generated text. In
this assignment, we compute perplexity on a held-out test set, and also sample model
outputs on a set of prompts and send them to a powerful LLM (e.g., GPT-4.1 or
Gemini) with a carefully designed evaluation prompt. The LLM returns scores for
coherence, relevance, fluency, etc., giving an LLM-based evaluation. We can then
compare and discuss whether lower perplexity corresponds to better LLM-based
scores, and how combining traditional and LLM-based metrics provides a more
complete picture of model quality.