Chapter 4: Machine Translation & Generation

Machine Translation (MT): Machine translation is a sub-field of computational linguistics that

focuses on developing systems capable of automatically translating text or speech from one
language to another. In Natural Language Processing (NLP), the goal of machine translation is
to produce translations that are not only grammatically correct but also convey the meaning
of the original content accurately.

 The automatic process of converting text or speech from one natural language (source
language) into another (target language) using computational methods.
 Goal: Achieve translation close to human quality, maintaining meaning, grammar, and
fluency.

Importance of Machine Translation

 Global Communication – Breaking language barriers in business, education, and

government.

 Accessibility – Enabling access to information in multiple languages.

 Efficiency – Faster and cheaper than human translation for large volumes of text.

 Applications: Google Translate

Real-time speech translation (Skype Translator, Zoom)

Types of Machine translation

1. Rule-Based Machine Translation

Rule-Based Machine Translation (RBMT) is the earliest approach to automatic translation.

It translates text using a set of linguistic rules (grammar, syntax, and dictionaries) defined by
experts for both the source and target languages.

Working steps:

[Link] (Source Language Processing):

 The source sentence is analyzed grammatically (morphology, syntax, semantics).

 Example: "The boy eats an apple." → subject (boy), verb (eats), object (apple).
[Link] (Mapping Source → Target):

 The analysed structure is mapped into an equivalent structure of the target language
using transfer rules.
 Example: English → French mapping of subject–verb–object structure.

[Link] (Target Language Processing):

 The target sentence is generated using target language grammar and vocabulary.
 Example: "The boy eats an apple." → "Le garçon mange une pomme."

Types of RBMT

There are three main sub-types of RBMT:

1. Direct Translation:
 Word-for-word translation using a dictionary.
 Fast but inaccurate (ignores grammar).
 Example: "The house is big" → "La maison est grande" (works fine), but o en
fails with idioms.
2. Transfer-Based Translation:
 Translates after analyzing the grammar of the source language, then transfers
it into target grammar using rules.
 More accurate than direct translation.
3. Interlingua-Based Translation:
 Converts the source text into an intermediate language-independent
representation (interlingua).
 Then generates the target sentence from this interlingua.
 Advantage: One interlingua → Many target languages.
 Example: English → Interlingua → French/Spanish/Chinese.

2. Statistical Machine Translation

SMT uses statistical models to translate text, based on bilingual corpora. Over time, several
SMT approaches were developed, each improving on the previous one.

The main types of SMT are:

1. Word-Based SMT

 Translates one word at a time, based on word alignment probabilities.

 Example: English "house" → French "maison" (highest probability from
dictionary).
2. Phrase-Based SMT (most widely used in practice)

 Translates phrases (contiguous sequences of words) instead of single words.

 Phrases are learned from bilingual corpora using alignment.
 Includes reordering models to handle different word orders between
languages.

3. Syntax-Based SMT

 Uses syntactic parse trees of source and target languages.

 Aligns sentences based on grammatical structures (subject, verb, object).
 Produces translations by mapping syntactic units.
 Captures grammatical structure → be er for languages with different word
orders (e.g., English ↔ Japanese).

3. Neural Machine Translation (NMT)

A neural network, inspired by the human brain, is a network of interconnected nodes

functioning as an information system. Input data passes through these nodes to produce an
output. Neural machine translation software utilizes neural networks to process vast
datasets, with each node contributing a specific change from source text to target text until
the final result is obtained at the output node.

Aspect SMT (Statistical) NMT (Neural)

Statistical & phrase-based Deep learning (seq2seq with
Approach
models attention/transformer)
Limited (phrases/short
Context Full sentence context
chunks)
Fluency Rigid, less natural Smooth, human-like
Data Requirement Works with smaller datasets Needs large datasets
Resource Usage Low–moderate High (GPU/TPU needed)
Transparent (phrase tables,
Interpretability Black-box, harder to interpret
alignments)
Handling Rare
Better with phrase tables Initially poor, improved with sub words.
Words
Development
Needs feature engineering End-to-end training
Effort
Seq2seq with Attention mechanism

Seq2Seq (Sequence-to-Sequence) is a neural network architecture used to transform one

sequence into another (e.g., English → French transla on).

It has two main parts:

 Encoder: Reads the input sequence and compresses it into a context (hidden state).
 Decoder: Uses that context to generate the output sequence step by step.

Before Attention: RNN and Seq2Seq Problem

 Sequence-to-Sequence (Seq2Seq) models

Encoder: Reads the input sentence → compresses into a single context vector.

Decoder: Uses this vector to generate the output sentence.

Limitation: For long sentences, squeezing everything into one vector loses important details.

The decoder forgets earlier words because RNNs/LSTMs struggle with long-term
dependencies.

Example: Input: “I live in India. It is a beautiful country.”

→ When predic ng the word “country”, the model might forget “India” because it was far
behind.

Attention: The Solution

 Instead of only using one vector, the decoder can look back at all encoder outputs and
decide which parts matter most for the current word.
 Attention acts like a searchlight/spotlight.

Text Generation Techniques: Text generation is the process of creating new sequences of text
(sentences, paragraphs, etc.) that resemble human-written language.

 Example: Input → “Once upon a me”, Model Output → “there lived a king in a
faraway land…”
 It is widely used in chatbots, story writing, translation, summarization, and code
generation.
LSTM: Long Short-Term Memory

An LSTM (Long Short-Term Memory) is a special type of Recurrent Neural Network (RNN) that
can remember information for long periods and decide what to keep or forget.
It uses gates to control the flow of information through time.

 Recurrent Neural Networks (RNNs) generate text by passing information from one
word (or character) to the next through hidden states.
 But when sequences are long, RNNs face the vanishing gradient problem:
o During training, the error signal (gradient) becomes extremely small as it is
backpropagated through many time steps.
o As a result, RNNs “forget” information from the distant past

How LSTMs Solve This

LSTMs (Long Short-Term Memory networks) were designed to fix the forgetting issue using
special gates:

1. Forget Gate

The Forget Gate decides which information from the previous cell state should be discarded
(forgotten).

[Link] Gate

The Input Gate decides what new information should be added to the cell state.

[Link] State

The Cell State acts as the long-term memory of the LSTM — it carries important information
through many time steps.

[Link] Gate

The Output Gate decides what information from the cell state should be output as the new
hidden state.

Transformers for text understanding and generation

Transformers are deep learning models introduced in the paper “Attention is All You Need”
They replaced older sequence models like RNNs and LSTMs, which process text step by step,
with a model that processes the entire sequence in parallel using self-attention.
Transformer Architecture

The full Transformer has two parts:

 Encoder: Reads input text and produces rich contextual representations.

 Decoder: Uses encoder outputs + its own attention to generate output text word by
word.

 For text understanding → we use the encoder (e.g., BERT).

 For text generation → we use the decoder (e.g., GPT).
 For translation or seq2seq tasks → we use encoder–decoder (e.g., original
Transformer, T5).

Transformers for Text Understanding

 Tasks: Classification, Sentiment Analysis, Question Answering, Named Entity

Recognition.
 Model Example: BERT (Bidirectional Encoder Representations from Transformers)

 Uses only the encoder.

 Learns contextual embeddings (each word’s meaning depends on its
neighbors).
 Example: “bank” in “river bank” vs “money bank” is understood differently.

Transformers for Text Generation

 Tasks: Machine Translation, Summarization, Story Generation, Chatbots.

 Model Example: GPT (Generative Pre-trained Transformer)

 Uses only the decoder.

 Predicts the next word given the previous words (causal/auto-regressive).
 Example: Input “The cat is on the” → Output “mat”.

 Encoder–Decoder models like T5 and BART are used for summarization and
translation.

Role of Auto encoders and decoders

Auto encoders are a special type of neural networks that learn to compress data into a
compact form and then reconstruct it to closely match the original input. They consist of
an:
 Encoder that captures important features by reducing dimensionality.
 Decoder that rebuilds the data from this compressed representation.

Architecture of Autoencoder
An autoencoder’s architecture consists of three main components that work together to
compress and then reconstruct data which are as follows:

1. Encoder
It compress the input data into a smaller, more manageable form by reducing its
dimensionality while preserving important information. It has three layers which are:
 Input Layer: This is where the original data enters the network. It can be images, text
features or any other structured data.
 Hidden Layers: These layers perform a series of transformations on the input data. Each
hidden layer applies weights and activation functions to capture important patterns,
progressively reducing the data's size and complexity.
 Output(Latent Space): The encoder outputs a compressed vector known as the latent
representation or encoding. This vector captures the important features of the input
data in a condensed form helps in filtering out noise and redundancies.
2. Bottleneck (Latent Space)
It is the smallest layer of the network which represents the most compressed version of the
input data. It serves as the information bottleneck which force the network to prioritize the
most significant features. This compact representation helps the model learn the underlying
structure and key patterns of the input helps in enabling better generalization and efficient
data encoding.
3. Decoder
It is responsible for taking the compressed representation from the latent space and
reconstructing it back into the original data form.
 Hidden Layers: These layers progressively expand the latent vector back into a higher-
dimensional space. Through successive transformations decoder attempts to restore
the original data shape and details
 Output Layer: The final layer produces the reconstructed output which aims to closely
resemble the original input. The quality of reconstruction depends on how well the
encoder-decoder pair can minimize the difference between the input and output during
training.