tags:

concept

Recurrent Networks (RNN)

Created: 18-02-2026
[!TLDR]
Brief explanation of the whole document in my own words.
Table of Contents:
1. Content
1. Temporal Sequence Processing with
Conventional NN
The most common application is language processing. In this context, using a conventional
NN, we make use of:
Corpus (X ): Training data (text). X = [x , ...., x , ...., x ] where N : size of the corpus in
1 t N

words/tokens.
​ ​ ​

Vocabulary (V ): Set of unique words/tokens in the corpus. V = [w , ..., w , ..., w ] where

1 t ∣V ∣

∣V ∣: size of the vocabulary.

​ ​ ​

Sequence: Set of words/token from which a prediction is made.

Context: Sliding window of duration T .
Image unavailable: Descripción
Figure 1: NLP with conventional NN.
1.1. Embedding
Words or tokens must be converted into fixed-length numeric vectors through the process of
embedding that increases the generalization capacity. This embedding can be:
 Learned: The embeddings are adjusted along with the weights of the network with the
goal of ensuring embeddings of similar words are similar.
Pretrained: Use pretrained embedding instead of learning them. Two types:
Fixed embedding: Loaded and aren't updated during training.
Adjustable embedding: Loaded and fine tuned during training.
1.2. Loss Function
y : desired output; one-hot vector of ∣V ∣ elements (number of classes). Example: y =
[0, ..., 0, 1, 0, ..., 0]∣V ∣×1 ​

y
^: model's output.
​

We use the cross-entropy loss function:

∣V ∣
​^, y) = − ∑ yk log y^k = − log y^c
LCE (y ​ ​ ​ ​ ​ ​ ​

k=1

where c is the correct class.

In NLP, y^ is the probability that the model assigns to the next word w being the word k from
k t

the vocabulary. Therefore, this probability is y^ , and L = − log p(w ∣w , ...w ).

​ ​ ​

c
​ ​
CE ​
t ​
t−1 ​
t−T ​

1.3. Limitations of Conventional NN for NLP tasks

Traditional NNs process data using fixed and static context. Each input is independent,
ignoring temporal relationships or dependencies between words. If the context is small (few
words), then it is very fast, but it cannot capture long-term dependencies; whereas if the
context is large (more words), the model can capture long-term dependencies, but it
becomes computationally expensive and may introduce noise.

2. Recurrent Networks
Recurrent Neural Networks are specialized in processing sequential data. They have
memory: this allows RNNs to retain information from previous elements in the sequence.
This happens because each output depends on the current input and the previous states.
2.1. Simple Recurrent Networks
These have a single hidden layer where the output of the nodes depends on the output of
those same nodes from the previous time step. The context is variable and the activation
function tends to be tanh (g) and softmax (f ).
 Image unavailable: Descripción
Figure 2: Simple Recurrent Network.
d , d , d : Input, hidden layer and output's dimensions.
in ​
h ​
out ​

W ∈ Rdh ×din ​ ​

U ∈ Rdh ×dh ​ ​

V ∈ Rdout ×dh ​ ​

RNNs can be seen as a deep network when it is unwrapped in time (Fig. 3). The matrices U,
W, and V are the same in different times. This matrices change during training, but they are
the same three for all the "layers".
Image unavailable: Descripción
Figure 3: RNN unwrapped in time.
2.2. Training
1. The matrices U, W, and V are initialized randomly (often using Xavier/Glorot
initialization).
2. The vector h is initialized with zeros. 0 ​

3. The weights are updated with gradient like in a common supervised NN.
4. Back-Propagation Through Time (BPTT) is applied for all time intervals.
5. The final gradient of the cost function with respect to U, W, and V is calculated by
summing the T gradients.
2.2.1. Training with mini-batches
Mini-batches are used. Each sequence has its own independent hidden state. Suppose a very
simple case: mini-batch with two short sequences:
s1 = [a, b, c, d]
​

s2 = [e, f ]
​

With padding we get:

s1 = [a, b, c, d]
​

s2 = [e, f , P AD, P AD]

​

The PAD token is an artificial token with an index in the vocabulary and embedding that is
excluded from the error function, ignored via masking and lacks any linguistic meaning, it's
just a computational tool.
In t = 1 we have:
h1−s1 = f (h0 , a)
​ ​

h1−s2 = f (h0 , e)
​ ​

In t = 2 we have:
h2−s1 = f (h1−s1 , b)
​

​
​

h2−s2 = f (h1−s2 , f )
​
​
​

2.2.1. Cost Function for Sequences

The cost function for the complete sequence is the sum of the individual losses for each step
t:
T
L = ∑ l(yt , y
^t ) ​ ​ ​ ​

t=1

where the loss function can be:

Cross-Entropy (Classification): l(y , y^ ) = L (y , y^ ) = − ∑ y log(y^ ) where k is the
t t CE t t
∣V ∣ t t

component, y ​is a one-hot vector where all components y are zero except for the target
​ ​ ​ ​ ​ ​ ​

k=1 ​

k ​

k
​ ​

t
t

class.
​

k ​

Mean Squared Error, MSE (Regression): l(y , y^ ) = L (y , y^ ) = (y − y^ )

t ​

t
​ ​
MSE ​
t ​

t
​ ​
1
2 ​
t ​

t
​ ​
2

[!Concept]
Teacher Forcing:
The model is provided with the ground-truth sequence history to predict the next word
instead of passing the model's best prediction.
Example: Model in t = 0 predicts long , the correct word is length ; in t = 1 we pass
length instead of long .

2.3. Gradients, Problems, and Solutions

Output layer weights: V(m) = V(m − 1) − μ (m) ∂L
∂V
​

Hidden layer weights (recurrent weights): U(m) = U(m − 1) − μ ∂L

∂U (m) ​

Input layer weights: W(m) = W(m − 1) − μ (m) ∂L

∂W
​

Where μ is the learning rate.

Causes of Vanishing Gradients:
Repeated multiplications with small values
Saturated activation functions: Functions such as Sigmoid and Tanh saturate for
extreme values where the derivatives are near zero (Fig. 4).
Long sequences
Image unavailable: Descripción
Figure 4: Saturated activation functions (Sigmoid, Tanh).
Solutions for Vanishing/Exploding Gradients:
Gated architectures (LSTM, GRU)
Alternative activation functions (ReLU)
Gradient clipping: Limit the maximum value of the gradients.
Truncated Back-Propagation Through Time (TBPTT): Limit the number of time steps in
time to perform Back-Propagation.
Data normalization
2.3. Architectures and Their Applications
2.3.1. Sequence to Vector (many-to-one)
Explanation: This architecture processes an input sequence of varying length and
produces a single fixed-size vector as the final output, typically by utilizing the hidden
state of the last time step.
Uses:
Text classification (like sentiment analysis).
Activity recognition based on data sequences.
Image unavailable: Descripción
Figure 5: Sequence to vector.
2.3.2. Sequence to Sequence (many-to-many)
Explanation: This architecture maps an input sequence to an output sequence, either
through a synchronized time-step mapping or an encoder-decoder framework.
Uses:
Prediction of the next word.
 Sequence labeling.
Image unavailable: Descripción
Figure 6: Sequence to sequence.
2.3.3. Vector to Sequence (one-to-many)
Explanation: This architecture takes a single non-sequential input (a vector) and
generates a sequence of outputs over multiple time steps.
Uses:
Image description: input (image) -> output (text description)
Image unavailable: Descripción
Figure 7: Vector to sequence.
2.4. Deep RNN
The lower level network's outputs are used as inputs for the higher-level layers, obtaining
different abstraction levels.
Image unavailable: Descripción
Figure 8: Deep RNN with 3 levels.