CSE465
Topic 3a
Recurrent Neural Network
(Basics)
CSE465: Pattern Recognition and Neural Network
Sec: 3
Faculty: Silvia Ahmed (SvA)
Spring 2025
Topics
1. What is a Convolutional Neural Network (CNN)?
2. Basic intuition.
3. Visual Cortex vs CNN
4. Operations:
1. Convolution
2. Padding
3. Stride
4. Pooling
5. CNN Architecture
Silvia Ahmed (SvA) CSE465 ECE@NSU 2
ANN and CNN so far..
Silvia Ahmed (SvA) CSE465 ECE@NSU 3
Sequential Data
• Sequential data is a type of data where the order of elements
matters.
• Each piece of information depends on previous and/or future
elements in the sequence.
• Unlike regular data, where items can be shuffled without losing
meaning, sequential data follows a specific order—changing
this order alters its meaning.
Silvia Ahmed (SvA) CSE465 ECE@NSU 4
Example
• Text (Natural Language Processing)
• Example: "I love programming" vs. "Programming love I“
• The meaning depends on the word order.
• Time Series Data
• Example: Daily stock prices, weather temperatures, or heart rate over
time.
• Yesterday’s stock price affects today's price.
• Speech and Audio
• Example: A sentence in speech must be heard in order; otherwise, it
becomes gibberish.
• Videos
• Frames in a video must play in the correct order for the story to make
sense.
Silvia Ahmed (SvA) CSE465 ECE@NSU 5
Sequential vs Other data types
Feature Sequential Data Non-Sequential Data
Order Matters? ✅ Yes ❌ No
Examples Text, speech, stock prices Images, customer records,
tabular data
Shuffling Allowed? ❌ No (Changes meaning) ✅ Yes (No impact on
meaning)
Memory Required? ✅ Yes (Past elements ❌ No (Each element is
influence the present) independent)
Silvia Ahmed (SvA) CSE465 ECE@NSU 6
Is ANN or CNN suitable for Sequential data?
Input Output hi ANN
Hi my name is NSU 0
I love Bangladesh 0 my
Bangladesh won the match 1
name
• Vectorize (OHE, Bag Of Words, TFIDF, Word2Vec)
• One Hot Encoding: is
• Unique words: 12
• Take a vector of 12 size (12x5)x4 = 240 weights
• hi: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] nsu
• my: [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Silvia Ahmed (SvA) CSE465 ECE@NSU 7
Sentence # 1 Sentence # 2
hi
i
my
Problem:
Varying input size
name love
is
bangladesh
(12x3)x4 = 144 weights
nsu
Silvia Ahmed (SvA) CSE465 ECE@NSU 8
Problem:
Sentence # 1
Varying input size i
Solution: love
Zero Padding
bangladesh
Dictionary: 10000 words
Sentence: 100 words
Input weights : 1,000,000 [0]
New problem: [0]
Unnecessary computation
Silvia Ahmed (SvA) CSE465 ECE@NSU 9
Overall problems using ANN and CNN
• Text input → Varying size
• Zero padding → Unnecessary computation
• Prediction Problem
• Totally disregarding the sequence information
• Recurrent Neural Network:
• ✅ Best suited for sequential data because:
• RNNs have loops that allow them to "remember" past information, making them
ideal for tasks where order matters.
• Example: In text generation, RNNs can remember previous words to generate
coherent sentences.
Silvia Ahmed (SvA) CSE465 ECE@NSU 10
Application of RNN
• Natural Language Processing (NLP)
• Machine Translation (e.g., Google Translate)
• Converts text from one language to another while maintaining context.
• Text Generation (e.g., Chatbots, AI Writing Assistants)
• Predicts the next word in a sentence to generate human-like text.
• Sentiment Analysis (e.g., Analyzing Customer Reviews)
• Determines whether a review is positive, negative, or neutral.
• Speech-to-Text (e.g., Voice Assistants like Siri, Alexa)
• Converts spoken words into written text.
Silvia Ahmed (SvA) CSE465 ECE@NSU 11
Application of RNN (contd.)
• Time Series Forecasting
• Stock Price Prediction
• Predicts future stock prices based on past trends.
• Weather Forecasting
• Uses past weather patterns to predict future weather conditions.
• Sales Forecasting
• Helps businesses predict future sales based on historical data.
Silvia Ahmed (SvA) CSE465 ECE@NSU 12
Application of RNN (contd.)
• Speech and Audio Processing
• Speech Recognition (e.g., Google Voice Search)
• Converts spoken language into text.
• Music Generation (e.g., AI-Generated Music)
• AI composes music by learning patterns in existing compositions.
• Voice Cloning
• Generates human-like speech from a small sample of a person’s voice.
Silvia Ahmed (SvA) CSE465 ECE@NSU 13
Application of RNN (contd.)
• Video and Image Processing
• Lip Reading AI
• Uses RNNs to predict words by analyzing lip movements in videos.
• Handwriting Recognition (e.g., Digitizing Handwritten Notes)
• Converts handwritten text into digital text.
• Action Recognition in Videos (e.g., Surveillance Systems)
• Identifies human activities like running, walking, or fighting in security footage.
Silvia Ahmed (SvA) CSE465 ECE@NSU 14
Application of RNN (contd.)
• Healthcare Applications
• Disease Progression Prediction
• Predicts how a disease (e.g., Parkinson’s, Alzheimer’s) will develop over time.
• Medical Report Generation
• Automatically summarizes medical findings from patient data.
• ECG and EEG Signal Analysis
• Helps detect heart or brain abnormalities by analyzing sequential health data.
Silvia Ahmed (SvA) CSE465 ECE@NSU 15
Application of RNN (contd.)
• Autonomous Systems
• Self-Driving Cars (e.g., Tesla Autopilot)
• RNNs help predict pedestrian movements and vehicle behavior.
• Robotics
• Enables robots to understand and react to sequential actions.
• Chatbots & Virtual Assistants
• Conversational AI (e.g., ChatGPT, customer service bots)
• Helps machines understand and respond in human-like conversations.
Silvia Ahmed (SvA) CSE465 ECE@NSU 16
Application of RNN (contd.)
• Fraud Detection & Anomaly Detection
• Credit Card Fraud Detection
• Identifies unusual spending patterns in transactions.
• Cybersecurity Threat Detection
• Detects suspicious network activity to prevent cyberattacks.
• Gaming and AI Agents
• AI Players in Games (e.g., OpenAI’s Dota 2 Bot)
• AI uses RNNs to predict opponent moves and react strategically.
• DNA Sequence Analysis
• Genomics & Drug Discovery
• Helps in analyzing genetic sequences for medical research.
Silvia Ahmed (SvA) CSE465 ECE@NSU 17
CSE465
Topic 4: Recurrent Neural Network
RNN Architecture
CSE465 18
Data for RNN (Keras)
• (timesteps, input_features)
Review Sentiment
• Vocabulary: Movie was good 1
• No of unique words: 5 Movie was bad 0
• movie: [1 0 0 0 0] Movie was not good 0
• was: [0 1 0 0 0]
• good: [0 0 1 0 0]
• bad: [0 0 0 1 0]
• not: [0 0 0 0 1]
• Review 1: [1 0 0 0 0], [0 1 0 0 0], [0 0 1 0 0] (3, 5)
• Review 2: [1 0 0 0 0], [0 1 0 0 0], [0 0 0 1 0] (3, 5)
• Review 3: [1 0 0 0 0], [0 1 0 0 0], [0 0 0 0 1], [0 0 1 0 0] (4, 5)
Silvia Ahmed (SvA) CSE465 ECE@NSU 19
How RNN works?
Vocabulary:
Review Sentiment
• movie: [1 0 0 0 0]
X1 Movie was good 1
• was: [0 1 0 0 0]
X11 X12 X13
• good: [0 0 1 0 0]
X2 Movie was bad 0
• bad: [0 0 0 1 0]
X3 Movie was not good 0
• not: [0 0 0 0 1]
state
ANN
t = 1: X11
t = 2: X12
input output t = 3: X13
hidden
Feed forward
Silvia Ahmed (SvA) CSE465 ECE@NSU 20
How RNN works? (contd.)
Review Sentiment state
X1 Movie was good 1
X11 X12 X13
X2 Movie was bad 0
X3 Movie was not good 0
Vocabulary:
• movie: [1 0 0 0 0] 𝑦ො
• was: [0 1 0 0 0]
• good: [0 0 1 0 0]
• bad: [0 0 0 1 0]
• not: [0 0 0 0 1]
Recurrent hidden unit
Silvia Ahmed (SvA) CSE465 ECE@NSU 21
How RNN works? (contd.)
t=1 Review Sentiment
1 X1 Movie was good 1
0 X11 X12 X13
𝑦ො 0
0
Recurrent 0
hidden unit t=2 For t = 1, the feedback
0 will be either zeros or
Number of weights and biases: 1 random numbers
• Input → hidden: 5 x 3 = 15
• Hidden feedback: 3 x 3 = 9 0
• Hidden → output: 3 x 1 = 3 0
• Biases: 3 + 1 = 4
0
Silvia Ahmed (SvA) CSE465 ECE@NSU 22
RNN Forward Propagation
Review Sentiment
X11, X12, X13 1
Forward → unfolding/unrolling through time
X21, X22, X23 0
Activation:
X31, X32, X33, X34 0 At t = 1: tanh/ReLU
𝑋11 𝑊𝑖 + 𝑏 𝑓(𝑋11 𝑊𝑖 + 𝑏)
Xij: Vectors of dim 5 Loop
Wi : (5,3) O1
(1,5),(5,3)
(1,3) (1,3) (1,3)
o1
t=1 Wi
X11
Silvia Ahmed (SvA) CSE465 ECE@NSU 23
RNN Forward Propagation
Review Sentiment
X11, X12, X13 1
Forward → unfolding/unrolling through time
X21, X22, X23 0
Activation:
X31, X32, X33, X34 0 tanh/ReLU
At t = 2: 𝑓(𝑋12 𝑊𝑖 + 𝑜1 𝑊ℎ + 𝑏)
Loop 𝑋12 𝑊𝑖 + 𝑏 + 𝑜1 𝑊ℎ
Wi : (5,3) O2
(1,3),(3,3)
o1 o2 (1,3) (1,3)
Wh
t=1 Wi t=2 Wi
X11 X12
Silvia Ahmed (SvA) CSE465 ECE@NSU 24
RNN Forward Propagation
Review Sentiment W0
X11, X12, X13 1
Forward → unfolding/unrolling through time
X21, X22, X23 0
Activation:
X31, X32, X33, X34 0 tanh/ReLU
At t = 3: 𝑓(𝑋13 𝑊𝑖 + 𝑜2 𝑊ℎ + 𝑏)
Loop 𝑋13 𝑊𝑖 + 𝑏 + 𝑜2 𝑊ℎ
Wi : (5,3) O3
(1,3),(3,3)
o1 o2 o3 (1,3) (1,3)
t=1 Wi t=2 Wi t=3 Wi After t = 3:
X11 X12 X13 𝑜3 𝑊2 + 𝑏 Activation:
sigmoid
(1,3),(3,1) 𝑓(𝑜3 𝑊0 + 𝑏)
(1,1) (1,1) (1,1)
Silvia Ahmed (SvA) CSE465 ECE@NSU 25
CSE465
Topic 4: Recurrent Neural Network
Types of RNN Architecture
CSE465 26
Types of RNN
RNN
Many-to- One-to- Many-to- One-to-
One Many Many One
Silvia Ahmed (SvA) CSE465 ECE@NSU 27
Many-to-One (M→1) RNN
• Multiple input time steps, but only one output.
• The RNN processes the entire sequence and generates a single output
based on all past inputs.
• Example Use Cases:
• Sentiment Analysis:
• Input: A sentence ("I love this movie, it was amazing!") 𝑦ො
• Output: A single sentiment label (e.g., positive)
• Spam Detection:
• Input: An email with multiple words o1 o2 o3
• Output: A single label (spam or not spam)
• Stock Price Prediction
t=1 Wi t=2 Wi t=3 Wi
• Input: Past 30 days of stock prices
• Output: Stock price on day 31 X11 X12 X13
Silvia Ahmed (SvA) CSE465 ECE@NSU 28
One-to-Many (1→M) RNN
• A single input produces a sequence of outputs over time.
• Used for sequence generation tasks where output unfolds over time.
• Example Use Cases
• Music Generation:
• Input: A starting note
• Output: A full melody
• Image Captioning
• Input: A single image
• Output: A sequence of words describing the image
• Text-to-Speech (TTS)
• Input: A single word
• Output: Multiple sound waves forming a spoken version
Silvia Ahmed (SvA) CSE465 ECE@NSU 29
Many-to-Many (M→M) RNN
• This category has two variations:
• Equal Length Many-to-Many (M→M)
• Unequal Length Many-to-Many (M→M with different input/output lengths)
• (a) Many-to-Many (Equal Input & Output Lengths):
• Every input time step corresponds to an output at the same time step.
• Example:
• Parts of Speech Tagging
• Input: “My name is Silvia”
• Output: (pronoun, noun, verb, noun)
• Named Entity Recognition (NER)
• Input: “Alice lives in Paris.”
• Output: (Name, Verb, Location)
Silvia Ahmed (SvA) CSE465 ECE@NSU 30
One-to-One (1→1) RNN
• Each input corresponds to a single output.
• This is the simplest form of an RNN and behaves similarly to a traditional
feedforward neural network (FNN).
• Example Use Cases:
• Simple Binary Classification
• Input: A single image
• Output: Label (e.g., "Cat" or "Dog")
• Why Not Use RNN for One-to-One?
• Since RNNs are designed for sequential data, using them for static one-to-one tasks
is inefficient.
• A simple Artificial Neural Network (ANN) or Convolutional Neural Network (CNN)
(for images) is better suited.
Silvia Ahmed (SvA) CSE465 ECE@NSU 31
CSE465
Topic 4: Recurrent Neural Network
Backpropagation Through
Time (BPTT)
CSE465 32
Example: M→1 Sentiment Analysis
Review Sentiment W0
𝑜1 = 𝑓(𝑋11 𝑊𝑖 + 𝑜1 𝑊ℎ + 𝑏)
X11, X12, X13 1
X21, X22, X23 0 𝑜2 = 𝑓(𝑋12 𝑊𝑖 + 𝑜1 𝑊ℎ + 𝑏)
X31, X32, X33, X34 0
𝑜3 = 𝑓(𝑋13 𝑊𝑖 + 𝑜2 𝑊ℎ + 𝑏)
Xij: Vectors of dim 5 Loop
𝑦ො = 𝜎(𝑜3 𝑊𝑜 + 𝑏)
o0 o1 o2 o3
t=1 Wi t=2 Wi t=3 Wi
X11 X12 X13
Silvia Ahmed (SvA) CSE465 ECE@NSU 33
Silvia Ahmed (SvA) CSE465 ECE@NSU 34
Silvia Ahmed (SvA) CSE465 ECE@NSU 35
Silvia Ahmed (SvA) CSE465 ECE@NSU 36