Page 1

Importance of Baseline Models and Hyperparameter Selec on in Deep Learning

Introduc on:
Deep Learning models are complex and require proper evalua on and tuning. Baseline models and
hyperparameters play a key role in improving model performance. A baseline gives a reference point,
while hyperparameters control how the model learns.

Baseline Models

Defini on:
A baseline model is a simple model used as a benchmark to compare the performance of advanced
models.

Importance:

• Provides a reference performance

→ Helps to know whether the new model is be er or not

• Helps in model evalua on

→ Ensures improvements are meaningful

• Easy to implement and understand

→ Requires less computa on and me

• Helps in detec ng data issues

→ Poor results indicate problems in dataset

• Avoids unnecessary complexity

→ Simple models may some mes perform well

Types of Baseline Models:

• Random model
• Mean/Median predic on
• Linear Regression / Logis c Regression
• Simple Neural Networks

Example:
In classifica on:
Baseline → Logis c Regression
Advanced → Deep Neural Network
The advanced model should outperform the baseline.

Hyperparameter Selec on
Defini on:
Hyperparameters are parameters set before training that control the learning process.

Page 2

Importance of Hyperparameters

• Controls learning process

→ Affects convergence and stability

• Improves accuracy
→ Proper tuning gives be er results

• Prevents overfi ng & underfi ng

→ Balances model complexity

• Affects training efficiency

→ Impacts me and memory usage

• Determines model structure

→ Number of layers, neurons, etc.

Common Hyperparameters

• Learning Rate
• Batch Size
• Number of Epochs
• Op mizer (SGD, Adam)
• Ac va on Func on
• Dropout Rate

Tuning Methods

• Grid Search
• Random Search
• Bayesian Op miza on
• Manual Tuning

Challenges

• High computa onal cost

• Large search space
• Parameters are interdependent

Rela on Between Baseline & Hyperparameters

Baseline models need minimal tuning, while deep learning models require proper hyperparameter
tuning to perform be er. Without tuning, even complex models may perform worse than baseline.

Conclusion

Baseline models and hyperparameter selec on are essen al in deep learning. Baseline models
provide a standard for comparison, while hyperparameters ensure efficient learning and be er
performance. Together, they help in building accurate and reliable models.

Different Performance Metrics in Deep Learning

Introduc on

Performance metrics are used to evaluate how well a model performs on a given task. Different
metrics are used depending on the problem type such as classifica on or regression.

1. Classifica on Metrics

1.1 Accuracy

Defini on:
Accuracy is the ra o of correctly predicted instances to total instances.

Formula:
Accuracy = (TP + TN) / (TP + TN + FP + FN)

Example:
If a model correctly predicts 90 out of 100 samples →
Accuracy = 90%

1.2 Precision

Defini on:
Precision measures how many predicted posi ve cases are actually posi ve.

Formula:
Precision = TP / (TP + FP)

Example:
Out of 50 predicted posi ve cases, 40 are correct →
Precision = 40/50 = 0.8

1.3 Recall (Sensi vity)

Defini on:
Recall measures how many actual posi ve cases are correctly iden fied.

Formula:
Recall = TP / (TP + FN)

Example:
Out of 60 actual posi ves, model iden fies 45 →
Recall = 45/60 = 0.75

1.4 F1 Score

Defini on:
F1 Score is the harmonic mean of Precision and Recall.

Formula:
F1 = 2 × (Precision × Recall) / (Precision + Recall)

Example:
Precision = 0.8, Recall = 0.75
F1 = 0.774

1.5 Confusion Matrix

Defini on:
A table showing correct and incorrect predic ons.

Predicted + Predicted -

Actual + TP FN

Actual - FP TN

Example:
Used in spam detec on to see correct and wrong classifica ons.

2. Regression Metrics

2.1 Mean Absolute Error (MAE)

Defini on:
Average of absolute differences between actual and predicted values.

Formula:
MAE = (1/n) Σ |yi – ŷi|

Example:
Actual: [10, 20]
Predicted: [12, 18]
MAE = (2 + 2)/2 = 2
2.2 Mean Squared Error (MSE)

Defini on:
Average of squared differences between actual and predicted values.

Formula:
MSE = (1/n) Σ (yi – ŷi)²

Example:
MSE = (4 + 4)/2 = 4

2.3 Root Mean Squared Error (RMSE)

Defini on:
Square root of MSE.

Formula:
RMSE = √MSE

Example:
RMSE = √4 = 2

2.4 R² Score (Coefficient of Determina on)

Defini on:
Measures how well the model explains variance in data.

Range: 0 to 1

Example:
R² = 0.9 → Model explains 90% of variance

Conclusion

Different performance metrics are used based on the problem type. Classifica on metrics like
accuracy, precision, recall, and F1-score evaluate categorical predic ons, while regression metrics like
MAE, MSE, RMSE, and R² measure predic on errors. Choosing the right metric is essen al for proper
model evalua on.
Computer Vision and Speech Recogni on

Introduc on

Computer Vision and Speech Recogni on are two major applica ons of Deep Learning that deal with
understanding visual and audio data. Computer Vision focuses on extrac ng meaningful informa on
from images, while Speech Recogni on converts spoken language into text. Both domains rely on
neural networks and require proper preprocessing techniques to improve performance and accuracy.

Part A: Computer Vision

Defini on

Computer Vision is a field of Ar ficial Intelligence that enables machines to interpret and analyze
visual informa on from images and videos. It aims to replicate human vision by automa cally
extrac ng features such as edges, shapes, textures, and objects from images.

Preprocessing in Computer Vision

Preprocessing is an essen al step in computer vision as raw images may contain noise, ligh ng
varia ons, and inconsistencies. These issues can nega vely affect model performance. Therefore,
preprocessing techniques such as contrast normaliza on and dataset augmenta on are applied to
standardize the input data and improve generaliza on.

Contrast Normaliza on

Contrast normaliza on is used to adjust the intensity values of an image so that all images have a
similar contrast level. Contrast is defined as the varia on in pixel intensi es and is usually measured
using standard devia on.

The mean intensity of the image is calculated as:

1
𝑋¯ = 𝑋, ,
3𝑟𝑐

where 𝑟and 𝑐represent rows and columns, and 𝑘represents color channels.

The contrast (standard devia on) is given by:

⃓
⃓
⃓
⃓
⃓ 1
⃓
⃓ (𝑋 , , − 𝑋¯)
⃓3𝑟𝑐
⃓
⎷
Global Contrast Normaliza on (GCN) is performed using:

𝑋, , − 𝑋¯
𝑋, , =𝑠⋅
1
max 𝜖 , 𝜆 + 3𝑟𝑐 ∑(𝑋 , , − 𝑋¯)

This process removes illumina on differences and scales all images to a common contrast level,
making learning more stable.

Local contrast normaliza on further improves image quality by normalizing each pixel using local
sta s cs:
𝑋−𝜇
𝑋 =
𝜎

This enhances edges and fine details in the image.

Dataset Augmenta on

Dataset augmenta on is a technique used to ar ficially increase the size of the training dataset by
applying transforma ons that do not change the label of the image. This helps in improving model
robustness and reducing overfi ng.

Common augmenta on techniques include rota on, transla on, flipping, scaling, and color
adjustments. For example, an image of a cat can be rotated or flipped, and it will s ll represent the
same class. By genera ng mul ple varia ons of the same image, the model learns to generalize
be er to unseen data.

Convolu onal Neural Networks (CNN)

CNNs are specialized neural networks designed for processing image data. They use convolu on
opera ons to extract features automa cally from images.

The convolu on opera on is defined as:

𝑆(𝑖, 𝑗) = 𝐼(𝑚, 𝑛) ⋅ 𝐾(𝑖 − 𝑚, 𝑗 − 𝑛)

where 𝐼is the input image and 𝐾is the kernel (filter).

An ac va on func on such as ReLU is applied:

𝑓(𝑥) = max (0, 𝑥)

Pooling layers are used to reduce spa al dimensions. For example, max pooling selects the maximum
value:
 𝑓(𝑥) = max (𝑥 , 𝑥 , . . . , 𝑥 )

CNNs reduce parameters through sparse connec vity and parameter sharing, making them efficient
for image processing tasks.

Applica ons of Computer Vision

Computer Vision is widely used in image classifica on, object detec on, facial recogni on, medical
image analysis, and autonomous driving systems. These applica ons demonstrate its ability to
extract meaningful insights from visual data.

Part B: Speech Recogni on

Defini on

Speech Recogni on is the process of conver ng spoken language into text. It enables machines to
understand and process human speech for various applica ons.

Problem Formula on

In speech recogni on, the input is a sequence of audio signals:

𝑋 = (𝑥 ( ) , 𝑥 ( ) , . . . , 𝑥 ( ) )

The output is a sequence of words:

𝑦 = (𝑦 , 𝑦 , . . . , 𝑦 )

The goal is to find the most probable output:

𝑓 ∗ (𝑋) = arg max 𝑃(𝑦 ∣ 𝑋)

This represents selec ng the word sequence that maximizes the probability given the input audio.

Preprocessing in Speech Recogni on

Speech signals are con nuous and must be converted into a suitable format for processing. The
audio is divided into small frames (typically 20 ms each), and features are extracted from each frame.
Noise reduc on techniques are also applied to improve signal quality.

Feature extrac on techniques such as MFCC convert audio signals into frequency-based
representa ons.

The Mel scale is computed as:

 𝑓
𝑀𝑒𝑙(𝑓) = 2595log 1+ 700

This helps in modeling how humans perceive sound frequencies.

Models Used

Speech recogni on systems use deep learning models such as RNNs and LSTMs to handle sequen al
data. CNNs can also be applied to spectrograms, which are visual representa ons of audio signals.

The RNN model is defined as:

ℎ = 𝑓(𝑊𝑥 + 𝑈ℎ + 𝑏)

where ℎ represents the hidden state at me 𝑡.

Applica ons of Speech Recogni on

Speech recogni on is used in voice assistants, speech-to-text systems, automated customer service,
and language transla on systems. It enables natural interac on between humans and machines.

Challenges

Speech recogni on systems face challenges such as background noise, varia ons in accents, speech
speed, and ambiguity in language. These factors make accurate recogni on difficult.

Conclusion

Computer Vision and Speech Recogni on are powerful applica ons of deep learning that enable
machines to interpret visual and audio data. Preprocessing techniques such as contrast normaliza on
and dataset augmenta on significantly improve performance. While CNNs are widely used in
computer vision, sequence models like RNNs are essen al for speech recogni on. With con nuous
advancements, these technologies are becoming more accurate and widely applicable in real-world
scenarios.
LSTM and Other Gated RNNs and Their Advantages in Sequence Learning

Introduc on

Recurrent Neural Networks (RNNs) are designed for modeling sequen al data such as speech, text,
and me-series signals. In sequence learning, the model must capture dependencies across me
steps. However, standard RNNs face difficul es in learning long-term dependencies due to gradient-
related issues. To address this, gated RNN architectures such as LSTM and GRU are introduced,
which regulate informa on flow using ga ng mechanisms.

Limita ons of Standard RNN

In a basic RNN, the hidden state is computed as:

ℎ = 𝑓(𝑊𝑥 + 𝑈ℎ + 𝑏)

Although this structure allows informa on to flow across me, it suffers from:

 Vanishing gradient problem: Gradients shrink during backpropaga on, preven ng learning
of long-term dependencies.

 Exploding gradient problem: Gradients grow excessively, causing unstable training.

As a result, standard RNNs are not effec ve for long sequences.

Long Short-Term Memory (LSTM)

Defini on

LSTM is a gated RNN architecture designed to overcome the limita ons of standard RNNs by
introducing a memory cell that maintains informa on over long me intervals.

LSTM Architecture

An LSTM unit consists of:

 Cell state 𝐶 (long-term memory)

 Hidden state ℎ (short-term output)

 Three gates: Forget gate, Input gate, Output gate

The cell state acts as a conveyor belt that carries informa on across me steps with minimal
modifica on.

LSTM Equa ons and Working

Forget Gate

The forget gate determines which informa on from the previous cell state should be removed:
 𝑓 = 𝜎(𝑊 [ℎ ,𝑥 ] + 𝑏 )

If 𝑓 is close to 0, informa on is forgo en; if close to 1, it is retained.

Input Gate

The input gate decides what new informa on should be stored:

𝑖 = 𝜎(𝑊 [ℎ , 𝑥 ] + 𝑏 )
𝐶 = tanh (𝑊 [ℎ , 𝑥 ] + 𝑏 )

The candidate values 𝐶 represent new informa on to be added.

Cell State Update

The new cell state is computed as:

𝐶 =𝑓 ⋅𝐶 +𝑖 ⋅𝐶

This equa on shows how old informa on is par ally forgo en and new informa on is added.

Output Gate

The output gate controls what informa on is passed to the next hidden state:

𝑜 = 𝜎(𝑊 [ℎ , 𝑥 ] + 𝑏 )
ℎ = 𝑜 ⋅ tanh (𝐶 )

Explana on

The ga ng mechanism allows LSTM to selec vely remember or forget informa on, making it
effec ve in capturing long-term dependencies in sequences.

Gated Recurrent Unit (GRU)

Defini on

GRU is a simplified gated RNN architecture that combines some of the gates in LSTM to reduce
complexity while maintaining performance.

GRU Architecture

GRU consists of:

  Update gate

 Reset gate

It does not maintain a separate cell state; instead, it directly updates the hidden state.

GRU Equa ons

Update Gate

𝑧 = 𝜎(𝑊 [ℎ , 𝑥 ])

This gate determines how much past informa on should be retained.

Reset Gate

𝑟 = 𝜎(𝑊 [ℎ , 𝑥 ])

This gate controls how much previous informa on is ignored.

Hidden State Computa on

ℎ = tanh (𝑊[𝑟 ⋅ ℎ , 𝑥 ])
ℎ = (1 − 𝑧 ) ⋅ ℎ +𝑧 ⋅ℎ

Explana on

GRU simplifies LSTM by merging the forget and input opera ons into a single update gate, making it
computa onally efficient while s ll capturing dependencies.

Other Gated RNN Variants

Bidirec onal RNN

Processes the sequence in both forward and backward direc ons, allowing the model to use both
past and future context.

Stacked (Deep) RNN/LSTM

Mul ple layers of RNN/LSTM are stacked to learn complex hierarchical pa erns in sequences.

Peephole LSTM
Allows gates to directly access the cell state, improving the model’s ability to learn precise ming
rela onships.

Advantages of Gated RNNs in Sequence Learning

Ability to Capture Long-Term Dependencies

Unlike standard RNNs, LSTM and GRU can retain informa on over long sequences, making them
suitable for tasks like language modeling.

Solves Vanishing Gradient Problem

The use of gates ensures smooth gradient flow, allowing effec ve training over long sequences.

Selec ve Memory Mechanism

Gates allow the model to decide what informa on to keep, update, or discard, improving learning
efficiency.

Improved Context Representa on

Gated RNNs can capture rela onships between distant elements in a sequence, leading to be er
understanding of context.

Computa onal Efficiency (GRU)

GRU requires fewer parameters than LSTM, resul ng in faster training and reduced computa onal
cost.

Be er Performance

These models achieve higher accuracy in tasks such as:

 Speech recogni on

 Machine transla on

 Text genera on

Applica ons

 Speech recogni on systems

 Natural language processing

 Time-series predic on
  Sen ment analysis

Conclusion

LSTM and other gated RNNs are powerful architectures designed to overcome the limita ons of
tradi onal RNNs. By introducing ga ng mechanisms, they effec vely control informa on flow and
capture long-term dependencies in sequen al data. This makes them highly suitable for sequence
learning tasks and widely used in modern deep learning applica ons.

Random and Unsupervised Features in CNN

Introduc on

In Convolu onal Neural Networks (CNNs), the most computa onally expensive part is learning the
convolu onal features (kernels). During supervised training using gradient descent, each update
requires both forward propaga on and backward propaga on through all layers, which makes
training costly. To reduce this computa onal burden, alterna ve approaches are used where features
are not learned in a fully supervised manner.

Cost of Feature Learning in CNN

In CNNs, the final (output) layer is rela vely inexpensive because it operates on a reduced number of
features a er mul ple pooling layers. However, learning convolu onal kernels is expensive since:

 Each training step requires full forward and backward propaga on

 Large datasets and deep architectures increase computa on

 Feature learning dominates training cost

To address this, non-supervised feature genera on methods are used.

Strategies for Obtaining Convolu on Kernels Without Supervised Training

There are three main strategies:

1. Random Ini aliza on of Kernels

In this approach, convolu on kernels are ini alized randomly and not updated during training.

Explana on

Surprisingly, random filters can s ll extract useful features such as edges and textures. When
convolu on is followed by pooling:
  The network becomes frequency selec ve

 It achieves transla on invariance

Thus, even without learning, useful representa ons can emerge.

Advantage

 Very low computa onal cost

 No training required for feature extrac on

Limita on

 Features are not op mized for the task

2. Handcra ed Kernels

In this approach, kernels are manually designed based on domain knowledge.

Examples

 Edge detectors (horizontal, ver cal)

 Texture filters

Explana on

Each kernel is designed to detect specific pa erns such as edges at different orienta ons or scales.

Advantage

 No need for training

 Interpretable features

Limita on

 Limited flexibility

 Requires expert knowledge

3. Unsupervised Learning of Kernels

In this approach, kernels are learned using unsupervised learning methods instead of labeled data.

Example: K-Means Clustering

Small image patches are extracted, and k-means clustering is applied:

min ∑ ∣∣ 𝑥 − 𝜇 ∣∣

where 𝜇 are cluster centroids.

These centroids are then used as convolu on kernels.

Explana on

 Features are learned independently of the classifier

 A er learning kernels, features are extracted once for the dataset

 The final layer (e.g., logis c regression or SVM) is trained separately

Since the last layer is o en convex, op miza on becomes easier.

Advantages

 Reduces training cost

 Allows separa on of feature learning and classifica on

 Works well with limited labeled data

Layer-wise Unsupervised Pretraining

An intermediate approach is greedy layer-wise pretraining.

Method

 Train first layer independently

 Extract features from first layer

 Train second layer using these features

 Con nue layer by layer

Explana on

This avoids full forward and backward propaga on through the en re network at every step,
reducing computa onal cost.

Example

Convolu onal Deep Belief Network (CDBN)

 Uses unsupervised learning for each layer

 Learns hierarchical features

Patch-Based Training Approach

Instead of training full convolu on layers:

 Train a model on small image patches

 Use learned parameters as kernels

Advantage

 No need to perform convolu on during training

 Convolu on is used only during inference

This allows training of very large models efficiently.

Observa ons on Random Filters

Research shows that:

 Random filters + pooling produce useful representa ons

 They provide transla on invariance

 They are useful for architecture selec on

Prac cal Approach

1. Try different architectures with random filters

2. Train only the last layer

3. Select best architecture

4. Fully train selected model

Use of Unsupervised Learning in Reinforcement Learning (RL)

Need for Unsupervised Learning in RL

Reinforcement Learning o en deals with:

 High-dimensional inputs (images, states)

 Sparse reward signals

Unsupervised learning helps in extrac ng useful representa ons.

Applica ons

1. Representa on Learning

Unsupervised models learn compact state representa on:

𝑧 = 𝑓(𝑥)
This reduces dimensionality and improves learning.

2. Feature Extrac on

Instead of raw inputs, RL agents use learned features, improving efficiency.

3. Pretraining

Unsupervised learning can ini alize models before RL training, reducing learning me.

4. Model Learning

Learn environment dynamics:

𝑠 = 𝑓(𝑠 , 𝑎 )

This supports model-based RL.

5. Improved Explora on

Unsupervised signals help the agent discover structure in the environment.

Advantages in RL

 Reduces dependency on rewards

 Improves sample efficiency

 Handles high-dimensional data

 Speeds up convergence

Conclusion

Random and unsupervised feature learning provide efficient alterna ves to expensive supervised
training in CNNs. Methods such as random ini aliza on, handcra ed filters, and unsupervised
learning (e.g., k-means) allow feature extrac on without full backpropaga on. These approaches
significantly reduce computa onal cost and enable training of large models. Furthermore,
unsupervised learning plays a crucial role in reinforcement learning by improving representa on,
explora on, and efficiency.
Efficient Convolu on Algorithms

Introduc on

Modern Convolu onal Neural Networks (CNNs) o en contain millions of units, making computa on
very expensive. Efficient implementa on of convolu on opera ons is essen al to reduce training and
inference me. While parallel computa on helps, selec ng efficient convolu on algorithms can
significantly improve performance.

Need for Efficient Convolu on

Convolu on is the most computa onally intensive opera on in CNNs because:

 It is applied repeatedly across mul ple layers

 Large input sizes and kernels increase computa on

 Deep networks require millions of opera ons

Thus, op mizing convolu on is crucial for real-world applica ons.

Convolu on Using Fourier Transform

Concept

Convolu on in the spa al domain can be converted into mul plica on in the frequency domain
using Fourier Transform.

Mathema cal Representa on

Convolu on opera on:

𝑦=𝑥∗𝑤

Using Fourier Transform:

𝑦=ℱ (ℱ(𝑥) ⋅ ℱ(𝑤))

where:

 ℱ= Fourier Transform

 ℱ = Inverse Fourier Transform

Explana on

Steps involved:
 1. Convert input 𝑥and kernel 𝑤into frequency domain

2. Perform element-wise mul plica on

3. Convert result back using inverse transform

Advantage

 Faster for large inputs and kernels

 Reduces computa onal complexity in some cases

Limita on

 Overhead of transforma on may not always be beneficial for small inputs

Separable Convolu on

Defini on

A kernel is called separable if it can be expressed as the outer product of vectors.

Mathema cal Representa on

For a 2D kernel:

𝐾 =𝑎⊗𝑏

where:

 𝑎and 𝑏are 1D vectors

 ⊗represents outer product

Explana on

Instead of performing one 2D convolu on:

 Perform two 1D convolu ons sequen ally

o First along rows

o Then along columns

Complexity Comparison

Let kernel size be 𝑤in 𝑑dimensions:

Naive Convolu on:

𝑂(𝑤 )

Separable Convolu on:

𝑂(𝑤 × 𝑑)

Advantages

 Significant reduc on in computa on

 Fewer parameters required

 Faster execu on

Limita on

 Not all kernels are separable

 Approxima on may be required in some cases

Other Efficiency Considera ons

Algorithm Op miza on

Researchers con nuously develop faster and approximate convolu on methods that:

 Reduce computa on me

 Maintain model accuracy

Prac cal Importance

In real-world applica ons:

 Inference speed is more cri cal than training speed

 Efficient forward propaga on is highly valuable

This is because deployment systems handle large-scale data in real me.

Summary

Efficient convolu on algorithms are essen al for handling large-scale CNNs. Techniques such as
Fourier Transform-based convolu on and separable convolu on reduce computa onal complexity
and improve performance. While not all methods apply to every case, selec ng appropriate
algorithms significantly enhances efficiency without compromising accuracy.