DSC 345/445

4.3 Short on Deep Learning

Data Representation
• In traditional learning algorithms (e.g., decision tree, KNN, etc.), it is assumed that
the features are given and fixed.

• There are algorithms for better feature extraction/selection and dimensionality

reduction, but these are usually separated from the learning models.

• In neural networks, by increasing the

number of layers, we let the model
extract features that are most important .

Figure: Chollet, Francois. Deep learning with Python.

Simon and Schuster, 2021.
Deep Learning
• The idea of having successive layers for obtaining increasingly meaningful
representations is at the core of deep learning.

• With the recent leaps in computational resources, many more layers could be used
in the neural network models  deeper architectures.

• As we increase the depth, number of parameters will explode:

• More data will be needed
• Higher flexibility of the model, hence higher risk of overfitting
 Different Models
• There are many different types of (deep) neural networks. Some of them are:
• Autoencoders
• Convolutional neural networks
• Recurrent neural networks
• Generative adversarial models
• Transformers
encoder decoder
• …

• Autoencoders: self-supervised models

for learning data representation.
• The loss would be the difference between
the original image and the reconstructed
one.
Deep Learning in Computer Vision
• Computer vision was probably the first domain where deep learning showed
strong results in early 2010’s.

• Vision models in deep learning usually use convolutional layers.

• Using regular networks, we should vectorize a 2D/3D image.

RGB channels

ℙ(cat)
ℙ(horse)
…

…
MxNx3
Convolution Layers
• By vectorizing we lose the spatial structure and local patterns existing in the image.
• Convolutional layers can preserve and decode these spatial feature efficiently.

After fully sweeping the …

… image using our filter …
…

…
5x5x3
filter
…

activation map
M’ x N’

MxNx3
Convolution Layers
• By vectorizing we lose the spatial structure and local patterns existing in the image.
• Convolutional layers can preserve and decode these spatial feature efficiently.

2 3 1 7 5
5 2 12 9 17 1 1 1 17 19 23
10
3
0
9
1
5
7
6
2
1
✽ 0
1
0
1
0
1
= 36 43 50
25 32 50
4 7 3 14 23

2x1 + 2x1
1x1
3x1
5x1 3x1 + 12x1
7x1
1x1 1x1 ++12x0
5x1
7x1 5x0
2x0 ++12x0
10x0 2x0
+9x0
++12x0
0x0 +17x0
9x0 10x1
1x1++1x1
1x0++0x1
3x1 0x1
7x1++7x1
9x1 1x1
2x1==19
5x1 17
23
36
Convolution Layer
• Number of channels in filters is the same as
the input image.
1 pixel d

…
filter 1 filter 2 filter k

MxNx3 k activation maps

M’ x N’ x k
Convolution Layer
• We usually feed the model with more than just one image.
• A batch of b images.

…
kx3x5x5

k activation maps k activation maps

for image 1 for image b
bx3xMxN b x k x M’ x N’
Convolution Neural Network (CNN)
• We can continue applying more convolution layers.
• We can also add non-linear activations after each layer.

Convolving with Convolving with

k1 filters k2 filters
+ 𝑎(.) + 𝑎(.)
…

bxdxMxN b x k1 x M’ x N’ b x k2 x M’’ x N’’

size of 2D images
batch size

# channels
Pooling
• Pooling is usually used to make the layers’ outputs smaller.
• It’s like a downsampling with no trainable parameters.

Max-pooling with
2x2 filter and
2 3 1 7 stride 2
5 2 12 9 5 12
10 0 1 7 10 7
3 9 5 6
b x d x M/2 x N/2
bxdxMxN
Fully Connected Layer
• After multiple convolution layers, we can have one or more fully connected layers
to get the final outputs.

Vectorizing the volume next layer

for each sample in the
batch
…

k2 x M’’ x N’’ Cx1

(M’’ N’’ k2) x 1

Fully Connected Layer
• After multiple convolution layers, we can have one or more fully connected layers
to get the final outputs.

Vectorizing the volume dout x b

for each sample in the
batch
…
x= Wx = Y
…
…
…

…
b x k2 x M’’ x N’’ (M’’ N’’ k2) x b dout x (M’’ N’’ k2)
CNN for Image Classification
• Here’s the model’s architecture for predicting the class label of any image:
Recurrent Neural Networks (RNN)
• Sometimes you want your model to have a notion of memory (e.g., in case of time-
series data).
• RNNs incorporate hidden state that get updated as we move forward in time.
• Hidden states change with time ht . And for each time t, ht depends on the state at t–1.

Same network
output y1 y2 y3 with the same
parameters

h0 h1 h2
RNN ≡ RNN RNN RNN …

input x1 x2 x3
Recurrent Neural Networks (RNN)
output

• Each hidden state is a functions of the previous hidden

state and the current input.
ht = f𝜃(ht–1, xt) RNN

• And the output is a function of the current state:

yt = g𝜃(ht) = W . ht input
• This could be used for different scenarios:

many-to-one one-to-many many-to-many

Generative Adversarial Networks (GAN)

Figure: [Link]

• Once the whole network is trained (including generator and discriminator), we

only use the generator to generate new samples.