Deep Learning

Deep Learning

Mukesh Saraswat
saraswatmukesh@[Link]

Department of Computer Science & Engineering and Information Technology

Jaypee Institute of Information Technology, Noida

July 16, 2020

Deep Learning

Outline

Introduction to Deep Learning

Deep Learning Success
Why Deep Learning and why now?
Fundamentals of Deep Learning

Convolutional Neural Networks (CNN)

Deep Learning for Computer Vision
Case Studies of CNN
Deep Learning
Introduction to Deep Learning
Deep Learning Success

Outline

Introduction to Deep Learning

Deep Learning Success
Why Deep Learning and why now?
Fundamentals of Deep Learning

Convolutional Neural Networks (CNN)

Deep Learning for Computer Vision
Case Studies of CNN
Deep Learning
Introduction to Deep Learning
Deep Learning Success

Deep Learning
Deep Learning
Introduction to Deep Learning
Deep Learning Success

Deep Learning Success

I Image Classification
I Machine Translation
I Speech Recognition
I Game Playing
. . . and many, many more
Deep Learning
Introduction to Deep Learning
Deep Learning Success

Deep Learning Success

I Image Classification
I Machine Translation
I Speech Recognition
I Game Playing
. . . and many, many more
Deep Learning
Introduction to Deep Learning
Deep Learning Success

Deep Learning Success

I Image Classification
I Machine Translation
I Speech Recognition
I Game Playing
. . . and many, many more
Deep Learning
Introduction to Deep Learning
Deep Learning Success

Deep Learning Success

I Image Classification
I Machine Translation
I Speech Recognition
I Game Playing
. . . and many, many more
Deep Learning
Introduction to Deep Learning
Deep Learning Success

Deep Learning Success

I Image Classification
I Machine Translation
I Speech Recognition
I Game Playing
. . . and many, many more
Deep Learning
Introduction to Deep Learning
Why Deep Learning and why now?

Outline

Introduction to Deep Learning

Deep Learning Success
Why Deep Learning and why now?
Fundamentals of Deep Learning

Convolutional Neural Networks (CNN)

Deep Learning for Computer Vision
Case Studies of CNN
Deep Learning
Introduction to Deep Learning
Why Deep Learning and why now?

Why Deep Learning?

I Hand-Engineered Features vs. Learned features

Deep Learning
Introduction to Deep Learning
Why Deep Learning and why now?

Why Now?
I Large Datasets
I GPU Hardware Advances + Price Decreases
I Improved Techniques
Deep Learning
Introduction to Deep Learning
Why Deep Learning and why now?

The Big Players

Deep Learning
Introduction to Deep Learning
Why Deep Learning and why now?

Startups
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Outline

Introduction to Deep Learning

Deep Learning Success
Why Deep Learning and why now?
Fundamentals of Deep Learning

Convolutional Neural Networks (CNN)

Deep Learning for Computer Vision
Case Studies of CNN
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

The Perceptron
I Invented in 1950s by Frank Rosenblatt
I Inspired by neurobiology
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

The Perceptron
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Perceptron Forward Pass

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Perceptron Forward Pass

output = g (XW + b)
X = x0 , x1 , . . . , xn
W = w0 , w1 , . . . , wn
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Sigmoid Activation

output = g (XW + b)
1
g (a) = σ(a) =
1 + e −a
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Common Activation Functions

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Importance of Activation Functions

I Activation functions add non-linearity to our network’s
function
I Most real-world problems + data are non-linear
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Perceptron Forward Pass

output = g (XW + b)
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Perceptron Forward Pass

output = g (
(2 ∗ 0.1)+
(3 ∗ 0.5)+
(−1 ∗ 2.5)+
(5 ∗ 0.2)+
(1 ∗ 3.0)
)
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Perceptron Forward Pass

output = g (3.2) = σ(3.2)

1
= = 0.96
(1 + e −3.2 )
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

How do we build neural networks with

perceptrons?
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Perceptron Diagram Simplified

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Multi-Output Perceptron
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Multi-Layer Perceptron (MLP)

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Multi-Layer Perceptron (MLP)

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Deep Neural Network

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Applying Neural Networks

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Example Problem: Will my Flight be Delayed?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Example Problem: Will my Flight be Delayed?

Temperature: -20 F

Wind Speed: 45 mph

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Example Problem: Will my Flight be Delayed?

[-20, 45]
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Example Problem: Will my Flight be Delayed?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Example Problem: Will my Flight be Delayed?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Example Problem: Will my Flight be Delayed?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Quantifying Loss

loss(f (x (i); θ), y (i))

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Total Loss

N
1 X
Total loss := J(θ) = loss(f (x (i) ; θ), y (i) )
N i
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

N
1 X
argθ min loss(f (x (i) ; θ), y (i) )
N
i
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

I Loss is a function of the model’s parameters

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

I How to minimize loss?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

I How to minimize loss?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

I How to minimize loss?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

I How to minimize loss?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

I How to minimize loss?

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks: Objective

I This is called Stochastic Gradient Descent (SGD)

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Stochastic Gradient Descent (SGD)

I Initialize θ randomly
I For N Epochs
I For each training example (x, y ):
I Compute Loss Gradient:

∂J(θ)
∂θ
I Update θ with update rule:

∂J(θ)
θ := θ − η
∂θ
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks In Practice

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Loss function can be difficult to optimize

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Loss function can be difficult to optimize

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Learning Rate & Optimization

I Small Learning Rate
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Learning Rate & Optimization

I Large Learning Rate
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

How to deal with this?

I Try lots of different learning rates to see what is ‘just right’

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

How to deal with this?

I Try lots of different learning rates to see what is ‘just right’

I Do something smarter
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

How to deal with this?

I Try lots of different learning rates to see what is ‘just right’

I Do something smarter : Adaptive Learning Rate
I ADAM
I Momentum
I NAG
I Adagrad
I Adadelta
I RMSProp
I For details: check out
[Link]
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Training Neural Networks In Practice 2: Fighting

Overfitting
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

The Problem of Overfitting

Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Popular Regularization Techniques

I Dropout
I Early Stopping
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Regularization I: Dropout
I During training, randomly set some activations to 0
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Regularization I: Dropout
I During training, randomly set some activations to 0
I Typically ‘drop’ 50% of activations in layer
I Forces network to not rely on any 1 node
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Regularization I: Dropout
I During training, randomly set some activations to 0
I Typically ‘drop’ 50% of activations in layer
I Forces network to not rely on any 1 node
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Regularization II: Early Stopping

I Don’t give the network time to overfit

I ...
I Epoch 15: Train: 85% Validation: 80%
I Epoch 16: Train: 87% Validation: 82%
I Epoch 17: Train: 90% Validation: 85%
I Epoch 18: Train: 95% Validation: 83%
I Epoch 19: Train: 97% Validation: 78%
I Epoch 20: Train: 98% Validation: 75%
Deep Learning
Introduction to Deep Learning
Fundamentals of Deep Learning

Regularization II: Early Stopping

I Don’t give the network time to overfit

I ...
I Epoch 15: Train: 85% Validation: 80%
I Epoch 16: Train: 87% Validation: 82%
I Epoch 17: Train: 90% Validation: 85% <—-Stop Here
I Epoch 18: Train: 95% Validation: 83%
I Epoch 19: Train: 97% Validation: 78%
I Epoch 20: Train: 98% Validation: 75%
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Outline

Introduction to Deep Learning

Deep Learning Success
Why Deep Learning and why now?
Fundamentals of Deep Learning

Convolutional Neural Networks (CNN)

Deep Learning for Computer Vision
Case Studies of CNN
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Computer Vision is Machine Learning

Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Images are Numbers

Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Computer Vision is Hard

Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Famous Computer Vision Datasets

Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Classify an Image of a Number

Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Neural Networks (CNN)

I The last few years, deep convolutional networks have

outperformed the state-of-the-art in many visual recognition
tasks
I Image Classification
I Classification and Localization
I Object Detection
I Semantic Segmentation
I Instance Segmentation
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Image Classification

I Given an image, a discrete label is required as an output,

resembling the main object in the image.
I Assumption: Only one object in the image and not multiple.
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Classification with Localization

I Along with discrete labelling, localization is required as an
output.
I Localization: The exact location of the object in the image,
using the concept of bounding box.
I Assumption: Only one object in the image and not multiple.
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Object Detection
I The task is to classify and localize all the objects in the
image.
I Localization: The exact location of the object in the image,
represented by a bounding box.
I No assumption. An image may consists of multiple objects.
Semantic Segmentation
I Label each pixel of an image with a corresponding class of
what is being represented
I Does not differentiate between the instances of a class. Only
care about pixels.
I A pixel level image classification task where the output is a
high resolution image, possibly of same size as input image.
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Instance Segmentation

I Along with pixel level classification, we expect the computer

to classify each instance of a class separately.
I Semantic segmentation does not differentiate between the
instances of a particular class.
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Neural Networks (CNN)

I Learn a complex representation of visual data using vast
amounts of data
I Inspired by the human visual system
I Learn multiple layers of transformations, which are applied on
top of each other to extract a progressively more sophisticated
representation of the input
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Neural Networks: Layers

I INPUT: [32×32×3] will hold the raw pixel values of the image, in this case an image of width 32, height
32, and with three color channels R, G, B
I CONV Layer: will compute the output of neurons that are connected to local regions in the input, each
computing a dot product between their weights and a small region they are connected to in the input
volume. This may result in volume such as [32×32×12] if we decided to use 12 filters
I RELU Layer: will apply an elementwise activation function, such as the max(0, x) thresholding at zero.
This leaves the size of the volume unchanged ([32×32×12])
I POOL Layer: will perform a downsampling operation along the spatial dimensions (width, height),
resulting in volume such as [16×16×12]
I Fully-Connected (FC) Layer: will compute the class scores, resulting in volume of size [1x1x10], where
each of the 10 numbers correspond to a class score, such as among the 10 categories of CIFAR-10. As with
ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the
numbers in the previous volume
Convolutional Layer
Convolution operation
I Convolve a 2D image with a function, termed as “kernel” or
“filter”.
I The kernel slides over the image.
I It calculates the sum of the element-wise multiplication
between the input matrix and kernel matrix

Figure: Convolution Operation

[[Link]
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolutional Layer
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolution
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Convolution:Representation Learning
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Max pooling
Max pooling operation
I Selecting the maximum pixel value from each region, defined
by a filter.
I Idea: Retain the important features from each region only.
I Adv.: Reduce the size of the feature map and results in fewer
parameters in the network.

Figure: Max-Pooling Operation [https : //www .[Link]/What − is −

max − pooling − in − convolutional − neural − networks#].
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

Effect of pooling: Down sampling

I Convolution and specially pooling operation reduce image size.

I Down sampling: Converting a high resolution image to a low
resolution image
I Adv.: Next layer convolution filters see larger context.
I Deeper layer filters able to focus on a larger receptive field
(context).
I Receptive field: Region in the input image that a particular
feature extractor (filter) looks at a particular instance.
I Conclusion: The model better understands “WHAT” is
present in the image, but it loses the information of
“WHERE” it is present.
Deep Learning
Convolutional Neural Networks (CNN)
Deep Learning for Computer Vision

CNN Architecture for Image Classification

Deep Learning
Convolutional Neural Networks (CNN)
Case Studies of CNN

Outline

Introduction to Deep Learning

Deep Learning Success
Why Deep Learning and why now?
Fundamentals of Deep Learning

Convolutional Neural Networks (CNN)

Deep Learning for Computer Vision
Case Studies of CNN
Deep Learning
Convolutional Neural Networks (CNN)
Case Studies of CNN

Object Recognition
Case Study: ImageNet
I ImageNet: dataset of 14+ million images (21,841 categories)

I Links to images not images

I Let’s take the high level category of fruit as an example:
I Total 188,000 images of fruit
I There are 1206 Granny Smith apples:
Deep Learning
Convolutional Neural Networks (CNN)
Case Studies of CNN

ImageNet

I Competition: ImageNet Large Scale Visual Recognition

Challenge (ILSVRC)
I Networks
I AlexNet (2012)
I ZFNet (2013)
I VGGNet (2014)
I GoogLeNet (2014)
I ResNet (2015)
I CUImage (2016)
Deep Learning
Convolutional Neural Networks (CNN)
Case Studies of CNN
Deep Learning
Convolutional Neural Networks (CNN)
Case Studies of CNN
Deep Learning
Convolutional Neural Networks (CNN)
Case Studies of CNN
Deep Learning
Convolutional Neural Networks (CNN)
Case Studies of CNN
Deep Learning
Convolutional Neural Networks (CNN)
Case Studies of CNN
Thank You