Deep Learning

S U N I L KU M A R V U P PA L A
S U N I L .V U P PA L A @ G M A I L .C O M
W W W. L I N K E D I N .C O M / I N / S U N I LV U P PA L A /

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Agenda
Session-1:
• DL - What, Where, Why, How? • Building DL models using Keras+Tensorflow
• Why deep learning now? • Convolutional Neural Network
• Applications of DL
• Machine learning vs Deep learning
• Fundamentals of Artificial neural network
• Tensorflow playground
•Session-2:
• Feed forward networks
• Various layers in DL
• Activation Functions
• Hyper parameters in DL
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
2
What is Deep learning (DL)?
Deep learning (DL) is a class of machine learning (ML)
algorithms that*:
• use a cascade of many layers of nonlinear processing units
for feature extraction and transformation
• Each successive layer uses the output from the previous
layer as input
• learn multiple levels of representations that correspond to
different levels of abstraction

DL is inspired by the structure and function of the brain called

artificial neural networks.
*wikipedia

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
3
Where it fits in?

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
4
Where to use DL?
• Finance domain (Categorical and Numerical data) :
• Identify the fraud detection in credit card transactions

•Healthcare domain (Image data):

• Lung cancer classification of images

•Social media(Image data):

• Face recognition and tag the people

•Across the domains (Text):

• Identify the potential cases of automation from historical ticket data using
• Build a chat bot

Few more applications of DL are: Personalized recommendations, Prediction, Anomaly detection, Drug
discovery, Autonomous cars, Video analytics etc...

•But is it NEW concept?

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
5
 Brief history

Convolution Neural Networks for Handwritten Recognition Google Brain Project on 16k
1958 Perceptron 1974 Backpropagation 1998 Cores
2012

awkward silence (AI Winter)

1969 1995 2006 2012

Perceptron criticized SVM reigns Restricted AlexNet wins
Boltzmann ImageNet
Machine

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
6
Why second wave?
•More data from systems and sensors (IoT)

•More compute power : GPUs, multi-core CPUs

Important property:

Results get better with more data + bigger models +

more computation

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
7
Machine Learning vs Deep Learning

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
8
Supervised & unsupervised learning (Recap)

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
9
Artificial neural network

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
10
Tensorflow platground demo
A neural net model is composed a set of Layers.
Run multiple examples in the increasing order of
complexity
◦ Linear
◦ Comlicated circle
◦ Spiral
◦ Shallow learning
◦ Deep learning

There are many types of layers available and each

layer has many parameters. Thus we can have
infinitely many different network architectures.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
11
Deep Learning Fundamentals

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Artificial neural network
Artificial neurons are elementary units in an
artificial neural network.

The artificial neuron receives one or more inputs

(representing dendrites) and sums them to produce
an output (or activation) (representing a neuron's
axon).

Usually the sums of each node are weighted, and

the sum is passed through a non-linear function
known as an activation function.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
13
Back propagation
With the concept of gradient descent
◦ Forward propagation: Activation [input * weight
matrix]
◦ To get optimal value of each weight:
◦ Direction is opp. to the gradient
◦ Find weight with minimum error
◦ Derivative (slope of a tangent line - rate of change of a
function)
◦ Partial derivative (wrt one of the variables)
◦ Chain rule (derivatives of composite functions)
◦ calculate the error wrt each weight
◦ New weight = old weight - Derivative Rate * learning
rate

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
14
Feed forward nets
Information flow is unidirectional
• Data is presented to Input layer
• Passed on to Hidden Layer
• Passed on to Output layer
 Information is distributed
 Information processing is parallel
 Backpropagation

• Requires training set (input / output pairs)

• Starts with small random weights
• Error is used to adjust weights (supervised learning)

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
15
Solving XOR with a neural net

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
16
Basic set of Layers
Dense Layer
Dropout Layer
Convolution1D
Convolution2D
MaxPooling1D
LSTM

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
17
Dense and Dropout layers
Dense Layer:
It creates a regular fully connected Neural net layer
Dense (output_dim , activation='linear')
◦ output_dim: (integer > 0 ) Specifies the size of the Layer ( Number of Neurons)
◦ activation: name of activation function
Dropout Layer:
Dropout: A Simple Way to Prevent Neural Networks from Over- fitting
Dropout ( p )
Applies Dropout to the input. Dropout consists in randomly setting a fraction p of the input units
to 0 at each update during the training phase, which helps prevent over-fitting.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
18
Convolution1D/2D
Convolution operator for filtering neighborhoods of one-dimensional inputs.
1D convolution layer (e.g. temporal convolution)
2D convolution layer (e.g. spatial convolution over images)
Convolution1D ( nb_filter, filter_length, activation='linear', border_mode='valid',
subsample_length=1 )
◦ nb_filter: Number of convolution kernels to use (dimensionality of the output).
◦ filter_length: The extension (spatial or temporal) of each filter.
◦ activation: name of activation function to use
◦ border_mode: 'valid' or 'same'.
◦ subsample_length: factor by which to subsample output.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
19
MaxPooling1D
Max pooling operation for temporal data.
The max-pooling layer would reduce the input Matrix into a down sampled size with max value for each block.
Please refer to the image shown below for an example.

MaxPooling1D (pool_length=2, stride=None, border_mode='valid' )

pool_length: size of the region to which max pooling is applied
stride: integer, or None. factor by which to downscale. 2 will halve the input. If None, it will default to
pool_length.
border_mode: 'valid' or 'same'

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
20
Activation Functions
In Neural Networks, the activation function of a node defines the output of that node given an
input or set of inputs.

A standard computer chip circuit can be seen as a digital network of activation functions that
can be "ON" (1) or "OFF" (0), depending on input.
This is similar to the behavior of the linear perceptron in neural networks.
It is the nonlinear activation function that allows such networks to compute nontrivial problems
using only a small number of nodes.
ReLu: max(0,x)

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
21
Properties of activation function
◦ Nonlinear:
◦ When the activation function is non-linear, then a two-layer neural network can be proven to be a universal function approximator.
◦ Continuously differentiable:
◦ This property is necessary for enabling gradient-based optimization methods.
◦ Range:
◦ When the range of the activation function is finite, gradient-based training methods tend to be more stable.
◦ Smaller learning rates are typically necessary.
◦ Monotonic:
◦ When the activation function is monotonic, the error surface associated with a single-layer model is guaranteed to be convex.
◦ Smooth
◦ Functions with a Monotonic derivative have been shown to generalize better in some cases.
Approximates identity near the origin:
◦ The neural network will learn efficiently when its weights are initialized with small random values.
◦ When the activation function does not approximate identity near the origin, special care must be used when initializing the
weights.
◦ [Link]

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
22
 Activation functions
2
20

Logistic
2
18 1.5

16

14
1.5

1
Hyperbolic tangent 1

1
0.5
12
0.5

y=
0
10

exp( x) − exp(− x)
8 0 -0.5

y= 1 + exp(− x)
6 -0.5 -1

exp( x) + exp(− x)
-1 -1.5
2
-1.5 -2
0 -10 -8 -6 -4 -2 0 2 4 6 8 10
0 2 4 6 8 10 12 14 16 18 20
-2
-10 -8 -6 -4 -2 0 2 4 6 8 10

Linear
y=x
Rectifier / ramp function
f(x) = max(0,x)
x is the input to a neuron.
smooth approximation to the rectifier
is softplus
f(x) = ln(1+ex)

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
23
 Activation functions
Unit step (threshold):
The transfer function translates the input signals to
output signals. Four types of transfer functions are
commonly used, Unit step (threshold), sigmoid,
piecewise linear, and Gaussian.

Sigmoid:
The sigmoid function consists of 2 functions, logistic
and tangential. The values of logistic function range
from 0 and 1 and -1 to +1 for tangential function.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
24
Deep Learning Algorithms
MLP – Multi Layer perceptron
◦ A multilayer perceptron (MLP) is a feed forward artificial neural network model that maps sets of input
data onto a set of appropriate outputs.
◦ An MLP consists of multiple layers of nodes in a directed graph, with each layer fully connected to the
next one.
◦ Had multiple hidden layers with logistic regression classifier transformation

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
25
Parameters to vary for tuning
•Number of layers •Batch size
•Number of neurons in each layer •Momentum parameter (weightage given to
earlier steps taken in the process of gradient
•Activation function in each layer
descent)
•Number of epochs
•Kernels
•Error/loss functions
•Number of features
•Iteration (equivalent to when a weight update is
done) •Number of filters for images

•Learning rate (α) •Filter sizes for images

• Size of the step in the direction of the •Gradient descent methods
negative gradient

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
26
 Recap of evaluation measures
Accuracy: Overall, how often is the classifier correct?
(TP+TN)/total = (100+50)/165 = 0.91

Misclassification Rate: Overall, how often is it wrong?

(FP+FN)/total = (10+5)/165 = 0.09
equivalent to 1 minus Accuracy also known as "Error Rate"

False Positive Rate: When it's actually no, how often does it predict
yes?
Precision: When it predicts yes, how often FP/actual no = 10/60 = 0.17
is it correct?
TP/predicted yes = 100/110 = 0.91 Specificity: When it's actually no, how often does it predict no?
TN/actual no = 50/60 = 0.83
"Sensitivity" or "Recall": When it's actually equivalent to 1 minus False Positive Rate
yes, how often does it predict yes?
TP/actual yes = 100/105 = 0.95
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
27
Keras and Demos
Why we need Keras?
◦ Keras: Deep Learning library for Theano and
TensorFlow
◦ An API spec for building DL models across many
platforms

Guiding principles: modularity, minimalism,

extensibility, and Python-nativeness Other alternate frameworks:
Simple • Caffe
• Tensorflow
Keras’ community is growing, while Theano’s is
• Torch/PyTorch
declining
Less flexible
Less projects available online than caffe
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
28
Keras+Tensorflow Demos

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
29
Convolutional Neural Networks

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Convolutional Neural Network
CNN - Convolutional Neural Network
◦ Feed-forward artificial neural network
◦ Convolutional networks were inspired by
biological processes

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
31
 Convolution operation

• Non-linearity is needed to learn complex (non-linear) representations of data, otherwise the NN

would be just a linear function

• Most deep networks use ReLU - max(0,x), since it trains much faster, is more expressive than logistic
function and prevents the gradient vanishing problem.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
32
Convolution operation
Raw Image Pixel Filter or Kernel or Feature detector

Convolved Feature or
Activation Map or the
Feature Map.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
33
Convolutional Neural Network
An image input constitutes a 3-dimensional structure called the Input Volume (255x255x3).

CNN’s use filters as kernels where the parameters or weights have to be learnt.A filter is a matrix of lower
size than the input to it.

The inputs are convolved with the filters and passed through the activation function.

The weights of the kernels are randomly initialized and are modified during training based on error-
minimization using backpropagation.

The real values of the kernel matrix change with each learning iteration over the training set, indicating that
the network is learning to identify which regions are of significance for extracting features from the data.

Stride: The shift of filter after each [Link] can be increased from 1 to a larger value to decrease
overfitting.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
34
ReLu and Max pooling

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
35
CNN in summary

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
36
RNN and LSTM

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Recurrent Networks
Feed forward networks:
◦ Information only flows one way
◦ One input pattern produces one output
◦ No sense of time (or memory of previous state)
Recurrency
◦ Nodes connect back to other nodes or themselves
◦ Information flow is multidirectional
◦ Sense of time and memory of previous state(s)

Possible applications of RNN’s are in domains where data is sequential.

For example:
Speech and Text (NLP)
Music
Protein and DNA sequences
Time series from trade data
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
38
RNN and LSTM

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
39
LSTM (Long Short Term Memory)
It create a layer of Long-Short Term Memory units .
LSTM (output_dim , activation='tanh', inner_activation='hard_sigmoid')
output_dim: dimension of the internal projections and the final output.
activation: name of activation function to use
Inner_activation: name of activation function to use for inner cells

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
40
Long short-term memory
LSTM - Long short-term memory
◦ Recurrent neural network (RNN)
◦ Take input not just the current input
example they see, but also what they
perceived one step back in time.
Feedback loop, ingesting their own
outputs moment after moment as input
◦ an LSTM network is well-suited to learn
from experience to classify, process and
predict time series
◦ LSTM blocks contain three or four "gates" that
they use to control the flow of information into
or out of their memory.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
41
Text analysis

[Link]
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Type of chatbots
Usecases:
• Pizza Hut to help you order a pizza
• Uber to book a taxi
• CNN to keep you up-to-date with news
content

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
43
Conversational interfaces

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
44
Pretrained models

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
45
Convnets and pretrained models
• LeNet (1990s)

• AlexNet (2012)

• ZF Net (2013)

• GoogLeNet (2014)

• VGGNet (2014)

• ResNets (2015)

• DenseNet (August 2016)

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
46
Architecture of Alex Krizhevsky et al.

● 8 layers total

● Trained on Imagenet Dataset (1000

categories, 1.2M training images,
150k test images)

● 16.4% top-5 error

○ Winner of the ILSVRC- 2012
challenge.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Impact on Computer Vision
ImageNet Challenge2012

1.2M images with 1000 object categories

• AlexNet of uni Toronto: 15% error rate vs 26% for

2th placed (traditional CV)

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Practical tips

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
DL Project ideas
Image Speech
◦ Captioning ◦ Alexa / Home APIs
◦ extract embedded text ◦ Local languages
◦ emoji - extract sentiment
Numerical / Categorical
◦ Volume, result prediction
Text
◦ Time series forecasting - weather / server
◦ Sarcasm
◦ Satillite data analysis - ISRO
◦ Chatbots - specific topic
◦ Govt data analysis
◦ Sentiment analysis
◦ [Link]
◦ Local languages

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
50
DL project ideas contd
Video Robot
◦ analytics - speed control ◦ Path planning / recommendations
◦ search ◦ reinforcement learning

◦ annotated data

Multimodal Recommender systems

◦ Chatbots ◦ Specific product / item category
◦ Get information from multiple sources
◦ Generative

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
51
Practical challenges and tips while using
ML / DL
◦ Data availability
◦ Rules based + ML

◦ Application revisited: Gien ticket data of client, identify the potential candidates for
automation
◦ Used ML for clustering the ticket data with preprocessing
◦ Tried ML algorithms including ensemble to reach
◦ Extended DL algorithms but the improvement is not more than 5%

◦ Discussion

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
52
Cheat sheets

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
53
Learning path
Building knowledge : Practise with assignments and projects:
K1. Refresh your fundamentals on statistics, P1. Kaggle challenges with available data
probability and linear algebra Cuisine prediction, lung cancer,
K2. Do Course era deep learning Choose text or image or
categorical/nurmerical probems
K3. Refer specific concepts in
[Link] by Goodfellow P2. Participate in hackathons and assesments
K4. Refer advanced topics of deep learning P3. Github profile and upload your codes
based on the need - Generative adversial P4. Define your problem with your domain
networks, Auto encoders, deep reinforcement experience and follow steps of data science
learning, visualization techniques project execution with github repositories
K5. Attend webinars and AV meets or
conferences to network and see latest trends

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
54
Learning path references
K1. Relearning K4. Advanced topics

[Link] [Link]
ud827 relmfu

[Link]
P1. Competitions
K2. Online courses:
[Link]
[Link]
[Link]
[Link]
P2. Assessement of your skills
K3: Books:
[Link]
[Link]
P3. Github profile
[Link]
[Link]

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
55
 References
• [Link]
• [Link]
• [Link]
• [Link]
• [Link]
• [Link]
• [Link]
• [Link]
• [Link]
• [Link]
• [Link]/blog
• [Link]
• [Link]
lZjM4NTRiOWY
• Machine learning, Deep learning courses in CourseEra by Andrew NG
• [Link]

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
56
Thank you

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.