Deep Learning

An overview using Multi Layer Perceptrons

What is Deep Learning
Deep Learning (DL) is field of machine learning concerned with
algorithms based on (artificial) neural networks and representation
learning

• (artificial) neural networks provide the computation backbone to construct deep

learning models

• representation learning involves (automatically) finding an appropriate

representation of data (e.g., features) in order to perform an ML/AI task
Multilayer Perceptrons (MLPs)
Multilayer Perceptrons (MLPs)
Perceptron:
The output is computed by a (sigmoid) activation function and its value is 0 and 1 based on, if the
weighted sum ∑ᵢwᵢxᵢ is less than or greater than some threshold value respectively.
Multilayer Perceptrons (MLPs)
Perceptron:
The output is computed by a (sigmoid) activation function and its value is 0 and 1 based on, if the
weighted sum ∑ᵢwᵢxᵢ is less than or greater than some threshold value respectively.

A bias term can be used to model unobservable factors

Multilayer Perceptrons (MLPs)
All the weights of one layer can be represented with matrix
Activation Functions for Deep Learning
Neural networks use non-linear activation functions, which help the
network learn complex data, compute and learn almost any function, and
provide accurate predictions. Popular activation function include:

• Sigmoid/logistic
• Tanh/Hyperbolic Tangent
• Rectified Linear Unit (ReLU)
• Leaky ReLU

• Sofmax : used for multiple classes

• More recent: ELU/GELU

Activation Functions for Deep Learning
Properties of Activation Functions
Activation Function Advantages Disadvantages

Sigmoid/logistic Smooth gradient Vanishing gradient

Output values bound Outputs not zero centered
Clear predictions Computationally expensive
Tanh/Hyperbolic Tangent Zero centered output Similar to Sigmoid
Otherwise, similar to Sigmoid

Rectified Linear Unit (ReLU) Computationally Efficient Dying ReLU problem

Leaku ReLU Prevents dying ReLU problem Results not consistent

Activation Functions for Deep Learning
Choosing an activation function

• ReLU function is a general activation function and is used in most cases these days
• If we encounter cases of dead neurons in the network the leaky ReLU function is the
best choice

• Sigmoid functions and their combinations generally work better in the case of
(binary) classifiers
• Sigmoid and tanh functions are sometimes avoided due to the vanishing gradient
problem

As a rule of thumb, one can begin with using ReLU function and then move over to
other activation functions in case ReLU doesn’t provide with optimum results
Model Training in Deep Learning
Model training involves determining the best set of weights for maximizing a
model’s accuracy for a particular task. It entails:

• Defining a suitable loss function, e.g.:

• Mean Square Error, Quadratic loss, L2 Loss:

• Cross entropy loss:

• Computing (finding) minimum of this loss function

Model Training in Deep Learning2
Backpropagation is employed to efficiently compute the gradient of the loss
function

• need to compute the partial derivatives ∂L/∂w and ∂L/∂b of the cost function L with
respect to any weight w or bias b in the network
Model Training in Deep Learning2,14
Backpropagation:
Forward pass to compute error (loss function value)
+
Backward pass to compute error gradient
Regularization for Deep Learning
Regularization is a technique which makes slight modifications to the
learning algorithm such that the model generalizes better. Regularization
techniques that are employed in deep learning include:

• Dropout
• probabilistically remove inputs during training
• Early Stopping
• monitor model performance on a validation set and stop training when performance
degrades
• Batch Normalization
• normalization step that, at each mini-batch, fixes the means and variances of each
layer's inputs
 Training data consist of many
28 by 28 pixel images of scanned
segmentation handwritten digits, and so the input
layer contains 784=28×28 neurons.

Source: [Link]
 Intuition of layers
Suppose we want to determine whether an
image shows a human face or not:

Source: [Link]
[Link]
KERAS
 If you want to learn theano:
[Link]
Keras ecture/Theano%[Link].mp4/[Link]
[Link]
cture/RNN%20training%20(v6).ecm.mp4/[Link]

Very flexible
or
Need some
effort to learn

Easy to learn and use

Interface of
TensorFlow or (still have some flexibility)
Theano You can modify it if you can write
keras TensorFlow or Theano
Keras
• François Chollet is the author of Keras.
• He currently works for Google as a deep learning engineer and
researcher.
• Keras means horn in Greek
• Documentation: [Link]
• Example:
[Link]
• Handwriting Digit Recognition

Machine “1”

28 x 28

MNIST Data: [Link]

Keras provides data sets loading function: [Link]
Keras
28x28 ……

500
……
softplus, softsign, relu, tanh,
hard_sigmoid, linear
……
500

Softmax

y1 y2
…… y10
Keras

Several alternatives: [Link]

Keras

Step 3.1: Configuration

SGD, RMSprop, Adagrad, Adadelta, Adam, Adamax, Nadam

Step 3.2: Find the optimal network parameters

Training data Labels In the following slides

(Images) (digits)
Keras
Step 3.2: Find the optimal network parameters

numpy array numpy array

28 x 28 …… 10 ……
=784

Number of training examples Number of training examples

[Link]
 We do not really minimize total loss!
Mini-batch ➢ Randomly initialize
network parameters
Mini-batch
x1 NN y1 𝑦ො 1 ➢ Pick the 1st batch
𝑙1 𝐿′ = 𝑙1 + 𝑙31 + ⋯
x31 NN y31 𝑦ො 31 Update parameters once
𝑙31 ➢ Pick the 2nd batch

……
𝐿′′ = 𝑙2 + 𝑙16 + ⋯
Update parameters once
x2 NN y2 𝑦ො 2
Mini-batch

…
𝑙2 ➢ Until all mini-batches
have been picked
x16 NN y16 𝑦ො 16
𝑙16 one epoch
……

Repeat the above process

 Batch size influences both speed and
Mini-batch performance. You have to tune it.

➢ Pick the 1st batch

x1 NN y1 𝑦ො 1
𝐿′ = 𝑙1 + 𝑙31 + ⋯
Mini-batch

𝑙1
Update parameters once
x31 NN y31 𝑦ො 31
…… 𝑙31 ➢ Pick the 2nd batch
𝐿′′ = 𝑙2 + 𝑙16 + ⋯
100 examples in a mini-batch Update parameters once

…
Batch size = 1
➢ Until all mini-batches
Stochastic gradient descent have been picked
Repeat 20 times one epoch
 Very large batch size can yield
Speed worse performance

• Smaller batch size means more updates in one epoch

• E.g. 50000 examples
• batch size = 1, 50000 updates in one epoch 166s 1 epoch

• batch size = 10, 5000 updates in one epoch 17s 10 epoch

Batch size = 1 and 10, update the same

166s amount of times in the same period.

Batch size = 10 is more stable, converge faster

GTX 980 on MNIST with 50000

17s training examples
Keras

Save and load models

[Link]

How to use the neural network (testing):

case 1:

case 2:
References
1. [Link]
2. I. GoodFellow, Y. Bengio, A. Courniville (2016): Deep Learning (Adaptive Computation and Machine Learning series)
3. [Link]
4. [Link]
5. [Link]
6. Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives”, in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8,
pp. 1798-1828, Aug. 2013
7. [Link]
8. [Link]
9. [Link]
10. [Link]
learning/#:~:text=4.,of%20the%20perceptron%20learning%20model
11. [Link]
12. [Link]
13. [Link]
14. [Link]
15. [Link]
16. [Link]
17. [Link]
18. P. Naraei, A. Abhari and A. Sadeghian, "Application of multilayer perceptron neural networks and support vector machines in classification of healthcare data," 2016 Future
Technologies Conference (FTC), San Francisco, CA, 2016, pp. 848-852
19. A. Vaswani, [Link], N. Parmar, J. Usktoreit, L. Jones, A.N. Gomez, L. Kaiser and I. Pushkin, “Attention is all you need”, NIPS 2017
20. [Link]
21. [Link]