Deep Learning Fundamentals with PyTorch
Translated by
ScribdTranslationsDeep Learning Fundamentals with PyTorch
Translated by
ScribdTranslationsBadge
Deep Learning
of the course
withPytorch
Juan Pablo Morales
@juanpamf
What is Deep
Learning?
Deep Learning
Deep learning is a field of Machine
Learning that you will use neural networks
multi-layered artificial
To automatically learn features
of different levels of abstraction,
And solving difficult data problems
such as: image, text, video, audio.
History
1950-1990: Creation of
main algorithms for
training of NNs
1990:Difficulties in
train deep models
Investigation
2010: Deepwater Horizon Explosion
Learning
The boom of deep learning
Achievements
Resolution with almost human performance of
a large part of the perception tasks:
Vision: object detection, segmentation
semantics
Listening: Speech-to-text, sound playback
Language: Machine translation
○Juegos: Ajedrez, Go, LoL, SC2
Increasingly impacting more fields, such as the
physics, medicine, and finance.
2018 Turing Award
Applications: Generative DL
Aplicaciones: MusicVAE
Applications: Pix2Pix
Applications: CycleGAN
Why the Deep
Learning now?
What changed?
Labeled datasets
large size
GPUs and hardware
Improvements in the
algorithms
Open Source Software
Labeled Datasets or
labeled
Growth of open datasets
Labeled Data
Performance of deep models
GPUs
Improvements in the algorithms
Open-source software
Software for
Deep Learning
Pytorch and Tensorflow
they are the leaders
Comparison
´
´
Work cycle in
Deep Learning
The importance of understanding
how a network learns
No free lunch theorems
There is no learning algorithm that is
a silver bullet
Leaky abstractions
You cannot expect a magic API that does
all the work for us (A. Karpathy)
Hard work
Complex model (e.g. computation time)
Understanding of generalization
Visualization
Learning algorithms
Model space H
A loss function L
An optimization method
The training of the network corresponds to
find the model within H whose
the loss is minimal over the dataset.
Learning algorithms
Model prediction:
Loss function:
Total loss function:
●Problema de minimización:
Model space
parameterized
A flow with similarities to ML
traditional
Objective
Data preparation
Features
Learning algorithm
Set the parameters (internal settings)
Tuning
Hyperparameter search (external settings)
Differences: Preparation of
the data
Differences: Algorithm of
learning
Differences: Algorithm of
learning
Diferencias: Tuning
Deep learning flow
Definition of the problem
○X -> Y
Error
Data preparation
Algorithm design for learning
Tuning
Model space in
Deep Learning
Model space
Architecture:
Layers
Model space
Model space
Given an architecture, for each value
from the parameters we obtain a network
distinct neuronal.
Therefore, an architecture defines a
model space.
We seek the optimal parameters
Model space and
capacity
The larger the space of
models we say have more
capacidad
The more capacity, the more potential
easy to learn, but harder to teach
Small deep red = 1M
parameters
Convolutional Layers
●Locally connected
They share parameters
Very efficient without having a high cost!
Convolutional architecture
Classical architecture:
(Conv - ReLU - Pool)* - FC
Formula for convolutions
The number of input channels C
The size of the filter/kernel F
The number of K filters
The padding P
●El stride S
Loss functions
for Deep Learning
Loss function
Types of problems
Types of Problems Example Target
Regresión Hotel price [Link] R
Binary Classification Medical Exam {0,1}
Multiclass Classification Image recognition {0,1,2,...N}
Multiclass Classification Genre of a movie {{0,1},{2,3,5},...}
Multilabel
Regression
Last FC Sin
Layer activation
x1 x1
L = Squared error
average
(MSE)
Binary Classification
Last
Sigmoid
FC Layer
x1 y1
L = Entropy
binary crusade
(BCE)
Multiclass Classification
Last FC Softmax
Layer
x1 x1
x2 x2
L = Negative Log-
x3 x3 Likelihood
(NLL)
x4 x4
x5 x5
Classification
Multiclass Multilabel
Ultima FC Sigmoid
Layer
x1 x1
x2 x2
x3 x3 L = BCE
x4 x4
x5 x5
In summary
Other classic losses:
Hinge Loss
Specific problems that may require
specific losses:
Probability distribution: KL divergence.
SGD and
Backpropagation
Aplicaciones: MusicVAE
Heuristics
Follow the direction of the gradient
Having a random component for
avoid local minima
Adjust the step length (learning rate)
Gradient descent
SGD
The loss is a sum above all
dataset
High computing cost
We introduce randomness by choosing the
to generate a data on which to calculate the
loss
We apply a step from the descent of
gradient
SGD
Excellent theoretical properties
Mini-batch SGD
Gradient descent (complete dataset)
Quick, it can get stuck in local minima
SGD (1 data)
Converge more slowly, avoid local minima
Mini-batch SGD (k data)
Good compromise between speed y
randomness.
Backpropagation
A neural network is a computation graph
Composed of inputs, outputs, parameters, and functions
linear and activation
Its structure allows us to calculate the
gradient in a simpler form
Backpropagation algorithm
Libraries like pytorch implement from
this algorithm is automatically formed
Autograd (reverse mode automatic differentiation)
Backpropagation
The pieces of the puzzle fit together.
Evaluate a model
of Deep Learning
Recap
We designed our learning algorithm.
a. Architecture (model space)
[Link] function
c. Optimization method
We minimize the loss over the dataset
a. This is equivalent to setting the good value of the
network parameters
Does this assure us?
a good performance
about new data?
Evaluation phase: First
idea
●Randomly separate dataset into:
Training (70%)
Here the minimization is carried out.
○Test (30%)
Here the performance of the model is evaluated (rate of
error
What do we do with the hyperparameters?
If we test several models, we will choose a good value.
but our performance estimate will be biased.
Evaluation phase:
Second idea
Randomly split dataset into:
Training (60%)
Here the minimization is performed
Validation (20%)
Based on the error rate, we chose the
best value of the hyperparameters.
Test (20%)
■Aquí se evalúa la performance del modelo
Error cup
Evaluation phase:
Second idea
In deep learning
(large datasets):
Training: 98%
Validation: 1%
Test: 1%
Underfitting-Overfitting
Regularization of
Neural Networks
deep
Build a
good model of
deep learning
The recipe for building a neural network
good performance is similar to the
machine learning models:
1. Build a simple first model that achieves
learn about the data.
2. This model probably despite having
learned to be very simple and have bias
(underfitting).
3. Increase the model capacity until it overfits.
4. Regularize to find the right balance
between underfitting and overfitting.
How to control capacity
of my model
To increase capacity:
Add layers
Add number of neurons per layer
Increase the number of epochs
Choose another optimization method.
complex (adaptability, momentum, etc.)
To decrease the capacity:
Train with a larger amount of data
Use regularization techniques
L2 or weight decay
● Penalize large parameters:
Update with an additional term of
decaimiento:
Batch Norm
Key innovation in deep learning
contemporary (2016).
Similar to normalizing the inputs (good
practice), but in intermediate layers.
To calculate the mean for a batch of data and
variance.
Update data by subtracting the average and dividing
for variance plus epsilon
○Transform with:
Why batch norm is
efficient?
Smooth the loss function
(MIT, 2019):
The minimums they lead to are
plans, and it is known that they have
good generalization
Controls the internal covariate
shift (ICS)
Dropout
Algorithm
With a certain probability p
we will deactivate certain
neurons (== 0)
In each iteration, the layer
it will recalculate if 'deactivates' one
neuron.
During the training phase
dropout acts, during the phase of
evaluation of dropout layers
they have no effect.
Data augmentation
Optimization of
hyperparameters
Babysitting (panda)
●Successive iterations of the data scientist,
testing parameters and waiting for the
result
Also known as Student
Gradient Descent
It is not sufficiently systematic (e.g.:
companies)
Grid Search (traditional ML)
We define the hyperparameters that
let's optimize
We define the range they can take.
the hyperparameters
We define a number of values by
to try
We separate the space regularly,
forming a grid
Random search
Deep learning has many more
hyperparameters that traditional ML
Random search
Random search is more efficient for
deep learning
The best way to use it is
defining the range of the parameters
with a correct scale
○LR: logarithmic scale
Batch_size: powers of 2
●For computational cost we can choose
to optimize a single hyperparameter
Higher hyperparameters
importance
Learning rate
Momentum
Mini-batch size
Neurons in each layer
Number of layers
The problem
what we will resolve
Style transfer
Types of transfer
of style
●Example based vs
Collection Based
Collection Based
Paired
Unpaired
A common tool:
the GAN
GANs
The greatest innovation in
network architectures
post-2010 neural
Ian Goodfellow, MIT Top 35
under 35, Forbes top 40
under 40, etc.
Architecture:
Generative Network
Discriminative Network
Zero-sum game (losses
opposite)
GANs
Loss of GANs
●Discriminative network: cross-entropy when the network
should/reject
Generative Red: inverse of the discriminative
(zero-sum game)
The task: Winter to Summer
Winter to ...
Applications of CycleGAN
Applications of CycleGAN
Defining CycleGAN
●4 networks:
○G_x_y
○G_y_x
○D_x
○D_y
Deep
Convolutional
GANs
DCGAN Tips
CycleGAN loss
Loss of the 2 GANs
○With MSE instead of cross-entropy
Cyclical Loss
Distance from cycle to original (L1 norm)
Loss of identity
The networks leading to your domain must leave
you fix the images of the domain (L1 norm)
Why the Deep
Learning is more
more powerful than the ML
traditional?
Going deeper: More layers
There are reasons why
which have more layers
we obtain greater
performance?
Capacity and quantity
of neurons
Depth and cost in parameters
(Telgarski et al. 2016)
There are functions that can be
approximately by a network of layers
to be approximated by networks of
layers need at least
At the same budget in parameters
(optimization!) deep networks are
highly more expressive.
Exponential oscillations
1 additional layer = 2x Profundidad 1 vs Profundidad 2
oscillations
Deep models are
more difficult to optimize
Residual networks
(He et al. 2016)
Innovations make things possible
ever deeper networks
To go further
Topics
Embeddings and architectures for NLP
Recursive Networks and LSTM
Deep Reinforcement Learning
Autoencoders and Deep Learning
Bayesian
Resources
●Google Scholar, Arxiv (sanity)
Deep Learning Book
●YouTube Channel: 2 Minute Papers
Deep Learning cheatsheets
The Incredible Pytorch on GitHub
People
Fei Fei Li, Justin Johnson, Andrej
Karpathy
Andrew NG
Roger Grosse
Yoshua Bengio and Ian Goodfellow
Ali Rahimi and Ben Recht
Gabriel Peyré
And many more...
A data scientist is a
programmer
Anonymous