Deep Learning
Chapter 3
Examples
Dr. Hossein Amirkhani
amirkhani@[Link]
[Link]
University of Qom
Spring 2018
Python review 2
Hossein Amirkhani
Python review 3
Hossein Amirkhani
Outline 4
•Classifying movie reviews: binary classification
•Classifying newswires: multiclass classification
•Predicting house prices: regression
Hossein Amirkhani
Problem and data 5
• Classify movie reviews as positive or negative
• Two-class classification (binary classification)
• We use the IMDB dataset
• 50,000 highly polarized reviews
• 25,000 training reviews and 25,000 testing reviews, each with
50% negative and 50% positive reviews
• Why use separate training and test sets?
Hossein Amirkhani
Load Data 6
• Just keep the top 10,000 most frequently occurring words in the
training data.
• The reviews have been turned into sequences of integers (each
integer stands for a word in a dictionary)
Hossein Amirkhani
Data 7
Hossein Amirkhani
Prepare Data 8
• We cannot feed list of integers into the neural network.
• One solution: One-hot encode lists to turn them into
vectors of 0s and 1s (bag-of-words model).
Hossein Amirkhani
Network architecture 9
What are shapes of W and b of each layer?
Hossein Amirkhani
Network architecture 10
• 16 is the number of hidden units (neurons) of the layer.
• A hidden unit is a dimension in the representation space of the
layer.
• This is “how much freedom you’re allowing the network to have
when learning representations.”
• Having more hidden units and layers:
• + Allows your network to learn more-complex representations,
• - Makes the network more computationally expensive
• - May lead to learning unwanted patterns (patterns that will improve performance
on the training data but not on the test data): overfitting.
Hossein Amirkhani
Network architecture 11
• We use sigmoid function for the final layer to
obtain something that can be interpreted as a
probability (in [0,1]): the probability that the
sentiment is positive.
• Here, we have only one output unit (compare
with 10 outputs and softmax function in the
MNIST example).
Hossein Amirkhani
Configuring learning process 12
Crossentropy is the default
choice when dealing with
models that output
probabilities:
binary_crossentropy
categorical_crossentropy
Configuring optimizer
Using custom losses and metrics
Hossein Amirkhani
Validation set 13
• To monitor during training the accuracy of the model on data it has never
seen before
Hossein Amirkhani
Training 14
Hossein Amirkhani
Validation 15
• Training loss decreases as we expect when running gradient descent.
• But it overfits from epoch 4: learning representations that are specific to the training data and don’t generalize to unseen data.
• Stop training after four epochs (early stopping).
Hossein Amirkhani
Validation 16
Hossein Amirkhani
Retrain 17
With state-of-the-art approaches, you should be able to get close to 95%.
Hossein Amirkhani
Using the trained network 18
Network is
confident
Network is not
confident
Hossein Amirkhani
Outline 19
•Classifying movie reviews: binary classification
•Classifying newswires: multiclass classification
•Predicting house prices: regression
Hossein Amirkhani
Problem and data 20
• Reuters dataset: a set of short newswires and their topics,
published by Reuters in 1986.
• Each data point should be classified into only one category from
46 topics.
• Some topics are more represented than others, but each topic
has at least 10 examples in the training set.
Hossein Amirkhani
Load and prepare data 21
The same as IMDB dataset
One-hot encoding of labels
Hossein Amirkhani
Network architecture 22
• A 16-dimensional space (as the previous example) may be too
limited to learn to separate 46 different classes.
• Such small layers may act as information bottlenecks,
permanently dropping relevant information.
Hossein Amirkhani
Configuring learning process 23
• use sparse_categorical_ crossentropy if you have encoded the
labels as an integer tensor instead of one-hot encoding:
Hossein Amirkhani
Training 24
Hossein Amirkhani
Validation 25
Hossein Amirkhani
Retrain 26
Hossein Amirkhani
Discussion 27
• Is 80% a good accuracy?
• With a balanced binary classification problem, the accuracy reached by a
purely random classifier would be 50%.
• With a balanced 46 class classification problem, the accuracy reached by a
purely random classifier would be 2%.
• But here, the accuracy of random system is 19%.
Hossein Amirkhani
Using the trained network 28
Hossein Amirkhani
Information bottleneck 29
• What happens when you have intermediate layers that are
significantly less than 46-dimensional?
• The network now peaks at ~71% validation accuracy, an 8%
absolute drop.
Hossein Amirkhani
First exercise 30
• Try
• Different number of words.
• Different number of hidden layers.
• Layers with more hidden units or fewer hidden units.
• The mse loss function instead of crossentropy.
• The tanh activation (an activation that was popular in the early days of neural
networks) instead of relu.
• Do these experiments on the MNIST, IMDB, and Reuters datasets.
• Send the scientific report (code snippets, plots, results, discussion) to my
email till 10:00 pm two weeks later (the day before the session). Delay is
not acceptable. Consider the internet problems, etc. beforehand J
[Link]
Hossein Amirkhani
Outline 31
•Classifying movie reviews: binary classification
•Classifying newswires: multiclass classification
•Predicting house prices: regression
Hossein Amirkhani
Problem 32
• Regression: Predicting a continuous value instead of a discrete
label, such as
• predicting the temperature tomorrow, given meteorological data
• predicting the time that a software project will take to complete,
given its specifications.
• Here, we predict the price of homes in a given Boston suburb in
the mid-1970s, given features about the suburb at the time, such
as the crime rate, accessibility to highways, average number of
rooms, and so on.
Hossein Amirkhani
Data 33
• Each feature has a different scale, some are between 0 and 1,
some between 1 and 12, others between 0 and 100, and so on.
• The targets are the median values, in thousands of dollars.
Hossein Amirkhani
Prepare Data 34
• Do feature-wise normalization so that each feature is centered
around 0 and has a unit standard deviation.
• Note that the quantities used for normalizing the test data are
computed using the training data.
Hossein Amirkhani
Network architecture 35
• The last layer has no
activation. It will be a
linear layer free to learn
to predict values in any
range.
• An MAE of 0.5 on this
problem would mean
your predictions are off
by $500 on average.
Hossein Amirkhani
K-fold cross validation 36
• Because you have so few data points, the validation set would end up
being very small. This would prevent you from reliably evaluating
your model. So, we use k-fold cross validation.
Hossein Amirkhani
K-fold cross validation 37
Hossein Amirkhani
K-fold cross validation 38
Keep a record of how well the
model does at each epoch
Hossein Amirkhani
Validation 39
Hossein Amirkhani
Validation 40
• Better visualization:
• Omit the first 10 data points, which are on a different scale.
• Replace each point with an exponential moving average of the previous points, to obtain a smooth
curve.
The validation MAE stops improving significantly after 80 epochs. Hossein Amirkhani
Smoothing 41
Hossein Amirkhani
Retrain 42
• Retrain on all of the training data, with the best hyper-parameters (such as
number of epochs), and then look at its performance on the test data.
Hossein Amirkhani
The End