Transfer learning
Keras & TensorFlow
Haythem Ghazouani
Transfer learning
Transfer learning (TL) is a research problem in ML that focuses on
storing knowledge gained while solving one problem and applying it
to a different but related problem. For example, knowledge gained
while learning to recognize Cats could apply when trying to
recognize Tigers.
Haythem Ghazouani 2
Building Complex Models - Functional API
• The Keras functional API is a way to create
models that is more flexible than the
[Link] API.
• The functional API can handle models with
non-linear topology, models with shared
layers, and models with multiple inputs or
outputs.
• This architecture makes it possible for the
neural network to learn both deep patterns
(using the deep path) and simple rules
(through the short path).
Haythem Ghazouani 3
Building Complex Models - Functional API
input_ = [Link](shape=X_train.shape[1:])
hidden1 = [Link](30, activation="relu")(input_)
hidden2 = [Link](30, activation="relu")(hidden1)
concat = [Link]()([input_, hidden2])
output = [Link](1)(concat)
model = [Link](inputs=[input_], outputs=[output])
Once you have built the Keras model, everything is exactly like earlier, so
there’s no need to repeat it here: you must compile the model, train it, evaluate
it, and use it to make predictions.
Haythem Ghazouani 4
Building Complex Models - Functional API
What if you want to send a subset of the
features through the wide path and a different
subset (possibly overlapping) through the
deep path?!
• In this case, one solution is to use
multiple inputs.
Haythem Ghazouani 5
Building Complex Models - Functional API
For example, suppose we want to send five features through the wide path
(features 0 to 4), and six features through the deep path (features 2 to 7) :
input_A = [Link](shape=[5], name="wide_input")
input_B = [Link](shape=[6], name="deep_input")
hidden1 = [Link](30, activation="relu")(input_B)
hidden2 = [Link](30, activation="relu")(hidden1)
concat = [Link]([input_A, hidden2])
output = [Link](1, name="output")(concat)
model = [Link](inputs=[input_A, input_B], outputs=[output])
• You should name at least the most important layers.
Haythem Ghazouani 6
Building Complex Models - Functional API
We can compile the model as usual, but when we call the fit()
method, instead of passing a single input matrix X_train, we must
pass a pair of matrices (X_train_A, X_train_B): one per input.
The same is true for X_valid, and also for X_test and X_new when
you call evaluate() or predict():
history = [Link]((X_train_A, X_train_B), y_train, epochs=20,
validation_data = ((X_valid_A, X_valid_B), y_valid))
model_evaluate = [Link]((X_test_A, X_test_B), y_test)
y_pred = [Link]((X_new_A, X_new_B))
Haythem Ghazouani 7
Building Complex Models - Functional API
Another use case is as a regularization technique (i.e., a training
constraint whose objective is to reduce overfitting and thus improve
the model’s ability to generalize).
For example, you may want to add some
auxiliary outputs in a neural network
architecture to ensure that the underlying
part of the network learns something useful
on its own, without relying on the rest of the
network.
Haythem Ghazouani 8
Building Complex Models - Functional API
[...] # Same as before, up to the main output layer
output = [Link](1, name="main_output")(concat)
aux_output = [Link](1, name="aux_output")(hidden2)
model = [Link](inputs=[input_A, input_B], outputs=[output,
aux_output])
• Each output will need its own loss function. Therefore, when we compile the
model, we should pass a list of losses.
[Link](loss=["mse", "mse"], loss_weights=[0.9, 0.1], optimizer="sgd")
Haythem Ghazouani 9
Building Complex Models - Functional API
• We care much more about the main output than about the auxiliary output,
so we want to give the main output’s loss a much greater weight.
• we need to provide labels for each output (In this example, we used the
same labels).
history = [Link]([X_train_A, X_train_B], [y_train, y_train], epochs=20,
validation_data=([X_valid_A, X_valid_B], [y_valid, y_valid]))
• When we evaluate the model, Keras will return the total loss, as well as all
the individual losses:
total_loss, main_loss, aux_loss = [Link]([X_test_A, X_test_B],
[y_test, y_test])
y_pred_main, y_pred_aux = [Link]([X_new_A, X_new_B])
Haythem Ghazouani 10
Reusing Pretrained Layers
• It is generally not a good idea to train a very large DNN from scratch:
instead, you should always try to find an existing neural network that
accomplishes a similar task to the one you are trying to tackle then reuse
the lower layers of this network.
• This technique is called transfer learning.
• It will not only speed up training considerably but also require significantly
less training data.
• The output layer of the original model should usually be replaced
because it is most likely not useful at all for the new task, and it may not
even have the right number of outputs for the new task.
Haythem Ghazouani 11
Reusing Pretrained Layers
• Transfer learning will work best when the inputs have similar low-level features.
Haythem Ghazouani 12
Reusing Pretrained Layers
• Try freezing all the reused layers first (i.e., make their weights non-trainable
so that Gradient Descent won’t modify them), then train your model and see
how it performs. Then try unfreezing one or two of the top hidden layers to
let backpropagation tweak them and see if performance improves.
model_A = [Link].load_model("my_model_A.h5")
model_B_on_A = [Link](model_A.layers[:-1])
model_B_on_A.add([Link](1, activation="sigmoid"))
# If you want to avoid affecting model_A
model_A_clone = [Link].clone_model(model_A)
model_A_clone.set_weights(model_A.get_weights())
Haythem Ghazouani 13
Reusing Pretrained Layers
• The new output layer was initialized randomly it will make large errors.
• Freeze the reused layers during the first few epochs, giving the new layer
some time to learn reasonable weights.
for layer in model_B_on_A.layers[:-1]:
[Link] = False
model_B_on_A.compile( ... )
history = model_B_on_A.fit( ..., epochs = 5, ... )
for layer in model_B_on_A.layers[:-1]:
[Link] = True
model_B_on_A.compile( ... )
history = model_B_on_A.fit( ... )
model_B_on_A.evaluate( ... )
Haythem Ghazouani 14
Reusing Pretrained Layers
Pretraining on an Auxiliary Task
If you do not have much labeled training data, one last option is
to train a first neural network on an auxiliary task for which you
can easily obtain or generate labeled training data, then reuse the
lower layers of that network for your actual task. The first neural
network’s lower layers will learn feature detectors that will likely be
reusable by the second neural network.
Haythem Ghazouani 15
CNN Variations
Over the years, variants of the CNN architecture have been developed,
leading to amazing advances in the field. A good measure of this progress is
the error rate in competitions such as the ILSVRC ImageNet challenge.
1. LeNet-5 (1998)
2. AlexNet (2012)
3. VGG-16 (2014)
4. Inception-v1
5. Inception-v3
6. ResNet-50
7. Xception (2016)
8. Inception-v4 (2016)
9. Inception-ResNets
10. ResNeXt-50 (2017) [Link]
Haythem Ghazouani 16
CNN Variations - Legend
Haythem Ghazouani 17
CNN Variations - LeNet-5
• It was created by Yann LeCun in 1998 and has been widely used for
handwritten digit recognition (MNIST).
• This architecture has become the standard “template”: stacking
convolutions and pooling layers and ending the network with one or more
fully-connected layers.
Haythem Ghazouani 18
CNN Variations - AlexNet
• The AlexNet CNN architecture won the 2012 ImageNet ILSVRC challenge.
• To reduce overfitting, the authors used two regularization techniques. First,
they applied dropout with a 50% dropout rate during training to the outputs
of layers F1 and F2. Second, they performed data augmentation.
Haythem Ghazouani 19
CNN Variations - VGG
• The runner-up in the ILSVRC 2014 challenge was VGG, developed by
Karen Simonyan and Andrew Zisserman from the Visual Geometry Group
(VGG) research lab at Oxford University.
• VGG-16 architecture and VGG-19 architecture
Haythem Ghazouani 20
CNN Variations - GoogLeNet (Inception)
Haythem Ghazouani 21
CNN Variations - GoogLeNet (Inception)
• Inception modules allow GoogLeNet to use parameters much more
efficiently than previous architectures.
• This 22-layer architecture with 5M parameters is called the Inception-v1.
• Having parallel towers of convolutions with different filters, followed by
concatenation, captures different features at 1×1, 3×3 and 5×5, thereby
"clustering" them.
• 1×1 convolutions are used for dimensionality reduction to remove
computational bottlenecks.
• Due to the activation function from 1×1 convolution, its addition also adds
nonlinearity.
Haythem Ghazouani 22
CNN Variations - GoogLeNet (Inception V3)
Haythem Ghazouani 23
CNN Variations - ResNet-50
• The basic building block for ResNets are the conv and identity blocks.
• It uses skip connections (also called shortcut connections).
• If you add many skip connections, the network can start making progress
even if several layers have not started learning yet.
Haythem Ghazouani 24
CNN Variations - Xception (2016)
Haythem Ghazouani 25
CNN Variations - Inception-v4 (2016)
Haythem Ghazouani 26
CNN Variations - Inception-ResNet-V2 (2016)
Haythem Ghazouani 27
CNN Variations - ResNeXt-50 (2017)
Haythem Ghazouani 28
Using Pretrained Models from Keras
In general, you won’t have to implement standard models like
GoogLeNet or ResNet manually, since pretrained networks are
readily available with a single line of code in the
[Link] package.
For example, you can load the ResNet-50 model, pretrained on
ImageNet, with the following line of code:
model = [Link].resnet50. ResNet50
(weights= "imagenet")
Haythem Ghazouani 29
Using Pretrained Models from Keras
Available models :
Models for image classification with weights trained on ImageNet:
1) Xception
2) VGG16
3) VGG19
4) ResNet, ResNetV2
5) InceptionV3
6) InceptionResNetV2
7) MobileNet
8) MobileNetV2
9) DenseNet [Link]
10) NASNet
Haythem Ghazouani 30
Using Pretrained Models from Keras
Haythem Ghazouani 31
Using Pretrained Models from Keras
model = [Link].resnet50.ResNet50(weights="imagenet")
# you first need to ensure that the images have the right size
# ResNet-50(224 × 224)
images_resized = [Link](images, [224, 224])
# Each model provides a preprocess_input() function
inputs = [Link].resnet50.preprocess_input(images_resized * 255)
# Now we can use the pretrained model to make predictions
Y_proba = [Link](inputs)
Haythem Ghazouani 32
Using Pretrained Models from Keras
import tensorflow as tf
from tensorflow import keras
model = [Link].vgg16.VGG16(weights=None)
# The model’s summary() method displays all the model’s layers
print([Link]())
include_top: whether to include the top layers of the network or not (False,
True).
weights: one of None (random initialization) or 'imagenet' (pre-training on
ImageNet).
Haythem Ghazouani 33
Pretrained Models for Transfer Learning
• If you want to build an image classifier but you do not have
enough training data, then it is often a good idea to reuse the
lower layers of a pretrained model.
• For example Xception model, we exclude the top of the
network by setting include_top=False: this excludes the global
average pooling layer and the dense output layer. We then add
our own layers. Finally, we create the Keras Model:
Haythem Ghazouani 34
Pretrained Models for Transfer Learning
base_model = [Link](weights="imagenet",include_top=False)
avg = [Link].GlobalAveragePooling2D()(base_model.output)
output = [Link](n_classes, activation="softmax")(avg)
model = [Link](inputs=base_model.input, outputs=output)
for layer in base_model.layers:
[Link] = False
optimizer = [Link](lr=0.2, momentum=0.9, decay=0.01)
[Link](loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = [Link](train_set, epochs=5, validation_data=valid_set)
for layer in base_model.layers:
[Link] = True
optimizer = [Link](lr=0.01, momentum=0.9, decay=0.001)
[Link](...)
history = [Link](...)
Haythem Ghazouani 35