C H A P T E R

9
Transfer learning

OUTLINE
• Introduction to transfer learning strategies
• Using the pre-trained Inception-v3 model through
TensorFlow Hub for transfer learning
• Comparing transfer learning and training from scratch

9.1 Transfer learning

9.1.1 Introduction to transfer learning

In Chapter 4, Section 4.1.3., we explained and demonstrated that the features extracted by convolutional
layers of different depths of the network are different. In particular, the shallower layers are mainly responsible for
extracting simple features of the input such as edges and lines, while the deeper layers are based on the extracted fea-
tures from the previous layers, so they can recognize more specific features such as nose, eyes, ears, and so on, as
shown in Fig. 9.1.

FIG. 9.1 Example of a convolutional neural network (CNN).

Principles and Labs for Deep Learning 219 Copyright © 2021 Elsevier Inc. All rights reserved.
[Link]
220 9. Transfer learning

The learned feature of convolutional neural networks (CNNs) can be applied to different tasks. For example, by
using the ImageNet dataset containing tens of thousands of images to train the CNNs, a large amount of training data
allows the network models to learn very diverse features. When we have a new dataset and want to solve a task of
interest, instead of training the entire network model from scratch, we use a trained model on a large dataset such as
ImageNet, called a pre-trained model, as an initialization or fixed feature extractor for the new task. This training
method is called transfer learning [1–9].
Transfer learning is actually very similar to our human learning method; the knowledge gained by completing
a task can also be used to solve other related tasks. The more related the tasks are, the easier the knowledge is
transferred. For example, when we know how to ride a bicycle, it will be easier for us to learn to ride a motorcycle.
According to the experimental results of many public works [10–14], using the transfer learning method for training
neural network models can help to obtain better performance while requiring less training data and training time
compared with the method of training from scratch. Fig. 9.2 shows how training from scratch differs from transfer
learning. As shown, in training from scratch, the models are trained separately for specific tasks and no knowledge
is retained for transferring from one model to another. In transfer learning, learning of new tasks is based on the
learned knowledge from previous tasks.

Pre-trained model

Learning
Dataset 1 Task 1
Model 1 Dataset 1

Learning Learning
Dataset 2 Task 2 Dataset 2 Task 2
Model 2 Model 1

(A) Training from scratch (B) Transfer learning

FIG. 9.2 Diagram of training a model from scratch and training a model using transfer learning.

9.1.2 Transfer learning strategies

There are four transfer learning strategies that can be applied based on the following two characteristics of the train-
ing data.

▪ The size of training data: If a new dataset contains tens of thousands of training samples, it is considered a large
dataset. If the new dataset only consists of hundreds or thousands of training samples, it is considered a small
dataset.
▪ The similarity of data: This is the similarity between the new training data and the data used for the pre-trained
model. For example, cats and tigers are considered as similar samples, while cats and tables are considered as
different samples.

(1) The first strategy: training network models with small dataset and similar training samples
Because a huge network model is trained on a small dataset, the overfitting problem is prone to occur. Thus, when
applying transfer learning, the weights of the pre-trained model must remain unchanged during training process.
Since the new dataset has similarity to the dataset used for the pre-trained model, they have similar features in hidden
layers of the network, especially in the deeper layers. Therefore, the layers for feature extraction do not need to change.
To learn a new task, it is only required to make changes to the last layer that handles output features of the network
model. The new network model is built based on the architecture of the pre-trained model by removing the output
layer (last layer), then adding new layers for the new task. The steps for transfer learning are as follows:
▪ Step 1 (removing the network layer): Choose to remove the last layer of the pre-trained model, as shown in
Fig. 9.3.
 9.1 Transfer learning 221

FIG. 9.3 Removing the last layer of the pre-trained model (1).

▪ Step 2 (adding new layer): Add one or multiple layers to the top of the original network structure for the new task, as
shown in Fig. 9.4.

FIG. 9.4 Adding layers for the new task (1).

▪ Step 3 (training new model): During training the new network model, the weights of most network layers are fixed;
only newly added layers are trained, as shown in Fig. 9.5.

Trainable False Training New Layer

Convolution Layer Convolution Layer Convolution Layer Output

(edge detector) (shape detector) (Higher level feature) (New task)

FIG. 9.5 Freezing layers for training (1).

(2) The second strategy: training network models with small datasets and dissimilar training samples
Because of using a small dataset for training a network model, the weights of the pre-trained model should be left
unchanged to prevent the overfitting problem. Since the samples of the new dataset are different from these of the
dataset used for the pre-trained model, they have similar features only in the shallower layers, while their features in
the deeper layers of the network model are very different. To build a network model based on the architecture of the
pre-trained model for learning a new task, only the shallower layers are retained, whereas the deeper layers should
be removed, and the output layer is replaced with a new one. The steps for transfer learning are as follows:
▪ Step 1 (removing the network layer): Remove most of the deep layers of the pre-trained model and leave only some
shallow layers that extract simple features such as lines and edges, as shown in Fig. 9.6.
222 9. Transfer learning

FIG. 9.6 Removing layers of the pre-trained model.

▪ Step 2 (adding new layer): Choose to add new layers to the top of the original network architecture for the new task,
as shown in Fig. 9.7.

FIG. 9.7 Adding layers for the new task (2).

▪ Step 3 (training new model): During training the new network model, the weights of most network layers are fixed;
only newly added layers are trained, as shown in Fig. 9.8.

FIG. 9.8 Freezing layers for training (2).

(3) The third strategy: training network models with a large dataset and similar training samples
Using a large dataset allows us to train the entire pre-trained model or a new network model from scratch. The
following provides two training methods:
(a) Using pre-trained model
Because the new dataset has similarity to the dataset used for the pre-trained model, they have similar features in
hidden layers of the network, especially in the deeper layers. Therefore, the layers for feature extraction do not need to
change. To learn a new task, it is only required to make changes to the last layer that handles the output of the network
model. The new model is built based on the architecture of the pre-trained model by replacing the last layer with new
one. Finally, because the dataset is large enough, the weights of the newly added layers as well as some or all other
 9.1 Transfer learning 223
layers of the new network model can be optimized to obtain better performance. The steps for fine-tuning are as
follows:
▪ Step 1 (removing the network layer): Choose to remove the last layer of the pre-trained model, as shown in Fig. 9.9.

FIG. 9.9 Removing the last layer of the pre-trained model (2).

▪ Step 2 (adding new layers): Choose to add one or more layers to the top of the original network architecture for a new
task, as shown in Fig. 9.10.

FIG. 9.10 Adding layer for the new task (3).

▪ Step 3 (training new model):

• First method: The weights of most network layers are fixed; only newly added layers are trained, as shown in
Fig. 9.11.

FIG. 9.11 Freezing layers for training (3).

• Second method: Train or fine-tune through the entire network model on the new dataset, as show in Fig. 9.12.

FIG. 9.12 Training or fine-tuning the entire network model (1).

224 9. Transfer learning

(b) Creating a new network model

Because the dataset is large enough, it allows us to create a new network model and train this model from scratch, as
shown in Fig. 9.13.

FIG. 9.13 Training the network model from scratch (1).

(4) The fourth strategy: training network models with large dataset and dissimilar training samples
Using a larger dataset for training a network model can avoid the overfitting problem. Therefore, the entire pre-
trained network model or a new network model can be trained on the new dataset for improving performance.
The following provides two training methods.
(a) Using a pre-trained model
Due to the difference between the new dataset and the dataset used for the pre-trained model, their features are not
similar in most layers of the network model. However, in practice, it is still very often beneficial to initialize with
weights from a pre-trained model for training a new network model. The steps for the training model are as follows:
▪ Step 1 (removing the network layer): Choose to remove the last layer of the pre-trained model, as shown in Fig. 9.14.

FIG. 9.14 Removing the last layer of the pre-trained model (3).

▪ Step 2 (adding new layer): Choose to add one or more layers to the top of the original network architecture for a new
task, as shown in Fig. 9.15.

FIG. 9.15 Adding layers for the new task (4).

 9.2 Experiment: Using Inception-v3 for transfer learning 225
▪ Step 3 (training new model): Train or fine-tune the entire network model on the new larger dataset with weights
initialization from the pre-trained model, as shown in Fig. 9.16.

FIG. 9.16 Training or fine-tuning the entire network model (2).

(b) Creating a new network model

Because the dataset is large enough, it allows us to create a new network model and train this model from scratch, as
shown in Fig. 9.17.

FIG. 9.17 Training the network model from scratch (2).

9.2 Experiment: Using Inception-v3 for transfer learning

This section introduces how to use the Inception-v3 network for improving image classification performance on the
Dogs vs. Cats dataset. We conduct two training scenarios. In the first scenario, we build the Inception-v3 model, named
Model-1, through Keras Applications and train it from scratch with random weight initialization. In the second sce-
nario, we build a new network model, named Model-2, by using the Inception-v3 network loaded through TensorFlow
Hub for feature extraction and adopting two fully connected layers for classification. By applying transfer learning to
Model-2, the weights of most of the layers are fixed; only two newly added fully connected layers are trained on the
Dogs vs. Cats dataset to learn a new task. The experimental results show that Model-2 obtains greater accuracy than
Model-1 does.

9.2.1 Introduction to Dogs vs. Cats dataset

The Dogs vs. Cats dataset, which includes two classes of dog and cat, was introduced for a Kaggle machine learning
competition in 2013. The Kaggle provides 25,000 labeled images of dogs and cats for training and 12,500 unlabeled
images for testing. Fig. 9.18 shows some of the images from the Dogs vs. Cats dataset. To build an optimized input
pipeline in the experiment, the Dogs vs. Cats dataset is loaded from the TensorFlow dataset through the “[Link].
Datasets” application programming interface (API) for training and testing network models.
226 9. Transfer learning

FIG. 9.18 Images from Dogs vs. Cats dataset.

9.2.2 Code examples

Fig. 9.19 is a flowchart of the source code for images classification models.

FIG. 9.19 Flowchart of the source code for images classification models.

1. Preparing dataset

a) Import packages

import os
import numpy as np
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_datasets as tfds
from tensorflow import keras
from [Link] import layers
import [Link] as plt
from preprocessing import flip, color, rotate, zoom
 9.2 Experiment: Using Inception-v3 for transfer learning 227

b) Loading data
Load Cats_vs_Dog dataset:

# Divide data with ratio of 8:1:1 for training, validation and testing
train_split, valid_split, test_split = [Link]([80, 10, 10])
# load the training set
train_data, info = [Link]("cats_vs_dogs", split=train_split, with_info=True)
# load validation set
valid_data = [Link]("cats_vs_dogs", split=valid_split)
# load test set
test_data = [Link]("cats_vs_dogs", split=test_split)

View name of class and create decoder:

print([Link]['label'].names) # display name of class

decoder = [Link]['label'].names # create decoder
Result: ['cat', 'dog']

Display the image from dataset:

for data in train_data.take(1):

img = data['image'] # read image
label = data['label'] # read lable
# get the class
[Link](decoder[label])
# Display image
[Link](img)

Result:
228 9. Transfer learning

c) Setting data
Creating function for data augmentation:

input_shape = (299, 299) # set the input size

def parse_aug_fn(dataset):
"
Image Augmentation function
"
# Image standardization
x = [Link](dataset['image'], tf.float32) / 255.
x = [Link](x, input_shape)
# Random horizontal flip
x = flip(x)
# color conversion (50%)
x = [Link]([Link]([], 0, 1) > 0.5, lambda: color(x), lambda: x)
# Image rotation (25%)
x = [Link]([Link]([], 0, 1) > 0.75, lambda: rotate(x), lambda: x)
# image scaling (50%)
x = [Link]([Link]([], 0, 1) > 0.5, lambda: zoom(x), lambda: x)
return x, dataset['label']

def parse_fn(dataset):
# Image standardization
x = [Link](dataset['image'], tf.float32) / 255.
x = [Link](x, input_shape)
return x, dataset['label']

Setting data for training, validation, and testing:

AUTOTUNE = [Link] # Automatic adjustment mode

buffer_size = 1000 # Because the image is larger, the cache space is set to 1000.
batch_size = 64 # Batch size

# Training data
train_data = train_data.map(map_func=parse_aug_fn, num_parallel_calls=AUTOTUNE)
# shuffle training data
train_data = train_data.shuffle(buffer_size)
# Set batch size and turn on prefetch mode
train_data = train_data.batch(batch_size).prefetch(buffer_size=AUTOTUNE)

#Validation data
valid_data = valid_data.map(map_func=parse_fn, num_parallel_calls=AUTOTUNE)
# Set batch size and turn on prefetch mode
valid_data = valid_data.batch(batch_size).prefetch(buffer_size=AUTOTUNE)

# Test data
test_data = test_data.map(map_func=parse_fn, num_parallel_calls=AUTOTUNE)
# Set batch size and turn on prefetch mode
test_data = test_data.batch(batch_size).prefetch(buffer_size=AUTOTUNE)
 9.2 Experiment: Using Inception-v3 for transfer learning 229
2. Building and training network models
a) Model-1: Training from scratch
Create a storage directory for saving model:

model_dir = 'lab9-logs/models' # set storage directory path

[Link](model_dir) # Create a storage directory

Set callback function:

# Save training log

log_dir = [Link]('lab9-logs', 'model-1')
model_cbk = [Link](log_dir=log_dir)
# early stopping during training
model_esp = [Link](monitor='val_binary_accuracy',
patience=30,
mode='max')

Create Inception-v3 network:

# Loading Inception-v3 network from [Link]

base_model = [Link].InceptionV3(include_top=False, # does not include the
fully connected layer.
weights=None, # random initialization
pooling='avg',
input_shape=input_shape+(3,))
# adding two fully connected layers to top of the base Inception-v3 model,
# using the Sigmoid activation function in the last layer
model_1 = [Link]([
base_model,
[Link](128, activation='relu'),
[Link](1, activation='sigmoid')
])

View model information via “[Link]” API:

model_1.summary()
230 9. Transfer learning

Result:

Set the optimizer, loss function, and metric function:

model_1.compile([Link](),
loss=[Link](),
metrics=[[Link]()])

Training network model:

history = model_1.fit(train_data,
epochs=200,
validation_data=valid_data,
callbacks=[model_cbk, model_esp])
Result˖

b) Model-2: Transfer Learning

Set callback function:

# Save training log

log_dir = [Link]('lab9-logs', 'model-2')
model_cbk = [Link](log_dir=log_dir)
# early stopping during training
model_esp = [Link](monitor='val_binary_accuracy',
patience=30,
mode='max')
9.2 Experiment: Using Inception-v3 for transfer learning 231
232 9. Transfer learning

Set the optimizer, loss function, and metric function:

model_2.compile([Link](),
loss=[Link](),
metrics=[[Link]()])

Training network model:

history = model_2.fit(train_data,
epochs=200,
validation_data=valid_data,
callbacks=[model_cbk, model_esp])
Reuslt:

Supplementary explanation
In Chapter 8, the Inception-v3 network was loaded through TensorFlow Hub at [Link]
inception_v3/classification/4, and in this chapter, it is loaded at [Link]
feature_vector/4. The difference between the two loaded networks is that the Inception-v3 loaded in Chapter 8 contains the
last classification layer (1000 categories), whereas this classification layer is removed in the latter network.

3. Comparison of Model-1 and Model-2

Both the best-trained weights of Model-1 and Model-2 are used for evaluating the Cats_vs_Dogs test set.
# Load the best trained weights of Model-1
model_1.load_weights(model_dir + '/Best-model-1.h5')
# Load the best trained weights of Model-2
model_2.load_weights(model_dir + '/Best-model-2.h5')
# Calculate the loss value and accuracy of Model-1 and Model-2
loss_1, acc_1 = model_1.evaluate(test_data)
loss_2, acc_2 = model_2.evaluate(test_data)
print("Model_1 Prediction: {}%".format(acc_1 * 100))
print("Model_2 Prediction: {}%".format(acc_2 * 100))
Result˖Model_1 Prediction: 97.97413945198059%
Model_2 Prediction: 99.39655065536499%

The results show that Model-2 with transfer learning obtained a classification accuracy of 99.39%, which is greater
than the 1.42% accuracy of Model-1 with training from scratch.
 References 233

References
[1] J. Yosinski, J. Clune, Y. Bengio, H. Lipson, How transferable are features in deep neural networks? Adv. Neural Inf. Proces. Syst. (2014)
3320–3328.
[2] M. Oquab, L. Bottou, I. Laptev, J. Sivic, Learning and transferring mid-level image representations using convolutional neural networks,
in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, 2014, pp. 1717–1724.
[3] R. Mormont, P. Geurts, R. Maree, Comparison of deep transfer learning strategies for digital pathology, in: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition Workshops, 2018, pp. 2343–234309.
[4] S. Kornblith, J. Shlens, Q.V. Le, Do better ImageNet models transfer better? Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (2019) 2661–2671.
[5] H. Shin, et al., Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learn-
ing, IEEE Trans. Med. Imaging 35 (5) (2016) 1285–1298.
[6] N. Tajbakhsh, et al., Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans. Med. Imaging 35 (5)
(2016) 1299–1312.
[7] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, V. Lempitsky, Domain-adversarial training of neural
networks, J. Mach. Learn. Res. 17 (1) (2016) 2030–2096.
[8] D. Hendrycks, K. Lee, M. Mazeika, Using pre-training can improve model robustness and uncertainty, in: International Conference on Machine
Learning, 2019.
[9] Z. Ding, Y. Fu, Deep transfer low-rank coding for cross-domain learning, IEEE Trans. Neural Netw. Learn. Syst. 30 (6) (2019) 1768–1779.
[10] F. Zhuang, et al., A comprehensive survey on transfer learning, Proc. IEEE 109 (1) (2021) 43–76.
[11] Z. Li, D. Hoiem, Learning without forgetting, IEEE Trans. Pattern Anal. Mach. Intell. 40 (12) (2018) 2935–2947.
[12] S.-C. Huang, T.-H. Le, D.-W. Jaw, DSNet: joint semantic learning for object detection in inclement weather conditions. IEEE Trans. Pattern Anal.
Mach. Intell. (2021), [Link]
[13] T.-H. Le, S.-C. Huang, D. Jaw, Cross-resolution feature fusion for fast hand detection in intelligent homecare systems, IEEE Sens. J. 19 (12) (2019)
4696–4704.
[14] Q.-V. Hoang, T.-H. Le, S.-C. Huang, An improvement of RetinaNet for hand detection in intelligent homecare systems, in: 2020 IEEE Interna-
tional Conference on Consumer Electronics - Taiwan (ICCE-Taiwan), 2020, pp. 1–2.