Transfer Learning

Mithun DJ
Agenda
• Motivation for Transfer Learning
• Understanding Transfer Learning
• Transfer Learning Strategies
• Transfer Learning for Deep Learning
• Deep Transfer Learning Strategies
• Types of Deep Transfer Learning
• Applications of Transfer Learning
Motivation
Motivation
• After supervised learning – Transfer
Learning will be the next driver of ML
commercial success
• Nuts and bolts of building AI applications using Deep
Learning’

Andrew Ng
renowned professor and data scientist, who has
been associated with Google Brain, Baidu, Stanford
and Coursera
Definition
• Transfer learning is a machine learning technique where a model
trained on one task is repurposed as the foundation for a second task.
This approach is beneficial when the second task is related to the first
or when data for the second task is limited.
Motivation
• Transfer learning can further our progress towards AGI
• History
• The Neural Information Processing Systems (NIPS) 1995 workshop Learning to Learn:
Knowledge Consolidation and Transfer in Inductive Systems.
• Learning to Learn, Knowledge Consolidation, and Inductive Transfer.
• Situation where what has been learned in one setting is exploited to
improve generalization in another setting.
• Deep Learning, Goodfellow et al
• Important:
• Most models which solve complex problems need a whole lot of data, and getting
vast amounts of labeled data for supervised models can be really difficult,
considering the time and effort it takes to label data points
 Broad Categories in ImageNet:
• Animals
• Mammals, birds, reptiles, fish, amphibians, insects, etc.
• Examples: dogs, cats, elephants, butterflies.
• Objects / Man-made artifacts
• Tools, vehicles, furniture, instruments, electronic devices, etc.
• Examples: cars, airplanes, chairs, laptops, guitars.
• Scenes / Environments
• Indoor and outdoor natural or man-made environments.
• Examples: kitchens, bedrooms, mountains, beaches.
• People & Body Parts
• Whole persons, professions, faces, hands, etc.
• Food & Everyday Items
• Fruits, vegetables, cooked dishes, packaged food.
• Examples: pizza, apples, burgers.
• Structures & Infrastructure
• Buildings, monuments, bridges, roads.
• Plants
• Flowers, trees, leaves, crops.
ImageNet characteristics
• Scale & Size
• Over 14 million images (as of 2017 release).
• Images are hand-annotated and categorized into thousands of classes.
• Classes
• Around 21,000 synsets (categories) based on the WordNet hierarchy.
• Each synset corresponds to a concept (e.g., “cat,” “car,” “apple”).
• Annotations
• Each image is labeled with the object it contains.
• Many images include bounding boxes marking the exact location of the object.
• ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
• Subset of ImageNet used for benchmarking in AI/ML.
• Contains 1.2 million training images, 50,000 validation images, and 100,000 test
images across 1,000 object categories. This challenge pushed forward the
development of deep learning and CNNs (e.g., AlexNet in 2012).
Existing Challenges
• Getting such a dataset for every domain is tough.
• Hyper specialization
• Domain Specific
• Task Specific
• Specific dataset
• Performance degradation
• Solution:
• Goes beyond specific tasks and domains
• how to leverage knowledge from pre-trained models and use it to solve new
problems
Understanding Transfer Learning
Note: Knowledge means features, weights
Key Takeaways
Questions
• What to Transfer?
• When to Transfer?
• How to Transfer?
Question1
• What to transfer
• We take the knowledge a model has already learned (like shapes, patterns, or
word meanings) and use it for a new problem.
• A model trained to recognize cats and dogs can transfer its knowledge of "edges
and curves" to help detect tumors in X-ray images.
• Transfer feature representations learned by ResNet on ImageNet to classify
medical X-rays.
Question 2
• When to transfer
• Transfer learning is useful when you have limited labeled data but a large
pre-trained model exists.
• Example: If you only have 1,000 hospital images, you can transfer knowledge
from a model trained on millions of general images.
• Negative Transfer
Question3
• How to transfer
• We transfer by reusing the old model in different ways—like freezing some
parts and training only the last layer.
• Transfer by freezing the backbone of BERT and fine-tuning only the
classification head for sentiment analysis.
• Example: Use BERT (a language model) and just train a small new layer to
detect positive vs. negative reviews.
Strategies
Strategies
Transfer Learning :
This is the basic approach where you have some labeled examples in both your original task and your new task, and
you transfer knowledge between them.
Inductive Transfer Learning :
This happens when you have labeled data in your source (original) domain and want to apply it to a target (new)
domain. It splits into two cases:
The source and target domains are the same, yet the source and target tasks are different from each other.
Case 1: You have no labeled data in your target domain → leads to Self-taught Learning
Case 2: You do have some labeled data in your target domain → can lead to Multi-task Learning
Transductive Transfer Learning:
This assumes you're working with different domains but the same type of task, leading to Domain
Adaptation.
Unsupervised Transfer Learning:
This is when you have no labeled examples in either domain - the computer has to figure things out on its
own.
Sample Selection Bias/Covariate Shift :
This deals with situations where your training data and real-world data don't match perfectly.
Summary
What to Transfer
• Instance transfer
• Feature-representation transfer
• Parameter transfer
• Relational-knowledge transfer
 Transfer Types
Instance Transfer
• Simple meaning: Reusing some data from an old problem to help with a new problem.
• Example: Using part of an English text dataset to improve performance in classifying Indian English tweets.
Feature-Representation Transfer
• Simple meaning: Using useful features learned earlier (like shapes, edges, patterns) to solve a new problem.
• Example: A model trained to recognize cats and dogs can reuse its “edge detection” features to identify
other animals.
Parameter Transfer
• Simple meaning: Reusing learned model parameters or settings (like weights) from one task to another.
• Example: Starting with a pre-trained language model (BERT) and fine-tuning it for sentiment analysis.
Relational-Knowledge Transfer
• Simple meaning: Transferring knowledge in data where items are connected to each other.
• Example: Using insights from one social network (like Facebook friendships) to predict connections in
another (like LinkedIn).
Importance
• Limited Data:
• Acquiring extensive labelled data is often challenging and costly. Transfer learning enables
us to use pre-trained models, reducing the dependency on large datasets.
• Enhanced Performance:
• Starting with a pre-trained model which has already learned from substantial data allows
for faster and more accurate results on new tasks ideal for applications needing high
accuracy and efficiency.
• Time and Cost Efficiency:
• Transfer learning shortens training time and conserves resources by utilizing existing
models hence eliminating the need for training from scratch.
• Adaptability:
• Models trained on one task can be fine-tuned for related tasks making transfer learning
versatile for various applications from image recognition to natural language processing.
Method
• Pre-trained Model:
• Start with a model already trained on a large dataset for a specific task. This pre-trained
model has learned general features and patterns that are relevant across related tasks.
• Base Model:
• This pre-trained model, known as the base model, includes layers that have processed data
to learn hierarchical representations, capturing low-level to complex features.
• Transfer Layers:
• Identify layers within the base model that hold generic information applicable to both the
original and new tasks. These layers often near the top of the network capture broad,
reusable features.
• Fine-tuning:
• Fine-tune these selected layers with data from the new task. This process helps retain the
pre-trained knowledge while adjusting parameters to meet the specific requirements of the
new task, improving accuracy and adaptability.
Frozen Vs Trainable Layers in Transfer Learning

• Frozen Layers: These layers from a pre-trained model remain

unchanged during fine-tuning. They retain general features learned
from the original task, extracting universal patterns from input data.
• Trainable Layers: These layers are adjusted during fine-tuning to learn
task-specific features from the new dataset, allowing the model to
meet the new task’s unique requirements.
 How to Decide Which Layers to Freeze or Train?
• Small, Similar Dataset:
• For smaller datasets that resemble the original dataset, you freeze most layers and only
fine-tune the last one or two layers to prevent overfitting.
• Large, Similar Dataset:
• With large, similar datasets you can unfreeze more layers allowing the model to adapt while
retaining learned features from the base model.
• Fine tune more and small learning rate
• Small, Different Dataset:
• For smaller, dissimilar datasets, fine-tuning layers closer to the input layer helps the model
learn task-specific features from scratch.
• Freeze most of the network, Fine tune only the last layers
• Large, Different Dataset:
• In this case, fine-tuning the entire model helps the model adapt to the new task while
using the broad knowledge from the pre-trained model.
• Fine tune the entire model
Applications
• Computer Vision: Transfer learning is prevalent in image recognition tasks
where models pre-trained on large image datasets are adapted to specific
tasks such as medical imaging, facial recognition and object detection.
• Natural Language Processing (NLP): In NLP models like BERT, GPT or ELMo
are pre-trained on vast text corpora and later fine-tuned for specific tasks
such as sentiment analysis, machine translation and question-answering.
• Healthcare: Transfer learning helps develop medical diagnostic tools using
knowledge from general image recognition models to analyze medical
images like X-rays or MRIs.
• Finance: Transfer learning in finance assists in fraud detection, risk
assessment and credit scoring by transferring patterns learned from related
financial datasets.
Advantages of Transfer Learning
• Speed up the training process:
• By using a pre-trained model the model can learn more quickly and effectively
on the second task, as it already has a good understanding of the features and
patterns in the data.
• Better performance:
• Transfer learning can lead to better performance on the second task, as the
model can use the knowledge it has gained from the first task.
• Handling small datasets:
• When there is limited data available for the second task, transfer learning can
help to prevent overfitting as the model will have already learned general
features that are likely to be useful in the second task.
Disadvantages of Transfer Learning

• Domain mismatch:
• The pre-trained model may not be well-suited to the second task if the two
tasks are vastly different or the data distribution between the two tasks is
very different.
• Overfitting:
• Transfer learning can lead to overfitting if the model is fine-tuned too much
on the second task, as it may learn task-specific features that do not
generalize well to new data.
• Complexity:
• The pre-trained model and the fine-tuning process can be computationally
expensive and may require specialized hardware.
Deep Transfer Learning
Deep Transfer Learning Strategies

• Off-the-shelf Pre-trained Models as Feature Extractors

• Fine Tuning Off-the-shelf Pre-trained Models
Examples
• Computer Vision • NLP
• VGG-16 • Word2Vec
• VGG-19 • GloVe
• Inception V3 • FastText
• XCeption • Universal Sentence Encoder by Google
• ResNet-50 • Bidirectional Encoder Representations
from Transformers (BERT) by Google
MobileNet
•Introduced by Google (2017)
• designed for efficient mobile & embedded vision applications.
•Lightweight CNN –
•optimized for devices with limited resources.
•Uses Depthwise Separable Convolutions –
• reduces computation & model size.
•Parameter Reduction –
• far fewer parameters than VGG/ResNet.
•Trade-off Control –
•width multiplier & resolution multiplier adjust speed vs. accuracy.
•Efficient on Mobile/IoT –
•low latency, suitable for smartphones, drones, AR/VR.
•Performance –
•achieves comparable accuracy to large CNNs with fewer resources.
•Model Variants –
• MobileNetV1, V2, V3 (progressively improved efficiency & accuracy).
•Training Dataset –
• trained on ImageNet (1.2M images, 1000 categories).
•Applications – image classification, detection, face recognition, NLP (via transfer learning).
 Alexnet(2012)
1. Introduced by:
1. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton.
2. Paper: "ImageNet Classification with Deep Convolutional Neural Networks" (NeurIPS 2012).
2. Competition:
1. Won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC-2012).
2. Achieved top-5 error rate of 15.3%, vs ~26% for the runner-up.
3. Architecture Highlights:
1. 8 layers total:
1. 5 convolutional layers.
2. 3 fully connected layers.
2. Final layer: Softmax classifier (1000 ImageNet classes).
4. Innovations:
1. ReLU activation → faster training than tanh/sigmoid.
2. Dropout regularization → prevented overfitting.
3. Data augmentation → image translations, reflections, crops.
4. GPU acceleration → trained using two NVIDIA GTX 580 GPUs.
5. Overlapping max-pooling → improved feature capture.
5. Size & Parameters:
1. ~62 million parameters.
2. Input: RGB images resized to 224×224×3.
6. Impact:
1. Sparked the deep learning revolution in computer vision.
VGG-16
•Developed by:
•Oxford’s Visual Geometry Group (VGG), led by Karen Simonyan & Andrew Zisserman.
•Year:
• Introduced in 2014, published in the paper “Very Deep Convolutional Networks for Large-
Scale Image Recognition”.
•Competition:
•Submitted to the ILSVRC-2014 (ImageNet Large Scale Visual Recognition Challenge).
•Layers:
•16 weight layers (13 convolutional + 3 fully connected).
•Input size:
•Standardized to 224×224 RGB images.
•Parameters:
•~138 million trainable parameters.
•Training data:
•Trained on ImageNet dataset (over 1.2 million images across 1000 categories).
•Performance:
• Achieved 7.3% top-5 error rate, a major improvement in 2014.
 VGG-19 (19-layer model)
•Also by:
•Simonyan & Zisserman, Visual Geometry Group, Oxford.
•Year:
•2014, alongside VGG-16, same research paper.
•Layers:
•19 weight layers (16 convolutional + 3 fully connected).
•Structure:
•Deeper version of VGG-16 with additional conv layers for richer feature extraction.
•Parameters:
•~144 million trainable parameters (slightly higher than VGG-16).
•Training data:
•ImageNet dataset (1.2M+ images, 1000 classes).
•Performance:
•Slightly better than VGG-16 but with much higher computational cost.
•Legacy:
•Forms the foundation of many transfer learning tasks in computer vision
• (e.g., feature extraction, style transfer).
 ResNet-50
•Introduced By:
•Developed by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun in 2015 (Microsoft
Research).
•Publication:
•Presented in the paper "Deep Residual Learning for Image Recognition" at CVPR 2016.
•Problem Solved:
•Tackles the vanishing gradient problem and performance degradation in very deep neural networks.
•Key Idea:
• Uses residual connections (skip connections) that allow gradients to flow directly across layers.
•Depth:
•50 layers deep – a lighter version compared to ResNet-101 and ResNet-152.
•Training Dataset:
•Originally trained on ImageNet (1.2 million images, 1,000 classes).
•Performance:
•Achieved 3.57% top-5 error rate on ImageNet, winning the ILSVRC 2015 classification task.
•Impact:
•Inspired deeper networks (ResNet-101, ResNet-152) and is widely used in transfer learning, object
detection, and medical image analysis.
BERT
•BERT Bidirectional Encoder Representations from Transformers.
•Developed by Google AI Language (2018).
•Uses self-attention to capture word context bidirectionally.
•Pre-training tasks: Masked Language Modeling (MLM) & Next Sentence
Prediction (NSP).
•Trained on BooksCorpus (800M words) + English Wikipedia (2.5B words).
•Variants: BERT-Base (12 layers) & BERT-Large (24 layers).
•Fine-tuned for tasks: Q&A, Sentiment, NER, Text Classification.
•Outperformed prior models on GLUE, SQuAD, and SWAG benchmarks.
•Sparked a wave of Transformer-based models (RoBERTa, ALBERT,
DistilBERT).
References
• Hands-On Transfer Learning with Python
• Dipanjan Sarkar (also credited as DJ Sarkar)
• Nitin Panwar
• Raghav Bali
• Tamoghna Ghosh
• Github link
• Amazon