Scribd
Upload
3 views18 pages

Transfer Learning with CNNs Overview

The document discusses transfer learning using Convolutional Neural Networks (CNNs), specifically focusing on the DeCAF model for generic visual tasks. It provides practical steps for utilizing large datasets to improve performance on smaller datasets and emphasizes the importance of monitoring loss curves to avoid issues like overfitting. The conclusion highlights the complexities of training neural networks and the necessity of experimentation and further study.

Uploaded by

devmbandhiya11
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
3 views18 pages

Transfer Learning with CNNs Overview

The document discusses transfer learning using Convolutional Neural Networks (CNNs), specifically focusing on the DeCAF model for generic visual tasks. It provides practical steps for utilizing large datasets to improve performance on smaller datasets and emphasizes the importance of monitoring loss curves to avoid issues like overfitting. The conclusion highlights the complexities of training neural networks and the necessity of experimentation and further study.

Uploaded by

devmbandhiya11
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Transfer Learning

Transfer Learning with CNNs

DeCAF: A Deep Convolutional Activation Feature for Generic Visual

Razavian -the-
CVPR Workshops 2014 Source: cs231n
Transfer Learning with CNNs

DeCAF: A Deep Convolutional Activation Feature for Generic Visual

Razavian -the-
CVPR Workshops 2014 Source: cs231n
Transfer Learning with CNNs

DeCAF: A Deep Convolutional Activation Feature for Generic Visual

Razavian -the-
CVPR Workshops 2014 Source: cs231n
Transfer Learning with CNNs

Source: cs231n
Transfer Learning with CNNs

Source: cs231n
Transfer Learning with CNNs

Source: cs231n
Transfer Learning with CNNs

Source: cs231n
Transfer Learning with CNNs

Source: cs231n
Transfer Learning with CNNs

Source: cs231n
Transfer Learning with CNNs
Takeaway for your projects and beyond:
Have some dataset of interest but it has < ~1M images?

1. Find a very large dataset that has similar data, train a big ConvNet there
2. Transfer learn to your dataset

Deep pretrained models so you

Caffe: [Link]
TensorFlow: [Link]
PyTorch: [Link]
Matconvnet: [Link]

Source: cs231n
Monitor and Visualize
the Loss Curve
Monitor and visualize the loss curve

Source: cs231n
Monitor and visualize the loss curve

Source: cs231n
Monitor and visualize the loss curve

big gap = overfitting


=> increase regularization
strength?

no gap
=> increase model
capacity?

Source: cs231n
Monitor and visualize the loss curve

Not learning: gradients not applied to Overfit: model too large/dataset too small More extreme case of overfitting
weights

slow start

Not converged yet: need longer training Slow start: initialization weights too small Applied the negative of gradients

Source: cs231n
Monitor and visualize the loss curve

Problem: val set too small, statistics not Get nans in the loss after a number of iterations: caused
meaningful by high learning rate and numerical instability in models

Source: cs231n
Attempt at a conclusion
Training neural networks is still a black art

For many techniques, the reasons why, when, and whether they work are in
active dispute
It all comes down to (principled) trial and error
Further reading: A. Karpathy, A recipe for training neural networks

You might also like