Copyright Notice

These slides are distributed under the Creative Commons License.

[Link] makes these slides available for educational purposes. You may not use or distribute
these slides for commercial purposes. You may make copies of these slides and use or distribute them for
educational purposes as long as you cite [Link] as the source of the slides.

For the rest of the details of the license, see [Link]

 Introduction to
ML strategy

Why ML
[Link]
Strategy?
Motivating example

Ideas:
• Collect more data • Try dropout
• Collect more diverse training set • Add !" regularization
• Train algorithm longer with gradient descent • Network architecture
• Try Adam instead of gradient descent • Activation functions
• Try bigger network • # hidden units
• Try smaller network • … Andrew Ng
 Introduction to
ML strategy

Orthogonalization
[Link]
TV tuning example
Car

Andrew Ng
Chain of assumptions in ML

Fit training set well on cost function

Fit dev set well on cost function

Fit test set well on cost function

Performs well in real world

Andrew Ng
 Setting up
your goal

Single number
[Link]
evaluation metric
Using a single number evaluation metric

Idea

Classifier Precision Recall F1 Score

A 95% 90% 92.4%
B 98% 85% 91.0%
Experiment Code

Andrew Ng
Another example

Algorithm US China India Other Average

A 3% 7% 5% 9% 6%
B 5% 6% 5% 10% 6.5%
C 2% 3% 4% 5% 3.5%
D 5% 8% 7% 2% 5.25%
E 4% 5% 2% 4% 3.75%
F 7% 11% 8% 12% 9.5%

Andrew Ng
 Setting up
your goal

Satisficing and
[Link]
optimizing metrics
Another cat classification example
Classifier Accuracy Running time
A 90% 80ms
B 92% 95ms
C 95% 1,500ms

Andrew Ng
 Setting up
your goal

Train/dev/test
[Link]
distributions
Cat classification dev/test sets
Regions:
• US
• UK
• Other Europe
• South America
• India
Idea
• China
• Other Asia
• Australia

Experiment Code
Andrew Ng
True story (details changed)

Optimizing on dev set on loan approvals for

medium income zip codes

Tested on low income zip codes

Andrew Ng
Guideline

Choose a dev set and test set to reflect data you

expect to get in the future and consider important
to do well on.

Andrew Ng
 Setting up
your goal

Size of dev
[Link]
and test sets
Old way of splitting data

Andrew Ng
Size of dev set
Set your dev set to be big enough to detect differences in
algorithm/models you’re trying out.

Andrew Ng
Size of test set
Set your test set to be big enough to give high confidence
in the overall performance of your system.

Andrew Ng
 Setting up
your goal

When to change
[Link] dev/test sets and
metrics
Cat dataset examples

Metric: classification error

Algorithm A: 3% error
Algorithm B: 5% error

Andrew Ng
Orthogonalization for cat pictures: anti-porn

1. So far we’ve only discussed how to define a metric to

evaluate classifiers.

2. Worry separately about how to do well on this metric.

Andrew Ng
Another example
Algorithm A: 3% error
Algorithm B: 5% error
Dev/test User images

If doing well on your metric + dev/test set does not

correspond to doing well on your application, change your
metric and/or dev/test set.

Andrew Ng
 Comparing to human-
level performance

Why human-level
[Link]
performance?
Comparing to human-level performance

accuracy

time

Andrew Ng
Why compare to human-level performance
Humans are quite good at a lot of tasks. So long as
ML is worse than humans, you can:
- Get labeled data from humans.

- Gain insight from manual error analysis:

Why did a person get this right?

- Better analysis of bias/variance.

Andrew Ng
 Comparing to human-
level performance

Avoidable bias
[Link]
Bias and Variance

high bias “just right” high variance

Andrew Ng
Bias and Variance
Cat classification

Training set error: 1% 15% 15% 0.5%

Dev set error: 11% 16% 30% 1%

Andrew Ng
Cat classification example

Training error 8% 8%
Dev error 10% 10 %

Andrew Ng
 Comparing to human-
level performance
Understanding
[Link] human-level
performance
Human-level error as a proxy for Bayes error
Medical image classification example:
Suppose:
(a) Typical human ………………. 3 % error

(b) Typical doctor ………………... 1 % error

(c) Experienced doctor …………... 0.7 % error

(d) Team of experienced doctors .. 0.5 % error

What is “human-level” error?

Andrew Ng
Error analysis example

Training error

Dev error

Andrew Ng
Summary of bias/variance with human-level
performance

Human-level error

Training error

Dev error

Andrew Ng
 Comparing to human-
level performance

Surpassing human-
[Link]
level performance
Surpassing human-level performance

Team of humans

One human

Training error

Dev error

Andrew Ng
Problems where ML significantly surpasses
human-level performance

- Online advertising

- Product recommendations

- Logistics (predicting transit time)

- Loan approvals

Andrew Ng
 Comparing to human-
level performance

Improving your model

[Link]
performance
The two fundamental assumptions of
supervised learning

1. You can fit the training set pretty well.

2. The training set performance generalizes pretty

well to the dev/test set.

Andrew Ng
Reducing (avoidable) bias and variance

Human-level Train bigger model

Train longer/better optimization algorithms

Training error NN architecture/hyperparameters search

More data
Dev error Regularization

NN architecture/hyperparameters search
Andrew Ng
 Orthogonalization

Orthogonalization or orthogonality is a system design property that assures that modifying an instruction
or a component of an algorithm will not create or propagate side effects to other components of the
system. It becomes easier to verify the algorithms independently from one another, it reduces testing and
development time.

When a supervised learning system is design, these are the 4 assumptions that needs to be true and
orthogonal.

1. Fit training set well in cost function

- If it doesn’t fit well, the use of a bigger neural network or switching to a better optimization
algorithm might help.
2. Fit development set well on cost function
- If it doesn’t fit well, regularization or using bigger training set might help.
3. Fit test set well on cost function
- If it doesn’t fit well, the use of a bigger development set might help
4. Performs well in real world
- If it doesn’t perform well, the development test set is not set correctly or the cost function is
not evaluating the right thing.
 Single number evaluation metric
To choose a classifier, a well-defined development set and an evaluation metric speed up the iteration
process.

Example : Cat vs Non- cat

y = 1, cat image detected
Actual class 𝑦

Predict class 𝑦ො
1 0
1 True positive False positive
0 False negative True negative
Precision
Of all the images we predicted y=1, what fraction of it have cats?
𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒
Precision (%) = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑥 100 = (𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒+𝐹𝑎𝑙𝑠𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒) 𝑥 100

Recall

Of all the images that actually have cats, what fraction of it did we correctly identifying have cats?
𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒
Recall (%) = 𝑥 100 = 𝑥 100
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 𝑎𝑐𝑡𝑢𝑎𝑙𝑙𝑦 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 (𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒+𝐹𝑎𝑙𝑠𝑒 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒)

Let’s compare 2 classifiers A and B used to evaluate if there are cat images:

Classifier Precision (p) Recall (r)

A 95% 90%
B 98% 85%

In this case the evaluation metrics are precision and recall.

For classifier A, there is a 95% chance that there is a cat in the image and a 90% chance that it has correctly
detected a cat. Whereas for classifier B there is a 98% chance that there is a cat in the image and a 85%
chance that it has correctly detected a cat.

The problem with using precision/recall as the evaluation metric is that you are not sure which one is
better since in this case, both of them have a good precision et recall. F1-score, a harmonic mean, combine
both precision and recall.
2
F1-Score= 1 1
+
𝑝 𝑟

Classifier Precision (p) Recall (r) F1-Score

A 95% 90% 92.4 %
B 98% 85% 91.0%

Classifier A is a better choice. F1-Score is not the only evaluation metric that can be use, the average, for
example, could also be an indicator of which classifier to use.
 Satisficing and optimizing metric

There are different metrics to evaluate the performance of a classifier, they are called evaluation matrices.
They can be categorized as satisficing and optimizing matrices. It is important to note that these evaluation
matrices must be evaluated on a training set, a development set or on the test set.

Example: Cat vs Non-cat

Classifier Accuracy Running time

A 90% 80 ms
B 92% 95 ms
C 95% 1 500 ms

In this case, accuracy and running time are the evaluation matrices. Accuracy is the optimizing metric,
because you want the classifier to correctly detect a cat image as accurately as possible. The running time
which is set to be under 100 ms in this example, is the satisficing metric which mean that the metric has
to meet expectation set.

The general rule is:

1 𝑂𝑝𝑡𝑖𝑚𝑖𝑧𝑖𝑛𝑔 𝑚𝑒𝑡𝑟𝑖𝑐
𝑁𝑚𝑒𝑡𝑟𝑖𝑐 : {
𝑁𝑚𝑒𝑡𝑟𝑖𝑐 − 1 𝑆𝑎𝑡𝑖𝑠𝑓𝑖𝑐𝑖𝑛𝑔 𝑚𝑒𝑡𝑟𝑖𝑐
 Training, development and test distributions

Setting up the training, development and test sets have a huge impact on productivity. It is important to
choose the development and test sets from the same distribution and it must be taken randomly from all
the data.

Guideline

Choose a development set and test set to reflect data you expect to get in the future and consider
important to do well.
 Size of the development and test sets
Old way of splitting data
We had smaller data set therefore we had to use a greater percentage of data to develop and test ideas
and models.

70 % 30 %

Training set Test set

Or

60 % 20 % 20 %

Training set Development set Test set

Modern era – Big data

Now, because a large amount of data is available, we don’t have to compromised as much and can use a
greater portion to train the model.

98 % 1% 1%
1

Training set Development set Test set

Guidelines
• Set up the size of the test set to give a high confidence in the overall performance of the system.
• Test set helps evaluate the performance of the final classifier which could be less 30% of the whole
data set.
• The development set has to be big enough to evaluate different ideas.
 When to change development/test sets and metrics
Example: Cat vs Non-cat
A cat classifier tries to find a great amount of cat images to show to cat loving users. The evaluation metric
used is a classification error.

Algorithm Classification error [%]

A 3%
B 5%

It seems that Algorithm A is better than Algorithm B since there is only a 3% error, however for some reason,
Algorithm A is letting through a lot of the pornographic images.

Algorithm B has 5% error thus it classifies fewer images but it doesn't have pornographic images. From a
company's point of view, as well as from a user acceptance point of view, Algorithm B is actually a better
algorithm. The evaluation metric fails to correctly rank order preferences between algorithms. The evaluation
metric or the development set or test set should be changed.

The misclassification error metric can be written as a function as follow:

𝑚𝑑𝑒𝑣
1
𝐸𝑟𝑟𝑜𝑟 ∶ ∑ ℒ{(𝑦̂ (𝑖) ≠ 𝑦 (𝑖) }
𝑚𝑑𝑒𝑣
𝑖=1

This function counts up the number of misclassified examples.

The problem with this evaluation metric is that it treats pornographic vs non-pornographic images equally. On
way to change this evaluation metric is to add the weight term 𝑤 (𝑖) .

1 𝑖𝑓 𝑥 (𝑖) 𝑖𝑠 𝑛𝑜𝑛 − 𝑝𝑜𝑟𝑛𝑜𝑔𝑟𝑎𝑝ℎ𝑖𝑐

𝑤 (𝑖) = {
10 𝑖𝑓 𝑥 (𝑖) 𝑖𝑠 𝑝𝑜𝑟𝑛𝑜𝑔𝑟𝑎𝑝ℎ𝑖𝑐

The function becomes:

𝑚𝑑𝑒𝑣
1
𝐸𝑟𝑟𝑜𝑟 ∶ ∑ 𝑤 (𝑖) ℒ{(𝑦̂ (𝑖) ≠ 𝑦 (𝑖) }
∑ 𝑤 (𝑖)
𝑖=1

Guideline

1. Define correctly an evaluation metric that helps better rank order classifiers
2. Optimize the evaluation metric
 Why human-level performance?

Today, machine learning algorithms can compete with human-level performance since they are more
productive and more feasible in a lot of application. Also, the workflow of designing and building a
machine learning system, is much more efficient than before.

Moreover, some of the tasks that humans do are close to ‘’perfection’’, which is why machine learning
tries to mimic human-level performance.

The graph below shows the performance of humans and machine learning over time.

Bayes optimal
error

Machine
Learning
Human
s

The

Machine learning progresses slowly when it surpasses human-level performance. One of the reason is
that human-level performance can be close to Bayes optimal error, especially for natural perception
problem.

Bayes optimal error is defined as the best possible error. In other words, it means that any functions
mapping from x to y can’t surpass a certain level of accuracy.

Also, when the performance of machine learning is worse than the performance of humans, you can
improve it with different tools. They are harder to use once its surpasses human-level performance.

These tools are:

- Get labeled data from humans

- Gain insight from manual error analysis: Why did a person get this right?
- Better analysis of bias/variance.
 Avoidable bias
By knowing what the human-level performance is, it is possible to tell when a training set is performing
well or not.

Example: Cat vs Non-Cat

Classification error (%)

Scenario A Scenario B
Humans 1 7.5
Training error 8 8
Development error 10 10

In this case, the human level error as a proxy for Bayes error since humans are good to identify images. If
you want to improve the performance of the training set but you can’t do better than the Bayes error
otherwise the training set is overfitting. By knowing the Bayes error, it is easier to focus on whether bias
or variance avoidance tactics will improve the performance of the model.

Scenario A
There is a 7% gap between the performance of the training set and the human level error. It means that
the algorithm isn’t fitting well with the training set since the target is around 1%. To resolve the issue, we
use bias reduction technique such as training a bigger neural network or running the training set longer.

Scenario B
The training set is doing good since there is only a 0.5% difference with the human level error. The
difference between the training set and the human level error is called avoidable bias. The focus here is
to reduce the variance since the difference between the training error and the development error is 2%.
To resolve the issue, we use variance reduction technique such as regularization or have a bigger training
set.
 Understanding human-level performance
Human-level error gives an estimate of Bayes error.

Example 1: Medical image classification

This is an example of a medical image classification in which the input is a radiology image and the output
is a diagnosis classification decision.

Classification error (%)

Typical human 3.0
Typical doctor 1.0
Experienced doctor 0.7
Team of experienced doctors 0.5

The definition of human-level error depends on the purpose of the analysis, in this case, by definition the
Bayes error is lower or equal to 0.5%.

Example 2: Error analysis

Classification error (%)
Scenario A Scenario B Scenario C
1 1
Human (proxy for Bayes error) 0.7 0.7 0.5
0.5 0.5
Training error 5 1 0.7
Development error 6 5 0.8
Scenario A
In this case, the choice of human-level performance doesn’t have an impact. The avoidable bias is between
4%-4.5% and the variance is 1%. Therefore, the focus should be on bias reduction technique.

Scenario B
In this case, the choice of human-level performance doesn’t have an impact. The avoidable bias is between
0%-0.5% and the variance is 4%. Therefore, the focus should be on variance reduction technique.

Scenario C
In this case, the estimate for Bayes error has to be 0.5% since you can’t go lower than the human-level
performance otherwise the training set is overfitting. Also, the avoidable bias is 0.2% and the variance is
0.1%. Therefore, the focus should be on bias reduction technique.

Summary of bias/variance with human-level performance

• Human - level error – proxy for Bayes error
• If the difference between human-level error and the training error is bigger than the difference
between the training error and the development error. The focus should be on bias reduction
technique
• If the difference between training error and the development error is bigger than the difference
between the human-level error and the training error. The focus should be on variance reduction
technique
 Surpassing human-level performance
Example1: Classification task

Classification error (%)

Scenario A Scenario B
Team of humans 0.5 0.5
One human 1.0 1
Training error 0.6 0.3
Development error 0.8 0.4
Scenario A
In this case, the Bayes error is 0.5%, therefore the available bias is 0.1% et the variance is 0.2%.

Scenario B
In this case, there is not enough information to know if bias reduction or variance reduction has to be
done on the algorithm. It doesn’t mean that the model cannot be improve, it means that the conventional
ways to know if bias reduction or variance reduction are not working in this case.

There are many problems where machine learning significantly surpasses human-level performance,
especially with structured data:

• Online advertising
• Product recommendations
• Logistics (predicting transit time)
• Loan approvals
 Improving your model performance

The two fundamental assumptions of supervised learning

There are 2 fundamental assumptions of supervised learning. The first one is to have a low avoidable bias
which means that the training set fits well. The second one is to have a low or acceptable variance which
means that the training set performance generalizes well to the development set and test set.

If the difference between human-level error and the training error is bigger than the difference between
the training error and the development error, the focus should be on bias reduction technique which are
training a bigger model, training longer or change the neural networks architecture or try various
hyperparameters search.

If the difference between training error and the development error is bigger than the difference between
the human-level error and the training error, the focus should be on variance reduction technique which
are bigger data set, regularization or change the neural networks architecture or try various
hyperparameters search.

Summary

• Train bigger model

• Train longer, better optimization algorithms
• Neural Networks architecture/hyperparameters search

• More data
• Regularization
• Neural Networks architecture/hyperparameters search
•
 Copyright Notice
These slides are distributed under the Creative Commons License.

[Link] makes these slides available for educational purposes. You may not use or distribute
these slides for commercial purposes. You may make copies of these slides and use or distribute them for
educational purposes as long as you cite [Link] as the source of the slides.

For the rest of the details of the license, see [Link]

 Error Analysis

Carrying out error

[Link]
analysis
Look at dev examples to evaluate ideas

Should you try to make your cat classifier do better on dogs?

Error analysis:
• Get ~100 mislabeled dev set examples.
• Count up how many are dogs.

Andrew Ng
Evaluate multiple ideas in parallel
Ideas for cat detection:
• Fix pictures of dogs being recognized as cats
• Fix great cats (lions, panthers, etc..) being misrecognized
• Improve performance on blurry images
Image
1
2
3
..
.
% of total
Andrew Ng
 Error Analysis

Cleaning up
[Link] Incorrectly labeled
data
Incorrectly labeled examples

y 1 0 1 1 0 1 1

DL algorithms are quite robust to random errors in the

training set.

Andrew Ng
Error analysis
Incorrectly
Image Dog Great Cat Blurry Comments
labeled
…
Labeler missed cat
98 in background

99
Drawing of a cat;
100 Not a real cat.

% of total 8% 43% 61% 6%

Overall dev set error

Errors due incorrect labels

Errors due to other causes

Goal of dev set is to help you select between two classifiers A & B.
Andrew Ng
Correcting incorrect dev/test set examples

• Apply same process to your dev and test sets to

make sure they continue to come from the same
distribution
• Consider examining examples your algorithm got
right as well as ones it got wrong.
• Train and dev/test data may now come from
slightly different distributions.
Andrew Ng
 Error Analysis

Build your first system

[Link]
quickly, then iterate
Speech recognition example

• Noisy background • Set up dev/test set

• Café noise and metric
• Car noise • Build initial
• Accented speech system quickly
Guideline:
• Far from microphone • Use Bias/Variance
Build your
• Young children’s first
speech analysis & Error
system quickly,
• Stuttering analysis to
• … prioritize next
then iterate steps.
Andrew Ng
 Mismatched training
and dev/test data
Training and testing
[Link] on different
distributions
Cat app example
Data from webpages Data from mobile app

Andrew Ng
Speech recognition example

Training Dev/test

Purchased data Speech activated

rearview mirror
Smart speaker control

Voice keyboard

Andrew Ng
 Mismatched training
and dev/test data

Bias and Variance with

[Link] mismatched data
distributions
Cat classifier example
Assume humans get ≈ 0% error.

Training error
Dev error

Training-dev set: Same

distribution as training
set, but not used for
training
Andrew Ng
Bias/variance on mismatched training and
dev/test sets

Andrew Ng
More general formulation

Andrew Ng
 Mismatched training
and dev/test data

Addressing data
[Link]
mismatch
Addressing data mismatch
• Carry out manual error analysis to try to understand difference
between training and dev/test sets

• Make training data more similar; or collect more data similar to

dev/test sets

Andrew Ng
Artificial data synthesis

+ =

“The quick brown Car noise Synthesized

fox jumps in-car audio
over the lazy dog.”

Andrew Ng
Artificial data synthesis

Car recognition:

Andrew Ng
 Learning from
multiple tasks

Transfer learning
[Link]
Transfer learning

x !"

x !"

Andrew Ng
When transfer learning makes sense

• Task A and B have the same input x.

• You have a lot more data for Task A than Task B.

• Low level features from A could be helpful for learning B.

Andrew Ng
 Learning from
multiple tasks

Multi-task
[Link]
learning
Simplified autonomous driving example

Andrew Ng
Neural network architecture

x !"

Andrew Ng
When multi-task learning makes sense
• Training on a set of tasks that could benefit from having
shared lower-level features.
• Usually: Amount of data you have for each task is quite
similar.

• Can train a big enough neural network to do well on all

the tasks.

Andrew Ng
 End-to-end deep
learning
What is
[Link] end-to-end
deep learning
What is end-to-end learning?
Speech recognition example

Andrew Ng
Face recognition

[Image courtesy of Baidu]

Andrew Ng
More examples

Machine translation

Estimating child’s age:

Andrew Ng
 End-to-end deep
learning

Whether to use
[Link]
end-to-end learning
Pros and cons of end-to-end deep learning
Pros:
• Let the data speak
• Less hand-designing of components needed

Cons:
• May need large amount of data
• Excludes potentially useful hand-designed
components
Andrew Ng
Applying end-to-end deep learning
Key question: Do you have sufficient data to learn
a function of the complexity needed to map x to y?

Andrew Ng
 Build system quickly, then iterate
Depending on the area of application, the guideline below will help you prioritize when you build your
system.

Guideline
1. Set up development/ test set and metrics
- Set up a target
2. Build an initial system quickly
- Train training set quickly: Fit the parameters
- Development set: Tune the parameters
- Test set: Assess the performance
3. Use Bias/Variance analysis & Error analysis to prioritize next steps
 Training and testing on different distributions
Example: Cat vs Non-cat
In this example, we want to create a mobile application that will classify and recognize pictures of cats
taken and uploaded by users.

There are two sources of data used to develop the mobile app. The first data distribution is small, 10 000
pictures uploaded from the mobile application. Since they are from amateur users, the pictures are not
professionally shot, not well framed and blurrier. The second source is from the web, you downloaded
200 000 pictures where cat’s pictures are professionally framed and in high resolution.

The problem is that you have a different distribution:

1- small data set from pictures uploaded by users. This distribution is important for the mobile app.
2- bigger data set from the web.

The guideline used is that you have to choose a development set and test set to reflect data you expect
to get in the future and consider important to do well.

The data is split as follow:

Web App App

205 000 5 000 5 000 5 000

1

Training set Development set Test set

The advantage of this way of splitting up is that the target is well defined.

The disadvantage is that the training distribution is different from the development and test set
distributions. However, this way of splitting the data has a better performance in long term.
 Bias and variance with mismatched data distributions
Example: Cat classifier with mismatch data distribution
When the training set is from a different distribution than the development and test sets, the method to
analyze bias and variance changes.

Classification error (%)

Scenario A Scenario B Scenario C Scenario D Scenario E Scenario F
Human (proxy for Bayes error) 0 0 0 0 0 4
Training error 1 1 1 10 10 7
Training-development error - 9 1.5 11 11 10
Development error 10 10 10 12 20 6
Test error - - - - - 6

Scenario A
If the development data comes from the same distribution as the training set, then there is a large
variance problem and the algorithm is not generalizing well from the training set.
However, since the training data and the development data come from a different distribution, this
conclusion cannot be drawn. There isn't necessarily a variance problem. The problem might be that the
development set contains images that are more difficult to classify accurately.
When the training set, development and test sets distributions are different, two things change at the
same time. First of all, the algorithm trained in the training set but not in the development set. Second of
all, the distribution of data in the development set is different.

It's difficult to know which of these two changes what produces this 9% increase in error between the
training set and the development set. To resolve this issue, we define a new subset called training-
development set. This new subset has the same distribution as the training set, but it is not used for
training the neural network.

Scenario B

The error between the training set and the training- development set is 8%. In this case, since the training
set and training-development set come from the same distribution, the only difference between them is
the neural network sorted the data in the training and not in the training development. The neural
network is not generalizing well to data from the same distribution that it hadn't seen before
Therefore, we have really a variance problem.

Scenario C
In this case, we have a mismatch data problem since the 2 data sets come from different distribution.

Scenario D
In this case, the avoidable bias is high since the difference between Bayes error and training error is 10 %.

Scenario E
In this case, there are 2 problems. The first one is that the avoidable bias is high since the difference
between Bayes error and training error is 10 % and the second one is a data mismatched problem.

Scenario F
Development should never be done on the test set. However, the difference between the development
set and the test set gives the degree of overfitting to the development set.
General formulation

Bayes error

Avoidable Bias

Training set error

Variance

Development - Training set error

Data mismatch

Development set error

Degree of overfitting to the development set

Test set error

 Addressing data mismatch

This is a general guideline to address data mismatch:

• Perform manual error analysis to understand the error differences between training,
development/test sets. Development should never be done on test set to avoid overfitting.

• Make training data or collect data similar to development and test sets. To make the training data
more similar to your development set, you can use is artificial data synthesis. However, it is
possible that if you might be accidentally simulating data only from a tiny subset of the space of
all possible examples.
 Transfer Learning
Transfer learning refers to using the neural network knowledge for another application.

When to use transfer learning

• Task A and B have the same input 𝑥
• A lot more data for Task A than Task B
• Low level features from Task A could be helpful for Task B

Example 1: Cat recognition - radiology diagnosis

The following neural network is trained for cat recognition, but we want to adapt it for radiology diagnosis.
The neural network will learn about the structure and the nature of images. This initial phase of training
on image recognition is called pre-training, since it will pre-initialize the weights of the neural network.
Updating all the weights afterwards is called fine-tuning.

For cat recognition

Input 𝑥: image
Output 𝑦 – 1: cat, 0: no cat

Images Image recognition - cat

Radiology diagnosis
Input 𝑥: Radiology images – CT Scan, X-rays
Output 𝑦 :Radiology diagnosis – 1: tumor malign, 0: tumor benign

Radiology images
𝑥

Radiology diagnosis
𝑦ො
Guideline
• Delete last layer of neural network
• Delete weights feeding into the last output layer of the neural network
• Create a new set of randomly initialized weights for the last layer only
• New data set (𝑥, 𝑦)
 Multi-task learning
Multi-task learning refers to having one neural network do simultaneously several tasks.

When to use multi-task learning

• Training on a set of tasks that could benefit from having shared lower-level features
• Usually: Amount of data you have for each task is quite similar
• Can train a big enough neural network to do well on all tasks

Example: Simplified autonomous vehicle

The vehicle has to detect simultaneously several things: pedestrians, cars, road signs, traffic lights, cyclists,
etc. We could have trained four separate neural networks, instead of train one to do four tasks. However,
in this case, the performance of the system is better when one neural network is trained to do four tasks
than training four separate neural networks since some of the earlier features in the neural network could
be shared between the different types of objects.

The input 𝑥 (𝑖) is the image with multiple labels

The output 𝑦 (𝑖) has 4 labels which are represents:
0 Pedestrians
(𝑖) 1 Cars
𝑦 = [ ]
1 Road signs - Stop
0 Traffic lights

| | | | 𝑌 = (4, 𝑚)
𝑌 = [𝑦 (1) (2) (3) (4) ]
𝑦 𝑦 𝑦
𝑌 = (4,1)
| | | |

Neural Network architecture Pedestrians

Cars

Road signs - Stop

To train this neural network, loss function is defined as follow: Traffic lights
𝑚 4
1 (𝑖) (𝑖) (𝑖) (𝑖)
− ∑ ∑ (𝑦𝑗 log (𝑦̂𝑗 ) + (1 − 𝑦𝑗 ) log (1 − 𝑦̂𝑗 ))
𝑚
𝑖=1 𝑗=1

Also, the cost can be compute such as it is not influenced by the fact that some entries are not labeled.
Example:
1 0 ? ?
0 1 ? 0
𝑌=[ ]
0 1 ? 1
? 0 1 0
 What is end-to-end deep learning
End-to-end deep learning is the simplification of a processing or learning systems into one neural
network.

Example - Speech recognition model

The traditional way - small data set

Audio Extract Features Phonemes Words Transcript

The hybrid way - medium data set

Audio Phonemes Words Transcript

The End-to-End deep learning way – large data set

Audio Transcript

End-to-end deep learning cannot be used for every problem since it needs a lot of labeled data. It is used
mainly in audio transcripts, image captures, image synthesis, machine translation, steering in self-driving
cars, etc.
 Whether to use end-to-end deep learning

Before applying end-to-end deep learning, you need to ask yourself the following question: Do you have
enough data to learn a function of the complexity needed to map x and y?

Pro:
• Let the data speak
- By having a pure machine learning approach, the neural network will learn from x to y. It will
be able to find which statistics are in the data, rather than being forced to reflect human
preconceptions.

• Less hand-designing of components needed

- It simplifies the design work flow.

Cons:
• Large amount of labeled data
- It cannot be used for every problem as it needs a lot of labeled data.

• Excludes potentially useful hand-designed component

- Data and any hand-design’s components or features are the 2 main sources of knowledge for a
learning algorithm. If the data set is small than a hand-design system is a way to give manual
knowledge into the algorithm.