FAQ for Deep Learning

Collections for SS20 and SS21

Lecturer: Prof. Bin Yang

Collected by: Qian Tu and Yiwen Liao

Institute of Signal Processing and

System Theory

University of Stuttgart

Version: 2022.06.25
 Table of Contents

Organizations

Lectures

Programming Practice

Assignments

Version: 2022.06.25 ISS – FAQ for DL – 2 / 110

 Table of Contents

Organizations

Lectures

Programming Practice

Assignments

Version: 2022.06.25 ISS – FAQ for DL – 3 / 110

 Is the DL exam open book (can we take notes with us for the exam)?
Yes, the DL exam is open book. All resources (e.g., books, exam papers) are permitted
but electronic devices (e.g. laptops, calculators, smartphones, smart watches) are
prohibited.

Version: 2022.06.25 ISS – FAQ for DL – 4 / 110

 What is necessary to pass the DL module?
In order to pass the DL module you should successfully pass the DL exam.

Version: 2022.06.25 ISS – FAQ for DL – 5 / 110

 Is the DL exam offered during winter semesters as well? Can I skip this summer
semester exam?
The DL exam is offered twice a year, one for the summer term and one for the winter
term. Students can freely choose to take the exam in either terms.

Version: 2022.06.25 ISS – FAQ for DL – 6 / 110

 Is the DL lab only for students who are interested or is it necessary for all to pass
the DL module?
The DL lab is a different course which is offered in every winter term. It is not
necessary for the DL exam or DL module. Successfully passing the DL exam and doing
the three assignments are required to apply for the DL lab.

Version: 2022.06.25 ISS – FAQ for DL – 7 / 110

 Do we have to upload the programming practice or is it voluntary?
You do NOT have to upload the programming practices but the contents of these
practices are relevant for the DL exam.

Version: 2022.06.25 ISS – FAQ for DL – 8 / 110

 Are uploading programming assignments compulsory for taking the deep learning
exam? Can we also submit all of the programming assignments together at the
end of the semester?
■ The submission of the assignments is not compulsory for taking the exam. It is only
required for participating in the DL-lab in winter semesters.
■ Since we provide the solutions to the assignments after the submission deadline, it
is not possible to submit all of them together at the end of the semester.

Version: 2022.06.25 ISS – FAQ for DL – 9 / 110

 Is it allowed to do the programming assignments in groups or should it be done
individually? Are there any theoretical assignments?
■ The assignments are expected to be accomplished individually. Nonetheless, you
can discuss with your fellow students and DL course participants.
■ There are no other theoretical assignment (e.g. calculations), just 4 programming
tasks.

Version: 2022.06.25 ISS – FAQ for DL – 10 / 110

 Are the references relevant for the exam?
Only the contents of the lecture slides and lecture videos are relevant for the exam.
Some parts of the lecture slides, which are marked with a star, are NOT relevant for
the exam. References are available only for your better and deeper understanding of
deep learning and are NOT relevant for the exam.

Version: 2022.06.25 ISS – FAQ for DL – 11 / 110

 Are the expert talks relevant for the DL exam?
The expert talks are NOT relevant to the DL exam.

Version: 2022.06.25 ISS – FAQ for DL – 12 / 110

 How to find the course material of Advanced Mathematics for Signal and
Information Processing.
The materials for AM can be found with the link AM-course. Alternatively, you can
also find the AM course in the list of lectures in winter 2020/21 of the ISS in ILIAS.

Version: 2022.06.25 ISS – FAQ for DL – 13 / 110

 Since the TensorFlow (TF) introduction is a relevant part of the exam, what kind
of questions we could be expect?
■ Concerning the TF introduction, only general questions towards understanding TF
are relevant for the exam, not specific implementations in the notebooks.
■ You will NOT be asked to write codes.

Version: 2022.06.25 ISS – FAQ for DL – 14 / 110

 In some exams, the students can have additional time to read through the exam
beforehand and ask questions, or additional time to transfer the solutions to the
answer sheet. Is this the case in the DL exam, or is this time already included in
the 60 min?
No, this is not the case for the DL exam. In order to be fair in all exams and all lecture
halls, we decided to unify our procedure: There will be NO reading of exam tasks
before. You will directly answer the single choice questions on provided sheets by
drawing crosses. This can save much time for you because you do not have to transfer
your answers from one sheet to another. You only have to answer the calculation
problem on your own sheet. All these hints are printed on the exam cover.

Version: 2022.06.25 ISS – FAQ for DL – 15 / 110

 Are wrong answers in single-choice questions marked negatively?
No, there are no negative points for wrong answers in the single choice problems.

Version: 2022.06.25 ISS – FAQ for DL – 16 / 110

 There are few tasks about RNN in the old exam. How can we review it? Could
you give us a guide?
That is true. The topic RNN is pretty short. If you understand what I explained, that is
enough for the exam.

Version: 2022.06.25 ISS – FAQ for DL – 17 / 110

 Table of Contents

Organizations

Lectures

Programming Practice

Assignments

Version: 2022.06.25 ISS – FAQ for DL – 18 / 110

 In “ch03_4_Kullback-Leibler_divergence_and_cross_entropy” at 36:00 (see
screenshot below), the equation of Gaussian normal distribution for |X| was used.
In this example, we take |X| from the Laplace distribution. Since X is the same,
q
does it mean that we can replace Gaussian with Laplace? How do we obtain π2 σ
at the end.

In the last two lines, we calculate EX∼p (|X|), i.e. the mean of |X| according to the
distribution p, it is a Gaussian distribution. That is the reason for the integral. In the
x2
last line, “...” means the simple calculation of 0∞ x exp (− 2σ
R
2 )dx.

Version: 2022.06.25 ISS – FAQ for DL – 19 / 110

 Can you please explain me the difference between the “,” and the “;” in the
distribution like q(x, y; θ)?
■ “;” is used to separate data arguments from parameters, so q(x, y; θ) is a
distribution function (PDF or PMF) of the random variables x and y. It is
described by the function parameters θ. In the end, q() is a function of both x, y
and θ. Since x, y and θ have different physical meanings, “;” is used to separate
them. In some text books, the simpler notation q(x, y, θ) is used.
■ In the ISS course Statistical and Adaptive Signal Processing (SASP):
□ If θ is given and you study q() as a function of x, y, it is called the distribution.
□ If x, y are given and you study q() as a function of θ, it is called the likelihood function.
This is important for parameter estimation for given x, y.

Version: 2022.06.25 ISS – FAQ for DL – 20 / 110

 In KL-divergence DKL(p||q) between p(x, y) and q(x, y|θ). Does q(x, y|θ) and
q(x, y; θ) mean the same thing? It seems that only the second notation with “;”
is correct. Same question for posterior probability q(y|x, θ) and q(x, y; θ).
Different people use different notations. In this course, the notation is p(x, y; θ) and
p(x|y; θ). ";" is used to show that θ as the parameter vector θ is somehow different
than the data x and y. Nevertheless, notations like p(x,y, θ) from others also exist.

Version: 2022.06.25 ISS – FAQ for DL – 21 / 110

 [SS20] In “ch03_2_Random_variable_and_probability_distribution_(1/2)” from
08:24, shall we use the the kronecker delta (a discrete analog of the Dirac delta
function) instead of the dirac-function when we want to express PMF using PDF?
By using dirac we have 0 or +∞ and not 0 and 1.
This slide talks about the PDF of a discrete-valued RV, not PMF. Hence the Dirac
function is correct because the PDF of such a RV consists of a number of weighted
Dirac functions.

Version: 2022.06.25 ISS – FAQ for DL – 22 / 110

 [SS20] In the Lecture video “ch04_3_Universal_approximation” at 11:40 Mins,
we have considered w1 and w2 as a row vector. But in section 4.3, while
discussing parameter vectors, we discussed how w is a weight matrix with
dimensions RM ×M . Can you please help me understand why was it changed to
the row vector here?

see next slide

Version: 2022.06.25 ISS – FAQ for DL – 23 / 110

 In general, for a layer with M1 inputs x and M2 neurons, you have the expression
W ∗ x where W is a M2 × M1 matrix. Now in your case:
■ The 1. layer has 1 input and M outputs. The 1. weight matrix W1 simplifies to a
column vector w1 of size M × 1, i.e. W1 = w1 .
■ The 2. layer has M inputs and one output. The 2. weight matrix W2 simplifies to a
row vector w2T . w2 is a column vector (default notation for vector) of size M × 1,
i.e. W2 = w2T .

Version: 2022.06.25 ISS – FAQ for DL – 24 / 110

 Why is there a “typically” in the table on slide 4-9? Has the use of activation
functions shifted slightly over the last year?
There is no change of opinion in the past years.
■ “typically” is added only for regression, not for classification.
■ For classification, the final layer ALWAYS uses softmax. It is a must to get
probability-like values.
■ In the video recording, there was “identity” only. It sounds like “a must” as softmax
for classification. But this is not true. You can use an activation function like
sigmoid, tanh in the final layer. It will not make the network better or worse. So
people drop the activation function in the final layer.
■ Hence, “typically” is added in the new slides to say: “typically identity, but you are
allowed to use another activation function”.
In comparison, the hidden layers (at least one) need some nonlinear activation functions
due to the fundamental approximation theorem of NN. For the final layer, this is no
longer necessary because you already have at lest one layer of nonlinear neurons.

Version: 2022.06.25 ISS – FAQ for DL – 25 / 110

 In the example below, for the calculation for x2 , must w1 be transposed too?

No. In this case, the input vector simplifies to a scalar x and the weight matrix of the
1. layer simplifies to a column vector w1 . The result of the 1. layer is a column vector
as well. A special case of the general formulation weight matrix W × input vector x,
with only one column in W .

Version: 2022.06.25 ISS – FAQ for DL – 26 / 110

 In slide 3-19 it proves that KL divergence is always larger than 0. But in 3-20
back KL divergence it is smaller than 0. Does it mean that the first equation only
valid for N ≥ 2?
Regardless forward/backward KL divergence, the integrand may be positive or negative
(slide 3-20), but the integral = KL divergence is always nonnegative.

Version: 2022.06.25 ISS – FAQ for DL – 27 / 110

 [E4.5 MNISTnet1] While defining the network, [Link](Dense(1024,
input_shape=784, activation="tanh")) gave an error. This was fixable by
using input_shape=(1,784). In this case the code ran through to the end.
However, the final results then differed completely from the results you showed for
both loss and accuracy. Any ideas as to why this might be the case?
If you use Keras, the parameter input_shape should receive a tuple rather than an
integer. Therefore, you got an error if you wrote input_shape=784. Second, if you
want to specify the shape of the input (where each sample is a vector) of a dense layer
in Keras, you should write input_shape=(784, ) in your case. Then Keras can
automatically know the shape of the entire training batch during training. What you
wrote here, input_shape=(1, 784), means that each sample is a tensor with the size
of 1 × 1 × 784 rather than a vector. That is why you had different (wrong) results. You
can refer to the API doc. Hint: It is always very important to read the API
documentation before using a function.

Version: 2022.06.25 ISS – FAQ for DL – 28 / 110

 Why is a gradient vector of a batch cost function less noisy than the gradient
vector of a minibatch cost function? Could you please explain this?
In our notation, a batch is the complete dataset containing all samples, while a
minibatch is a small subset of samples. Since each sample is random, so is the
corresponding gradient vector. Hence the average of the gradient vector over a large
number of samples is less noisy than over a small number of samples.

Version: 2022.06.25 ISS – FAQ for DL – 29 / 110

 Each sample being random, has some noise. What is the nature of the
distribution of the noise? Are they i.i.d noise?
The origin of randomness is not solely noise. In real life applications, input signal even
without any noise is typically random, e.g. different written digits: different digits,
written by different people, even when you write “0” ten times, you will have 10
different digit images. Take a look at the ISS course “Advanced mathematics for ...” to
have a better feeling about random variables and stochastic signals.

Version: 2022.06.25 ISS – FAQ for DL – 30 / 110

 What is the meaning of the dynamic range of a neural layer? And what is the
contribution of it in a neural network?
It is what we typically understand: the value range. The dynamic range of a layer is
here a loose name that means collectively the dynamic ranges of all neurons.

Version: 2022.06.25 ISS – FAQ for DL – 31 / 110

 (1/2) Why Normal Random Distribution and Uniform Random Distribution are
not optimum solutions for randomizing the initial parameters? Because of the
reasons for non optimality, why constant activation flow is necessary for the
randomization?)
Random normalization (Gaussian or uniform) is not bad, much better than zero
normalization because it is symmetry-breaking. “not optimum” means that random
normalization of the weight matrices Wl with a constant variance for all layers l is not
optimum. It is better to choose the distribution parameter σ depending on the fan-in
and fan-out of the layer, see He and Glorot initialization.

Version: 2022.06.25 ISS – FAQ for DL – 32 / 110

 (2/2) Why is it better to have different variances? What is meant by constant
activation flow and constant gradient flow?
■ Just as explained in the lecture notes: constant activation flow means that the
variance of an activation of a neuron at a certain layer l after a random initialization
is constant for all layers l. Like in the input and batch normalization, strongly
different data dynamic ranges in different neurons and at different layers will slow
down the convergence of the training. Similarly, constant gradient flow means the
same vector norm of the gradient vector of the loss function L w.r.t different layers.
■ Note that the theoretical calculations shown apply only for the first forward-pass
after initialization. It is not a general proof. In deep learning, many researchers
observed a faster learning by using this kind of initialization in experiments and then
they tried to find an argument. This applied to many methods in Ch. 5 and 6.

Version: 2022.06.25 ISS – FAQ for DL – 33 / 110

 ̸ k, the product
When we derive the He initialization, why we say that for j =
Wij · Wjk is 0? Why is the combined initialization √ 1
and not
Ml +Ml−1
√ 1
which seems more logic to me because if Ml = Ml−1 , the
int(mean(Ml +Ml−1 ))
He’s initialization and Glorot’s initialization are equal.
■ For j ̸= k, Wij and Wik are two independent random weights. Hence
E(Wij Wik ) = E(Wij )E(Wik ). Since we assumed zero mean of Wij , the above term
is zero.
■ The difference between (M1+M2) and mean(M1,M2) is a scaling factor of 2. The
initialization means “∼ ...”, proportional. The absolute scaling is not so critical
because all weights are affected by the same scaling.

Version: 2022.06.25 ISS – FAQ for DL – 34 / 110

 Regarding L1/L2-regularization and the vanishing gradient problem. Since the
regularization adds an additional term to the gradient calculation. Does this
compensate the vanishing-gradient or amplify it, because the regularization
dominates the gradient and reduces learning?
■ By using L2 regularization, the primary effect is the factor 0 < 1 − 2λγ t < 1 instead
of 1. This leads to a weight decaying. This is the case even λ is chosen very small
in order not to change the original optimization problem seriously.
■ There is an additional term in the gradient calculation. Since λ is pretty small and
also θ is not large (due to weight norm penalty), the additional term is also quite
small. So to my knowledge, this regularization does not solve the vanishing gradient
problem as shortcuts do.

Version: 2022.06.25 ISS – FAQ for DL – 35 / 110

 Question on slide 7-6: Why is the red star located in the last column of X_l and
the blue star in the first column? Shouldn’t the results of the first kernel (o = 1)
be located in the first column and the results of the last kernel (o = Dl ) in the
last column?
Yes, it is correct. In this illustration, the signal flows from left to right. So the first
feature map of Xl is interpreted to be the most right one (red star) and the last feature
map to be the most left one (blue star).

Version: 2022.06.25 ISS – FAQ for DL – 36 / 110

 In “ch07_1_Convolutional_layer” at time 1:05:26, we talk about # operations
required for activation signal. Why we don’t count that number to the total
complexity?
■ In the lecture, the number of elements in the kernel W and bias b is counted for
memory complexity and the number of multiplications required for the calculation
of the activation tensor A is counted for computational complexity. The number of
additions is quite similar.
■ We didn’t count the number of activation function calculations because
□ This is trivial. If the layer l has Ml neurons, you need to calculate l-times the function
ϕl .
□ This number is much smaller than Nx and is negligible.

Version: 2022.06.25 ISS – FAQ for DL – 37 / 110

 ′
In ch07_2, the first major modification, we use 1 × 1 × Dl−1 × Dl−1 to reduce
′
the depth. But kernel size is 1 × 1, so Dl−1 maps we got are correlated, (e.g. the
first kernel w1 = 1, the second kernel w2 = 2, in this case each element in second
′ ′
map is 2 times first map). So are these Dl−1 useless? Dl−1 = 1 might be ok.
■ ′
“so Dl−1 maps we got are correlated”: This is in general correct. For any type of
layer, different channels at the output are always functions of the same input tensor.
Hence they are in general correlated, but not always. For special kernels, the first
output channel may only depend on the first input channel etc.
■ “the first kernel w1 = 1, the second kernel w2 = 2, in this case each element in
second map is 2 times first map”: This is wrong. Even for kernel width 1, the value
of a certain pixel of a certain output channel is ϕ(linear combination of all input
channels at the same pixel position). Hence the kernel w is a vector, not a scalar.
Different output channels at the same pixel position are from different linear
combination of the same input channels. "x of first map" will not happen.

Version: 2022.06.25 ISS – FAQ for DL – 38 / 110

 Regarding Property 4 of convolutional layers: ‘We are using the same kernel for
all output neurons.’ Shouldn’t it be ‘kernel tensor’ instead of just ‘kernel’? On
slide 7-5 for example we used the term ’kernel’ just for one part (pink or blue) of
the kernel tensor. Sometimes it seems that we use ‘kernel’ and ‘kernel tensor’ as
synonymous and sometimes ‘kernel’ is just one part of the the kernel tensor.
Yes. The precise formulation is “use the same kernel tensor”. Each 4D kernel tensor
contains Dl different 3D kernels for Dl different output feature maps. Hence we can
also say “use the same kernels ...”. In the literature, not all people distinguish clearly
between kernel tensor and kernel. The meaning should be clear from the context.

Version: 2022.06.25 ISS – FAQ for DL – 39 / 110

 When to use a convolutional layer with stride (e.g. s = 2) and when to use a
(e.g. 2 × 2) max pooling layer. Both reduce the spatial size of the output but in
most CNN architectures max pooling layers are preferred. Is there even a practical
difference besides that the "reduction factor" is not exactly equal? So couldn’t we
“save” one layer by including the spatial size reduction in the convolutional layer?
I mean both variants do not add any multiplications or parameters to the NN?
■ It is an interesting question and a little bit beyond this course. From the practical
view, there is no absolute answer to “when to use strided convolution or pooling”. It
depends on specific experiments, datasets and tasks. You may try this by your own
and check whether there exists some significant differences.
■ The statement “in most CNN architectures max pooling layers are preferred” is not
true. Modern architectures, on the contrary, favour strided convolutions, e.g.
ResNet(2015), MobileNets(2017), MobileNetsV2(2019), MnasNet(2019). Using
pooling layers is more or less a historical issue.

Version: 2022.06.25 ISS – FAQ for DL – 40 / 110

 We have a formula for calculating the activation tensor in lecture ch07_1. If we
have 2 spatial dimensions as in this formula, that it is a 2D
convolution/correlation operation. However, in chapter 7.2 (Major modifications
on W_l), it is a 3D convolution because of the triple sum. So if we want to know
the dimension of the convolutional operation in a convolutional layer, do we have
to count the sums or the spatial dimensions?
In the long formula, we have 2 spatial dimensions and a 2D convolution/correlation
along these dimensions. The input is a 3D tensor and we have a 3rd sum over the input
feature maps. Hence we need a 3D kernel tensor for the triple sum. “3D convolution”
is, however, not a precise term because the 3. sum is not a convolution/correlation (no
shift). But occasionally, people say 3D convolution because of the 3D kernel. Since we
want to generate multiple output feature maps, the final kernel is a 4D tensor. So be
careful: The notations in DL are not always consistent.

Version: 2022.06.25 ISS – FAQ for DL – 41 / 110

 Should we add max-pooling layer after each convolutional layer or 2 layers (the
example in the script)? What’s the difference?
No. There are no fixed pattern. In general: you need conv. layers for feature learning
and max-pooling or stride for downsampling. They have to be harmonized. Too less
conv. layers could lead to badly learned features. Too many conv. layers will lead to a
higher complexity. This is a difficult trade-off. It is an individual decision, often by
trial-and-error.

Version: 2022.06.25 ISS – FAQ for DL – 42 / 110

 When Deconvolution is performed, the size of image increases and the number of
features maps too. The same with Upsampling. When to use one and not the
other or if both are the same, because in some papers about UNet, the authors
describe Upsampling and not Deconvolution. What is the difference and when to
use them?
This is one of the most confusing part in the DL notations. There are different
meanings for “upsampling, deconvolution, ...” for signal processing and DL. You should
always read the paper carefully to see in which sense they are meant.
■ Upsampling (as in signal processing): zero insertion + a fixer low pass filter
(convolution kernel). Here, #outpt maps=#input maps when upsampling applied
to each input map.
■ Deconvolution (as in DL): zero insertion + learnable convolution kernel. In this
case, you can choose any number of output maps = number of convolution kernels.
In DL, mostly deconvolution or upsamling are meaning the same. But again, take a
careful look at the paper what did the authors mean.

Version: 2022.06.25 ISS – FAQ for DL – 43 / 110

 What do we mean below?

You have the loss L of one minibatch containing B samples and want to compute its
gradient w.r.t. one particular weigth matrix Ws or Wx . In a feedforward NN, L
depends on this Wx over B paths, one path for each sample x(n) in the minibatch. In
a RNN, you have much more paths from L through Ws or Wx due to backpropagation
through time. These 2 slides show you the more complicated gradient calculation in a
RNN. If you don’t see it, draw an unfolded RNN with B = 2 samples per minibatch.
Version: 2022.06.25 ISS – FAQ for DL – 44 / 110
 Recurrent layer definition: Is a feedback only for dense layer possible/useful or
also for other layer types such as CNN-layer?
■ Theoretically, this is possible. But for which purpose? What is the goal of a
convolutional layer and a recurrent layer? Convolutional layer: process single image
as input. Recurrent layer: process time series as input by taking its temporal
correlation into account.
■ What happens for video, a sequence of images? You would say, you need both
layers in this case. The answer is: Yes and No. You need both layers to process
video as input, but no in the same layer. Typically, the first layers are convolutional
layers to extract features from individual images. Then recurrent layers are applied
to the extracted feature vectors. So both layers do not happen together.
■ Of course, you can design a layer doing both operations together. This network is
much more complicated (computation, memory) than above and nobody uses this
up to now.

Version: 2022.06.25 ISS – FAQ for DL – 45 / 110

 There are two loss in VAE to minimize, one is lrec and one is lKL . I understand
that the minimization of lKL will force the q(z|x) to resemble to the Gaussian
Distribution. However, I still don’t understand why minimizing the lrec will give us
an optimum reconstruction of x from z? And what is the meaning of optimum in
this scenario? Is that meaning q(z|x) nearly equal to 1 for a given Z ∗ and be 0
for other Z?
This is indeed a statement which is non-trivial to see because the first expression lrec
looks so complicated. Try to simplify this expression to −EZ∼... ln q(x|Z) or
−EX∼p(x) ln q(x|Z ∼ qe ), because Z is calculated from X by the encoder and the
distribution Z ∼ ... originates from the distribution X ∼ p(x). Then you see it is
exactly the KLD KLD(p||q) = const − EX∼p ln q(X; θ). A few slides ago as we tried to
approximate the true unknown distribution p(x) by a parametric model q(x; θ). It is
always the general expression cross entropy = −EX∼p ln q(X; θ) or
cross entropy = −EX,Y ∼p ln q(Y |X; θ) to be minimized to approximate an unknown
distribution by a parametric model (DNN).

Version: 2022.06.25 ISS – FAQ for DL – 46 / 110

 Regarding slide 9-7, Adam optimizer does adapt the learning rate automatically,
so what is actually meant by learning rate gamma = 0.001?
If you take a look at the Adam optimizer syntax below, you will see that it has a
number of parameters including learning_rate. Adam adapts this basic
learning_rate by taking recursively averaged 1. and 2. order moments of gradients
into account. But you still need this basic learning_rate whose default value is
0.001. Please take a look at the document of Adam.

Version: 2022.06.25 ISS – FAQ for DL – 47 / 110

 On slide 9-10 it says that generative models would learn the joint distribution
p(x, y). In what way this is realized? A cGAN is supposed to learn the
distribution p(x, y), but the generator is just a neural net which gets a variable y
as input. It seems to me that the generator would thereby only learn the
distribution p(x|y). Or can you say that because we know the empirical
distribution of p(y), we thereby also know p(x, y) = p(x|y) ∗ p(y)? To add to
this, another question of mine is what distribution a normal autoencoder learns
and whether it is discriminative. It says in the slides that a normal autoencoder is
not generative but it seems to me that it cannot differentiate between different y
given a x so it also cannot be discriminative. Is it a class of its own? If we are
asked to name e.g. an activation function that would fit a given problem, are we
required to write its definition down to receive all points or is it enough to say
that you need e.g. ReLU and why it is used here?
see next slide

Version: 2022.06.25 ISS – FAQ for DL – 48 / 110

 ■ Activation function: just the name. No need to copy the definition from the lecture.
■ AE: Yes. It is neither generative (cannot generate x̂ without x) nor discriminative
(no decision ŷ). Please do not think that discrinminative and generative models
cover ALL models of the world. At the beginning, there were only discriminative
models in ML/DL. Then generative models came in. Slide 9-10 tries to distinguish
generative from discriminative models.
■ cGAN: Yes. G alone indeed learns only p(x|y). Since p(y) is given by data, it is
indeed true and even better to say “learn p(x, y) = p(x|y)p(y)”. This is important
because you can calculate p(y|x) from p(x, y) and hence use a generative model for
discriminative task (possible, but rarely done in practice). If you only learned p(x|y)
while ignoring p(y), you cannot calculate p(y|x).

Version: 2022.06.25 ISS – FAQ for DL – 49 / 110

 Given p(x|y) = p(y|x) ∗ p(x)/p(y), where p(y) is similarly given as empirical
distribution of the data, could not in theory a discriminative model, which learns
the distribution p(y|x), be used for generating data if p(x) were given? Or is it
that one cannot usually infer p(x) from the data? Or would it be only possible
very approximately, and not be feasible at all?
■ If you are given p(x), you are actually already able to generate data (more formally,
sampling from a distribution). In this case, the ways to sample from p(x) is another
story in the field of statistics.
■ Inferring p(x) from the data is feasible for low-dimensional data, e.g., by using the
kernel density estimation. However, when it turns to high-dimensional complex data
like images, it is difficult to infer p(x) from the data.

Version: 2022.06.25 ISS – FAQ for DL – 50 / 110

 On slide 9-14, it is said that q(z|x) is unknown, but close to Gaussian
N (µ(x), C(x)) due to [Link](q(z|x)||p(z)). p(z) is Gaussian N (0, I). Why is
then q(z|x) ∼ N (µ(x), C(x)) and not approximately ∼ N (0, I)?
VAE minimizes lrec + lKL , i.e. two different cost functions lrec and lKL jointly. Only a
compromise is possible. You will never achieve lKL =0.

Version: 2022.06.25 ISS – FAQ for DL – 51 / 110

 In slide 9-21 regarding the minibatch size B. For updating D we draw B samples
z and select B samples x, so in total we should have 2B samples in total. Why
do we average over B and not 2B when computing the gradient.
■ The difference between division by B or 2B is a factor 2. Such a fixed factor of the
gradient is not a issue because you can always compensate it by changing the
learning rate correspondingly. To formulate it more dramatically, you can even drop
the division by B and use the sum of gradients instead of average for update. Of
course you have to compensate B by changing the learning rate. Since B can be
pretty large and can be varying during try-and-error, it is more comfortable to use a
certain average of gradients for update.

Version: 2022.06.25 ISS – FAQ for DL – 52 / 110

 For what tasks it is better to use VAE and for what tasks it is GAN?
Purpose: Explained briefly on slides 9-10 and 9-11. VAE vs. GAN: See slide 9-23 "VAE
vs GAN"

Version: 2022.06.25 ISS – FAQ for DL – 53 / 110

 Is the output in VAE is controllable? Since it seems like we could control it by
using the joint distribution map of latent variable and true number just like in
E9.5, but we still draw random inputs from standard Gaussian without these
maps by inference right? So is it more similar as GAN or cGAN by inference?
■ VAE is indeed more silimar to GAN/cGAN than to AE. It is generative and is able
to generate “realistic” outputs upon random input, see E9.5.
■ VAE generates in a random way based on the underlying distribution. So its output
is not controllable in a deterministic way. In E9.5, by sampling the latent space
(z1,z2) in a certain region, the learned VAE will generate images of a certain digit.
But there is no control to generate a specific digit image.

Version: 2022.06.25 ISS – FAQ for DL – 54 / 110

 Could you please explain the difference between transfer learning and few-shot
learning? From the lecture, they use pre-trained models and then use this training
to detect similar test samples.
Very roughly, transfer learning is a name for the task to transfer a learned DNN from
one task (domain) to another. There are many different methods of transfer learning:
■ Fine tuning is the simplest one. You take a well trained DNN for task A, keep the
first layers for feature extraction and train the last layer for task B.
■ Few-shot learning is a method of or requirement for transfer learning in the sense
that only a few labeled samples for task B is required. This is possible if task A and
B are not too different and the feature extractor of task A is still useful for task B.

Version: 2022.06.25 ISS – FAQ for DL – 55 / 110

 Does it mean that in few-shot learning, given that task A and task B are similar,
if the NN is already trained for task A we do not train it again(not even the last
layers) for task B?
No, just like fine tuning, we retrain the trained DNN or parts of it using a few labeled
samples from task B.

Version: 2022.06.25 ISS – FAQ for DL – 56 / 110

 [Exam task 3.2] How do we get from “(X − µ1 + µ1 − µ2 )2 ” to “σ12 + (µ1 − µ2 )2 ”?

It should be noted that we are calculating the expectation. A more detailed calculation
is attached below.

Version: 2022.06.25 ISS – FAQ for DL – 57 / 110

 Table of Contents

Organizations

Lectures

Programming Practice

Assignments

Version: 2022.06.25 ISS – FAQ for DL – 58 / 110

 (1/2) [Understanding of the definition of n-dimensional datasets] In exercise 2.2
for example, it is assumed that the dataset contains all x and all y. Since the
dataset is defined to be one-dimensional, it is further assumed all x and all y need
to be stored in one common vector. But in the available template it seems as x
and y both have their own one-dimensional vector. But how can the dataset (a
combination of both?) be one-dimensional then?
In 2.2, xi refers to the input data (for a network or a certain machine learning
algorithm) and xi is scalar (one-dimensional), while yi is the corresponding ground
truth (label). Specifically, in 2.2, italic is used as the font for xi (this also means that
xi is scalar). On the contrary, in 2.4, xi is in bold, meaning that each xi is a vector.
In 2.4, 2D means that each xi is a 2-dimensional vector as xi = [xi,1 , xi,2 ]T . In these
three practices, the the number of dimensions denotes how many features a sample
has, when we talk about the data dimension.

Version: 2022.06.25 ISS – FAQ for DL – 59 / 110

 (2/2) [Understanding of the definition of n-dimensional datasets] In exercise 2.2
for example, it is assumed that the dataset contains all x and all y. Since the
dataset is defined to be one-dimensional, it is further assumed all x and all y need
to be stored in one common vector. But in the available template it seems as x
and y both have their own one-dimensional vector. But how can the dataset (a
combination of both?) be one-dimensional then?
Attention! The meaning of the term dimension above is different from the description
for vectors/matrices/tensors. We often say: voice signals are 1D; black-white images
are 2D; MRI images are 3D ⇒ we describe the dimension in a spatial perspective (see
the lecture slide 1-1). However, this dimension (1D/2D/3D) does not tell us how many
features (e.g. pixels in an image) are there in an input sample.

Version: 2022.06.25 ISS – FAQ for DL – 60 / 110

 Does exercise 2.4 refer to the 2-norm or the square of the 2-norm?
In 2.4, the notation is ||xi ||2 and it denotes the squared Euclidean norm (2-norm).
(Note: in many papers/books, we often omit the subscript 2 for Euclidean norm.)

Version: 2022.06.25 ISS – FAQ for DL – 61 / 110

 In the solution that 2.4 ||xi ||2 is written in code as [Link]([Link](X),
axis=-1). I have 2 questions here: 1) why is it using [Link] instead of
[Link]? Isn’t [Link] used as calculating norms of a vector?
2) why is it using axis = -1 instead of 1? They seem to produce the same
result.
Actually, there is a saying that can answer both questions: All roads lead to Rome.
What we provide is a sample solution and there are many other possibilities that can
solve these easy programming tasks.
Q1: You can also use [Link] but you have to additionally pay attention that
a squared 2-norm was required in 2.4.
Q2: In this case, you can also use axis=1. In NumPy, axis=-1 denotes the only axis
which is not explicitly specified. You can refer to the NumPy quickstart for more
information.

Version: 2022.06.25 ISS – FAQ for DL – 62 / 110

 [Programming Practice I] Q1: if we are considering 2 or n features from input, it
means from a sampled point of normal distribution curve we are taking 2
magnitudes. But in general, a sampled point in normal distribution will have only
one magnitude. So in that case, we can draw only one feature. Practically, if we
plot a bell curve, there will be only one magnitude for say mean = 0 and so on.
Please correct me if am wrong.
Your first statement is not clear to me: If we are considering 2 or n features from input,
it means from a sampled point of normal distribution curve we are taking 2 magnitudes.
But I guess, you might have some misunderstanding the term “feature”. As an easy
way, you can take a look at the API documents of [Link] with the link. .
In this task, given a 2D data point x = [x1 , x2 ], both features x1 and x2 are
independent of each other. Then, each feature is sampled from a standard normal
distribution.

Version: 2022.06.25 ISS – FAQ for DL – 63 / 110

 [Programming Practice II] In the formula for y_hat, the sigmoid activation
function is applied to the input in order to get the hidden layers variables (output
of the first layer) but why no activation function is applied to the hidden layer to
get the final output (y_hat)?
The network here is defined for a regression task (see 2.8) and a common activation of
the output layer for a regression is the identity (also known as linear activation). You
can also refer to the lecture slide 4-9 for a short summary for different activation
functions.

Version: 2022.06.25 ISS – FAQ for DL – 64 / 110

 In sample solution, dimension of -1 is used in a lot of places. Like axis=-1 and
[Link](1, -1). What does -1 do when we use it as dimensions?
If you use -1 in the reshape functions, the actual value of this dimension can be
automatically calculated by NumPy. It is an easier way to deal with multi-dimensional
arrays. You can also refer to the link for more information. For example, if you have an
array x with 10 elements and want to reshape this into a 2 × 5 array, you can of course
use [Link](2, 5). However, you can also use [Link](2, -1) and NumPy
can automatically know the exact shape value for the second axis (the -1 axis).

Version: 2022.06.25 ISS – FAQ for DL – 65 / 110

 In 2.5, the factor 0.5 is used for easier calculation of the derivative. But when we
divide it by the number of samples as in the lecture (factor N1 ), we get a bad
approximation (see attached plot). Why is this the case?

see next slide

Version: 2022.06.25 ISS – FAQ for DL – 66 / 110

 Dividing by 2 and by N are two different things.
■ The original MSE is something like (y − ŷ)2 . If you calculate the derivative w.r.t ŷ,
you obtain −2(y − ŷ). This “2” in the derivative looks “ugly” and by adding a 0.5
before the original MSE we can remove this 2 in the derivative. From the
perspective of optimization, there is almost no differences between with and
without the factor of 0.5 before MSE. On the other hand, the gradient calculated
during each iteration is different when using with or without this factor. That is
why you got different results from the sample solution (I guess you kept all other
settings such as learning rate and the number of neurons).
■ The factor N1 means that you calculate the gradient over the entire batch. This is
related to the fact that you use batch gradient descent algorithm to optimize the
model. Actually, this step is already done in 2.8 of the sample solution. Therefore,
what you have done was actually dividing the factor of N1 twice during each training
epoch.

Version: 2022.06.25 ISS – FAQ for DL – 67 / 110

 In the sample solution, there is the following: 0.5 *sum([Link](y_true -
y_pred), axis=-1), while y_true, y_pred are both of the shape (100,1), the
sum() over rows makes none sense, which means it only make a change of the
shape of the total function to (1,100). When testing the codes without sum(),
just return 0.5 * [Link](y_true - y_pred), the codes also works well
with no errors.
It is right if we do not specify axis=-1 in this practice, because each sample is a
scalar. In common cases, samples are of higher dimensions. Therefore, axis=-1 is
more general. This was actually to emphasize that we should sum up the values along
the feature-axis. Using axis=-1 does not result in a shape of (1, 100) but a shape of
(100,). Therefore, the sample solution also works. You can specify keepdims=True to
guarantee a shape of (100, 1) but it is not necessary.

Version: 2022.06.25 ISS – FAQ for DL – 68 / 110

 Why are we doing regularization in programming practices because we are not
overfitting (according to slide 6-2). In 2.5 we cannot see any improvement by
regularization.
The programming practices aim to let students to have an opportunity to understand
the theory the learned from the lecture in a practical way. Furthermore, here the
regularization with different factors just give us an overview how this term can affect
the training procedure and the final prediction.

Version: 2022.06.25 ISS – FAQ for DL – 69 / 110

 Table of Contents

Organizations

Lectures

Programming Practice

Assignments

Version: 2022.06.25 ISS – FAQ for DL – 70 / 110

 How will the solution evaluated, i.e. is it important to write always along the
rules of clean code, and is it important to have a very smart or compact solution?
Or does it only count that we come to a working solution?
It is important that the students have DONE the four assignments if they want to
apply for the DL-lab in the winter term. Many students might be beginners to Python,
so smart or compact solutions are not expected. In other words, a working solution to
the assignments is already enough.

Version: 2022.06.25 ISS – FAQ for DL – 71 / 110

 How does the batch work? For example, in the first assignment simple regression:
if we do not use minibatch, the input of our neural network is ‘x’ , a scalar, and
the output ’y_pred’ is also a scalar. But when we set the batch size is 8, then is
the input of our network still a scalar ‘x’, or a vector [x1 , ...x8 ]? 1) if the input is
still ‘x’, what is the use of batch? 2) if the input is vector [x1 , ...x8 ], the shape of
W adapt to the input vector, and we can’t apply our Neural Network on the
scalar input. How to solve this problem?
(1/2) For understanding how batch operations work you have to think about how the
data is typically stored in tensors for processing. Contrary to the notation that is often
used in the literature, the input examples x are typically stored as row vectors in a
matrix X. This means that if you have 10 input examples each being a vector of length
784, they will be stored in a matrix X of size 10 × 784.

Version: 2022.06.25 ISS – FAQ for DL – 72 / 110

 (2/2) If you want to implement a dense layer with 100 neurons, you then have to apply
an affine mapping followed by the activation function a(). For the implementation, you
input the matrix X with size 10 × 784 into the layer and as an output you want a
matrix Y with size 10 × 100. The output of the layer is then calculated as
Y = a(XW + B). In order for this operation to be defined the shapes of the matrices
have to match. Here is the same operation with the shapes of the operands in brackets:
Y [10 × 100] = a(X[10 × 784]W [784 × 100] + B[10 × 100]). As you can see the shapes
match and the overall function is defined. This might be confusing to you, since we are
using row instead of column vectors but this allows for the batch dimension to be the
first dimension and as far as I know all major DL frameworks implement layers that
expect the first dimension to be the batch dimension. Hint: In Numpy and Tensorflow
broadcasting is available, so if you add the bias you do not have to use a matrix B of
size 10 × 100. You can just add a vector of size 100. So to summarize: You just have
to work with row vectors and not with column vectors.

Version: 2022.06.25 ISS – FAQ for DL – 73 / 110

 Regarding Assignment I-A: What is the result of our model suppose to be? The
model needs one single input value to calculate one single output value and
during the training we give the model a number of 600 input data as single value
each to calculate 600 predictions/single outputs. Therefore, how to calculate the
y_pred without a loop?
see next slide

Version: 2022.06.25 ISS – FAQ for DL – 74 / 110

 The model should output one scalar output for every scalar input. But usually we want
to process a whole batch of data. A batch of 600 examples is then represented by a
data matrix X with shape (600, 1). So each datapoint is represented by one row of the
data matrix. If the first layer of your network then has 10 neurons, you will get an
activation a of this layer that has shape (600, 10), so your weight matrix needs to have
shape (1, 10) and the bias needs to have shape (1, 10). This is necessary for the affine
transformation X*W+b=a. The same idea holds true for every layer until your output
layer which should give a result with shape (600, 1). Note that the order of the data
and weight matrix is flipped when compared with the lecture and many textbooks. But
since in many DL frameworks the batch dimension is always the first dimension, this
flipped order is necessary for the computations to be defined.

Version: 2022.06.25 ISS – FAQ for DL – 75 / 110

 [Assignment I-A] In the regularized_train_step, when we define the l2-loss in this
function, why the l2-norm is the squared sum of all train variables? In the lecture
it is said that we only add the squared sum of weight, not bias.
This is simpler to implement. Otherwise you would have to go through all the variables
and only pick out the matrices and ignore the biases.

Version: 2022.06.25 ISS – FAQ for DL – 76 / 110

 (1/2) [Assignment I-A] 1) For Regularisation, how to recognise the overfitting
and recognise the improvement after regularisation. With regularisation, the
result is even worse because the amplitude is bigger than before (see attached
screenshots in the next slide). And when doing training without regularisation,
the loss in training data set is decreased, but the loss in validation data set is
constant. It is different from the comments. Is it because of the precision of
output of float number? 2) If the initial values of the weights are the same and
using the same architecture of neuronal network, the weights after training are
totally different. Why?

Version: 2022.06.25 ISS – FAQ for DL – 77 / 110

 (2/2) attached screenshots:

(a) no regularization (b) regularization

see next slide

Version: 2022.06.25 ISS – FAQ for DL – 78 / 110

 ■ The overfitting is correctly reproduced by your network. It follows the training data
exactly and shows oscillation between the points. It is expected that the weights
can differ significantly, since there are weight symmetries, as you have learned in the
lecture, and SGD is inherently random. This leads to different weights almost every
time you train the network.
■ It seem that your regularization is not working correctly. If it would the oscillations
should vanish. Have a look again at you code for computing the regularization.
Hint: tf.reduce_sum expects its input to be a tensor, not a list of tensors. Here is
a link to the documentation.

Version: 2022.06.25 ISS – FAQ for DL – 79 / 110

 [Assignment I-A] Do we have to feed the data explicitly to the model or does it
get its data from some hidden mechanism?
When you are going over the iterator of the training data set (the for loop) you get one
batch in each iteration from it. This batch has to be fed to the model explicitly by
calling it, i.e. either with mdl(x) or mdl.__call__(x).

Version: 2022.06.25 ISS – FAQ for DL – 80 / 110

 [Assignment I-A] 1) Should the weights be initialized using [Link]()
or do we need to use some other method? 2) The epoch completion condition
was set as batch_size * train_iters == N_train_samples. Is this correct?
■ For the weights you can use [Link]() as an initial value. The biases
are normally initialized with [Link]().
■ Yes this is correct, since every iteration processes one batch and the number of
examples in a batch is the batch size.

Version: 2022.06.25 ISS – FAQ for DL – 81 / 110

 [Assignment I-A] Towards the end of the assignment that it is mentioned that the
oscillations of the predicted regularized model should not have oscillations
especially in the beginning of the curve. However what is observed is that indeed
the frequency of those oscillations have decreased but one the oscillations has an
amplitude higher than the unregularized model which is strange. So is this normal
or is there something wrong with the implementation and predictions? Also when
increasing the number of epochs to 20000 instead of only 1000 the oscillations
vanished and the regularized model was actually quite good. Is it possible to tune
the number of epochs or is it not allowed to change it?
see next slide

Version: 2022.06.25 ISS – FAQ for DL – 82 / 110

 The regularization implemented seems to be working. You can tune the strength of the
regularization until the oscillations vanish more or less completely. But since this
example is rather artificial, you will see that the model will not be able to follow the
true curve anymore. This is especially true for the last part of the interval. you can of
course also tune the number of epochs.
Version: 2022.06.25 ISS – FAQ for DL – 83 / 110
 Hhow epoch works in detail?
An epoch is completed if you have iterated over the complete training data. Then the
complete training data set is reshuffled again and a new epoch started again. There is
no such thing as a formula for regularization. Regularization can be implemented in
many different ways. Examples are weight or activation regularization or drop out.

Version: 2022.06.25 ISS – FAQ for DL – 84 / 110

 At the [Link]() the model shows at every layer the output-shape
“multiple”. Is this normal for this problem or should the shape be explicit in the
form of (X, X, X)? Because an error occured during the training, so maybe the
problem was in implementation. The error during the training is: “unsupported
format string passed to [Link].__format__”
It is not a problem if the summary shows shapes with multiple. But this does not mean
that there still could be a problem with your model. Please look again.

Version: 2022.06.25 ISS – FAQ for DL – 85 / 110

 When feeding images to convolutional layers, it shows up an error message saying
that the dimensions from the flatten to the conv1 layer don’t fit. (“Input 0 of
layer conv2d_1 is incompatible with the layer: expected ndim=4, found ndim=2.
Full shape received: [64, 120000]”). Why is this?
Convolutional layers expect images as inputs. This is also reflected in the error message
saying that the layers expects ndim=4, i.e. a tensor of shape [batch_size, height,
width, channels]. A flatten layer is only necessary at the transition from convolutional
to dense layers. So just remove the first flatten layer and pass the images directly into
the convolutional layers.

Version: 2022.06.25 ISS – FAQ for DL – 86 / 110

 [Assignment I-B] The test MAE=7.6, which seems reasonable? However when
with family pictures, the estimate is wrong by roughly +/ − 25 years. For my
tests I even recreated the “style” of the pictures of the dataset (usually zoomed in
so that the face fills the pic, looking into the camera, pictures originally taken in
1 : 1 ratio etc). What do you think about these results?
The MAE of 7.6 sounds correct. You can also verify that the model is working by
predicting on samples of the test dataset. As the faces in the UTK faces data set were
preprocessed by cropping and centering the face in the middel of the image, you can
only expect reasonable results if you use the same preprocessing. Unfortunately they do
not provide a preprocessing algorithm with the dataset so you can not easily apply their
preprocessing to your own images. But you can look in their original paper on what
they used for preprocessing. Besides preprocessing there are also other factors like an
unbalance of ethnicity in the dataset that can influence the generalization of the model.

Version: 2022.06.25 ISS – FAQ for DL – 87 / 110

 [Assignment I-B] The test loss is around 7.2 but when extracting the accuracy of
the model, a test accuracy around 0.03 can be obtained.
Accuracy is not applicable to this problem, since it is not classification that we are
doing. We are doing regression. The test loss seems reasonably good and the model
seems to be making good predictions on it. A possible explanation for this is that the
preprocessing is missing and therefore the predictions are off. Even if you think that
you have centered and cropped your face correctly it might still be different than what
the preprocessing applied to the UTK-Faces data set would have done.

Version: 2022.06.25 ISS – FAQ for DL – 88 / 110

 [Assignment I-B] The first epoch works just fine, it also prints the epoch, the
training loss and the validation loss. But when starting the second epoch the first
call of the train_step function gets the last batch, which consists only of 32
images instead of 64 as 20000/64 = 312.5. This results in an error. train_ds =
train_ds.repeat() was added to indefinitely repeat the training data set but
this doesn’t seem to resolve the issue.
Normally the model should be able to work with varying batch sizes. If it does not, this
might hint to a problem in the model definition. If you cannot fix it, you can also use
the train_ds.batch(batch_size, drop_remainder=True). This will drop the last
part of the data set that does not fit in a complete batch.

Version: 2022.06.25 ISS – FAQ for DL – 89 / 110

 [Assignment I-B] There was a TypeError in a code line: When running the Test
model segment the first code line: Num_test_batches = int((len(files) -
N_training_examples - N_validation_examples)/batch_size) produces
the error “object of type ‘module’ has no len()”.
There seems to be something wrong with the list files. After reading the files with the
load_file_names() function files should be an object of type list and not a module. So
please check if you are somewhere in your code are modifying the list files.

Version: 2022.06.25 ISS – FAQ for DL – 90 / 110

 [Assignment I-B] 1) When training the model and trying to print the
val_loss/val_iters an error occurs. Which makes sense because val_loss is a
tensor with 64 and val_iters a tensor with 4 values. What is wrong with this? 2)
When testing the model on the test data, the test_loss cannot be computed
because of incompatible shapes ([64] and [60]).
■ Both val_loss and val_iters should be scalars and not tensors. You want to
accumulate the validation loss in the variable val_loss and count the number of
validation iterations in val_iters.
■ There seems to be something wrong with your model. Normally, if you input a
batch of size 60 you would expect the model to output a prediction for each of
these 60 images. So the output tensor should have a shape like [60,] or [60, 1].

Version: 2022.06.25 ISS – FAQ for DL – 91 / 110

 In the training dataset, we give the command to repeat the training samples
indefinitely. If we repeat indefinitely, then how does the program go to the next
step? shouldn’t it keep looping/ working on the infinitely long data set and never
be able to execute the next line of code?
Yes, applying the [Link]() transformation with no arguments will repeat
the input indefinitely. However, by each training step we consume only one batch of
samples. And we can use “if” statement to check whether all entries have already been
read from the whole dataset. If so, we can start a new epoch and when the number of
epochs reaches our default setting, we can break out the training loop. You might refer
to part “Training workflows” on this page for more information about repeat().

Version: 2022.06.25 ISS – FAQ for DL – 92 / 110

 When calling the train_step function I get the following error: "AttributeError:
’[Link]’ object has no attribute
’_in_graph_mode’".How to fix this?
This error usually happens when you try to update a constant tensor. You might have
initiated weights and biases as [Link]() instead of [Link]() inside
definition of MyModel()?

Version: 2022.06.25 ISS – FAQ for DL – 93 / 110

 When trying to run the training in Assignment I-B, we get the following error:
"Unknown image file format. One of JPEG, PNG, GIF, BMP required. [[node
DecodeJpeg]]". What could be the reason for this error?
As the error implies, your input format for the node DecodeJpeg is invalid. The error, if
it happens during training, might be related to the parser function “parse_func” which
is responsible for the data input pipeline.

Version: 2022.06.25 ISS – FAQ for DL – 94 / 110

 Am I allowed to import other python packages such as scikit-learn or are we
restricted to the imports done in the first cell?
Yes, sure. You can use other packages, although we did not really see a necessity of
using additional packages.

Version: 2022.06.25 ISS – FAQ for DL – 95 / 110

 In the previous solution of Assignment I-B and I have a small doubt. What is the
significance of the “outer” tf.reduce_sum in the highlighted area of the image?

The inner tf.reduce_sum([Link]()) computes the squared sum of one single

trainable variable, for example, for a weight matrix, it first takes the square of each
element in this matrix and then sum them up. The argument of the outer
tf.reduce_sum is a list, which stores the squared sum of each trainable variable in this
model, using this outer tf.reduce_sum we can get the regularization term.

Version: 2022.06.25 ISS – FAQ for DL – 96 / 110

 The training process of MobileNetV2 takes quite a long time (about 30 minutes
for each epoch), during which time the Colab always went disconnected, because
Colab reminds that we shouldn’t occupy its resource for such a long time
(Perhaps part of the reasons depends on the GPU capacity of my Notebook).
What should we do against this problem? (Except for an upgrade to Colab pro or
using other cloud servers)

see next slide

Version: 2022.06.25 ISS – FAQ for DL – 97 / 110
 This might be a bottleneck between Google Drive and Colab. Loading large-scale data
from Google Drive is slow. Hence, a temporary solution for you might be this: You can
upload the .[Link] file into the content directory of your Colab page and directly upzip
the file there. In this case, the access to the data would be much faster and each epoch
may take only one or two minutes. (One more hint is that the time consumption of the
first epoch is typically longer due to compiling models in Keras.)

Version: 2022.06.25 ISS – FAQ for DL – 98 / 110

 [Assignment II] Since val_split = 0.1 , number of samples = 9144, the number of
validation samples should be around 914, not 866. Is there something wrong? or
866 does not mean the number of validation sample?
It seems like Keras is doing the split not exactly. It seems like this behavior is caused by
Keras.

Version: 2022.06.25 ISS – FAQ for DL – 99 / 110

 [Assignment II] During fine tuning of the base model, do we have to fine tune the
full base model or any specific number of top layers?
As explained in the text, we first want to fine tune the top layer that we appended to
the MobileNetV2. Otherwise the high magnitude gradients caused by its random
initialization might effect the MobileNetV2 in a bad way. When the top layer was
trained for some epochs, we reduce the learning rate and fine tune the complete model.

Version: 2022.06.25 ISS – FAQ for DL – 100 / 110

 [Assignment II] After defining my base_model without the top layer and with the
imagenet as weights and after defining MyTransferModel, it is to assign it to
tf_mdl as such: tf_mdl =
MyTransferModel(pretrained_model=base_model()) but then this error
pops up “ValueError: The first argument to ‘[Link]‘ must always be passed.”
Any hints on where the problem is?
You are supposed to pass the object of the base model to the transfer model. What
you are doing however, is to pass the result of a call of the base model to the transfer
model. You should remove the brackets and pass the object and the the call result, i.e.
tf_mdl = MyTransferModel(pretrained_model=base_model).

Version: 2022.06.25 ISS – FAQ for DL – 101 / 110

 [Assignment II] As long as only the new layers are trained
(base_model.trainable = False), the accuracy develops as expected. As
soon as fine tuning is then started (base_model.trainable = True), the
accuracy drops to a low value at the beginning instead of continuing the training
with the same value. 1) Even if this is unlikely: is this the desired behaviour? 2)
Randomly comparing the weighting factors of individual layers after training only
the new layers and before fine tuning the complete model and they were identical.
What else could cause the drop in accuracy?

(c) Question (d) Answer

see next slide

Version: 2022.06.25 ISS – FAQ for DL – 102 / 110

 A slight drop in accuracy is to be expected, since when freezing the weights of the base
model and training only the last layer there is limited flexibility. But when we unfreeze
the model and train all layers the gradients can suddenly be back propagated into the
network and cause changes in deeper layers. This might lead to a small decrease in
accuracy. But it should quickly rise again and surpass the accuracy of the model with a
frozen base model. Besides the weights there are also batch normalization layers that
learn some scaling and additive constants. These layers are very important for the
performance of the model and if they are changed they can cause performance decrease
even when all weights are identical. The drop in accuracy that you are seeing is,
however, too large. It is attached the plot from the solution. Make sure that you are
training with a much smaller learning rate after you unfreeze the base model and check
the overall model and data pipeline.

Version: 2022.06.25 ISS – FAQ for DL – 103 / 110

 In the document we can always find some “same” losses, for example the
BinaryCrossentropy and the binary_crossentropy. Principally from the
naming they should be same thing, but there must be some reasons for their
existence. And by changing one of them into another to calculate the loss I do
get totally different results. So what is the difference between them?
The BinaryCrossentropy is a class and binary_crossentropy is a function. You
can not simply replace [Link].binary_crossentropy(y_true, y_pred)
calls with [Link](y_true, y_pred) and expect
the same outcome. In the case of
[Link].binary_crossentropy(y_true, y_pred) you are calling a
function that computes the binary crossentropy loss, which is what you want. In the
case of [Link](y_true, y_pred), however, you
are calling the constructor of the class BinaryCrossentropy with wrong arguments.
Have a look into their corresponding pages in the API reference for more detailed
information on how to use them.

Version: 2022.06.25 ISS – FAQ for DL – 104 / 110

 There is a hint in the Transfer Learning part saying that we should specify
steps_per_epoch = int((1.0 - val_split) * N_samples_Caltech101 /
tf_batch_size) as well as a suitable number of validation_steps. 1) Is this
really necessary? The model is fitted without specifying it: tf_history_1 =
tf_mdl.fit(x=train_gen, epochs=tf_epochs,
validation_data=val_gen) and it still works. The necessary parameters seem
to be inferred directly, unless we would repeat the generator indefinitely.
Yes. Tensorflow seems to be able to derive the necessary data from the generator.
Normally the variable steps_per_epoch has to be specified for an infinite long data
set, so that the fit function knows when to stop. But with the generator the fit
function seems to be able to derive the contained images through the directory and the
batch size.

Version: 2022.06.25 ISS – FAQ for DL – 105 / 110

 Google colab always disconnects and then needs to re-run the program/train the
model. Is there any way to keep connected?
You could try the following steps:
1. Right-click on the connect button (on the top-right side of the colab)
2. Click on inspect
3. Get the HTML id of the button and substitute in the following code
[Link](“Put ID here”).click(). The rest of the code
remains unchanged.

Version: 2022.06.25 ISS – FAQ for DL – 106 / 110

 Regarding question 4 of Assignment II: “How could you quantify how similar two
domains are?” Which domain is about?
This question is rather general and refers to the domains in a transfer learning setting
described in Section 4 of the task description file.

Version: 2022.06.25 ISS – FAQ for DL – 107 / 110

 The likelihood is is defined as p(x|y), where x is the input and y the output (e.g.
the class). In the task description however the likelihood is defined as p(y|x)
which seem to be the posterior. Is it right?
■ The likelihood measures the quality of a fit of a model p(y|x), which depends on
the parameters of the DNN, to the data. In this context the random variables are
given and fixed, like our dataset, and the likelihood depends only on the parameters
of our model. We then typically want to maximize the likelihood by optimizing the
parameters.
■ A posterior, on the other hand, arises in Bayesian statistics when you have some
prior belief and use it in combination with the evidence and a likelihood to form
p(y|x).
■ In general p(y|x) is just a conditional probability. Not every conditional probability
can be classified into a posterior or likelihood. The context in which it is used
determines how it is called. Since in the context of this assignment we are
maximizing the probability of some observed data via the parameters of the DNN,
we can call p(y|x) a likelihood.
Version: 2022.06.25 ISS – FAQ for DL – 108 / 110
 How to call and use the pretrained model MobileNetV2?
You can just load the pretrained model using [Link].MobileNetV2(),
it will return a [Link]() instance. There are a lot of arguments in this API,
please refer to this for more information. When fine-tuning the model, please set
include_top=False. Put the pretrained model as an additional argument in
__init__ function and call it normally as calling a layer. You can also check tensorflow
official website for a detailed tutorial about transfer learning.

Version: 2022.06.25 ISS – FAQ for DL – 109 / 110

 After computation on the CPU for 3 epochs the connection is lost, because the
computation takes too long (∼ 11 min/epoch). How to tackle this problem of no
GPU support.
You can turn on your GPU support by clicking on “Runtime” in the top left menu and
selecting “Change runtime type”. Afterwards, you can choose “GPU” in the drop down
list under “Hardware accelerator”. You can refer to the first assignment part II as well.
If you successfully get support from GPU, each epoch may take only about 15 seconds.

Version: 2022.06.25 ISS – FAQ for DL – 110 / 110