QUESTION – 1: Multiple Choice

For each of the following questions, circle the letter of your choice. There is only ONE
correct choice unless explicitly mentioned. No explanation is required.

(a) Which of the following is true about max-pooling?

(i) It allows a neuron in a network to have information about features in a larger part of the
image, compared to a neuron at the same depth in a network without max pooling.
(ii) It increases the number of parameters when compared to a similar network without max
pooling.
(iii) It increases the sensitivity of the network towards the position of features within an
image.
Solution: (i)

(b) In order to backpropagate through a max-pool layer, you need to pass information about
the positions of the max values from the forward pass.
(i) True
(ii) False
Solution: (i)

(c) Consider a simple convolutional neural network with one convolutional layer. Which of
the following statements is true about this network? (Check all that apply.)
(i) It is scale invariant.
(ii) It is rotation invariant.
(iii) It is translation invariant.
(iv) All of the above.
Solution: (iii)

(d) Mini-batch gradient descent is a better optimizer than full-batch gradient descent to avoid
getting stuck in saddle points.
(i) True
(ii) False
Solution: (i)

(e) Using "neural style transfer", you want to generate an RGB image of the Great Wall of
China that looks like it was painted by Picasso. The size of your image is 100x100x3 and you
are using a pretrained network with 1,000,000 parameters. At every iteration of gradient
descent, how many updates do you perform?
(i) 10,000
(ii) 30,000
(iii) 1,000,000
(iv) 1,030,000
Solution: (ii) You only update the image, i.e. 10,000 pixels and each pixel has 3 channels.

(f) You are building a model to predict the presence (labeled 1) or absence (labeled 0) of a
tumor in a brain scan. The goal is to ultimately deploy the model to help doctors in hospitals.
Which of these two metrics would you choose to use?
(i) Precision = True positive examples / Total predicted positive examples
(ii) Recall = True positive examples / Total positive examples
Solution: (ii) Increase recall, because we don't want false negatives.
(g) You want to map every possible image of size 64x64 to a binary category (cat or non-cat).
Each image has 3 channels and each pixel in each channel can take an integer value between
(and including) 0 and 255. How many bits do you need to represent this mapping?

Solution: (ii)

(h) The mapping from question (g) clearly can not be stored in memory. Instead, you will
build a classifier to do this mapping. Imagine a simple single hidden layer classifier for
classifying images as cat vs non-cat. You use a single hidden layer of size 100 for this task.
Each weight in the W[1] and W[2] matrices can be represented in memory using a float of size
64 bits. How many bits do you need to store your two layer neural network (you may ignore
the biases b[1] and b[2])?

Solution: (ii)

QUESTION – 2: Short Answers

The questions in this section can be answered in 3-4 sentences. Please be short and
concise in your responses.
(a) Consider an input image of shape 500x500x3. You flatten this image and use a fully
connected layer with 100 hidden units.

(i) What is the shape of the weight matrix of this layer?

Solution: Weight matrix which is 750,000 x 100, giving 75 million.
(ii) What is the shape of the corresponding bias vector?
Solution: 100 x 1

(b) Consider an input image of shape 500x500x3. You run this image in a convolutional
layer with 10 filters, of kernel size 5 x 5. How many parameters does this layer have?
Solution: 5x5x3x10 and a bias value for each of the 10 filters, giving 760 parameters.

Solution: The derivative represents how much the output changes when the input is changed.
In other words, how much the input has influenced the output.

(d) Why is it necessary to include non-linearities in a neural network?

Solution: Without nonlinear activation functions, each layer simply performs a linear
mapping of the input to the output of the layer. Because linear functions are closed under
composition, this is equivalent to having a single (linear) layer. Thus, no matter how many
such layers exist, the network can only learn linear functions.
(e) The universal approximation theorem states that a neural network with a single hidden
layer can approximate any continuous function (with some assumptions on the activation).
Give one reason why you would use deep networks with multiple layers.
Solution: While a neural network with a single hidden layer can represent any continuous
function, the size of the hidden layer required to do so is prohibitively large for most
problems. Also, having multiple layers allows the network to represent highly nonlinear (e.g.
more than what a single sigmoid can represent) with fewer number of parameters than a
shallow network can.

(f) You want to use the figure below to explain the concept of early stopping to a friend. Fill-
in the blanks. (1) and (2) describe the axes. (3) and (4) describe values on the vertical and
horizontal axis. (5) and (6) describe the curves. Be precise.

(g) Look at the grayscale image at the top of the collection of images below. Deduce what
type of convolutional filter was used to get each of the lower images. Explain briefly and
include the values of these filters. The filters have a shape of (3,3).

Solution: Left image: Vertical edge detector. Filter: [[1,0,-1][1,0,-1][1,0,-1]] Right image:
Horizontal edge detector. Filter: [[1,1,1][0,0,0][-1,-1,-1]]

(h) When the input is 2-dimensional, you can plot the decision boundary of your neural
network and clearly see if there is overfitting.
How do you check overfitting if the input is 10-dimensional?
Solution: Compute cost function in the validation and training set. If there is a significant
difference, then you have a variance problem.
QUESTION – 3: Loss Functions

(a) Suppose you're given an example image of an iguana. If the model correctly predicts the
resulting probability distribution as
 = (0.25, 0.25, 0.3, 0.2), what is the value of the cross-
entropy loss? You can give an answer in terms of logarithms.
Solution: -log 0.3
(b) After some training, the model now incorrectly predicts mouse with distribution (0.0, 0.0,
0.4, 0.6) for the same image. What is the new value of the cross-entropy loss for this
example?
Solution: -log 0.4
(c) Suprisingly, the model achieves lower loss for a misprediction than for a correct
prediction. Explain what implementation choices led to this phenomenon.
Solution: This is because our objective is to minimize CE-loss, rather than to directly
maximize accuracy. While CE-loss is a reasonable proxy to accuracy, there is no guarantee
that a lower CE loss will lead to higher accuracy.

(d) Given your observation from question (c), you decide to train your neural network with the
accuracy as the objective instead of the cross-entropy loss. Is this a good idea? Give one
reason. Note that the accuracy of a model is defined as
Accuracy = (Number of correctly-classified examples) / (Total number of examples)
Solution: It's difficult to directly optimize the accuracy because
- it depends on the entire training data, making it impossible to use stochastic
gradient descent.
- the classification accuracy of a neural network is not differentiable with respect to
its parameters.

(e) After tuning the model architecture, you find that softmax classifier works well.
Specifically, the last layer of your network computes logits z = (z1, ... , zny ), which are then
fed into the softmax activation. The model achieves 100% accuracy on the training data.
However, you observe that the training loss doesn't quite reach zero. Show why the cross-
entropy loss can never be zero if you are using a softmax activation.
(f) The classifier you trained worked well for a while, but its performance suddenly dropped. It
turns out that the biology lab started producing chimeras (creatures that consist of body parts
of different animals) by combining different animals together. Now each image can have
multiple classes associated with them; for example, it could be a picture of a dog with mouse
whiskers, cat ears and an iguana tail! Propose a way to label new images, where each
example can simultaneously belong to multiple classes.
Solution: Use multi-hot encoding, e.g. (1, 0, 0, 1) would be dog and mouse.

(g) The lab asks you to build a new classifier that will work on chimeras as well as normal
animals. To avoid extra work, you decide to retrain a new model with the same architecture
(softmax output activation with cross-entropy loss). Explain why this is problematic.

(h) Propose a different activation function for the last layer and a loss function that are better
suited for this multi-class labeling task.
Solution: We can formulate this as ny independent logistic regression tasks, each trying to
predict whether the example belongs to the corresponding class or not. Then the loss can
simply be the average of ny logistic losses over all classes.
QUESTION – 4: Batch Normalization

Solution: It prevents division by 0 for features with variance 0.

(b) Give 2 benefits of using a batch normalization layer.

Solution: (i) accelerates learning by reducing covariate shift, decoupling dependence of
layers, and/or allowing for higher learning rates/ deeper networks, (ii) accelerates learning by
normalizing contours of output distribution to be more uniform across dimensions, (iii)
Regularizes by using batch statistics as noisy estimates of the mean and variance for
normalization (reducing likelihood of overfitting), (iv) mitigates poor weights initialization
and/or variability in scale of weights, (v) mitigates vanishing/exploding gradient problems,
(vi) constrains output of each layer to relevant regions of an activation function, and/or
stabilizes optimization process, (vii) mitigates linear discrepancies between
batches, (viii) improves expressiveness of the model by including additional learned
parameters, , producing improved loss.

(c) Explain what would go wrong if the batch normalization layer only applied the first
transformation (znorm).
Solution: Normalizing each input of a layer may change what the layer can represent. For
instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the
nonlinearity. ̃ makes sure that the transformation inserted in the network can represent the
identity transform

Recall that during training time, the batch normalization layer uses the mini-batch statistics
to estimate  and   . However, at test time, it uses the moving averages of the mean and
variance tracked (but not used) during training time.
(d) Why is this approach preferred over using the mini-batch statistics during training and at
test time?
Solution: There were two correct answers:
(1) Moving averages of the mean and variance produce a normalization that's more consistent
with the transformation the network used to learn during training than the mini-batch
statistics. You need to support variable batch sizes at test time, which includes small batch
sizes (as small as a single example). The variability/noisiness between input images means
batches with small batch sizes at test time will be less likely to have the same mini-batch
statistics that produce the normalized activations trained on at training time. Using the moving
averages of mean and variance as estimates of the population statistics addresses this issue.
(i) Mini-batches might be small at test time. (ii) Smaller mini-batches mean the mini-batch
statistics are more likely to differ from the mini-batch statistics used at training. (iii) Moving
averages are better estimates.

(2) Moving averages of the mean and variance produce a consistent normalization for an
example, that's independent of the other examples in the batch. (i) A single example might be
part of different batches at test time. (ii) That example should receive the same prediction at
test time, independent of the other examples in its batch. (iii) Mini-batch statistics vary per
batch but moving averages do not.

Suppose you have the following dataset:

You make the mistake of constructing each mini-batch entirely out of one of the two groups
of data, instead of mixing both groups into each mini-batch. As a result, even though your
training loss is low, when you start using the trained model at test time in your new mobile
app, it performs very poorly.

(e) Explain why inclusion of the batch normalization layer causes a discrepancy between
training error and testing error when mini-batches are constructed with poor mixing.
Solution: First, consider the distributions for the mini-batch statistics  and   over the mini-
batches used during training. If batches were constructed IID, both of the distributions would
be normal centered on the population statistic (for which the moving averages are accurate
estimates). However, since the batches were constructed as described, both the distributions
would be compositions of two normal distributions (i.e. bimodal)|one per group of data.
During training, batches on average get normalized according to a statistic drawn from each
of the distributions; however, at test time, batches get normalized according to the mean of
both of the distributions, which never occurred during training. The following figure
demonstrates this point in one dimension.
QUESTION – 5: Numpy coding
In this question, you will implement a fully-connected network. The architecture is LINEAR
RELU DROPOUT BATCHNORM. This is a dummy architecture that has been
made up for this exercise.
The code below implements the forward propagation, but some parts are missing. You will
need to fill the parts between the tags (START CODE HERE) and (END CODE HERE)
based on the given instructions. Note that we are using only numpy (not tensorflow), and the
relu function has been imported for you.
QUESTION – 6: X-Network
An X-neuron, as opposed to a neuron, takes vectors as input, and outputs vectors. There
are three stages in the forward propagation for an X-neuron.

Consider the following 3-layer X-network, made of X-neurons exclusively.

Solution: D x D

Solution: D x nx

Solution: n[1]

Solution: Dimension D.

Solution: Dimension (D, 1)

QUESTION – 7: Practical case study: Online advertising
Learning and predicting user response plays a crucial role in online advertising.
(a) Let's say you have a website with a lot of traffic. You would like to build a network that
computes the probability of a user clicking on a given advertisement on your website. This is
a supervised learning setting. What dataset do you need to train such a network?

(b) Choose an appropriate cost function for the problem and give the formula of the cost.

(c) Your website sells sport footwear and you have already collected a dataset of 1 million
examples from past visits. Your friend, who works in the high fashion industry, offers to let
you use their dataset as it has similar descriptors. However, you are concerned about the
impact of this different distribution dataset in the performance of your predictive system.
Explain how you would use your friend's dataset.
Solution: Training set. Reasons include:
- Neural networks are very data intensive. Therefore, opportunities to increase the
dataset should not be missed.
- The new data in the training set could help the neural network to learn better lower
level features (as you have more data)
- Using the new data in the validation/test set would change the goal/target of the
optimization process. Thus, the optimized system would not perform well with real
world data.

(d) How would you assess the impact of the new dataset?
Solution: Split the training set in a new training set + validation set (made exclusively of old
training examples). Measure the difference of accuracy between the new training set and
validation set. If there is a significant difference, then the distribution mismatch between the
old and new images is a problem.

(e) Initially, you decide to build a fully connected neural network for the problem. This
baseline model would have L hidden layers, one input layer and an output layer. The number
of neurons in the lth layer is n[l].Write down the number of parameters of this model as a
function of L and n[l] for l = 0...L.
Note: The input layer is considered to be layer number 0

(f) Based on the information provided in the graph below, what type of problem will you
encounter as the number of hidden layer approaches 10? Mention possible solutions to this
problem.
Solution: Overfitting problem. Possible solutions are:
- Regularization: Parameter norm penalties (L2/L1), Dropout.
- Reduce complexity of the neural network (decrease number of neurons and hidden
layers)
- Increase the training dataset. Data Augmentation

QUESTION – 8: The following code describes the forward pass and gradient computations
of a fully connected network with one hidden layer (input: x_mat, output: y_pred):
# First, compute the new predictions `y_pred`
z_2 = [Link](x_mat, W_1)
a_2 = sigmoid(z_2)
z_3 = [Link](a_2, W_2)
y_pred = sigmoid(z_3).reshape((len(x_mat),))

# Now compute the gradient

J_z_3_grad = -y + y_pred
J_W_2_grad = [Link](J_z_3_grad, a_2)
a_2_z_2_grad = sigmoid(z_2)*(1-sigmoid(z_2))
J_W_1_grad = ([Link]((J_z_3_grad).reshape(-1,1), W_2.reshape(-1,1).T)*a_2_z_2_grad).[Link](x_mat).T
gradient = (J_W_1_grad, J_W_2_grad)
a) Write down the equations of forward pass and gradient calculations using matrix notation,
partial derivatives and sigmoid function σ(.).
b) Suppose that another fully connected layer is added at the input as the initial hidden layer,
with weight matrix W_0. Modify the code above to compute y_pred and J_W_0_grad.

Solution:
a)
  ,    ,
   ,
   ,

   
 

 ,    ,    1   
   
  

   
 
b)
# First, compute the new predictions `y_pred`
z_1 = [Link](x_mat, W_0)
a_1 = sigmoid(z_1)
z_2 = [Link](a_1, W_1)
a_2 = sigmoid(z_2)
z_3 = [Link](a_2, W_2)
y_pred = sigmoid(z_3).reshape((len(x_mat),))

# Now compute the gradient

J_z_3_grad = -y + y_pred
J_W_2_grad = [Link](J_z_3_grad, a_2)
a_2_z_2_grad = sigmoid(z_2)*(1-sigmoid(z_2))
temp = ([Link]((J_z_3_grad).reshape(-1,1), W_2.reshape(-1,1).T)*a_2_z_2_grad).T
J_W_1_grad = [Link](a_1).T
a_1_z_1_grad = sigmoid(z_1)*(1-sigmoid(z_1))
J_W_0_grad = ([Link]([Link](-1,1), W_1.reshape(-1,1).T)*a_1_z_1_grad).[Link](x_mat).T
gradient = (J_W_0_grad, J_W_1_grad, J_W_2_grad)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
ADDITIONAL PROBLEMS
1. The tanh activation usually works better than sigmoid activation function for hidden units because
the mean of its output is closer to zero, and so it centers the data better for the next layer.
True/False?
True. The output of the tanh is between -1 and 1, it thus centers the data which makes the learning
simpler for the next layer.

2. You are building a binary classifier for recognizing cucumbers (y=1) vs. watermelons (y=0). Which
activation function would you recommend using for the output layer?
Solution:
Sigmoid outputs a value between 0 and 1 which makes it a very good choice for binary classification. You
can classify as 0 if the output is less than 0.5 and classify as 1 if the output is more than 0.5. It can be done
with tanh as well but it is less convenient as the output is between -1 and 1.

3. Suppose you have built a neural network. You decide to initialize the weights and biases to be zero.
Which of the following statements are True? (Check all that apply)
(i) Each neuron in the first hidden layer will perform the same computation. So even after
multiple iterations of gradient descent each neuron in the layer will be computing the same
thing as other neurons.
(ii) Each neuron in the first hidden layer will perform the same computation in the first iteration.
But after one iteration of gradient descent they will learn to compute different things
because we have “broken symmetry”.
Solution: (i)

4. If your Neural Network model seems to have high variance, what of the following would be promising
things to try?
(i) Add regularization
(ii) Make the Neural Network deeper
(iii) Get more training data
(iv) Increase the number of units in each hidden layer
(v) Get more test data

Solution: (i), (iii)

5. What is weight decay?
(i) A regularization technique (such as L2 regularization) that results in gradient descent
shrinking the weights on every iteration.
(ii) Gradual corruption of the weights in the neural network if it is trained on noisy data.
(iii) The process of gradually decreasing the learning rate during training.
(iv) A technique to avoid vanishing gradient by imposing a ceiling on the values of the weights.
Solution: (i)

6.

(i)

(ii)

(iii)

(iv)

Solution: (iv)

7. Consider this figure:

These plots were generated with gradient descent; with gradient descent with momentum ( β= 0.5) and
gradient descent with momentum (β = 0.9). Which curve corresponds to which algorithm?

Solution: (1) is gradient descent. (2) is gradient descent with momentum (small β). (3) is gradient descent
with momentum (large β)

8. Suppose batch gradient descent in a deep network is taking excessively long to find a value of the
parameters that achieves a small value for the cost function . Which of the
 following techniques could help find parameter values that attain a small value for cost function?
(Check all that apply)
(i) Try better random initialization for the weights
(ii) Try mini-batch gradient descent
(iii) Try using Adam
(iv) Try initializing all the weights to zero
(v) Try tuning the learning rate α
Solution: (i), (ii), (iii), (v)

9. A convolutional neural network has 4 consecutive 3x3 convolutional layers with

stride 1 and no pooling. How large is the support of (the set of image pixels which activate) a
neuron in the 4th non-image layer of this network?
Solution: With a stride of 1, and a 3x3 filter, and no pooling, this means the “outer ring” of
the image gets chopped off each time this is applied. Hence, this reduces the dimension from
(nxn) to ((n-2)x(n-2)). We get, working backwards:
1x1 <- 3x3 <- 5x5 <- 7x7 <- 9x9
Thus, the support is 9x9=81 pixels.

10. ConvNet basics: A 1D convolutional net has an input of size p x n (i.e. p features of size n
each). The firrst layer is a convolutional layer with f output feature maps and kernels of
size k that connect all p input feature maps to all f output feature maps.
(a) give a formula for the size of the output feature maps m?
(b) give a formula for the number of parameters (independent weights) in this layer (not
counting biases)?
(c) give a formula for the number of multiply-accumulate operations to do a forward
propagate (not counting biases)?
(d) We decide to use a convolutional layer with stride s. Give a formula for the size of the
output feature map

11. We decided to enter the ImageWeb competition in which we are given 10 million labeled
images, each containing one roughly-centered object, and labeled with a single category
among 10,000 possible categories. We will use a large convolutional network for this.
(a) name and describe the four or five basic module types that must be assembled to
construct a convolutional net.
(b) each input image is 200 by 200 pixels. Design a typical convolutional net architecture that
could be applied to these images. Give the number of layers/stages, the sizes of the kernels,
number of feature maps, pooling sizes, pooling strides, etc.
(c) what loss function do we typically use with neural nets for classification tasks?

12. Someone gives you a large dataset with 10 million vectors of size 1000. One half of the
features seems distributed according to a normal (Gaussian) distribution of various means
and standard deviations, while the other half seems to take only positive values and seems
distributed according to a log-normal distribution, i.e. a distribution over x such that log x is
Gaussian of various means and standard deviations.
(a) why can it be a problem to have features with widely varying means and standard
deviations?
(b) how would you pre-process the normal-distributed features?
(c) how would you pre-process the log-normal-distributed features?