DATA7703 Week 8 Tutorial Solutions
2025 Semester 2
1. (a) The network has two layers, one hidden layer, 5 input neurons, 7 hidden neurons,
and 2 output neurons.
(b) The plots are shown below.
The range of 𝜎(𝑢) is (0, +1), and that for (𝑢)+ is [0, +∞).
(c) Since all the parameters have value 0, the weighted input sum for both output
neurons are 0, and the output values are both 𝜎(0) = 1+𝑒1−0 = 0.5.
(d) i. The neural net predicts the class with the highest score, thus for the 1st example,
it predicts class 2, as the scores for class 1 and 2 are -2 and 3 respectively. Simi-
larly, the neural net predicts 1 and 2 for the 2nd and 3rd examples respectively.
Since the classes for the three examples are 1, 1, 2 respectively, the accuracy is
2/3.
ii. Given class scores 𝑜𝑖1 and 𝑜𝑖2 , we can turn them into a class distribution
𝑒𝑜𝑖1 𝑒𝑜𝑖2
(︂ )︂
(𝑝𝑖1 , 𝑝𝑖2 ) = , ,
𝑒𝑜𝑖1 + 𝑒𝑜𝑖2 𝑒𝑜𝑖1 + 𝑒𝑜𝑖2
where the first number is the probability of class 1, and second the probability
of class 2.
The log-loss of the dataset is thus
𝑒𝑜11 𝑒𝑜21 𝑒𝑜32
(︂ )︂
1
− ln 𝑜11 − ln 𝑜21 − ln 𝑜31
3 𝑒 + 𝑒𝑜12 𝑒 + 𝑒𝑜22 𝑒 + 𝑒𝑜32
𝑒−2 𝑒2 𝑒3
(︂ )︂
1
= − ln −2 − ln 2 − ln 0
3 𝑒 + 𝑒3 𝑒 + 𝑒−3 𝑒 + 𝑒3
= 1.687
1
(e) No. Typically, if there are 𝐶 classes, we need to have 𝐶 output neurons (when 𝐶 = 2,
we can use just one output neuron as described in lecture). Thus we need to add
one more output neuron to the MLP.
(f ) Strictly speaking, no, because we only need one output neuron. However, you can
also just ignore one of the output neuron when training the network and making
predictions — this is similar to training some of the neurons in our brain to solve a
problem. Any well-justified answer is acceptable.
(g) No. The net income can be a negative number, but with ReLU output unit, the
network is only able to predict non-negative values.
(h) Yes, we can represent the function as an MLP that has an input layer with 𝑑 input
neurons, a single hidden layer with 𝑛1 ReLU hidden neurons, and an output layer
with 𝑛2 sigmoid output neurons.
For 1 ≤ 𝑖 ≤ 𝑛1 and 1 ≤ 𝑗 ≤ 𝑑, the 𝑖-th hidden neuron is connected to the 𝑗-th input
neuron with a weight 𝑤1,𝑖𝑗 , that is, the (𝑖, 𝑗)-th entry of 𝑊1 . In addition, the 𝑖-th
hidden neuron has a bias of 𝑏1,𝑖 , that is, the 𝑖-th element of b1 .
Similarly, for 1 ≤ 𝑖 ≤ 𝑛2 and 1 ≤ 𝑗 ≤ 𝑛1 , the 𝑖-th output neuron is connected to the
𝑗-th hidden neuron with a weight 𝑤2,𝑖𝑗 , that is, the (𝑖, 𝑗)-th entry of 𝑊2 . In addition,
the 𝑖-th hidden neuron has a bias of 𝑏2,𝑖 , that is, the 𝑖-th element of b2 .
In fact, when 𝑑 = 5, 𝑛1 = 7 and 𝑛2 = 2, the neural network’s structure is the one
shown in (a).
2. (a) The following argument is from the lecture slides. Consider minimizing a function
𝐿(w), then for small 𝜂, we have
𝐿(w − 𝜂 ∇ 𝐿(w))
≈ 𝐿(w) + (−𝜂 ∇ 𝐿(w))⊤ ∇ 𝐿(w)
= 𝐿(w) − 𝜂 ‖∇ 𝐿(w)‖22
≤ 𝐿(w).
The last inequality is strict if ∇ 𝐿(w) ̸= 0. That is, the function value decreases if
we move along the negative gradient direction by a small step.
Note that the above explanation is very rough: it doesn’t show that the function
actually decreases, and even if that’s the case, it doesn’t imply that gradient descent
will converge to a minimizer (it can converge to something else). There are many
theoretical results on when gradient descent is guaranteed to find a minimizer under
some conditions. These are beyond the scope of this course.
(b) i. Since 𝑥2 and 𝑥4 are both non-negative, we have 𝑓 (𝑥) ≥ 0 + 0 + 1 = 1 for any 𝑥.
The equality holds only when 𝑥 = 0, thus the minimizer is 𝑥 = 0.
ii. We have 𝑓 ′ (𝑥) = 4𝑥3 + 2𝑥.
2
𝑡 𝑥𝑡 𝑓 ′ (𝑥𝑡 )
0 1 6
1 0.4 1.056
2 0.2944 0.690864
3 0.22532 -
iii. Given the 𝑡-th iterate 𝑥𝑡 , the next iterate is
𝑥𝑡+1 = 𝑥𝑡 − 0.1𝑓 ′ (𝑥𝑡 )
= 𝑥𝑡 − 0.4𝑥3𝑡 − 0.2𝑥𝑡
= 𝑥𝑡 (0.8 − 0.4𝑥2𝑡 )
We can show by induction that for all 𝑡 ≥ 0, 0 ≤ 𝑥𝑡 ≤ 1, and
𝑥𝑡+1 ≤ 0.8𝑥𝑡 .
Clearly, the above inequalities hold for 𝑡 = 0. If for some 𝑡 the above inequalities
hold, then from 0 ≤ 𝑥𝑡 ≤ 1, we have 0 < 0.8−0.4𝑥2𝑡 < 0.8, thus 0 ≤ 𝑥𝑡+1 ≤ 0.8𝑥𝑡 .
It also follows easily that 𝑥𝑡+2 ≤ 0.8𝑥𝑡+1 . Hence 0 ≤ 𝑥𝑡 ≤ 0.8𝑡 𝑥0 . Thus as 𝑡 → ∞,
we have 𝑥𝑡 → 0.