Scribd
Upload
4 views1 page

Utils

The document outlines the use of NumPy functions for implementing activation functions and loss calculations in neural networks. Key functions include np.maximum for ReLU, np.where for derivatives, and np.clip for numerical stability in activations and cross-entropy. The document emphasizes the efficiency of vectorized operations in handling large data sets and preventing mathematical errors.

Uploaded by

cs25mtech02007
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views1 page

Utils

The document outlines the use of NumPy functions for implementing activation functions and loss calculations in neural networks. Key functions include np.maximum for ReLU, np.where for derivatives, and np.clip for numerical stability in activations and cross-entropy. The document emphasizes the efficiency of vectorized operations in handling large data sets and preventing mathematical errors.

Uploaded by

cs25mtech02007
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Justification of NumPy Functions in Activations and Losses

To implement the neural network’s activation functions and loss calculations efficiently from scratch, I
relied heavily on NumPy’s vectorized operations. Here is a breakdown of the specific np functions used
in [Link] and the mathematical reasoning behind them.

1. Handling Piecewise Functions


• [Link](0, x): Used in relu and multiclass hinge loss. It performs an element-wise
maximum operation. Mathematically, it applies f (x) = max(0, x) instantly across an entire matrix,
which avoids slow Python for loops when dealing with batches of data.

• [Link](condition, x, y): Used for the derivatives of ReLU and Leaky ReLU. The derivative
of ReLU is 1 if x > 0 and 0 otherwise. [Link] acts as a vectorized if-else statement, allowing
me to map these gradients mathematically across large tensors in a single step.

2. Numerical Stability
• [Link](x, min, max): Used heavily in sigmoid, tanh, and cross entropy.
– In activations, calculating e1000 throws an overflow error (NaN) because the number is too
large for 64-bit memory. Clipping inputs between -500 and 500 prevents this while keeping
the math accurate, since σ(500) is effectively 1 anyway.
– In Cross-Entropy, calculating log(0) throws a math error. I clipped the predictions between
1e−15 and (1−1e−15 ) to ensure the log function always receives a valid, strictly positive input.
• [Link](x, axis=-1, keepdims=True): Used in softmax. By subtracting the maximum logit
from all inputs before applying the exponential (exi −max(x) ), it mathematically shifts the values to
prevent exponential overflow, while keeping the final probabilities identical.

3. Mathematical Core Operations


• [Link](x): Used in sigmoid, tanh, and softmax to calculate ex . It computes the exponential
element-wise for the entire matrix.

• [Link](x): Used in cross entropy to compute the natural logarithm, which mathematically
penalizes confident but incorrect predictions.
• [Link] & [Link]:
1
(ypred − ytrue )2 .
P
– [Link] is used in Mean Squared Error (MSE) to compute N
– [Link] is used in the Cross-Entropy and HingePloss to aggregate the error margins. In
Softmax, it is used to calculate the denominator ( exi ) to normalize the outputs into a valid
probability distribution.

You might also like