Deep Learning Concepts and Calculations
Uploaded by
Prabhupada SamantarayDeep Learning Concepts and Calculations
Uploaded by
Prabhupada SamantarayCommon questions
Powered by AIDimensionality reduction techniques like PCA reduce data complexity while retaining essential structures by projecting data to a lower-dimensional space along principal components of maximum variance. The role in feature learning involves noise reduction, improving algorithm runtime, and preventing overfitting. Variance capture relates to practical interpretation by quantifying how much of the original data's total variance each principal component explains, guiding which components can be discarded without significant information loss. Retaining sufficient variance ensures the reduced data's fidelity remains high, critical for effective model building .
Adam combines momentum and RMSProp to provide adaptive learning rates for parameter updates. It computes individual adaptive learning rates using estimates of first (mean) and second (uncentered variance) moments of gradients. Adam updates parameters using: mt = β1mt−1 + (1−β1)gt (momentum) vt = β2vt−1 + (1−β2)g^2t (RMSProp aspect) The adjustments account for sparseness and noise in gradients, which contributes to faster convergence and robustness. Default parameters β1 = 0.9, β2 = 0.999 are typically effective, making Adam suitable for various deep models .
The transformation from logits to softmax probabilities involves applying the softmax function, which rescales raw logits to probabilities that sum to 1. Given logits z = [z1, z2,...,zk], softmax is applied as: softmax(zi) = e^zi / Σ(e^zj). For cross-entropy loss, if the true class is encoded in a vector (1 among k-classes), the loss for predicted probabilities (p) is: L = -Σ(y_true[i] * log(p[i])). Mathematically, it measures the divergence between the true class and predicted probabilities, aiding in minimization during training .
Convolutional layers improve computational efficiency by using shared weights, reducing the parameters compared to fully connected layers that have unique weights for every input-output pair. For image data, these layers exploit spatial hierarchies by learning local patterns (e.g., edges, textures) that transform into complex representations (features) as layers deepen. Due to spatial weight sharing, they require fewer computations, hence, are more memory efficient. Moreover, convolutional operations are parallelizable due to their sliding window nature over the input data .
Increasing the style weight in NST emphasizes capturing style characteristics more prominently, shifting the content-style balance towards stylization. As a result, the generated image reflects textures, colors, and patterns of the style reference image more than retaining structural fidelity to the content image. However, excessive style weight could obscure content-specific details, leading to less identifiable rendering of the original content scene, highlighting the need to balance these weights carefully for desired artistic output .
Numerical gradient checking ensures the correctness of backpropagation by estimating gradients using finite difference approximations and comparing these values against analytically computed gradients. For a function f(w), the numerical gradient is approximated as f(w+ε)−f(w−ε)/2ε. By computing this on perturbed weights and comparing with gradients from backprop, discrepancies indicate potential errors in implementation. This checks the implementation's accuracy since numerical calculations are less error-prone than complex analytical derivations, given small ε .
The choice between ReLU and Sigmoid significantly impacts convergence. ReLU is preferred because of its ability to mitigate the vanishing gradient problem due to its linear growth in positive input regions, allowing for effective gradient flows in deep networks. In contrast, Sigmoid squashes input to [0,1], resulting in gradient shrinkage (vanishing gradient) as inputs push into saturation regions, impeding deeper layer training. ReLU's sparsity-inducing property (outputs zero for any negative input) also aids efficient learning .
LSTMs and GRUs tackle the vanishing gradient problem present in RNNs through their gating mechanisms. LSTMs use forget, input, and output gates to control cell state preservation, allowing gradients to flow across many time steps. This structure prevents gradient diminishing, maintaining informative signal propagation. GRUs simplify this by merging some gate functions while maintaining similar benefits of capturing long-term dependencies efficiently, thus further addressing computational inefficiency while maintaining comparable performance .
The KL divergence term in VAEs measures the difference between the learned latent distribution and a prior distribution (usually standard normal). It acts as a regularizer, ensuring the latent space captures diverse yet structured data characteristics by penalizing deviations from the prior. This alignment promotes variability in the generations while preventing overfitting. It is crucial for ensuring that samples drawn from the latent space remain coherent and capable of reconstructing varied input data faithfully .
To perform one forward pass through a three-layer MLP with architecture, e.g., 784 → 128 → 64 → 10 and ReLU activations, follow these steps: 1. Input layer: Feed the input data (dim 784) to the network. 2. First hidden layer: Compute Z1 = W1X + b1, where W1 is weight matrix (128x784), and b1 is bias (128). Apply ReLU: A1 = ReLU(Z1). 3. Second hidden layer: Compute Z2 = W2A1 + b2, where W2 is (64x128), and b2 is (64). Apply ReLU: A2 = ReLU(Z2). 4. Output layer: Compute Z3 = W3A2 + b3, where W3 is (10x64), and b3 is (10). Output is A3 = softmax(Z3). Each step uses matrix multiplication and activation application based on architecture specifics.