Deep Learning: Feedforward Networks Guide
Uploaded by
sanjana smDeep Learning: Feedforward Networks Guide
Uploaded by
sanjana smCommon questions
Powered by AIDropout prevents overfitting by randomly deactivating neurons during training with a probability p, which effectively creates different network architectures for each training batch. This process prevents the network from becoming overly reliant on any single node, promoting generalization. During training, the network becomes robust to variations, mimicking ensemble learning. It is typically applied only during training because, during inference, all neurons are used to maintain the full model architecture, scaling the activations appropriately to reflect the effects of dropout during training .
ReLU (Rectified Linear Unit) benefits over sigmoid and tanh include computational efficiency and helping solve the vanishing gradient problem because it does not saturate in the positive region, unlike sigmoid and tanh. ReLU also avoids the vanishing gradient problem common with sigmoid and tanh, where gradients become very small, making training slower or causing it to stall. However, ReLU can suffer from the dying ReLU problem, in which some neurons can stop activating during training. Leaky ReLU helps address this issue by allowing a small, non-zero gradient when the unit is not active .
L1 regularization adds the absolute value of weights to the loss term, encouraging sparsity in the model by driving some weights to zero, which can serve a feature selection purpose. Consequently, it often results in models that are simpler and less sensitive to noise. In contrast, L2 regularization adds the squared value of weights to the loss, preventing large weights but not necessarily making weights zero. This form does not perform feature selection but rather spreads out the learning across more features, leading to smoother and more stable training. The difference in their mathematical formulation, L1 = λΣ|w| and L2 = λΣw², underlies their distinct impact, with L1 promoting sparse solutions and L2 discouraging overly complex models without loss of detail .
Backpropagation plays a critical role in training neural networks by calculating the gradient of the loss function with respect to each weight by applying the chain rule. This process is done layer by layer, propagating the error from the output layer back to the input. The chain rule is used to compute the derivatives of composite functions, essential for evaluating the contribution of each weight to the total error. It enables precise updates to the weights and biases by iteratively adjusting them in the direction that reduces error, making it the cornerstone of neural network learning .
Early stopping involves monitoring the validation error during training and halting the learning process if the error begins to increase, indicating potential overfitting. By doing so, early stopping captures the model at a point before overfitting intensifies, effectively acting as a regularizer. It not only promotes better generalization by preventing learning of noise in training data but also reduces training time by stopping when further improvements are unlikely. This makes it an automatic model selection mechanism, balancing performance and training efficiency without excess computations .
Mini-batch gradient descent is commonly used because it offers a compromise between the stability of batch gradient descent and the efficiency of stochastic gradient descent (SGD). It processes small batches of data, typically ranging from 32 to 128 samples, allowing it to converge faster than batch gradient descent while being less noisy than SGD. This approach helps in efficiently using memory and computational resources, providing more stable updates and reducing the oscillations that occur in stochastic updates, ultimately leading to faster and smoother convergence in practice .
Batch normalization involves normalizing inputs to a layer by adjusting and scaling based on the batch's mean and variance. The pseudo-code involves computing the mean and variance, normalizing the input, then scaling and shifting it using learned parameters gamma and beta. This normalization process reduces internal covariate shifts and speeds up training by allowing higher learning rates. Additionally, it helps regularize the model by introducing slight noise into the activations, effectively improving training stability and convergence speed. By addressing the shifting input distributions during training, batch normalization results in faster and more stable convergence .
The Huber loss function blends the Mean Squared Error (MSE) and Mean Absolute Error (MAE) by behaving like MSE when the error is small and like MAE when the error is large, defined by a threshold δ. This combination allows it to be less sensitive to outliers than MSE (which penalizes large errors more heavily) while maintaining gradient continuity for small deviations (like MSE) enabling efficient training. The blend provides robustness to outliers while retaining the properties of being smoothly differentiable, making it suitable for applications that need balanced sensitivity to noise and outliers .
Poor weight initialization can lead to slow convergence, vanishing/exploding gradient problems, and failure to learn complex data patterns, especially in deep networks. Xavier initialization addresses this by setting the variance of the weights as 2/(nin + nout), which is well-suited for symmetric activations like tanh, ensuring that the network layers neither blow up magnitudes nor dwarf them. He initialization, setting variance as 2/nin, is tailored for ReLU activations, which benefits from handling positive skewness by promoting variance that suits rectified linear outputs. These initialization strategies help stabilize variance across layers during both forward and backward passes, fostering a healthier training path .
Hyperparameter selection critically impacts neural network performance. The learning rate determines the step size in weight updates; too high can cause divergence, and too small can result in slow convergence. Batch size influences stability and convergence—smaller batches induce noisy updates but might generalize better than large ones. Regularization strength affects model complexity; excessive regularization can underfit by oversuppressing parameters, while insufficient regularization can overfit by insufficiently penalizing complexity. Balancing these hyperparameters through careful tuning is key to achieving optimal performance, ensuring efficient training without overfitting or underfitting .