Normalization,
Regularization
Outline
• Normalization
• Regularization
• Interaction of optimization, initialization, normalization, regularization
Initialization vs. optimization
Initialization vs. optimization
Normalization
• Initialization matters a lot for training, and can vary over the course
of training to no longer be “consistent” across layers / networks
• But remember that a “layer” in deep networks can be any
computation at all…
• …let’s just add layers that “fix” the normalization of the activations to
be whatever we want!
Layer normalization
• First idea: let’s normalize (mean zero and variance one) activations at
each layer; this is known as layer normalization
Also common to add an additional scalar weight and bias to each term (only changes representation e.g., if we put
normalization prior to nonlinearity instead)
LayerNorm illustration
Batch normalization
Minibatch dependence
Outline
Regularization of deep networks
• Typically deep networks are overparameterized models: they contain
more parameters (weights) than the number of training examples
• This means (formally, under a few assumptions), that they are capable of
fitting the training data exactly
• In “traditional” ML/statistical thinking (with a number of big caveats),
this should imply that the models will overfit the training set, and not
generalize well
• but they do generalize well to test examples
• but not always (many larger models will often still overfit)
Regularization
ℓ2 Regularization a.k.a. weight decay
Dropout
Outline
Many solutions … many more questions
BatchNorm: Other benefits?
References
• 10-414/714 CMU