Deep Learning — Study Notes
Original educational study material prepared as structured class notes.
1. Neural Network Basics
A neural network consists of layers of interconnected computational units. Each unit computes a
weighted sum of inputs, adds a bias, and applies an activation function.
The forward pass produces predictions. A loss function measures the difference between predictions
and targets. Backpropagation computes gradients of the loss with respect to model parameters, allowing
an optimizer to update the parameters.
Key review points
• Define the main concepts in your own words.
• Identify one practical example where the concept would be useful.
• Consider common limitations, failure modes, and evaluation methods.
2. Activation Functions
The sigmoid function maps values to a range between zero and one and is useful for binary output
probabilities. Tanh produces values between negative one and one.
ReLU is widely used in hidden layers because it is simple and helps reduce some gradient-related
problems. Variants such as Leaky ReLU can keep a small gradient for negative inputs.
Key review points
• Define the main concepts in your own words.
• Identify one practical example where the concept would be useful.
• Consider common limitations, failure modes, and evaluation methods.
3. Optimization
Gradient descent updates parameters in the direction that reduces the loss. Stochastic and mini-batch
gradient descent use subsets of training examples and are standard in modern deep learning.
Learning rate, batch size, optimizer choice, and regularization strongly affect training. Adam is a popular
adaptive optimizer, while momentum-based methods can accelerate progress in useful directions.
Key review points
• Define the main concepts in your own words.
• Identify one practical example where the concept would be useful.
• Consider common limitations, failure modes, and evaluation methods.
4. Convolutional Neural Networks
CNNs are designed to exploit local structure in grid-like data such as images. Convolutional filters learn
patterns such as edges, textures, and increasingly complex visual structures.
Pooling can reduce spatial dimensions and computational cost. Modern image models often use
convolutional blocks together with normalization, skip connections, or attention mechanisms.
Key review points
• Define the main concepts in your own words.
• Identify one practical example where the concept would be useful.
• Consider common limitations, failure modes, and evaluation methods.
5. Recurrent and Sequence Models
Recurrent neural networks process sequences while maintaining a hidden state. LSTM and GRU
architectures were developed to better handle long-range dependencies than simple recurrent networks.
Sequence modeling is used for language, speech, time series, and other ordered data. In many modern
language applications, transformer architectures have replaced recurrent networks.
Key review points
• Define the main concepts in your own words.
• Identify one practical example where the concept would be useful.
• Consider common limitations, failure modes, and evaluation methods.
6. Transformers and Attention
Attention allows a model to compute relationships between elements of a sequence. Self-attention lets
each token consider other tokens when constructing its representation.
Transformers use repeated attention and feed-forward blocks, positional information, and residual
connections. They have become a major architecture for language, vision, audio, and multimodal
systems.
Key review points
• Define the main concepts in your own words.
• Identify one practical example where the concept would be useful.
• Consider common limitations, failure modes, and evaluation methods.
7. Training Best Practices
Use a held-out validation set, track training and validation loss, and save checkpoints. Data
augmentation can improve generalization for suitable domains.
Regularization methods include weight decay, dropout, early stopping, and augmentation.
Hyperparameters should be tuned using validation data while keeping the final test set untouched until
the end.
Key review points
• Define the main concepts in your own words.
• Identify one practical example where the concept would be useful.
• Consider common limitations, failure modes, and evaluation methods.