Deep Learning Exam Guide
1. McCulloch-Pitts Neuron
The McCulloch-Pitts (M-P) neuron is the earliest mathematical model of an artificial neuron.
It takes multiple binary inputs, applies fixed weights, and compares the weighted sum to a threshold.
If the sum exceeds the threshold, the neuron fires (output = 1), else it doesn-t (output = 0).
It can simulate logic gates like AND, OR, and NOT. Limitation: weights and threshold are fixed -
cannot learn.
Deep Learning Exam Guide
2. Least Mean Squares (LMS) Algorithm
The LMS algorithm is an adaptive learning rule used to update the weights of a neuron to minimize
the mean square error (MSE) between predicted and target outputs.
Weight update rule:
w(t+1) = w(t) + - * e(t) * x(t), where e(t) = d(t) - y(t).
This ensures convergence towards minimum error through small adjustments each iteration.
Deep Learning Exam Guide
3. Perceptron Model
A Perceptron is the simplest feedforward network that classifies linearly separable data.
It computes the sum of weighted inputs plus bias and passes through an activation function (step
function).
Learning rule: w_i - w_i + -(d - y)x_i. The perceptron fails for non-linear problems like XOR.
Deep Learning Exam Guide
4. Multi-Layer Perceptron (MLP)
An MLP consists of input, hidden, and output layers.
Hidden layers with non-linear activations (ReLU, Sigmoid, Tanh) allow learning complex mappings.
Trained using the backpropagation algorithm, it adjusts weights via gradients to minimize error.
Deep Learning Exam Guide
5. Gradient Descent and Backpropagation
Gradient Descent minimizes a cost function by iteratively updating weights opposite to the gradient.
Backpropagation computes gradients layer-wise using the chain rule, propagating error backward
from output to input.
Deep Learning Exam Guide
6. CNN (Convolutional Neural Networks)
CNNs are specialized for spatial data (images). They use convolution layers to extract features,
pooling layers for dimensionality reduction, and fully connected layers for classification.
Feature maps capture local patterns like edges and textures.
Deep Learning Exam Guide
7. Pooling and Convolution Kernels
Pooling (max or average) reduces spatial dimensions and computation while retaining key
information.
Convolution kernels (filters) slide over input to extract specific features like edges, corners, etc.
Deep Learning Exam Guide
8. Recurrent Neural Networks (RNNs) and LSTM
RNNs handle sequential data, maintaining temporal memory through hidden states.
LSTMs solve vanishing gradient problems using gates: forget, input, and output gates controlling
memory flow.
They capture long-term dependencies effectively.
Deep Learning Exam Guide
9. Regularization in Deep Learning
Regularization techniques (L1, L2, Dropout, Early Stopping) prevent overfitting.
They add penalties or remove neurons during training to ensure generalization and robustness.
Deep Learning Exam Guide
10. Autoencoders
Autoencoders are unsupervised models that compress and reconstruct data.
They consist of an encoder (compress) and a decoder (reconstruct).
Variants:
- Sparse AE - penalizes large activations.
- Denoising AE - reconstructs original input from noisy version.
- Contractive AE - penalizes sensitivity to input variations.
- Variational AE (VAE) - learns latent probabilistic distribution.
- Regularized AE - prevents identity mapping for better latent features.
Deep Learning Exam Guide
11. Variational Autoencoder (VAE)
VAE combines probabilistic modeling and neural networks.
Encoder produces mean and variance vectors for latent z.
Loss = Reconstruction Error + KL Divergence between posterior and prior distributions.
Allows new sample generation through z~N(0,1).
Deep Learning Exam Guide
12. Generative Stochastic Networks (GSN)
GSNs extend denoising autoencoders by training a stochastic transition function between corrupted
and clean samples.
Over multiple iterations, samples from this transition approach the true data distribution, forming a
generative Markov chain.
Deep Learning Exam Guide
13. Transfer Learning & Domain Adaptation
Transfer Learning (TL) transfers knowledge from a source task/domain to a related target
task/domain.
It helps when labeled target data are scarce.
Types:
- Inductive TL - tasks differ, labeled target available.
- Transductive TL (Domain Adaptation) - tasks same, domain distributions differ.
- Unsupervised TL - no labeled data.
Methods: Instance transfer (reweight samples), Feature transfer (TCA, SCL), Model transfer
(fine-tuning), Relational transfer.
Deep Learning Exam Guide
14. Domain Adaptation Techniques
When P(X) differs across source and target but task is same.
Instance-based: KMM, KLIEP adjust sample weights.
Feature-based: learn domain-invariant space (TCA, DIF, EDT).
EDT aligns subspaces via eigen-decomposition; DIF minimizes Maximum Mean Discrepancy
(MMD).
Deep Learning Exam Guide
15. GANs (Generative Adversarial Networks)
GANs consist of two competing networks:
- Generator (G): creates fake samples.
- Discriminator (D): distinguishes real from fake.
Objective (minimax):
min_G max_D V(D,G) = E_x[log D(x)] + E_z[log(1 - D(G(z)))].
Equilibrium occurs when generated data distribution matches real data.
Deep Learning Exam Guide
16. GAN Variants
- DCGAN - uses deep convolutional layers and batch normalization for stable image generation.
- Conditional GAN (cGAN) - conditions generation on labels or attributes.
- InfoGAN - maximizes mutual information to learn interpretable features.
- Energy-based GAN - uses autoencoder as discriminator.
- WGAN - minimizes Wasserstein distance for stable training.
Deep Learning Exam Guide
17. Advanced GANs: CoGAN, LAPGAN, ALI
- CoGAN - trains two GANs with shared layers to learn joint distributions across domains (e.g.,
photo & sketch) without paired data.
- LAPGAN - generates images progressively through a Laplacian pyramid (low - high resolution)
improving details and stability.
- ALI (Adversarially Learned Inference) - jointly trains generator and encoder to match joint
distributions p(x,z) and q(x,z), enabling both generation and inference.
Deep Learning Exam Guide