AlexNet

Detailed Long-Answer Notes

Module 3 — Convolutional Neural Networks | Course: IT702

1. Introduction to AlexNet

AlexNet is a landmark deep Convolutional Neural Network (CNN) architecture that revolutionized the
field of computer vision and deep learning. It was designed by Alex Krizhevsky, Ilya Sutskever, and
Geoffrey Hinton, and it won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012
by a significant margin, achieving a top-5 error rate of just 15.3%, compared to 26.2% achieved by the
second-best entry. This landmark result reignited widespread interest in deep neural networks and set
the stage for the modern deep learning era.

Key Fact Details

Introduced by Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton (2012)
Published in "ImageNet Classification with Deep CNN" — NeurIPS 2012
Competition ImageNet LSVRC-2012
Total Parameters Over 62.3 million learnable parameters
Total Layers 5 Convolutional + 3 Fully Connected + 1 Softmax = 8 layers
Input Size 227 × 227 × 3 (RGB image)
Output 1000 classes (Softmax probabilities)
Key Innovation First successful use of ReLU, Dropout, Data Augmentation in deep CNN

2. Architecture Overview

AlexNet consists of 8 learnable layers in total: 5 convolutional layers and 3 fully connected layers,
followed by a final Softmax output. Additionally, it has 3 Max-Pooling layers and 2 Local Response
Normalization (LRN) layers. The network processes input images of size 227×227×3 (even though
224×224×3 is often cited, the effective size including padding is 227×227×3).
The general flow of the architecture is:
• Input Image → Conv1 → Pool1 → LRN1
• → Conv2 → Pool2 → LRN2
• → Conv3 → Conv4 → Conv5 → Pool3
• → FC6 (Dropout) → FC7 (Dropout) → FC8 → Softmax Output

2.1 Layer-by-Layer Summary Table

Layer Input Size Filter / Kernel Stride Output Size Activation

Conv 1 227×227×3 96 @ 11×11 4 55×55×96 ReLU

MaxPool 1 55×55×96 3×3 2 27×27×96 —

LRN 1 27×27×96 — — 27×27×96 —

Conv 2 27×27×96 256 @ 5×5 1 (pad 2) 27×27×256 ReLU

MaxPool 2 27×27×256 3×3 2 13×13×256 —

LRN 2 13×13×256 — — 13×13×256 —

Conv 3 13×13×256 384 @ 3×3 1 (pad 1) 13×13×384 ReLU

Conv 4 13×13×384 384 @ 3×3 1 (pad 1) 13×13×384 ReLU

Conv 5 13×13×384 256 @ 3×3 1 (pad 1) 13×13×256 ReLU

MaxPool 3 13×13×256 3×3 2 6×6×256 —

Dropout — Rate: 0.5 — — —

FC 6 6×6×256=9216 4096 neurons — 4096 ReLU

Dropout — Rate: 0.5 — — —

FC 7 4096 4096 neurons — 4096 ReLU

FC 8 4096 1000 neurons — 1000 Softmax

3. Detailed Layer-by-Layer Description

3.1 Input Layer

The input to AlexNet is an RGB image of size 227×227×3 (height × width × channels). Although many
sources cite 224×224, the actual computation requires 227×227 due to padding applied in the first
convolution layer. Each pixel value in the three color channels (Red, Green, Blue) forms the raw input
features fed into the network.
3.2 Convolutional Layer 1 (Conv1)
Conv1 Specs Input: 227×227×3 | Filters: 96 | Kernel: 11×11 | Stride: 4 | Output:
55×55×96 | Activation: ReLU

The first convolutional layer applies 96 filters, each of size 11×11×3, with a stride of 4. The large 11×11
filter size is used to capture broad, coarse-grained features like colors, gradients, and edges across a
wide receptive field. The stride of 4 significantly reduces the spatial dimensions from 227×227 to
55×55. The ReLU activation function is applied after convolution, producing 96 feature maps of size
55×55.

The output feature map size is calculated as: (227 - 11) / 4 + 1 = 55.

3.3 Max-Pooling Layer 1 (Pool1)

Pool1 Specs Input: 55×55×96 | Kernel: 3×3 | Stride: 2 | Output: 27×27×96

The first max-pooling layer uses a 3×3 window with stride 2, which introduces overlapping pooling — a
technique used in AlexNet to reduce overfitting compared to non-overlapping pooling. This layer
reduces the spatial dimensions from 55×55 to 27×27, while retaining the depth of 96. Max-pooling
selects the maximum value within each window, preserving the most prominent features detected by
the convolutional filters.

3.4 Local Response Normalization 1 (LRN1)

Local Response Normalization (LRN) is applied after Pool1. It implements a form of lateral inhibition
inspired by the human visual cortex. LRN normalizes neuron activations across adjacent channels
(feature maps), which helps with generalization by creating competition between neurons that detect
similar features. The output remains 27×27×96 — the spatial dimensions and depth are unchanged.

3.5 Convolutional Layer 2 (Conv2)

Conv2 Specs Input: 27×27×96 | Filters: 256 | Kernel: 5×5 | Stride: 1 | Padding: 2 |
Output: 27×27×256 | Activation: ReLU

The second convolutional layer uses 256 filters of size 5×5, with a stride of 1 and zero-padding of 2 on
all sides. The padding ensures the output feature map maintains the same spatial dimensions (27×27).
This layer detects finer patterns like textures and more detailed edge configurations. ReLU activation is
applied. The depth increases to 256 feature maps.

3.6 Max-Pooling Layer 2 (Pool2) and LRN2

A 3×3 max-pooling layer with stride 2 reduces the spatial dimensions from 27×27 to 13×13, while the
depth remains at 256. Another round of Local Response Normalization (LRN2) is applied after pooling,
further regularizing the feature activations. Output: 13×13×256.

3.7 Convolutional Layer 3 (Conv3)

Conv3 Specs Input: 13×13×256 | Filters: 384 | Kernel: 3×3 | Stride: 1 | Padding: 1 |
Output: 13×13×384 | Activation: ReLU

The third convolutional layer applies 384 filters of size 3×3 with stride 1 and padding 1 (same padding).
This is the only convolutional layer in AlexNet that does not have pooling or normalization applied
immediately after it. The filter depth increases to 384, allowing the network to learn more complex and
abstract feature combinations from the preceding layer's output.

3.8 Convolutional Layer 4 (Conv4)

Conv4 Specs Input: 13×13×384 | Filters: 384 | Kernel: 3×3 | Stride: 1 | Padding: 1 |
Output: 13×13×384 | Activation: ReLU

The fourth convolutional layer is identical to Conv3 in terms of filter size, stride, and padding. It applies
384 filters of size 3×3 with stride 1 and padding 1. The output dimensions remain 13×13×384. This
layer further deepens the feature extraction hierarchy, allowing the network to learn highly abstract
representations.

3.9 Convolutional Layer 5 (Conv5)

Conv5 Specs Input: 13×13×384 | Filters: 256 | Kernel: 3×3 | Stride: 1 | Padding: 1 |
Output: 13×13×256 | Activation: ReLU
The fifth and final convolutional layer reduces the filter count to 256 while maintaining the 3×3 kernel
size with stride 1 and padding 1. After this layer, a Max-Pooling layer (Pool3) of size 3×3 with stride 2 is
applied, reducing the dimensions from 13×13 to 6×6, giving a final feature map of 6×6×256.

3.10 Fully Connected Layers (FC6, FC7, FC8) and Dropout

After the five convolutional layers and three pooling layers, the output feature map of 6×6×256 is
flattened into a one-dimensional vector of 9,216 neurons (6 × 6 × 256 = 9,216). This forms the input to
the fully connected layers.

• Applied before FC6 to reduce overfitting. Randomly sets 50% of neuron outputs to zero
during training, forcing the network to learn more robust, distributed representations.:
Dropout (Rate 0.5)
• The first fully connected layer with 4,096 neurons and ReLU activation. Each neuron is
connected to all 9,216 inputs from the flattened layer.: FC6 (4096 neurons)
• Applied again before FC7.: Dropout (Rate 0.5)
• The second fully connected layer with 4,096 neurons and ReLU activation.: FC7 (4096
neurons)
• The final fully connected layer (output layer) with 1,000 neurons — one per class in the
ImageNet dataset. The Softmax activation function converts the raw scores into
probability distributions across all 1,000 classes.: FC8 (1000 neurons)

4. Key Innovations and Features of AlexNet

4.1 ReLU Activation Function

AlexNet was one of the first networks to use the Rectified Linear Unit (ReLU) as its activation function,
replacing the traditional sigmoid and tanh functions. ReLU is defined as f(x) = max(0, x). It offers
several advantages:
• Unlike sigmoid/tanh, ReLU does not saturate for large positive values, which means
gradients remain strong and do not vanish during backpropagation.: Non-saturation
• Networks using ReLU converge up to 6 times faster than those using tanh, as
demonstrated in the original AlexNet paper.: Faster Convergence
• ReLU is computationally inexpensive, requiring only a simple thresholding operation.:
Computational Simplicity
• ReLU naturally creates sparse activations (many neurons output zero), which is
computationally efficient and often improves generalization.: Sparsity
4.2 Dropout Regularization
Dropout is a powerful regularization technique introduced prominently by AlexNet. During training, each
neuron in the dropout layers has a 50% probability of being temporarily "dropped out" (set to zero). This
prevents neurons from co-adapting too closely to each other, which reduces overfitting. Dropout
effectively trains an ensemble of different network architectures simultaneously. At test time, all
neurons are active but their outputs are scaled down by the dropout rate.

4.3 Data Augmentation

To prevent overfitting on the training data, AlexNet employed two major data augmentation techniques:
• 256×256 images were randomly cropped to 227×227 patches, and horizontal flips were
applied, effectively multiplying the training dataset size by a factor of 2,048.: Random
Cropping and Horizontal Flipping
• Principal Component Analysis (PCA) was applied to the RGB pixel values of training
images, and random multiples of the principal components were added to the images.
This captures the important property that object identity is invariant to changes in the
intensity and color of illumination.: PCA Color Augmentation

4.4 Local Response Normalization (LRN)

Local Response Normalization was applied after the first and second convolutional layers in AlexNet.
LRN implements a form of lateral inhibition by normalizing the activity of neurons that detect similar
features in adjacent channels. This is inspired by the concept of contrast normalization in the visual
cortex. While later architectures (like VGGNet) found LRN less effective and stopped using it, it
contributed to AlexNet's performance in 2012.

4.5 Overlapping Max-Pooling

Traditional pooling uses non-overlapping windows (stride = pool size). AlexNet introduced overlapping
pooling where stride (2) is less than the pool size (3×3). Overlapping pooling reduces the top-1 and top-
5 error rates slightly and makes the model harder to overfit, as demonstrated in the original paper.

4.6 GPU-Parallelized Training

AlexNet was among the first networks to be trained on two GPUs simultaneously (NVIDIA GTX 580,
3GB memory each). The network was split across two GPUs: some feature maps were computed on
one GPU and others on the second GPU. Cross-GPU communication only happened at certain layers.
This parallelism was necessary because the network was too large to fit on a single GPU at the time,
and it significantly reduced training time.

5. AlexNet vs ZFNet — Comparative Overview

ZFNet (Zeiler and Fergus Net, 2013) was a direct improvement upon AlexNet, winning the ILSVRC
2013 competition. It refined AlexNet's architecture based on visualization of learned features.

Feature AlexNet ZFNet

Year 2012 2013

First Layer Filter 11×11, Stride 4 7×7, Stride 2
Subsequent Filters 5×5, 3×3 3×3 only
Normalization Local Response Normalization Local Contrast Normalization
(LRN)
Visualization Not used Deconvolution layers for feature
visualization
Architecture 5 Conv + 3 FC layers 5 Conv + 3 FC layers (same,
refined)
Key Improvement — Smaller strides capture finer
features in early layers

6. Significance and Impact of AlexNet

AlexNet's victory at ILSVRC 2012 marked a pivotal turning point in the history of artificial intelligence
and computer vision. Its impact can be summarized as follows:

• AlexNet demonstrated conclusively that deep neural networks with many layers, trained
on large datasets with GPU acceleration, significantly outperform traditional machine
learning approaches like SVMs with hand-crafted features.: Proof of Deep Learning
• Prior to 2012, neural networks had fallen out of favor in the research community.
AlexNet's success reinvigorated interest in deep learning worldwide.: Revival of Neural
Networks
• AlexNet's architectural principles — stacked convolutional layers, ReLU activations,
dropout, data augmentation — became the foundational template for nearly all
subsequent CNN architectures including VGGNet, GoogLeNet, ResNet, and beyond.:
Foundation for Modern CNNs
 • AlexNet showed that GPUs were essential for training large neural networks efficiently,
catalyzing the development of GPU-based deep learning frameworks such as
TensorFlow, PyTorch, and Caffe.: GPU-Driven AI Research
• The ImageNet competition became the de facto benchmark for image classification, and
AlexNet's performance set a new standard that every subsequent architecture has
attempted to improve upon.: ImageNet Benchmark

7. Limitations of AlexNet

Despite its revolutionary contributions, AlexNet has several limitations that were addressed by later
architectures:
• With over 62.3 million parameters, AlexNet is computationally expensive and requires
significant memory. Approximately 90% of these parameters reside in the fully connected
layers.: Large Number of Parameters
• Local Response Normalization was later found to have minimal benefit and was replaced
by Batch Normalization in more modern architectures.: Use of LRN
• The 11×11 kernel in the first convolutional layer is very large and captures coarse
features. VGGNet showed that stacking multiple smaller 3×3 kernels achieves better
performance with fewer parameters.: Large Kernel Sizes
• While innovative, this was found to have only marginal benefits in later work.:
Overlapping Pooling
• With only 8 learnable layers, AlexNet is relatively shallow compared to modern
architectures like ResNet-152 or VGG-19.: Depth

8. Expected Long-Answer Exam Questions & Model Answers

Q1: Describe the architecture of AlexNet in detail. Include the number of layers,
filter sizes, and the role of each layer.

Answer: AlexNet is a deep Convolutional Neural Network architecture consisting of 8 learnable layers
— 5 convolutional layers and 3 fully connected layers. It was introduced in 2012 by Krizhevsky,
Sutskever, and Hinton, and won the ILSVRC-2012 image classification challenge.

Input: The network takes a 227×227×3 RGB image as input.

Conv1: 96 filters of size 11×11 are applied with stride 4. This produces 55×55×96 feature maps. ReLU
activation is applied. The large kernel captures broad visual features (edges, colors). Max-Pooling (3×3,
stride 2) reduces dimensions to 27×27×96. LRN is applied for normalization.

Conv2: 256 filters of size 5×5 with stride 1 and padding 2 are applied. Output is 27×27×256. ReLU is
applied. Max-Pooling and LRN follow, reducing output to 13×13×256.

Conv3: 384 filters of size 3×3, stride 1, padding 1. Output: 13×13×384. ReLU activation.

Conv4: 384 filters of size 3×3, stride 1, padding 1. Output: 13×13×384. ReLU activation.

Conv5: 256 filters of size 3×3, stride 1, padding 1. Output: 13×13×256. Max-Pooling (3×3, stride 2)
reduces to 6×6×256. ReLU activation.

FC6: The 6×6×256 = 9,216 neuron flat vector feeds into 4,096 fully connected neurons with ReLU
activation. Dropout (rate 0.5) is applied before this layer.

FC7: 4,096 neurons with ReLU activation. Dropout (rate 0.5) applied before this layer.

FC8 (Output): 1,000 neurons with Softmax activation, producing probabilities for 1,000 ImageNet
classes.

AlexNet has over 62.3 million trainable parameters in total. The network was trained on two GPUs due
to its size, and it introduced critical innovations such as ReLU, Dropout, Data Augmentation, and
Overlapping Pooling.

Q2: What are the key innovations introduced by AlexNet? Explain each in detail.

Answer: AlexNet introduced several groundbreaking innovations that have since become standard
practice in deep learning:

1. ReLU Activation Function: AlexNet replaced the traditional sigmoid and tanh activation functions with
ReLU (Rectified Linear Unit), defined as f(x) = max(0, x). ReLU does not saturate for positive inputs,
allowing gradients to flow without vanishing during backpropagation. This led to networks converging
up to 6× faster than with tanh, as demonstrated in the paper.
2. Dropout Regularization: To combat overfitting on the large ImageNet dataset, AlexNet applied
dropout with a rate of 0.5 in the fully connected layers FC6 and FC7. During training, each neuron has
a 50% chance of being randomly deactivated. This forces the network to learn multiple independent
representations of the same feature, effectively creating an ensemble of different sub-networks and
drastically reducing overfitting.

3. Data Augmentation: AlexNet artificially expanded the training dataset through random 227×227
crops from 256×256 images, horizontal flipping, and PCA color augmentation. This multiplied the
effective training data size by thousands, reducing overfitting.

4. Local Response Normalization (LRN): Inspired by lateral inhibition in neuroscience, LRN normalizes
the activations of neurons sharing the same spatial position across adjacent feature maps, creating
competition between feature detectors. This was used after Conv1 and Conv2.

5. Overlapping Max-Pooling: Unlike standard non-overlapping pooling, AlexNet used 3×3 pooling
windows with stride 2, creating overlapping regions. This marginally reduces error rates and makes
learned representations more robust.

6. GPU-Parallel Training: AlexNet was trained across two NVIDIA GTX 580 GPUs. The model was split
between GPUs, with inter-GPU communication at specific layers. This made training feasible within a
reasonable time frame (about 5-6 days) on large-scale data.

Q3: Compare AlexNet and ZFNet. What improvements did ZFNet introduce?

Answer: ZFNet (Zeiler and Fergus Network) was introduced in 2013 and directly improved upon
AlexNet, winning the ILSVRC-2013 competition. The key differences are:

Filter Size: AlexNet used 11×11 filters in the first convolutional layer with stride 4. ZFNet reduced this to
7×7 filters with stride 2. Smaller filters and smaller strides in the first layer allow ZFNet to capture more
fine-grained spatial details that AlexNet misses due to its aggressive downsampling.
Normalization: AlexNet used Local Response Normalization (LRN), which normalizes across channels
based on local activity. ZFNet replaced this with Local Contrast Normalization, which is a more
principled form of normalization that better accounts for spatial contrast in feature maps.

Feature Visualization: ZFNet introduced Deconvolutional Layers (transposed convolutions) as a

visualization tool. These allowed the researchers to project feature activations back to the pixel space,
providing insight into what features each layer of the network had learned. This interpretability tool
helped in diagnosing and improving the architecture.

Architecture Similarities: ZFNet retained AlexNet's overall structure of 5 convolutional layers followed
by 3 fully connected layers, same use of ReLU, dropout, and data augmentation. The improvements
were targeted and surgical rather than wholesale redesigns.

9. Important Terms and Definitions

Term Definition

Convolution Mathematical operation where a filter slides over input to produce a

feature map highlighting specific patterns.
ReLU Rectified Linear Unit — activation f(x)=max(0,x). Non-saturating, enables
fast convergence.
Max Pooling Downsampling operation that selects the maximum value in each pooling
window, reducing spatial dimensions.
Dropout Regularization technique: randomly deactivates neurons during training at
a set rate (e.g., 0.5).
LRN Local Response Normalization — normalizes neuron outputs across
adjacent feature maps to implement lateral inhibition.
Softmax Activation function converting raw output scores into a probability
distribution over classes.
Feature Map The output of a convolutional layer — a 2D representation of detected
features at each spatial location.
Receptive Field The region of the input image that influences a particular neuron's
activation.
Overlapping Pooling Pooling where stride < kernel size, causing pool windows to overlap.
Data Augmentation Techniques to artificially increase training data by applying transformations
like flipping, cropping, color shifts.
ILSVRC ImageNet Large Scale Visual Recognition Challenge — benchmark
competition for image classification on 1000 classes.
Parameters Learnable weights and biases in a neural network. AlexNet has ~62.3
million parameters.

10. Quick Revision — Points to Remember

• AlexNet = 5 Conv + 3 FC + 1 Softmax (8 learnable layers total).
• Total parameters: ~62.3 million.
• Input size: 227×227×3. Output: 1000 classes.
• Conv1: 96 filters, 11×11, stride 4. Conv2: 256 filters, 5×5. Conv3/4: 384 filters, 3×3. Conv5: 256
filters, 3×3.
• Three Max-Pooling layers: after Conv1, Conv2, and Conv5.
• Two LRN layers: after Pool1 and Pool2.
• Two Dropout layers (rate 0.5): before FC6 and FC7.
• Key innovations: ReLU, Dropout, Data Augmentation, LRN, Overlapping Pooling, Dual-GPU
Training.
• AlexNet won ILSVRC-2012 with top-5 error of 15.3% vs second place's 26.2%.
• ZFNet improved AlexNet by using 7×7 stride-2 first layer instead of 11×11 stride-4.

End of AlexNet Notes | IT702 — Module 3