ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey
Hinton
무하메드 (2023120216), 티머만크리스토프 (2023126767), 민하살림 (2023120344)
01 Introduction
02 The Architecture
03 Reducing Overfitting
04 Details of learning
05 Results
06 Conclusion and Discussion
Introduction
Evolution of Realistic Settings: Object classification faces
increased variability in realistic settings.
Dataset Scale: Larger training sets are imperative, with datasets
like ImageNet boasting 15 million
Q&A labeled high-resolution images
across 22,000 categories.
CNN Flexibility: CNNs offer scalability through varying depth and
breadth, accommodating diverse datasets effectively.
GPU Optimization: Optimized GPU implementation of 2D
convolution allows for training large CNNs on sizable datasets
without severe overfitting.
Training Infrastructure: Two GTX 580 3GB GPUs, coupled with
optimized convolution, reduce training time to 5-6 days.
Introduction
Brief Overview of Image Classification
Q&A
The Architecture
ReLU Nonlinearity
- Neurons often modeled with saturating nonlinearities like tanh(x)
or sigmoid f(x) = tanh(x) or f(x) = (1 + e−x)−1.
Q&A
- In gradient descent, these saturating nonlinearities are much
slower to train than the used ReLU function
- The iterations needed to reach 25% training error on CIFAR-10,
highlighting the efficiency of ReLUs.
- Enables experimentation with larger neural networks on datasets
like CIFAR-10.
The Architecture
Training on Multiple GPUs
GPU Memory Limitation: Single GTX 580 GPU has 3GB memory, restricting network
size.
Expanding Network Size: 1.2 million training examples prove sufficient for networks
Q&A
too large for a single GPU.
Two-GPU Configuration: Utilizing two GPUs for parallelization to overcome memory
constraints.
Direct GPU Communication: GPUs can read from and write to each other's memory
directly, optimizing parallelization.
Communication Strategy: Kernels distributed across GPUs with selective
communication, enhancing efficiency.
The Architecture
Local Response Normalization
Local Response Normalization (LRN) is a
technique used in the context of neural
networks
It enhances the generalization and performance of the Q&A
network by mitigating values becoming extremely large
or small
Each neuron is normalized based on its adjacent
neurons in the layer
A sum of squares is calculated using a gaussian-like
weighing function. The original neuron value is divided
by this sum, which normalizes the neuron’s response
Additionally, scaling and bias parameters are added to
enable optimization during training
The Architecture
Local Response Normalization
Q&A
The Architecture
Overlapping Pooling
Q&A
The Architecture
Overall Architecture
Q&A
The Architecture
Comprehensive Architecture of the
CNN
Layer Composition: Eight layers with
weights, comprising five convolutional
and three fully-connected layers. Q&A
Softmax Output: The final fully-
connected layer feeds into a 1000-way
softmax for class label distribution.
Objective Function: Maximizes the
multinomial logistic regression
objective.
Training Objective: Equivalent to
maximizing the log-probability of
correct labels across training cases.
The Architecture
Details of Convolutional Layers
First Convolutional Layer: 96 kernels of
size 11x11x3 with a stride of 4 pixels.
Second Convolutional Layer: 256
kernels of size 5x5x48 applied to the Q&A
(response-normalized and pooled)
output of the first layer.
Third-Fifth Convolutional Layers:
Interconnected without pooling or
normalization, with varying kernel
sizes and connectivity.
The fully-connected layers have 4096 neurons
each
Reducing Overfitting
What is
Overfitting?
-Overfitting occurs when a machine
learning model learns the training data
Söhne
too well.
- It often happens with complex models
or when the training dataset is small.
- Including too many features in the
model, especially irrelevant or noisy
features, can lead to overfitting.
Reducing Overfitting
Data Augmentation Strategies
- Overfitting Challenge: Mitigating overfitting on image data through
artificial dataset enlargement.
Q&A
- Common Method: Label-preserving transformations applied for dataset
augmentation.
- Two Forms of Augmentation: Computationally efficient, generated in
Python code on the CPU during GPU training.
- Real-time Transformation: Augmented images generated in real-time
during GPU training, minimizing storage requirements.
Reducing Overfitting
Image translations and horizontal reflections
- Image Translations and Reflections: Extracting random 224x224 patches and their
horizontal reflections from 256x256 images.
Q&A
- Training Set Expansion: Increases the training set size by a factor of 2048.
- Overcoming Overfitting: Essential for preventing substantial overfitting, enabling the use
of larger networks.
- Test-time Prediction: Network predictions averaged over ten patches (five and their
horizontal reflections).
Reducing Overfitting
Intensity Alteration with PCA
- RGB Channel Intensity Alteration: Performing PCA on ImageNet training
set RGB pixel values.
Q&A
- Proportional Magnitudes: Adding multiples of principal components
based on eigenvalues and random variables.
- Illumination Variations: Capturing natural image property of object
identity invariance to intensity changes.
- Error Rate Reduction: This scheme reduces the top-1 error rate by over
1%.
Reducing Overfitting
What is Dropout?
- Dropout acts as a form of regularization,
preventing overfitting and improving the model's
ability to generalize to new data.
Q&A
- During each training iteration, a random subset
of neurons is dropped out or set to zero. This
means that different sets of neurons are dropped
out in each iteration.
- During each training iteration, a dropout
probability (typically between 0.2 and 0.5) is
chosen.
Reducing Overfitting
Dropout
Dropout layer
- This Dropout technique randomly sets the output of
each hidden neuron to zero, with a probability of 0.5
Q&A
- Every time an input is presented, the neural network
samples a different architecture, but they all share the
same weights
- Applied to the first two fully-connected layers of the
neural network.
Details of learning
Training Configuration
- Optimization Method: Stochastic Gradient
Descent (SGD) utilized for training.
- Batch Size: SGD performed with a batch Q&A
size of 128 examples.
- Momentum: Momentum set to 0.9 for
smoother convergence.
96 convolutional kernels of size 11×11×3 learned by the first convolutional
- Weight Decay: Weight decay of 0.0005 layer on the 224×224×3 input images. The top 48 kernels were learned on
applied, crucial for learning and reducing GPU 1 while the bottom 48 kernels were learned on GPU 2.
training error.
Details of learning
Weight Update Rule
Q&A
96 convolutional kernels of size 11×11×3 learned by the first
convolutional layer on the 224×224×3 input images. The top 48 kernels
were learned on GPU 1 while the bottom 48 kernels were learned on
GPU 2
Details of learning
Weight Initialization Strategy
- Gaussian Distribution: Weights in each
layer initialized from a zero-mean Gaussian
distribution with standard deviation $0.01$.
Q&A
- Biases Initialization: Biases in specific
layers initialized differently to accelerate
early learning stages.
96 convolutional kernels of size 11×11×3 learned by the first convolutional
layer on the 224×224×3 input images. The top 48 kernels were learned on
GPU 1 while the bottom 48 kernels were learned on GPU 2
Details of learning
Learning Rate Adjustment
- Equal Learning Rate: Equal learning rate
applied to all layers, manually adjusted
during training.
Q&A
- Manual Adjustment: Learning rate
adjusted manually based on validation
error rate.
- Learning Rate Reduction: When validation
96 convolutional kernels of size 11×11×3 learned by the first convolutional
error rate plateaus, learning rate divided by layer on the 224×224×3 input images. The top 48 kernels were learned on
$10$. GPU 1 while the bottom 48 kernels were learned on GPU 2
- Initialization: Learning rate initialized at
$0.01$ and reduced three times during
training.
Details of learning
Training Duration and
Hardware Used
- Training Duration: Network trained for
approximately 90 cycles on 1.2 million
images.
Q&A
- Hardware Configuration: Training
performed on two NVIDIA GTX 580 3GB
GPUs.
- Time Taken: Training process took five to
96 convolutional kernels of size 11×11×3 learned by the first convolutional
six days. layer on the 224×224×3 input images. The top 48 kernels were learned on
GPU 1 while the bottom 48 kernels were learned on GPU 2
Results
ILSVRC-2010 results
Q&A
Results
ILSVRC-2012 results
Q&A
Results
Fall 2009 ImageNet
- 10,184 categories and 8.9 million images
- Half the images are used for training, half for testing
Q&A
- Splits differ from other authors, which does not affect results
- Top-1 and top-5 error rates are 67.4% and 40.9%
- The net above is used, but with an additional sixth convolutional layer over the last pooling layer
- The best published results were 78.1% and 60.9%
Results
Qualitative Evaluations
Q&A
Conclusion and Discussion
Supervised Learning Success: The paper demonstrates that a large, deep convolutional neural network can
achieve groundbreaking results on a highly challenging dataset through purely supervised learning.
Importance of Depth: The significance of depth in the network's performance is emphasized. Removing even a
single convolutional layer results in a notable loss (approximately
Q&A 2%) in the top-1 performance of the network.
This highlights the importance of the network's depth in achieving the reported results.
Impact of Middle Layers: Specifically, removing any of the middle layers of the network leads to a reduction in
top-1 performance, underscoring the importance of these layers in the learning process.
Absence of Unsupervised Pre-training: Despite acknowledging the potential benefits of unsupervised pre-
training, the authors simplify their experiments by not using it. They express the expectation that unsupervised
pre-training could be particularly helpful with increased computational power, allowing for a larger network
without a proportional increase in labeled data.
Conclusion and Discussion
- In conclusion, this paper presents a pioneering Convolutional
Neural Network (CNN) architecture, showcasing its remarkable
efficacy in object classification tasks. The integration of novel
elements such as rectified linear units
Q&A(ReLUs), dropout for
regularization, and strategic data augmentation contributes to the
CNN's outstanding performance
Thank you for
Q&A
attention