Scribd
Upload
17 views12 pages

Understanding Convolutional Neural Networks

The document provides an overview of Convolutional Neural Networks (CNN), detailing their architecture, inspiration, and applications, particularly in image and video processing. It contrasts CNNs with Feedforward Neural Networks (FNN) and outlines the limitations of FNNs in handling large image data. Additionally, it describes the CNN structure, including layers such as convolution, pooling, and fully connected layers, and discusses the use of the MNIST dataset for image classification tasks.

Uploaded by

T SANKARA RAO
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
17 views12 pages

Understanding Convolutional Neural Networks

The document provides an overview of Convolutional Neural Networks (CNN), detailing their architecture, inspiration, and applications, particularly in image and video processing. It contrasts CNNs with Feedforward Neural Networks (FNN) and outlines the limitations of FNNs in handling large image data. Additionally, it describes the CNN structure, including layers such as convolution, pooling, and fully connected layers, and discusses the use of the MNIST dataset for image classification tasks.

Uploaded by

T SANKARA RAO
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Convolutional neural networks (CNN)

Questions
 What is a convolutional neural network (CNN)?
 What are some applications of CNN?
Objectives
 Understand the inspiration behind CNN and learn the CNN architecture
 Learn the convolution operation and its parameters
 Learn how to create a CNN using Galaxy’s deep learning tools
 Solve an image classification problem on MNIST digit classification dataset
using CNN in Galaxy
Requirements
 tutorial Hands-on: Introduction to deep learning
 slides Slides: Deep Learning (Part 1) - Feedforward neural networks (FNN)
 tutorial Hands-on: Deep Learning (Part 1) - Feedforward neural networks
(FNN)
 slides Slides: Deep Learning (Part 2) - Recurrent neural networks (RNN)
 tutorial Hands-on: Deep Learning (Part 2) - Recurrent neural networks
(RNN)
last_modification Published: May 19, 2021
last_modification Last Updated: Jun 2, 2022

What is a convolutional neural network (CNN)?


Speaker Notes
What is a convolutional neural network (CNN)?

Convolutional Neural Network (CNN)


 Increasing popularity of social media in past decade
o Image and video processing tasks have become very important

 FNN could not scale up to image and video processing tasks


 CNN specifically tailored for image and video processing tasks

Feedforward neural networks (FNN)


 In FNN all nodes in a layer connected to all nodes in next layer
o Each connection has a weight, must be learned by learning
algorithm

Limitations of FNN
 If input is 64 pixel by 64 pixel grayscale image
o Each grayscale pixel represented by 1 value, usually between 0 to
255
o Where 0 is black, 255 is white, and values in between are shades of
gray
 Since each grayscale pixel represented by 1 value, we say channel size is
1
 Image represented by 64 x 64 x 1 = 4,096 values (rows x columns x
channels)
o Hence, input layer of FNN has 4096 nodes

 Lets assume next layer has 500 nodes


o Since FNN fully connected, we have 4,096 x 500 = 2,048,000
weights
Limitations of FNN
 For complex problems, we need multiple hidden layers in our FNN
o Compunds the problem of having many weights

 Having too many weights


o Makes learning more difficult as dimension of search space is
increased
o Makes training more time/resource consuming

o Increases the likelihood of overfitting

 Problem is further compunded for color images


o Each pixel in color image represented by 3 values (RGB color mode)

o Since each pixel represented by 3 values, we say channel size is 3

o Image represented by 64 x 64 x 3 = 12,288 values (rows x columns


x channels)
o Number of weights is now 12,288 x 500 = 6,144,000

Limitations of FNN
 Clear that FNN cannot scale to larger images (Too many weights)
 Another problem with FNN
o 2D image represented as 1D vector in input layer

o Any spatial relationship in the data is ignored

Inspiration for CNN


 In 1959 Hubel & Wiesel did an experiment to understand how visual cortex
of brain processes visual info
o Recorded activity of neurons in visual cortex of a cat

o While moving a bright line in front of the cat

 Some cells fired when bright line is shown at a particular angle/location


o Called these simple cells

 Other cells fired when bright line was shown regardless of angle/location
o Seemed to detect movement

o Called these complex cells

 Seemed complex cells receive inputs from multiple simple cells


o Have an hierarchical structure

 Hubel and Wiesel won Noble prize in 1981

Inspiration for CNN


 Inspired by complex/simple cells, Fukushima
proposed Neocognitron (1980)
o Hierarchical neural network used for handwritten Japanese
character recognition
o First CNN, had its own training algorithm

 In 1989, LeCun proposed CNN that was trained by backpropagation


 CNN got popular when outperformed other models at ImageNet Challenge
o Competition in object classification/detection

o On hundreds of object categories and millions of images

o Run annually from 2010 to present

 Notable CNN architectures that won ImageNet challenge


o AlexNet (2012), ZFNet (2013), GoogLeNet & VGG (2014), ResNet
(2015)

Architecture of CNN
 A typical CNN has 4 layers
o Input layer

o Convolution layer

o Pooling layer

o Fully connected layer

 We will explain a 2D CNN here


o Same concepts apply to a 1 (or 3) dimensional CNN

Input layer
 Example input a 28 pixel by 28 pixel grayscale image
 Unlike FNN, we do not “flatten” the input to a 1D vector
o Input is presented to network in 2D as 28 x 28 matrix

o This makes capturing spatial relationships easier


Convolution layer
 Composed of multiple filters (kernels)
 Filters for 2D image are also 2D
 Suppose we have a 3 by 3 filter (9 values in total)
o Values are randomly set to 0 or 1

 Convolution: placing 3 by 3 filter on the top left corner of image


o Multiply filter values by pixel values, add the results

o Move filter to right one pixel at a time, and repeat this process

o When at top right corner, move filter down one pixel and repeat
process
o Process ends when we get to bottom right corner of image

3 by 3 Filter

Convolution operator parameters


 Filter size
 Padding
 Stride
 Dilation
 Activation function

Filter size
 Filter size can be 5 by 5, 3 by 3, and so on
 Larger filter sizes should be avoided
o As learning algorithm needs to learn filter values (weights)

 Odd sized filters are preferred to even sized filters


o Nice geometric property of all input pixels being around output pixel

Padding
 After applying 3 by 3 filter to 4 by 4 image, we get a 2 by 2 image – Size of
the image has gone down
 If we want to keep image size the same, we can use padding
o We pad input in every direction with 0’s before applying filter

o If padding is 1 by 1, then we add 1 zero in evey direction

o If padding is 2 by 2, then we add 2 zeros in every direction, and so


on

3 by 3 filter with padding of 1

Stride
 How many pixels we move filter to the right/down is stride
 Stride 1: move filter one pixel to the right/down
 Stride 2: move filter two pixels to the right/down

3 by 3 filter with stride of 2

Dilation
 When we apply 3 by 3 filter, output affected by pixels in 3 by 3 subset of
image
 Dilation: To have a larger receptive field (portion of image affecting filter’s
output)
 If dilation set to 2, instead of contiguous 3 by 3 subset of image, every
other pixel of a 5 by 5 subset of image affects output

3 by 3 filter with dilation of 2


Activation function
 After filter applied to whole image, apply activation function to output to
introduce non-linearlity
 Preferred activation function in CNN is ReLU
 ReLU leaves outputs with positive values as is, replaces negative values
with 0

Relu activation function

Single channel 2D convolution


Triple channel 2D convolution
Triple channel 2D convolution in 3D

Change channel size


 Output of a multi-channel 2D filter is a single channel 2D image
 Applying multiple filters results in a multi-channel 2D image
 E.g., if input image is 28 x 28 x 3 (rows x columns x channels)
o We apply a 3 x 3 filter with 1 x 1 padding, we get a 28 x 28 x 1
image
o If we apply 15 such filters, we get a 28 x 28 x 15

 Number of filters allows us to increase or decrease channel size

Pooling layer
 Pooling layer performs down sampling to reduce spatial dimensionality of
input
 This decreases number of parameters
o Reduces learning time/computation

o Reduces likelihood of overfitting

 Most popular type is max pooling


o Usually a 2 x 2 filter with a stride of 2

o Returns maximum value as it slides over input data


Fully connected layer
 Last layer in a CNN
 Connect all nodes from previous layer to this fully connected layer
o Which is responsible for classification of the image

An example CNN

An example CNN
 A typical CNN has several convolution plus pooling layers
o Each responsible for feature extraction at different levels of
abstraction
o E.g., filters in first layer detect horizontal, vertical, and diagonal
edges
o Filters in the next layer detect shapes

o Filters in the last layer detect collection of shapes

 Filter values randomly initialized, learned by learning algorithm


 CNN not only do classification, but can also automatically do feature
extraction
o Distinguishes CNN from other classification techniques (like Support
Vector Machines)

MNIST dataset
 MNIST dataset of handwritten digits
o Composed of training set of 60,000 and test set of 10,000 images

 Digits have been size-normalized/centered in a fixed-size image (28 by 28


pixels)
 Images are grayscale
o Each pixel is represented by a number between 0 and 255

o 0 for black, 255 for white, and other values for shades of gray

 MNIST dataset is a standard image classification dataset


o Used to compare various Machine Learning techniques

Classification of MNIST images with CNN


 We define a CNN and train it using MNIST dataset training data
 Goal is to learn a model such that given image of a digit we predict the
digit (0 to 9)
 We then evaluate the trained CNN on test dataset and plot the confusion
matrix

Common questions

Powered by AI

Notable CNN architectures like AlexNet, ZFNet, GoogLeNet, VGG, and ResNet, which have won the ImageNet Challenge, greatly advanced feature extraction and classification capabilities. AlexNet introduced deeper networks with dropout for regularization, ZFNet optimized hyperparameters for improved performance, GoogLeNet deployed inception modules for efficient computation, VGG emphasized depth with very small filters, and ResNet introduced residual learning to train even deeper networks without degradation. These architectures significantly improved object recognition tasks across various industries .

The pooling layer conducts down-sampling of the spatial dimensions of the input, which decreases the number of parameters the network must learn, thereby reducing the risk of overfitting. The most common form is max pooling, which captures the most significant features by selecting the maximum value from each region covered by the filter. This also helps in minimizing computation and learning time .

Padding involves adding extra pixels (usually with zero values) around the input image's border before applying the filter, which helps maintain the spatial dimensions of the input image in the output. This ensures that the edges of the input are adequately represented throughout the convolution process, which is crucial for building deeper networks without reducing image size .

Hubel & Wiesel's experiments on the visual cortex of a cat revealed the presence of simple and complex cells, which detect specific features like lines and movement at various angles. This inspired the hierarchical, layered structure of CNNs, where different layers of neurons detect increasingly complex features of input images . Fukushima's Neocognitron and LeCun's CNN developments were directly influenced by this understanding .

Odd-sized filters, such as 3x3, are preferred because they have the geometric property of a central pixel, facilitating symmetric weight distribution around the output pixel. This symmetry aids in neural network training and helps in uniformly capturing feature information from the input image .

CNNs are specifically designed to handle image data more effectively than FNNs. Unlike FNNs, which require a flattened 1D vector input and lose spatial information, CNNs accept 2D image inputs, preserving spatial relationships. CNNs use convolutional layers to apply local filters that capture spatial hierarchies and reduce the number of parameters compared to the fully connected layers of FNNs, thus alleviating the issue of having too many weights .

The ReLU activation function introduces non-linearity into the CNN, which is essential for learning complex patterns. It does so by setting any negative output values to zero while passing positive values unchanged. This simplicity reduces computational cost and helps mitigate the vanishing gradient problem, leading to faster training and improved model performance .

The convolution operation in CNNs applies local filters to the input image without requiring it to be flattened into a 1D vector, thus preserving spatial hierarchies. These filters, by sliding over the 2D input, can capture local spatial patterns, which are then aggregated across layers to build complex features. This structure contrasts with FNNs, which lose spatial information by treating images as 1D vectors .

CNNs are employed on the MNIST dataset to classify handwritten digits by learning spatial features via its multiple layers (convolution, pooling, and fully connected layers). The MNIST dataset is a benchmark for evaluating machine learning models because it is well-structured, size-normalized, and provides a standardized measure for comparing model performance, given its widespread use and simplicity as an image classification task .

Dilation in convolution refers to increasing the receptive field of the filter by skipping input pixels according to a set pattern, often allowing for larger context capturing without increasing the filter size or sacrificing image resolution. This technique helps in capturing more global features of an image without an increase in model complexity .

You might also like