Digital Signal Processing
Module 2.1
CNN – Part 1
Dr. John E. Ball
Electrical and Computer Engineering Department ‐ Mississippi State University
Module 2.1 – CNN Part 1
Module 2 Overall Objectives:
1. Explain biological motivation for Convolutional Neural Networks
(CNNs)
2. Explain basic CNN building blocks, how they work, how to code
them, and how to determine how many parameters each layer
required
3. Code and test a NN, a simple CNN, and a more complex CNN to
recognize digits (Mini‐Project 1)
Note: ECE offers classes such as Mathematical Foundations of Machine Learning
(Sr/MS), Neural Networks (PhD), Digital Image Processing (PhD), etc. These are good
classes to take that would compliment this class well.
Digital Signal Processing 2.1 ‐ 1
Module 2.1 – CNN Part 1
Module Objectives:
1. Explain the biological motivation for CNNs.
2. Discuss a typical CNN architecture.
3. Introduce Mini Project 1.
Digital Signal Processing 2.1 ‐ 2
Module 2.1 – CNN Part 1
Reading for this Module
1. DSP4 – 2.1 to 2.3.
Digital Signal Processing 2.1 ‐ 3
Module 2.1 – CNN Part 1
In‐class computation:
None.
Digital Signal Processing 2.1 ‐ 4
Module 2.1 – CNN Part 1
Announcements
Digital Signal Processing 2.1 ‐ 5
Module 2.1 – CNN Part 1
Motivation
Many tasks we encounter require image processing and object
recognition.
Name some?
Convolutional Neural Networks are very adept at this task!
Digital Signal Processing 2.1 ‐ 6
Module 2.1 – CNN Part 1
Student Exercise 1
(20 Minutes) Watch the TED Talk – Teaching Computers to Understand
Pictures.
TED Talk Video
[Link]
_to_understand_pictures?language=en
What did you learn? Post results to discussion board. Also post group
member names.
* Distance students do this as individual assignment.
Digital Signal Processing 2.1 ‐ 7
Module 2.1 – CNN Part 1
Student Exercise 1
Digital Signal Processing 2.1 ‐ 8
Module 2.1 – CNN Part 1
What is Deep Learning?
A Deep Neural Network (DNN) is a Neural Network (NN) with typically
at least five layers. DNNs can have hundreds of layers and millions of
parameters. Shallow networks have smaller numbers of layers.
We will look a powerful subset of DNNs called Convolutional Neural
Networks (CNNs).
Blue italics generally indicates a term you
need to add to your technical vocabulary.
Digital Signal Processing 2.1 ‐ 9
Module 2.1 – CNN Part 1
Why Neural Networks?
Why are Neural Networks so Powerful?
NNs are Universal Function Approximators:
Theorem: Given a continuous function f x and a parameter 0 ,
then a NN exists (requires nonlinearities, at least one hidden layer, and
sufficient neurons) that can approximate the function to any desired
level, that is,
f x g x , x ,
where g x is the NN output.
Digital Signal Processing 2.1 ‐ 10
Module 2.1 – CNN Part 1
Why Neural Networks?
Why is Universal Function Approximation important? Consider this 2D
case with three classes.
1
1 1 11
1 1 1
1 1 1 11 1 1 3 3
33
1
1 1 11 1 11 1
1 1 11 1 3 33 33
1
3 3 33 3
3 3
2 2 333
2
2 2 2 22 3 33
2 2 2 2 2 3
22 2 2 33 3
22 2 2 2 22 3
33 3 3
Digital Signal Processing 2.1 ‐ 11
Module 2.1 – CNN Part 1
Why Neural Networks?
Consider this 2D case with four classes and much harder problem space.
1
1 1 11
1 1 1
1 1 1 11 1 1 3 3
1
1 11 11 1 1 1 33
4 1 1 11 1
3 33 33
4 4 2 1 2 3 3
4 4 4 22 2
2 2 2 22 3 3 3 3
4 44 3 3
4 4 4 2 22 2 2 2 22 33
4 4 4 4 2 2 2 2 2 2 2 2 2 2 33 33
4 2 33 3
33 3 3 3
3
3 33 3 3
3
3 3
Digital Signal Processing 2.1 ‐ 12
Module 2.1 – CNN Part 1
Why Convolution?
Again, we turn to a math theorem.
Theorem: Given a signal x n corrupted with additive IID Gaussian
noise, the optimal detector is convolving the noisy signal with the
matched filter m n x n .
Here, the * operator means complex conjugate (e.g., a radar has a
complex signal with real and imaginary parts).
Note: This is also the correlation operation (correlation is just like
convolution, but you don’t time reverse one of the terms).
Digital Signal Processing 2.1 ‐ 13
Module 2.1 – CNN Part 1
Combining Convolution and Neural Networks
We get a powerful one‐two punch!
The convolution layers learn masks to detect features, and the NN
layers learn how to interpret those features.
Digital Signal Processing 2.1 ‐ 14
Module 2.1 – CNN Part 1
Limitations
Well, even though NNs are general function approximators, there does
not exist a recipe that says use a certain network architecture for a
certain problem.
We know there is a network that can give good results, but what does it
look like?
There is the fun part – making your own network to solve a particular
problem!
Digital Signal Processing 2.1 ‐ 15
Module 2.1 – CNN Part 1
DL – Biological Motivations
Hubel & Weisel discovered that neurons in the primary visual cortex respond to
oriented edges.
They also discovered simple and complex cells. Simple cells respond to their
preferred orientation at only specific spatial locations, while complex cells are more
spatially variant in their responses. Complex cells pool over multiple simple cells.
Researchers have shown that simple cells produce linear outputs while complex
cells aggregate outputs in a nonlinear fashion.
Digital Signal Processing 2.1 ‐ 16
Module 2.1 – CNN Part 1
DL – Biological Motivations
Hubel & Weisel discovered that
neurons in the primary visual cortex
respond to oriented edges.
Here, V1 neuron spiking activity
reaches maximum when edge is
oriented.
Image source: [Link]
Digital Signal Processing 2.1 ‐ 17
Module 2.1 – CNN Part 1
DL – Biological Motivations
Hubel & Weisel also discovered that simple cells sum LGN (lateral geniculate
nucleus of the thalamus) inputs and have orientation selectivity.
Image sources: [Link]
Digital Signal Processing 2.1 ‐ 18
Module 2.1 – CNN Part 1
DL – Biological Motivations
The Neocognitron developed in 1980 (Fukushima) recognizes simple images and is
based on the knowledge of how the biological system functions. LeCun initially
developed CNNs. He credits the Neocognitron for leading to the first CNN.
Neocognitron
Image source: [Link], [Link] CNN
Digital Signal Processing 2.1 ‐ 19
Module 2.1 – CNN Part 1
DL – Convolutional Neural Network
A Convolutional Neural Network (CNN) is a DNN that is loosely based on the animal
visual system. We will now look at the building blocks of a typical CNN.
Image source: [Link]
Digital Signal Processing 2.1 ‐ 20
Module 2.1 – CNN Part 1
DL – Biological Motivations
Cichy et al. compared a DNN to temporal
(magnetoencephalography) and spatial (functional
MRI) visual brain representations and concluded
that the DNN captured the stages of human visual
processing in both time and space from early
visual areas towards the dorsal and ventral streams.
Digital Signal Processing 2.1 ‐ 21
Module 2.1 – CNN Part 1
CNN and the Brain
This figure shows feature
maps at various stages of
neural processing
From left to right:
Photoreceptor
LGN cell
V1 simple cell
V1 complex cell
Image source: [Link]
Digital Signal Processing 2.1 ‐ 22
Module 2.1 – CNN Part 1
CNN and the Brain
Convolutional neural networks have three main
traits that support their use as models of
biological vision:
(1) they can perform visual tasks at near‐
human levels.
(2) they do this with an architecture that
replicates basic features known about the
visual system.
(3) they produce activity that is directly
relatable to the activity of different areas
in the visual system.
Image source: [Link]
Digital Signal Processing 2.1 ‐ 23
Module 2.1 – CNN Part 1
Student Exercise 2
Two minutes. How are CNNs loosely based on the brain and eye
functionality?
Post to discussion board.
Digital Signal Processing 2.1 ‐ 24
Module 2.1 – CNN Part 1
DL & ML – Timeline (selected items)
2014
1960 1989 1997 VGGnet, GAN,
Control Theory 1969 1974 ML system LSTM 2012 GoogleNet,
Kelley XOR Problem Backprop reads digits Schmidhuber & Alexnet DeepFace
Minsky&Papert (many) LeCun Hochreiter Krishevsky (many)
1959
Simple/Complex
Cells 1980 2009
1943 1958
Hubel & Wiesel Neocognitron ImageNet
Neural Nets Perceptron
McCulloch& Rosenblatt Fukushima Li
Pitt
Digital Signal Processing 2.1 ‐ 25
Module 2.1 – CNN Part 1
Some Important DL & ML Milestones
1960 Control Theory
Henry J. Kelley
Kelley was a professor of aerospace and ocean engineering at the Virginia Polytechnic Institute. In
1960, he published “Gradient Theory of Optimal Flight Paths,” itself a major and widely recognized
paper in his field. Many of his ideas about control theory – the behavior of systems with inputs, and
how that behavior is modified by feedback – have been applied directly to AI and ANNs over the
years. They were used to develop the basics of a continuous backpropagation model (aka the
backward propagation of errors) used in training neural networks.
1980 Neocognitron
Synthesized knowledge in biological vision to make an artificial vision system – recognizes simple
images via unsupervised learning. This network was an inspiration that lead to development of CNN
by LeCun.
Digital Signal Processing 2.1 ‐ 26
Module 2.1 – CNN Part 1
Some Important DL & ML Milestones
1989 First CNN
LeCun developed the CNN for handwritten digit recognition
1993 – (note no star on timeline)
Jürgen Schmidhuber
German computer scientist Schmidhuber solved a “very deep learning” task in 1993 that required more than
1,000 layers in the recurrent neural network. It was a huge leap forward in the complexity and ability of neural
networks.
1997 – Long short‐term memory was proposed ‐ Jürgen Schmidhuber and Sepp Hochreiter
A recurrent neural network framework, long short‐term memory (LSTM) was proposed by Schmidhuber and
Hochreiter in 1997. They improve both the efficiency and practicality of recurrent neural networks by
eliminating the long‐term dependency problem (when necessary information is located too far “back” in the
RNN and gets “lost”). LSTM networks can “remember” that information for a longer periods of time. Refined
over time, LSTM networks are widely used in DL circles, and Google recently implemented it into its speech‐
recognition software for Android‐powered smartphones.
Digital Signal Processing 2.1 ‐ 27
Module 2.1 – CNN Part 1
Some Important DL & ML Milestones
2009 ‐ Launch of ImageNet, Fei‐Fei Li
A professor and head of the Artificial Intelligence Lab at Stanford University, Fei‐Fei Li launched ImageNet in
2009. As of 2017, it’s a very large and free database of more than 14 million (14,197,122 at last
count) labeled images available to researchers, educators, and students. Labeled data – such as these images –
are needed to “train” neural nets in supervised learning.
2014 – DeepFace
Monster platforms are often the first thinking outside the box, and none is bigger than Facebook. Developed
and released to the world in 2014, the social media behemoth’s deep learning system – nicknamed DeepFace –
uses neural networks to identify faces with 97.35% accuracy. That’s an improvement of 27% over previous
efforts, and a figure that rivals that of humans (~97.5%).
Digital Signal Processing 2.1 ‐ 28
Module 2.1 – CNN Part 1
Mini‐Project 1
In MP1, you will:
• Do some prep work (a.k.a. homework) related to module 2 material [individual]
• Find a CNN paper and write a one‐page summary [individual]
• Use three NNs to classify digits [group]
1. A generic NN (given in assignment)
2. A simple CNN (given in assignment)
3. Modify the simple CNN to see if you can improve results
Digital Signal Processing 2.1 ‐ 29
Module 2.1 – CNN Part 1
MNIST
MNIST is a dataset of handwritten digits.
The MNIST CNN was the first CNN and was
invented by Yann LeCun.
MNIST is provided as part of Tensorflow.
Examples of the digits are shown here.
Each image is 28 x 28.
The dataset is also available here:
[Link]
Digital Signal Processing 2.1 ‐ 30
Module 2.1 – CNN Part 1
MP1 – MNIST NN # 1
This is the base NN for MNIST you will be using in MP1.
batch_size = 128
hidden_units = 256
dropout = 0.45
model = Sequential()
[Link](Dense(hidden_units, input_dim=input_size))
[Link](Activation(‘relu’))
[Link](Dropout(dropout))
[Link](Dense(hidden_units))
[Link](Activation(‘relu’))
[Link](Dropout(dropout))
[Link](Dense(num_labels))
[Link](Activation(‘softmax’))
Digital Signal Processing 2.1 ‐ 31
Module 2.1 – CNN Part 1
MP1 – MNIST NN # 1
This is the base CNN for MNIST you will be using in MP1. Code from Altienza
Advanced Deep Learning with TensorFlow 2 and Keras.
batch_size = 128
kernel_size = 3
pool_size = 2
filters = 32
dropout = 0.2
num_epochs = 3
model = Sequential()
[Link](Conv2D(filters, kernel_size, activation='relu’,
input_shape=input_shape))
[Link](MaxPooling2D(pool_size))
[Link](Conv2D(filters, kernel_size, activation='relu'))
[Link](MaxPooling2D(pool_size))
[Link](Conv2D(filters, kernel_size, activation='relu'))
[Link](Flatten())
[Link](Dropout(dropout))
[Link](Dense(num_labels))
[Link](Activation('softmax'))
Digital Signal Processing 2.1 ‐ 32
Module 2.1 – CNN Part 1
Minute Paper
A minute paper is an anonymous way to ask questions or get clarification on
something you didn’t understand.
The instructor will take minute papers and address questions in future classes,
study sessions, or discussion boards.
Take one minute and fill out an anonymous minute paper (on Canvas).
Digital Signal Processing 2.1 ‐ 33