Introduction to Deep Learning
Ayesha Abduljalil
Artificial Neural
Networks (ANNs)
An Artificial Neural Network (ANN) is
a Machine Learning model inspired by
the networks of biological neurons
found in the human brain.
Artificial Neural
Networks (ANNs)
ANNs are at the very core of Deep Learning. They are
versatile, powerful, and scalable, making them ideal to
tackle large and highly complex Machine Learning tasks
such as classifying billions of images (e.g., Google
Images), powering speech recognition services (e.g.,
Apple’s Siri), recommending the best videos to watch to
hundreds of millions of users every day (e.g., YouTube),
or learning to beat the world champion at the game of
Go (DeepMind’s AlphaGo)
Artificial Neural
Networks (ANNs)
Artificial Neural Networks (ANNs) were first introduced
in 1943 by the neurophysiologist, Warren McCulloch, and
the mathematician, Walter Pitts, in their landmark paper,
‘A Logical Calculus of Ideas Immanent in Nervous
Activity’.
Rise of ANNs
In recent decades, Artificial Neural Networks (ANNs) have evolved from a promising idea
to one of the most powerful tools in modern Machine Learning. Their success is not due
to a single breakthrough, but rather the convergence of several key factors: the explosive
growth of data, advances in computing hardware, refinements in training algorithms, and
increasing investment driven by real-world successes. Together, these developments
have created the right conditions for ANNs to flourish, allowing them to outperform
traditional Machine Learning methods on large and complex problems while continuously
fueling further progress.
Rise of ANNs
1. There is now a huge quantity of data available to train
neural networks, and ANNs frequently outperform
other ML techniques on very large and complex
problems.
2. The tremendous increase in computing power since
the 1990s now makes it possible to train large neural
networks in a reasonable amount of time. This is in part
due to Moore’s Law, but also thanks to the gaming
industry, which has stimulated the production of
powerful GPU cards by the millions. Moreover, cloud
platforms have made this power accessible to
everyone.
Rise of ANNs
3. The training algorithms have improved.
4. Some theoretical limitations of ANNs have turned
out to be benign in practice. For example, many
people thought that ANN training algorithms were
doomed because they were likely to get stuck in
local optima, but it turns out that this is rather rare in
practice.
5. ANNs seem to have entered a virtuous circle of
funding and progress. Amazing products based on
ANNs regularly make the news, which pulls more
and more attention and funding towards them,
resulting in more and more progress and even more
amazing products.
The Artificial Neuron
The artificial neuron proposed by McCulloch and Pitts is
a very simple model of the biological neuron. It has one
or more binary (on/off) inputs and one binary output.
The artificial neuron activates its output when more than
a certain number of its inputs are active.
The Artificial Neuron
Let’s take the example of a few ANNs that perform various logical computations. Here we
are assuming that a neuron is activated when at least two of its inputs are active.
ANNs performing simple logical computations
The Artificial Neuron
• The first network is the identity function. If neuron A is activated, then neuron C gets
activated as well. Neuron C receives two input signals from neuron A. If neuron A is off,
then neuron C is off as well.
• The second network performs a logical AND operation. Neuron C is activated only when
both neurons A and B are activated. A single input signal is not enough to activate neuron
C.
• The third network performs a logical OR operation. Neuron C is activated if either neuron
A or neuron B is activated.
• The last network computes a slightly more complex logical proposition. Neuron C is
activated only if neuron A is active and neuron B is off. If neuron A is active all the time,
then we get a logical NOT, that is, neuron C is active when neuron B is off, and vice versa.
The Perceptron
The Perceptron is one of the simplest ANN
architectures, invented in 1957 by Frank Rosenblatt. It is
based on a slightly different artificial neuron called a
threshold logic unit (TLU), or sometimes a linear
threshold unit (LTU).
The Perceptron
The inputs and output are numbers (instead of binary on/off values), and each input
connection is associated with a weight.
The TLU computes a weighted sum of its inputs,
z = w1x1 + w2x2 + … + wnxn = XT W
The TLU then applies a step function to that sum and outputs the result,
hw(X) = step(z), where z = XT W.
The Perceptron
Threshold Logic Unit: An artificial neuron which computes a weighted sum of its inputs then applies a step function
The Perceptron
The most common step function ued in Perceptrons is the Heaviside step function.
Sometimes, the sign function is also used.
0 if z < 0 -1 if z < 0
Heaviside (z) = sgn (z) = 0 if z = 0
1 if z >= 0 +1 if z > 0
A single TLU can be used for simple linear binary classification. It computes a linear
combination of the inputs, and if the results exceeds a threshold, it outputs the positive
class. Otherwise it outputs the negative class.
The Perceptron
The Perceptron is simply composed of a single layer of TLUs, with each TLU connected to
all the inputs. When all the neurons in a layer are connected to every neuron in the
previous layer (i.e., its input neurons), the layer is called a fully connected layer, or a dense
layer.
The inputs of the Perceptron are fed to special passthrough neurons called input neurons:
they output whatever input they are fed. All the input neurons form the input layer.
An extra bias feature is generally added (x0 = 1): it is typically represented using a special
type of neuron called a bias neuron, which outputs 1 all the time.
The Perceptron
Architecture of a Perceptron with two input neurons, one bias neuron, and three output neurons.
The Perceptron
Computing the outputs of a fully connected layer for several training instances,
hW,b (X) = ϕ(XW + b)
In this equation, X represents the matrix of input features. It has one row per instance and
one column per feature. The weight matrix, W, contains all the connection weights except
for the ones from the bias neuron. It has one row per input neuron and one column per
artificial neuron in the layer. The bias vector, b, contains all the connection weights
between the bias neuron and the artificial neurons. It has one bias term per artificial
neuron. The function, ϕ, (phi), is called the activation function: when the artificial neurons
are TLUs, it is a step function.
How is the Perceptron trained?
The Perceptron learns by adjusting its connections when it makes mistakes. It looks at one
training example at a time, makes a prediction (that is the Perceptron computes weighted
sums then applies a step function, and outputs 0 or 1), and checks if the prediction is
correct. If the prediction is wrong, it updates the weights of the inputs that should have led
to the correct answer.
If the Perceptron predicted too low (that is 0 instead of 1), the weights are increased for
the active inputs. The sum, z, is higher next time. If the Perceptron predicted too high (that
is 1 instead of 0), the weights are decreased for the active inputs. The sum, z, is lower next
time.
The Perceptron learns by trial and error. If the predictions are correct, the weights are left
as they are. If the predictions are wrong, the weights are ‘nudged’ in the right direction.
How is the Perceptron trained?
For example,
Input – x = [5, 6]
Weights – w = [-0.1, -0.2]
Correct Output = 1
Computing prediction for one TLU,
z = w1x1 + w2x2 = (-0.1)5 + (-0.2)6 = -1.1
Since -1.1 is less than 0, applying the step function results in 0.
In order to correct this, the weights will have to be made less negative and more positive,
so that z becomes larger and crosses the threshold.
How is the Perceptron trained?
For example,
Input – x = [5, 6]
Weights – w = [0.1, 0.2]
Correct Output = 1
Computing prediction for one TLU,
z = w1x1 + w2x2 = 0.1(5) + 0.1(6) = (0.5) + (0.6) = 1.1
Since 1.1 is greater than 0, applying the step function results in 1.
The prediction is now correct, and no change will made to the weights.
Perceptron Learning Rule
The following is the Perceptron Learning Rule with weight update,
𝒏𝒆𝒙𝒕 𝒔𝒕𝒆𝒑
𝒘𝒊, 𝒋 = 𝒘𝒊, 𝒋 + 𝜼 𝒚𝒋 − 𝒚ෝ𝒋 𝒙𝒊
wi,j is the connection weight between the ith input neuron and the jth output neuron.
xi is the ith input value of the current training instance.
𝑦ෝ𝑗 is the output of the jth output neuron for the current training instance.
yj is the target output of the jth output neuron for the current training instance.
𝜂 is the learning rate. (Greek letter eta)
Perceptron Learning Rule
The decision boundary of each output neuron is linear, so Perceptrons are incapable of
learning complex patterns. However, if the training instances are linearly separable, this
algorithm would converge to a solution. This is called the Perceptron Convergence
Theorem.