EELE 0531: Machine Learning
Lecture 7
Instructor: Waleed Ejaz
wejaz@[Link]
Slides adopted from Prof. Andrew Ng Founder of [Link], Co-founder Coursera, Adjunct Professor at Stanford University 1
Neural Networks
2
The Brain: The Inspiration for Neural Networks
▪ Unmatched Intelligence:
• Biological brains demonstrate superior intelligence.
▪ Neural Network’s Goal:
• Replicate the brain's learning and thinking capabilities.
• Develop software that mimics how the human brain learns and thinks.
▪ Today's Reality:
• Neural networks have evolved significantly from biological brains.
▪ Key Point:
• Biological motivations still influence the design of artificial neural networks.
3
Neural Network
What is a neuron? What is a network?
Fundamental unit (of the brain) Connected elements
Neural networks are connected elementary (computing) units
4
Biological Neurons
▪Biological neurons are the fundamental units of the
brain that
• Receive sensory input from the external world or from
other neurons
• Transform and relay signals
• Send signals to other neurons and also motor commands
to the muscles
5
Neurons in the brain
6
Ref: US National Institutes of Health, National Institute of aging
Artificial Neurons
▪ Artificial neurons replicate the basic computations of biological
neurons.
▪ Artificial neurons are the fundamental units of artificial neural
networks that
• Receive inputs/activations from the sensors or other neurons
• Combine/transform information
• Create an output/activation
Inputs Outputs
7
Why Neural Networks Now?
▪ Explosion of Data:
• The digital age has led to a massive
increase in available data.
▪ Limitations of Traditional
Algorithms:
• Traditional algorithms struggle to
utilize large datasets effectively.
▪ Neural Networks' Scalability:
• Neural networks excel at
leveraging big data to achieve
superior performance.
8
Neurons as Functions
▪ We can see a neuron as a function
• Input given by 𝑥 ∈ ℝ𝑁
• A function f can describe transformation of the input data
• Output 𝑓 𝑥 = 𝑦ො ∈ ℝ
9
Neural Network
▪ NN is a network/graph of neurons
▪ Nodes are neurons
▪ Edges represent input-output connections of the data flow
10
Neural Network as a Function
▪ The whole network is again a function
▪ Multi-layer perceptron or MLP is often seen as the neural network
11
Neural Networks are Functions
▪ Neural networks are functions
▪ Consist of connected artificial neurons
▪ Input layer takes (sensor) data
▪ Output layer provides the function result (information or command)
▪ Hidden layers do some computations
12
Deep Learning
▪ More layers=deep learning
13
What Depth Enables
▪Each layer is a processing step
▪Having multiple processing steps allows complex
functions
▪Metaphor: NN and computing circuits
• computer = sequence of Boolean gates
• neural computer = sequence of layers
▪Deep neural networks can implement complex
functions e.g., sorting on input values
14
Deep Learning’s Impact
▪ Speech Recognition
• Early success with improved speech recognition systems.
▪ Computer Vision
• Breakthrough with the ImageNet moment in 2012.
▪ Natural Language Processing
• Subsequent advancements in text understanding and generation.
15
Neural Networks for Demand Prediction
▪ Example: Predicting if a T-shirt will be a top seller.
▪ Application: Used by retailers for inventory management and
marketing.
▪ Input feature (x): Price of the T-Shirt
▪ Output: Probability of being a top seller (yes or no)
16
Logistic Regression as a Neuron
▪ Input: x=price
1
▪ Output: a = f(x) =
1+𝑒 −(𝑤𝑥+𝑏)
▪ ‘a’ represents activation function
▪ Output: Activation(a), representing the probability of being a top
seller
▪ Neuron’s Function:
• Takes one or more numbers as input. a
x
• Performs a computation.
• Outputs one or more numbers. price Probability of
being top seller
17
Building a Neural Network
▪ Combining Neurons
• A neural network is formed by connecting multiple neurons.
▪ More Complex Example:
Price Affordability
Probability of being
Shipping cost Awareness a top seller
Marketing
Material quality Perceived quality
18
Neural Network in practice
• We go through the neurons one at a time and decide what inputs to take from
previous layer.
• For large neural network, it is challenging to do it manually.
• In practice, each neuron in a certain layer will have access to every feature.
Price Affordability
Probability of being
Shipping cost Awareness a top seller
Marketing
Material quality Perceived quality
19
Layers and Activations
▪Layer:
• A group of neurons that take similar inputs and produce outputs
together
• Input Layer: Raw features (price, shipping, etc.)
• Hidden Layer: Intermediate computations (affordability,
awareness, etc.)
• Output Layer: Final prediction
▪Activations: The output values of neurons in a layer
20
Simplifying the Network
▪ Full Connectivity: Each neuron in a layer has access to all features from the
previous layer
▪ Vectorized Representation:
• Input features are represented as a vector (𝑥)
Ԧ
• Hidden layer outputs are represented as a vector (activations 𝑎)
Ԧ
• Output layer produces the final activation/prediction (a)
Price Affordability
Probability of being a
Shipping cost 𝑥Ԧ Awareness 𝑎Ԧ top seller
a
Marketing
Material quality Perceived quality 21
Activation Function
▪ Transform the weighted sum of inputs from a neuron
▪ Determine the output activation of a neuron
▪ Activation functions introduce non-linearity into the network, allowing it to
model complex relationships within data. Without activation functions,
ANNs would be limited to linear relationships, significantly restricting their
capabilities.
▪ A neuron receives weighted inputs from the previous layer.
• These weighted inputs are summed together.
• The activation function then processes this summed value.
• The output of the activation function determines the neuron's activation level, which
is then passed on to the next layer in the network.
22
Common Activation Function
▪ Sigmoid: This function outputs a value between 0 and 1, often used for tasks
involving probability-like outputs. (s(y)=1/1+e^{-y})
▪ ReLU (Rectified Linear Unit): Outputs the input directly if positive, otherwise
outputs 0. ReLUs are popular due to their efficiency in training.
▪ tanh (Hyperbolic tangent): Outputs a value between -1 and 1, and is useful
for tasks where the output can range from negative to positive values.
23
Neural Network as Logistic Regression
▪ Hidden Layer Learns Features:
• The neural network can be seen as logistic regression using learned features
(affordability, awareness, etc.) instead of raw features.
▪ Automated Feature Engineering:
• The network learns the best features for prediction, eliminating the need for
manual feature engineering
24
Neural Networks with Multiple Hidden Layers
▪Deep Neural Networks
• Networks with more than one hidden layer
▪Architecture Choices
• Number of hidden layers
• Number of neurons per hidden layer
• Impact on performance
25
Structure of ANNs
▪ Input Layer: Receives raw data (images, text, numbers)
▪ Hidden Layers: Perform computations and information processing
• These layers are the heart of the network, where complex computations and
information processing occur. There can be one or more hidden layers, each containing
a number of artificial neurons. The number of hidden layers and neurons significantly
impacts the network's capacity to learn complex patterns.
▪ Output Layer: Produces the final result (classification, prediction)
26
Example: Face Recognition
▪ Let’s assume the actual image is 1000 X1000=1Million
27
Neural Network Layer: Hidden Layer Computation
28
Neural Network Layer: Output Layer Computation
14
+
S=
1914 +
8
&
29
Neural Network Layer: Prediction
30
🔗
More Complex Neural Network
31
More Complex Neural Network: Layer 3 computation
𝑎1
𝑎Ԧ 3
= 𝑎2
𝑎3
l: Layer number
j: Unit number (neuron)
g: Activation function (sigmoid in this
case)
▪ General Representation a^[l]_j: Activation of layer 'l' unit 'j’
[𝑙] 𝑙 [𝑙] w^[l]_j: Weight vector of layer 'l' for unit 'j’
𝑎𝑗 = 𝑔(𝑤𝑗 ∙ 𝑎Ԧ 𝑙−1 + 𝑏𝑗 ) a^[l-1]: Output vector from the previous
layer (l-1)
b^[l]_j: Bias for layer 'l' unit 'j'
32
Handwritten digit recognition using Forward Propagation
▪ Binary Classification problem: Is this the digit
‘0’ or ‘1’
Probability of being a handwritten ‘1’
33
Handwritten digit recognition using Forward Propagation
▪ Digit images 0 and 1
Probability of
being a
handwritten ‘1’
34