•
•
•
•
•
•
•
•
Source: The following slides contain content, slides, and images adapted from:
1. [Link]
2. Dive into Deep Learning [Link]
3. NVIDIA Edge AI and Robotics course
4. Various internet sources
Source: Internet
Source: Internet
Source: Internet
Source: Internet
Output
Activation function
(sum)
(weights) is the step function
Input
Source: [Link]
Source: Internet
Source: Internet
Source: Internet
Source: Internet
Output
Activation function
(sum)
a nonlinear activation function (sigmoid, tanh, ReLU, …)
(weights)
Input
Source: [Link]
Feature Perceptron Artificial Neuron (in ANNs)
A simple, single-layer neural model used for A general term for computational units in
Definition
binary classification. artificial neural networks (ANNs).
Consists of input weights, a summation Consists of input weights, a summation
Structure function, and a step activation function function, and an activation function (ReLU,
(threshold-based). Sigmoid, Tanh, etc.).
Uses various non-linear activation functions
Activation Function Uses a step function (outputs only 0 or 1). (ReLU, Sigmoid, etc.), enabling more complex
learning.
Can solve both linearly and non-linearly
Can only solve linearly separable problems
Learning Capability separable problems when used in multi-layer
(e.g., AND, OR). Fails on XOR.
networks.
Used in multi-layer perceptrons (MLPs) and
Network Type Used in single-layer networks.
deep neural networks (DNNs).
Early neural networks for basic classification
Example Usage Used in modern deep learning architectures.
tasks.
Regression Models
Output
(sum)
(weights)
Input
Source: [Link]
Regression Models
Output
Activation function
(sum) Linear activation function
(weights)
Input
Source: [Link]
Regression Models
Output
Activation function
(sum)
(weights) is the logistic function
Input
Non-linear activation function / binary classifier
Source: [Link]
• “How to get from linear weighted sum input to non-linear output?”
Name Plot Function Description
1 The most common activation
Logistic (sigmoid) 0
function. Squashes input to
x
(0,1).
1
Hyperbolic tangent 0 x Squashes input to (-1, 1).
(tanh) -1
Popular activation function.
Rectified Linear Unit Anything less than 0, results
(ReLU) 0 in zero activation.
x
Derivatives of these functions are also important (gradient descent).
Source: [Link]
• “How to output/predict a result”
Problem Description Name Function
Binary • Output probability for each class, in (0,1)
classification • Logistic regression of output of last layer Sigmoid
• Output probability for each class, in (0,1)
Multi-class
• Sum of outputs to be 1 (probability distribution)
classification • Training drives target class values up, others down Softmax
Regression Linear/ ReLU
Source: [Link]
Source: Internet
1
0.73
0.5
0.12
-2 0 1
Source: Internet
𝒐(𝒐𝒖𝒕)
We build a neural network for a binary
Output Layer classification task, with:
(𝒊𝒏)
𝒐
• (no bias, for simplicity)
• 2 inputs: = 0.5 and = 0.1
𝒉𝟏(𝒐𝒖𝒕) 𝒉𝟐(𝒐𝒖𝒕) Hidden Layer • 1 hidden layer with 2 neurons
𝒉𝟏 (𝒊𝒏)
𝒉𝟐 (𝒊𝒏)
• 1 output neuron in the output layer
• All neurons have sigmoid activation function:
Input Layer
Source: [Link]
𝒐(𝒐𝒖𝒕)
Output Layer
(𝒊𝒏)
𝒐
0.4 0.45
0.52 0.53
Hidden Layer
0.1 0.13
0.25 0.2
0.15 0.4 Similarly,
0.5 0.1 Input Layer
Source: [Link]
0.61
Output Layer
0.44
0.4 0.45
0.52 0.53
Hidden Layer
0.1 0.13
0.25 0.2
0.15 0.4
For binary classification, we would classify
0.5 0.1 Input Layer this (0.5, 0.1) input data point, as class 1
(as 0.61 > 0.5).
Source: [Link]
• “How to compare the outputs with the truth?”
Problem Name Function Notes
Notations for Classification
Binary Cross entropy for
• = training examples
classification logistic • = classes
• = prediction (probability)
• = true class (1/yes, 0/no)
Multi-class Cross entropy for
classification Softmax
Notations for Regression
• = training examples
Regression Mean Squared
• = prediction (numeric, )
Error • = true value
Source: [Link]
Training
C(w)
•
• w
Source: [Link]
•
: Cost
Gradient with respect to
Source: [Link]