TRAINING SIMPLE MACHINE LEARNING ALGORITHMS
Perceptron, Adaline & Gradient Descent
From rule-based updates to continuous optimization of a cost function
Dr. Md. Shahidur Rahman
Professor, DoCSE, SUST
rahmanms@[Link]
TODAY'S SESSION
Roadmap
1 Perceptron & Adaline
Why a continuous activation function changes everything
Learn weights by minimizing a
2 The Cost Function differentiable cost function
Sum of squared errors as an optimization objective
3 Gradient Descent
Climbing down the cost surface, one step at a time
4 Feature Scaling
Standardization speeds up convergence
5 SGD & Mini-Batch
Scaling learning to large, streaming datasets
2
O RI GI NS
A brief history of the artificial neuron
1943 Today
McCulloch & Pitts Foundations
Proposed the McCulloch–Pitts (MCP) neuron: These ideas underpin every modern
a simplified model of a brain cell, aiming to classifier: a weighted sum of inputs
understand how biological neurons could compared against a threshold.
inspire artificial intelligence.
1957
Frank Rosenblatt
Published the perceptron learning rule,
building on the MCP model — the first
algorithm that could automatically learn
optimal weights from data.
3
THE INSPIRATION
From biological neuron to logic gate
Biological neuron
The MCP reframing
McCulloch and Pitts described this behavior in
computational terms:
A simple logic gate with binary output
Inputs are integrated (summed) in the cell
body
An output fires only if the accumulated signal
exceeds a threshold
Multiple signals arrive at the dendrites and are
integrated in the cell body. When the accumulated This binary, threshold-based logic gate is the seed
signal exceeds a threshold, the neuron fires and passes from which the perceptron grows.
a signal along the axon.
4
THE MATH
The net input: a weighted sum
For an input vector x and weight vector w, the perceptron
computes a net input z as their dot product: Decision rule
Compare z to a threshold θ:
z = w₀x₀ + w₁x₁ + … + wₘxₘ = wᵀx z ≥ θ → predict class +1
x₀ = 1 is a constant input tied to the bias weight w₀ (the
z < θ → predict class –1
negative threshold).
Worked example Folding –θ into w₀ lets us write the rule
compactly as z ≥ 0 vs. z < 0. The value w₀
[1 2 3] × [4 5 6]ᵀ = 1×4 + 2×5 + 3×6 = 32 = –θ is called the bias unit.
5
THE MATH
Squashing z into a binary output
Discriminating two linearly separable classes
decision
boundary
If the accumulated net input meets the threshold,
the neuron outputs +1; otherwise, it outputs –1.
6
THE ALGORITHM
The perceptron learning rule
A reductionist approach mimicking a single neuron: it either fires, or it doesn't.
1 2a 2b →
Initialize Predict Update Repeat
For each training Pass over the training
Update every weight wⱼ
Set weights to 0 or example x⁽ⁱ⁾, compute set (epoch) until
simultaneously based
small random numbers. the predicted output ŷ convergence or a max-
on the prediction error.
using the step function. epoch limit.
7
THE ALGORITHM
Updating the weights
Each weight is updated simultaneously as:
Two quick examples
Misclassified as –1, with xⱼ⁽ⁱ⁾ = 0.5:
wⱼ := wⱼ + Δwⱼ
Δwⱼ = (1 – (–1))(0.5) = 1
Misclassified as –1, with a larger xⱼ⁽ⁱ⁾ = 2:
Δwⱼ = η (y⁽ⁱ⁾ – ŷ⁽ⁱ⁾) xⱼ⁽ⁱ⁾
Δwⱼ = (1 – (–1))(2) = 4
η learning rate, typically between 0.0 and 1.0
Larger input values push the decision boundary
y⁽ⁱ⁾ the true class label of example i
by a proportionally larger amount — the update
ŷ⁽ⁱ⁾ the predicted class label of example i scales with xⱼ⁽ⁱ⁾.
8
THE ALGORITHM
When do weights actually change?
Correct Correct Wrong Wrong
True label +1, True label –1, True label +1, True label –1,
predicted +1 predicted –1 predicted –1 predicted +1
Δwⱼ = η(1–1)x = 0 Δwⱼ = η(–1–(–1))x = 0 Δwⱼ = η(1–(–1))x = 2ηx Δwⱼ = η(–1–1)x = –2ηx
No update — the No update — the Weights pushed Weights pushed
prediction was prediction was toward the positive toward the negative
already right. already right. class. class.
9
A CAVEAT
Convergence depends on separability
Convergence is only guaranteed if the two classes are linearly separable and the learning rate is small
enough. Otherwise, the perceptron will never stop updating — so we cap training with a maximum
number of epochs and/or a tolerated error threshold.
Linearly separable Not linearly separable Not linearly separable
– +
– –
– – +
–
– + –
– –
– + + –
+ –
+ – + – + + +
+ + + –
– + + –
– + – + – –
– –
–
10
PUTTING IT TOGETHER
The perceptron, end to end
Error
Output
(+1 or –1)
Inputs are combined with weights into a net input, passed through a threshold function, and turned into
a binary prediction. During learning, the resulting error flows back to update every weight.
11
SECTION 1
Perceptron vs. Adaline
Same inputs, same weighted sum — a different signal drives the weight update.
PERCEPTRON
Inputs Net input Unit step Predicted
(x, 1) Σ wᵢxᵢ threshold label
Weights update using the PREDICTED CLASS LABEL vs. the true label (integer comparison).
ADALINE
Inputs Net input Linear Threshold
(x, 1) Σ wᵢxᵢ activation ϕ(z) (prediction)
Weights update using ( true label − ϕ(z) ) — the CONTINUOUS activation output, before thresholding.
3
SECTION 2
Defining the Cost Function
Every supervised learner needs an objective
COST FUNCTION
function to optimize during training.
● Adaline defines cost J(w) as the Sum of Squared
Errors (SSE) between the true label and the J(w) = ½ Σᵢ ( y⁽ⁱ⁾ − ϕ(z⁽ⁱ⁾) )²
continuous output ϕ(z)
● The ½ term is a convenience factor — it cancels y⁽ⁱ⁾ true class label for example i
cleanly when we differentiate z⁽ⁱ⁾ net input, wᵀx⁽ⁱ⁾
● Because ϕ(z) is a linear (identity) function, J(w) ϕ(z) linear activation, ϕ(z) = z
is differentiable and convex
Why not just count misclassifications?
● A convex, differentiable cost function
A 0/1 misclassification count is flat almost
guarantees gradient descent can find the global everywhere — its gradient is zero, so there is no
minimum signal to guide learning. SSE is smooth instead.
4
SECTION 3
Gradient Descent: Climbing Down the Hill
The step-by-step idea
1 Start from an initial (often random) weight
vector w
2 Compute the gradient ∇J(w) — the slope of the
cost surface at the current point
3 Take a step in the OPPOSITE direction of the
gradient
4
Step size is controlled by the learning rate η
5 Repeat until the cost stops decreasing
meaningfully
w := w + Δw Δw = −η∇J(w)
5
SECTION 3
Deriving the Weight Update Rule
Partial derivative of cost w.r.t. wⱼ ∂J/∂wⱼ = − Σᵢ ( y⁽ⁱ⁾ − ϕ(z⁽ⁱ⁾) ) xⱼ⁽ⁱ⁾
Weight change for wⱼ Δwⱼ = −η (∂J/∂wⱼ) = η Σᵢ ( y⁽ⁱ⁾ − ϕ(z⁽ⁱ⁾) ) xⱼ⁽ⁱ⁾
Simultaneous update, all weights w := w + Δw
Key distinction from the perceptron: ϕ(z⁽ⁱ⁾) here is a real-valued number, not an integer class label — and
the update sums over the entire training set before any weight changes, which is why this is called batch
gradient descent.
6
SECTION 3
Choosing the Learning Rate Matters
Too large → overshoots the minimum; cost Too small → converges safely, but needs far too many
grows every epoch instead of shrinking. epochs to be practical.
7
SECTION 4
Improving Convergence with Feature Scaling
STANDARDIZATION
x′ⱼ = ( xⱼ − μⱼ ) / σⱼ
Zero-mean, unit-variance features reshape
the cost surface into rounder, more
symmetric contours.
Result: gradient descent takes fewer,
more direct steps to reach the minimum.
In NumPy: X_std[:, j] = (X[:, j] - X[:, j].mean()) / X[:, j].std()
Note: standardization does not make the data normally distributed — it only recenters and rescales each feature.
After scaling, Adaline converges within about 15 epochs at η = 0.01, and every example is classified correctly —
though SSE never quite reaches zero.
8
SECT I O N 5
Batch Gradient Descent Doesn't Scale
Every single weight update re-evaluates the ENTIRE training set — fine for the
100-row Iris dataset, but costly once we reach millions of examples.
Batch GD Stochastic GD
Δw = η Σᵢ (y⁽ⁱ⁾ − ϕ(z⁽ⁱ⁾)) x⁽ⁱ⁾ Δw = η (y⁽ⁱ⁾ − ϕ(z⁽ⁱ⁾)) x⁽ⁱ⁾
One update per epoch, using all m examples One update per training example — noisy but
— stable but slow at scale. fast, and enables online learning.
Both share the same underlying idea — the difference is how much data each weight update looks at
before moving.
9
SECT I O N 5
Mini-Batch: The Practical Middle Ground
Applies batch gradient descent to small subsets of the data (e.g., 32 examples at a time) — combining
faster convergence with vectorized, efficient computation.
Batch GD Stochastic GD Mini-Batch GD
Update frequency Per epoch (all m) Per example Per small batch
Gradient noise None — smooth High — noisy Moderate
Convergence speed Slow per update Fast, erratic path Fast & efficient
Best suited for Small datasets Streaming / online Large-scale training
For most modern deep learning workloads, mini-batch gradient descent is the default choice — it
balances speed, stability, and hardware efficiency.
11
RECAP
Key Takeaways
Adaline updates weights using a continuous linear activation, not the thresholded prediction
— making its cost function differentiable
SSE is a convex, differentiable cost function — the ideal setting for gradient descent
The learning rate η trades off stability against speed of convergence
Standardizing features reshapes the cost surface and speeds up convergence dramatically
Batch, stochastic, and mini-batch gradient descent trade computation cost against update
frequency and noise
12