Learning and Training
Learning
• A neural network changes when its components are changing
• a neural network could learn by
1. developing new connections,
2. deleting existing connections,
3. changing connecting weights,
4. changing the threshold values of neurons,
5. varying one or more of the three neuron functions (remember: activation
function, propagation function and output function),
6. developing new neurons, or
7. deleting existing neurons (and so, ofcourse, existing connections).
Paradigms of Learning
• Supervised Learning
• Unsupervised Learning
• Reinforcement Learning
Unsupervised Learning
• Only the input patterns are given
• the network tries to identify similar patterns and to classify them into
similar categories
• The training set only consists of input patterns, the network tries by
itself to detect similarities and to generate pattern classes.
• Example : self-organizing maps
Reinforcement learning
• The training set consists of input patterns,
• after completion of a sequence
• a value is returned to the network indicating whether the result was
right or wrong
Supervised learning
• The training set consists of input patterns with correct results
• network can receive a precise error vector
• Entering the input pattern
• Forward propagation
• Compare the output with the desired output
• Corrections of the network are calculated based on the error vector
• Corrections are applied
Training patterns and teaching input
• input vector
• output vector
• training sample
• teaching input (Desired Output)
• error vector
Using training samples
• Selection of training data and the learning curve.
• Task :
• exactly mark the colored areas around the training samples with the output 1
Visualization of training results of the same
training set on networks with a capacity too LOW
Visualization of training results of the same
training set on networks with a capacity - correct
Visualization of training results of the same
training set on networks with a capacity Too High
Generalization
• A model’s ability to perform well on new, unseen data that was not
part of the training set.
• During training a model, it learns patterns from the training data.
• But the real goal is not just to "memorize" training data
• It should capture the underlying patterns so that it can make accurate
predictions on test data or real-world data.
• This ability to "transfer" what was learned from training data to unseen
data is called generalization
Example
• Suppose you train a model to recognize cats and dogs using 1000
pictures.
• If the model does well only on those same 1000 pictures, but fails
badly on new ones, it has poor generalization
• If the model correctly classifies new cat/dog images it hasn’t seen
before, it has good generalization.
Generalization
• Underfitting → Model is too simple, cannot capture training patterns,
poor generalization.
• Overfitting → Model memorizes training data too much, poor
generalization.
• Bias–Variance trade-off → Finding the balance between underfitting
and overfitting to improve generalization.
• Generalization is the model’s ability to make accurate predictions
on unseen data, not just the data it was trained on.
Divide the set of training samples
• Poor Generalization can be overcome by divide
• one training set really used to train ,
• one verification set to test progress
• 70% for training data
• 30% for verification data
Learning curve and error measurement
• Learning Curve
• A learning curve shows how error changes as training progresses.
• It visualizes if the network is improving or stuck.
• The error is treated as a distance between the correct output and the network’s
output.
Types of Error Measurement
• Specific Error (online)
• Root Mean Square (RMS) Error
• Total Error (offline)
Specific Error (online)
• It refers to the error computed for a single training sample (one
input-output pair).
• It is called online error because it is measured immediately during
training for each sample
Specific Error (online)
Specific error (online) = error calculated for one training sample at a time, used in online learning to immediately update
network weights.
Root Mean Square (RMS) Error
• The RMS error is a way of measuring the average error between the
network’s predicted outputs and the target outputs, but it gives more
weight to larger errors
Root Mean Square (RMS) Error
Total Error (Offline)
• The total error is the sum of all specific errors (or RMS errors) over
the entire training dataset.
• It is called offline because it is computed after processing all training
samples in one epoch (not after each individual sample).
Total Error (Offline)
Euclidean Distance (Error Measure)
• The Euclidean distance measures the straight-line distance between the
target output vector 𝑡 and the network output vector 𝑦.
Euclidean Distance (Error Measure)
• It is the L2 norm (Euclidean norm) of the error vector.
• It measures the distance between predicted output and the actual
output
• RMS error also uses squared differences, but averages them (divides
by number of outputs) and then takes square root.
• Euclidean distance is similar but does not average — it directly sums
squared errors before taking the square root.
Euclidean Distance (Error Measure)
low-dimensional outputs – Euclidean Distance - visualize the error as a straight-line distance in
space.
high-dimensional outputs – RMS -normalizes the error by the number of outputs.
Learning Curve and Its Behavior
• General Exponential Decay (Perfect Learning Curve)
• A learning curve shows how the error (or loss) changes with training epochs
(rounds).
• X-axis → Epochs (time or training progress)
• Y-axis → Error (mistakes made by the model)
General Exponential Decay (Perfect Learning Curve)
Logarithmic Scale Representation
• Learning curves are plotted on a logarithmic scale for error.
• Exponential curves become straight lines in a log scale.
• Visualize constant rates of improvement
Spikes at the Bottom
• While training a model, the error curve (training or validation error)
usually decreases smoothly.
• But near the bottom of the curve (close to minimum error), instead of
being perfectly flat, we will visualize small up-and-down fluctuations
(spikes).
Spikes at the Bottom
Flat Areas in Learning Curves
• In a learning curve (Error vs. Epochs or Accuracy vs. Epochs), expect
to see error decrease (or accuracy increase) steadily as training
progresses.
• However, the curve shows horizontal flat regions — where the error
doesn’t improve (or accuracy doesn’t increase) for some epochs.
• These are called flat areas or plateaus.
Flat Areas in Learning Curves
• Flat regions can be caused by:
1. Learning Rate Issues
2. Optimization Landscape
3. Data Complexity
4. Regularization or Early Bottlenecks
• Temporary vs. Permanent Flat Areas
• Temporary Flat Areas: The curve is flat for a while but then continues
improving after some epochs
• Permanent Flat Areas: The curve stays flat and does not improve further
→ indicates convergence
Flat Areas in Learning Curves
Gradient Descent
• Optimization Algorithm used in neural network
• To find the best weights
• Minimizes the loss function
• Gradient descent is the process of adjusting weights step by step to
reduce error and make the neural network learn.
• local minima
• local maxima
• global minimum
Procedure
• Start with random weights
• Make a prediction (Forward Pass)
• Calculate the error (Loss Function)
• Compute the gradient (Backpropagation)
• Update weights
Gradient Descent
Types
• Batch Gradient Descent
• Stochastic Gradient Descent (SGD)
• Mini-Batch Gradient Descent
Batch Gradient Descent
• Uses the entire dataset to compute the gradient at each step.
• Very accurate updates, but slow for large datasets.
• Often used in small/medium datasets.
• Example:
• If you have 1,000 training samples, all 1,000 are used to compute the gradient
before taking one step.
Stochastic Gradient Descent (SGD)
• Uses only one sample at a time to update parameters.
• Much faster, but updates are noisy (zig-zag path).
• Helps escape local minima due to randomness.
• Example:
• If you have 1,000 samples, SGD updates the weights 1,000 times per epoch
(once per sample).
Mini-Batch Gradient Descent
• Divides dataset into mini-batches (e.g., 32, 64, 128 samples).
• Each update uses one batch → faster and less noisy than SGD.
• Most commonly used in deep learning.
• Example:
• If you have 1,000 samples and a batch size of 100 → you get 10 updates per
epoch.
Hebbian Learning
• Neurons that fire together
• if two neurons activate at the same time, the connection (weight)
between them is strengthened.
Delta Learning Rule
• It’s a supervised learning rule
• used to adjust the weights of a neuron
• so that the predicted output is closer to the desired target output.
• limitation of Hebbian learning is overcome by using Delta Learning
Rule
• Hebbian Learning : only strengthens weights when both input &
output are active
• Delta Learning Rule : adjusts weights based on the error, so it learns
correct mappings
Delta Learning Rule
Convergence
• After some number of iterations, the weights and bias stop changing
significantly
• they reach stable values.