1.
Hyperparameter tuning
Hyperparameter tuning is the process of finding the optimal values of hyperparameters to
make a machine learning model perform best.
Parameters → learned from data (e.g., weights in neural networks)
Hyperparameters → set before training (e.g., learning rate)
Goal: Improve accuracy, efficiency, generalization and avoid overfitting or underfitting and
achieve higher accuracy on unseen data.
1.1 Types of Hyperparameters
(A) Model Hyperparameters
Model hyperparameters are external configuration variables set by data scientists before
training to control the machine learning process, such as learning rate, batch size, and number
of layers. Unlike parameters learned from data, these define structure and optimization
efficiency.
(B) Optimization Hyperparameters
The process of finding the optimal configuration variables such as learning rate, batch size, or
network layers to maximize a machine learning model's performance.
It involves systematically testing different hyperparameter combinations to minimize the
error or maximize accuracy
(C) Regularization Hyperparameters
Regularization hyperparameters are configuration variables set by data
scientists before training a machine learning model to manage its complexity, prevent
overfitting, and improve generalization to new data.
Learning Rate
Controls how fast the model updates weights
Too high → overshooting
Too low → slow learning
Example:
Learning rate: 0.001 → 0.1
Batch Size
Number of samples processed at once
Small → noisy but faster learning
Large → stable but needs more memory
Batch size: 16, 32, 64
Epochs
Number of times the entire dataset is used
Too many → overfitting
Epochs: 10 → 100
Regularization
Adds penalty to reduce model complexity
Momentum
Momentum is the degree to which models update parameters in the same direction as
previous iterations, rather than reversing course. Most data scientists begin with a
lower hyperparameter value for momentum and then tweak upwards as needed to
keep the model on course as it takes in training data.
1.2 Techniques for Hyperparameter Tuning
1. Grid Search
Brute-force technique for hyperparameter tuning
It trains the model using all possible combinations of specified hyperparameter values to find
the best-performing setup.
It is slow and uses a lot of computer power which makes it hard to use with big datasets or
many settings.
2. Random Search
Picks random combinations of hyperparameters from the given ranges
In each iteration it tries a new random combination of hyperparameter values.
It records the model’s performance for each combination.
After several attempts it selects the best-performing set.
3. Bayesian Optimization
It treats hyperparameter tuning like a mathematical optimization problem and learns from
past results to decide what to try next.
Build a probabilistic model (surrogate function) that predicts performance based on
hyperparameters.
Update this model after each evaluation.
Use the model to choose the next best set to try.
Repeat until the optimal combination is found.
Workflow of Hyperparameter Tuning
1. Select algorithm
2. Define hyperparameters
3. Choose value ranges
4. Select tuning method
5. Train models with combinations
6. Evaluate using validation set
7. Select best model
8. Test on unseen data
Advantages
Improves accuracy
Better model generalization
Reduces bias and variance
Disadvantages
Time-consuming
Requires computational power
Needs expertise