Hyperparameter Optimization:
Hyperparameters are the parameters that define a model's architecture.
Do not change with model training.
Do not learn from model training; instead, they control the learning process.
Hyperparameters are set before training an ML model.
Each model has its own set of hyperparameters.
Examples:
Linear / Logistic Regression: C, optimization algorithm, max_itr for gradient descent
Decision trees: max depth, min leaf, criterion, min sample split
Random Forest: no. of trees, max features, learning rate, subsample
Neural network: learning rate, batch size, epochs, activation function, optimizer
Why Hyperparameter Tuning is important ?
Optimal Solution Improve Performance Avoid Poor Results
To find the optimal solution for Improve model performance, i.e., Bad settings may lead to worse
your model. accuracy, F1 score. results even if data is good.
Prevent Faster Convergence Reduce Fluctuations
Over/Underfitting Can cause faster convergence, i.e., Reduces unnecessary and random
Wrong hyperparameters can cause learning rate. fluctuations in prediction if
overfitting and underfitting. hyperparameters are tuned well.
Sensitive Models
Sensitive models like neural
networks rely on hyperparameters.
Searching for hyperparameters.
1. GridSearch:
GridSearch is a technique to tune hyperparameters. A predefined set of hyperparameters are given, and it finds the best
combination for the ML model.
There are 4 steps in GridSearch:
01 02
Define Range Evaluate Performance
Define the range of possible values for all hyperparameters. Evaluate the model performance for each hyperparameter
combination.
param_grid = {
"C": [0.01, 0.1, 1, 10],
"solver": ["liblinear", "saga"]
}
03 04
Cross Validate Pick Top Performers
Cross validate. Pick top performing hyperparameters.
grid_search = GridSearchCV(
estimator=model,
param_grid=param_grid,
scoring="roc_auc",
cv=5
)
Advantages:
Finds optimal combinations.
Any model with hyperparameters.
Cross-validation reduces overfitting risk.
Disadvantages:
Time consuming when it comes to large grids.
More hyperparameters cause exponential growth in combinations.
2. Random Search:
Tries n number of random combinations from a given hyperparameter space.
The hyperparameter values can be passed as a list or as a continuous distribution.
It doesn't matter how huge the parameter space is. We set the number of random combinations that need to be tried out.
There are 4 steps in random search:
01 02
Define Sampling Space Sample Randomly
Define the random sampling space: Sample hyperparameters randomly.
Explicit values
Or continuous distribution param_dist = { "C":
uniform(0.01, 10), "solver": ["liblinear", "saga"] }
03 04
Evaluate Performance Cross Validate & Pick
Evaluate the model performance for each randomly sampled Cross validate and pick top performing hyperparameters.
combination.
random_search = RandomizedSearchCV(
estimator=model,
param_distributions=param_dist,
n_iter=10,
combinations to try scoring="roc_auc",
cv=5,
random_state=42,
verbose=1
)
3. Bayesian optimization:
It is a probabilistic method for hyperparameter optimization.
Instead of trying all possible combinations in sample space blindly, it builds a probabilistic model of function mapping.
How it works ?
Hyperparameter sample is defined
Build a probabilistic model.
Select the next hyperparameters to evaluate.
Then evaluate selected hyperparameters
Update probabilistic model.
Repeat until convergence.
Advantages
High cost evaluations are done efficiently.
Compared to grid search, requires few evaluations
Can find optimal hyperparameters faster than random search
Disadvantages:
More complex to implement.
Choice of probabilistic model defines performance.
Not ideal for very large models.
Why random search efficient than grid search ?
Non sequential sampling: random search samples only random points, not sequential.
Random search does not give equal importance to every hyperparameter combination.
Fine Tuning:
Process of making adjustments on a pretrained model to improve its performance.
We don't train from scratch; we just manipulate parameters.
It improves accuracy, robustness, and generalization.
Fine tuning is done according to the need of the application.
Types of fine tuning:
i. Hyperparameter fine tuning:
Making adjustments in hyperparameters like learning rate, batch size, no. of epochs.
ii. Model parameter fine tuning:
Adjusting weight and biases. Usually done in deep learning.
iii. Layerwise fine tuning:
Some layers of a pretrained model are frozen, and only the last layers are trained.
iv. Optimizer fine tuning:
Choosing the right optimizers and their parameters.
We can also think of types of fine tuning in a much broader way:
1. Full parameter fine tuning:
Instead of adjusting only a few parameters, it refers to updating all trainable parameters of a model during training.
Eg: ChatGPT is likely to be full parameter fine-tuned on top of its different GPT model.
2. Domain specific fine tuning:
Instead of training on a large amount of data and heavy parameter combinations, it refers to adapting a pretrained model to a
certain domain, industry.
Goal is to make the model perform better in the target domain.
3. Specific task fine tuning:
It is the process of making a model capable of performing a specific task with high precision. The goal is to make the model excel
in one task. Eg: customer support chatbot.