1.
Examples: How Hyperparameter Tuning Helps in Decision Making
Example 1: Medical Diagnosis Model
Problem: Predict whether a patient has a disease using a Random Forest model.
Hyperparameters tested: n_estimators = [100, 200], max_depth = [5, 10]
Validation Results:
(100, 5) → 88%
(100, 10) → 90%
(200, 5) → 89%
(200, 10) → 93%
Decision: Choose n_estimators = 200 and max_depth = 10 because it provides the highest
accuracy.
Impact: Improves reliability of clinical decision support systems.
------------------------------------------------------------
Example 2: Spam Detection System
Model: Logistic Regression
Hyperparameter tuned: C = [0.01, 0.1, 1, 10]
Results:
0.01 → 85%
0.1 → 90%
1 → 94%
10 → 92%
Decision: Choose C = 1 for best performance.
Impact: Improves email filtering decisions.
------------------------------------------------------------
Example 3: Self‑Driving Car Object Detection (CNN)
Hyperparameters tuned: learning rate and batch size.
Results:
(0.01, 16) → Unstable training
(0.001, 16) → Good performance
(0.0001, 32) → Slow convergence
(0.001, 32) → Best performance
Decision: learning_rate = 0.001, batch_size = 32.
Impact: Improves object detection accuracy.
------------------------------------------------------------
Example 4: Credit Risk Prediction
Model: Support Vector Machine (SVM)
Hyperparameters tuned: C and gamma.
Decision: Select hyperparameters that produce the highest validation accuracy.
Impact: Helps make better loan approval and fraud detection decisions.
------------------------------------------------------------
Summary:
Hyperparameter tuning improves decision making by selecting the model configuration
that produces the best validation performance, leading to more accurate predictions
in real-world applications.
2. Hyperparameter Tuning Algorithms
1. Grid Search
Idea: Try all possible combinations of hyperparameters.
If we define:
learning_rate = [0.001, 0.01, 0.1]
batch_size = [16, 32]
Grid search tries:
(0.001,16)
(0.001,32)
(0.01,16)
(0.01,32)
(0.1,16)
(0.1,32)
Steps
1. Define hyperparameter values
2. Train model for each combination
3. Evaluate using validation set
4. Select best result
Pros
Simple
Guarantees best combination in search space
Cons
Very slow
Not scalable
When used
Small models / small search space
2. Random Search
Idea: Try random combinations instead of all combinations.
Instead of checking every point in the grid, we sample randomly.
Example:
Trial 1 → lr=0.03, batch=16
Trial 2 → lr=0.008, batch=32
Trial 3 → lr=0.05, batch=16
Steps
1. Define parameter ranges
2. Randomly sample values
3. Train model
4. Keep best result
Pros
Faster than grid search
Works well in high dimensions
Cons
May miss optimal region
Key concept
Most models are sensitive to few hyperparameters, not all.
3. Bayesian Optimization
Idea: Use past results to decide next hyperparameters.
Instead of guessing randomly, it learns:
"Which hyperparameters are likely to perform better?"
It builds a probability model of performance.
Common methods:
Gaussian Process
Tree-structured Parzen Estimator (TPE)
Used in:
Optuna
Hyperopt
Scikit-optimize
Steps
1. Try random parameters initially
2. Build surrogate model
3. Predict promising region
4. Evaluate there
5. Repeat
Pros
Efficient
Fewer experiments needed
Cons
More complex
4. Genetic Algorithm (Evolutionary Search)
Idea: Inspired by natural evolution.
Hyperparameters = “chromosomes”
Process:
selection
crossover
mutation
Steps
1. Initialize random population
2. Train models
3. Select best performers
4. Create new population
5. Repeat
Pros
Works for complex search spaces
Global optimization
Cons
Computationally expensive
5. Hyperband
Idea:
Train many models for a short time, keep only promising ones.
It uses early stopping.
Instead of fully training bad models, it stops them early.
Steps
1. Train many models briefly
2. Remove worst performers
3. Continue training best models
Pros
Very efficient
Saves compute
Cons
Requires iterative training models
Used in:
Deep learning
Neural architecture search
6. Optuna (Modern Practical Approach)
Optuna uses:
Bayesian optimization
pruning (early stopping)
Example conceptually:
trial.suggest_float("lr", 1e-5, 1e-1)
trial.suggest_int("layers", 2, 6)
Then Optuna:
runs trials
learns from results
searches smarter
Quick Comparison
Method Speed Intelligence Compute Cost
Grid Search Slow Low High
Random Search Medium Low Medium
Bayesian Fast High Low
Genetic Slow High High
Hyperband Very Fast Medium Low
Optuna Very Fast High Low
Decision Tree — GridSearchCV
from [Link] import DecisionTreeClassifier
model = DecisionTreeClassifier()
param_grid = {
"max_depth": [3, 5, 10, None],
"min_samples_split": [2, 5, 10]
grid = GridSearchCV(model, param_grid, cv=5)
[Link](X, y)
print(grid.best_params_)
How Hyperparameter Tuning is Done
Hyperparameter tuning follows four main steps.
Step 1: Choose a model
Example:
Logistic Regression
Random Forest
CNN
Suppose we choose Random Forest.
Step 2: Select hyperparameters to tune
These are parameters not learned from data.
Example (Random Forest):
n_estimators
max_depth
min_samples_split
Example search space:
n_estimators = [100, 200]
max_depth = [5, 10]
Step 3: Train multiple models
The tuning algorithm tries different combinations.
Example combinations:
Model 1 → (100, 5)
Model 2 → (100, 10)
Model 3 → (200, 5)
Model 4 → (200, 10)
Each model is trained separately.
Step 4: Evaluate using validation or cross-validation
Each trained model is evaluated on validation data.
Example:
Parameters Accuracy
(100,5) 0.89
(100,10) 0.91
(200,5) 0.90
(200,10) 0.93
Step 5: Select the best hyperparameters
Best result:
(200,10) → Accuracy = 0.93
These become the final hyperparameters.
Then we train the final model using them.
Visual Workflow
Choose model
↓
Define hyperparameter range
↓
Train multiple models
↓
Evaluate performance
↓
Select best configuration
Real Example (Deep Learning)
Suppose we tune:
learning_rate = [0.001, 0.0001]
batch_size = [16, 32]
Models trained:
(0.001,16)
(0.001,32)
(0.0001,16)
(0.0001,32)
Then choose the best validation loss.
Important Concept
Hyperparameter tuning is basically:
Training many models and selecting the best one.
The only difference between algorithms is how combinations are chosen:
Grid Search → all combinations
Random Search → random combinations
Bayesian → intelligent search
Hyperband → early stopping
Important Concept
Hyperparameter tuning is basically:
Training many models and selecting the best one.
The only difference between algorithms is how combinations are chosen:
Grid Search → all combinations
Random Search → random combinations
Bayesian → intelligent search
Hyperband → early stopping
Hyperparameter Tuning for Deep Learning
Deep learning has more hyperparameters, so smarter tuning methods are preferred.
1. Neural Networks (ANN)
Hyperparameters:
learning rate
number of layers
neurons per layer
batch size
optimizer
activation function
Example:
learning_rate = [0.001, 0.0001]
batch_size = [16, 32, 64]
Recommended tuning:
Random Search
Bayesian Optimization
Optuna
Grid search is too expensive.
2. Convolutional Neural Networks (CNN)
Hyperparameters:
number of convolution layers
filter size
number of filters
stride
learning rate
batch size
Recommended tuning:
Random Search
Hyperband
Optuna
CNN training is costly → avoid grid search.
3. RNN / LSTM
Hyperparameters:
hidden units
sequence length
learning rate
dropout
batch size
Recommended tuning:
Bayesian optimization
Random search
4. Transformer / LLM Fine-tuning
Hyperparameters:
learning rate
batch size
epochs
warmup steps
weight decay
Typical learning rate:
1e-5 to 5e-5
Recommended tuning:
Bayesian optimization
Hyperband
Optuna
Grid search is rarely used here.
Quick Comparison
Model Type Tuning Strategy
Small ML models Grid Search
Tree models Random Search
Neural networks Random Search
CNN Hyperband / Optuna
LSTM Bayesian
Transformers Bayesian / Optuna