Hyperparameter Tuning in
Machine Learning
Optimization Strategies for Model Performance
Presented by:
1. RAMSHA ABID (FA23-BCS-027)
2. RABAB ZAHRA (FA23-BCS-425)
Hyperparameters
Hyperparameters (in Machine Learning) are the configuration settings or values that are set before the training process begins
and control how a model learns from data, rather than being learned from the data itself.
They define aspects such as:
the structure of the model
the learning process
and the training behavior
Unlike model parameters (e.g., weights in a neural network), hyperparameters are externally chosen—often through techniques
like grid search or random search—to optimize model performance.
The Need for Tuning
Default settings are heuristic 4 rarely optimal across datasets and
models.
Accuracy impact: proper tuning reduces underfitting and overfitting by
balancing bias3variance tradeoffs.
Objective: maximize validation/generalization score while minimizing
compute and wall-clock time.
Tuning is essential for reproducible, production-grade model
performance.
What is Grid Search CV?
Grid Search CV exhaustively evaluates every combination from a user-
defined parameter grid, using cross-validation at each point to estimate
performance.
Definition Mechanism
Systematic, exhaustive search Enumerate grid → run cross-
across specified validated training for each
hyperparameter values. combination → pick best
validation score.
Analogy
Like trying every key on a keyring until the lock opens.
Iterate
Define Grid Run CV Select Best
Combos
The diagram shows a grid of parameter values evaluated systematically. Each grid point runs cross-validation; results are
compared to select the optimal hyperparameter tuple. High-contrast neon cyan marks grid points for clarity.
Pros & Cons of Grid Search
Pros Cons
Reliable: It checks every single Very Slow: If you add more
option you give, so it won't miss settings, it takes long time to
the best result. finish.
Simple to Use: Easy to set up Expensive: It uses a lot of
and always gives the same computer power (CPU/RAM).
result. Inflexible: It wastes time
Thorough: It is very detailed checking settings that might
and leaves no stone unturned not even be important.
in the list.
What is Random Search CV?
Random Search picks random settings and tests them. It tries a fixed
number of combinations.
Definition
A random way to explore and find good settings.
Mechanism
Pick random combinations → test each one → pick the best result.
High Coverage
Dense Grid: Tests close Random Scatter: Tests
together, small area spread out, big area
Narrow Range Broad Range
Random Finds: Better
Missed Zones: Grid
chance to find good
misses some areas
settings
Low Coverage
Random Search spreads tests across a wider area. Grid Search tests in a tight cluster. Random has a better chance to find good
settings.
Grid vs. Random — Direct Comparison
Feature Grid Search Random Search
Approach Systematic (exhaustive) Stochastic (random sampling)
Speed Slow — scales poorly Faster for large/continuous spaces
Best for Small, discrete search spaces Large or continuous search spaces
Computational Cost High (exponential with dims) Lower for same trial budget
Visual emphasis: chart and diagram dots use neon cyan for Grid and neon purple for Random to ensure high-contrast distinction
across visuals.
Conclusion
Grid Search: precise within a bounded discrete grid — use for low-
dimensional or final fine-grained tuning.
Random Search: efficient exploration in high-dimensional or
continuous spaces — often finds strong configurations faster.
Practical tip: combine approaches — coarse random search to identify
regions, then local grid or Bayesian methods for refinement.
Thank you for your attention!