Scribd
Upload
10 views2 pages

Hyperparameter Optimization

The document discusses hyperparameter optimization in the context of large-scale machine learning systems, highlighting the importance of setting hyperparameters correctly to avoid issues such as slow convergence and poor generalization. It defines hyperparameters and lists several examples, including step size, batch size, and regularization, along with the potential negative impacts of inefficient settings. The document also mentions the role of theory in guiding hyperparameter selection for optimal performance in learning algorithms.

Uploaded by

balaramamurty.ece
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
10 views2 pages

Hyperparameter Optimization

The document discusses hyperparameter optimization in the context of large-scale machine learning systems, highlighting the importance of setting hyperparameters correctly to avoid issues such as slow convergence and poor generalization. It defines hyperparameters and lists several examples, including step size, batch size, and regularization, along with the potential negative impacts of inefficient settings. The document also mentions the role of theory in guiding hyperparameter selection for optimal performance in learning algorithms.

Uploaded by

balaramamurty.ece
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Hyperparameter Optimization

CS4787/5777 --- Principles of Large-Scale Machine Learning


Systems
 Prelim next week
 Practice prelim exams released

So far in class we've talked about:

 learning algorithms, such as SGD;


 ways to accelerate these learning algorithms, such as momentum;
 and deep learning.

Each one of these methods introduces many hyperparameters that we need to set
for the algorithms to run efficiently.

And we saw that setting the hyperparameters suboptimally can lead to slow
convergence!
Formally: what is a hyperparameter?

A hyperparameter is a parameter that is set before the main learning algorithm is


run, in contrast to other model parameters that are assigned during learning.

Question: What hyperparameters have we seen so far in this course? What bad
things can happen if we set them inefficiently?

 step size/learning rate


 divergence
 take too long to learn
 get lost in a suboptimal region (for nonconvex opt)
 batch size
 can affect "noise ball"
 can blow up memory
 can result in suboptimal use of hardware
 regularization λ
 high bias or high variance
 generalize poorly
 choice of nonlinearity in a neural network
 architectural choices in NN
 bad stuff
 number of epochs
 high loss vs poor generalization
 training time
 momentum β
 adam parameters ρ1 and ρ2
 make you converge slower
 initialization
 affects learning & generalization (for deep nets)

Examples of hyperparameters:
 step size α
 momentum β
 batch size B
 etc.

One way to get a feeling for what hyperparameters do is theory. We've seen some
amount of theory in this course. One thing that theory can do is give us a recipe
for assigning hyperparameters.

 For example, theory can tell us how to set hyperparameters for strongly
convex optimization if we know the strong convexity constant μ and the
Lipschitz constant L.
 e.g. α=1L for gradient descent
 Setting hyperparameters according to theory generally gives us the right
asymptotic rate.

 And it gives us a good sense of what order of magnitude the hyperparameters


should probably have, and how they should scale with problem parameters
and other hyperparameters.

 We can even (sometimes) get the optimal setting of hyperparameters for a


given problem class. For example, for convex optimzation, we can find the
best setting of hyperparameters that makes the algorithm converge for any
strongly convex, Lipschitz continuous function with that μ and L.
 But it can be loose relative to the best setting of the hyperparameters for a
particular problem/dataset!

You might also like