NERSCPowering Scientific Discovery Since 1974

Hyperparameter Optimization

In machine learning, parameters are the values that describe a machine learning model and are usually chosen by a learning algorithm, like the weights in linear regression or a neural network. Hyperparameters, on the other hand, are parameters for the learning algorithm, like the learning rate for gradient descent or the number of layers of a neural network.  

The process of looking for the most optimal hyperparameters for a machine learning algorithm is called hyperparameter optimization

There are several common hyperparameter optimization algorithms:

  • Grid Search
    • A brute force search of every combination of hyperparameters
  • Random Search
    • Randomly sample and evaluate sets of hyperparameters specified by a probability distribution
  • Bayesian Hyperparameter Optimization
    • Use a statistical mode to help choose which set of hyperparameters to explore next based on the performance of past sets of hyperparameters

We support a few pieces of software for hyperparameter optimization. 

 

Single Node

Multinode

 

 

 

Spearmint - Bayesian Hyperparameter Optimization

Spearmint is a Python Bayesian optimization codebase. Read More »