In machine learning, parameters are the values that describe a machine learning model and are usually chosen by a learning algorithm, like the weights in linear regression or a neural network. Hyperparameters, on the other hand, are parameters for the learning algorithm, like the learning rate for gradient descent or the number of layers of a neural network.
The process of looking for the most optimal hyperparameters for a machine learning algorithm is called hyperparameter optimization
There are several common hyperparameter optimization algorithms:
- Grid Search
- A brute force search of every combination of hyperparameters
- Random Search
- Randomly sample and evaluate sets of hyperparameters specified by a probability distribution
- Bayesian Hyperparameter Optimization
- Use a statistical mode to help choose which set of hyperparameters to explore next based on the performance of past sets of hyperparameters
We support a few pieces of software for hyperparameter optimization.
- Great for:
- Bayesian Optimization