Bayesian optimization
Beginner
A smart way to search for good settings when each try is slow. It keeps a running guess about which settings are good, and picks the next try to learn the most.
Novice
A way to search for good settings when each try is slow. It fits a surrogate model to the tries so far, then picks the next try where the model predicts a good result or is most unsure.
Expert
Usually a Gaussian process as the surrogate plus an acquisition function, such as expected improvement, that scores where to sample next. Works best below about 20 continuous settings with slow, noisy evaluations. TPE, a related method, models where good and poor samples fall instead.
Explained in AI in EDA (Extras).
See also: Surrogate model, Design-space exploration (DSE).