Bayesian Hyperparameter Optimization

A neural net you must tune takes twelve minutes per epoch. Budget: 50 trials. Random search will hand 50 points to the wind; by trial 20 Optuna has already noticed that every configuration with learning rate above 0.01 and dropout below 0.2 collapsed, and it starts spending the budget where the surviving evidence is. That difference — remembering what past trials taught you — is the entire gap between sequential model-based optimization and the random search you started with in Grid and Random Search. The penalty strengths and dropout rates you chose by hand in Regularization with L1, L2, and Dropout are exactly the knobs a study searches jointly: same levers, one budget, and a surrogate that remembers.

Optuna's default is Tree-structured Parzen Estimator (TPE), and its logic is almost embarrassingly simple once stated. After several trials, sort completed trials by objective value and split them at a quantile γ (default 0.25): the top 25% are the "good" set G, the rest are "bad" set B. Then build two independent density models per parameter: l(x) from the good trials, g(x) from the bad trials. Each model is a kernel density estimate — a mixture of Gaussians centered on the observed values (a Parzen/window density), fit separately for each parameter because TPE assumes tree-structured independence.

Selection flips from "maximize the predicted objective" to "maximize the probability of landing in a good region", because TPE never models f(x) directly — it models where good and bad trials live:

acquisition ∝ [ γ + (1 − γ)·g(x)/l(x) ]⁻¹

Maximize that and you are, by construction, maximizing l(x)/(l(x) + (1−γ)g(x)) — the estimated posterior probability that x belongs to the good set. High l(x) (many good trials nearby) and low g(x) (few bad trials there) wins. Notice what the formula rewards: regions with proven good neighbors and unexplored space, since a region with no trials at all has unremarkable l and g and still competes.

import optuna
from optuna.samplers import TPESampler
from optuna.pruners import MedianPruner

def objective(trial: optuna.Trial) -> float:
    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)      # log scale, as always
    wd = trial.suggest_float('weight_decay', 1e-6, 1e-2, log=True)
    n_layers = trial.suggest_int('n_layers', 1, 4)
    width = trial.suggest_categorical('width', [128, 256, 512])
    dropout = trial.suggest_float('dropout', 0.1, 0.5)
    # ... build and train, reporting per epoch ...
    return best_val_auc

study = optuna.create_study(
    direction='maximize',
    sampler=TPESampler(seed=42, n_startup_trials=10),
    pruner=MedianPruner(n_startup_trials=10, n_warmup_steps=20),
)
study.optimize(objective, n_trials=64, timeout=3 * 3600, show_progress_bar=False)

print(study.best_value, study.best_params)
optuna.visualization.plot_optimization_history(study)   # vs the random-startup baseline

Three parameters in that snippet are policy, not plumbing:

• n_startup_trials=10 — the first 10 trials are sampled uniformly at random, because fitting l(x) and g(x) to four data points is curve-fitting noise. Too few and TPE chases outliers; too many and you paid random-search prices for a Bayesian method.
• log=True — TPE's Gaussian kernels operate on the transformed scale; a learning rate sampled linearly recreates the decade-skew problem from the grid-search topic.
• seed — a search is an experiment; an unseeded study cannot be reproduced in a design review.

TPE handles the parameter types a Gaussian process cannot handle cheaply: suggest_int, suggest_float(log=True), and suggest_categorical all work, and the tree structure means choosing n_layers=1 prunes the sub-space below it rather than wasting model capacity on the width parameter of a layer that does not exist. That is what the "tree-structured" in the name buys.

The honest baseline question an interviewer will ask: how do you know the surrogate helped? Run 10 random trials, record the best-so-far value, and compare it against the TPE curve from the same seed — Optuna's n_startup_trials gives you that baseline for free inside every study. If TPE's curve does not separate from the random prefix, your objective is too noisy, your search space is mis-specified, or the signal you are hunting does not exist. The mechanics of modeling that objective — Gaussians, expected improvement, and when Hyperopt's variant is the better tool — are next.

TPE is one member of a family. Every method in it follows the same four-step loop — fit a surrogate of the objective from completed trials, define an acquisition function, pick the trial that maximizes it, evaluate and append — and the members differ in which surrogate they fit and how expensive that fit is.

The classical Bayesian optimization surrogate is a Gaussian process: a prior over functions where any finite set of inputs has a joint multivariate normal distribution. Given n observed (x, f(x)) pairs, the GP posterior at a new x is itself Gaussian with mean μ(x) and standard deviation σ(x), tight near observed points and wide far from them. The acquisition function turns mean-plus-uncertainty into a number to maximize; expected improvement is the standard one:

EI(x) = (μ(x) − f⁺)·Φ(z) + σ(x)·φ(z), where z = (μ(x) − f⁺)/σ(x)

with f⁺ the best value seen so far, Φ and φ the standard normal CDF and PDF. The first term rewards predicted improvement; the second rewards uncertainty — the exploration bonus. When σ(x) → 0, EI collapses to the expected gain, pure exploitation; when the surrogate knows nothing there, EI is large regardless of μ, pure exploration. This trade-off is why a single high-σ trial can beat five careful ones early in a study.

The O(n³) row is the one that decides most choices: a GP refits its n×n kernel matrix on every suggestion, so by 150 evaluations a suggest call is slower than short training runs. TPE's per-parameter density models have no such cliff, which is why production Optuna studies run thousands of trials while GP-based tools are reserved for genuinely expensive objectives — 30 trials of a 6-hour simulation, where a 2-second fit is invisible.

Hyperopt is TPE before Optuna and still appears in older codebases; the vocabulary transfer directly:

from hyperopt import fmin, tpe, hp, Trials, STATUS_OK

space = {
    'lr':      hp.loguniform('lr', -5, -1),          # exp(uniform(-5, -1))
    'depth':   hp.choice('depth', [4, 6, 8]),        # categorical, returns 0/1/2 index
    'dropout': hp.uniform('dropout', 0.1, 0.5),
    'layers':  hp.choice('layers', [[64], [64, 128], [64, 128, 256]]),  # conditional
}

trials = Trials()
best = fmin(fn=objective, space=space, algo=tpe.suggest,
            max_evals=50, trials=trials, rstate=np.random.default_rng(42))

Note hp.choice('depth', ...) returns the index of the chosen option, not the option — a bug that silently trains every model at depth 4 while you believe you searched three values. Note also hp.loguniform for the learning rate: log-scale sampling remains mandatory here for the same reason it was in random search.

Three practical questions separate a working study from a decorative one. Parallelism: TPE and random search parallelize by simply running workers against one shared study object (SQLite or PostgreSQL storage), whereas GP-EI serializes — its next point depends on the last result. Objective noise: validation AUC computed once varies across trials from data order and initialization; averaging k seeds per trial or fixing seeds reduces the noise the surrogate otherwise chases. Budget accounting: 64 trials × 100 epochs × 12 s = 21 hours, the same configurations × folds arithmetic as always — and it is precisely that budget line that pruning attacks, which is the last piece.

Run the numbers before defending a pruning setup. Your study is 60 trials × 100 epochs × 12 seconds = 20 hours. Half of those trials are configuration failures — a learning rate of 0.03 that diverges by epoch 10, a dropout of 0.5 whose validation AUC has been flat at 0.55 since epoch 5. Without pruning, you pay full price for every one of them. With pruning, those trials stop at epoch 12–20 and the freed hours are spent on configurations that were still improving. Same wall-clock budget, effectively 80–100 trials instead of 60.

Pruning requires the objective to be reportable during training, not just at the end:

import optuna
from optuna.pruners import MedianPruner

def objective(trial):
    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
    model, opt = build(trial, lr)
    for epoch in range(100):
        train_one_epoch(model, opt)
        val_auc = evaluate(model, val_loader)
        trial.report(val_auc, epoch)                 # step = epoch, value = metric
        if trial.should_prune():                     # true → raise, don't return
            raise optuna.TrialPruned()
    return val_auc

study = optuna.create_study(
    direction='maximize',
    sampler=optuna.samplers.TPESampler(seed=42, n_startup_trials=10),
    pruner=MedianPruner(n_startup_trials=10, n_warmup_steps=20, interval_steps=5),
)
study.optimize(objective, n_trials=64, n_jobs=4, catch=(ValueError,))

MedianPruner implements one rule: at a given step, compare this trial's reported value against the median value of all trials that reached that step — prune if it is worse (direction-aware). The three parameters are where the judgment lives:

• n_startup_trials=10 — no pruning until 10 trials have completed; the median of three points is not a threshold, it is a coin flip.
• n_warmup_steps=20 — every trial gets at least 20 epochs. Neural nets often look identical for the first epochs and only separate after the learning rate has done its early work; pruning before that punishes slow starters, not bad configurations.
• interval_steps=5 — check every 5 epochs rather than every epoch, so a single noisy evaluation at epoch 33 does not kill a good trial. Each check is an adaptive query against the validation set, and fewer queries means less of the winner's-curse contamination you already know about.

Hyperband and ASHA are the aggressive cousins: they treat training as a resource, allocate it in halving rungs (5 → 20 → 80 epochs), and at each rung keep only the top η (η = 3 or 4) survivors — ASHA (asynchronous successive halving) does this across parallel workers without a coordinator, which is what optuna.pruners.ASHAPruner and optuna.pruners.HyperbandPruner automate. The math of what they save: if only the top 1/3 of configurations earn promotion at each of three rungs, average epochs per trial drops from 100 toward 5 + 15 + 30 ≈ 37 — a 2.7× effective budget multiplier for the trials that matter.

Pruning has four failure modes, and each is a design error rather than a tuning error:

1. Pruning on a different metric than the objective. If you prune by training loss but return validation AUC, you kill trials that were converging toward your actual goal. Report exactly what you return.
2. Pruning with the wrong direction. direction='maximize' with a reported loss inverts every comparison: the worst trials survive to completion. Optuna will not save you from this.
3. Curves that cross late. A trial that is mediocre at epoch 20 but wins at epoch 90 gets pruned at 20 under any median rule. Mitigate with generous n_warmup_steps, or prune only on a smoothed value over the last few epochs.
4. Pruning enabled while the metric is nearly uninformative. If validation AUC on a 400-example validation split swings ±0.02 epoch to epoch, every pruning decision is noise-driven. Fix the evaluation set first; a pruner amplifies an unstable metric.

Sanity-check the study when it finishes: study.trials should show a healthy mix of TrialState.COMPLETE and TrialState.PRUNED, and pruning should never dominate — if 70% of trials are pruned, your grace period is too short or your space is mostly garbage, which is a search-space problem, not a pruner problem.

You now have the full tuning stack: budget discipline from grid versus random search, a surrogate that spends trials where information is, and a pruner that stops paying for lost causes. What none of them can fix is a model that fits training data well and validation data badly for reasons no hyperparameter will reach — reading that evidence is Diagnosing the Generalization Gap.