Grid and Random Search

You are tuning an SVM classifier on 40,000 rows. You pick four hyperparameters — C, gamma, class_weight, and the kernel — with five sensible values each. That is 5^4 = 625 configurations, and with 5-fold cross-validation at roughly 20 seconds per fit, the whole sweep is 625 x 5 = 3,125 fits, about 17 hours of compute. You launch it overnight. The next morning a colleague reports that a 60-configuration random search found a better model in 2 hours. Same team, same data, wildly different outcomes. That is not luck. It is geometry.

Grid search carries a hidden assumption: every dimension of the search space matters equally. In real problems one or two hyperparameters drive the metric and the rest only bend it slightly. A grid allocates trials uniformly across all dimensions, so nearly every trial varies parameters that do not matter, while the influential parameter takes only five distinct values — the five you guessed. If those guesses land in a dead zone (all C values above 10 when the useful range is 0.01 to 1), no amount of grid resolution rescues you. The answer was never on the grid.

Random search decouples budget from resolution. Every trial draws the influential parameter afresh from its entire range, so N trials give N distinct values of it. The probability that at least one draw lands in a good band occupying fraction q of the range is:

P(hit) = 1 - (1 - q)^N

With q = 0.1, N = 30 gives 1 - 0.9^30 which is about 95.8%. Note what is missing from that bound: the dimension count. The nuisance parameters can sit anywhere, because varying them costs nothing extra in expectation. This is the 2012 Bergstra and Bengio result that settled the debate — grid search evaluates the response surface on a lattice and wastes almost all trials re-evaluating unimportant parameters, while random search covers more distinct values of every important parameter at equal budget. Their experiments showed random matching or beating grid on most benchmark problems.

The arithmetic for our SVM example:

18,750 configurations to guarantee what 30 random trials deliver with 95% probability. That table is the entire argument.

Grid search is not useless. It wins when you have one or two dimensions and need a readable response curve — "report AUC at C in {0.01, 0.1, 1, 10, 100}" is a deliverable, not a search — and in regulated settings where the trial list must be pre-declared and auditable. As the default for three or more hyperparameters under a fixed budget, though, random search is the correct first choice. When you pitched the tuning plan to your product manager in Communicating Results to Stakeholders, the credibility of the number you quoted rested on this budget arithmetic. Every more advanced method — including the surrogate-guided search in Bayesian Hyperparameter Optimization — is built on the same insight: spend trials where they buy information, not on a lattice.

Two rules before you launch anything. First, count budget in configurations, not fits: 60 configurations times 5 folds is 300 model fits, and you should be able to state that number before the job starts. Second, seed every search (random_state=42). You will need to reproduce the winner, and "the search was random" is not an answer you can give in a design review.

Random search fixes budget allocation. It creates a second problem the moment you draw from the wrong distribution: the sampling distribution decides where your trials actually go, and most hyperparameters are useless when sampled uniformly on their raw scale.

Take a learning rate. You want anywhere from 1e-5 for careful fine-tuning to 1e-1 for aggressive training. A uniform draw on [1e-5, 0.1] puts 90% of its mass above 0.01, because the interval is 0.09 wide and 0.089 of that width is the top decade. The chance of a uniform sample landing below 1e-3 is 1e-3 / 0.1 = 1%. Run 60 uniform trials and you get roughly 0.6 samples in the low-learning-rate band, while 54 trials go to values that make training diverge or oscillate. Now sample on the log scale — draw u uniform from log10(1e-5) to log10(0.1), return 10^u — and every decade gets equal probability: 60 trials spread about 15 per decade across four decades. Same budget, orders-of-magnitude better coverage of the range that matters.

The rule: anything that acts multiplicatively must be sampled on a log scale. Learning rates, regularization strengths (C, alpha, lambda), kernel gamma, weight decay, and batch sizes all live on multiplicative scales where the difference between 1e-4 and 1e-3 matters more than the difference between 0.4 and 0.5. Truly bounded, near-linear quantities — dropout rate between 0.1 and 0.5, tree depth between 3 and 12, momentum between 0.8 and 0.99 — can be sampled linearly without waste.

scikit-learn makes the distinction explicit with scipy.stats.loguniform inside RandomizedSearchCV. The full pattern, with the scaling sealed inside a pipeline:

from scipy.stats import loguniform
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold

pipe = Pipeline([('scaler', StandardScaler()), ('clf', SVC(probability=True))])

dists = {
    'clf__C':      loguniform(1e-3, 1e3),   # six decades, equal mass each
    'clf__gamma':  loguniform(1e-4, 1e-1),
    'clf__kernel': ['rbf', 'poly'],
    'clf__degree': [2, 3, 4],               # only affects 'poly' trials
}

search = RandomizedSearchCV(
    pipe, dists, n_iter=60, scoring='roc_auc',
    cv=StratifiedKFold(5, shuffle=True, random_state=42),
    random_state=42, n_jobs=-1, refit=True,
)
search.fit(X_train, y_train)
print(search.best_params_, round(search.best_score_, 4))

Two details in that snippet are load-bearing. The pipeline: if you scale before the split, validation-fold statistics leak into training and your score flatters the model. Putting the scaler inside the pipeline guarantees it is fit on fold-train data only. And refit=True: best_params_ is a dictionary, not a model. The flag re-trains the winning configuration on the entire training set and stores it as best_estimator_, which is the artifact you actually ship — a discipline you will lean on again when each trial becomes an iterative training run in Early Stopping and Learning Rate Schedules.

The parameter ranges themselves are domain knowledge, not plumbing. A C range of [1e-3, 1e3] only makes sense because C is the inverse regularization strength and the useful regime usually straddles 1. Ranges that are too narrow guarantee a search that never had a chance; ranges that are too wide waste trials on absurd corners. Write the range, then write one sentence justifying each endpoint. If you cannot justify an endpoint, you are guessing, and the search will spend your budget proving the guess wrong.

Finally, remember what n_iter counts: configurations, not fits. 60 configurations times 5 folds is 300 model fits, every one billed against the budget. When a single trial is expensive, cut n_iter and widen the log ranges rather than shrinking back to a comfortable-looking grid.

Random search is the default for finding a best configuration. But one scenario still belongs to the grid, and four failure modes sabotage tuning jobs regardless of which method you pick.

The scenario first. With two hyperparameters and a need to present the shape of the response — a heatmap of AUC over (n_estimators, max_depth) for a slide — you want the lattice, because readers will parse monotonicity and interactions off the axes. Same for sensitivity analysis: "how much does the score move if C drifts by 10x?" is a sweep question, not a search question. Decide which job you are doing before choosing a method: finding the best point (search) versus describing a surface (sweep).

Mistake one: tuning on the test set. Every configuration evaluated on the test split is one more adaptive query against it, and after enough queries the test score becomes a biased estimate of generalization — the winner's curse. The fix is structural: a three-way split (train / validation for the search / untouched test for the final report), or nested cross-validation when data is scarce. Your write-up should state which one you used.

Mistake two: comparing configurations on different footings. If one trial runs with early stopping at epoch 40 while another trains the full 200, or a fold's scaler was fit before the split, the scores are not comparable. Everything inside the search must be identical except the hyperparameters under test.

Mistake three: shipping the wrong artifact. best_params_ is a dict. The deployable model is best_estimator_, which exists only because refit=True retrained the winner on the full training set. Set refit=False to save compute and you get parameters with nothing to serve — or worse, someone grabs the last fold's estimator from a hand-rolled loop and ships a model trained on 80% of the data.

Mistake four: declaring a winner inside the noise band. Two configurations scoring 0.8712 and 0.8726 across 5 folds are usually indistinguishable; the standard error of a 5-fold AUC on tens of thousands of rows is often around 0.003. Compute the spread of the fold scores and apply the one-standard-error rule: among configurations within one SE of the best, prefer the simpler one. This is also the honest answer to "how do you know your tuning helped?" — a confidence interval, not a leaderboard position:

import numpy as np
from sklearn.model_selection import cross_validate, StratifiedKFold

cvres = cross_validate(search.best_estimator_, X_train, y_train,
                       cv=StratifiedKFold(5, shuffle=True, random_state=42),
                       scoring='roc_auc', return_train_score=True)
mean = cvres['test_score'].mean()
se = cvres['test_score'].std(ddof=1) / np.sqrt(5)
print(f'AUC {mean:.4f} +/- {se:.4f}  (train {cvres["train_score"].mean():.4f})')
if cvres['train_score'].mean() - mean > 0.10:
    print('large train/val gap - diagnose before tuning further')

That last check is the bridge to the next failure state. A search can only pick the best model among the candidates you offered it. If every candidate shows train AUC 0.99 and validation 0.83, you do not have a tuning problem — you have a generalization problem, and the tuning budget is wasted until it is fixed. That diagnosis is the subject of Diagnosing the Generalization Gap, and both the random search here and the surrogate-guided version in Bayesian Hyperparameter Optimization assume the candidate family is capable of solving the task at all.

For the interview, expect "grid or random search?" and answer in one line: random search, because only a few parameters matter, and random gives each of them N distinct values in N trials instead of k values in a k^d grid — then name the exception (low-dimensional sweeps you must present as a surface) to show you are reasoning, not reciting.