Benchmarks with Criterion

Your parser "feels slower" after a refactor. Feelings do not ship; a benchmark that produces a number with a confidence interval does. Criterion is the standard tool, and wiring it up is mostly a Cargo.toml ceremony:

[dev-dependencies]
criterion = { version = "0.5", features = ["html_reports"] }

[[bench]]
name = "parse_bench"
harness = false   # Criterion replaces libtest's bench harness

harness = false is the line people miss. Without it, Cargo wraps your bench target in libtest's unstable #[bench] machinery and the build fails; with it, your main is the runner, and Criterion's macros provide it:

// benches/parse_bench.rs
use criterion::{criterion_group, criterion_main, Criterion, Throughput};
use mylib::parse;

fn bench_parse(c: &mut Criterion) {
    let input = "timeout=5&retries=3&cache=off";
    let mut group = c.benchmark_group("parse");
    group.throughput(Throughput::Bytes(input.len() as u64));
    group.bench_function("query", |b| b.iter(|| parse(input)));
    group.finish();
}

criterion_group!(benches, bench_parse);
criterion_main!(benches);

cargo bench compiles that file under the bench profile — which inherits release settings, opt-level 3 — and runs it. Each benchmark goes through a warm-up phase (3 s by default) so the CPU settles and caches fill, then a measurement phase (5 s, at least 100 samples), and reports:

parse/query     time: [1.2417 µs 1.2509 µs 1.2605 µs]
                 change: [-2.31% -1.14% -0.02%] (p = 0.06 > 0.05)
                 No change in performance detected.

Read that middle line carefully: the three numbers are the 95% confidence interval around the median, and Criterion's verdict is a statistical test against the previous run. "No change in performance detected" means the observed difference was indistinguishable from noise — not that nothing happened.

Useful controls:

cargo bench                      # run everything
cargo bench -- parse             # substring filter, like cargo test
cargo bench -- --sample-size 200 # more samples for a noisy benchmark
cargo bench -- --warm-up-time 5  # longer warm-up on a busy machine

Two structural rules keep benchmarks honest. First, put setup outside b.iter — the closure runs thousands of times, so anything it does is measured; if setup must be redone per iteration (a consumed String, a fresh temp file), use iter_batched with a BatchSize instead. Second, name benchmarks after the case you would defend in a review: parse/empty, parse/1k-params, parse/already-valid. A single bench_parse number tells you nothing about which input regressed.

Parameterized benchmarks deserve their own shape, because one scalar hides which input got slower:

use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};

fn bench_parse_sizes(c: &mut Criterion) {
    let mut group = c.benchmark_group("parse");
    for size in [16usize, 256, 4096] {
        let input = "a=1&".repeat(size);
        group.bench_with_input(BenchmarkId::from_parameter(size), &input, |b, input| {
            b.iter(|| parse(input));
        });
    }
    group.finish();
}

That yields parse/16, parse/256, parse/4096 — so an O(n²) regression sneaking into an O(n) parser shows up as the shape of the curve across parameters instead of as one mystery aggregate.

The optimizer is the other half of the story — a benchmark can silently measure nothing at all, and Criterion's answer to it is black_box, covered in the next chapter, The Benchmark That Measured Nothing. If you have come from Test Doubles with Traits, note the shared move: both topics replace a judgment call with something mechanical.

You run a benchmark for a hash function and get 0.4 ns per call. Real hashing of a 64-byte key cannot be 0.4 ns. Nothing is broken — the compiler proved your measured work was irrelevant and deleted it.

That is the central hazard of benchmarking in an optimizing compiler's language. Two optimizations are responsible:

1. Dead-code elimination. If the return value of parse(input) is never observed, the call — and everything it does — may be removed entirely. Your loop then measures an empty loop.
2. Constant folding. If input is a literal known at compile time, the compiler can precompute the whole result, or specialize the function for that constant, so the "measurement" covers an optimized special case rather than the code users run.

std::hint::black_box is the fence. It takes a value, returns it unchanged, and the compiler is forbidden from making any assumption about what happened to it:

use std::hint::black_box;

fn bench_hash(c: &mut Criterion) {
    let key = vec![0xABu8; 64];
    c.bench_function("hash/64-bytes", |b| {
        b.iter(|| black_box(hash(black_box(&key))));
    });
}

The two black_box calls solve different problems:

Be precise about what Criterion already does for you: b.iter black-boxes the value your closure returns, so the output side is covered by default. What it cannot protect is the input side — a literal argument the optimizer can fold straight through your function. Wrapping both is the conservative habit; it costs a nanosecond-scale compiler barrier and removes an entire class of phantom results.

What to measure, and what to exclude

The closure inside b.iter runs tens of thousands of times. Everything in it is measured, including allocation you did not intend to time:

// measures parsing AND allocating the input every iteration
b.iter(|| parse(&format!("timeout={}", 5)));

// measures only parsing
let input = String::from("timeout=5");
b.iter(|| parse(black_box(&input)));

When the fixture must be rebuilt per iteration — a consumed Vec<u8>, a temp file that the code deletes — use iter_batched, which separates setup from the timed section:

use criterion::BatchSize;

b.iter_batched(
    || make_document(),            // not timed
    |doc| parse(&doc),             // timed
    BatchSize::SmallInput,
);

BatchSize decides how many setups are created per batch; LargeInput (one setup per iteration) is right when setup is expensive or the input is consumed.

Timing resolution is the last trap. Very fast operations — a handful of nanoseconds — sit at the edge of the platform timer's precision, so Criterion automatically scales up measurement time to accumulate enough iterations per sample. If a benchmark is still wide-interval after that, the fix is usually more work per sample (a larger input, or b.iter_custom running the loop N times yourself and returning the elapsed Duration) rather than more samples of an equally fuzzy measurement.

Sanity-check the number. Compare against a mental model: hash/64-bytes at ~50 ns is plausible; 0.4 ns means the loop body is empty or the work folded. A quick way to expose a lying benchmark is to change the input contents — if the time does not move at all, the compiler is not running your code. Cross-check the order of magnitude against perf or a flamegraph from Profiling with Flamegraphs and Perf, where the actual instruction counts are visible.

Interviews do not usually ask for black_box by name — they ask "how do you know your benchmark measured the right thing?" Answer with these two optimizations, the input/output placement table above, and the habit of sanity-checking against a physical lower bound.

Criterion's real value shows up on the second run. Day one you measure; day thirty you need to know whether a pull request made anything slower — and eyeballing two time: lines is not a method.

Baselines are how you compare runs. Save the state of a known-good commit, then ask future runs to compare against it:

cargo bench -- --save-baseline main     # on main: record the reference
git checkout feature/fast-path
cargo bench -- --baseline main          # compare this run against `main`

Output becomes a verdict:

parse/query  time: [1.3821 µs 1.3905 µs 1.3996 µs]
             change: [+11.24% +12.87% +14.51%] (p = 0.00 < 0.05)
             Performance has regressed.

Read the array as before — median with 95% confidence bounds — then read change as a confidence interval on the difference. The rule that keeps you honest: if the interval crosses zero, the change is indistinguishable from noise, and Criterion prints change has inconclusive significance rather than declaring a win. p = 0.00 with a wholly positive interval is a regression; a wide interval spanning negative and positive means your benchmark needs more samples, not a rollback.

Baselines live in target/criterion/, one directory per benchmark, so they survive between runs but not between clean checkouts — CI jobs that start from a fresh workspace must either re-run --save-baseline on the merge base or store target/criterion/ as a cache artifact.

cargo bench -- --save-baseline pr      # candidate branch
cargo bench -- --baseline main         # compare, keep both for inspection
cargo bench -- --discard-baseline      # forget a bad reference

Turning down the noise. Microbenchmarks move for reasons unrelated to your change: turbo-boost ramps, background processes, page cache. Two settings cut the false alarms:

# Criterion.toml at the crate root
noise-threshold = 0.03   # ignore changes under 3%
sample-size = 200        # more samples per benchmark

A noise-threshold tells Criterion not to report a change smaller than the machine's own jitter, so a 0.4% "regression" stops waking anyone up. Raise sample_size when confidence intervals stay wide; raise measurement_time when individual iterations are so short that timer resolution dominates.

Making it fail the build. Criterion exits 0 regardless of verdicts — statistical judgment is left to you. For CI gating, add a small step that fails on a regression signal: parse the change: line for benchmarks you care about, or compare saved baselines with critcmp:

critcmp main +0.5%     # baseline vs candidate, changes >= 0.5%

Keep the gate to a handful of load-bearing benchmarks (the ones users hit per request); gating all forty benches turns every run into a coin flip.

With the html_reports feature enabled, every run writes target/criterion/report/index.html — per-benchmark pages with the time distribution, the line chart of every saved run, and the change plot. Opening that chart before approving a "regression" is the fastest way to see whether the jump is a step (your commit) or a single noisy outlier (someone's browser build on the same runner).

Finally, a benchmark says that it got slower. It never says why — which is the job of sampling profilers, so continue with Profiling with Flamegraphs and Perf.