Profiling with Flamegraphs and Perf

A Benchmarks with Criterion run reports a 40% regression on report/aggregate, and the diff touches twelve files. You need the one function where the extra time goes — and a stack of numbers will not tell you. A flamegraph will, in one picture.

Get symbols into the binary first. Profiling a stripped release binary produces frames named unknown. Enable debug info without giving up optimization:

# Cargo.toml
[profile.release]
debug = 1          # line tables; enough for good frames, small size cost

Then record. The flamegraph crate wraps perf on Linux and dtrace on macOS:

cargo install flamegraph

cargo flamegraph --release --bin server      # runs the binary, records samples
cargo flamegraph --release --test orders     # a single test
cargo flamegraph --release --bench parse     # a criterion benchmark

It prints flamegraph.svg in the working directory — an interactive document where bar width is share of samples, and hovering a frame tells you its function, its own percentage, and its cumulative percentage. The hottest path is the widest stack of frames from the bottom up. On Windows, run the same tools under WSL, where perf is available.

Flags worth knowing before you need them:

cargo flamegraph --release --bin server -- --port 9000   # args after --
cargo flamegraph --non-interactive --title "report/aggregate" -o agg.svg
cargo flamegraph --deterministic                          # stable colors run to run
cargo flamegraph --flamechart                             # x-axis becomes sample order

--deterministic matters the moment you keep two SVGs side by side for review; --flamechart trades the nested layout for a time-ordered one, which is how you spot "phase 1 then phase 2" behavior in a batch job.

Without the wrapper, the underlying command is worth knowing:

perf record --call-graph dwarf -F 99 -- ./target/release/server
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

The flags each buy something:

One Rust-specific detail: release builds do not guarantee frame pointers, so if you skip --call-graph dwarf, add RUSTFLAGS="-C force-frame-pointers=yes" and use -g; otherwise stacks unwind as a single tall frame and the graph says nothing.

Each rectangle in the picture comes from one sample: the profiler interrupted the process, walked the stack, and recorded where execution was. Nothing is instrumented, so overhead scales with the sampling rate — at 99 Hz the process loses well under a percent of its time to the profiler, which is why you can record production-shaped load without the timings shifting. It also means the graph is statistical: a function taking 2% of total time earns roughly two samples in a hundred, so anything under a few percent is noise. Trust plateaus; ignore slivers.

Profile the right build. This is the rule people break first: profile release, always. An unoptimized debug build spends its time inside core/alloc helpers that do not exist in production, so you will optimize a bottleneck your users never hit. Keep debug = 1 so you keep symbols — optimization plus symbols is exactly the combination profilers want.

For a quick iteration loop, run the workload under load while recording — a benchmark harness, a wrk/hey loop against a local server, or a single test through cargo flamegraph --test. Thirty seconds of samples at 99 Hz is roughly three thousand stacks; that is plenty to separate a 60% hotspot from noise.

Reading the picture — what width, depth, and color actually mean, and how perf report cross-checks it — is the next chapter.

The SVG opens and you are staring at hundreds of colored rectangles. Two minutes of careful reading beats an afternoon of guessing, but only if you know what the axes actually claim.

The axes are not what people assume. The y-axis is stack depth: each rectangle is one function, stacked on its caller, rooted at the bottom. The x-axis is not time — frames are laid out in an arbitrary (roughly alphabetical-by-stack) order purely so rectangles nest without overlap. Nothing scrolls left to right as the program runs. The only quantity that encodes data is width: a frame's share of all samples.

                        ┌─────────────┐
        ┌───────────────┤ format_row  │  ← wide AND deep: hot
┌───────┤ serialize                              ┌──────┤ clone ─┐
│ render└───────────────┤ pretty_print │         │ to_string      │
┴───────────────────────┴─────────────┴─────────┴────────────────┴
 root                                              narrow: rare path

Read it bottom-up along the widest column. Three patterns cover most real profiles:

Do not read meaning into color by default. Palettes either assign warm hues by frame heat or hash the function name for distinguishability; neither tells you anything width does not already. The exception is a differential flamegraph, generated by comparing two recordings, where color encodes the change — red hotter, blue cooler — and that is the fastest way to see what a commit did to a profile.

Cross-check with the text view. The same data without picture-reading:

perf report --stdio --sort symbol,dso     # ranked list of symbols
perf report --stdio --children            # with inherited (cumulative) time
perf diff old.data new.data               # weight change between recordings

perf report is what you use when a frame's name in the graph is truncated or mangled. Speaking of which: if symbols come out as _ZN7mycrate…, pipe the symbolized output through rustfilt — and remember from the setup chapter that debug = 1 in the release profile is what makes any of these names exist at all.

The rendered SVG is searchable: type into the search box at the top and matching frames highlight while the rest dim, which turns "find where parse_query is called from" into a two-second lookup across thousands of frames. Double-click a frame to zoom into that subtree — useful when one hot root dominates and you need to see what it calls.

Async code reads differently. Your async fn compiles to a state machine, so the sampled stack shows poll frames rather than the logical await chain; the hot frame is typically mycrate::handler::{{closure}}::poll plus whatever it called synchronously inside. Read the innermost frames and ignore the executor noise (tokio::runtime, scheduler). And a CPU flamegraph only samples running threads — a request that spends 500 ms waiting on a database shows almost no samples at all. For that shape of problem you want wall-clock sampling or distributed tracing; for "the CPU is pegged," the flamegraph is exact.

Interview framing: "where is the time going?" answered with width-not-time, self versus cumulative, and one concrete next command (perf report or a re-record with debug = 1) is a complete, senior answer. When the profile instead points at the allocator, the next chapter takes over.

The flamegraph's widest column is alloc::alloc::alloc and its callers are alloc::raw_vec::finish_grow. CPU sampling has done its job — it proved the machine is spending its time allocating — but it cannot say which line of your code asked for the bytes, nor how many are live at once. That needs a heap profiler.

The distinction matters because there are two different questions:

heaptrack is the practical default on Linux. It intercepts the allocator and records every allocation with a backtrace:

heaptrack ./target/release/server --once
heaptrack_print server.heaptrack.gz | less
heaptrack_gui server.heaptrack.gz   # charts + per-call-site table

heaptrack_print answers the questions directly: top memory consumers ranked by call site, temporary allocations (bytes allocated and freed immediately — often 60–80% of all allocations in a parser), peak consumption, and leaked allocations reported at exit. A Vec<String> rebuilt on every loop iteration shows up as one call site owning hundreds of megabytes of churn, which no amount of code reading would surface.

dhat is the lightweight, in-process alternative — a crate, not a system tool, so it works anywhere including Windows:

fn main() {
    if std::env::var("DHAT").is_ok() {
        let _profiler = dhat::Profiler::new_builder()
            .file_name("dhat.json")
            .build();
    }
    run();
}

DHAT=1 cargo run --release        # writes dhat.json when the process exits

Open the JSON in the dhat web UI and you get total blocks and bytes, the state at t-gmax (the moment of peak memory), and the state at the end of the run, each broken down by allocation backtrace. That pair is the diagnostic: heavy at t-gmax but clean at the end is a transient spike — normal for batch work; heavy at the end is a leak, and the backtrace names the owner.

Rust makes leaks harder, not impossible: Rc<RefCell<…>> cycles, tasks that are spawned and forgotten, and mem::forget (a safe function) all retain memory the type system will not reclaim. So the honest interview answer for "how do you find a memory leak in a Rust service?" is three steps: profile RSS over time to confirm growth, take a heap profile at peak and at exit to attribute bytes to call sites, then fix the owner — usually an unbounded cache or a dropped-signal channel.

For a long-running server you cannot pause, the allocator itself can be wired in: build with tikv-jemalloc-prof (prof:true), trigger a dump from an admin endpoint or signal, and parse the heap profile with jemalloc_pprof — same backtrace-attribution model, no process restart, which is how you catch a leak that only appears after six hours.

Practical habits that keep you out of these tools:

• Prefer iter reuse over per-iteration to_owned() in hot loops; heaptrack's temporary-allocation percentage is the scoreboard.
• Watch for clone frames under your own code — a String cloned inside a formatting path is the single most common self-inflicted allocation.
• Measure after an optimization with the same tool; allocation counts are only meaningful against a saved baseline, exactly like the timing numbers from Benchmarks with Criterion.

With tests, benchmarks, and profiles in hand, the last step in this phase is packaging those checks into the release itself: Production Crate Baseline.