Profiling & Performance

cProfile: Finding the Hotspot

Start broad — measure before changing anything:

python -m cProfile -s cumtime app.py        # sort by cumulative time
python -m cProfile -s tottime app.py        # sort by time spent in the function itself

Or profile a callable in-process:

import cProfile, pstats
cProfile.run("solve(data)", "prof.out")
pstats.Stats("prof.out").strip_dirs().sort_stats("cumulative").print_stats(10)

Read the two time columns carefully: tottime excludes sub-calls (the place to optimize inside), cumtime includes them (the place to find a slow call chain). A function with huge call counts and small tottime still costs time through per-call overhead — that pattern points to unnecessary loops rather than expensive bodies.

Line and Memory Profilers

cProfile reports per function; hot loops need per line:

pip install line_profiler memory_profiler
kernprof -l -v script.py    # runs @profile functions and prints per-line timing
python -m memory_profiler script.py          # line-by-line memory growth

For allocation sites rather than totals, use the stdlib tracemalloc:

import tracemalloc
tracemalloc.start()
# ... workload ...
snap = tracemalloc.take_snapshot()
for stat in snap.compare_to(tracemalloc.take_snapshot(), "lineno")[:5]:
    print(stat)

The diff shows which source lines allocated the memory still alive — the right tool when RSS grows steadily or a cache never evicts.

Benchmarking Discipline and Pitfalls

Microbenchmarks lie easily. Rules that keep them honest:

• Use timeit (or time.perf_counter()) — never time.time(), which jumps with clock adjustments
• Warm up first; discard the first run so caches and lazy imports settle
• Vary input sizes to see complexity, not one lucky sample
• Remember cProfile adds per-call overhead — tiny functions look relatively slower under it

Interview angle: the winning narrative is measure → hypothesize → change → re-measure. State the profile evidence, apply the fix (often an algorithmic change: set membership instead of list scans), then show the before/after numbers. Micro-tweaks like local-variable binding only matter after the algorithm is right.