Skip to main content

Profiles and flame graphs

Record a profile in your runs, and see where the time went against the reference: the web app draws the new run's profile as a differential flame graph, colored by how each function changed, with a table of the top functions. Most profilers' files work as they are.

A differential CPU flame graph of the gateway's steady-high run, rooted at "all": under ingress and worker, the shared cache's lookup and respond and the response rendering are new, in orange; the model's forward pass is blue, a smaller share. Under it, the Top functions table: self and total time per function, in ms and in %, and the change in percentage points against main, the biggest increase first.A differential CPU flame graph of the gateway's steady-high run, rooted at "all": under ingress and worker, the shared cache's lookup and respond and the response rendering are new, in orange; the model's forward pass is blue, a smaller share. Under it, the Top functions table: self and total time per function, in ms and in %, and the change in percentage points against main, the biggest increase first.
Rendering responses under the shared cache's lock is new (orange); the model's forward pass runs less (blue): cache hits skip it.

Show a profile next to the reference's​

From a CI job, without code

When a job already records a profile, qa upload cpu.folded saves it as a run, and the run's card shows the flame graph: see Upload results from any tool. The steps below are for profiles recorded in the runs of a batch.

With py-spy, for Python code:

  1. Record the profile in run(), into the run's output directory:

    qa/main.py
    import subprocess

    def run(context):
    subprocess.run(['py-spy', 'record', '--format', 'raw', '--nolineno', '-o', context.output_dir / 'cpu.folded',
    '--', 'python', 'my_script.py', context.input_path], check=True)
  2. Show the file in the run cards:

    qaboard.yaml
    outputs:
    visualizations:
    - name: CPU flame graph
    path: cpu.folded
  3. Run a batch on this commit and on the reference.

How people read the flame graph, search it and compare shares or absolute values is in the user guide: Flame graphs.

Record a profile with your profiler​

qaboard.flamegraphs.save_flame_graph(path, source) saves a profile as folded stacks, from perf script's output, a pprof file (.pb.gz), folded stacks, or a {"outer;...;inner": value} dict your code collected. It returns the stacks, so profile.share("lock") can be a metric. With total=, values are rescaled to add up to it; unit= says what they are.

qa/main.py
import subprocess
from qaboard.flamegraphs import save_flame_graph

def run(context):
perf_data = context.output_dir / 'perf.data'
subprocess.run(['perf', 'record', '-F', '99', '-g', '-o', perf_data, '--', './my-binary', context.input_path], check=True)
script = subprocess.run(['perf', 'script', '-i', perf_data], check=True, capture_output=True).stdout
save_flame_graph(context.output_dir / 'cpu.folded', script)
perf_data.unlink()

Like stackcollapse-perf.pl: the command's name first, offsets and C++ arguments removed, unknown functions shown as their module ([libc.so.6]). pid=True, tid=True and annotate=True (_[k] after kernel frames) are options.

Files the viewer reads​

Save what profilers write in the run's output directory, as it is. The viewer reads the file's content; the name tells QA-Board to use it:

FormatFile namesWritten by
Folded stacks, main;parse;read 12 per line*.folded, *.collapsed, *.stacks.txt, gzipped too (.gz)stackcollapse-*.pl, inferno, py-spy --format raw, async-profiler, qaboard.flamegraphs
speedscope JSON, sampled or evented*.speedscope.jsonpy-spy --format speedscope, rbspy, speedscope's converters
Chrome/V8 CPU profile*.cpuprofilenode --cpu-prof, Chrome DevTools
A tree, {"name": "main", "value": 1200, "children": [...]}, value including the children's*flame.jsond3-flame-graph, your code

Other names need type: flame/folded (any flame/... type) on the visualization. perf script's text and pprof files are too big or binary for the browser: save_flame_graph turns them into folded stacks (recipes).

The viewer reads profiles up to 200 MB (after decompression), and draws the first 1,000 levels of a stack: the search and the table of functions count deeper frames too.

Get profiles you can compare​

  • Keep frame names stable between runs: the viewer matches frames by their path of names. No addresses, hashes, thread or process ids: py-spy --nolineno leaves line numbers out, and perf stacks start with the command's name, without its pid.
  • Shares or absolute values. Two runs rarely take the same number of samples: by default the viewer compares each function's share of its profile. To compare absolute values too, save values that compare across runs, e.g. microseconds of CPU per 1,000 requests (total=, see below).
  • The unit: a # unit: nanoseconds line in folded stacks (other tools skip it), or the visualization's unit option, which wins over the file's. Units the viewer formats: samples (the default), nanoseconds, microseconds, milliseconds, seconds, bytes; any other word shows as is. Words after the unit describe it: microseconds of CPU per 1,000 requests. speedscope and .cpuprofile files say their unit themselves; the unit option still wins.
qaboard.yaml
outputs:
visualizations:
- name: CPU flame graph
path: cpu.folded
- name: Allocations
path: heap.collapsed
unit: bytes # the values' unit

The serving latency example saves its profiler's folded stacks, and rescales them to microseconds of CPU per 1,000 requests, in qa/main.py:

save_flame_graph(output_dir / 'cpu.folded', output_dir / 'cpu.folded', unit='microseconds of CPU per 1,000 requests',
total=bench['profiled_cpu_s'] * 1e6 / bench['profiled_requests'] * 1000)

Where to go next​