Skip to main content

History

The History page shows how a metric evolved over a branch's commits, test by test. Use it to answer "which commit made this worse?" or "is this input getting better?".

Find the commit that caused a regression​

Say the inference gateway's p99 latency got worse on one load profile, heavy-tail-high, while the median over all profiles barely moved. To find the commit that did it:

  1. Open History from the sidebar of the commit list. It shows the branch you're looking at, or the reference branch. Or, on the commit page where you saw the regression, hover the name of the run that regressed and click History: History opens already filtered on that input, from 3 days before that commit, and you can skip step 3.
  2. Pick the metric at the top left of the chart, e.g. Latency p99.
  3. Type the test's name in the filter box at the top right, e.g. heavy-tail-high. Only its line is left.
  4. If the regression may be older than 3 days, pick earlier dates at the top left, e.g. the last two weeks.
  5. Hover along the line. Below the chart, the hovered commit shows as New and the commit before it as Reference, with the run's outputs compared. Where the line jumps, you found the commit: here "batcher: copy payloads into fixed slots", where p99 more than triples.
  6. Click that point to keep it as the reference, then hover the later commits to compare them with it: the commit that borrows payloads again brings p99 back near 3 ms. Double-click the chart to let go of the reference.
Filtered on one test, hovering along its p99 line until the commits below change at the one that copies payloads; a click keeps it as the reference for the later commits.

Below the chart, Metrics distribution and Metrics per test compare all the metrics of the two commits (see below). To look at everything the commit changed, open its page with the N runs button of its row.

tip

Keep Breakdown per test on: a regression on one test can hide in the median over all tests.

Follow one input over time​

On a commit page, hover the name of a run on its card and click History (it's also in the card's ☰ menu, and in the run's ⋯ menu in the Logs view): History opens on that input, on the commit's branch, already filtered on it, from 3 days before the commit to today. Hover its points to compare each commit's outputs with the commit before.

PSNR of the facade photo over six weeks of main, a point per commit, with a jump at the merge of the bilateral denoiser. Below, that commit and the one before, and their output images: only the new one outlines the leaves.PSNR of the facade photo over six weeks of main, a point per commit, with a jump at the merge of the bilateral denoiser. Below, that commit and the one before, and their output images: only the new one outlines the leaves.
Hover a point to compare its outputs with the commit before: the PSNR jumps when the bilateral denoiser is merged, and the leaves get halos.

You can also go to /<project>/history/<branch>. From the History page, the sidebar has the project's navigation: the branch's commits, Code, Integrations and Milestones.

Choose what the chart shows​

Performance over time has one point per commit, for the selected metric (pick another at the top left; the project's configuration chooses which ones are offered, else all the metrics of the runs).

Performance over time on main, the last two weeks: Latency p99, a line per test, and the 12 ms target dashed. One line more than triples and comes back four commits later; hovered, its point at the jump names heavy-tail-high, and the commit below copies payloads into fixed slots.Performance over time on main, the last two weeks: Latency p99, a line per test, and the 12 ms target dashed. One line more than triples and comes back four commits later; hovered, its point at the jump names heavy-tail-high, and the commit below copies payloads into fixed slots.
p99 per test over the last two weeks of main, with its 12 ms target dashed: hover the jump to see which test it is, and the commit below.
  • Breakdown per test: one line per test (input and configuration). Turn it off for a single line, the median, average or sum over all runs, but beware: commits may not have been evaluated on the same tests. Use the sum for counts that add up, like the tests of each test suite: filter on one configuration first, else a test run in two configurations counts twice. See tracking tests.
  • Relative to start (@100): rescales each line so that its first value is 100, to compare tests with very different values.
  • Show output files: below the chart, show the files a run created instead of its visualizations.

The last two show when Breakdown per test is on.

The same chart Relative to start: every test's p99 as a percentage of its value at the first commit, so that the 200 ms bursty test and the 2 ms steady ones share a scale. Heavy-tail-high climbs to about four times its start for four commits.The same chart Relative to start: every test's p99 as a percentage of its value at the first commit, so that the 200 ms bursty test and the 2 ms steady ones share a scale. Heavy-tail-high climbs to about four times its start for four commits.
Relative to start, every line begins at 100: tests with very different latencies share one scale.

The bar at the top of the page picks which runs the chart follows:

  • the date range and the refresh button, like on the commit list. By default, the last 3 days;
  • the batch selector: which batch to follow, CI by default;
  • the filter box, with the same syntax as on the commit page: input path, configuration, platform, tag, tuning parameters (key:value)...

Each branch's commit list starts with the same chart, aggregated over the runs (see Follow a branch).

Compare all the metrics of two commits​

Below the chart, for the hovered (or clicked) commit compared with its reference:

  • Metrics distribution: the same summary as the commit page's Summary, with medians, changes and box plots.
  • Metrics per input tag, if the project turns it on: the summary for each tag of the inputs, to see which kinds of inputs drive a metric.
  • Metrics per test: the Metrics Table and Metrics Diff tabs, with options to sort runs by name, batch, status, configuration, id or metric.

Pick which metrics to show with the metrics selector in the tables.

Share a history​

The chart options (metric, aggregation, breakdown, relative scale, output files) and the filter are saved in the page's link: copy it to share the same view.

Where to go next​

  • Reading metrics
    The commit's Summary, Metrics Diff and Metrics Table, against its reference.
  • The image viewer
    Find where the images of two commits differ, down to the pixel.
  • What's in a link
    What the link you share keeps, and what stays in your browser.