New and reference

QA-Board shows your results next to a reference: the results you compare them with. This page helps you choose that reference, and read what each run is compared with.

A number alone says little: is a PSNR of 29.6 dB good? Next to the reference's, it says what your change did: +5%. The new bar at the top of a commit page says which results you look at, the ref bar below it what they're compared with: a commit, one of its batches, maybe with a filter. Each run is shown against the reference's run of the same input.

The camera ISP's bilateral denoiser, in the top bar and in blue, against the release before it, isp-4.1, below and in purple: each image says which one it is. Here the new results were named “bilateral” (see Name what you compare).

The default reference

When you open a commit, the reference is the latest commit with results on the project's reference branch, often main (the project sets it in qaboard.yaml): what your change does to the code everyone uses. The page's link saves that commit, so a link you share today compares with the same commit next week.

Choose another reference

To compare with a release, another branch or a colleague's commit, change the ref bar. The new bar works the same way.

• Type a commit hash in the commit field. Short hashes work.
• Click the branch name next to it to select that branch's latest commit.
• Click Change next to the commit field: swap the bars, pick the reference branch's latest commit, or a milestone.

The reference's Change menu: swap the bars, pick the reference branch (main), or a milestone, like the detector's releases.

Each bar also has its own batch selector and filter box: you can compare any batch of any commit with any other, or only some of their runs.

Every item of the Change menu

Menu item	What it does
Change project	Pick results from another project, e.g. another sub-project of a monorepo, or a fork.
Swap the bars	Swaps the top and bottom bars' results, with their batches, filters and names.
Also show in the bottom bar / top bar	Copies this commit, batch and filter to the other bar, instead of what it shows.
Stack another reference…	In the ref bar: compare with several references, one at a time.
Remove from comparison	Clears the commit. With several references, removes the one shown.
The reference branch	Selects its latest commit.
Milestones	Your favorites, your milestones, the project's, and those from qaboard.yaml. Their batch and filter are selected too. See Milestones.

Compare two batches of the same commit

A tuning experiment runs on the same commit as CI. To see what it changed, compare its batch with the commit's CI batch:

1. In the new bar's batch selector, pick the experiment, e.g. tune-denoise.
2. Click Change, then Also show in the bottom bar. The ref bar now shows the same commit and batch.
3. In the ref bar's batch selector, pick CI.

Each run of the experiment is now shown against the CI run of the same input.

Compare with a milestone

A milestone keeps results worth comparing with, like a release or your best tuning so far. Save one with the star next to a commit's hash, then pick it in the ref bar's Change menu. See Milestones.

Name what you compare

Click new or ref at the start of a bar to rename it, for example ISP v2 and ISP v1. The whole page then uses these names: the labels of images and videos, the legends of plots, the headers of text diffs and HTML outputs, the flame graphs' choices, the Summary, and the columns of the Metrics Table and Metrics Diff (ISP v2 − ISP v1). The names are saved in the link, so people you share it with see them too. Erase a name to go back to "new" or "ref".

Results that are a milestone are named after it, like isp-4.1 in the picture at the top of this page. You can still rename them on the page: the milestone's name then shows next to its star. Erase the name to go back to the milestone's.

Each side also has its color, in its bar, in the viewers' labels and in plots: new is blue, and each reference has its own color. Results keep their color wherever they go: when you swap the bars, the colors swap too. Green and red never stand for a side: they say what got better or worse (see what the colors mean).

Compare with several references

To compare with the last release, a tuning experiment and the latest main on one page, stack them. The new results are compared with one reference at a time, and you switch between them.

1. Hover the ref bar and click Stack a reference, below it.
2. Add a commit, a branch (its latest commit), a milestone, or the same reference again to change its batch or filter. It shows right away, and the reference you were looking at stays in the list.

Stack a reference, under the reference's bar: type a commit or a branch, pick the reference branch, the same reference again, or a milestone.

3. Switch references with their tabs, below the bars, or press r (next) and Shift r (previous). Each keeps its batch, filter and name.

Two references stacked: the isp-4.1 milestone, shown in the bottom bar, and ref 2, the latest commit of main, hovered.

The tabs show everything you compare: first the top bar's results, then, after "vs", the references, each with its name, in its color (purple, magenta, teal, violet, then gray), with a star for milestones. The two tabs in the bars are lit. The first reference is ref, the ones you add ref 2, ref 3...: the tab, the ref bar and the viewers all use that name, and you can rename it in any of them. Next to the name, a tab says only what tells it apart from the others: its batch or filter, its commit when others are on the same branch, its branch, its project. Hover a tab for what its bar would show: the commit and its message, who made it and when, the batch, the filter, and how many runs it has and how many failed.

The references are saved in the page's link: share it, and others can switch between the same references.

Reorder, rename and remove references

• Drag tabs to reorder them, or press Alt ← →. While you drag, a bar shows where the tab would go, and the right of the tabs says it in words: "Move ref 3 between ref and ref 2".
• Double-click a tab to rename it, press Delete to remove it.
• Right-click a tab, or click its ⋯, to compare with it, rename it, Duplicate it (to change its batch or filter), move or remove it.
• Swap the bars, in the menus of the two lit tabs, or drag the first tab on a reference (or a reference on the first tab): the top results go in the bottom bar. It's the same comparison, reversed, and nothing is replaced. To stop comparing with what was on top, swap, then remove it.
• A reference keeps its name and its color when you reorder or remove the others. The color of a reference you remove goes to the next one you add (or to the first gray one). A copy of a reference you named "night" is "night copy", in the next free color.
• When you swap the bars, the colors follow their results: the top bar takes the reference's color. Picking another commit in a bar keeps its color.
• When you swap the bars, the names you gave follow their results, while "new", "ref" and "ref 2" stay in their bars.
• When there isn't room, the tabs' details give way first, then the tabs scroll sideways (with the mouse wheel too).
• Only the reference shown is loaded with all its runs. Once the page is idle, the next and previous ones load in the background (unless they have thousands of runs), and hovering a tab loads it too.

How runs are matched

Each new run is compared with one reference run: the most similar one. QA-Board looks for the same input first, then the closest configuration, the closest tuning parameters, and the same platform. The closest is the one with the fewest different settings; between as many, the one whose values are the most similar: for base-v1, base-v2 rather than other, for gain: 3, gain: 4 rather than gain: 100.

When the best match is not identical, an orange tag with a ≠ warns you on the run: Input, Config or Platform. Hover it to see what the reference run used. Config shows how the configurations and tuning parameters differ, line by line, and says when other reference runs were as close. A run with no match at all has no reference values or visualizations.

The Config tag of the int8 release's run at 416 px: compared with the fp32 release's run at 320 px, the closest, it removes fp32, adds int8, and changes the size and the model.

caution

If you filter the new runs, filter the reference the same way (Filter the reference's runs…). Runs are only matched with the reference runs its filter leaves, and the Summary compares the new runs with every reference run of the same inputs, whatever their configuration.

Read configurations

A run's configuration is what it ran with: configuration files like base or low-light, values like isp.nr.strength: 2, and tuning parameters. Run cards show each of these settings as a tag, compared with the reference run's.

Take the detector's int8 release, kd-1.3, against the fp32 release, kd-1.2. Its run on slices/small-objects says:

The int8 run against the fp32 release: fp32 removed, int8 added, size-320 the same, and the model a changed tuning parameter. The orange Config tag says the reference run had other settings.

• − fp32, red and struck through: the reference run used fp32, this one doesn't.
• + int8, green: this run added int8.
• size-320, gray: both runs have it.
• model: "kd-1.2-fp32-320" → "kd-1.3-int8-320", orange, with a wrench: a tuning parameter whose value changed, the reference's value first.
• ≠ Config, a filled orange tag: no reference run had the same settings, so this run is compared with the closest one (see how runs are matched).

Each run is compared with its own reference run, and says how they differ. The release also ran its models at 416 px, which the fp32 release didn't: those runs are compared with its 320 px runs, and their tag size-320 → 416 marks only the part that changed. Above the runs, vs. reference sums up the changes, with how many runs have each one.

The int8 release's runs at 320 and 416 px against the fp32 release: each run shows what it added, removed and changed, and vs. reference sums it up above them.

With many runs on the page, most settings are the same for all of them. To show you what matters, each run only shows its own settings: what all the runs share hides behind a dashed tag like ··· 9 shared. Hover it to see those settings, click it to show them in this run. The top of the page lists them once, next to All 12 runs. Hover any tag to see what it means and copy it.

• What all the runs share follows the filter: filter runs, and what the remaining ones share hides too. The ··· shared tags sit where the settings they hide are when those come first or last (··· 2 shared night.yaml ··· 7 shared), else in one tag, first.
• Changes all runs have, e.g. when the reference commit used other default settings, are listed once at the top of the page, next to vs. reference, with how many runs have each change. Runs don't repeat them: their ··· shared tag lists them, and the Config tag's diff shows them.
• Long strings that vary between runs start with … when they share a long beginning, e.g. …scenes/night.yaml. Hover them for the full string.
• Long settings, like commands, paths or lists, are shortened with …. Hover one to read it, or click it to show all the settings in full, in every run, wrapped on several lines. Click one again to shorten them. The web app remembers your choice, also in Settings → Preferences and the Configuration section of the View options.
• To always see full configurations, turn off Hide them in each run, at the top of the page or in the Configuration section of the View options. Runs then show all their settings, the shared ones outlined.

Every kind of tag, what counts as a setting, and copying settings

Tag	Means
gray	The same as the reference run.
green, with +	Added: the reference run doesn't have it.
red, struck through, with −	Removed: only the reference run has it.
orange, gain: 3 → 4	Changed: the reference's value, then this run's. For strings, only the part that changed: conf/[day → night].yaml.
with a wrench	A tuning parameter (qa --tuning), shown after the configuration, in full, even when all runs share it.
dashed, ··· 9 shared	Settings all the runs shown share, hidden in each run.

• Each string of a configuration (base, conf/night.yaml) is a setting, and so is each value of its JSON objects (isp.nr.strength: 2).
• A list of values is one setting. Long ones show their first values and their size: tone_lut: [0, 0.016, 0.032, …] 64 values, ccm: [[1.6, -0.4, -0.2], …] 3×3.
• A list of objects has a setting per value (layers[1].stride: 2), so that you see which one changed.
• To copy settings, hover them: copy a setting as text or JSON, or the run's settings as qa options (-c base -c '{"gain": 4}' --tuning '{"nr": 2}'), as JSON, or as a tuning search space. The Config tag's diff copies the changes, or the reference's qa options.
• Short configurations (one or two short settings) are never hidden, and the Logs view always shows full configurations.

Compare the outputs

Below its metrics, each run shows its output files next to the reference's, in viewers made to compare them. Images zoom and pan together, and show side by side, with a wipe, flipped in place, or as their difference.

Wipe shows new and reference in one image, on each side of a divider you drag.

Videos play in sync, frame by frame, with the same modes. Plots draw the reference over the new results, text files and flame graphs show what changed.

• The image viewer

  Find where two images differ, read their pixel values, run a blind test.

• The video viewer

  Step through both videos frame by frame, find the frames that differ.

• Plots, flame graphs, 3D and more

  How the other viewers show the reference.