Skip to main content

QA-Board in 5 minutes

Tests tell you that your code runs. QA-Board shows you whether it got better: every commit runs on your test set, and the web app puts its outputs and metrics next to a reference's, so your team can see what changed and agree that it's better.

This tour follows the teams of Kitefin Robotics, a fictional company that builds camera robots. They use QA-Board for a camera pipeline, an object detector and an inference server. Their projects are real code, in examples/: the history is made up, but every number and image comes from running them. Some clips are still being recorded: they show as placeholders until then.

1. Run​

Wrap your code with qa, then run it on a batch of inputs: on your computer, on a cluster, or in CI on every push. Here Ines, on the camera team, runs its ci batch from her computer (9 images in 2 configurations) and shares the results:

qa batch runs your code on each input and configuration of a batch; with --share, it prints the link to the results.

qa runs your code where it already runs. QA-Board doesn't need to be the entrypoint of your build or your CI. See Running your code.

2. Compare​

Every commit is compared with a reference: by default, the latest results of the reference branch, or a milestone like the last release. The Summary shows each metric's change, the Metrics Diff each run's.

When the detector moved to int8, its model got 3.5 times smaller, and its median mAP barely moved. Sorting the Metrics Diff by AP on small objects brings the slice that got worse to the top.

From the Summary to each test's change, sorted to bring the worst slice to the top.

See The commit page.

3. Look​

Metrics tell you that something changed; the outputs tell you what. Images show side by side, with synced zoom and pan, a perceptual diff, pixel values and histograms.

A new denoiser in the camera pipeline won more than 1 dB of PSNR in low light. Zooming into the images, then turning on Image Diff and raising its threshold, leaves only what PSNR missed: halos along the edges.

Synced zoom, then Image Diff and its threshold.

Plots, flame graphs, text, videos, point clouds and HTML reports have viewers too. See Comparing results and Visualizations.

4. Find​

History follows a metric across commits, input by input. A change to the inference server made its p99 latency worse on one input only, so the median over all inputs hid it. Hovering along that input's line shows each commit's results, until they change at the commit that caused it.

Hover along a test's line to find the commit that changed it, click to compare later ones with it.

See History.

5. Tune​

Start tuning experiments from the web app: pick parameters and values, or let qa optimize search for the best ones. The Analysis view then shows how each parameter matters, and the trade-offs between metrics.

Start a sweep from the browser, then compare its runs in the Analysis view.

See Tuning from the web app and Auto-optimization.

6. Gate​

Experimental

The quality gate may change without notice.

Coding agents write a growing share of changes, and "it's faster" needs checking. The inference server team's coding agent added a cache that saved CPU and took p99 from milliseconds to seconds. qa gate compares the commit with the reference, rule by rule, and fails the CI job.

qa gate compares a commit's CI batch with a reference, rule by rule, and fails the job when one breaks.

See Quality gate.

And more​

  • Logs: follow a run's log live, count its errors and jump to the next one. See The commit page.
  • Files: browse every run's output folder from the browser, with the keyboard. See Browsing files.
  • Bit accuracy: check that a change didn't alter any output file, e.g. before handing a model to a hardware team. See Bit accuracy.

A run's log, live: it follows the end, counts the errors, and jumps to the next one.

While a run runs, its log follows the end; click the error count to jump from error to error.

Try it​