Running from the web app

From a commit's page, you can run its code again without a terminal:

• Redo runs that failed, were deleted or got stuck.
• Run the commit on other tests or settings, like the int8 detector on the slices where it loses small objects.
• Start a tuning experiment and see which values win: try a grid of parameters, or let an optimizer search them.

What you need

• Sign in, at the bottom of the sidebar. Signed out, the buttons that start runs are grayed out, next to a Sign in button. You also need the developer role on the project: see Project access.
• The commit's artifacts: the build and configuration files your CI saved. If they were deleted, QA-Board can bring them back first (see If runs don't start).
• A task runner for your project, like a compute cluster. If there is none, the form says so: ask whoever integrated QA-Board in your project.

Run tests or a tuning experiment

On the detector's quant/int8 branch, the commit that calibrates int8 with percentiles still loses 7% of the AP on small objects. Can the post-processing win it back? Here is how to try 2 score thresholds and 2 NMS IoUs on quant-search, its outdoor and small-objects slices:

1. On the commit page, select Run Tests / Tuning in the sidebar, under Tuning.
2. In Batch label, name the new batch, e.g. int8-nms. Its runs stay apart from the CI's.
3. In Batch of inputs+configurations, type the batch of tests to run: quant-search. The form shows 2 tests below it: hover it to see the inputs and their configurations. To find a batch, or write your own, see Choosing the tests to run.
4. If the form lists platforms (like Linux and Android), pick the one to run on.
5. Under Manual tuning, write each parameter with the values to try:

   {
     "score_threshold": [0.2, 0.3],
     "nms_iou": [0.5, 0.7]
   }

   With All combinations, the form says it Explores 4 combinations. To run the tests with the commit's own settings, click the no tuning example instead.
6. Check the total, 8 total runs (4 combinations on 2 tests), and click Start 8 runs.

Every combination of 2 score thresholds and 2 NMS IoUs on the 2 tests of quant-search: the form says 8 runs before you start them.

A message confirms that int8-nms started, with Show its logs. Its runs show up in the new batch, and when the last one ends, QA-Board emails you if your server sends emails. Next: look at the results.

QA-Board checks the form as you fill it: Fix this to start lists what would make the batch fail, like an unknown batch, and Before you start warns about the rest, like artifacts it must restore first. The form remembers what you entered, per project. If the batch doesn't start, the commit page says why: see If runs don't start.

Start from a run's settings

Hover a run's configuration tags and choose Copy as a tuning search space. Paste it in the Manual tuning editor, then add the values you want to try.

Every option of the form

Option	What it does
Batch label	The new batch's name: letters, digits and _.@:=, other characters become -. Reusing the label of an existing batch adds runs to it.
Batch of inputs+configurations	A batch, or several with a pattern like batch-*. Below it: how many tests, and the files that define the batch, e.g. Defined in qa/batches.yaml. Click a file to open it on your git server at this commit, or, for the shared and private batches, in Available Tests at the batch. When several files define a batch, the last one wins, and the form says which.
Platforms	Shown when the project has several. The runs use qa --platform with the one you pick.
Start from an example	no tuning: the tests with the commit's settings. simple-combinations: a grid. list-of-combinations: a list of parameter sets, each its own grid, whose runs add up. 1x2 matrix: a parameter whose value is a matrix. function: a JavaScript function that returns the parameters, for ranges and loops.
All combinations or Sampling	Every combination of the values (a grid), or a uniform sample of as many as you choose.
Pasting key=value lines	They become the JSON the editor expects: block.threshold=10 becomes "block.threshold": [10], and space-separated matrices like [1 2 3] become JSON arrays.
Overwrite previous identical runs	On, the default: runs that already exist in the batch, with the same input, configuration and parameters, run again. Off: they're kept, and only the missing runs start.
Will run as	With LSF, the user the runs start as, with their shell environment. It's you.
Limits	At most 5,000 runs per batch. Over 1,000, the form asks you to check that the cluster has room.

To see which tests a batch runs before you start it, or to write your own batch, open the Available Tests view: see Choosing the tests to run.

Let the optimizer search the parameters

When a grid would be too big, open the Automated tuning tab. It sets up a search in three steps: the Parameters to tune and their ranges, What “better” means with the metrics your runs return, and the Budget, the number of evaluations. Click Start automated tuning: each evaluation runs the batch, first with random parameters, then with parameters picked from the results of the previous ones. To choose the parameters and the budget, and read what the search found, see Find the best parameters automatically.

For the detector, tune-int8 searched the calibration percentile, the layers to keep in fp32, the score threshold and the NMS IoU on quant-search, for the best AP on small objects and mAP with a small model. There is no email when an automated tuning ends yet. See Automated tuning with qa optimize for every setting.

The tune-int8 experiment in the guided form: the parameters to tune, the metrics to improve and the budget.

Look at the results

1. In the new bar's batch selector, pick the new batch, int8-nms. Its runs show up as they start, and the page refreshes by itself while some are pending.
2. Select Tuning Analysis in the sidebar, under Tuning.

The form starts 8 runs. Once they're done, pick their batch: Tuning Analysis draws one line per combination of score threshold and NMS IoU.

Tuning Analysis shows how the metrics picked at its top, like in the tables, depend on the parameters. Most plots use each combination's median over the inputs (or its average):

• Parallel coordinates: one line per combination, across an axis for each metric and parameter that varies, colored by the metric you pick below it. Drag along an axis to highlight the lines in a range.
• Sensitivity to tuning parameters: the metric against one parameter, and with two parameters, a map over both.
• Breakdown by test: one line per input, to check that every input agrees.
• Efficient frontier & tradeoffs: two metrics against each other, one point per combination. The best trade-offs are on the edge of the cloud.

For the inference gateway, a sweep tried the batching settings of the sharded cache: max batch 1, 16 and 64, by max delay 0, 250, 1000 and 4000 µs, on the steady-high and heavy-tail-high load profiles. With CPU per request added to the metrics, the frontier shows which settings are worth it:

Each configuration's CPU per request against its p99: the defaults picked, max batch 16 and 250 µs of delay, are on the frontier.

For an automated tuning, the view is a report: what the search found, which parameters matter, and what to do next. These plots are under More plots. The sidebar highlights Tuning Analysis when the selected batch is an automated tuning. See Read the results.

What tune-int8 found: its status and best objective, then what to do next and the best parameters within the ranges searched.

Every plot and setting of Tuning Analysis

Part	What it shows
Automated tuning report	For an automated tuning: a report first, see Read the results, then these plots under More plots.
N runs · M inputs	How many runs and inputs the plots use (manual tuning).
The metrics, at the top	The metrics the plots use, the same choice as the tables'. Add one to plot it, like CPU per request.
Parallel coordinates	One line per combination of parameters, with its metrics aggregated over the inputs. Only metrics and parameters that vary get an axis. For an automated tuning, the iteration and the objective are axes too.
Metric and aggregation	The metric for the colors above and the Y axis below, and Median aggregation or Average aggregation.
Sensitivity to tuning parameters	The metric against the parameter you pick (X axis, Log-scale available). With two parameters that vary, a contour map over the two you pick. With several inputs, a box plot per value.
Breakdown by test	One line per input and configuration, against the parameter. Relative to Best shows each value as a percent of that input's best.
Efficient frontier & tradeoffs	With two parameters that vary: one point per combination, the metric against a second one (Metric on Y-axis), colored by the parameter. Hover a point for its parameters.

• The view uses the runs that match the filter, and leaves out failed and pending runs.
• The metrics, parameters and aggregation you pick are in the link: share it, and others see the same plots.
• On a batch without tuning parameters, it says How to start tuning?, with a Start a tuning experiment button.

Compare with the CI batch

To see what the experiment changed against the commit's CI runs, show int8-nms in the new bar and the same commit's CI batch in the reference bar: see compare two batches of the same commit. To look at one combination, filter the new bar's runs, e.g. nms_iou:0.7 score_threshold:0.3: the Summary otherwise aggregates all of them.

Redo, stop and clean up runs

To...	Where
Redo one run	On its card (Visualizations or Output Files views), ☰ menu → Redo. In the Logs view, its ⋯ menu
Redo the failed runs of a batch	Commit bar menu (☰) → Redo failed runs
Redo deleted runs	Commit bar menu → Redo deleted runs, or the button in the "N of the runs below were deleted" message
Redo all runs of a batch	Commit bar menu → Redo all runs…
Stop pending runs	The Stop runs button above the views, shown while runs are pending
Unblock a run stuck as pending	On its card, ☰ menu → Mark as failed, then Redo it
Delete runs, or only their files	Card ☰ menu (Delete…, Delete output files…), or the commit bar menu: failed runs, the whole batch, several batches

A run card's menu: redo it, delete it, or delete only its output files. While a run is pending or running, Mark as failed replaces Redo.

Redone runs use the commit's saved artifacts and the same input, configuration and parameters. They also use the runner they were started with, and the batch's runner settings, like how many runs can run at the same time: QA-Board starts them together, with a single qa batch. Follow them with the counters under the filter box, or in the Logs view, under Started from QA-Board.

danger

Deleting runs can't be undone: their output files and metrics are gone. QA-Board asks you to confirm, and won't delete batches or commits saved as milestones.

Filters, stuck runs and other details

• When you filter the batch's runs, the commit bar menu only redoes the runs that match: Redo the N runs that match the filter…, Redo filtered failed runs, Redo filtered deleted runs. Moving runs follows the filter too: Move the filtered runs to another batch. Deleting doesn't: it applies to the whole batch, and the confirmation says so.
• Redo all runs… asks you to confirm: the runs' current results are replaced when they start.
• Mark as failed is for runs stuck as pending or running whose job is gone, e.g. killed by the cluster: they would never report.
• Delete files, keep metadata, in the commit bar menu, makes its delete actions free storage only: the runs and their metrics stay. A box then limits it to files matching patterns like *.png.
• A batch with pending runs can't be renamed.

Get an email when a batch ends

Runs can take hours, or wait for a busy cluster. When you start a batch from the form, QA-Board emails you once, when its last run ends: how many runs failed, the main metrics against their targets, the best combinations when you tuned parameters, and links to the failed runs' logs. Under the Start button, the form says when it will email you.

What the email says, and when you get one

The email says:

• whether all runs succeeded, how many failed, and how long the batch took,
• the median of the project's main metrics over the successful runs, and whether they meet their targets,
• when you tuned parameters, the best combinations on the first main metric,
• which runs failed, with links to their logs,
• the commit, the batch and the inputs, with links to them in QA-Board.

Good to know:

• You get one email per batch, when no run is left pending: not when the first run fails. If you stop the batch from QA-Board, there is no email.
• For pipelines, whose steps start other batches, you get one email when the whole pipeline is done, not one per step.
• QA-Board emails the address of your account, or the one your administrators configured for your username. The confirmation message after you start says where.
• Your QA-Board server must be configured to send emails. There is no email for automated tuning yet.
• Batches you start from a terminal with qa --share batch email you too: see QA_NOTIFY to turn it off.

Start CI jobs and open your tools

The sidebar's Integrations menu (hover it to open it) has what your project added: links to reports, docs and dashboards, badges like the pipeline status, CI jobs (GitHub Actions workflows, GitLab CI manual jobs, Jenkins builds) and webhooks that call other tools. Click a job or a webhook to start it for this commit: a tag then shows whether it's pending, running, successful or failed. Click the tag to open the job's page and its logs.

Links to the project's reports and guides, and a webhook: after a click, its tag shows that the job is queued.

Good to know about integrations

• On the commit list, integrations act on the branch you're looking at. On a commit page, on the selected commit and batch.
• You need to be signed in to start GitHub Actions workflows and GitLab CI jobs. Once a job started, its icon changes to repeat, to start it again.
• GitHub runs workflows on the latest commit of the branch (or on the ref the project chose), not necessarily on the commit you're looking at; QA-Board then follows the run you started. Workflows show the status of their latest run for the commit. Only the workflows declared in the project's qaboard.yaml can be seen and started.
• Statuses refresh every few seconds while the menu is open, and your browser remembers the jobs you started on each commit.
• A link is disabled when its target doesn't answer. Badges that don't load are hidden.
• With many entries, a search box appears at the top of the menu.
• If the project configured nothing, the menu shows its git host's defaults on branch pages: GitLab's pipeline status, or a link to GitHub Actions.
• Some projects show their most used integrations directly in the sidebar, outside the menu.

To configure them for a project, see Integrations menu.

If runs don't start

The commit page tells you why, in a message above the views or in the form.

The commit's artifacts are missing

Runs start from the commit's artifacts. When some are missing, a message above the views says which ones, and when and by whom they were deleted:

• This commit's artifacts were deleted (or are incomplete, when files went missing without QA-Board knowing). If your project configured it, Recreate the artifacts asks your CI to rebuild them, and redoing runs or starting a batch does it too. Otherwise, Restore the artifacts restores the files from the source code, but not build outputs like binaries.
• This commit's artifacts are being recreated: QA-Board asked your CI, with a link to follow the build. It doesn't start anything until the build is done: try again then.
• Some of this commit's artifacts are missing: you can still redo runs and start tuning, but the runs that need these files will fail.
• Running tests from the web app needs build artifacts, instead of the form: the project doesn't declare any artifacts in its qaboard.yaml. Ask whoever integrated QA-Board in your project, and see Tuning from QA-Board.

Until the runs of a batch you started show up, a message at the top of the page says how it's going, e.g. Your batch is waiting for LSF: see Batches started from QA-Board.

Every message of the Run Tests / Tuning form

Message	What it means
Sign in to start runs	Runs started from QA-Board run as you. Once you're signed in, the form also lists the batch's tests and checks your settings.
Fix this to start	What would make the batch fail, e.g. an unknown batch, an invalid automated tuning configuration, a platform the project doesn't have.
Before you start	What to know first, e.g. missing artifacts QA-Board will restore.
Tuning may not work	The project's CI only saves artifacts on some branches, and not on this commit's: the message gives the commands to save them yourself, with qa save-artifacts.
Please configure async runners	The project has no task runner: runs may be killed if they take too long.
Why the batch has no tests, Check the batch files	Under the batch: see every problem flagged.
Under the Start button	What's missing (a batch label, a batch with tests), parameters that are not valid JSON or a function that fails, or too many runs.
Started …, after you click	It started, and where QA-Board will email you. Show its logs opens the Logs view.
… failed to start. Its logs say why.	Show its logs opens them.
No email when it ends: …	The batch started, but QA-Board can't email you, and says why.

Where to go next

• Logs and failed runs
  Read a run's log while it runs, find why it failed, see the commands that started a batch.
• Reading metrics
  The Summary, Metrics Table and Metrics Diff of your experiment against the CI.
• Project integrationLet people tune from the web app
  What a project needs for runs from the web app: artifacts, a task runner, emails.