Skip to main content

Let people tune from the web app

With a few settings, anyone on your project can run a commit's code on other tests, or with other parameters, from the commit page's Run Tests / Tuning form, without a terminal. This page is how a project enables it.

Looking for how to start runs?

See Running from the web app in the user guide.

A grid of 2 score thresholds by 2 NMS IoUs on 2 tests: Start 8 runs, then compare the runs in the Tuning Analysis.

On the inference gateway, a performance engineer swept the batching settings this way: 3 maximum batch sizes by 4 maximum delays. The Tuning Analysis view plotted each configuration's CPU per request against its p99 latency, and the settings on that frontier became the new defaults.

The efficient frontier of the tune-batching sweep: one point per configuration of max batch and max delay, CPU per request against p99, colored by max delay. The hovered point, max batch 16 and max delay 250 µs, sits on the lower left edge.The efficient frontier of the tune-batching sweep: one point per configuration of max batch and max delay, CPU per request against p99, colored by max delay. The hovered point, max batch 16 and max delay 250 µs, sits on the lower left edge.
Each configuration's CPU per request against its p99: the defaults picked, max batch 16 and 250 µs of delay, are on the frontier.

Enable it​

  1. Save the build in the CI. Runs started from the web app use the commit's artifacts: the files your code needs to run, like a build, saved by qa save-artifacts (see CI integration). qaboard.yaml, qa/ and the batches files are always saved.
  2. Choose where the runs execute. The server starts each batch with the runner set by its QABOARD_TUNING_RUNNER environment variable: local (the default, on the server itself), celery or lsf (via QA_RUNNERS_LSF_BRIDGE). See Celery, LSF and the server's environment variables.
  3. Log in. Starting runs needs a logged-in user, and runs are started as that user. See user management.
  4. (Optional) Emails. If the server can send emails, people get one when the batches they start end: see Emails.

To check: open a commit of your CI on its commit page, then Run Tests / Tuning. Pick your CI batch and the no tuning example, and send. The batch's runs start, and the Logs view lists the batch under "Started from QA-Board", with the qa batch command QA-Board ran and its log.

The Logs view of the tune-batching sweep: under "Started from QA-Board", the qa batch command kaito's tuning request ran, with its grid of max batch and max delay, and the end of its log; then the two runs left by the filter, and the Batch section with the command again.The Logs view of the tune-batching sweep: under "Started from QA-Board", the qa batch command kaito's tuning request ran, with its grid of max batch and max delay, and the end of its log; then the two runs left by the filter, and the Batch section with the command again.
A sweep started from the web app: the Logs view shows the qa batch command QA-Board ran, its log, and its runs.

How the batch starts​

For each request, the server writes a script in the batch's directory and runs it with the chosen runner. The script:

  • goes to the commit's artifacts folder, and sources the .envrc files of that folder and of its parents: activate your environment there (a virtualenv, modules...), it's saved with the artifacts;
  • sets CI=true, GIT_COMMIT to the commit, QA_USER to the user who started it, and QABOARD_TUNING=true;
  • runs qa --label <label> batch --batches-file <the batches> --batch <the batch> --tuning-search ..., or qa optimize for automated tuning.

When the batch doesn't start, the Logs view says why, with the end of that script's log: a ModuleNotFoundError points to the environment, an LSF report to the cluster. See Logs and failed runs.

How parameters reach your code​

Tuning parameters are merged into context.params, and appended to context.configs as a dict: in most cases you have nothing to do. The detector's run() lets them override its defaults and the flags each commit builds with:

qa/main.py
def read_flags(params):
    flags = dict(DEFAULT_FLAGS)
    path = ROOT / 'build' / 'flags.json'
    if path.exists():
        flags.update({k: v for k, v in json.loads(path.read_text()).items() if v is not None})
    flags.update({k: v for k, v in params.items() if k in DEFAULT_FLAGS})   # tuning parameters
    return flags

From examples/detection-eval/qa/main.py

The same parameters work from a terminal, with --tuning for one set of values:

qa --label int8-nms --tuning '{"score_threshold": 0.3, "nms_iou": 0.5}' batch quant-search

Settings in qaboard.yaml​

qaboard.yaml
inputs:
  # Let users choose a platform, passed to `qa --platform`. By default runs use "linux".
  platforms:
    - name: linux
      label: Linux
    - name: hw-model
      label: Fixed-point model

tuning:
  # If your CI only saves artifacts on some branches, list them (globs, "*" matches "/" too):
  # on the commits of other branches, the tuning form warns that tuning may not work, and how to save the artifacts.
  # In monorepos, {subproject} is the project's folder in the repository and {subproject.name} its last part.
  branches:
    - main
    - '*/*{subproject.name}*'  # e.g. alice/denoise-v2 for the project in isp/denoise

People choose the tests from the batches files of the commit, and can add their own in the Available Tests view (see the user guide). For the batches' syntax, see Batches of inputs.

Shortcuts for people who tune
  • To start from an existing run's parameters, hover its configuration tags and choose Copy as a tuning search space, then paste it in the Manual tuning editor.
  • Pasting lines of key=value in the Manual tuning editor converts them to the tuning format: block.threshold=10 becomes "block.threshold": [10], and space-separated matrices like [1 2 3] become JSON arrays.
  • In the Available Tests view, Ctrl+S (or Cmd+S) saves the batches file you are editing.
  • Once runs are done, the Tuning Analysis view shows how the metrics depend on the parameters: the sensitivity to each parameter and to pairs of parameters, a parallel coordinates plot, and the trade-off between two metrics. See the user guide. For an automated tuning, it starts with a report of what the search found and what to do next: see Read the results.

Other tuning workflows​

Tuning from the web app runs the code of a commit. Depending on where you are in your work, other workflows fit better:

  • Locally, with your own tools: if you already have good development and debugging tools (an IDE, tensorboard for neural networks, one-off scripts...), keep using them. It's still worth having them call your code through qa, so that runs behave the same everywhere: see debugging with IDEs.
  • Your local code, results in QA-Board: tweak, compile and run your code, then compare the results across runs in the web app. Runs started with --share appear in a batch named after you, with a warning that they may not match the committed code: 
    qa --share run --input slices/small-objects
    qa --share --label testing-some-logic-tweaks batch quant-search
  • Commit, CI, QA-Board: to change configuration files that live in your repository, commit and push the changes, and look at the results of the CI runs.
  • Automated tuning: let qa optimize search the parameters for you, from the web app or the CLI. See Find the best parameters automatically to start one and read its results, and Automated tuning with qa optimize for every setting.