Tips for CLI usage

CLI flags worth knowing

qa --help

All commands have some help:

qa --help
qa batch --help

qa --share

When you run qa batch or qa run on your terminal, results are saved locally under output/, and they are not visible in QA-Board. To make them visible, add --share: qa --share run ....

tip

If you don't like this default, make --share the default via

# .bashrc or other shell config
alias qa="qa --share"

# you can also use an environment variable (all options can be set as QA_<OPTION>)
export QA_SHARE=true

With --share, qa batch and qa optimize also save what they print, errors included, in the batch directory, under qa-batch-logs/. QA-Board shows these logs with the batch, in the Logs view. They are not saved in the CI, which keeps the logs. qa optimize saves the output of its iterations in its own log.

qa --dryrun

qa commands support a --dryrun mode, where they print actions they would take, but don't actually do anything. In particular it helps see quickly what inputs you defined in a batch:

qa --dryrun batch my-batch
# qa run --input image/A.jpg
# qa run --input image/B.jpg

note

For qa --dryrun run, you are expected to handle if context.dryrun: ... yourself in run(). The use-case is usually printing how you would call an executable, for debugging.

qa --label my-label

Every time you qa run, it erases previous results. So if you want compare different versions by tweaking doing qa run, it won't work. Fortunately, qa lets you give a "label", or "experiment name" to runs. Results with different labels are stored separately:

qa --label without-optimizations batch validation-images
qa --label    with-optimizations batch validation-images

Each label is a batch of its own: pick it in the web app's batch selector.

The CI batch, someone's local runs (🏠) and a tuning experiment, each with how many of its runs succeeded.

tip

To keep previous output files, use qa run --keep-previous (or export QABOARD_RUN_KEEP=true). It can be useful if you are debugging long runs and implemented a caching mechanism. (Experimental)

qa batch

Batch Runners

While qa run uses the local environment, qa batch will offload computation to a "runner" backend.

Available runners:

• local: local multiprocessing with joblib. Default if no other runner is configured, and on Windows instead of lsf. Set the concurrency with --local-concurrency, QA_BATCH_CONCURRENCY or runners.local.concurrency in qaboard.yaml.
• lsf: 1 LSF job per run. Limit how many run at the same time with --lsf-concurrency or runners.lsf.concurrency.
• dask: a pool of long-lived Dask workers, started as LSF jobs. Best for many short runs. Set the concurrency with --dask-concurrency or runners.dask.concurrency.
• celery: Celery workers.
• windows: runs sent to Windows hosts via Jenkins.

Choose the runner with --runner=local, and set a default with runners.default: local in qaboard.yaml. Without runners.default, qa batch uses the first runner configured under runners (other than local), else local.

Dealing with existing results

When you try to re-run already existing results, The behaviour of qa batch can be changed with the --action-on-existing flag:

• --action-on-existing=run: overwrite the old results (default).
• postprocess: only call the postprocess() function, not run()+postprocess() as usual. (Note: it's also provided by qa postprocess)
• sync: update the output file manifest and read metrics from $output_dir/metrics.json. (Note: it's also provided by qa sync)
• skip: do nothing
• assert-exists: fail if there are no results

The default can be set in qaboard.yaml with outputs.action_on_existing.

Similarly, --action-on-pending (or outputs.action_on_pending) decides what to do when the same runs are already pending: wait (default, then run), sync (use those runs' results), skip or run (run as usual, can cause races).

Running a list of runs

Instead of batches, qa batch --runs-file runs.json does the runs listed in a JSON file. QA-Board uses it to redo runs: they are started together, so the batch's runner settings (e.g. its concurrency) apply to all of them.

runs.json

{
  "runs": [
    {
      "input_path": "images/A.jpg",
      "database": "/datasets",
      "type": "image",
      "platform": "linux",
      "configurations": ["base", {"threshold": 0.5}],
      "extra_parameters": {"iterations": 3},
      "job_options": {"type": "dask", "concurrency": 20}
    }
  ]
}

Only input_path (relative to database) and database are required. Runs are done with the runner and options in job_options, completed by the defaults from qaboard.yaml. Options you give on the command line, like --runner or --lsf-queue, win.

---

Connecting to a custom QA-Board instance

Use qa --offline to ensure you don't connect to a QA-Board instance. It's useful if... you don't have one (?). The default connection settings can be overridden by environment variables. For example:

export QABOARD_PROTOCOL=http
export QABOARD_HOST=my-server:5151
# or: export QABOARD_URL=http://my-server:5151

See Client Installation.