The qa command line

qa runs your code: on one input with qa run, on a batch with qa batch. This page covers the options you use every day: sharing results, naming experiments, previewing a batch, choosing where it runs.

qa's own options, like --share, --label, --config, --platform, --dryrun and --offline, go before the command. The command's options, like --input or --runner, go after it:

qa --share --label ines-halo batch ci --runner local

Every command has its help: qa --help, qa batch --help...

The commands

Command	What it does	Read more
qa init	Adds qaboard.yaml and qa/ to your project.	Project setup
qa run --input <input>	Runs your code on one input.	Running your code
qa batch <batch>	Runs the inputs and configurations of a batch.	Batches
qa postprocess --input <input>	Redoes only your postprocess() function, on a run's files.	Postprocessing
qa sync --input <input>	Sends a run's metrics.json to QA-Board again, with --share or in CI.
qa get <setting>	Prints a setting, e.g. qa get config, qa get --input A.jpg output_dir.	API
qa runs <url>	Lists the runs a web app page shows: their folders, or everything with --json.	Runs in scripts
qa save-artifacts	Saves what your runs need to be redone from the web app, e.g. your build.	Artifacts
qa check-bit-accuracy	Compares a commit's output files with a reference's, byte for byte.	Bit accuracy
qa gate	Experimental: fails the CI when metrics regress.	Quality gate
qa optimize	Searches for the best parameters, and continues a search it saved.	Automated tuning

See results in the web app: --share

From your computer, qa run and qa batch save results in output/, in your project: the web app doesn't see them. Add --share to save them in the shared storage and send them to QA-Board. qa then prints the link to them:

qa --share batch ci

In CI, qa always shares results. To share them by default from your computer too:

# in your shell's profile
export QA_SHARE=true

With --share, qa batch and qa optimize also save what they print, errors included, in the batch's folder, under qa-batch-logs/. The web app shows these logs with the batch, in the Logs view (see batch commands). In CI, they aren't saved: the CI keeps the logs. qa optimize saves the output of its iterations in its own log.

Name experiments: --label

Each qa run replaces the previous results of the same input and configuration. To keep several versions, e.g. before and after a change, give each one a label:

qa --share --label ines-before batch ci
qa --share --label ines-halo batch ci

Each label is a batch of its own: pick it in the web app's batch selector. Shared from your computer, a batch shows your name and 🏠.

The CI batch, someone's local runs (🏠) and a tuning experiment, each with how many of its runs succeeded.

tip

To keep the previous output files of a run, e.g. when your code caches intermediate results while you debug a long run, use qa run --keep-previous (or export QABOARD_RUN_KEEP=true).

Preview: --dryrun

With --dryrun, qa prints what it would do, without doing it. For qa batch, it prints the qa run command of each run, e.g. to check what a batch runs. In the detector example:

qa --dryrun batch ci

qa --configuration fp32:size-320 run --input "slices/indoor"
   /home/arjun/kitefin/detector/output/7622c337-20/slices-indoor
[...]

For qa --dryrun run, run() is called with context.dryrun set to True: it's up to your code not to do anything, e.g. print the command it would run.

Choose where batches run: runners

qa run runs on your machine. qa batch hands its runs to a runner:

• local: runs on this machine, one at a time unless you set a concurrency: --local-concurrency, QA_BATCH_CONCURRENCY or runners.local.concurrency in qaboard.yaml (-1: one run per CPU). It's the default when no other runner is configured, and on Windows instead of lsf.
• lsf: one LSF job per run. Limit how many run at the same time with --lsf-concurrency or runners.lsf.concurrency.
• dask: a pool of long-lived Dask workers, started as LSF jobs or on an existing Dask cluster. Best for many short runs. Set the concurrency with --dask-concurrency or runners.dask.concurrency.
• celery: Celery workers.
• windows: runs sent to Windows hosts via Jenkins.

Choose the runner with --runner, e.g. qa batch --runner local ci, and set a default with runners.default in qaboard.yaml. Without runners.default, qa batch uses the first runner configured under runners other than local, else local. Batches can change their runs' settings, like LSF's memory: see runner settings per batch.

Runs that already ran

By default, qa batch runs everything again. --action-on-existing changes what it does with runs that already succeeded:

• run: run them again (the default).
• postprocess: only call your postprocess() function, like qa postprocess.
• sync: send their metrics.json to QA-Board again, like qa sync.
• skip: do nothing.
• assert-exists: fail if some runs have no results. Like skip, it does nothing with those that succeeded.

Runs that failed, or never ran, always run. The default can be set in qaboard.yaml, with outputs.action_on_existing.

When the same runs are already pending, e.g. started by an earlier command, --action-on-pending (or outputs.action_on_pending) decides: wait for them, then run (the default), sync (use their results), skip, or run as usual, which can cause races.

To redo a list of runs, give them in a file: see --runs-file.

Email when the runs end

QA-Board emails you when the runs of qa --share batch end:

qa --share batch ci

QA-Board will email ines@kitefin.example when the runs end (`export QA_NOTIFY=none` to stop).

It's the same email users get when they start a batch from the web app (what it says): one per qa batch, when its last run ends.

• To stop them, export QA_NOTIFY=none.
• It needs --share: QA-Board only knows about the runs it saves.
• It's never sent in CI, nor for qa batch commands started by other ones, like the steps of a pipeline or the iterations of qa optimize: their runs count as the first command's runs, and its email comes when they are all done. Nested qa batch commands know the first one from the QA_BATCH_ROOT_COMMAND_ID environment variable, which all runners forward to the runs (local, LSF, Dask and Celery).

Work without a server: --offline

With qa --offline, qa never contacts QA-Board: results are only saved on disk. To use another server, see Connect to your server.

Outside CI, qa also tells the server which command ran, so that admins see how QA-Board is used (what it sends). QA_NO_ANALYTICS=1 or --offline keep it quiet.

Options as environment variables

Every option can be set in the environment: QA_<OPTION> for qa's own options, QA_<COMMAND>_<OPTION> for a command's, in capitals and with _ for -:

export QA_SHARE=true                  # qa --share
export QA_BATCH_RUNNER=lsf            # qa batch --runner lsf
export QA_BATCH_LSF_CONCURRENCY=20    # qa batch --lsf-concurrency 20

Options given on the command line win. More variables that change what qa batch does are in Advanced batches.