Skip to main content

Tips for CLI usage

CLI flags worth knowing​

qa --help​

All commands have some help:

qa --help
qa batch --help

qa --share​

When you run qa batch or qa run on your terminal, results are saved locally under output/, and they are not visible in QA-Board. To make them visible:

tip

If you don't like this default, make --share the default via 

# .bashrc or other shell config
alias qa="qa --share"

# you can also use an environment variable
export QA_SHARE=true

qa --dryrun​

qa commmands support a --dryrun mode, where they print actions they would take, but don't actually do anything. In particular it helps see quickly what inputs you defined in a batch:

qa --dryrun batch my-batch
# qa run --input image/A.jpg
# qa run --input image/B.jpg
note

For qa --dryrun run, you are expected to handle if context.dryrun: ... yourself in run(). The use-case is usually printing how you would call an executable, for debugging.

qa --label my-label​

Everytime you qa run, it erases previous results. So if you want compare different versions by tweaking doing qa run, it won't work. Fortunately, qa lets you give a "label", or "experiment name" to runs. Results with different labels are stored separately:

qa --label without-optimizations batch validation-images
qa --label    with-optimizations batch validation-images
select-batch
tip

To keep previous output files, use qa batch/run --keep-previous or EXPORT QA_KEEP_PREVIOUS=true. It can be useful if you are debugging long runs and implemented a caching mecanism. (Experimental)

qa batch​

Batch Runners​

While qa run uses the local environment, qa batch will offload computation to a "runner" backend.

Currently:

  • On Windows we use joblib for parallel computing. You can set the concurrency with QATOOLS_BATCH_CONCURRENCY and other environment variables from joblib. runners.local.concurrency in qaboard.yaml also works...
  • On linux we use SIRC's LSF cluster

You can also set the runner via --runner=local, and even set a default with runners.default: local in qaboard.yaml.

Dealing with existing results​

When you try to re-run already existing results, The behaviour of qa batch can be changed with the --action-on-existing flag:

  • --action-on-existing=run: overwrite the old results (default).
  • postprocess: only call the postprocess() function, not run()+postprocess() as usual. (Note: it's also provided by qa postprocess)
  • sync: update the output file manifest and read metrics from $output_dir/metrics.json. (Note: it's also provided by qa sync)
  • skip: do nothing

Connecting to a custom QA-Board instance​

Use qa --offline to ensure you don't connect to a QA-Board instance. It's useful if... you don't have one (?). The default connection settings can be overriden by environment variables. For example:

export QABOARD_DB_PROTOCOL=http
export QABOARD_DB_HOST=qa
export QABOARD_DB_PORT=5000