Batches of inputs

A batch is a named list of inputs and configurations, in a YAML file. qa batch ci runs them all, in parallel or on a cluster, and the web app shows their runs together. Your CI runs a batch on every commit; others run nightly, before a release, or to try parameters.

Write a batch

The inference gateway example measures its latency on 5 load profiles in CI:

qa/batches.yaml

# CI, on every commit: about 45 s on a 4-CPU runner. Five tests, so that the median over them (what the gate and
# the commit list show) is one test's value, and a regression in a single test doesn't move it much.
ci:
  inputs:
  - profiles/steady-low.yaml
  - profiles/steady-mid.yaml
  - profiles/steady-high.yaml
  - profiles/bursty-high.yaml
  - profiles/heavy-tail-high.yaml

From examples/serving-latency/qa/batches.yaml

Inputs are paths relative to your database, inputs/ here. A batch can also list folders and wildcards like profiles/*.yaml: qa finds the inputs in them with inputs.globs.

Batches are in qa/batches.yaml, or in the files that inputs.batches lists in qaboard.yaml.

Run a batch

To see what a batch would run, without running anything, use --dryrun. It prints the qa run command of each run, and where its outputs go:

qa --dryrun batch ci

qa run --input "profiles/steady-low.yaml"
   /home/kaito/kitefin/inference-gateway/output/linux/profiles-steady-low
qa run --input "profiles/steady-mid.yaml"
   /home/kaito/kitefin/inference-gateway/output/linux/profiles-steady-mid
[...]
qa run --input "profiles/heavy-tail-high.yaml"
   /home/kaito/kitefin/inference-gateway/output/linux/profiles-heavy-tail-high

Then run it, with --share to see the results in the web app. qa prints the link to the commit's results first:

qa --share batch ci

qa --share batch ci on the camera pipeline: qa prints the link to the results, then runs each input and configuration of the batch.

• Run several batches at once: qa batch ci capacity, or with a wildcard: qa batch 'sweep-*'.
• Arguments after -- are given to each run, and to your code as context.forwarded_args: qa batch ci -- --verbose runs qa run --input "profiles/steady-low.yaml" --verbose...
• Runs remember the name of their batch: the web app shows it, and you can filter runs by it.
• By default, qa batch runs on this machine. To run several runs at the same time, or on a cluster, see runners.

Configurations in a batch

configs gives the configurations the batch's inputs run with. The camera ISP example checks the fixed-point model of its camera chip, the hw-model platform, at ISO 100 (base):

qa/batches.yaml

# The fixed-point model of the camera chip, compared commit to commit by `qa check-bit-accuracy`
bitexact:
  platform: hw-model
  configs: [base]
  inputs:
  - charts/*.png
  - photos/facade.png
  - photos/night-street.png

From examples/image-quality/qa/batches.yaml

To run each input with several configurations, give them in a matrix. Its CI runs 9 images at ISO 100 and 3200: 18 runs.

qa/batches.yaml

# CI: the charts and 4 photos, in good and low light
ci:
  inputs:
  - charts/*.png
  - photos/facade.png
  - photos/backlit-window.png
  - photos/night-street.png
  - photos/sky-gradient.png
  matrix:
    configs: [base, low-light]

From examples/image-quality/qa/batches.yaml

A matrix can also combine values, which ${matrix.<name>} puts in the configurations. The detector example runs 4 slices of its test set with the fp32 model and the int8 model:

qa/batches.yaml

# CI, on every commit: 4 slices, fp32 and int8, at the release's input size
ci:
  inputs:
  - slices/indoor
  - slices/outdoor
  - slices/warehouse
  - slices/small-objects
  configs: ['${matrix.model}', size-320]
  matrix:
    model: [fp32, int8]

From examples/detection-eval/qa/batches.yaml

qa --dryrun batch ci

qa --configuration fp32:size-320 run --input "slices/indoor"
   /home/arjun/kitefin/detector/output/7622c337-20/slices-indoor
[...]
qa --configuration int8:size-320 run --input "slices/small-objects"
   /home/arjun/kitefin/detector/output/6d41e2b9-20/slices-small-objects

Its sizes batch adds size: [320, 416, 512] to the matrix: 4 slices × 2 models × 3 sizes, 24 runs. See matrices for more.

An input can add its own configurations to the batch's:

qa/batches.yaml

my-batch:
  configs: [int8]
  inputs:
  - slices/indoor                                      #=> [int8]
  - slices/outdoor: [size-416, {score_threshold: 0.2}] #=> [int8, size-416, {score_threshold: 0.2}]
  - slices/small-objects: {nms_iou: 0.7}               #=> [int8, {nms_iou: 0.7}]

The same input twice, with other configurations

An input can appear several times, each time with its own configurations. These forms are the same:

qa/batches.yaml

crops:
  configs: [base]
  inputs:
  - image.raw: {crop: A}       #=> [base, {crop: A}]
  - image.raw: {crop: B}       #=> [base, {crop: B}]
  - [image.raw, {crop: C}]     #=> [base, {crop: C}]

Database and platform

A batch can set its own database, where its inputs are, and its platform, like the camera ISP's bitexact batch above. Runs are compared with the reference's runs on the same platform. At the top of a batch file, database, configs and platform are defaults for the file's batches that don't set their own:

qa/batches.yaml

database: /mnt/datasets          # for all the batches of this file

my-batch:
  inputs:
  - images/A.jpg                 #=> /mnt/datasets/images/A.jpg

archive:
  database:                      # this batch's inputs are elsewhere
    linux: /mnt/archive
    windows: '\\storage\archive'
  inputs:
  - images/A.jpg

Runner settings per batch

Under the name of a runner, a batch changes the settings of its runs, e.g. how much memory LSF gives them. The camera ISP's 113 MP test wall needs more memory than the other images:

qa/batches.yaml

deep-zoom:
  configs: [base]
  inputs:
  - charts/test-wall-113mp.tiff
  lsf:
    max_memory: 3072   # MB

Runner settings are merged in this order, the last one wins:

1. runners.<runner> in qaboard.yaml (e.g. runners.lsf.queue),
2. runners.<runner> in the input type,
3. <runner>: in the batch, as above,
4. command line options, like qa batch --lsf-queue.

A batch can also limit how many of its runs run at the same time, with concurrency under local:, lsf: or dask:, and set the local runner's timeout. How batches with their own limit share the resources depends on the runner:

• local: they run one after the other.
• LSF: each gets its own job arrays (or waves), sent at the same time, so their limits add up.
• Dask: their runs share the workers, and qa batch never has more runs in flight than the largest concurrency.

caution

A batch can't choose its runner yet: qa batch --runner, or runners.default in qaboard.yaml, does, for all batches. A batch's runner: doesn't change it. With Celery, a batch's celery: settings are ignored too: the Celery configuration comes from qaboard.yaml.

Groups of batches

Name batches you often run together with aliases. In the camera ISP example, qa batch nightly runs every input at every ISO, and the fixed-point model:

qa/batches.yaml

aliases:
  nightly: [all-isos, bitexact]

From examples/image-quality/qa/batches.yaml

Aliases can list other aliases, and wildcards like sweep-*.

Edit and check batches in the web app

The Available Tests view edits private and shared batches, flags YAML mistakes and unknown keys as you type, and lists the tests each batch resolves to on a commit:

A YAML mistake is underlined and listed in the problems below the editor; the Batches panel lists the file's batches.

The ci batch resolves to 8 tests on this commit: 4 inputs, each with the fp32 and the int8 model.

Avoid mistakes

• Inputs are paths. Put in quotes those YAML would read as something else, like numbers ("20260223"), dates ("2026-02-23") or yes/no. A - with nothing after it is an empty input: qa batch stops and says on which line it is.
• Check which inputs a batch finds with qa batch <batch> --list-inputs: it prints each input once per configuration. --list and --list-output-dirs print more, see the API page.
• Get completion and warnings in your editor with the batch files' schema.

• Advanced batches

  Matrices, YAML anchors, inputs selected by their metadata, several batch files, lists of runs.

• The qa command line

  Share results, name experiments, choose a runner, skip what already ran.