Batches of inputs

Reminder

Make sure you read the section on input files

As we discussed, you can define batches of inputs in file(s) whose paths are given by inputs.batches in your qaboard.yaml configuration.

qa/batches.yaml (default)

my-batch:
 inputs:
   - A.jpg
   - B.jpg

qa batch my-batch
#=> qa run --input A.jpg
#=> qa run --input B.jpg

qa batch first-batch second-batch

Runs started by qa batch remember the name of their batch: the web application shows it, and you can use it in the filter.

qa --share batch ci on the camera pipeline: qa prints the link to the results, then runs each input and configuration of the batch.

Arguments after -- are forwarded to each qa run, and to your code as context.forwarded_args (it works the same with qa run and qa postprocess):

qa batch my-batch -- --my-flag value
#=> qa run --input A.jpg --my-flag value
#=> qa run --input B.jpg --my-flag value

Use qa batch my-batch --list-inputs to only print the inputs, and see the API page for --list and --list-output-dirs.

tip

Inputs are paths. Put in quotes those YAML would read as something else, like numbers ("20260223"), dates ("2026-02-23") or yes/no. A - with nothing after it is an empty input: qa batch stops and says on which line it is.

In the web app, the Available Tests view edits private and shared batches, flags YAML mistakes as you type, and lists the tests each batch resolves to on a commit:

A YAML mistake is underlined and listed in the problems below the editor; the Batches panel lists the file's batches.

The ci batch resolves to 8 tests on this commit: 4 inputs, each with the fp32 and the int8 model.

Setting a custom database

In QA-Board relative input paths are considered relative to a "database" folder (defaults to / unless changed with inputs.database in qaboard.yaml).

You can set a default database for all the batches in define in each YAML file:

database: /some/path

my-batch:
  inputs: # will be relative to the database
  - Images/Demo3/A.jpg
  - Images/Demo2

Setting a custom database per batch

you-can-override-the-default-database:
  database:
    linux: /mnt/database
    windows: '\\\\storage\\database'
  inputs:
  - Images/Demo3/A.jpg
  - Images/Demo2

Specifying test configurations

Reminder

Make sure you read the section on configurations

Let's look at examples from an image processing project to illustrate how configurations can be given:

using-a-custom-configuration:
  configs:
  - base
  inputs:
  - Bayer/A.dng
  - Bayer/B.dng
# => configs == ["base"]
# => the code would load base/config.yaml

multiple-configs:
  configs:
    - base
    - low-light
  inputs:
  - Bayer/A.dng
  - Bayer/B.dng
#=> configs == ["base", "low-light"]
#=> the code merges 2 configs

configs-can-be-complex-objects:
  configs:
    - base
    - low-light
    - simulator:
      - "-w 9920"
      - "-h 2448"
      - "-it BAYER10"
  inputs:
  - Bayer/A.dng
  - Bayer/B.dng
# configs == ["base", "low-light", {"simulator": ["-w 9920", "-h 2448", "-it BAYER10"]}]
# => Here we use the "simulator" config parameter to pass CLI arguments to a hardware simulator.

each-input-can-have-its-own-configuration:
  configs:
    - base
  inputs:
  - Bayer/A.dng:
    #=> configs == ["base"]
  - Bayer/B.dng:
      - low-light
      - simulator:
        - "-DD"
    #=> configs == ["base", "low-light", {"simulator": ["-DD"]}]


each-input-can-have-its-own-configuration-and-appear-twice:
  configurations:
    - base
  inputs:
  - a.txt: {crop: A}
    #=> configurations == ["base", {"crop": ["A"]}]
  - a.txt: {crop: B}
    #=> configurations == ["base", {"crop": ["B"]}]
  - [a.txt, {crop: C}]
    #=> configurations == ["base", {"crop": ["C"]}]

You can also set top-level default configs (and platform) in your YAML files. Batches can also set their own platform.

Runner settings per batch

Under a key named after the runner, batches can change the runner's settings. They are used when the batch runs with this runner:

my-batch:
  inputs:
  - A.jpg
  lsf:
    queue: my_long_queue
  local:
    concurrency: 2

See in what order runner settings are merged.

Matrix batches

You can use "matrix" batches to run on combinations of configurations (and platfornms):

multiple-configs:
  inputs:
  - a.raw
  matrix:
    configs:
      -
          - base
          - tuning
      -
          - base

#=> `qa batch multiple-configs` will run
#   with [base, tuning] and [base]

Per-input extra configurations still work.

The syntax was inspired from DroneCI, GithubActions, AzurePipelines.

Interpolation

Matrix variables are interpolated using the ${matrix.variable} syntax:

my-batch-multiple-configs:
  inputs:
  - image.raw
  matrix:
    version: [1, 2]
  configs:
    - base-v${matrix.version}

# => will run with [base-v1] and [base-v2]

And elaborate complex testing strategies:

my-batch-multiple-values:
  inputs:
  - image.raw
  matrix:
    threshold: [1, 2, 3, 4]
    mode: ["a", "b"]
  configs:
    - base
    - block.threshold: ${matrix.threshold}
    - block.mode: ${matrix.mode}

# => will start 8 runs run with
#     [base, block.threshold: 1, block.mode: a
#     [base, block.threshold: 1, block.mode: b
#     [base, block.threshold: 2, block.mode: a
#     etc

Advanced users can use indexing: ${matrix.param[0]} or ${matrix.param[key]} according to the python formatting miniformat, if they use "complex" matrix parameters with objects/arrays/strings:

my-batch-matrix-object:
  inputs:
  - image.raw
  matrix:
    point: [
      {x: 10, y: 40},
      {x: 20, y: 50},
    ]
  configs:
    - base
    - flags: "-x=${matrix.point[x]} -y=${matrix.point[y]}"

Aliases for groups of batches

For convenience you can define aliases for batches you often run together. For instance you can do:

qa/batches.yaml

aliases:
  two-batches:
  - first-batch
  - second-batch

qa batch two-batches

Dynamic groups of batches

You can use wildcards:

qa/batches.yaml

batch-1:
   # ...

batch-2:
   # ...

qa batch 'batch-*'
#=> runs both batch-1 and batch-2

Aliases also understand wildcards:

aliases:
  all:
  - batch-*

qa batch all
#=> runs both batch-1 and batch-2

YAML anchors and aliases

For convenience you can define YAML anchors for common configurations

.base: &base
  - base
  - partial

hdr:
  configs:
    - *base
    - hdr
  inputs:
    - A
    - B
    - C
#=> configs == ["base", "partial", "hdr"]

note

YAML "aliases" and "anchors" are standard YAML feature. Read more here.

Reusable configurations/inputs

Sometimes you want to mix and match reusabe definitions of configs and inputs. YAML anchors let you do it:

.inputs_hdr: &inputs_hdr
  inputs:
    - A
    - B

.lots_of_inputs_hdr: &lots_inputs_hdr
  inputs:
    - A
    - B
    - C
    - D
    - E
    - F

.HDR: &HDR
  configs:
    - *base
    - hdr_base
    - hdr_motion

.HDR-disabled: &HDR-disabled
  configs:
    - *base

hdr:
  <<: *HDR
  <<: *inputs_hdr
no-hdr:
  <<: *HDR-disabled
  <<: *inputs_hdr
# qa --label hdr    batch hdr
# qa --label no-hdr batch no-hdr

# Maybe on nightly runs you want to run lots of inputs
lots-of-hdr-inputs:
  <<: *HDR
  <<: *lots_inputs_hdr
lots-of-no-hdr-inputs:
  <<: *HDR-disabled
  <<: *lots_inputs_hdr

Environment variables

Variable	Usage
QA_BATCH_ALLOW_MANY_RUNS	qa batch refuses to send more than 1000 runs to QA-Board at once. Set it to allow it (or use qa --offline)
QA_BATCH_FILTER_INPUT_REGEX	Only keep the inputs whose path matches this regular expression (when using inputs.globs)
QA_BATCH_FAIL_IF_EMPTY	Fail instead of warning when a batch has no inputs
QA_BATCH_QUIET	Don't print the qa run command of each run
QA_BATCH_CONCURRENCY, QA_BATCH_TIMEOUT	Defaults for the local runner