Batches of inputs

Reminder

Make sure you read the section on input files

As we discussed, you can define batches of inputs in file(s) whose paths are given by inputs.batches in your qaboard.yaml configuration.

qa/batches.yaml (default)
my-batch:
inputs:
- A.jpg
- B.jpg
qa batch my-batch
#=> qa run --input A.jpg
#=> qa run --input B.jpg
qa batch --batch first-batch --batch second-batch

Setting a custom database per batch

you-can-override-the-default-database:
database:
linux: /mnt/database
windows: '\\\\storage\\database'
inputs:
- Images/Demo3/A.jpg
- Images/Demo2

Specifying test configurations

Reminder

Let's look at examples from one of Samsung's projects to illustrate how configurations can be given. In short, each image is processed by a tool named "CDE". We need to feed it config files and CLI arguments.

using-a-custom-configuration:
configs:
- base
inputs:
- Bayer/A.dng
- Bayer/B.dng
# => configs == ["base"]
# => the code would load base/config.cde
multiple-configs:
configs:
- base
- low-light
inputs:
- Bayer/A.dng
- Bayer/B.dng
#=> configs == ["base", "low-light"]
#=> we merge 2 CDE configs
configs-can-be-complex-objects:
configs:
- base
- low-light
- cde:
- "-w 9920"
- "-h 2448"
- "-it BAYER10"
inputs:
- Bayer/A.dng
- Bayer/B.dng
# configs == ["base", "low-light", {"cde": ["-w 9920", "-h 2448", "-it BAYER10"]}]
# => Here we use the "cde" config parameter to pass CLI arguments to CDE.
each-input-can-have-its-own-configuration:
configs:
- base
inputs:
- Bayer/A.dng:
#=> configs == ["base"]
- Bayer/B.dng:
- low-light
- cde:
- "-DD"
#=> configs == ["base", "low-light", {"cde": ["-DD"]}]

Matrix batches

You can use "matrix" batches to run on combinations of configurations (and platfornms):

multiple-configs:
inputs:
- a.raw
matrix:
configs:
-
- base
- tuning
-
- base
#=> `qa batch multiple-configs` will run
# with [base, tuning] and [tuning]

Per-input extra configurations still work.

The syntax was inspired from DroneCI, GithubActions, AzurePipelines.

Interpolation

Matrix variables are interpolated using the ${matrix.variable} syntax:

my-batch-multiple-configs:
inputs:
- image.raw
matrix:
version: [1, 2]
configs:
- base-v${matrix.version}
# => will run with [base-v1] and [base-v2]

And elaborate complex testing strategies:

my-batch-multiple-values:
inputs:
- image.raw
matrix:
threshold: [1, 2, 3, 4]
mode: ["a", "b"]
configs:
- base
- block.threshold: $matrix.threshold
- block.mode: $matrix.mode
# => will start 8 runs run with
# [base, block.threshold: 1, block.mode: a
# [base, block.threshold: 1, block.mode: b
# [base, block.threshold: 2, block.mode: a
# etc

Aliases for groups of batches

For convenience you can define aliases for batches you often run together. For instance you can do:

qa/batches.yaml
aliases:
two-batches:
- first-batch
- second-batch
qa batch two-batches

YAML anchors and aliases

For convenience you can define YAML anchors for common configurations

.base: &base
- base
- partial
hdr:
configs:
- *base
- hdr
inputs:
- A
- B
- C
#=> configs == ["base", "partial", "hdr"]
note

YAML "aliases" and "anchors" are standard YAML feature. Read more here.

Reusable configurations/inputs

Sometimes you want to mix and match reusabe definitions of configs and inputs. YAML anchors let you do it:

.inputs_hdr: &inputs_hdr
inputs:
- A
- B
.lots_of_inputs_hdr: &lots_inputs_hdr
inputs:
- A
- B
- C
- D
- E
- F
.HDR: &HDR
configs:
- *base
- hdr_base
- hdr_motion
.HDR-disabled: &HDR-disabled
configs:
- *base
hdr:
<<: *HDR
<<: *inputs_hdr
no-hdr:
<<: *HDR-disabled
<<: *inputs_hdr
# qa --batch-label hdr batch hdr
# qa --batch-label no-hdr batch no-hdr
# Maybe on nightly runs you want to run lots of inputs
lots-of-hdr-inputs:
<<: *HDR
<<: *lots_inputs_hdr
lots-of-no-hdr-inputs:
<<: *HDR-disabled
<<: *lots_inputs_hdr