Inputs

Inputs are what your code runs on: an image, a video, a recording, a load profile, a folder of files. qa finds them as paths, relative to a folder you set in qaboard.yaml: the database. This page sets it up, so that qa run --input and qa batch find your inputs.

Set where your inputs are

The camera ISP example keeps its inputs in the project, in its inputs/ folder:

qaboard.yaml

inputs:
  # relative to this folder: generated charts, downloaded photos, and videos made from them (clips/*.yaml)
  database:
    linux: inputs
    windows: inputs
  globs: ['*.png', '*.tiff', 'clips/*.yaml']
  batches: qa/batches.yaml

From examples/image-quality/qaboard.yaml

Inputs are then paths relative to that folder:

qa run --input charts/slanted-edge.png
#=> runs on inputs/charts/slanted-edge.png

• database is relative to the project's root, where qaboard.yaml is, or absolute, e.g. /mnt/datasets. Give a linux and a windows path when the folder is mounted at different places. Without database, it's /.
• {project}, {subproject} and environment variables like $DATASETS are replaced in it.
• An absolute input path works too: qa run --input /mnt/datasets/images/A.jpg.
• qa --database <folder> run ... uses another database for one command.

Recognize inputs inside folders

To run on every image of a folder, qa must know what an input looks like. Give it glob patterns in inputs.globs: above, the camera ISP's inputs are the PNG and TIFF images, and the YAML files that describe its videos. Batches can then list folders and wildcards, and qa finds the inputs in them, in subfolders too:

qa/batches.yaml

my-photos:
  inputs:
  - photos             # every input under inputs/photos
  - charts/*.png

When each input is a folder, recognize it by a file it contains, and set use_parent_folder. In the detector example, each slice of the test set is a folder with its images and a labels.json:

qaboard.yaml

inputs:
  # relative to this folder: each slice of the test set is a folder with its images and labels.json
  database:
    linux: inputs
  globs: labels.json
  use_parent_folder: true
  batches: qa/batches.yaml

From examples/detection-eval/qaboard.yaml

qa run --input slices/indoor

tip

To see which inputs a batch finds, before running anything: qa batch <batch> --list-inputs. Inputs are sorted by path, so a batch runs them in the same order on every machine.

Inputs in the example projects

Each example project in the repository shows a kind of input, with its configurations and metrics:

Project	An input	Configurations	Metrics
Camera ISP	An image: a test chart, a photo, the 113 MP test wall	The ISO: base, low-light, night	PSNR, SSIM, MTF50, edge overshoot, ΔE 2000
Detector	A slice of the test set: a folder of images and their labels	The model (fp32, int8) and its input size	mAP, AP on small objects, latency
Inference gateway	A load profile: the request rate, bursts, payload sizes	How long to measure	p50 and p99 latency, throughput
SLAM	A sequence: a folder with a trajectory through a warehouse and its ground truth		Trajectory error (ATE), drift
Fleet assistant	A task: the operator's request and the checks to pass		Pass rate, unsafe actions, cost
Fleet SDK tests	A test suite: a folder of tests	The mode: default, strict (warnings are errors)	Tests, failed, flaky and skipped tests, coverage

Batches list these inputs with their configurations, to run them together: see Batches of inputs.

Input metadata

Your entrypoint can give each input metadata: a label, tags, the regions where to measure... run() gets them in context.input_metadata, and batches can select inputs by their metadata. The camera ISP example reads them from a YAML file next to each photo:

qa/main.py

def metadata(context):
    """Input metadata: ROI presets for the image viewer, and the regions charts are measured in."""
    path = context.input_path.with_suffix('.metadata.yaml')
    return yaml.safe_load(path.read_text()) if path.exists() else {}

From examples/image-quality/qa/main.py

See Metadata and external databases, also for inputs that aren't files: unit tests, rows of a database...

Several types of inputs

Some projects have inputs of different kinds, recognized and processed differently, e.g. images and movies stored as folders of frames. Give each type its settings under inputs.types:

qaboard.yaml

inputs:
  types:
    default: image
    image:
      globs: '*.raw'
    movie:
      globs: frame_000.jpg
      use_parent_folder: true
      database:                # the settings of inputs can change per type
        linux: /mnt/movies
        windows: F://movies

Choose the type of a batch with type, or on the command line with qa --type:

qa/batches.yaml

my-images:
  inputs:
  - my/image.raw

my-movies:
  type: movie
  inputs:
  - folder/of/movies

qa batch my-images                       # images, the default type
qa --type movie run --input folder/of/movies/A

run() gets the type in context.type:

qa/main.py

def run(context):
    if context.type == "image":
        ...
    elif context.type == "movie":
        ...

A type can also have its own runner settings, e.g. another LSF queue for movies. They apply after those of qaboard.yaml, and before the batch's: see runner settings.

qaboard.yaml

inputs:
  types:
    movie:
      runners:
        lsf:
          queue: <your-long-queue>