Configurations

A configuration says how to run your code on an input: at ISO 3200, with the int8 model, with a debug build, with these parameters. qa gives run() a list of configurations, and your code decides what they mean. The web app shows them as tags on each run, and compares each run with the reference run of the same input and the closest configuration (see how runs are matched).

Give a configuration

On the command line, with --config (or -c), before the command:

qa --config low-light run --input photos/night-street.png
#=> context.configs == ['low-light']

In the camera ISP example, run() reads the configuration's name, which sets the ISO of the simulated camera:

qa/main.py

def run(context):
    start = time.time()
    flags = read_flags(context.params)
    config = next((c for c in context.configs if isinstance(c, str)), 'base')

From examples/image-quality/qa/main.py

To give several, repeat the option, or separate them with :. They stay in order:

qa --config int8 --config size-320 run --input slices/indoor
qa --config int8:size-320 run --input slices/indoor
#=> context.configs == ['int8', 'size-320']

A configuration can also be a mapping of parameters, in JSON:

qa --config base --config '{"denoise_strength": 2.0}' run --input photos/night-street.png
#=> context.configs == ['base', {'denoise_strength': 2.0}]
#=> context.params == {'denoise_strength': 2.0}

Most of the time, batches give the configurations: the camera ISP's ci batch runs each image at base and low-light. See configurations in batches. Without any, context.configs is [], or what inputs.configs says in qaboard.yaml.

Read them in run()

• context.configs is the list: names, paths, mappings... in the order they were given.
• context.params merges all the mappings of the list, the last ones winning. It's often all you need.

A common convention: names are files of parameters kept with your code, and mappings override some of their parameters:

qa/batches.yaml

my-batch:
  configs:
  - base                       # parameters from configs/base.yaml, in the repository
  - /mnt/tuning/delta.yaml     # a file anywhere
  - {threshold: 0.4}           # parameters given directly
  inputs:
  - images

qa/main.py

from pathlib import Path

import yaml


def run(context):
    parameters = {}
    for config in context.configs:
        if isinstance(config, str):  # a file: an absolute path, or a name in configs/
            path = Path(config) if Path(config).is_absolute() else Path("configs") / f"{config}.yaml"
            parameters.update(yaml.safe_load(path.read_text()))
        elif isinstance(config, dict):
            parameters.update(config)
    return my_code(context.input_path, context.output_dir, parameters)

The detector example's configurations name the model and its input size, int8 and size-320. The inference gateway's are mappings: {repeats: 3, duration_s: 4} says how long to measure.

Pass them to an executable

Write the parameters to a file in the output directory, and give its path to your executable:

qa/main.py

import json
import subprocess


def run(context):
    config_path = context.output_dir / "config.json"
    config_path.write_text(json.dumps(context.params))
    process = subprocess.run([
        "build/my-tool",
        "--input", str(context.input_path),
        "--output", str(context.output_dir),
        "--config", str(config_path),
    ])
    return {"is_failed": process.returncode != 0}

The file stays with the run's outputs, so you can see in the web app what the run used.

Tuning parameters

--tuning (JSON) and --tuning-filepath (a JSON file) add parameters on top of the configurations. Both can be repeated:

qa --config int8 --tuning '{"score_threshold": 0.2}' run --input slices/indoor
#=> context.params['score_threshold'] == 0.2

They are merged in context.params, and come last in context.configs. The web app shows them apart from the configurations, as tuning tags. A JSON string or list, instead of a mapping, is added to context.configs:

qa --config base --tuning '"low-light"' --tuning '{"key": "value"}' run --input my/test
#=> context.configs == ['base', 'low-light', {..., 'key': 'value'}]
#=> context.params['key'] == 'value'

This is what tuning from the web app and qa batch --tuning-search use to try parameter values.

Set environment variables

When context.params has an ENV mapping, qa sets those environment variables before the run. Write their values in quotes: environment variables are strings.

qa/batches.yaml

my-batch:
  configs:
  - ENV: {OMP_NUM_THREADS: "4"}
  inputs:
  - images