Input metadata and external databases

Inputs often come with what you know about them: the scene, the lighting, the sensor, where a test chart's patches are. Give that to QA-Board as metadata, and you can filter batches by it, show labels instead of paths, preset regions of interest in the image viewer, and let your code decide what to measure. When your inputs aren't files, read them from your own database.

Give inputs metadata

Define a metadata(context) function in your project's entrypoint: it returns a dict for each input. The camera ISP reads a .metadata.yaml file next to each input, when there is one:

qa/main.py

def metadata(context):
    """Input metadata: ROI presets for the image viewer, and the regions charts are measured in."""
    path = context.input_path.with_suffix('.metadata.yaml')
    return yaml.safe_load(path.read_text()) if path.exists() else {}

From examples/image-quality/qa/main.py

context is the same as run(context) gets: context.input_path, context.rel_input_path, context.database, context.configurations, context.params, context.platform, context.batch... When qa batch filters inputs, the runs don't exist yet: context.output_dir isn't set, and context.input_metadata is empty.

Older projects: metadata(absolute_input_path, database, input_path)

metadata() used to get the input's absolute path, the database, and the input's path relative to the database. Entrypoints written that way still work: qa calls metadata(context) when your function takes a single argument.

The night street's gives the image viewer 3 regions of interest:

inputs/photos/night-street.metadata.yaml

# Regions of interest for the image viewer (pixels of the 1224x816 photo)
roi:
- x: 630
  y: 0
  w: 140
  h: 120
  label: Street lamp
- x: 0
  y: 380
  w: 260
  h: 220
  label: Shadows (noise)
- x: 840
  y: 400
  w: 320
  h: 120
  label: Parked cars

Source: examples/image-quality/inputs/photos/night-street.metadata.yaml

Your run() gets the metadata as context.input_metadata. The camera ISP's test charts say where to measure: the slanted edge's region for the MTF and the overshoot, the color patches for ΔE 2000, a flat region for the noise. So run() only computes the metrics that make sense for each input:

qa/main.py

    measure = context.input_metadata.get('measure', {})
    if 'edge' in measure:
        edge = measures.slanted_edge(output, measure['edge'])
        metrics.update(mtf50=edge['mtf50'], overshoot=edge['overshoot'])
        plots.edge_profile(output_dir / 'edge.plotly.json', edge)
        plots.mtf(output_dir / 'mtf.plotly.json', edge)

From examples/image-quality/qa/main.py

You can also set metadata in run(), with context.input_metadata = {...}.

What metadata does

Key	What QA-Board does with it
label	Shown instead of the input's path, on run cards and in tables.
roi	Regions of interest preset in the image viewer, see Visualizations.
tags	A list, like [low-light, motion]: the History page summarizes metrics per tag with dashboard_breakdown_by_tag: true in your metrics file.
id	Runs on different inputs are compared when they have the same metadata.id: for instance images from two sensors, taken in the same conditions.
anything else	Filters for batches (below), and shown in the run card's menu, under Metadata.

Keys that start with _ only serve to filter batches: run() doesn't get them, and they're not saved with the runs.

Filter batches with metadata

only keeps the inputs that match all its conditions, exclude leaves out those that match all of its own:

qa/batches.yaml

night-outdoor:
  inputs:
    - photos
  only:                     # all these conditions
    location: outdoor
    camera:                 # one of these values
      - k3
      - k3-v2
    scene: 'night*'         # wildcards
    stabilized: false       # booleans and numbers
    lux: '<10'              # also >, >=, =, ==, <=
    rel_input_path: 'photos/*'  # the path relative to the database (input_path: the absolute path)
  exclude:
    scene: night-rain

qa batch night-outdoor

Without inputs, only and exclude pick from all the inputs of the database. Text matches ignore case. On big databases, keep metadata() fast: qa batch calls it for every input it lists when the batch has only or exclude, and each run calls it again for its input.

Read inputs from your own database

Inputs are not always files. They can be rows of a test database, or the names of unit tests, listed by something like gtest_project --gtest_list_tests. Define an iter_inputs() function in your project's entrypoint, that yields the inputs matching a query:

qa/main.py

def iter_inputs(path, database, only, exclude, inputs_settings):
  """
  path: what the batch lists under `inputs` (None: all the inputs of the database)
  database: the database's path, from qaboard.yaml or the batch
  only, exclude: the batch's filters
  inputs_settings: qaboard.yaml's `inputs` settings (globs, use_parent_folder, your own keys...)
  """
  # For instance, query your database:
  #   SELECT test, metadata FROM tests WHERE path LIKE ...
  # Filtering with only/exclude here is optional: qa filters what you return again, on its metadata.
  for test in tests:
    yield {"input_path": test.path, "database": database, "metadata": test.metadata}

qa reads the input_path (relative to database), database and metadata keys.