Metrics

Your run() function returns the numbers that say how good a run is: PSNR, mAP, p99 latency, a run time. QA-Board shows them in summaries, tables, run cards and plots over time, compared with the reference's. This page shows how to return them, then how to tell QA-Board which way is better, so that it colors improvements and regressions.

Return metrics

run() returns a dict:

qa/main.py

def run(context):
    # --snip--
    return {
        "psnr": psnr,
        "overshoot": overshoot,
        "runtime": runtime_s,
        "is_failed": False,
    }

That's all QA-Board needs. qa run prints the metrics, qa sends them to QA-Board with the run (with --share, or in CI), and the web app shows every metric with a number. qa adds compute_time, the run's duration in seconds, and is_failed says whether the run failed.

Code in another language or process

It can write a metrics.json file in context.output_dir: qa merges it with what run() returns (run() wins for the same key). It can declare the metrics in a metrics_definitions.json.

What QA-Board shows by default

Until you declare them, QA-Board doesn't know whether higher or lower values of a metric are better, so it shows them in neutral colors:

• the Summary shows up to 8 metrics: those whose median changed the most vs the reference (without reference runs, the first ones). Their changes are gray, neither green nor red. compute_time is left out: pick it if you need it;
• the Metrics Diff shades cells in blue: the darker, the larger the change;
• the Metrics Table shades cells in blue: the darker, the larger the value compared with the other runs;
• both tables show the declared metrics, and others up to 20 columns: pick more above them;
• each run card shows the run's first 6 metrics;
• the History page lets you plot any metric of its commits.

Hovering a metric's name says when QA-Board doesn't know which way is better, and the Summary lists those metrics with a link to this page. How people read the metrics: Reading metrics.

Declare your metrics

Declaring a metric tells QA-Board how to show it: its label and unit, its target, and whether smaller is better. Wrap the values you return in a Metric:

qa/main.py

from qaboard import Metric

def run(context):
    # --snip--
    return {
        "psnr": Metric(psnr, label="PSNR", suffix=" dB", smaller_is_better=False, target=30),
        "overshoot": Metric(overshoot, label="Edge overshoot (halos)", short_label="Overshoot",
                            suffix="%", scale=100, smaller_is_better=True, target=0.05),
        "runtime": Metric(runtime_s, label="Runtime", suffix=" s", smaller_is_better=True),
        "is_failed": False,
    }

The web app now shows improvements in green and regressions in red, values with their units, and how far each run is from the target:

Declared metrics get colors: the int8 model's smaller size is green, its AP on small objects red, and the bars under the metrics show the share of runs that meet their target.

A Metric behaves like a float: your postprocess() function can still compute with it. The result of a computation is a plain float, without the definition: if postprocess() replaces a value, wrap it again. Declaring needs a qa newer than the latest release on PyPI (see Installation), and an up-to-date QA-Board server: older servers ignore the definitions.

All the fields are optional:

Field	What it does
smaller_is_better	True if smaller values are better (an error, a run time), False if larger values are (PSNR, accuracy). Without it, changes stay neutral, e.g. for a count of frames. In a metrics file, leaving it out means True: write null when neither is better.
target	The value to reach. Table cells are colored by how far they are from it, and the Summary shows the share of runs that meet it. Needs smaller_is_better.
target_passfail	With True, values that meet the target are green, the others red, without shades.
label	A readable name. Default: the metric's key.
short_label	For tables and small spaces. Default: the label.
description	Shown when hovering the metric's name.
suffix	Shown after the values, e.g. "%" or " ms".
scale	Values are multiplied by it when shown, e.g. 100 to show a fraction in percent.
precision	The number of significant digits shown. Default: 3.
plot_scale	"log" or "linear", for the axis of the plots. Default: log in the Summary's box plots, linear on the History page. The box plots are linear when some values are 0 or negative.

Quality gates use the definitions too: rules about regressions need to know which way is better.

Once, next to the code that computes them

The best place to declare a metric is where your code computes it, so that the metric and its definition change together. When a function computes the same metric in many places, or when a module computes many metrics, declare them once with define_metric, at the top of the module:

my_algo/quality.py

from qaboard import define_metric

define_metric("psnr", label="PSNR", suffix=" dB", smaller_is_better=False, target=30)
define_metric("ssim", label="SSIM", smaller_is_better=False)

def quality_metrics(output, reference):
    return {"psnr": psnr(output, reference), "ssim": ssim(output, reference)}

define_metric only counts when it runs in qa's process: the module must be imported, directly or not, by your qa/main.py. Calls in other processes, like a subprocess or multiprocessing workers, are not seen. If the module is also used without qa, keep the import optional:

try:
    from qaboard import define_metric
except ImportError:
    define_metric = lambda *args, **kwargs: None

From any language: metrics_definitions.json

If a program in another language computes the metrics, it can write a metrics_definitions.json file next to its metrics.json, in context.output_dir:

metrics_definitions.json

{
  "psnr": {"label": "PSNR", "suffix": " dB", "smaller_is_better": false, "target": 30},
  "latency": {"suffix": " ms", "smaller_is_better": true}
}

Editors can check the file and complete its fields with its JSON schema: https://samsung.github.io/qaboard/metrics_definitions.schema.json.

How QA-Board combines the definitions

At the end of each run, qa gathers the definitions of the metrics the run returned, field by field: from define_metric, then metrics_definitions.json, then Metric, the last one winning. Fields with a wrong type are left out, with a warning. qa sends the definitions to QA-Board with the run's metrics.

QA-Board keeps them with the commit: commit pages show the metrics as they were defined at the time. Lists of commits use the definitions from the latest runs on the reference branch, and the History page those of its latest commit. QA-Board writes a commit's definitions only when they're new or changed: give all the runs the same definitions, e.g. don't compute a different target for each input.

Definitions in a metrics file apply too: for a metric declared in both, the fields from your code win.

Choose the metrics shown by default

Without configuration, QA-Board picks the metrics it shows, see above. To choose them, list them in a metrics file, and give its path in qaboard.yaml. The metrics you list don't have to be declared:

qaboard.yaml

outputs:
  metrics: qa/metrics.yaml

qa/metrics.yaml

# the metric used to sort runs, shown in lists of commits and plotted over time on their page
default_metric: psnr
# shown in the Summary
summary_metrics: [psnr, ssim, overshoot]
# shown in the tables and on the run cards, and offered in the plot over time of the lists of commits
main_metrics: [psnr, overshoot, runtime]

# Optional, for the History page: the metrics in its summary and tables (default: main_metrics, else QA-Board picks),
# those you can plot over time (default: main_metrics, else all the metrics of its commits)
# dashboard_metrics: [psnr, runtime]
# dashboard_evolution_metrics: [psnr, ssim, runtime]
# also summarize the metrics per tag of the inputs, from the "tags" of their metadata
# dashboard_breakdown_by_tag: true

People pick other metrics in the Summary and the tables. Run cards always show the main_metrics. Lists of commits only show metrics over time with a default_metric or main_metrics.

The tables show the main metrics, one row per test: each cell is colored by how far it is from the metric's target, given in the header.

Declaring metrics in a metrics file

The metrics file can also declare metrics, under available_metrics, with the fields above:

qa/metrics.yaml

# yaml-language-server: $schema=https://samsung.github.io/qaboard/metrics.schema.json
available_metrics:
  psnr:
    label: PSNR
    suffix: " dB"
    smaller_is_better: false
    target: 30
  loss:
    label: Loss function
    description: Mean error on the validation set

It works with the definitions from your code: for a metric declared in both, the fields from your code win, in the web app, in quality gates and in automated tuning, whose configuration can also say which way is better (psnr: max). The camera ISP example's metrics file declares all its metrics.

caution

In metrics files, metrics that leave out smaller_is_better are "smaller is better", like loss above. Write smaller_is_better: null for metrics where neither is better. Until November 2026, null (or smaller_is_better: with nothing after it) also meant "smaller is better": if your metrics file has some, write true.

Moving metrics from a metrics file to your code

Long metrics files are hard to maintain: nobody owns the hundreds of metrics they list, and they're out of date as soon as the code changes. You don't have to move everything at once:

1. Keep the metrics file: it keeps working, and its definitions still apply.
2. Delete the definitions of metrics your runs don't return anymore: search your code for their names.
3. Declare new metrics in your code, and move the others when you change the code that computes them: once your code declares a metric, delete it from the metrics file.
4. Keep in the metrics file the lists of metrics to show by default (summary_metrics, main_metrics...). Once QA-Board's choices suit you, delete the file and outputs.metrics in qaboard.yaml.

A definition in a metrics file without smaller_is_better means "smaller is better". When you move it to your code, write smaller_is_better=True, else QA-Board shows the metric in neutral colors, and quality gate rules about its regressions fail.

Failed runs

Return "is_failed": True when a run didn't work: it crashed, or its output is unusable. The run's card says Failed: see the logs, and qa run exits with an error. Returning it is often simpler than raising an exception, and when you split your code between run() and postprocess(), qa doesn't call postprocess() after a failed run().

If run() raises an exception, or doesn't return a dict, qa marks the run failed. If it returns metrics without is_failed, qa adds "is_failed": False.

Text and tags

Metrics can also be strings: tables and run cards show them as tags. To customize a tag, return a dict:

qa/main.py

def run(context):
    # --snip--
    return {
        "loss": 1.345,          # a number
        "status": "unstable",   # a string, shown as a tag
        "rich_status": {        # a tag with an icon and a color, that links somewhere
            "text": "unstable",
            "href": "https://example.com",
            "icon": "warning-sign",   # a Blueprint icon: https://blueprintjs.com/docs/#icons
            "intent": "warning",      # or primary, success, danger
            # also: "minimal": True, "large": True, "style": {...} (CSS)
        },
    }

Parameters and badges next to the configuration

A run can add parameters to its configuration, under a params key. They show with the run's configuration, as key: value tags, and the filter box finds runs by them. Use it to make visible a few key settings buried in a configuration file, or to link to another tool, like a model registry or a training run, with badges: tags like the rich metrics above. The detector example returns the model it evaluated, and a link to its model card:

qa/main.py

    return {
        **metrics,
        'is_failed': False,
        'params': {
            'model': model_id,
            'badges': [{'text': f"Model card kd-{flags['model_version']}", 'icon': 'id-number',
                        'href': f"https://models.kitefin.example/detector/kd-{flags['model_version']}#{kind}-{size}"}],
        },
    }

From examples/detection-eval/qa/main.py

The run's model shows with its configuration tags, followed by its badge, a link to the model card.