Skip to main content

How a project integrates QA-Board

This guide is for whoever adds QA-Board to a project: you wrap your code with qa, and the web app shows every commit's outputs and metrics next to a reference's. This page shows how the pieces fit, then the steps, in order. A coding agent can do these steps too: see Coding agents. Looking at results that someone else set up? Read the user guide.

QA-Board was born at Samsung in 2018, where it is the central quality platform for ~50 projects in production, with more than 40 million runs.

What you get​

For each commit, the web app compares your runs with the reference's: metrics first, then the files your code wrote, like images with synced zoom and their difference.

A new denoiser: PSNR goes up, and so does the edge overshoot. The images show why: halos around the window frame.

How it works​

  Where your code runs: your computer, a CI runner, a cluster
  ┌───────────────────────────────────────────────────────────┐
  │ qa batch ci                                               │
  │  └─ for each input and configuration of the batch:        │
  │       run(context) in qa/main.py ──► your code            │
  └─────┬───────────────────────────────────────────┬─────────┘
        │ output files                              │ metrics, status, commit
        ▼                                           ▼
  ┌────────────────────────────┐  reads  ┌─────────────────────────┐
  │ storage, e.g. /mnt/qaboard │ ◄────── │ QA-Board server         │
  │ the same path everywhere   │         │ database, API, web app  │
  └────────────────────────────┘         └────────────┬────────────┘
                                                      ▼
                                    people: each commit next to its reference
  • qa runs your code where you start it. It's a command line, in Python. For each run (one input, one configuration) it calls the run(context) function of your qa/main.py, which calls your code: a Python function, or any executable.
  • Output files go on a shared storage. Your code writes them in context.output_dir, a folder on a storage that the server and every machine running qa mount at the same path, by default /mnt/qaboard, often an NFS share. Uploading outputs over HTTP is on the roadmap.
  • Metrics go to the server. qa sends what run() returns, with the run's status and the commit it ran on, to the server's API.
  • The web app compares. It shows a commit's runs next to the reference's: by default, the latest results of your reference branch.
  • qaboard.yaml, at the root of your repository, tells qa how to run your code, where your inputs are, and how to show your metrics and files.

The words QA-Board uses (batch, run, configuration, milestone...) are explained in Key concepts.

The steps​

To try each step on your own code, follow the tutorial: it walks through them for your stack (GitHub or GitLab, Docker Compose or Kubernetes, Python or any executable). The times below are rough, for a project whose code already runs from a script.

  1. Install qa, about 5 minutes: pip install qaboard where your code's dependencies are, and set QABOARD_URL to your server. You get: qa --help works. See Installation. No server yet? The quickstart starts one on your computer; for your team, see Starting a server.

  2. Create qaboard.yaml, about 5 minutes: run qa init at the root of your repository, then check the project's name, its reference branch, and storage: the results folder your server reads. You get: qaboard.yaml and a qa/ folder with main.py, batches.yaml and metrics.yaml. See Project setup.

  3. Point to your inputs, about 10 minutes: set the folder they're in, inputs.database. You get: qa finds an input from its path in that folder. See Inputs.

  4. Wrap your code, from 30 minutes to a few hours: in run(context), call your code on context.input_path, with the settings in context.params, and write its files in context.output_dir. You get: qa run --input <an-input> runs your code and prints its log and metrics. See Running your code and Configurations.

  5. Return metrics, about 15 minutes: return a dict from run(), and declare each metric's label, unit, target, and whether smaller is better. You get: metrics in the Summary, the tables and History, with regressions in red. See Metrics.

  6. Show your files, about 15 minutes: list them under outputs.visualizations in qaboard.yaml. You get: images side by side with synced zoom, plots over the reference's, videos in sync. See Visualizations and Viewers and file formats.

  7. Run a batch, about 10 minutes: list inputs and configurations in qa/batches.yaml, then run qa --share batch <batch>. You get: a link to the commit page, with your runs. See Batches of inputs.

    qa --share batch prints the link to the results, then runs the code on each input and configuration.

  8. Pick a runner, from 15 minutes to an hour: by default, qa batch does the runs on the machine where you start it, one per CPU with the qaboard.yaml of qa init (the local runner). Other runners send them to a cluster: LSF, Dask, Celery or Jenkins. You get: a batch's runs on many machines at once.

  9. Run in CI, about 30 minutes: run qa batch ci on every push, on CI runners that mount the storage, and connect your git host. You get: every commit's results, compared with the reference branch, shown as CI. See CI integration.

When you need them​

  • Tuning from the web app: people run batches with other parameters from the web app, without a checkout. It needs your build's artifacts and a runner the server can use. See Tuning from the web app and Artifacts.
  • Automated tuning: qa optimize searches the parameters that give the best metrics, from the CLI or the web app. See Automated tuning with qa optimize, and, for the people who tune, Find the best parameters automatically.
  • A quality gate (experimental): qa gate fails the CI job when metrics regress against the reference. See Quality gate.
  • Integrations: links and buttons in the web app's Integrations menu, to start CI jobs, call webhooks or open your other tools. See Integrations.
  • Bit accuracy: check that a commit's output files didn't change at all. See Bit accuracy.
  • Storage cleanup: delete old results and artifacts automatically. See Deleting old data.

Where to go next​

  • Hands-onTutorial: your project

    Wrap your code, run it on your test set and in CI, and compare every commit with your reference branch.

  • The wordsKey concepts

    Projects, batches, runs, inputs, configurations, references and milestones, in plain words.

  • ExamplesUse cases

    Six demo projects, with their qaboard.yaml and what they look like in the web app.

  • Try itQuickstart

    A QA-Board server on your computer, with the demo projects.