Add QA-Board to your project

qa init adds QA-Board to a project: a qaboard.yaml file that describes your project to qa, and a qa/ folder with the Python code that runs yours. This page creates them, says what to fill in first, and connects your git host.

Create the files

1. At the root of your git repository, run:

   qa init

   It creates these files, next to yours:

   your-repository/
   ├── qaboard.yaml       # your project: its name, where the inputs are, what to show, where results go
   └── qa/
       ├── main.py        # runs your code: its run() function is what you edit
       ├── batches.yaml   # batches: lists of inputs and configurations to run together
       └── metrics.yaml   # optional: how to show your metrics

   In a git repository, qa init fills in the project's name and URL from your git remote, and the reference branch from the remote's default branch. When it can't reach the remote (no network or credentials), the reference branch stays master, and it says so. Without a remote, it says Please edit qaboard.yaml with your project name and url: set them yourself, see below.

2. Check that it works. Until you edit it, qa/main.py prints what it gets, then runs the command you give after the input:

   qa run --input /etc/hostname 'echo "{input_path} => {output_dir}"'

   Outputs: <your-repository>/output/linux/etc-hostname
   Edit qa/main.py to run *your* code using the context
     .output_dir:  <your-repository>/output/linux/etc-hostname
     .input_path:  /etc/hostname
   [...]
   /etc/hostname => <your-repository>/output/linux/etc-hostname
   {'is_failed': False, 'compute_time': 0.017203330993652344}

   <your-repository> is where you ran qa init. The last line is what the run returned: its metrics.

   If qa warns that The storage path does not exist, it couldn't create the folder for shared results, storage in qaboard.yaml (/mnt/qaboard). Local runs like this one don't need it. To share results, set it to your server's results folder, or for the quickstart's server, export QA_STORAGE=/tmp/qaboard.

3. Commit qaboard.yaml and qa/ with your code: every commit runs with its own.

Next, make run() run your code: see Running your code.

What to fill in qaboard.yaml

qa init writes a qaboard.yaml full of comments. Here are the settings of the camera ISP example, a complete project in the repository:

qaboard.yaml

# The camera ISP of the K3 robots: a QA-Board demo project, see README.md.
project:
  name: kitefin/camera-isp
  url: https://git.kitefin.example/kitefin/camera-isp
  description: Image pipeline of the K3 camera, from raw sensor data to the displayed image, in float and in the fixed-point model of the camera chip.
  entrypoint: qa/main.py
  reference_branch: main

inputs:
  # relative to this folder: generated charts, downloaded photos, and videos made from them (clips/*.yaml)
  database:
    linux: inputs
    windows: inputs
  globs: ['*.png', '*.tiff', 'clips/*.yaml']
  batches: qa/batches.yaml

storage:
  linux: /mnt/qaboard

From examples/image-quality/qaboard.yaml

Setting	What it is	Read more
project.name	The project's id in QA-Board, often the git repository's path.
project.url	Where your git repository is hosted.	Connect your git host
project.entrypoint	The Python file with your run() function.	Running your code
project.reference_branch	The branch whose latest results every commit is compared with.	References
inputs	Where your inputs are, how to recognize them, and the files that list your batches.	Inputs, Batches
outputs	Your metrics file, and which output files the web app shows, and how.	Metrics, Visualizations
storage	Where shared results are saved: a folder that the QA-Board server mounts too. $QA_STORAGE wins over it.	Where is the data?
runners	Where qa batch runs: on this machine, or on a cluster.	Runners

To see the configuration qa ends up with, including site defaults, run:

qa get config

tip

Several projects in one repository each get their own qaboard.yaml, in their folder. See Monorepos.

Check qaboard.yaml and batch files in your editor

QA-Board publishes JSON schemas for qaboard.yaml, batch files, metrics files and the metrics_definitions.json files runs can write (see Declaring metrics). Editors that understand them complete the keys, show their docs on hover, and flag typos and wrong types.

The files qa init creates already declare their schema, on their first lines. For your other files, with VS Code's YAML extension, add to your settings:

.vscode/settings.json

{
  "yaml.schemas": {
    "https://samsung.github.io/qaboard/qaboard.schema.json": ["qaboard.yaml", "**/qaboard.yaml"],
    "https://samsung.github.io/qaboard/batches.schema.json": ["qa/batches.yaml", "**/*batches*.yaml"],
    "https://samsung.github.io/qaboard/metrics.schema.json": ["qa/metrics.yaml"]
  }
}

Or add a comment at the top of the file, which works in any editor that uses the YAML language server:

qa/batches.yaml

# yaml-language-server: $schema=https://samsung.github.io/qaboard/batches.schema.json

Without internet access, use your QA-Board server's copy, e.g. https://<your-qaboard>/docs/batches.schema.json, or the files installed with qa, in the folder this prints:

python -c "import qaboard, pathlib; print(pathlib.Path(qaboard.__file__).parent / 'schemas')"

qa accepts keys the schemas don't know: they become input settings your entrypoint's iter_inputs can read, or options for viewers. So treat what the schemas flag as warnings. The web app's Available Tests view checks batches with the same schema.

Connect your git host

QA-Board shows the commits that qa sends results for, with the message and author it reads from git. When it also knows your git host, you can:

• see your commits with their author's avatar, and find them by branch or tag,
• get links to your code,
• start and follow CI jobs from the web app: GitHub Actions workflows and GitLab CI manual jobs,
• wait for the reference commit's CI to end before comparing with it, e.g. in bit-accuracy checks,
• delete old results.

QA-Board works with GitHub (and GitHub Enterprise), GitLab, Gitea/Forgejo and Bitbucket Cloud:

1. Your QA-Board server must know your git host, and have a token if the repository is private: ask its admins, or see Git hosts.
2. Your repository sends its push events to QA-Board, with a webhook: see below.
3. In CI, give qa a token for your git host too: qa batch then sets a commit status that links to the results, on GitHub and GitLab, and qa can ask whether a commit's CI ended. See the CLI's tokens.

tip

project.url in qaboard.yaml (e.g. git@github.com:org/repo.git) tells qa and QA-Board where your repository is hosted, even before the first push event.

GitHub

1. In the repository's Settings > Webhooks, add a webhook:
   • Payload URL: https://<your-qaboard>/webhook/github
   • Content type: application/json
   • Secret: the server's webhook secret, if it has one (recommended)
   • Events: Just the push event, as GitHub calls it
2. GitHub sends a "ping" right away: in the webhook's Recent Deliveries, it got a 200 response.

GitLab

1. As a maintainer of the project, add a webhook in Settings > Webhooks:
   • URL: https://<your-qaboard>/webhook/gitlab
   • Secret token: the server's webhook secret, if it has one (recommended)
   • Trigger: push events and tag push events
2. Click Test > Push events: GitLab shows that the hook answered 200 OK.

Gitea, Forgejo and Bitbucket

Like GitHub, with the payload URL https://<your-qaboard>/webhook/gitea (or /webhook/forgejo) and the push event. For Bitbucket Cloud, use https://<your-qaboard>/webhook/bitbucket with the Repository push trigger.

If a delivery fails, its HTTP status says why: see webhooks.

Site defaults for every project

Organizations can give all their projects default settings, like where results are stored or which runner to use, with a base qaboard.yaml. Each project's qaboard.yaml is merged on top of it:

• Mappings are merged key by key, and a project can override any setting.
• Lists are replaced, unless they contain super, which stands for the site's list: batches: [super, qa/batches.yaml].
• Locations (storage, inputs.database) are replaced as a whole.
• qa init doesn't copy the sample project's values for settings the site defines: it shows the site's values as comments instead.
• qa get config prints the configuration a project ends up with.

qa finds the base configuration with the QABOARD_SITE_CONFIG environment variable, which site packages set (e.g. a qaboard-site-mysite package points to its own qaboard_site_mysite/qaboard.yaml). Set QABOARD_SITE_CONFIG="" to ignore it.

If you set QABOARD_SITE_CONFIG yourself, e.g. as a CI variable shared by Linux and Windows runners, it can depend on the platform like other locations:

QABOARD_SITE_CONFIG='{"linux": "/mnt/qaboard/site.yaml", "windows": "//server/qaboard/site.yaml"}'

Add a base configuration to your site package

Add a qaboard.yaml next to your site package's __init__.py, and add to its defaults:

"QABOARD_SITE_CONFIG": str(Path(__file__).with_name("qaboard.yaml")),

If projects need someone to create their storage folders, or to raise a quota, say how in QABOARD_STORAGE_HELP: qa prints it when it can't create a folder for results or artifacts.

"QABOARD_STORAGE_HELP": "To get storage for a new project, ask your IT team: https://helpdesk.example.com",