Run QA-Board in your CI

Run your tests on every commit you push: each commit then has results, and the web app compares them with the reference, the latest results of your reference branch. qa reads the commit, the branch and the CI job from the CI's environment variables, so the job only installs qaboard and runs qa batch.

Before you start

• A QA-Board server, and its address for QABOARD_URL (see Installation). The quickstart's server only answers on the computer it runs on: for your team's CI, see Starting a QA-Board server.
• A batch of tests to run on every commit, like ci (see Batches of inputs).
• A CI runner that can write in the results storage, at the same path as the server (like /mnt/qaboard, an NFS share), the path of storage in qaboard.yaml: a self-hosted GitHub Actions runner, a GitLab runner or a Jenkins agent on your network. GitHub-hosted runners can't save results yet: see the roadmap.

Run your batch on every push

1. Add a CI job that installs qaboard and runs your batch:

   GitHub Actions

   .github/workflows/qaboard.yml

   # Runs your test set on every push, and saves the results to QA-Board.
   # qa saves results where the server reads them (e.g. /mnt/qaboard, an NFS share), at the same
   # path: use a self-hosted runner that mounts it. GitHub-hosted runners can't reach it yet.
   name: QA-Board
   on: push
   permissions:
     contents: read
     statuses: write  # for qa batch's commit status
   
   jobs:
     qaboard:
       runs-on: self-hosted
       env:
         QABOARD_URL: http://<your-qaboard>
         # qa batch sets a commit status that links to the results
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         # results sent with a token show who sent them (Settings > Access tokens: read and ingest scopes)
         QABOARD_TOKEN: ${{ secrets.QABOARD_TOKEN }}
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with:
             python-version: "3.12"
         - run: pip install qaboard -r requirements.txt
         # qa reads the commit and the branch from GITHUB_SHA and GITHUB_REF_NAME
         - run: qa batch ci

   Not run in our CI: adapt it to your setup.

   GitLab CI

   .gitlab-ci.yml

   # Runs your test set on every push, and saves the results to QA-Board.
   # qa saves results where the server reads them (e.g. /mnt/qaboard, an NFS share), at the same
   # path: the runner must mount it. With the docker executor, add it to the runner's volumes.
   qaboard:
     stage: test
     image: python:3.12
     variables:
       QABOARD_URL: http://<your-qaboard>
       # QABOARD_TOKEN: a masked CI/CD variable, with a token from Settings > Access tokens (read and ingest scopes)
     script:
       - pip install qaboard -r requirements.txt
       # qa reads the commit and the branch from CI_COMMIT_SHA and CI_COMMIT_REF_NAME
       - qa batch ci

   Not run in our CI: adapt it to your setup.

   Jenkins

   Jenkinsfile

   // Runs your test set on every push, and saves the results to QA-Board.
   // qa saves results where the server reads them (e.g. /mnt/qaboard, an NFS share), at the same
   // path: run it on agents that mount it.
   pipeline {
     agent { label 'qaboard' }
     environment {
       QABOARD_URL = 'http://<your-qaboard>'
     }
     stages {
       stage('QA-Board') {
         steps {
           sh '''
             python3 -m venv .venv
             . .venv/bin/activate
             pip install qaboard -r requirements.txt
             # qa reads the commit and the branch from GIT_COMMIT and GIT_BRANCH (Jenkins' git plugin)
             qa batch ci
           '''
         }
       }
     }
   }

   Not run in our CI: adapt it to your setup.

   <your-qaboard> is your server's address, like localhost:5151. If your CI already builds your code, run qa batch in the same job, after the build.

2. Push a commit. When the job ends, the commit is in your project's commit list, with its CI batch's results. The first push also adds the project to the home page.

3. Run the job on your reference branch too, main for instance: every other commit is compared with its latest results (see References and milestones).

Each commit shows how its CI batch went: the main metrics' medians, and the runs that crashed, in red.

On the commit page, the CI batch is shown as CI: it's the batch with the label default. Other batches the CI runs with qa --label nightly batch ... are listed next to it, like the runs people share from their computer.

The commit's batches: the CI batch, runs someone shared from their computer (🏠) and a tuning experiment, each with how many of its runs succeeded.

Link commits to their results

When qa has a token for your git host, qa batch sets a commit status named QA on the commit, with a link to its results and the number of runs. It's failed when runs failed. Batches with another label than default get their own status, like QA | nightly, and in a subproject the name is QA <folder>, with the name of the subproject's folder.

CI	Token
GitHub Actions	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} in the job's env, and statuses: write in its permissions. GitHub Actions doesn't put its token in the environment by itself.
GitLab CI	A GITLAB_ACCESS_TOKEN CI/CD variable (masked), a project access token with the api scope.
Other CIs	The token of the git host in QABOARD_GIT_HOSTS, else GITLAB_ACCESS_TOKEN or GITHUB_ACCESS_TOKEN (see other CI tools).

Without a token, qa batch works the same and sets no status.

Send verified results

Give your CI jobs a token, so that the results they send show who sent them, and so that they keep working once your server requires tokens to send results, or requires signing in (see Results sent by qa):

1. In the web app, open your project's Access tab, create a service account (e.g. ci, as a developer), and create a token for it with the read and ingest scopes. Runs then say they were sent by group/repo/ci[bot], and the token keeps working when you leave the team (service accounts). You need to be an owner of the project: else, ask its owners, or create a token as yourself in Settings → Access tokens, restricted to your project. Long-lived tokens are fine for CI, stored as protected secrets.
2. Store it in your CI's secrets as QABOARD_TOKEN: qa sends it. In GitHub Actions, add QABOARD_TOKEN: ${{ secrets.QABOARD_TOKEN }} to the job's env. In GitLab CI, add a masked, protected CI/CD variable named QABOARD_TOKEN.

For CI jobs that send results to many projects, admins create service accounts for the whole server, in Admin → Service accounts, that projects' owners add as members. Tokens tied to the CI itself (no stored secret) are coming next.

GitHub Actions

qa reads the commit from GITHUB_SHA, and the branch from GITHUB_HEAD_REF or GITHUB_REF_NAME. For tags, it records the tag. It calls the API at GITHUB_API_URL (else derived from GITHUB_SERVER_URL) for the repository GITHUB_REPOSITORY, so it works with GitHub Enterprise too. The token is GITHUB_TOKEN, else GITHUB_ACCESS_TOKEN, or the host's token in QABOARD_GIT_HOSTS.

Pull requests. Results belong to commits, so on: push is enough: pull requests from the repository's own branches get a run for each commit pushed to them. With on: [push, pull_request], those would run twice. If you add pull_request (for forks, for instance), check out the pull request's last commit: GitHub runs on a temporary merge commit, and qa records the pull request's last commit, whose details (message, author, parents) it reads from git.

.github/workflows/qaboard.yml

# Runs your test set on pushes and on pull requests. For pull requests, qa records the pull request's
# last commit, not the temporary merge commit GitHub runs on: check that commit out, so that qa can read
# its message, author and parents from git.
# Pull requests from the repository's own branches then run twice: `on: push` is usually enough.
name: QA-Board
on: [push, pull_request]
permissions:
  contents: read
  statuses: write  # for qa batch's commit status

jobs:
  qaboard:
    runs-on: self-hosted
    env:
      QABOARD_URL: http://<your-qaboard>
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha || github.sha }}
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install qaboard -r requirements.txt
      - run: qa batch ci

Not run in our CI: adapt it to your setup.

Waiting for the reference. qa check-bit-accuracy waits for the end of the reference commit's CI: give the job checks: read too.

GitLab CI

qa reads the commit and the branch from CI_COMMIT_SHA and CI_COMMIT_REF_NAME, and records tags from CI_COMMIT_TAG. For commit statuses, give it a GITLAB_ACCESS_TOKEN with the api scope (read_api is enough to wait for the CI of reference commits), as a CI/CD variable or in your site's shared secrets. It calls the API of GITLAB_HOST if it's set, else of CI_SERVER_URL.

The GitLab runner must mount the results storage: with the docker executor, add it to the runner's volumes.

Jenkins and other CI tools

qa knows that it runs in a CI when the CI environment variable is set (GitLab CI, GitHub Actions, CircleCI and Travis CI set it), or GIT_COMMIT (set by Jenkins' git plugin). It reads the commit and the branch from:

CI	Commit	Branch	Tag
Jenkins	GIT_COMMIT	GIT_BRANCH, or gitlabBranch (GitLab plugin)	GIT_TAG_NAME
CircleCI	CIRCLE_SHA1	CIRCLE_BRANCH	CIRCLE_TAG
Travis CI	TRAVIS_COMMIT	TRAVIS_BRANCH	TRAVIS_TAG

With another CI, set CI=true and GIT_COMMIT, GIT_BRANCH in the job.

Outside of GitHub Actions and GitLab CI, qa finds your git host from project.url in qaboard.yaml and the hosts listed in QABOARD_GIT_HOSTS (see Git hosts), and uses the host's token there, or GITLAB_ACCESS_TOKEN, or GITHUB_ACCESS_TOKEN. It only sends GITHUB_ACCESS_TOKEN to github.com and the GitHub hosts listed in QABOARD_GITHUB_HOSTS or QABOARD_GIT_HOSTS. Projects whose project.url is on an unknown host are assumed to be on GITLAB_HOST (if the hostname contains "github", qa warns instead and sends no token).

Fail the CI on regressions

qa batch fails when runs fail. To also fail when results get worse, add a step after it:

• qa gate (experimental) fails when metrics regress against the reference, with rules in qaboard.yaml.
• qa check-bit-accuracy fails when output files differ from the reference's.

In GitHub Actions, qa gate writes its table to the job summary: which rule failed, with a link to the Metrics Diff.

Save the build for runs started from the web app

To let people tune and run more tests from the web app, the CI saves what runs need, like your build, with qa save-artifacts after it builds (see Artifacts):

make
qa save-artifacts
qa batch ci

Without a CI

Runs started from your computer with --share are saved like CI runs, and show in the web app in a batch named after you, with a warning that they may not match the committed code:

qa --share batch ci

If you don't have a CI yet, see GitHub Actions or GitLab CI.

Run some tests only on some branches, from a Python script

This logic is usually better in your CI's configuration (rules, if: conditions). If your CI can't express it, qaboard.ci_helpers runs the functions you decorate only on the branches they name:

ci.py

import sys
from qaboard.ci_helpers import on_branch, run_tests

@on_branch('develop')
def my_tests():
  return 0 # 0 for success

# Also supported:
# @on_branch(["develop", "master"])
# @on_branch("feature/*")

if __name__ == '__main__':
    # run_tests() returns True if all the tasks succeeded
    sys.exit(0 if run_tests() else 1)

python ci.py