Integrating QA-Board with your CI

CI tools run automated scripts and tests everytime someone pushes a new commit.

tip

If you don't have a CI, try GitHub Actions or GitLab CI.

This said, you can still view your results in the web application by using qa --share run/batch. Note: It will only work with commits that were pushed to your git host!

Requirement

• Make sure your repository sends its push events to QA-Board. If you're not sure if/how, review the setup guide. You should be able to see your project in the QA-Board web application.

Projects show up on the home page once they have results.

Running QA-Board in your CI

1. Have your CI launch QA-Board, e.g. qa batch ci, assuming you defined a batch named ci (examples below).

note

You CI is responsible for setting up an environment ($PATH...) in which qaboard is installed! Consider using docker, or sourcing a configuration file...

2. Push a commit. If your CI is successful, the commit will appear in your project's page:

Each commit's row shows how its CI batch went: the main metric's median, a crashed batch in red, the other batches it ran.

3. If qa has a token for your git host, qa batch also sets a commit status named QA (QA <subproject> in a subproject, with the name of its folder), with a link to the results. Batches run with a --label other than default get their own status, e.g. QA | my-label.

Example with GitHub Actions

.github/workflows/qa.yml

# Every commit pushed to the repository, on any branch or tag
on: push
permissions:
  contents: read
  statuses: write  # for qa batch's commit status
  checks: read     # for bit-accuracy checks that wait for the reference commit's CI
jobs:
  qa:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
      with:
        # For pull_request events: the pull request's last commit, not the temporary merge commit
        ref: ${{ github.event.pull_request.head.sha || github.sha }}
    - run: pip install qaboard
    - run: qa batch ci
      env:
        QABOARD_URL: https://qaboard.example.com
        # GitHub Actions doesn't put its token in the environment: map it
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Results in QA-Board belong to commits, so running on push is enough: pull requests from the repository's own branches already get a run for each of their commits. With on: [push, pull_request], those would run twice. If you add pull_request (e.g. for forks), keep the ref: above: qa records the pull request's last commit, but by default actions/checkout only fetches the temporary merge commit, so qa couldn't read the commit's details (message, author, parents) from git.

qa reads the commit from GITHUB_SHA, and the branch from GITHUB_HEAD_REF or GITHUB_REF_NAME. For pull requests, it uses the pull request's last commit, not the merge commit. For tags, it records the tag. It calls the API at GITHUB_API_URL (else derived from GITHUB_SERVER_URL) for the repository GITHUB_REPOSITORY, so it works with GitHub Enterprise too. The token is GITHUB_TOKEN, else GITHUB_ACCESS_TOKEN, or the host's token in QABOARD_GIT_HOSTS.

Example with GitLab CI

.gitlab-ci.yml

stages:
  - build
  - qa

build-linux:
  stage: build
  script:
  - make
  - qa save-artifacts

qa-tests:
  stage: qa
  script:
  - qa batch ci

qa reads the commit and branch from CI_COMMIT_SHA and CI_COMMIT_REF_NAME. For commit statuses, give it a GITLAB_ACCESS_TOKEN with the api scope (read_api is enough to wait for the CI of reference commits), e.g. a CI/CD variable or the site's shared secrets. It calls the API of GITLAB_HOST if it's set, else of CI_SERVER_URL.

Other CI tools

QA-Board knows how to work with the most common CI tools: GitHub Actions, GitLab CI, Jenkins, CircleCI, TravisCI...

Elsewhere, qa finds your git host from project.url in qaboard.yaml and the hosts listed in QABOARD_GIT_HOSTS (see Git hosts), and uses the host's token there, or GITLAB_ACCESS_TOKEN, or GITHUB_ACCESS_TOKEN. It only sends GITHUB_ACCESS_TOKEN to github.com and the GitHub hosts listed in QABOARD_GITHUB_HOSTS or QABOARD_GIT_HOSTS. Projects whose project.url is on an unknown host are assumed to be on GITLAB_HOST (if the hostname contains "github", qa warns instead and sends no token).

note

qa detects that it runs in a CI when the CI environment variable is set (by GitlabCI, GitHub Actions, CircleCI, TravisCI...), or GIT_COMMIT (by Jenkins' git plugin).

Failing the CI on regressions

qa batch fails when runs fail. To also fail when metrics regress against a reference, run qa gate after it. It's experimental, see Quality gate.

In GitHub Actions, qa gate writes its table to the job summary: which rule failed, with a link to the Metrics Diff.

Optionnal CI helpers

QA-Board is not a CI tool, but it provide some utilities to run code only in some branches:

warning

This logic is usually better expressed in your CI tool itself. But if you're stuck with stone-edge tooling sometimes you roll your own.

# ci.py
from qaboard.ci_helpers import on_branch, run_tests

@on_branch('develop')
def my_tests():
  pass

# Also supported:
# @on_branch(["develop", "master"])
# @on_branch("feature/*")

if __name__ == '__main__':
    run_tests()

python ci.py