Skip to main content

Upload results from any tool

Save the files a tool already produced as a run, with one command: a test report, a benchmark file, a profile, a coverage report. There is no run() function to write. The run shows in the web app like the runs of qa run, compared with the reference's, with the viewer each file needs.

Experimental

qa upload is new: its options may change. Tell us what your CI needs.

Use qa run when QA-Board starts your code on each input of a batch (Running your code), qaboard.start_run() from Python code that logs as it goes, and qa upload when the results are already files.

Upload a test report from a CI job​

Before you start, you need a QA-Board server that takes files: one that accepts uploads, or a storage shared with it. In a project that keeps a shared storage, qa upload saves the files there where it is mounted, and sends them to the server where it isn't, e.g. on GitHub-hosted runners. The server asks who sends files: a token in the job's QABOARD_TOKEN variable (create one), or a job that signs in as itself, once a service account trusts the repository.

  1. Write the report with your test runner, as you may already do:

    pytest --junitxml=junit.xml
  2. Upload it:

    pip install qaboard
    export QABOARD_URL=https://qaboard.example.com
    export QABOARD_TOKEN=<a token with the ingest scope> # or, on your machine: qa auth login
    qa upload junit.xml

    qa prints the run's address in the web app.

  3. Open it. The run's card shows the tests as a tree, against the reference run's: new failures, fixed, added and removed tests (test reports). Its metrics count the tests: tests, passed, failed, errors, skipped, flaky and test_time. Follow them on the History page, and let the quality gate fail the CI when tests disappear.

This works in any repository: it doesn't need a qaboard.yaml. In a CI job, qa reads the project (the repository's name) and the commit from the CI's environment. Elsewhere, run it from a project, or give --project group/repo.

On your computer, sign in with qa auth login. qa --dryrun upload ... says what would be sent, without a server: the run's project, commit, batch and name, the folder it is saved in, its files and its metrics.

What you can upload​

Any file. The web app opens each with the viewer its name picks (viewers and file formats). Three kinds show on the run's card without anything in qaboard.yaml, and two give the run its metrics:

FilesNamed likeOn the cardMetrics
Test reports (JUnit XML)junit.xml, *.junit.xml, *-junit.xml, TEST-*.xmlThe tests, against the reference'sThe counts of tests, and their time
Benchmark filesbenchmarks.json, *.benchmarks.json, *-bench.jsonEach benchmark, against the reference'sOne per benchmark
Profiles*.folded, *.speedscope.json, *.cpuprofileA differential flame graph
Anything elseIn Output Files; on the card if qaboard.yaml lists it
pytest --junitxml=junit.xml --benchmark-json=pytest.benchmarks.json
py-spy record --format raw -o cpu.folded -- python serve.py
qa upload --input codec junit.xml pytest.benchmarks.json cpu.folded build/coverage/
  • A file keeps its name in the run, and a folder its name and what is in it: build/coverage/index.html is saved as coverage/index.html.
  • --as renames the only file given, e.g. when a tool writes results.xml: qa upload results.xml --as junit.xml.
  • Add metrics with -m name=value, or from a JSON file with --metrics metrics.json. They win over those read from the files. --no-read reads none.
  • Several test reports add up in the metrics, like the report per class that Maven and Gradle write. The card shows the first four files of each kind; the others are in Output Files.
  • In a folder, hidden folders (.git, .venv) are left out, and so is output/, where qa keeps a copy of the runs it shares from this computer.
  • Some names are QA-Board's own in every run: metrics.json, run.json, manifest.outputs.json. Give metrics with --metrics, and save other files under another name with --as. A log.txt of yours is the run's log.
  • A test report under another name is saved as any file: qa tells you, and --as junit.xml fixes it.

Name the run​

A run is identified like any run: a project, a commit, a batch, an input and configurations. Runs with the same input and configurations are compared from a commit to the next, and with the reference.

OptionDefault
What was tested: the input--input codecIn a CI job: the job's name, then the file's name without its extension (unit-tests/junit), or the job's name alone with several paths. Elsewhere: the file's name without its extension. In a GitHub matrix, give each job its own, e.g. --input unit-tests-${{ matrix.os }}: jobs of a matrix have the same name
How: the configurationsqa -c O3 -c '{"threads": 4}' upload ...None
The batch, in the web appqa --label nightly upload ...default: the CI's results. Outside the CI, your own batch, like qa --share
The group of tests, on the run's card--batch unit-testsNone
The project and the commit--project group/repo, --commit <id>The qaboard.yaml's or the CI's repository; the git HEAD or the CI's commit
  • Name runs yourself for a history that lasts: the default names change when a job is renamed, or uploads another file. --input doesn't.
  • One run per file: qa upload --each 'build/test-results/*.xml' names each run after its file.
  • Uploading the same run again replaces it: the web app shows the latest, and remembers the earlier attempts.
  • --failed says that what made the files couldn't finish, e.g. the test runner crashed. Failing tests don't fail the run: the run measured them.

In your CI​

.github/workflows/tests.yml
jobs:
unit-tests:
runs-on: ubuntu-latest
permissions:
id-token: write # the job signs in to QA-Board as itself, without a stored secret
contents: read
env:
QABOARD_URL: https://qaboard.example.com
steps:
- uses: actions/checkout@v4
- run: pip install qaboard pytest
- run: pytest --junitxml=junit.xml
- run: qa upload --input unit-tests junit.xml
if: always() # also when tests failed

How jobs sign in, as themselves or with a token: CI integration.

From Python​

qa upload is qaboard.start_run() from the shell. In a script, do the same and more:

import qaboard as qb
from qaboard.junit import junit_metrics

with qb.start_run(input="codec", label="nightly", batch="unit-tests") as run:
run.log_artifact("junit.xml")
run.log_metrics(junit_metrics("junit.xml"))

What it doesn't do​

  • Start the runs again from the web app. QA-Board doesn't know how the files were made. For runs it can redo and tune, wrap your code with qa run.
  • Add files to a run that ended. Upload everything a run has at once, or make it two runs.
  • Convert files. The web app reads perf script output and pprof files once converted to folded stacks, and a Criterion directory once written as one file: see profiles and benchmark suites.

Where next​