Upload results from any tool
Save the files a tool already produced as a run, with one command: a test report, a benchmark file, a profile, a
coverage report. There is no run() function to write. The run shows in the web app like the runs of qa run,
compared with the reference's, with the viewer each file needs.
qa upload is new: its options may change. Tell us what your CI needs.
Use qa run when QA-Board starts your code on each input of a batch (Running your code),
qaboard.start_run() from Python code that logs as it goes, and qa upload when the results
are already files.
Upload a test report from a CI job
Before you start, you need a QA-Board server that takes files: one that accepts uploads, or
a storage shared with it. In a project that keeps a shared storage, qa upload saves
the files there where it is mounted, and sends them to the server where it isn't, e.g. on GitHub-hosted runners. The server asks who sends files: a token in the job's
QABOARD_TOKEN variable (create one), or a job that
signs in as itself, once a service account trusts the repository.
-
Write the report with your test runner, as you may already do:
pytest --junitxml=junit.xml -
Upload it:
pip install qaboardexport QABOARD_URL=https://qaboard.example.comexport QABOARD_TOKEN=<a token with the ingest scope> # or, on your machine: qa auth loginqa upload junit.xmlqaprints the run's address in the web app. -
Open it. The run's card shows the tests as a tree, against the reference run's: new failures, fixed, added and removed tests (test reports). Its metrics count the tests:
tests,passed,failed,errors,skipped,flakyandtest_time. Follow them on the History page, and let the quality gate fail the CI when tests disappear.
This works in any repository: it doesn't need a qaboard.yaml. In a CI job, qa reads the project (the
repository's name) and the commit from the CI's environment. Elsewhere, run it from a
project, or give --project group/repo.
On your computer, sign in with qa auth login. qa --dryrun upload ... says what would be sent, without a server:
the run's project, commit, batch and name, the folder it is saved in, its files and its metrics.
What you can upload
Any file. The web app opens each with the viewer its name picks (viewers and file formats). Three kinds show on the run's card without anything in qaboard.yaml, and two give the run its metrics:
| Files | Named like | On the card | Metrics |
|---|---|---|---|
| Test reports (JUnit XML) | junit.xml, *.junit.xml, *-junit.xml, TEST-*.xml | The tests, against the reference's | The counts of tests, and their time |
| Benchmark files | benchmarks.json, *.benchmarks.json, *-bench.json | Each benchmark, against the reference's | One per benchmark |
| Profiles | *.folded, *.speedscope.json, *.cpuprofile | A differential flame graph | |
| Anything else | In Output Files; on the card if qaboard.yaml lists it |
pytest --junitxml=junit.xml --benchmark-json=pytest.benchmarks.json
py-spy record --format raw -o cpu.folded -- python serve.py
qa upload --input codec junit.xml pytest.benchmarks.json cpu.folded build/coverage/
- A file keeps its name in the run, and a folder its name and what is in it: build/coverage/index.html is saved as coverage/index.html.
--asrenames the only file given, e.g. when a tool writes results.xml:qa upload results.xml --as junit.xml.- Add metrics with
-m name=value, or from a JSON file with--metrics metrics.json. They win over those read from the files.--no-readreads none. - Several test reports add up in the metrics, like the report per class that Maven and Gradle write. The card shows the first four files of each kind; the others are in Output Files.
- In a folder, hidden folders (.git, .venv) are left out, and so is output/, where
qakeeps a copy of the runs it shares from this computer. - Some names are QA-Board's own in every run: metrics.json, run.json, manifest.outputs.json. Give metrics
with
--metrics, and save other files under another name with--as. A log.txt of yours is the run's log. - A test report under another name is saved as any file:
qatells you, and--as junit.xmlfixes it.
Name the run
A run is identified like any run: a project, a commit, a batch, an input and configurations. Runs with the same input and configurations are compared from a commit to the next, and with the reference.
| Option | Default | |
|---|---|---|
| What was tested: the input | --input codec | In a CI job: the job's name, then the file's name without its extension (unit-tests/junit), or the job's name alone with several paths. Elsewhere: the file's name without its extension. In a GitHub matrix, give each job its own, e.g. --input unit-tests-${{ matrix.os }}: jobs of a matrix have the same name |
| How: the configurations | qa -c O3 -c '{"threads": 4}' upload ... | None |
| The batch, in the web app | qa --label nightly upload ... | default: the CI's results. Outside the CI, your own batch, like qa --share |
| The group of tests, on the run's card | --batch unit-tests | None |
| The project and the commit | --project group/repo, --commit <id> | The qaboard.yaml's or the CI's repository; the git HEAD or the CI's commit |
- Name runs yourself for a history that lasts: the default names change when a job is renamed, or uploads another
file.
--inputdoesn't. - One run per file:
qa upload --each 'build/test-results/*.xml'names each run after its file. - Uploading the same run again replaces it: the web app shows the latest, and remembers the earlier attempts.
--failedsays that what made the files couldn't finish, e.g. the test runner crashed. Failing tests don't fail the run: the run measured them.
In your CI
- GitHub Actions
- GitLab CI
jobs:
unit-tests:
runs-on: ubuntu-latest
permissions:
id-token: write # the job signs in to QA-Board as itself, without a stored secret
contents: read
env:
QABOARD_URL: https://qaboard.example.com
steps:
- uses: actions/checkout@v4
- run: pip install qaboard pytest
- run: pytest --junitxml=junit.xml
- run: qa upload --input unit-tests junit.xml
if: always() # also when tests failed
unit-tests:
variables:
QABOARD_URL: https://qaboard.example.com
id_tokens:
QABOARD_ID_TOKEN:
aud: https://qaboard.example.com
script:
- pip install qaboard pytest
# upload the report even when tests fail, and fail the job if it isn't sent (after_script can't)
- pytest --junitxml=junit.xml || status=$?
- qa upload --input unit-tests junit.xml
- exit ${status:-0}
How jobs sign in, as themselves or with a token: CI integration.
From Python
qa upload is qaboard.start_run() from the shell. In a script, do the same and more:
import qaboard as qb
from qaboard.junit import junit_metrics
with qb.start_run(input="codec", label="nightly", batch="unit-tests") as run:
run.log_artifact("junit.xml")
run.log_metrics(junit_metrics("junit.xml"))
What it doesn't do
- Start the runs again from the web app. QA-Board doesn't know how the files were made. For runs it can redo
and tune, wrap your code with
qa run. - Add files to a run that ended. Upload everything a run has at once, or make it two runs.
- Convert files. The web app reads
perf scriptoutput and pprof files once converted to folded stacks, and a Criterion directory once written as one file: see profiles and benchmark suites.
Where next
- Use caseTests and coverage over timeWhat a team sees once its test reports are runs.
- Without a shared storageHow files reach the server from any machine.
- The quality gateFail the CI when tests disappear or benchmarks slow down.