Skip to main content

Send results without a shared storage

Runs can send their files to the QA-Board server over HTTP, instead of saving them in a folder that the server mounts too. Use it where a shared storage is hard to get: CI runners hosted by GitHub or GitLab, a laptop outside the office network, a cluster without the shared volume.

Experimental

Sending results over HTTP is new. Tell us how it goes with your projects.

Set it up​

  1. Ask your server's administrators whether it accepts files. They turn it on with one setting. In a project, qa doctor says whether it does.

  2. Leave storage out of qaboard.yaml, or say it. qa init writes a storage: replace it.

    qaboard.yaml
    storage: upload

    For one machine only, like a CI job in a project that otherwise uses a shared storage, set export QA_STORAGE=upload.

  3. Sign in: files are always sent with a token. Run qa auth login, and in CI let the job sign in as itself or give it a token.

Then use qa as usual: qa --share run, qa batch in CI, qa upload and qaboard.start_run() all send their runs' files.

export QABOARD_URL=https://qaboard.example.com
qa --share run --input images/a.jpg
# ...
# Sent 12 files to QA-Board (3.4 MB)

What is sent, and when​

A shared run is saved like a local one, in the project's output/ folder on the machine that runs it. qa sends its files from there, and the web app shows the run like any other: same pages, same viewers, same links.

When
The run's log, and the series it logsWhile it runs, every few seconds
Its metricsWhile it runs for start_run(), at the end for qa run
Files matching outputs.live in qaboard.yamlWhile it runs, each time they change
All its other filesWhen it ends, before it shows as finished

To look at some files before a long run ends, like plots it rewrites, list them:

qaboard.yaml
outputs:
live:
- "plots/*.png"
- "*.html"
  • A run that fails or is interrupted still sends what it wrote, with its log.
  • A run that is done again replaces the files the server had.
  • Files the server can't take, e.g. larger than its limit (1 GB by default): qa names them, and the run still shows with its other files. In CI the command fails; elsewhere it prints the error and goes on. Save big outputs in a shared storage.
  • A server that doesn't accept files refuses the run when it starts: in CI, qa stops there.
  • When the server doesn't answer, qa tries again a few times, then says what is missing. The files stay in output/.

What still needs a shared storage​

The server only has what runs sent it. It can't start your code where it has no build and no inputs.

Without a shared storage
See, compare, export and delete runs; the quality gate; HistoryYes
qa save-artifactsYes: it sends the commit's artifacts to the server
Start runs and tuning from the web app, redo runsYes, on the server's machine or on Kubernetes, from the artifacts qa save-artifacts sent (artifacts). Not with the LSF or Celery runners yet
qa check-bit-accuracyYes: it compares the reference's manifests (its files' hashes), fetched from the server. Not with QA_BITACCURACY_CMP, which compares files
qa optimizeYes: it reads the iterations' metrics from the server. Its journal, to resume, stays on the machine that runs it
qa batch with a cluster runnerWhen the cluster's nodes share the project's folder with the machine that starts the batch
Runs done againIn place: the server keeps the latest attempt's files

A project can mix both: most machines save in the shared storage, and the ones that can't mount it set QA_STORAGE=upload.

How it works
  • qa creates the run on the server, which chooses its folder under the server's own uploads folder.
  • Each file is a PUT /api/v1/output/<id>/files/<path>, with its SHA-256: the server checks what it received before the file shows. The log is sent by what was added. See the HTTP API.
  • The run's files are all there before the run ends: from then on they don't change.
  • A commit's artifacts are saved the same way, next to its runs, with PUT /api/v1/commit/<commit>/artifacts/files/<path>: see the HTTP API.
  • Runs send the state of their folder, not events: after a failed request, the next one sends what is missing.

Where next​