Skip to main content

Choose where runs execute

qa batch hands its runs to a runner: your computer, an HPC cluster, Kubernetes, a pool of workers. Pick the runner that matches the compute you already have, set it in qaboard.yaml, and every batch goes there. Whatever the runner, the runs are the same qa run commands, and the web app shows their results the same way.

Pick a runner​

Where you have computeRunnerEach run isRead
Your computer, or one big machinelocala process, several at a timeYour machine
An LSF clusterlsfan LSF job, in job arrays with a limit on how many run at onceLSF
A Slurm, PBS, SGE, HTCondor, OAR or Moab clusterdaska task on a few worker jobs that qa batch starts on the clusterDask on HPC schedulers
Many short runs on an HPC clusterdaskthe same: workers start once, and the runs don't wait for the schedulerDask on HPC schedulers
A Kubernetes clusterkubernetesa pod of a Kubernetes Job, or several runs one after the other in each podKubernetes
A Kubernetes cluster, with many short runsdaska task on Dask workers that run as podsDask on Kubernetes
Machines that are always on (GPU boxes, lab machines)celerya task that a Celery worker picks upCelery
Windows machines, through Jenkinswindowsa build of a Jenkins jobWindows with Jenkins

Already running batches with MLflow, W&B Launch, ClearML, Slurm job arrays or Kubernetes Jobs? See Coming from another tool: each one, mapped to qaboard.yaml.

What every runner needs
  • The same files at the same paths. The machines that do the runs read your code in the folder where you start qa batch, your inputs, and write to the results storage, at the same paths as the machine where you start it, e.g. on an NFS share. With Kubernetes, you mount them in the pods.
  • The same environment. They run qa run with your code's dependencies: the same Python environment on a shared drive, or an image that has them.
  • Your environment variables. Runners forward them to the runs. Celery and Kubernetes leave out secrets (names with TOKEN, PASSWORD, SECRET...), Dask and Kubernetes those specific to a host, like CUDA_VISIBLE_DEVICES. Kubernetes also leaves out PATH, HOME and the paths the pods don't mount: see the runs' environment.

Some runners need optional packages, where you run qa: pip install "qaboard[kubernetes]", "qaboard[dask]" ("qaboard[dask-kubernetes]" for Dask workers on Kubernetes) or "qaboard[celery]".

Set the runner​

Set the default runner in qaboard.yaml, with its settings:

qaboard.yaml
runners:
default: local
local:
concurrency: -1 # one run per CPU

See Run batches on your machine.

<your-queue>, <your-namespace>, <your-image>, <your-claim> and <your-qaboard-host> are your scheduler's queue, your Kubernetes namespace, an image with qa and your code's dependencies, the PersistentVolumeClaim of your shared storage, and your QA-Board server's hostname.

Then check the setup, and run a batch:

qa doctor --test-job
qa --share batch ci

qa doctor checks qaboard.yaml, the server, the storage and the runner, and says how to fix what doesn't work. To use another runner for one command: qa batch --runner local ci.

How qa picks the runner, and settings per batch
  • --runner wins, then runners.default. Without runners.default, qa batch uses the first runner configured under runners other than local, else local.
  • Every runner setting can also be set per input type, per batch and on the command line: see runner settings per batch.
  • A batch can't choose its runner yet: runner: in a batch is ignored.

Compare the runners​

What changes from one runner to the other: whether qa batch --no-wait returns once the runs are sent, and what the web app can do with them.

Runner--no-waitStop runs from the web appStart and redo runs from the web app
localNo: qa batch waits for the runsYes when the server started them, on the server itself. Started elsewhere: marks them as failed, the runs go onYes, where the server runs qa batch (its admins choose, e.g. on the server itself, for small batches)
lsfYesYes, with bkillYes, as the user who starts them
daskYes: a driver job keeps the runs goingYes, except with cluster.type: local. Slurm, PBS, SGE, HTCondor and OAR need their commands on the server; an existing cluster, its scheduler in the server's settingsYes
kubernetesYesYes: deletes the batch's JobsYes, as the user who starts them
celeryYesYes: revokes the tasks, when they were sent to the server's broker, with Celery 5.3 or laterYes
windowsYesNoRedo isn't supported

Stopping and starting runs from the web app needs settings on the server: your QA-Board admins follow Run batches from the web app.

Logs of each runner

Every run has its log.txt, written by qa run. Runners add what they see, in the run's output directory:

RunnerExtra logWhat it has
lsflog.lsf.txtEverything the job printed, then LSF's report: exit code, memory, why LSF killed it
dasklog.dask.txtEverything the run printed; worker logs are in the batch's folder
kuberneteslog.k8s.txt, log.pending.txtEverything the pod printed. While the pod waits, why, e.g. no node with a free GPU

The web app's Logs view shows them, next to log.txt.

Where to go next​