Run batches on Celery workers
With the celery runner, qa batch sends its runs to Celery workers through a
message broker, and waits for them. Workers run each qa run in the folder where you started qa batch, with your
environment variables, except those that look like secrets (see secrets): their machines need that
folder, your inputs and the results storage at the same paths, e.g. on shared drives.
The runs get the worker's environment, with the variables of qa batch on top: set what runs need that qa batch
doesn't forward, like secrets or CUDA_VISIBLE_DEVICES, where the workers start. To send results to a QA-Board server
that requires a token, either set QABOARD_TOKEN for the workers, or sign in once with qa auth login in a home
folder that the workers' machines share: the runs get your HOME, and with it your login.
Start workers and send them runs
-
Get a broker. QA-Board's Docker Compose deployment starts RabbitMQ on port
5672(not the quickstart's), only reachable from the server's host by default: for workers orqaon other hosts, the server's admins publish it and give you an account (see sharing the broker). Anyone who can connect to the broker can run commands on the workers. To start another one:docker run --detach -p 5672:5672 rabbitmq -
Start at least one worker, in the Python environment of your code:
pip install "qaboard[celery]"export CELERY_BROKER_URL=pyamqp://guest:guest@<broker-host>:5672//celery -A qaboard.runners.celery_app worker --concurrency=10 --loglevel=info<broker-host>is the broker's hostname, e.g. your QA-Board server's. Run workers as services, so that they restart after failures and reboots: see Celery's daemonization guide. -
Make
qa batchuse Celery, in qaboard.yaml:qaboard.yamlrunners:default: celerycelery:broker_url: pyamqp://guest:guest@<broker-host>:5672// # or $CELERY_BROKER_URLWithout
broker_urlorCELERY_BROKER_URL,qalooks for the broker on a host namedqaboard. Install the runner where you runqa batchtoo:pip install "qaboard[celery]". -
Check it:
qa doctor --runner celery --test-jobchecks the broker, that workers answer and listen to the runs' queue, then has a worker run a task. See qa doctor. -
Run a batch:
qa --share batch ci. To see the runs' logs live in your terminal instead, run them on your machine:qa batch --runner local ci.
Without --share (or a CI), runs write their files in output/, in the folder where you started qa batch: use
--share so that they go to the results storage that the server reads.
Stop a batch
Press Ctrl+C in qa batch, or click Stop runs in the web app: the workers drop the batch's tasks that wait, and
stop those that run, with everything their qa run started. The web app only stops batches sent to the server's
broker (its CELERY_BROKER_URL): batches sent to another broker, or another virtual host, go on.
What workers need, and the limits
- Celery 5.3 or later, and the default
preforkpool: with thethreadsorsolopools, workers can't stop running tasks, they finish (and when such a worker gets SIGTERM, its warm shutdown waits for them too). Older workers only stop batches fromCtrl+C, not from the web app. - Workers remember stopped batches in memory. A worker that starts afterwards may still run their waiting tasks,
unless the workers save their state:
celery -A qaboard.runners.celery_app worker --statedb=/var/lib/celery/worker.state.
Give the runs secrets
qa batch doesn't send the variables that look like secrets to the workers: set them where the workers start, the
runs get them. To send some anyway, e.g. a key that each user has, name them in qaboard.yaml:
runners:
celery:
forward_secrets: [WANDB_API_KEY, HF_TOKEN]
They travel in the tasks' messages: anyone who can read the broker's queues can read them, see sharing the broker.
The variables qa batch doesn't forward
The same as the Kubernetes runner:
- Names that contain
TOKEN,ACCESS,AUTH,SECRET,PASSWORD,PASSWD,CREDENTIAL,PRIVATE,COOKIE,JWTorAPIKEY, in any case. - Names with one of the words
KEY,PASS,PWorSESSION, between_or other characters that aren't letters or digits:OPENAI_API_KEY,DB_PASSorSESSION_ID, but notKEYBOARDorPASSAGE. - Values with credentials in a URL, anything between
://and@:https://user:password@host/...,https://gitlab-ci-token:xxx@gitlab.example.com/....
Share the server's broker between teams
Workers run the commands of the tasks they get, so anyone who can send tasks to a worker can run commands on its host. When several teams use the QA-Board server's RabbitMQ, give each team its own virtual host and user: its workers only get its tasks, and other accounts can't send them any. The server's admins create them:
team=camera
password=$(openssl rand -hex 16)
docker compose exec rabbitmq rabbitmqctl add_vhost $team
docker compose exec rabbitmq rabbitmqctl add_user $team $password
docker compose exec rabbitmq rabbitmqctl set_permissions -p $team $team ".*" ".*" ".*"
# the team's broker URL
echo "pyamqp://$team:$password@<qaboard-host>:5672/$team"
qaboard.yaml is in git: keep the password in the environment, where you run qa and where the workers run.
Settings ending with _url (and result_backend) expand environment variables:
runners:
celery:
broker_url: pyamqp://camera:${QA_CELERY_PASSWORD}@qaboard:5672/camera
With qa versions from before October 2026, leave broker_url out of qaboard.yaml and set the whole URL in
CELERY_BROKER_URL instead. To see who still connects with another account:
docker compose exec rabbitmq rabbitmqctl list_connections user vhost peer_host.
Monitor the workers
QA-Board's Docker Compose deployment includes Flower, for QA-Board's
admins at http://<your-qaboard>/flower/, where <your-qaboard> is your server's address: it can revoke tasks and
shut the workers down. It shows the default virtual host only. See also Celery's monitoring guide.
Celery settings
Settings under runners.celery are given to Celery, except QA-Board's own forward_secrets (secrets):
runners:
default: celery
celery:
# assuming your QA-Board server's hostname is qaboard
broker_url: pyamqp://guest:guest@qaboard:5672// # also read from ENV vars with CELERY_BROKER_URL
result_backend: rpc:// # also read from ENV vars with CELERY_RESULT_BACKEND
# Any of Celery's settings: https://docs.celeryq.dev/en/stable/userguide/configuration.html
timezone: Europe/Paris
# By default tasks will be named "qaboard" unless you define
qaboard_task_name: qaboard
To send the runs to specific workers, e.g. if you manage different types of resources (GPUs, Windows/Linux/Android...), start the workers with their queues, and route the runs there:
# On a server with a GPU:
celery -A qaboard.runners.celery_app worker --concurrency=1 --queues gpu,large-gpu
runners:
celery:
task_routes:
qaboard: gpu
Batches can't have their own Celery settings yet: qa batch ignores a batch's celery: settings, and uses
runners.celery from qaboard.yaml for all its runs.
Celery's workers guide explains how to set their concurrency, queues and limits.