Run batches on Celery workers

With the celery runner, qa batch sends its runs to Celery workers through a message broker, and waits for them. Workers run each qa run in the folder where you started qa batch, with your environment variables (except those whose names contain TOKEN, ACCESS, AUTH or SECRET): their machines need that folder, your inputs and the results storage at the same paths, e.g. on shared drives.

Start workers and send them runs

1. Get a broker. QA-Board's Docker Compose deployment starts RabbitMQ on port 5672 (not the quickstart's), only reachable from the server's host by default: for workers or qa on other hosts, the server's admins publish it and give you an account (see sharing the broker). Anyone who can connect to the broker can run commands on the workers. To start another one:

   docker run --detach -p 5672:5672 rabbitmq

2. Start at least one worker, in the Python environment of your code:

   pip install celery qaboard
   export CELERY_BROKER_URL=pyamqp://guest:guest@<broker-host>:5672//
   celery -A qaboard.runners.celery_app worker --concurrency=10 --loglevel=info

   <broker-host> is the broker's hostname, e.g. your QA-Board server's. Run workers as services, so that they restart after failures and reboots: see Celery's daemonization guide.

3. Make qa batch use Celery, in qaboard.yaml:

   qaboard.yaml

   runners:
     default: celery
     celery:
       broker_url: pyamqp://guest:guest@<broker-host>:5672//  # or $CELERY_BROKER_URL

   Without broker_url or CELERY_BROKER_URL, qa looks for the broker on a host named qaboard.

4. Run a batch: qa --share batch ci. To see the runs' logs live in your terminal instead, run them on your machine: qa batch --runner local ci.

Without --share (or a CI), runs write their files in output/, in the folder where you started qa batch: use --share so that they go to the results storage that the server reads.

Share the server's broker between teams

Workers run the commands of the tasks they get, so anyone who can send tasks to a worker can run commands on its host. When several teams use the QA-Board server's RabbitMQ, give each team its own virtual host and user: its workers only get its tasks, and other accounts can't send them any. The server's admins create them:

team=camera
password=$(openssl rand -hex 16)
docker compose exec rabbitmq rabbitmqctl add_vhost $team
docker compose exec rabbitmq rabbitmqctl add_user $team $password
docker compose exec rabbitmq rabbitmqctl set_permissions -p $team $team ".*" ".*" ".*"
# the team's broker URL
echo "pyamqp://$team:$password@<qaboard-host>:5672/$team"

qaboard.yaml is in git: keep the password in the environment, where you run qa and where the workers run. Settings ending with _url (and result_backend) expand environment variables:

qaboard.yaml

runners:
  celery:
    broker_url: pyamqp://camera:${QA_CELERY_PASSWORD}@qaboard:5672/camera

With qa versions from before October 2026, leave broker_url out of qaboard.yaml and set the whole URL in CELERY_BROKER_URL instead. To see who still connects with another account: docker compose exec rabbitmq rabbitmqctl list_connections user vhost peer_host.

Monitor the workers

QA-Board's Docker Compose deployment includes Flower, for QA-Board's admins at http://<your-qaboard>/flower/, where <your-qaboard> is your server's address: it can revoke tasks and shut the workers down. It shows the default virtual host only. See also Celery's monitoring guide.

Celery settings

Settings under runners.celery are given to Celery:

qaboard.yaml

runners:
  default: celery
  celery:
    # assuming your QA-Board server's hostname is qaboard
    broker_url: pyamqp://guest:guest@qaboard:5672//  # also read from ENV vars with CELERY_BROKER_URL
    result_backend: rpc://                  # also read from ENV vars with CELERY_RESULT_BACKEND
    # Any of Celery's settings: https://docs.celeryq.dev/en/stable/userguide/configuration.html
    timezone: Europe/Paris

    # By default tasks will be named "qaboard" unless you define
    qaboard_task_name: qaboard

To send the runs to specific workers, e.g. if you manage different types of resources (GPUs, Windows/Linux/Android...), start the workers with their queues, and route the runs there:

# On a server with a GPU:
celery -A qaboard.runners.celery_app worker --concurrency=1 --queues gpu,large-gpu

qaboard.yaml

runners:
  celery:
    task_routes:
      qaboard: gpu

Batches can't have their own Celery settings yet: qa batch ignores a batch's celery: settings, and uses runners.celery from qaboard.yaml for all its runs.

Celery's workers guide explains how to set their concurrency, queues and limits.