Deployments, rollbacks and high availability

With docker compose, deploy and upgrade QA-Board with deployments/deploy.py. It gives:

• Zero-downtime deploys: the backend runs several replicas behind nginx. New replicas start next to the old ones; nginx switches to them once they are healthy, then the old ones finish their in-flight requests and stop. If the new replicas are not healthy, they are removed and the old version keeps serving.
• The tested images are the deployed images: if your CI builds the images once per commit and tags them with the commit (e.g. <registry>/qaboard:backend-1a2b3c4d), you deploy those same images to staging, then production.
• Fast rollbacks: the images of the last 3 versions stay on the host.
• Migrations run once, before the backend is updated: if they fail, nothing running is changed.
• The database, redis, etc. are only restarted if their configuration changed. nginx configuration changes are reloaded without a restart.

It has no dependencies: it runs with python3 (or uv run), from the root of your checkout.

Environments

An environment is a docker compose env file that sets the compose files, the compose project and a few settings. Start from deployments/production.env, and add your site overlay (see Starting a server) to COMPOSE_FILE:

deployments/mysite/production.env

COMPOSE_PROJECT_NAME=qaboard
COMPOSE_FILE=docker-compose.yml:production.yml:deployments/mysite/mysite.yml
QABOARD_BACKEND_REPLICAS=2

The env file replaces the long -f docker-compose.yml -f production.yml -f ... lists:

docker compose --env-file deployments/mysite/production.env ps
# or, with the deployed version's images:
deployments/deploy.py deployments/mysite/production.env compose logs -f backend

Keep secrets out of the env files: put them in .env on the host.

Deploying

# The first time: --init allows deploying into a new compose project (with an empty database)
deployments/deploy.py deployments/mysite/production.env up --build --init
# Upgrades: git pull, then
deployments/deploy.py deployments/mysite/production.env up --build
# With images built by your CI and pushed to a registry, tagged with a version
deployments/deploy.py deployments/mysite/production.env up --version 1a2b3c4d

up:

1. Pulls the images for the version (or builds them with --build).
2. Runs the database migrations once. If they fail, nothing running is changed.
3. Updates the infrastructure services (database, redis, proxy...). They are only recreated if their configuration or image changed.
4. Updates the backend, one version at a time: it starts the new replicas next to the old ones, waits until they are healthy, points nginx at them, then stops the old ones gracefully. If the new replicas are not healthy, they are removed and the old ones keep serving.
5. Updates the web app and docs bundles. Old bundles are kept, so clients with an old version keep working.
6. Records the version for rollback and status, and removes old images so the disk doesn't fill up.

If a deploy fails, its output says which step. The images are named arthurflam/qaboard:<service>-<version> by default: to use your own registry, set image: in your site overlay.

Deploying from CI

Build and push the images in CI, tagged with the commit, then run deploy.py <env file> up --version <commit> on the host (e.g. over ssh). Deploy to a staging environment first, test it, then deploy the same images to production.

Rollbacks

deployments/deploy.py deployments/mysite/production.env status     # deployed version, history
deployments/deploy.py deployments/mysite/production.env rollback   # back to the previous version
# rollback --no-pull if the registry is down

Database migrations

Rolling back the code does not roll back database migrations. Write migrations that the previous version can live with (add columns, don't drop/rename them in the same release), since the old and the new backends also run side by side during a deploy.

Since mid-October 2026, a version that finds migrations it doesn't know (made by a newer version) keeps them and starts. Older versions refuse to start instead: a rollback to them stops before changing anything, with Can't locate revision. To go back to such a version, first undo the newer migrations with the running version, down to the latest migration the older version knows (alembic heads in its image; 3b7e9c1d4a52 for the versions just before git hosts became configurable):

deployments/deploy.py deployments/mysite/production.env compose run --rm --no-deps backend \
  bash -c 'cd /qaboard/backend/backend && alembic downgrade 3b7e9c1d4a52'
deployments/deploy.py deployments/mysite/production.env rollback

Staging with recent production data

A staging environment runs the production images and configuration, with its own database. To fill it with production's recent data:

deployments/deploy.py deployments/mysite/staging.env seed-db             # last 30 days
deployments/deploy.py deployments/mysite/staging.env seed-db --days 90

A production database can be big, so seed-db doesn't restore a full backup. It copies the schema and the small tables (projects, users...) in full, but only the recent commits (authored, or with outputs, in the last QABOARD_SEED_DAYS days) with their batches and outputs, and the test inputs those outputs use. Deleted commits and outputs are skipped. It reads production in a single read-only snapshot, with indexed queries, so only the recent rows are transferred. It replaces staging's database: if production can't be read, staging is left as it was. Older commits referenced by milestones are not copied.

The source is QABOARD_SEED_SOURCE in the staging env file (or --source). For a full copy, restore a backup instead: restore-db latest.

caution

If staging uses the same storage and task runners as production, runs started from staging are real runs.

Backups

The cron-backup-db service in production.yml makes daily backups of the database to /backups. Mount a host directory there in your site overlay, in the db service too (restore-db reads the backups from the database container), and copy the backups off the host:

deployments/mysite/mysite.yml

services:
  cron-backup-db:
    volumes:
    - /srv/qaboard/backups:/backups
  db:
    volumes:
    - /srv/qaboard/backups:/backups

To restore a backup (the backend is stopped meanwhile):

deployments/deploy.py deployments/mysite/production.env restore-db 2026-10-01.dump   # or "latest"

Host maintenance

Before upgrading the host (CPU, memory, storage, OS...):

1. Pause your automatic deploys, if you have some.
2. Check that the latest daily backup worked: list your backups directory.
3. Stop the server and make a fresh backup:

   # disconnect clients so that no one writes
   docker compose --env-file deployments/mysite/production.env stop
   # the backup needs the database
   docker compose --env-file deployments/mysite/production.env up -d db
   docker compose --env-file deployments/mysite/production.env run --rm cron-backup-db /etc/periodic/daily/backup before-upgrade.dump

4. After the maintenance, start everything:

   deployments/deploy.py deployments/mysite/production.env up
   deployments/deploy.py deployments/mysite/production.env status

   Check that you can open the web application, that status shows all the containers running and the backends healthy, and that the logs look all right (see Troubleshooting). Then resume the automatic deploys.
5. If something went wrong with the data, restore the backup: restore-db before-upgrade.dump.

High availability: what you get, what you don't

Failure	With docker compose (1 host)	With kubernetes
A deploy	✅ no downtime (nginx itself is only restarted if its image changes: ~1s)	✅
A bad release	✅ unhealthy replicas never get traffic, then rollback	✅ (helm --atomic)
A backend crash / OOM	✅ the other replicas serve, docker restarts it	✅
redis down	✅ slower, no errors	✅
The database container restarts	❌ errors for a few seconds	✅ with CloudNativePG (failover)
The host goes down / reboots	❌ down until it's back	✅ pods move to other nodes
The shared storage (NFS) is down	❌	❌

Going further on a single host doesn't buy much: the host is the single point of failure. A warm standby (a second host with a PostgreSQL streaming replica, and a manual failover) is possible but costs about as much effort as moving to kubernetes, which gives automatic failover.

First-time setup of a host

1. Docker Engine with docker compose ≥ 2.24 (docker compose version). If your CI builds the images on another host, it needs it too.
2. Clone the repository.
3. Fill the secrets in .env, and the SSL certificates if you use HTTPS (see Starting a server).
4. If you pull images from a private registry, docker login to it once, as the user who deploys.
5. When you move an existing deployment to deploy.py, check that the compose project name matches the existing one: docker compose ls. It must be COMPOSE_PROJECT_NAME in the env file, or compose would start with new, empty volumes. deploy.py refuses to deploy into a project that doesn't exist, unless you pass --init.

Settings

In the env file, or the shell's environment (which takes precedence):

Variable	Default	Usage
QABOARD_VERSION	the deployed version	Image version (tag suffix), e.g. the commit. up --version sets it.
QABOARD_BACKEND_REPLICAS	1	Backend replicas. 2+ for zero-downtime deploys.
QABOARD_UWSGI_PROCESSES, QABOARD_UWSGI_CHEAPER, QABOARD_UWSGI_CHEAPER_INITIAL	16, 8, 16	uwsgi processes per replica (max, min, initial)
QABOARD_DEPLOY_PRE_HOOK	none	A command run before deploying, e.g. to mount network volumes
QABOARD_RUN_MIGRATIONS	1	Set to 0 to not run migrations when the backend starts
QABOARD_SEED_SOURCE	none	seed-db: the database to copy recent data from, postgresql://user:password@host:port/db
QABOARD_SEED_DAYS	30	seed-db: keep the commits of the last N days