Skip to main content

Deployments, rollbacks and high availability

With docker compose, deploy and upgrade QA-Board with deployments/deploy.py. It gives:

  • Zero-downtime deploys: the backend runs several replicas behind nginx. New replicas start next to the old ones; nginx switches to them once they are healthy, then the old ones answer the requests they already received and stop. If the new replicas are not healthy, they are removed and the old version keeps serving.
  • The tested images are the deployed images: if your CI builds the images once per commit and tags them with the commit (e.g. <registry>/qaboard:backend-1a2b3c4d), you deploy those same images to staging, then production.
  • Fast rollbacks: the images of the last 3 versions stay on the host.
  • Migrations run once, before the backend is updated: if they fail, nothing running is changed.
  • The database, redis, etc. are only restarted if their configuration changed. nginx configuration changes are reloaded without a restart.

It has no dependencies: it runs with python3 (or uv run), from the root of your checkout.

Environments​

An environment is a docker compose env file that sets the compose files, the compose project and a few settings. Start from deployments/production.env, and add your site overlay (see Starting a server) to COMPOSE_FILE:

deployments/mysite/production.env
COMPOSE_PROJECT_NAME=qaboard
COMPOSE_FILE=docker-compose.yml:production.yml:deployments/mysite/mysite.yml
QABOARD_BACKEND_REPLICAS=2

The env file replaces the long -f docker-compose.yml -f production.yml -f ... lists:

docker compose --env-file deployments/mysite/production.env ps
# or, with the deployed version's images:
deployments/deploy.py deployments/mysite/production.env compose logs -f backend

Keep secrets out of the env files: put them in .env on the host, which the backend reads. deploy.py gives docker compose your env file instead of .env, so the settings docker compose reads itself, and those that docker-compose.yml sets from them (QABOARD_LOGIN_*, QABOARD_ADMINS, QABOARD_INGEST_AUTH), go in the env file or the overlay: see where each setting goes. The passwords the compose files use are the exception: deploy.py passes them from .env (QABOARD_DB_PASSWORD, RabbitMQ's, Flower's and pgAdmin's, see Passwords and admin services).

Deploying​

# A release's images (ghcr.io/samsung/qaboard:<service>-1.2.0), from a checkout of v1.2.0.
# The first time, --init allows deploying into a new compose project (with an empty database)
deployments/deploy.py deployments/mysite/production.env up --version 1.2.0 --init
# Images built by your CI and pushed to a registry, tagged with a version, e.g. the commit
deployments/deploy.py deployments/mysite/production.env up --version 1a2b3c4d
# Or build the images from your checkout
deployments/deploy.py deployments/mysite/production.env up --build

up:

  1. Pulls the images for the version (or builds them with --build).
  2. Runs the database migrations once. If they fail, nothing running is changed, and the deploy stops: the output says which migration failed and why. Fix the cause (e.g. a lock timeout: deploy again when the database is quieter), then deploy again.
  3. Updates the infrastructure services (database, redis, proxy...). They are only recreated if their configuration or image changed.
  4. Updates the backend, one version at a time: it starts the new replicas next to the old ones, waits until they are healthy, points nginx at them, then stops the old ones gracefully. If the new replicas are not healthy, they are removed and the old ones keep serving.
  5. Updates the web app and docs bundles. Old bundles are kept, so clients with an old version keep working.
  6. Records the version for rollback and status, and removes old images so the disk doesn't fill up.

If a deploy fails, its output says which step. The images are named ghcr.io/samsung/qaboard:<service>-<version> by default: to use another registry (a mirror, a fork's ghcr.io/<owner>/qaboard, or where your CI pushes), set QABOARD_IMAGE in the env file. The compose files and nginx's configuration come from your checkout, the code from the images: check out the version you deploy.

Deploying from CI

Build and push the images in CI, tagged with the commit, then run deploy.py <env file> up --version <commit> on the host (e.g. over ssh). Deploy to a staging environment first, test it, then deploy the same images to production.

Upgrading​

  1. Read the release notes of the versions since yours, "Before you upgrade" first: settings to check and data to move before the database migrations run.
  2. Check that the latest daily backup worked (see Backups).
  3. Check out the new release, and deploy it: 
    git fetch --tags && git checkout v1.2.1
    deployments/deploy.py deployments/mysite/production.env up --version 1.2.1
  4. Check it worked: status shows the new version, all the containers running and the backends healthy, and the web app opens. If not, roll back.

Rollbacks​

deployments/deploy.py deployments/mysite/production.env status     # deployed version, history
deployments/deploy.py deployments/mysite/production.env rollback   # back to the previous version
# rollback --no-pull if the registry is down
Database migrations

Rolling back the code does not roll back database migrations. Write migrations that the previous version can live with (add columns, don't drop/rename them in the same release), since the old and the new backends also run side by side during a deploy.

Since mid-October 2026, a version that finds migrations it doesn't know (made by a newer version) keeps them and starts. Older versions refuse to start instead: a rollback to them stops before changing anything, with Can't locate revision. To go back to such a version, first undo the newer migrations with the running version, down to the latest migration the older version knows (alembic heads in its image; 3b7e9c1d4a52 for the versions just before git hosts became configurable):

deployments/deploy.py deployments/mysite/production.env compose run --rm --no-deps backend \
  bash -c 'cd /qaboard/backend/backend && alembic downgrade 3b7e9c1d4a52'
deployments/deploy.py deployments/mysite/production.env rollback

Staging with recent production data​

A staging environment runs the production images and configuration, with its own database. To fill it with production's recent data:

deployments/deploy.py deployments/mysite/staging.env seed-db             # last 30 days
deployments/deploy.py deployments/mysite/staging.env seed-db --days 90

A production database can be big, so seed-db doesn't restore a full backup. It copies the schema and the small tables (projects, users...) in full, but only the recent commits (authored, or with outputs, in the last QABOARD_SEED_DAYS days) with their batches and outputs, and the test inputs those outputs use. Deleted commits and outputs are skipped. It reads production in a single read-only snapshot, with indexed queries, so only the recent rows are transferred. It replaces staging's database: if production can't be read, staging is left as it was. Older commits referenced by milestones are not copied.

The source is QABOARD_SEED_SOURCE in the staging env file (or --source). For a full copy, restore a backup instead: restore-db latest.

caution

If staging uses the same storage and task runners as production, runs started from staging are real runs.

Backups​

The cron-backup-db service in production.yml makes daily backups of the database to /backups. Mount a host directory there in your site overlay, in the db service too (restore-db reads the backups from the database container), and copy the backups off the host:

deployments/mysite/mysite.yml
services:
  cron-backup-db:
    volumes:
    - /srv/qaboard/backups:/backups
  db:
    volumes:
    - /srv/qaboard/backups:/backups

After each successful daily backup (YYYY-MM-DD.dump), the daily backups older than QABOARD_BACKUP_RETENTION_DAYS days (default 14, 0 keeps them all) are deleted, but never the newest 3. Other dumps, like before-upgrade.dump, are never deleted: remove them yourself. Before upgrading from an older version, move the old daily backups you want to keep to another name or folder.

To restore a backup (the backend is stopped meanwhile):

deployments/deploy.py deployments/mysite/production.env restore-db 2026-10-01.dump   # or "latest"

Database settings​

PostgreSQL starts with its defaults, sized for a small machine. To tune it, use services/db/init.sh as the db service's entrypoint, with a file of settings next to it, or set QABOARD_DB_CONFIG to your own file. services/db/qaboard.conf is an example, sized for an 8-core, 32 GB host shared with other services, with the reason for each value:

deployments/mysite/mysite.yml
services:
  db:
    # docker-compose.yml mounts services/db/ at /opt
    entrypoint: /opt/init.sh
    environment:
    - QABOARD_DB_CONFIG=/opt/mysite.conf  # default: /opt/qaboard.conf

At every start, init.sh copies the file into the database's folder and includes it at the end of postgresql.conf. For a new database, it does it right after creating it, and creates the pg_stat_statements extension (for an existing database, run CREATE EXTENSION IF NOT EXISTS pg_stat_statements; once). QABOARD_DB_CONFIG=none removes the include, to go back to the previous settings.

  • Settings like shared_buffers or max_connections need a restart of PostgreSQL: docker compose restart db, about a minute of errors. A deploy doesn't restart the database when only the file changed. Run CHECKPOINT; first, so that PostgreSQL stops faster.
  • Values set with ALTER SYSTEM (in postgresql.auto.conf) win over the file. Check where each value comes from with SELECT name, setting, sourcefile FROM pg_settings WHERE source NOT IN ('default', 'override');, and remove them with ALTER SYSTEM RESET ALL; (applied at the next restart).
  • max_connections must cover the backends' connection pools: each uwsgi process opens up to QABOARD_DB_POOL_SIZE + QABOARD_DB_MAX_OVERFLOW connections (default 4 + 2), times QABOARD_UWSGI_PROCESSES, times QABOARD_BACKEND_REPLICAS, and twice that during a deploy, when the old and new backends run side by side. Add your other clients (backups, dashboards, admins).

Database password​

The database is created with QABOARD_DB_PASSWORD from .env (default: password), and the backend and the backups connect with it. It's only used when the database is created: to change the password of an existing database,

  1. Change it in PostgreSQL: deployments/deploy.py <env file> compose exec db psql -U qaboard -c "ALTER USER qaboard PASSWORD '<new password>'" (connections from inside the container don't need the password).
  2. Set QABOARD_DB_PASSWORD=<new password> in .env, and deploy: deployments/deploy.py <env file> up. The backend replicas connect with the new password; the backups too.
  3. Update the other clients: dashboards (e.g. Metabase), QABOARD_SEED_SOURCE of a staging environment, services/pgadmin/passfile.

Host maintenance​

Before upgrading the host (CPU, memory, storage, OS...):

  1. Pause your automatic deploys, if you have some.
  2. Check that the latest daily backup worked: list your backups directory.
  3. Stop the server and make a fresh backup: 
    # disconnect clients so that no one writes
    docker compose --env-file deployments/mysite/production.env stop
    # the backup needs the database
    docker compose --env-file deployments/mysite/production.env up -d db
    # with deploy.py: it passes the database password from .env
    deployments/deploy.py deployments/mysite/production.env compose run --rm cron-backup-db /etc/periodic/daily/backup before-upgrade.dump
  4. After the maintenance, start everything: 
    deployments/deploy.py deployments/mysite/production.env up
    deployments/deploy.py deployments/mysite/production.env status
    Check that you can open the web application, that status shows all the containers running and the backends healthy, and that the logs look all right (see Troubleshooting). Then resume the automatic deploys.
  5. If something went wrong with the data, restore the backup: restore-db before-upgrade.dump.

High availability: what you get, what you don't​

FailureWith docker compose (1 host)With kubernetes
A deploy✅ no downtime (nginx itself is only restarted if its image changes: ~1s)✅
A bad release✅ unhealthy replicas never get traffic, then rollback✅ (helm --atomic)
A backend crash / OOM✅ the other replicas serve, docker restarts it✅
redis down✅ slower, no errors✅
The database container restarts❌ errors for a few seconds✅ with CloudNativePG (failover)
The host goes down / reboots❌ down until it's back✅ pods move to other nodes
The shared storage (NFS) is down❌❌

Going further on a single host doesn't buy much: the host is the single point of failure. A warm standby (a second host with a PostgreSQL streaming replica, and a manual failover) is possible but costs about as much effort as moving to kubernetes, which gives automatic failover.

First-time setup of a host​

Starting a QA-Board server walks through the first deploy. In short:

  1. Docker Engine with docker compose ≥ 2.24 (docker compose version). If your CI builds the images on another host, it needs it too.
  2. Clone the repository, and check out the version you deploy.
  3. Fill the secrets in .env, and the SSL certificates if you use HTTPS (see Starting a server).
  4. If you pull images from a private registry, docker login to it once, as the user who deploys.
  5. When you move an existing deployment to deploy.py, check that the compose project name matches the existing one: docker compose ls. It must be COMPOSE_PROJECT_NAME in the env file, or compose would start with new, empty volumes. deploy.py refuses to deploy into a project that doesn't exist, unless you pass --init.

Settings​

In the env file, or the shell's environment (which takes precedence):

VariableDefaultUsage
QABOARD_VERSIONthe deployed versionImage version (tag suffix), e.g. the commit. up --version sets it.
QABOARD_IMAGEghcr.io/samsung/qaboardWhere the images are pulled from: a mirror, a fork's ghcr.io/<owner>/qaboard, your CI's registry
QABOARD_PORT_HTTP5151The host's port of the web app (nginx)
QABOARD_BACKEND_REPLICAS1Backend replicas. 2+ for zero-downtime deploys.
QABOARD_UWSGI_PROCESSES, QABOARD_UWSGI_CHEAPER, QABOARD_UWSGI_CHEAPER_INITIAL16, 8, 16uwsgi processes per replica (max, min, initial)
QABOARD_DEPLOY_PRE_HOOKnoneA command run before deploying, e.g. to mount network volumes
QABOARD_RUN_MIGRATIONS1Set to 0 to not run migrations when the backend starts
QABOARD_SEED_SOURCEnoneseed-db: the database to copy recent data from, postgresql://user:password@host:port/db
QABOARD_SEED_DAYS30seed-db: keep the commits of the last N days
QABOARD_BACKUP_RETENTION_DAYS14Daily backups older than this are deleted, except the newest 3. 0 keeps them all
QABOARD_DB_CONFIGqaboard.conf next to init.shWith services/db/init.sh as the db's entrypoint: the PostgreSQL settings file, none to remove it (see Database settings)