Skip to main content

Troubleshooting common issues

The examples use docker compose and the env file of your environment, here deployments/production.env: it tells docker compose which compose files and project to use (see Deployments). Run them from the root of your checkout, on the QA-Board host.

Talking to the different services​

# status: deployed version, containers and their health
deployments/deploy.py deployments/production.env status

# read logs from a specific service
deployments/deploy.py deployments/production.env compose logs -f --tail 100 backend

# get a shell in a service
deployments/deploy.py deployments/production.env compose exec backend bash
# check the backend directly, without nginx
deployments/deploy.py deployments/production.env compose exec backend wget -qO- localhost:3001/api/v1/health

deploy.py ... compose is docker compose --env-file ... with the images of the deployed version. Plain docker compose --env-file deployments/production.env ... works for anything that doesn't (re)create containers.

tip

The backend can run several replicas. compose exec/logs pick one; add --index N for another. With old docker compose versions, exec can complain that "container #1" is not running after a deploy: use --index with the numbers from docker compose ps.

Questions to ask if things don't work​

  • Is the container even running? Is it restarting all the time? Is it unhealthy?
docker compose --env-file deployments/production.env ps -a
  • Is the disk full?
df -h
  • Did a deploy just happen? Roll back if needed (see Rollbacks).

How to restart the containers​

Symptoms:

  • the web application doesn't load;
  • 500 errors;
  • often needed after the disk got full.
docker compose --env-file deployments/production.env restart backend
# or everything (downtime!)
docker compose --env-file deployments/production.env restart

If your site needs something before the containers start (e.g. mounting network volumes), do it first: deploy.py runs your QABOARD_DEPLOY_PRE_HOOK, but plain docker compose doesn't.

How to recreate all the containers​

# Redeploys the current version: only what changed is recreated, the backend without downtime
deployments/deploy.py deployments/production.env up
# From scratch (downtime!). Never add --volumes/-v: it deletes the database
docker compose --env-file deployments/production.env down
deployments/deploy.py deployments/production.env up

Quick wins when the disk is full​

Symptoms:

  • 500 errors;
  • database unreachable in the logs;
  • no space left on device in the logs.

Remove the IIIF image cache:

docker compose --env-file deployments/production.env stop cantaloupe
docker compose --env-file deployments/production.env rm -v cantaloupe
deployments/deploy.py deployments/production.env compose up -d cantaloupe

Remove unused docker images (deploy.py already removes the images of old versions, it keeps the last 3):

docker image prune # -a

Re-build and start the containers​

To build the images on the host and deploy them:

deployments/deploy.py deployments/production.env up --build