Run batches from the web app
When users click Start in Run Tests / Tuning, Redo on a run, or Stop runs, the QA-Board server runs
qa for them. This page sets up where it does, so your users can run batches without a terminal, on your cluster,
as themselves.
By default, the server starts no batches: QABOARD_TUNING_RUNNER isn't set, so the web app and
qa batch --commit tell users to ask you, and that qa batch from CI or a terminal still works. Redo still goes
through the LSF bridge if you set one. Setting QABOARD_TUNING_RUNNER is the opt-in.


How the server starts a batch
For each request, the server writes a script in the batch's folder, in the results storage. The script goes to the
commit's artifacts, sources their .envrc files, and runs qa batch --no-wait, or
qa optimize for an automated tuning, for the user who asked. Two settings decide where things run:
QABOARD_TUNING_RUNNER, on the server: where thatqa batchcommand runs.- The project's runner,
runners.defaultin its qaboard.yaml: whereqa batchsends the runs, like from a terminal. See Choose where runs execute.
QABOARD_TUNING_RUNNER | qa batch runs | As | For |
|---|---|---|---|
| not set (default) | Nowhere: the server refuses to start batches, and redoes runs only through the LSF bridge, if you set one | Servers where only CI and terminals run batches | |
local | In the backend's container | The server's account | Trying QA-Board on your laptop, or a server whose every user you trust with it |
lsf | In an LSF job, sent through the LSF bridge | The user who started it | LSF sites |
kubernetes | In a pod, in the project's namespace | The user's UID and GID | Kubernetes sites |
celery | In the backend's container, with the broker's address | The server's account; the runs, as the workers' | Projects whose runner is celery |
Projects need their artifacts saved by the CI: see Let people tune from the web app for what projects do.
The form lists a batch's tests before it starts. For projects that list their inputs with their own iter_inputs,
that runs their code: the server does it only with local or celery, in its container without its secrets.
Otherwise the form says it can't list them, and users can still start the batch.


With local and celery, qa batch runs in the backend's container, as the server's account: everyone with the
developer role on a project, and anyone with a token that sends its artifacts (qa save-artifacts), can run code
there. QA-Board keeps the server's secrets out of its environment, but the code can read what the server's account
can. Use lsf or kubernetes when you don't trust every developer with the server: see Security.
Admin → Settings and the security checklist in Admin → Overview warn about it.
On the server itself: local
To try batches from the web app on your laptop, set it in .env (or the overlay's services.backend.environment,
or the Helm chart's config), then restart the backend:
# Batches started from the web app run in the backend's container, as its account: see the warning above
QABOARD_TUNING_RUNNER=local
qa batch runs in the backend's container, so the container needs what the projects' .envrc files
activate, e.g. their Python environments on a shared drive.
qa batch runs in the background, in its own process group: the server answers users once it started, and it runs
as long as it needs. If it fails within a few seconds (QABOARD_LOCAL_START_WAIT, 3 by default), users see why right
away; later, the web app shows it from its logs. Redoing runs works the same way, unless the server has an
LSF bridge.
- With the project's
localrunner, the runs execute in the backend's container too, and share its CPUs. Restarting the backend's container, e.g. to deploy, stops them. The web app warns users of projects without another runner. - With the project's
lsfrunner,qasends the runs through the LSF bridge if the server has nobsub. - With the project's
celeryrunner, the backend needs to reach the broker.
On LSF
The server can't run bsub from its container: it goes through a host that can, the bridge.
-
Get a bridge host that has LSF, where the server can log in with ssh, and run commands as your users, e.g. with a
sudorule for its account. -
Set the bridge, in the backend's environment. It's a command template: QA-Board replaces
{bsub_command}with the command to run (bsub,bwait,bkill, orbash <script>) and{user}with the user who started the runs:QABOARD_TUNING_RUNNER=lsfQA_RUNNERS_LSF_BRIDGE="ssh qaboard@<lsf-host> 'qaboard-as-user {user} {bsub_command}'"<lsf-host>is the bridge host, andqaboard-as-usera wrapper there that runs the rest of its arguments as{user}: see send jobs through a bridge host for one. Mount the ssh key in the backend's container. -
Try it: start a batch from the web app. The Logs view shows the
bsubcommand and its output.
What the server does with the bridge:
- Start:
qa batchruns in an LSF job, on the project'srunners.lsf.queue(runners.lsf.long_queuefor automated tuning), as the user. LSF's report is in the batch's logs. - Redo:
qa batchruns on the bridge host, as the user, then sends the runs with the runner they had. - Stop:
bkillon the batch's jobs, including the workers of Dask batches.
The commands QA-Board puts in {bsub_command} have quotes and here-documents: check that they reach the bridge
host unchanged, e.g. by running the template with bjobs -w in place of {bsub_command} from the backend's
container.
On Kubernetes
With QABOARD_TUNING_RUNNER=kubernetes, the server starts qa batch in a driver pod: a Job of one pod, named
qa-<first 8 characters of the command id>-driver-<random>. It runs in the namespace of the commit's
runners.kubernetes settings (else the first allowed namespace), with their image and mounts, as the signed-in
user's UID and GID, from the server's users' accounts. Its
qa batch --no-wait creates the runs' Jobs with the namespace's runs ServiceAccount, then ends.
With the Helm chart:
-
List the namespaces where batches run, in your values. They must exist:
my-values.yamlrunners:kubernetes:enabled: truenamespaces: [algo-qa, camera-qa]In each namespace, the chart creates the runner's Role (
<release>-qaboard-runs), the runs ServiceAccount, and a RoleBinding for it and for the backend's ServiceAccount. It setsQABOARD_TUNING_RUNNER=kubernetes,QABOARD_KUBERNETES_NAMESPACES,QABOARD_KUBERNETES_SERVICE_ACCOUNTandQABOARD_KUBERNETES_SERVER_URL(the proxy's Service). Installing it needs the permission to create Roles there. -
Know your users' UIDs. The server reads them from its users' accounts: mount
/etc/passwd, or setQABOARD_ACCOUNTS_COMMAND. Only users who sign in with the directory (LDAP, SAML, OIDC) can start batches, unless you setrunAsorQABOARD_KUBERNETES_TRUST_LOCAL_ACCOUNTS. -
Enforce who runs as whom in each namespace, with its Pod Security level and a ValidatingAdmissionPolicy on the UIDs: see one namespace per team.
-
Try it: start a batch from the web app. Its logs say where the driver pod runs, and why it waits if it's pending.
Projects need, in the qaboard.yaml of the commits they run:
runners.default: kubernetes, or Dask on Kubernetes: otherwise the driver pod does the runs itself.runners.kubernetes.image, withbashandqaboard[kubernetes], andmountsfor the results storage and the artifacts, at the same paths as the server's. See Run batches on Kubernetes.- Or no mounts, for projects without a shared storage: the pods download the commit's artifacts from QA-Board and send their results back with the batch's token, and the driver sends its log. That token only sends this batch's results (uploads).
Limits without a shared storage
- What
qa batchreads (the batches as they were defined, the runs to redo) goes in the driver's ConfigMap: at most 768 KiB. Larger batches are refused when they start: select fewer, or start them from CI. - The driver's log reaches the web app while the batch's token is valid: until 10 minutes after its last run ends.
Anything later, or refused by the server, is only in the pod's logs: the log's first line says how to read them
(
kubectl logs -n <namespace> job/<driver>), and a line says when some output is missing. qa optimizecan't resume an experiment started again: its journal stays in the driver pod.
Without the Helm chart
The backend talks to the cluster with its ServiceAccount when it runs in a pod, else with its kubeconfig. In each namespace where batches run:
-
Create the runner's Role, from Get the permissions.
-
Create a ServiceAccount for the driver pods, e.g.
qaboard-runs, and bind the Role to it and to the server's account:qaboard-runner-rolebinding.yamlapiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata:name: qaboard-runnernamespace: <runs-namespace>roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: qaboard-runnersubjects:- kind: ServiceAccountname: qaboard-runsnamespace: <runs-namespace>- kind: ServiceAccountname: <qaboard-service-account>namespace: <qaboard-namespace><runs-namespace>is a namespace where projects run,<qaboard-service-account>and<qaboard-namespace>the backend's ServiceAccount and namespace. The server only gets these permissions, in these namespaces. -
Set
QABOARD_TUNING_RUNNER=kubernetes,QABOARD_KUBERNETES_NAMESPACES,QABOARD_KUBERNETES_SERVICE_ACCOUNT=qaboard-runs, andQABOARD_KUBERNETES_SERVER_URL: the address where pods reach QA-Board.
What the driver pod gets, and what the server doesn't use
- The commit's
namespace,context(only those inQABOARD_KUBERNETES_CONTEXTS),image,image_pull_secrets,queue,priority_class,node_selector,tolerations,ttl_seconds_after_finished,envandmounts. - With an image per commit (
{commit}inimage), the commit's image. The server first asks its registry whether it has it, with the credentials ofimage_pull_secretsonly if its ServiceAccount may read Secrets in the namespace, never its own: No image for this commit fails the batch right away. The server's requests to the registry go to the host the project's qaboard.yaml names. - Not the commit's
job_template,workdir(the driver runs in the commit's artifacts),forward_env(the pod doesn't get the server's environment),resources(it getsQABOARD_KUBERNETES_DRIVER_RESOURCES),timeoutandpod_timeout(none:qa optimizeruns as long as its experiment), norrun_as(imageonly if you allow it, withQABOARD_KUBERNETES_ALLOW_RUN_AS_IMAGE). - No QA-Board token: if the server requires one to save results (
ingest.authset totoken), the runs need one, see give the runs a token. - The server retries the Kubernetes API for about 15 seconds when it starts or stops a batch, so that users don't wait for minutes when the API is down: they get its error.
- After it's submitted, the server watches the driver pod for about 20 seconds. An invalid image name, or a pod the namespace refuses, fails the batch right away; other reasons it waits, like an image it can't pull yet or no free node, are written in the batch's logs.
- Redoing runs works the same way: a driver pod runs
qa batchwith the runs to redo.
One namespace per team
A namespace is a trust domain. The pod that runs qa batch runs the commit's code with the namespace's runs
ServiceAccount, which can create Jobs: any pod the namespace accepts, as any UID it accepts. So everyone who can start
batches from the web app in a namespace can act as anyone else in it, and a policy on the qaboard.io/user label
can't tell them apart: they share the ServiceAccount. Give each team its own namespace, like the queues of W&B Launch
or ClearML.
To keep a namespace's batches away from other namespaces' users, run them all as a team account:
QABOARD_KUBERNETES_RUN_AS=<uid>:<gid> (the chart's runAs), with a ValidatingAdmissionPolicy that pins runAsUser
for the runs ServiceAccount.
What the server checks, and the settings that make it stricter
The server starts qa batch as the signed-in user, never as root nor as a UID from qaboard.yaml (nor as the
image's user, unless you allow run_as: image), and only for users who signed in with the directory (LDAP, SAML, OIDC). It doesn't use the commit's job_template, the
server's environment, environment variables that change how bash, Python, the Kubernetes client or qa behave, nor
mounts other than PVCs and NFS. What runs may do is up to the namespace: its Pod Security level, admission policies
and quotas.
| Variable | Helm value | What it does |
|---|---|---|
QABOARD_KUBERNETES_NAMESPACES | runners.kubernetes.namespaces | Comma-separated namespaces where the server may start and stop runs. The first is the default. Empty: none |
QABOARD_KUBERNETES_SERVICE_ACCOUNT | runners.kubernetes.serviceAccount | The runs ServiceAccount, of the driver pods. Chart default: <release>-qaboard-runs. Unset: the namespace's default ServiceAccount |
QABOARD_KUBERNETES_SERVER_URL | runners.kubernetes.serverUrl | How the pods reach QA-Board, e.g. http://qaboard-proxy.qaboard.svc. Chart default: the proxy's Service. Unset: QABOARD_HOST and QABOARD_PROTOCOL |
QABOARD_KUBERNETES_RUN_AS | runners.kubernetes.runAs | user (default): driver pods run as the signed-in user. <uid>:<gid>: all of them run as this account, e.g. a team's |
QABOARD_KUBERNETES_ALLOW_HOST_PATH | runners.kubernetes.allowHostPath | 1: projects may mount host_path volumes. They need a "privileged" namespace |
QABOARD_KUBERNETES_MIN_UID | Users with a lower UID can't start batches on Kubernetes. Default: 1000 | |
QABOARD_KUBERNETES_TRUST_LOCAL_ACCOUNTS | 1: also map local accounts to the UID of the Unix account with the same name. Only if admins create them: anyone can sign up | |
QABOARD_KUBERNETES_ALLOW_RUN_AS_IMAGE | runners.kubernetes.allowRunAsImage | 1: projects with run_as: image run as their image's user, never root, still only for users who signed in with the directory. Whoever can push to a project then picks the UID its batches run as, with its mounts: allow it only where admission policies pin the UIDs. Default: refused |
QABOARD_KUBERNETES_DRIVER_RESOURCES | The driver pods' resources, as JSON. Default: {"cpu": "500m", "memory": "2Gi"} | |
QABOARD_KUBERNETES_CONTEXTS | The kubeconfig contexts the server may use, comma-separated. Others are ignored: the server uses its default (in a pod, its ServiceAccount) |
The chart's runners.kubernetes.dask also lets the server delete the DaskClusters of the
Dask batches on Kubernetes users stop, and runners.kubernetes.extraRules adds
rules to the Role.
On Celery workers
With QABOARD_TUNING_RUNNER=celery, qa batch runs in the backend's container like with local, and gets the
broker's address (CELERY_BROKER_URL, in its environment: it's not written in the script saved with the logs) and the
server's (QABOARD_HOST, QABOARD_PROTOCOL). Projects send the runs to the workers with runners.default: celery.
The Docker Compose deployment and the Helm chart (celery.enabled) start RabbitMQ and Flower, not workers: start
workers on your machines, see Run batches on Celery workers. Anyone who can reach
RabbitMQ can run commands on the workers: set its password, see
passwords and admin services.
Stop runs
Stop runs, on a commit page, stops each command that started the batch's runs, with its runner, then marks the pending runs as failed:
| Runner | What the server does | It needs |
|---|---|---|
lsf | bkill on the batch's jobs | QA_RUNNERS_LSF_BRIDGE |
dask, workers on LSF | bkill on the workers and the driver | QA_RUNNERS_LSF_BRIDGE |
dask, workers on Slurm, PBS, SGE, HTCondor, OAR or Moab | Cancels the user's worker and driver jobs by their name (scancel, qdel...) | The scheduler's commands in the backend's image: it doesn't have them |
dask, workers on Kubernetes | Deletes the batch's DaskCluster and its driver's Job | The namespace in QABOARD_KUBERNETES_NAMESPACES, and the chart's runners.kubernetes.dask |
dask, an existing cluster | Stops the batch's runs on the scheduler | The scheduler in QABOARD_DASK_SCHEDULERS, and distributed installed in the backend |
kubernetes | Deletes the batch's Jobs, in its project, and the driver pod of batches started from the web app | The RoleBinding above, and the namespace in QABOARD_KUBERNETES_NAMESPACES. When it finds no Job there, e.g. in a kubeconfig context it may not use, it warns |
celery | Revokes the batch's tasks: workers drop those that wait, and stop those that run with what they started | The broker in CELERY_BROKER_URL, and workers with Celery 5.3 or later. Batches sent to another broker, e.g. a team's virtual host, go on |
local, qa batch started by the server | Stops qa batch and the runs it started on the server | Several replicas of the backend: the replica that started it. The others say where it runs |
local, from a terminal or the CI | Nothing: the runs go on where they run | |
windows | Nothing |
With QABOARD_TUNING_RUNNER=celery, the server stops both: its qa batch, and the tasks it sent.
Details on stopping Celery tasks
qa batchstamps each task with the batch's command id, and Stop runs revokes the tasks with it: it only needs the broker, not a result backend. Workers need Celery 5.3 or later.- Workers remember revoked tasks in memory: a worker that starts afterwards may still run the batch's waiting tasks,
unless the workers save their state (
celery worker --statedb). - A running task is stopped with SIGTERM: the worker stops its
qa runand what it started. Workers with thethreadsorsolopools can't stop running tasks: they finish.
Security
The server only runs qa batch itself, in its container, when you set QABOARD_TUNING_RUNNER to local or
celery. Then, to start or redo runs, qa batch gets the server's environment minus its secrets:
- Variables whose names contain
TOKEN,ACCESS,AUTH,SECRET,PASS(PASSWORD...),KEY,PRIVATE,CREDENTIAL,DSNorDATABASE_URL, and those the runners never forward to the runs. - The settings of the database, LDAP, SMTP, sign-in (
QABOARD_OIDC_*,QABOARD_SAML_*), Sentry and analytics. - The broker's address,
CELERY_BROKER_URL, unlessqa batchmay send runs to Celery: withcelery, or a project whose qaboard.yaml hasrunners.celery.
It keeps what qa needs: PATH, HOME, the locale, QABOARD_HOST and QABOARD_PROTOCOL, proxies, CA bundles,
QA_RUNNERS_LSF_BRIDGE.
This is defense in depth. The projects' code still runs in the server's container, as its account: it can read the
files the server can, like its SAML keys, the LSF bridge's ssh key or a Kubernetes ServiceAccount's token, and reach
what the server can, like its database. Running the projects' code on the server means trusting every developer of
every project with the server's container: when you don't, use lsf or kubernetes, where qa batch runs as the
user who started it, away from the server.
Users and files
- Who can start runs: signed-in users with the developer role on the project. See Project access.
- Whose runs they are: QA-Board records the user who started them, and when your storage's paths have
{user}, saves their outputs in that user's folder instead ofQABOARD_DEFAULT_USER's. Withlsfandkubernetes, they also run as that user, so the files are theirs. - The storage: the backend writes the scripts and logs in the batch's folder, and runs code from the commit's
artifacts, in the results storage. It refuses any folder outside
QABOARD_STORAGE_ROOTS. The machines that runqa batchmount the storage at the same path as the server. - Emails: users get one when their batch ends, if the server sends emails: see Emails.
Environment variables
| Variable | What it does |
|---|---|
QABOARD_TUNING_RUNNER | Where qa batch runs for batches started from the web app: kubernetes, lsf, or in the backend's container with local or celery. Not set (default): the server starts none, and redoes runs only through QA_RUNNERS_LSF_BRIDGE, if set |
QABOARD_LOCAL_START_WAIT | With local and celery, how many seconds the server waits for qa batch before it answers, so users see right away when it fails to start. Default: 3 |
QA_RUNNERS_LSF_BRIDGE | The command template that runs LSF commands as {user}: starts, redoes and stops batches on LSF |
QABOARD_DASK_SCHEDULERS | Comma-separated addresses of the Dask schedulers the server may connect to, to stop batches |
CELERY_BROKER_URL | The broker, given to qa batch with celery |
QABOARD_HOST, QABOARD_PROTOCOL | The server's address, given to qa batch with celery |
QABOARD_ACCOUNTS_COMMAND | Where the server finds users' UIDs, for kubernetes. See users' accounts |
QABOARD_KUBERNETES_* | The namespaces, accounts and mounts the server may use on Kubernetes: see One namespace per team |
All the server's variables are in Environment variables.
- Project integrationChoose where runs execute
The runners, what each needs, and what works from the web app.
- User guideRunning from the web app
What your users see: the form, redo and stop, and why a batch didn't start.