Skip to main content

Run batches from the web app

When users click Start in Run Tests / Tuning, Redo on a run, or Stop runs, the QA-Board server runs qa for them. This page sets up where it does, so your users can run batches without a terminal, on your cluster, as themselves.

By default, the server starts no batches: QABOARD_TUNING_RUNNER isn't set, so the web app and qa batch --commit tell users to ask you, and that qa batch from CI or a terminal still works. Redo still goes through the LSF bridge if you set one. Setting QABOARD_TUNING_RUNNER is the opt-in.

The Run Tests / Tuning form: experiment int8-nms on the quant-search batch (2 tests), the Manual tuning tab with a grid of score_threshold 0.2 and 0.3 by nms_iou 0.5 and 0.7, "Explores 4 combinations", "8 total runs", and the Start 8 runs button.The Run Tests / Tuning form: experiment int8-nms on the quant-search batch (2 tests), the Manual tuning tab with a grid of score_threshold 0.2 and 0.3 by nms_iou 0.5 and 0.7, "Explores 4 combinations", "8 total runs", and the Start 8 runs button.
What users see: they choose tests and parameters, and Start sends the batch to the runner this page sets up.

How the server starts a batch​

For each request, the server writes a script in the batch's folder, in the results storage. The script goes to the commit's artifacts, sources their .envrc files, and runs qa batch --no-wait, or qa optimize for an automated tuning, for the user who asked. Two settings decide where things run:

  • QABOARD_TUNING_RUNNER, on the server: where that qa batch command runs.
  • The project's runner, runners.default in its qaboard.yaml: where qa batch sends the runs, like from a terminal. See Choose where runs execute.
QABOARD_TUNING_RUNNERqa batch runsAsFor
not set (default)Nowhere: the server refuses to start batches, and redoes runs only through the LSF bridge, if you set oneServers where only CI and terminals run batches
localIn the backend's containerThe server's accountTrying QA-Board on your laptop, or a server whose every user you trust with it
lsfIn an LSF job, sent through the LSF bridgeThe user who started itLSF sites
kubernetesIn a pod, in the project's namespaceThe user's UID and GIDKubernetes sites
celeryIn the backend's container, with the broker's addressThe server's account; the runs, as the workers'Projects whose runner is celery

Projects need their artifacts saved by the CI: see Let people tune from the web app for what projects do.

The form lists a batch's tests before it starts. For projects that list their inputs with their own iter_inputs, that runs their code: the server does it only with local or celery, in its container without its secrets. Otherwise the form says it can't list them, and users can still start the batch.

The Logs view of the tune-batching sweep: under "Started from QA-Board", the qa batch command kaito's tuning request ran, with its grid of max batch and max delay, and the end of its log; then the two runs left by the filter, and the Batch section with the command again.The Logs view of the tune-batching sweep: under "Started from QA-Board", the qa batch command kaito's tuning request ran, with its grid of max batch and max delay, and the end of its log; then the two runs left by the filter, and the Batch section with the command again.
The batch's Logs view shows the qa batch command the server ran and its log: the first place to look when a batch started from the web app doesn't run.
Runs started on the server run the projects' code in its container

With local and celery, qa batch runs in the backend's container, as the server's account: everyone with the developer role on a project, and anyone with a token that sends its artifacts (qa save-artifacts), can run code there. QA-Board keeps the server's secrets out of its environment, but the code can read what the server's account can. Use lsf or kubernetes when you don't trust every developer with the server: see Security. Admin → Settings and the security checklist in Admin → Overview warn about it.

On the server itself: local​

To try batches from the web app on your laptop, set it in .env (or the overlay's services.backend.environment, or the Helm chart's config), then restart the backend:

.env
# Batches started from the web app run in the backend's container, as its account: see the warning above
QABOARD_TUNING_RUNNER=local

qa batch runs in the backend's container, so the container needs what the projects' .envrc files activate, e.g. their Python environments on a shared drive.

qa batch runs in the background, in its own process group: the server answers users once it started, and it runs as long as it needs. If it fails within a few seconds (QABOARD_LOCAL_START_WAIT, 3 by default), users see why right away; later, the web app shows it from its logs. Redoing runs works the same way, unless the server has an LSF bridge.

  • With the project's local runner, the runs execute in the backend's container too, and share its CPUs. Restarting the backend's container, e.g. to deploy, stops them. The web app warns users of projects without another runner.
  • With the project's lsf runner, qa sends the runs through the LSF bridge if the server has no bsub.
  • With the project's celery runner, the backend needs to reach the broker.

On LSF​

The server can't run bsub from its container: it goes through a host that can, the bridge.

  1. Get a bridge host that has LSF, where the server can log in with ssh, and run commands as your users, e.g. with a sudo rule for its account.

  2. Set the bridge, in the backend's environment. It's a command template: QA-Board replaces {bsub_command} with the command to run (bsub, bwait, bkill, or bash <script>) and {user} with the user who started the runs:

    QABOARD_TUNING_RUNNER=lsf
    QA_RUNNERS_LSF_BRIDGE="ssh qaboard@<lsf-host> 'qaboard-as-user {user} {bsub_command}'"

    <lsf-host> is the bridge host, and qaboard-as-user a wrapper there that runs the rest of its arguments as {user}: see send jobs through a bridge host for one. Mount the ssh key in the backend's container.

  3. Try it: start a batch from the web app. The Logs view shows the bsub command and its output.

What the server does with the bridge:

  • Start: qa batch runs in an LSF job, on the project's runners.lsf.queue (runners.lsf.long_queue for automated tuning), as the user. LSF's report is in the batch's logs.
  • Redo: qa batch runs on the bridge host, as the user, then sends the runs with the runner they had.
  • Stop: bkill on the batch's jobs, including the workers of Dask batches.
Test the template by hand

The commands QA-Board puts in {bsub_command} have quotes and here-documents: check that they reach the bridge host unchanged, e.g. by running the template with bjobs -w in place of {bsub_command} from the backend's container.

On Kubernetes​

With QABOARD_TUNING_RUNNER=kubernetes, the server starts qa batch in a driver pod: a Job of one pod, named qa-<first 8 characters of the command id>-driver-<random>. It runs in the namespace of the commit's runners.kubernetes settings (else the first allowed namespace), with their image and mounts, as the signed-in user's UID and GID, from the server's users' accounts. Its qa batch --no-wait creates the runs' Jobs with the namespace's runs ServiceAccount, then ends.

With the Helm chart:

  1. List the namespaces where batches run, in your values. They must exist:

    my-values.yaml
    runners:
    kubernetes:
    enabled: true
    namespaces: [algo-qa, camera-qa]

    In each namespace, the chart creates the runner's Role (<release>-qaboard-runs), the runs ServiceAccount, and a RoleBinding for it and for the backend's ServiceAccount. It sets QABOARD_TUNING_RUNNER=kubernetes, QABOARD_KUBERNETES_NAMESPACES, QABOARD_KUBERNETES_SERVICE_ACCOUNT and QABOARD_KUBERNETES_SERVER_URL (the proxy's Service). Installing it needs the permission to create Roles there.

  2. Know your users' UIDs. The server reads them from its users' accounts: mount /etc/passwd, or set QABOARD_ACCOUNTS_COMMAND. Only users who sign in with the directory (LDAP, SAML, OIDC) can start batches, unless you set runAs or QABOARD_KUBERNETES_TRUST_LOCAL_ACCOUNTS.

  3. Enforce who runs as whom in each namespace, with its Pod Security level and a ValidatingAdmissionPolicy on the UIDs: see one namespace per team.

  4. Try it: start a batch from the web app. Its logs say where the driver pod runs, and why it waits if it's pending.

Projects need, in the qaboard.yaml of the commits they run:

  • runners.default: kubernetes, or Dask on Kubernetes: otherwise the driver pod does the runs itself.
  • runners.kubernetes.image, with bash and qaboard[kubernetes], and mounts for the results storage and the artifacts, at the same paths as the server's. See Run batches on Kubernetes.
  • Or no mounts, for projects without a shared storage: the pods download the commit's artifacts from QA-Board and send their results back with the batch's token, and the driver sends its log. That token only sends this batch's results (uploads).
Limits without a shared storage
  • What qa batch reads (the batches as they were defined, the runs to redo) goes in the driver's ConfigMap: at most 768 KiB. Larger batches are refused when they start: select fewer, or start them from CI.
  • The driver's log reaches the web app while the batch's token is valid: until 10 minutes after its last run ends. Anything later, or refused by the server, is only in the pod's logs: the log's first line says how to read them (kubectl logs -n <namespace> job/<driver>), and a line says when some output is missing.
  • qa optimize can't resume an experiment started again: its journal stays in the driver pod.
Without the Helm chart

The backend talks to the cluster with its ServiceAccount when it runs in a pod, else with its kubeconfig. In each namespace where batches run:

  1. Create the runner's Role, from Get the permissions.

  2. Create a ServiceAccount for the driver pods, e.g. qaboard-runs, and bind the Role to it and to the server's account:

    qaboard-runner-rolebinding.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
    name: qaboard-runner
    namespace: <runs-namespace>
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: qaboard-runner
    subjects:
    - kind: ServiceAccount
    name: qaboard-runs
    namespace: <runs-namespace>
    - kind: ServiceAccount
    name: <qaboard-service-account>
    namespace: <qaboard-namespace>

    <runs-namespace> is a namespace where projects run, <qaboard-service-account> and <qaboard-namespace> the backend's ServiceAccount and namespace. The server only gets these permissions, in these namespaces.

  3. Set QABOARD_TUNING_RUNNER=kubernetes, QABOARD_KUBERNETES_NAMESPACES, QABOARD_KUBERNETES_SERVICE_ACCOUNT=qaboard-runs, and QABOARD_KUBERNETES_SERVER_URL: the address where pods reach QA-Board.

What the driver pod gets, and what the server doesn't use
  • The commit's namespace, context (only those in QABOARD_KUBERNETES_CONTEXTS), image, image_pull_secrets, queue, priority_class, node_selector, tolerations, ttl_seconds_after_finished, env and mounts.
  • With an image per commit ({commit} in image), the commit's image. The server first asks its registry whether it has it, with the credentials of image_pull_secrets only if its ServiceAccount may read Secrets in the namespace, never its own: No image for this commit fails the batch right away. The server's requests to the registry go to the host the project's qaboard.yaml names.
  • Not the commit's job_template, workdir (the driver runs in the commit's artifacts), forward_env (the pod doesn't get the server's environment), resources (it gets QABOARD_KUBERNETES_DRIVER_RESOURCES), timeout and pod_timeout (none: qa optimize runs as long as its experiment), nor run_as (image only if you allow it, with QABOARD_KUBERNETES_ALLOW_RUN_AS_IMAGE).
  • No QA-Board token: if the server requires one to save results (ingest.auth set to token), the runs need one, see give the runs a token.
  • The server retries the Kubernetes API for about 15 seconds when it starts or stops a batch, so that users don't wait for minutes when the API is down: they get its error.
  • After it's submitted, the server watches the driver pod for about 20 seconds. An invalid image name, or a pod the namespace refuses, fails the batch right away; other reasons it waits, like an image it can't pull yet or no free node, are written in the batch's logs.
  • Redoing runs works the same way: a driver pod runs qa batch with the runs to redo.

One namespace per team​

A namespace is a trust domain. The pod that runs qa batch runs the commit's code with the namespace's runs ServiceAccount, which can create Jobs: any pod the namespace accepts, as any UID it accepts. So everyone who can start batches from the web app in a namespace can act as anyone else in it, and a policy on the qaboard.io/user label can't tell them apart: they share the ServiceAccount. Give each team its own namespace, like the queues of W&B Launch or ClearML.

To keep a namespace's batches away from other namespaces' users, run them all as a team account: QABOARD_KUBERNETES_RUN_AS=<uid>:<gid> (the chart's runAs), with a ValidatingAdmissionPolicy that pins runAsUser for the runs ServiceAccount.

What the server checks, and the settings that make it stricter

The server starts qa batch as the signed-in user, never as root nor as a UID from qaboard.yaml (nor as the image's user, unless you allow run_as: image), and only for users who signed in with the directory (LDAP, SAML, OIDC). It doesn't use the commit's job_template, the server's environment, environment variables that change how bash, Python, the Kubernetes client or qa behave, nor mounts other than PVCs and NFS. What runs may do is up to the namespace: its Pod Security level, admission policies and quotas.

VariableHelm valueWhat it does
QABOARD_KUBERNETES_NAMESPACESrunners.kubernetes.namespacesComma-separated namespaces where the server may start and stop runs. The first is the default. Empty: none
QABOARD_KUBERNETES_SERVICE_ACCOUNTrunners.kubernetes.serviceAccountThe runs ServiceAccount, of the driver pods. Chart default: <release>-qaboard-runs. Unset: the namespace's default ServiceAccount
QABOARD_KUBERNETES_SERVER_URLrunners.kubernetes.serverUrlHow the pods reach QA-Board, e.g. http://qaboard-proxy.qaboard.svc. Chart default: the proxy's Service. Unset: QABOARD_HOST and QABOARD_PROTOCOL
QABOARD_KUBERNETES_RUN_ASrunners.kubernetes.runAsuser (default): driver pods run as the signed-in user. <uid>:<gid>: all of them run as this account, e.g. a team's
QABOARD_KUBERNETES_ALLOW_HOST_PATHrunners.kubernetes.allowHostPath1: projects may mount host_path volumes. They need a "privileged" namespace
QABOARD_KUBERNETES_MIN_UIDUsers with a lower UID can't start batches on Kubernetes. Default: 1000
QABOARD_KUBERNETES_TRUST_LOCAL_ACCOUNTS1: also map local accounts to the UID of the Unix account with the same name. Only if admins create them: anyone can sign up
QABOARD_KUBERNETES_ALLOW_RUN_AS_IMAGErunners.kubernetes.allowRunAsImage1: projects with run_as: image run as their image's user, never root, still only for users who signed in with the directory. Whoever can push to a project then picks the UID its batches run as, with its mounts: allow it only where admission policies pin the UIDs. Default: refused
QABOARD_KUBERNETES_DRIVER_RESOURCESThe driver pods' resources, as JSON. Default: {"cpu": "500m", "memory": "2Gi"}
QABOARD_KUBERNETES_CONTEXTSThe kubeconfig contexts the server may use, comma-separated. Others are ignored: the server uses its default (in a pod, its ServiceAccount)

The chart's runners.kubernetes.dask also lets the server delete the DaskClusters of the Dask batches on Kubernetes users stop, and runners.kubernetes.extraRules adds rules to the Role.

On Celery workers​

With QABOARD_TUNING_RUNNER=celery, qa batch runs in the backend's container like with local, and gets the broker's address (CELERY_BROKER_URL, in its environment: it's not written in the script saved with the logs) and the server's (QABOARD_HOST, QABOARD_PROTOCOL). Projects send the runs to the workers with runners.default: celery.

The Docker Compose deployment and the Helm chart (celery.enabled) start RabbitMQ and Flower, not workers: start workers on your machines, see Run batches on Celery workers. Anyone who can reach RabbitMQ can run commands on the workers: set its password, see passwords and admin services.

Stop runs​

Stop runs, on a commit page, stops each command that started the batch's runs, with its runner, then marks the pending runs as failed:

RunnerWhat the server doesIt needs
lsfbkill on the batch's jobsQA_RUNNERS_LSF_BRIDGE
dask, workers on LSFbkill on the workers and the driverQA_RUNNERS_LSF_BRIDGE
dask, workers on Slurm, PBS, SGE, HTCondor, OAR or MoabCancels the user's worker and driver jobs by their name (scancel, qdel...)The scheduler's commands in the backend's image: it doesn't have them
dask, workers on KubernetesDeletes the batch's DaskCluster and its driver's JobThe namespace in QABOARD_KUBERNETES_NAMESPACES, and the chart's runners.kubernetes.dask
dask, an existing clusterStops the batch's runs on the schedulerThe scheduler in QABOARD_DASK_SCHEDULERS, and distributed installed in the backend
kubernetesDeletes the batch's Jobs, in its project, and the driver pod of batches started from the web appThe RoleBinding above, and the namespace in QABOARD_KUBERNETES_NAMESPACES. When it finds no Job there, e.g. in a kubeconfig context it may not use, it warns
celeryRevokes the batch's tasks: workers drop those that wait, and stop those that run with what they startedThe broker in CELERY_BROKER_URL, and workers with Celery 5.3 or later. Batches sent to another broker, e.g. a team's virtual host, go on
local, qa batch started by the serverStops qa batch and the runs it started on the serverSeveral replicas of the backend: the replica that started it. The others say where it runs
local, from a terminal or the CINothing: the runs go on where they run
windowsNothing

With QABOARD_TUNING_RUNNER=celery, the server stops both: its qa batch, and the tasks it sent.

Details on stopping Celery tasks
  • qa batch stamps each task with the batch's command id, and Stop runs revokes the tasks with it: it only needs the broker, not a result backend. Workers need Celery 5.3 or later.
  • Workers remember revoked tasks in memory: a worker that starts afterwards may still run the batch's waiting tasks, unless the workers save their state (celery worker --statedb).
  • A running task is stopped with SIGTERM: the worker stops its qa run and what it started. Workers with the threads or solo pools can't stop running tasks: they finish.

Security​

The server only runs qa batch itself, in its container, when you set QABOARD_TUNING_RUNNER to local or celery. Then, to start or redo runs, qa batch gets the server's environment minus its secrets:

  • Variables whose names contain TOKEN, ACCESS, AUTH, SECRET, PASS (PASSWORD...), KEY, PRIVATE, CREDENTIAL, DSN or DATABASE_URL, and those the runners never forward to the runs.
  • The settings of the database, LDAP, SMTP, sign-in (QABOARD_OIDC_*, QABOARD_SAML_*), Sentry and analytics.
  • The broker's address, CELERY_BROKER_URL, unless qa batch may send runs to Celery: with celery, or a project whose qaboard.yaml has runners.celery.

It keeps what qa needs: PATH, HOME, the locale, QABOARD_HOST and QABOARD_PROTOCOL, proxies, CA bundles, QA_RUNNERS_LSF_BRIDGE.

This is defense in depth. The projects' code still runs in the server's container, as its account: it can read the files the server can, like its SAML keys, the LSF bridge's ssh key or a Kubernetes ServiceAccount's token, and reach what the server can, like its database. Running the projects' code on the server means trusting every developer of every project with the server's container: when you don't, use lsf or kubernetes, where qa batch runs as the user who started it, away from the server.

Users and files​

  • Who can start runs: signed-in users with the developer role on the project. See Project access.
  • Whose runs they are: QA-Board records the user who started them, and when your storage's paths have {user}, saves their outputs in that user's folder instead of QABOARD_DEFAULT_USER's. With lsf and kubernetes, they also run as that user, so the files are theirs.
  • The storage: the backend writes the scripts and logs in the batch's folder, and runs code from the commit's artifacts, in the results storage. It refuses any folder outside QABOARD_STORAGE_ROOTS. The machines that run qa batch mount the storage at the same path as the server.
  • Emails: users get one when their batch ends, if the server sends emails: see Emails.

Environment variables​

VariableWhat it does
QABOARD_TUNING_RUNNERWhere qa batch runs for batches started from the web app: kubernetes, lsf, or in the backend's container with local or celery. Not set (default): the server starts none, and redoes runs only through QA_RUNNERS_LSF_BRIDGE, if set
QABOARD_LOCAL_START_WAITWith local and celery, how many seconds the server waits for qa batch before it answers, so users see right away when it fails to start. Default: 3
QA_RUNNERS_LSF_BRIDGEThe command template that runs LSF commands as {user}: starts, redoes and stops batches on LSF
QABOARD_DASK_SCHEDULERSComma-separated addresses of the Dask schedulers the server may connect to, to stop batches
CELERY_BROKER_URLThe broker, given to qa batch with celery
QABOARD_HOST, QABOARD_PROTOCOLThe server's address, given to qa batch with celery
QABOARD_ACCOUNTS_COMMANDWhere the server finds users' UIDs, for kubernetes. See users' accounts
QABOARD_KUBERNETES_*The namespaces, accounts and mounts the server may use on Kubernetes: see One namespace per team

All the server's variables are in Environment variables.