# QA-Board > QA-Board is an open-source (Apache 2.0), self-hosted quality tracking platform. Every change runs on a test set, and QA-Board puts its outputs (images, videos, point clouds, plots, flame graphs, benchmarks, test reports, logs) and metrics next to a reference's, for the people and the coding agents who review it. The `qa` command line (the `qaboard` Python package) wraps your code, runs it on batches of inputs, locally, on clusters or in CI, and saves the results. The server (Docker Compose or a Helm chart) stores them, and its web app compares every commit with the reference branch or a milestone. Coding agents: to set up QA-Board or integrate a project, read [Integrate with a coding agent](https://samsung.github.io/qaboard/docs/agents/) first, then the quickstart. The source code is at https://github.com/Samsung/qaboard. The JSON schemas of qaboard.yaml and the batch files are under Optional. ## Get started - [Run QA-Board on your computer](https://samsung.github.io/qaboard/docs/get-started/quickstart/): Start a QA-Board server on your computer, then save the results of an example project to it and look at them in the web app. - [Your project on QA-Board](https://samsung.github.io/qaboard/docs/get-started/tutorial/): Wrap your code with qa, run it on your test set, connect your repository, run it in CI, and compare every commit with your reference branch. - [Key concepts](https://samsung.github.io/qaboard/docs/get-started/concepts/): The words QA-Board uses (projects, commits, batches, runs, inputs, configurations, references, milestones, metrics) and who does what between the qa command line and the server. - [The regressions QA-Board catches](https://samsung.github.io/qaboard/docs/use-cases/): The regressions that pass/fail tests and averages miss, in images, models, latency, robotics, test suites and LLM agents, and what QA-Board shows to catch them. - [Image quality: catch what the metrics miss](https://samsung.github.io/qaboard/docs/use-cases/image-quality/): For camera, ISP and imaging teams. Every commit's images next to the release's, synced down to the pixel, to catch the artifacts a better PSNR hides. - [Model evaluation: see what one mAP hides](https://samsung.github.io/qaboard/docs/use-cases/detection-eval/): For computer vision and ML teams. Metrics per slice, class and object size against the release model, and the images where the new model went wrong. - [Performance: catch tail latency before users do](https://samsung.github.io/qaboard/docs/use-cases/serving-latency/): For backend and performance teams. Latency percentiles, flame graph diffs and benchmarks against the reference's on every commit, to check any "it's faster" claim. - [Robotics: see where the trajectory went wrong](https://samsung.github.io/qaboard/docs/use-cases/robotics-3d/): For robotics, perception and automotive teams. Trajectories against the ground truth, 3D maps and sensor videos in sync with the reference's, on every commit. - [LLM agents: review every task, not an average](https://samsung.github.io/qaboard/docs/use-cases/agent-evals/): For teams shipping LLM and agent features. Each eval task against the reference, transcripts diffed, pass rates and cost per commit, a gate on what must never happen. - [Software quality: what a green CI doesn't tell you](https://samsung.github.io/qaboard/docs/use-cases/software-quality/): For teams that follow their tests in CI. Failures, flaky and skipped tests and coverage per suite against the reference, and a gate that catches deleted tests. - [QA-Board compared with other tools](https://samsung.github.io/qaboard/docs/alternatives/): Where QA-Board fits next to experiment trackers, FiftyOne and Voxel51, Rerun and Foxglove, DVC, LLM eval platforms, test dashboards, benchmarking tools and in-house dashboards. What each does best, where QA-Board differs, and when teams use both. ## User guide: reviewing results in the web app - [The QA-Board user guide](https://samsung.github.io/qaboard/docs/user-guide/overview/): What QA-Board does for you, the few words it uses, a map of the commit page, and the page to read for each task. - [Your first review](https://samsung.github.io/qaboard/docs/user-guide/first-review/): Review one commit of a demo project step by step: its metrics against the last release, its images in sync, their difference, the commit that changed them, a milestone and a link to share. - [Finding results](https://samsung.github.io/qaboard/docs/user-guide/finding-results/): Find a project, a commit, a branch or a batch of runs, and filter what you see. - [The commit page](https://samsung.github.io/qaboard/docs/user-guide/commit-page/): Find your way around a commit's results, the new and reference bars, the view options, and which view answers which question. - [Reading metrics](https://samsung.github.io/qaboard/docs/user-guide/metrics/): Did this commit make things better, and where? The Summary, the Metrics Diff and the Metrics Table, targets, and what the colors mean. - [New and reference](https://samsung.github.io/qaboard/docs/user-guide/comparing-results/): Choose what your results are compared with, name both sides, compare with several references, and read how each run is matched with a reference run. - [Pick each run's reference run](https://samsung.github.io/qaboard/docs/user-guide/reference-runs/): Compare runs on different inputs, like two sensors' shots or a register sweep's captures, pair them by name, and pick another reference run for a run or for all of them. - [Milestones](https://samsung.github.io/qaboard/docs/user-guide/milestones/): Save results worth comparing with, like a release or the best tuning, compare with them and share them. - [History](https://samsung.github.io/qaboard/docs/user-guide/history-and-dashboards/): Follow metrics over time on a branch, find the commit that caused a regression, and see the history of a single input. - [The image viewer](https://samsung.github.io/qaboard/docs/user-guide/image-viewer/): Compare images side by side, with a wipe or their difference, zoom to the pixel, read pixel values, jump to preset regions of interest and find where images differ. - [The video viewer](https://samsung.github.io/qaboard/docs/user-guide/video-viewer/): Compare videos frame by frame, in sync, with a wipe or their difference, and find the frames where they differ. - [Plots, flame graphs, 3D and more](https://samsung.github.io/qaboard/docs/user-guide/other-viewers/): Compare plots, training curves, flame graphs, test reports, benchmarks, text, HTML reports, point clouds and trajectories with the reference's in the web app. - [Running from the web app](https://samsung.github.io/qaboard/docs/user-guide/running-from-the-webapp/): Redo failed runs, run a commit on other tests or settings, and start tuning experiments from the web app, without a terminal. - [Tune parameters, step by step](https://samsung.github.io/qaboard/docs/user-guide/tuning/): From a first grid of values to parameters you can trust, from the web app. What to tune, how to read the results, and the mistakes to avoid, without an optimization background. - [Find the best parameters automatically](https://samsung.github.io/qaboard/docs/user-guide/automated-tuning/): Let QA-Board search the parameters that give the best metrics, read what it found, and decide what to do next, without an optimization background. - [Choosing the tests to run](https://samsung.github.io/qaboard/docs/user-guide/available-tests/): See which inputs and configurations a batch of tests runs on a commit, and write your own batches in the web app, before you run them. - [Logs and failed runs](https://samsung.github.io/qaboard/docs/user-guide/logs/): Find out why a run failed, follow the runs still running: live logs, errors, cluster reports and the commands that started a batch. - [Runs activity](https://samsung.github.io/qaboard/docs/user-guide/runs-activity/): See what waits and runs now, what ended recently and what starts soon, grouped by batch, user, queue or any job option, and clean up runs that are stuck. - [Sharing results](https://samsung.github.io/qaboard/docs/user-guide/sharing/): Share what you see with a link, copy paths and images, export files, and report bugs. - [Browsing files](https://samsung.github.io/qaboard/docs/user-guide/files/): Browse the output and artifacts folders in your browser, copy Linux and Windows paths, and find files fast with the keyboard. - [Storage usage](https://samsung.github.io/qaboard/docs/user-guide/storage-usage/): See how much storage your runs and exports use, what QA-Board deletes automatically, find what it doesn't track, and delete what you don't need. - [Your account and access tokens](https://samsung.github.io/qaboard/docs/user-guide/account/): Sign in, keep your preferences everywhere, create API tokens for scripts and CI, and sign in from the qa CLI. - [Who can access a project](https://samsung.github.io/qaboard/docs/user-guide/project-access/): See your role on a project, ask for access, and, as an owner, choose who can see it and act on its runs. - [Set a project's gate rules](https://samsung.github.io/qaboard/docs/user-guide/project-gate/): As a project's owner, set the quality rules every commit must meet, that no commit can change, and decide what happens when a commit loosens its own rules. - [Tips & FAQ](https://samsung.github.io/qaboard/docs/user-guide/tips-and-faq/): Answers to common questions about the QA-Board web app, the keyboard shortcuts worth learning, and lesser-known tricks. - [What's new](https://samsung.github.io/qaboard/docs/user-guide/whats-new/): How QA-Board's release notes reach you in the web app, and where to find all of them. ## Project integration: qa, qaboard.yaml, runners, CI - [How a project integrates QA-Board](https://samsung.github.io/qaboard/docs/introduction/): How QA-Board runs your code and where its results go, then the steps to add it to a project, from qa init to CI, with what you get after each. - [Install the qa command line](https://samsung.github.io/qaboard/docs/installation/): Install qa on the machines that run your code, point it to your QA-Board server, and give it an API token for your scripts. - [Add QA-Board to your project](https://samsung.github.io/qaboard/docs/project-init/): Create qaboard.yaml and the qa/ folder with qa init, fill in what qa needs to know about your project, connect your git host, and check your files in your editor. - [Running your code](https://samsung.github.io/qaboard/docs/running-your-code/): Write the run() function that runs your code on one input, saves its output files and returns its metrics, for Python code or an executable. - [Log from your code](https://samsung.github.io/qaboard/docs/log-from-code/): Create runs from a training loop, a notebook or an evaluation script with qaboard.start_run(): log values at each step, metrics, parameters and files, and see them in the web app next to the reference's while the code runs. - [Upload results from any tool](https://samsung.github.io/qaboard/docs/upload-results/): Save files other tools produced as a run, with qa upload: test reports, benchmark files, profiles, coverage. No run() function to write, from any CI job and any language, compared with the reference like every run. - [Inputs](https://samsung.github.io/qaboard/docs/inputs/): Tell qa where your inputs are, and how to recognize them inside folders, so that qa run and qa batch find them. - [Configurations](https://samsung.github.io/qaboard/docs/specifying-configurations/): Run your code on the same inputs with other settings, an ISO, a model, a build, parameters, and read them in run() as context.configs and context.params. - [Metrics](https://samsung.github.io/qaboard/docs/computing-quantitative-metrics/): Return metrics from your run() function, declare their label, unit, target and which way is better where your code computes them, and choose the metrics QA-Board shows by default. - [Visualizations](https://samsung.github.io/qaboard/docs/visualizations/): Show your runs' output files on the commit page, next to the reference's. List them in qaboard.yaml, hide the debug ones, offer a file per frame or stage, and preset regions of interest. - [Viewers and file formats](https://samsung.github.io/qaboard/docs/viewers/): The files each viewer of the web app reads, how to write them from your code, and the options you can give each viewer in qaboard.yaml. - [Benchmark suites](https://samsung.github.io/qaboard/docs/benchmarks/): Bring a pytest-benchmark, Google Benchmark, hyperfine, Criterion, JMH or go test suite into QA-Board, see each benchmark against the reference's, and follow it as a metric. - [Profiles and flame graphs](https://samsung.github.io/qaboard/docs/flame-graphs/): Save what perf, py-spy, async-profiler, Go's pprof or Node.js record in your runs, and see a differential flame graph against the reference's profile. - [References and milestones](https://samsung.github.io/qaboard/docs/references-and-milestones/): Choose what your project's results are compared with by default, list reference branches, and use milestones from the CI and scripts. - [Batches of inputs](https://samsung.github.io/qaboard/docs/batches-running-on-multiple-inputs/): List inputs and configurations in a batch file, run them all with qa batch, and see their runs together in the web app. - [Choose where runs execute](https://samsung.github.io/qaboard/docs/runners/): Pick the runner that sends the runs of qa batch to your computer, an HPC cluster, Kubernetes, Celery workers or Windows hosts, and know what each one needs. - [Coming from another tool](https://samsung.github.io/qaboard/docs/coming-from-other-tools/): Run the evaluations you run with MLflow Projects, W&B Launch, ClearML Agent, Slurm or LSF job arrays, Kubernetes Jobs, Ray, Dask or CI matrices with QA-Board, and see what you gain. - [Run batches on your machine](https://samsung.github.io/qaboard/docs/local-multiprocessing/): Run the runs of qa batch on the machine where you start it, several at a time, with a timeout, for all batches or batch by batch. - [Run batches on an LSF cluster](https://samsung.github.io/qaboard/docs/lsf-integration/): Send each run of qa batch to an LSF cluster as a job, choose its queue, memory and threads, limit how many run at the same time, and read LSF's reports in the logs. - [Run batches with Dask, on Slurm, PBS, SGE, LSF or Kubernetes](https://samsung.github.io/qaboard/docs/dask-integration/): Run batches on a few long-lived Dask workers that qa batch starts on Slurm, PBS, SGE, HTCondor, OAR, Moab, LSF or Kubernetes, or on an existing Dask cluster, with a bounded concurrency. - [Run batches on Kubernetes](https://samsung.github.io/qaboard/docs/kubernetes-runner/): Run each run of qa batch as a pod of a Kubernetes Job, with your shared storage mounted, as your own user, within your cluster's queues and quotas. - [Run batches on Celery workers](https://samsung.github.io/qaboard/docs/celery-integration/): Send the runs of qa batch to Celery workers through a message broker, like the RabbitMQ of QA-Board's Docker Compose deployment, and route them to the right machines. - [Run batches on Windows hosts with Jenkins](https://samsung.github.io/qaboard/docs/jenkins-integration/): Send the runs of qa batch to Windows hosts, as builds of a Jenkins job that runs one command, with the windows runner. - [Check your setup with qa doctor](https://samsung.github.io/qaboard/docs/doctor/): Run qa doctor to check everything a batch needs, from qaboard.yaml to the runner's permissions, and get how to fix each problem. - [Advanced batches](https://samsung.github.io/qaboard/docs/batches-advanced/): Matrices of parameters, definitions reused with YAML anchors, inputs selected by their metadata, several batch files, lists of runs, and the environment variables of qa batch. - [The qa command line](https://samsung.github.io/qaboard/docs/using-the-qa-cli/): The qa options you use every day, to share results, name experiments, preview a batch, choose where it runs, skip what already ran, and get an email when it ends. - [Run QA-Board in your CI](https://samsung.github.io/qaboard/docs/ci-integration/): Run your batch on every push with GitHub Actions, GitLab CI or Jenkins, so that every commit has results to compare with the reference. - [Quality gate](https://samsung.github.io/qaboard/docs/quality-gate/): Fail the CI when a commit's metrics regress against a reference, with qa gate and rules in qaboard.yaml. - [The gate on pull requests](https://samsung.github.io/qaboard/docs/quality-gate-pull-requests/): Post the quality gate's verdict where changes are reviewed, as a commit status and one comment on the pull request or merge request that every gate job updates. - [Integrations: links, badges and CI jobs](https://samsung.github.io/qaboard/docs/triggering-third-party-tools/): Add links, badges, webhooks, GitHub Actions workflows, GitLab CI jobs and Jenkins builds to the web app's Integrations menu. - [Results in scripts](https://samsung.github.io/qaboard/docs/apis/): Every way to use QA-Board's results from scripts: the runs a web app page shows with qaboard.runs(url) or qa runs, adding metrics and files to them, runs' folders with the qa CLI, the quality gate, and the HTTP API from any language. - [HTTP API](https://samsung.github.io/qaboard/docs/http-api/): Read QA-Board's results over HTTP from any language: authenticate with a token, get a commit's batches, runs, metrics and files, and compare them with a reference. - [Integrate with a coding agent](https://samsung.github.io/qaboard/docs/agents/): Start a server and integrate a project with a coding agent. Skills teach agents to wrap a project with qa, declare metrics and visualizations, set up CI and the gate, and to read results before claiming a change is better. - [Input metadata and external databases](https://samsung.github.io/qaboard/docs/metadata-integration-external-databases/): Give inputs metadata to filter batches, label runs and preset regions of interest, and read inputs from your own database. - [Let people tune from the web app](https://samsung.github.io/qaboard/docs/tuning-from-the-webapp/): Let everyone on your project run a commit's code on more tests, or with other parameters, from the web app. - [Automated tuning with qa optimize](https://samsung.github.io/qaboard/docs/auto-optimization/): Let qa optimize search the parameters that give the best metrics on a batch, from the CLI or the web app, and resume where it stopped. - [Bit accuracy tests](https://samsung.github.io/qaboard/docs/bit-accuracy/): Fail the CI when a commit changes output files byte for byte, and see which files changed in the web app. - [Deep learning and MLOps workflows](https://samsung.github.io/qaboard/docs/deep-learning/): Evaluate your models' inference in QA-Board next to the training tracker you already use, log trainings from your code, and gate model releases in the CI. - [Monorepos and many repositories](https://samsung.github.io/qaboard/docs/monorepo-subprojects/): Give each folder of a monorepo its own QA-Board project, and share one configuration between many repositories. - [Debugging runs in an IDE](https://samsung.github.io/qaboard/docs/debugging-runs-with-an-IDE/): Set breakpoints in your run() function by debugging one qa run in VS Code or PyCharm. - [Artifacts](https://samsung.github.io/qaboard/docs/storage/artifacts/): Save what a commit needs to run again, like its build, with qa save-artifacts in CI, so that people can redo runs and tune from the web app. - [Where is the data saved?](https://samsung.github.io/qaboard/docs/storage/where-is-the-data/): Where qa saves your runs' files and your builds, how to set the results storage in qaboard.yaml or with QA_STORAGE, and how to find a commit's folders from scripts. - [Send results without a shared storage](https://samsung.github.io/qaboard/docs/storage/uploads/): Runs on GitHub-hosted CI, a laptop or a cluster that doesn't mount the server's storage send their files to QA-Board over HTTP: what to set, what is sent and when, and what still needs a shared storage. - [Deleting old outputs and artifacts](https://samsung.github.io/qaboard/docs/storage/deleting-old-data/): Delete the outputs and artifacts of old commits automatically, keep those of your reference branch and milestones, delete results from the web app, and have your CI recreate missing artifacts. - [Frequently asked questions](https://samsung.github.io/qaboard/docs/faq/): Answers to the questions people ask when they integrate a project with QA-Board. ## Admin: running the server - [Starting a QA-Board server](https://samsung.github.io/qaboard/docs/deploy/): Set up QA-Board for your team on a Linux host with Docker Compose, from the results storage to the first results, then secure it, back it up and configure it. - [Connecting git hosts (GitHub, GitLab...)](https://samsung.github.io/qaboard/docs/backend-admin/git-hosts/): Connect QA-Board to GitHub, GitLab, Gitea/Forgejo or Bitbucket: tokens, push webhooks and their secrets, avatars, and upgrading older settings. - [Deployments, rollbacks and high availability](https://samsung.github.io/qaboard/docs/backend-admin/deployment/): Deploy, upgrade and roll back QA-Board with docker compose without downtime, back up and restore its database, and tune PostgreSQL. - [Running QA-Board on Kubernetes](https://samsung.github.io/qaboard/docs/backend-admin/kubernetes/): Install QA-Board on Kubernetes with its Helm chart: a quickstart on your laptop, then the results storage, database, ingress, sign-in, backups and upgrades on a real cluster. - [User Management](https://samsung.github.io/qaboard/docs/backend-admin/user-management/): Manage QA-Board's users, admins, API tokens and settings, require signing in, authenticate the results qa sends, and restrict who can read files. - [Settings and secrets](https://samsung.github.io/qaboard/docs/backend-admin/settings/): Change QA-Board's settings in Admin → Settings, without a deploy: Jenkins and git host tokens, email, sign-in... Where each value comes from, how secrets are encrypted, and the encryption key. - [Sign-in and single sign-on](https://samsung.github.io/qaboard/docs/backend-admin/sign-in/): Let users sign in with single sign-on (OpenID Connect) next to LDAP or local accounts, decide who gets an account, and fix sign-in problems. - [Project access and roles](https://samsung.github.io/qaboard/docs/backend-admin/project-access/): Decide who can see and act on each project: visibility, roles, owners, groups from your directory or identity provider, and access requests. - [Cleaning the storage](https://samsung.github.io/qaboard/docs/backend-admin/storage/): Free disk space and keep the database small from the admin panel: hard-delete deleted runs, delete artifacts and old results, scan the storage, and schedule these jobs. - [Accept results over HTTP](https://samsung.github.io/qaboard/docs/backend-admin/uploads/): Let runs send their files to the QA-Board server instead of saving them in a shared storage: turn it on, choose where files go, require tokens, set limits, and use object storage like S3 through a file system. - [Run batches from the web app](https://samsung.github.io/qaboard/docs/backend-admin/runners/): Let users start, redo and stop batches from the web app, on the server, LSF, Kubernetes or Celery workers, as themselves. - [Showing annotations from another app](https://samsung.github.io/qaboard/docs/backend-admin/annotations/): Show the regions people mark on images in another app (defects, labels...) in QA-Board's image viewer, read-only, with an annotation source: a small Python package. - [Monitoring with Sentry](https://samsung.github.io/qaboard/docs/backend-admin/monitoring/): Report the errors and traces of QA-Board's web app, backend, jobs and qa CLI to Sentry or a self-hosted Sentry. - [Usage analytics](https://samsung.github.io/qaboard/docs/backend-admin/usage-analytics/): See who uses QA-Board and how: active users, commits looked at, tuning started from the web app and the CLI, per user, group and project. Export the numbers, or query them from Metabase. - [Troubleshooting common issues](https://samsung.github.io/qaboard/docs/backend-admin/troubleshooting/): Find what's wrong with a QA-Board server: container status and logs, restarts, and what to do when the disk is full. ## Optional - [Where QA-Board is going](https://samsung.github.io/qaboard/docs/roadmap/): The principles behind QA-Board and what we're building next, with the status of each item. - [qaboard.yaml JSON schema](https://samsung.github.io/qaboard/qaboard.schema.json): The project configuration's schema, for editors and agents - [Batch files JSON schema](https://samsung.github.io/qaboard/batches.schema.json): The schema of the batch files qa batch runs - [Metrics file JSON schema](https://samsung.github.io/qaboard/metrics.schema.json): The schema of qa/metrics.yaml, which describes the metrics runs return - [Release notes](https://samsung.github.io/qaboard/release-notes/): What's new, month by month