Skip to main content

2026-10

qaboard v1.1.1

A new image viewer, readable configuration diffs, a dark theme and a commit page that stays fast with 1000 runs. QA-Board now works with GitHub and other git hosts, and comes with a quickstart, a tutorial and a user guide. Admins: read "Before you upgrade" first; at SIRC, also install `qaboard-site-sirc` when you upgrade qaboard.

Users

A new image viewer

Compare images side by side, with a wipe, one at a time in place, or as their difference, measured perceptually or in ΔE 2000. Zoom in to read the pixels' own values, even 16-bit or float, run a blind test, and rank the runs of a page by how much their images changed. Videos compare the same way, frame by frame.

Learn more →
Users

See what changed in a configuration

Runs show how their configuration differs from their reference run's, like gain: 3 → 4, and hide the settings all runs share. Lists and tuning parameters are readable, and runs are compared with the reference run whose configuration is closest. Copy a run's settings as qa options.

Learn more →
Users

Fast with 1000 runs and years of commits

On a commit with 1000 runs, filtering takes half a second instead of freezing the page for 7, and the Metrics Table opens in a third of a second instead of 7. The commit search box finds any of a project's commits, and long commit lists and the projects list load as you scroll.

Learn more →
Users

Dark theme and more room for results

Switch to a dark theme, or follow your system's. Collapse the sidebar to icons, dock the View options on the right (o), and switch commit lists to Compact. Press ? for the keyboard shortcuts and the filter syntax.

Learn more →
Users

Logs, files and reruns

Logs update live, with search, and failed LSF runs say why LSF stopped them. Browse a run's files under /s/, edit your batches in Available Tests with errors flagged as you type, and redo only the runs you filtered, with the batch's runner and concurrency.

Learn more →
Project integration

GitHub, other git hosts and a quality gate

QA-Board works with GitHub and GitHub Enterprise as well as GitLab, and receives the push events of Gitea, Forgejo and Bitbucket Cloud. qa batch sets a commit status that links to the results, you can start GitHub Actions workflows from the web app, and the experimental qa gate fails the CI when quality regresses.

Learn more →
Project integration

Get started in minutes

Try QA-Board on your computer with one Compose file and demo projects, then follow the tutorial: it shows the steps for your server, git host, CI and runners. A user guide explains the web app, and release notes pop up in it once a month is over.

Learn more →
Admins

Deploy, monitor and upgrade safely

deployments/deploy.py rolls out new backend replicas next to the old ones and rolls back instantly, a Helm chart runs QA-Board on Kubernetes, and Sentry traces requests from the browser to the database. Before upgrading, check your git host, QABOARD_STORAGE_ROOTS and QABOARD_PROXY_HOSTS settings, and your backups.

Learn more →

Web app​

Runs activity​

  • See what waits and runs now, what ended recently and the batches that start soon in Runs activity, from your profile menu or a project's sidebar: your runs or everyone's, with a timeline of the load and the busiest projects, users and queues (runs activity).
  • Group runs by batch, commit, project, user, qa batch command, runner, queue, platform, status or any job option, and open each group's runs with their job options, when they were queued, started and ended, and links to their results, log and files.
  • See the runs from before the upgrade too: they count as started when they were created, and ended after the time qa run measured them computing.
  • Spot runs pending for over a day, mark your stuck runs as failed (their jobs aren't stopped), or stop a batch, from Runs activity.

Storage usage​

  • See how much storage your runs use in Storage usage, from your profile menu or a project's sidebar: by project, batch, commit, branch and folder, the biggest first, filtered by project, branch, age or search (storage usage).
  • Paste the folder from a quota alert in Folders to see what QA-Board knows there, and open its subfolders.
  • Delete the files of many runs at once: check batches, commits or branches, or everything that matches the filters. QA-Board says how much space it frees and what it keeps (milestones, other people's runs, running runs), asks you to type a confirmation for big deletions, and shows the deletion's progress and errors.
  • Find out why IT's quota says something else, with the commands that measure a folder and find your biggest files and exports.
  • See who uses the most storage in Storage usage → Users: a leaderboard with each person's share and biggest project, for everyone's runs or a project's. Click someone to see their runs; admins then delete them. Users see other people's usage only in the projects where they're developers or owners (who sees what).
  • Show all projects again in one click when Storage usage was opened from a project.
  • Explore folders faster in Storage usage → Folders: it starts where your runs split, and skips folders that only lead to one subfolder.
  • See each batch's and commit's message, author and date in Storage usage, and each branch's latest commit. Click a project, branch or person in a row to filter by it, and a row to select it.
  • See how much space your deleted runs still take: deleting runs keeps their manifests, and the files written after they ended. QA-Board now measures what's left when it deletes runs.
  • See what QA-Board deletes automatically, when, and what it keeps, on Storage usage and on the project's commit list (e.g. Results deleted after 1 month), with what the project's qaboard.yaml says. Several rules are summed up in a line: open it to see them all.
  • Find your exports in Storage usage → Exports: their size, the runs they come from, and delete them without a terminal. Exports are links to the runs' files by default: QA-Board says how much space only frees once the runs are deleted too (exports).
  • See what you own on disk that QA-Board doesn't track, from the latest storage scan, in Storage usage → Not tracked: old exports, runs QA-Board doesn't know, what deleted runs left, other files. Delete the folders that are all yours (files QA-Board doesn't track).
  • Storage usage replaces the Quota links: QABOARD_QUOTA_URL_TEMPLATE is no longer used.

Comparing images​

  • Compare images like videos: side by side, with a wipe, one at a time in place (Single), or as their difference. Pick the mode above any image, or with the keys 1 to 4: it applies to all the images of the page, and links keep it. The swap button, or t, swaps the sides (around the difference too), or shows the other image in Single, and never changes the mode. It does it in every viewer of the page, so they all show the same one, and the keys work anywhere on the page, not only over an image. See comparing images.
  • Compare an image with another file of the same run, like its ground truth or the input, side by side or as their difference: pick it in vs reference ▾ above the image, where the likeliest files come first. Every run of the page then compares with its own file of that name, or only the run you're looking at with Apply to: This run, and c switches back and forth with the reference. The pick is for the page: another commit or batch compares with the reference again. See comparing with another file.
  • Run a blind test: Blind test above the images (or h) shows each viewer's images in a random order, called A and B everywhere, then Reveal shows which was which, and Done goes back to new first. See blind test.
  • See the difference between the new and reference images, all three in sync, with a cross on the pixel under your mouse in each, measured perceptually (YIQ), as ΔE 2000, per channel or as any change at all. With the images next to the modes, Difference again, or 4 again, shows it alone and larger. See the difference.
  • Zoom in and read the values of the pixels, written on them: the image's own values (10-bit, 16-bit, float, raw...) or the 8-bit colors on screen, picked from the 123 button above the images. Hover a pixel to see both, in both images, and their difference; p hides them. 8-bit images show one set of values: the screen shows them as they are. See pixel values.
  • Compare images in full screen: the button at the right of the image viewer's toolbar, or f, gives the screen to the images, with all the viewer's tools, and the difference under them when that makes them larger. See full screen.
  • Adjust how images look with brightness, contrast, gamma, sharpening, one channel (R, G, B or luma), false colors or inverted colors. They apply to both images and to the same visualization of the other runs; the pixel values, the difference and the histograms stay the images'.
  • Plot the row and column profiles of the view next to its histogram: the average of each column and row, with a ±1 std dev or min–max band, or the values along the row and column under the pointer. Hover a profile to see where it is on the images. The analysis button, in the toolbar above the images, opens a panel with three views, and remembers the one you picked last (a shows or hides it). See profiles.
  • Analyze raw images by channel of their sensor: pick Image values in the analysis panel to get the statistics, row and column profiles and histograms of R, Gr, Gb and B, or of each sub-pixel, on the image's own values at full resolution. The card above shows the color filter array with a diagram, and where each value comes from: the image's metadata, guessed from the pixels (Bayer, Tetra, Nona… up to 9×9 cells), or what you set. Set the pattern and the black level once per project: your browser remembers them. See raw images.
  • Read the statistics of the view per channel and image: mean, standard deviation, SNR, min, percentiles, max and the shares clipped at 0 and 255, with the differences between the images. Copy the table to paste it in a spreadsheet. See statistics.
  • Highlight a range of values on the images: drag on the histogram and stripes show the pixels in the range, with the share of each image they cover. Click R, G, B or Y above the histogram to show one channel alone and pick its values. The channels' areas blend where they overlap, and the axis has ticks. See histograms.
  • Find your way in large images: a minimap shows where you are once you zoom in, the zoom level says how many screen pixels an image pixel takes, 1:1 goes to 100%, and the histogram button overlays the histograms of the new and reference images in view.
  • Step through the regions of interest with their thumbnails above the images, or with [ and ]. The regions menu finds where images differ most, by perceptual difference, SSIM or ΔE 2000: one region per place that changed, with how much it changed (e.g. #1 · 3.2%). The region set you pick applies to every viewer of the same file on the page, and searches like Where they differ only run on the page where you pick them.
  • Copy or save the new, reference or difference image as you see them, or open the original files, from the copy button above the images.
  • Find which runs' images changed most: Where they differ, in all N runs… in the image viewer's regions menu compares every run of the page with its reference and ranks them, from the largest difference to the identical ones. Show takes you to a run, on the region that changed most. The list says which runs it's comparing and for how long, and which wait for their turn. See which runs changed most.
  • Pick how images are compared in the View options (Compare images), which replaces the Image diff switch. Links made with Image diff on open in difference mode.
  • Use QA-Board's image viewer in your own React apps: it builds as a package (npm run build:image-viewer), with its props documented in its README.

Videos​

  • Compare videos with a wipe, one at a time in place (Single), or as their difference, besides side by side. Zoom with the mouse wheel or a pinch and drag to pan: both videos follow, and zoomed in you see the actual pixels. See comparing videos.
  • Step through videos frame by frame with the arrow keys, scrub, slow down to 0.1× or loop: new and reference stay on the same frame, and wait for each other when one is still downloading.
  • Copy the current frame of a video (new, reference or their difference) to the clipboard, or save it as a PNG.
  • Compare videos like images: the difference between the videos or alone, perceptual or in ΔE 2000, display adjustments, a blind test, full screen (f) and a minimap when zoomed in.
  • Highlight a range of values on the videos, like on images: drag on the histogram of the frames, and pick its channels with R, G, B or Y. See histograms.
  • See which frame each video shows when paused, numbered like in the file even with variable frame rates, and whether both show the same frame. See which frames you are looking at.
  • Read the decoded values of the pixels (e.g. 10-bit Y′, Cb, Cr) of the very frame shown, under the pointer and written on the pixels when zoomed in, next to the 8-bit values your browser displays.
  • See what's in the video files side by side (codec, pixel format, chroma subsampling, color range and matrix, frame rate, keyframes, dropped frames), with what differs highlighted and what it means for what you see.
  • Find where two videos differ: the timeline shows how much each frame differs, n and N go to the frames that differ most, and regions show where and when they differ most, in the whole video or in the frame shown. See where the videos differ.
  • Change the speed of videos with < and > (they were [ and ], which now step through regions, like in the image viewer).

Point clouds and trajectories​

  • See point clouds (*.pcd files) in 3D: the new run's, the reference's and a ground truth on the same axes, each in its own color, that you can show or hide, orbit, pan and zoom. See point clouds and trajectories.
  • See 6DoF trajectories across the run's card, with the path in 3D next to the plots of each axis, at the same scale on every axis.

Test reports​

  • See a run's JUnit XML test report (pytest, GoogleTest, CTest, Maven, Gradle...) as a tree of its tests, compared with the reference's: new failures, fixed, added, removed, newly flaky and skipped tests at a glance. Click the counts or the changes to filter, search tests and messages, press n for the next failure, and read each failure's message, trace and output next to the reference's. Files named junit.xml or TEST-*.xml open in it (test reports).

Configurations and reference runs​

  • See at a glance what's specific to each run: settings that all the runs shown share are hidden behind "··· N shared" tags, and listed once at the top of the page. It follows the filter. Turn it off with Hide them in each run (reading configurations).
  • Read lists in configurations: [1, 0.98, 0.95], long lists with their size (… 64 values, 3×3) and where they changed (…0.619, 0.7 → 0.635, 0.651…). Tuning parameters show as settings like the others, with a wrench.
  • Copy a run's settings as qa options (-c … --tuning …), as JSON or as a tuning search space, and a setting as text or JSON, from their hover menu.
  • See how a run's configuration differs from its reference run's: added settings in green (+), removed ones struck through in red (−), changed values as gain: 3 → 4, and only the changed part of strings: conf/[day → night].yaml. Changes all runs share are listed once, at the top of the page. Hover the Config tag for a git diff-like view.
  • Runs are compared with the reference run whose configuration is closest: the fewest different settings, then the most similar values (base-v1 is closer to base-v2 than to other). The Config tag says when other reference runs were as close.

The commit page​

  • The commit page's view options are now a panel docked on the right that makes room for the results instead of covering them. Collapsed, a rail shows a dot on the sections that differ from the project's defaults, and hovering lists the changes. Press o to show or hide it, or drag its edge to resize it. See View options.

  • See and change what the commit page shows from Files to show (formerly Dynamic options): each option says which visualization it's for and lists the values of all runs. Options are linked across runs: a run that doesn't have the selected value shows the closest one and says so, and you can pick a value per run. See the user guide.

  • Open the export box from the new Export button next to the Visualizations and Output Files titles; press Enter to export, and find the last export's paths when you reopen it.

  • Run cards fit the window when qaboard.yaml doesn't set their width (outputs.style.width): they used to be 1500 px wide, wider than a laptop's screen.

  • Metrics Diff colors small changes and shows them precisely ("-0.37%" instead of "-0%"). The Summary's bars say how many runs met their metric's target.

  • Find more explanations on hover: run statuses, aggregated metrics, new/ref, targets in the metrics tables, and milestones. The reference's bar says when it's the same commit as the new one.

  • Pages use the width next to the sidebar instead of jumping between fixed widths, and the commit page fits laptop screens: its header cuts long commit messages and branches with "…".

  • The Tuning Analysis view of a qa optimize experiment plots its convergence itself: each iteration's objective and the best one so far, even when matplotlib isn't installed where qa optimize ran.

  • The commit page tells you when a commit's artifacts were deleted (when, and by whom) or are incomplete, what's missing, and lets you recreate them with your CI.

  • Compare with several references, one at a time: hover the reference's bar and Stack a reference (a commit, a branch, a milestone, or the same one with another batch or filter), then switch with their tabs or r / Shift r. Drag tabs to reorder them, right-click them or click their ⋯ to duplicate, rename or remove them. They are in the URL. See several references.

  • See each stacked reference under one name, in its tab, the reference's bar and the viewers: the first is "ref", the ones you add "ref 2", "ref 3"..., copies are <name> copy, and they keep their names when you reorder or remove references. Tabs say only what tells them apart: their batch or filter, their commit or branch, when the others' differ. Many tabs scroll sideways, with the mouse wheel too. See several references.

  • Hover a reference's tab to see its commit, message, author, branch, batch, filter, and how many runs it has and how many failed.

  • See everything you compare in the tabs: the top bar's results come first, then "vs" the references, and the two in the bars are lit. Swap the bars from their tabs' menus, or by dragging the first tab on a reference: the same comparison, reversed, and nothing is replaced. "Switch new/reference" is now Swap the bars.

  • Name what you compare: click new or ref to call them ISP v2 and ISP v1. The names are in the URL, and the page uses them: the images and videos' labels, the legends of plots, text diffs, flame graphs, HTML outputs, the Summary and the columns of Metrics Table and Metrics Diff. Milestones are called by their name, and you can rename them on the page too: the milestone's name then shows next to its star.

  • See which is which by color: new is blue, references purple, magenta, teal and violet, in their bars, tabs, the viewers' labels, and the plots (the Summary's box plots, overlaid Plotly plots), in the light and dark themes. Results keep their color when you reorder, switch or remove references, or swap the bars, and a color freed by a removed reference goes to the next one you add. The history, the dashboard and the artifacts' files use the names and colors of what you compare too. The reference is no longer orange, a color that says "warning".

  • Errors from redo, delete and tuning actions show a readable message instead of the raw server response.

  • When you are logged out, actions that need a login (redo, delete, rename, tuning, restoring artifacts...) are greyed out next to a Log in button, instead of failing with an error. If your session expires, one message asks you to log in again.

  • Filter output files from one line above them: File options (show all files, expand all folders, color-blind friendly...) moved to the View options, and the button says which are on. See Output Files.

  • See where a dragged reference tab will go: a bar marks the gap between the two tabs, and the tabs' row says "Move ref 3 between ref and ref 2".

  • Open an input's History from its run's menu, or by hovering its name: the name is no longer a link you click by mistake.

  • See tuning parameters in full in the run cards: they are never hidden behind "N shared" or cut short, and come first in the Logs view's one-line summaries.

Logs, batches and redoing runs​

  • See how long each run ran in the Logs view, and find what you know about it in its ⋯ menu: its input's History, batch, input path, who ran it, its compute time, date and storage. See Logs.
  • The Logs view filters runs (all, failed, running) and tags failed runs with the reason (e.g. "LSF: out of memory"). The sidebar's Logs entry shows how many runs failed, or are still running.
  • Show only the runs that succeeded: with Successful in the Logs view, or with status:success in the commit page's filter boxes. See the filter syntax.
  • In the Logs view, each run fits on one line, with buttons to open its output directory, copy its Linux or Windows path, redo it, mark it as failed or delete it. Its full configuration shows when you open its logs.
  • Switch between a run's log.txt, log.lsf.txt and log.dask.txt. For LSF runs, a summary shows the exit reason, host, run time and memory used vs requested, with hints that point to the batch's lsf: options.
  • Batches started from QA-Board (tuning, "Run Tests") show whether qa batch is waiting for LSF, running, failed (with the error) or didn't start any run, with a link to its logs.
  • See where the logs of each command that started a batch are: in the CI job (with a link), or in QA-Board (one click shows them). The batch's own log.txt only shows when it exists. See the user guide.
  • Show more of very long logs: they show their last 50,000 lines to stay fast, and say if the hidden lines have errors or matches for your search.
  • When you filter a batch's runs, the commit bar menu redoes only the runs that match the filter. Follow redone runs in the Logs view, under "Started from QA-Board". See the user guide.
  • The Available Tests view is a real editor for your private and shared batches: it flags YAML mistakes and inputs qa batch can't use as you type, shows whether your changes are saved (Ctrl+S), keeps them if you leave, and warns instead of overwriting someone else's changes. See the user guide.
  • Available Tests warns about what qa batch would ignore or misread: unknown keys like inpts (Did you mean "inputs"?), values of the wrong type, unknown runners and runner options, and input types the commit's qaboard.yaml doesn't define. Values defined at the top level and reused in batches with YAML anchors (database: *lab-db) aren't flagged as batches. See the user guide.
  • The Available Tests view lists the batches in each file: go to one, list the tests it resolves to on the commit, or run it in one click.
  • When a batch has no tests in Available Tests or Run Tests / Tuning, see why: a batch file missing from the commit's artifacts (with the path QA-Board looked for), invalid YAML, a batch that no file defines, or inputs the server didn't find. See the user guide.
  • Redoing runs or starting tuning on a commit with missing artifacts first brings them back, instead of failing or running from a broken folder.
  • Redo buttons work again on commits whose artifacts were deleted.
  • The tuning form checks your settings while you type, and lists what would make the batch fail (unknown batch, invalid automated tuning configuration, unknown platform...) before you start the runs.
  • The tuning form warns when the commit's artifacts are missing and will be restored first.
  • See which batch files a batch started from the tuning form came from: the Logs view names them next to its qa batch command, e.g. from qa/batches.yaml, and says which file defines each batch when you open it. When several files define a batch, they're listed in order: the last one wins. See the user guide.
  • Get an email when a batch you start from the tuning form ends: one email when its last run ends, with how many runs failed, the main metrics vs their targets, the best parameters and links to the failed runs' logs. See the user guide.

Finding commits and projects​

  • See your own work first: signed in, a project's page lists the batches you started (from the web app or your terminal), the latest first, with their progress, and where the reference branch is. Everyone and the N commits by everyone count show everyone's, and the page remembers your choice (yours or everyone's).
  • Hide the CI's batches from commit lists with Hide the CI, to see what people ran from the web app or their terminal.
  • Compare two batches from a project's Batches list: tick them, then Compare.
  • Add the other emails of your git config in Settings → Profile, so that your commits count as yours.
  • Search all of a project's commits from the commit list's search box, not only those in the date range: by message, committer, branch, batch or hash. Use branch:, committer:, batch:, message: or id: to look at one field, and -word to exclude (searching commits).
  • Jump to a branch from the commit list's search box: when your search can be a branch name (one word, or a branch: term), it suggests the most recently active branches that match, even in projects with thousands of branches.
  • Find commits by where their batches come from or who started them: origin:webapp, origin:local or origin:ci, and user:kaito in the commit list's search box (searching commits).
  • Scroll through long commit lists: they show the 50 latest commits, then more as you scroll, so big projects open much faster, and Older commits at the end looks a week further back. Branch pages still get their whole date range at once (up to 1000 commits), for their metrics chart.
  • Switch commit lists to Compact to see one commit per line, in aligned columns, with the full message, metrics and other batches on hover. The web app remembers your choice.
  • Open the projects list much faster on sites with hundreds of projects: it downloads less, and shows more projects as you scroll.
  • Start from a home page about you: signed in, / shows your projects, your batches running now with their progress and logs, those with failed runs this week (dismiss them once looked at), your recent batches with their main metric, and on demand what your teammates ran in your projects. New users see how to get their runs there (your home page).
  • Browse projects by their path in Explore, at /projects (the home page when you're signed out): subprojects go under their repository, projects that share a prefix under a folder, and each card shows when people last used the project and its runs over 30 days. Open and close groups: the web app remembers them (pick a project).
  • See only the projects people work on: those that only the CI ran in 90 days, like subprojects whose tests nobody looks at, fade under N quiet projects. The filter still finds them, and pinned projects never fade (quiet projects).
  • Go anywhere with the keyboard: press Ctrl+K (⌘K on macOS) or / on any page, or click Search…, then type to jump to a project, your recent batches, a page like Settings → Access tokens, or an action like switching the theme. In a project, it also finds its branches and goes to a commit from the start of its hash (search from anywhere).
  • Go to a project's subprojects from the Subprojects card above its commits, and to its parent projects from the breadcrumbs above its name in the sidebar.
  • Know right away when a link or a typo leads nowhere: you get a Page not found instead of an empty project, with a link to the closest project above the URL, or the projects below it when it's a folder.
  • Find your way around from the History page: the sidebar shows the project's navigation (the branch's commits, Code, History, Integrations, Milestones), and moving between these pages no longer reloads the app.
  • Add up a metric over a batch's runs: the charts over commits, on History and above a branch's commits, offer sum next to median and average, e.g. to follow the number of tests of all your test suites (History).

File browser​

  • Open a folder under /s/ in your browser to get the new file browser: sort and filter files (e.g. *.png), copy their Linux or Windows paths, paste a path to go there, and navigate with the keyboard (press ?). Scripts and curl still get nginx's plain index.
  • In the file browser, a bar shows the run whose folder you are in, with its status and a link to its results: redo it, delete it or mark it as failed from there.
  • Sign in from the file browser with any of your server's sign-in methods, single sign-on included: Sign in opens the sign-in page, then brings you back to the file.

Git hosts and integrations​

  • Follow links to code, commits and branches on your project's own git host, with the right URLs for GitHub, GitLab, Gitea/Forgejo and Bitbucket, instead of a hardcoded GitLab server.
  • The sidebar's Code menu links to the commit, its files, your subproject's folder, the branch and the repository on your git host.
  • Start the GitHub Actions workflows declared in your project's qaboard.yaml from the Integrations menu, follow the run you started, and see the status of each workflow's latest run for the commit. Only users who can access the project can see or start them (GitHub Actions).
  • See your git host's defaults in the Integrations menu of branch pages, when your project configures none and QA-Board knows its git host: GitLab's pipeline status, or a link to GitHub Actions (default integrations).
  • See commit authors' avatars from GitHub, Gitea/Forgejo and Bitbucket, learned from push events, or else their Gravatar. Avatars and badges from public git hosts load directly, which is faster.
  • The Jenkins integration accepts params (what the server used) and parameters (what the docs said), and no longer fails without them.

Look and feel​

  • Switch to a dark theme with the theme button next to "What's new" in the sidebar, the projects list or the file browser. By default the web app follows your system's theme; plots, editors and viewers follow it too, and metrics tables only highlight clear improvements and regressions.
  • Collapse the sidebar to a rail of icons for more room, with Collapse at its bottom or Ctrl+\ (⌘\ on macOS). Hover an icon for its name; the web app remembers your choice.
  • A calmer, more consistent look: every view's title sits above its flat card, configurations are grey tags, run statuses are small icons, and filters have their help and a clear button inside the box. Hover a view in the commit page's sidebar to see what it shows; your login moved to its bottom.
  • Pages say what's happening instead of staying blank or showing empty views: while they load, when they fail (with a Retry button), and when there's nothing to show yet, with what to do next and a link to the docs. For example, a commit without results explains how to get some, and an empty Metrics Diff says why there is nothing to compare.
  • Deleting runs, batches or a commit's results, redoing a whole batch and resetting the tuning form ask you first, with how many runs are affected. Notifications name what was acted on, and starting runs offers to show their logs.
  • The commit, batch and run action menus open on click instead of hover, so they no longer open by accident and work on touch screens. When a run has no reference and there's no Windows path to choose, its folder and copy buttons act directly.
  • The footer stays at the bottom of the window, with links to What's new, the cheat sheet and QA-Board on GitHub.
  • The web app says "runs" everywhere, like the docs ("8 runs", "Redo failed runs"), and page titles match the sidebar's names: Metrics Table instead of "Quality report", Metrics Diff instead of "Improvement report", and the Analysis view is now Tuning Analysis.
  • See who made a commit at a glance: when there's no avatar picture, committers, users and projects show their initials on a color that stays the same everywhere.
  • Buttons that were only icons have labels for screen readers, and the menus they open can be reached with the keyboard.
  • The "Quotas" links show only when the server has a quota dashboard, and the error page links to your server's support page.

Milestones​

  • Save milestones in your account: they follow you on every computer. The star next to a commit's hash opens a dialog that shows what you save (the commit, its batch, the filter and how many runs match), suggests a name, and asks for an optional group, notes, and whether everyone on the project sees it. See milestones.
  • Organize milestones in groups, like releases or tuning/denoiser: menus list them by group, and remember the groups you collapse.
  • Find milestones faster: the Change menu and the sidebar's Milestones menu list your favorites first, then your milestones, then the project's, with a search box that matches their name, group, notes, branch, batch, filter or owner.
  • Add milestones to your favorites with the heart: the sidebar lists them, one click away.
  • Share a link to a milestone, /<project>/milestone/<id>-<name>: it opens its results, and keeps working when the milestone is renamed. Add ?vs=<id>-<name> to link to a comparison of two milestones.
  • Manage milestones in a table: sort them by update, creation, name or group, open one, use one as the reference, compare two, or move, share and delete several at once. See managing milestones.
  • See when each milestone was last updated, on the right of the milestones menus, to find your way in long lists. Manage milestones… is now first in the menus.
  • Add a whole group of milestones to your favorites: its milestones, even future ones, are listed first.
  • Find milestones by dataset: milestones remember the datasets, qa batches and platforms their runs used. Search dataset:indoor or qa-batch:nightly, or pick a dataset in Manage milestones…, which also shows each milestone's notes.
  • Find out what happened when a milestone link is broken: the page says which milestone was deleted, and still opens the other one of a comparison.
  • Save several milestones on the same batch, for example one per filter. When the results you look at are a milestone, its name shows next to the star.
  • Milestones you saved in your browser before (the "private" ones) move to your account the next time you log in with that browser.

Accounts, tokens and admins​

  • Create API tokens in Settings → Access tokens, with only the scopes they need (read, send results, act), for some projects only, and with an expiry date or none. The token is shown once, with how to use it (APIs).
  • See when and from where your tokens were last used, and revoke them. Settings → CLI sign-ins lists the logins of qa auth login, and Activity your recent sign-ins and token changes.
  • Approve qa auth login from any browser: open the link it prints and check the code.
  • Keep your preferences (theme, list density, favorites...) from browser to browser and computer to computer when you're signed in.
  • See who really sent a run: runs sent with a token show it, the others can be marked as unverified.
  • Admins: see users, their tokens and last sign-ins, disable accounts and make admins, revoke any token, read the audit log and change settings in the new Admin panel, with a security checklist (user management).
  • Read why you can't do something: errors say what's wrong and what to do, e.g. that a token expired or lacks a scope.
  • Sign in with your company's single sign-on, when your admins set it up: its button comes first on the new sign-in page.
  • See where you're signed in in Settings → Sessions, and sign out of the browsers you don't use anymore, or of all of them but this one.
  • Add another way to sign in to your account, or remove one, in Settings → Sign-in methods.
  • See your role on a project in its new Access tab, and ask for access when you need more: owners get an email, and you get one when they decide (project access).
  • Choose who can see your project and act on its runs, as its owner: make it public, internal or private, and add users or groups as viewers, developers or owners, optionally until a date. Members of a group's projects are members of every project under it.
  • Read why you can't see a project or act on its runs, with its owners and a Request access button.
  • Give your CI a token that isn't yours, as a project's owner: create a service account in the project's Access tab, choose its role, and create its tokens. They only work on the project, keep working when you leave, and runs show they were sent by group/repo/ci[bot] (service accounts).
  • Ask for access: when your admins approve new accounts, you can sign in and say why you need QA-Board, and you get an email when they decide.
  • Read the release notes in the new "What's new" panel (/#whats-new opens it). Its search box shows only the highlights and changes that match, under their section headings: search "video" to see what changed in the video viewer. It pops up once a month is over; the current month's notes show as a preview, tagged "In progress".
  • The sidebar's help button opens the new user guide.
  • Press ? for a cheat sheet: keyboard shortcuts, QA-Board's words, the filter syntax and links to the docs. The sidebar's ? button opens it too.
  • The URL keeps everything you select (date range, metrics shown, highlighted commits, filters, view and plot options), but only what differs from the defaults, so links stay short. Share it, reload, or use the back button: you get the same view. Typing filters no longer adds entries to the browser's history.
  • Link to History with or without the breakdown per test, or relative to the first commit: ?breakdown_per_test=false and ?relative=true in the URL now work.

CLI and project setup​

Signing in​

  • Sign in from the CLI with qa auth login: open the link it prints in any browser, even when qa runs over SSH, and approve the code. The token is saved per server in your config folder, never in your projects, so it works in every checkout and worktree. qa auth status, qa auth logout and qa auth token (signing in).
  • Set QABOARD_TOKEN in CI to send verified results (CI tokens). QA_TOKEN and ~/.qaboard/secrets.yaml still work.
  • Read why the server refused your token (expired, revoked, missing a scope...) and what to do, instead of a stack trace. The advice fits where you are: in CI it's about QABOARD_TOKEN, never about signing in from a browser.
  • CI jobs fail when QA-Board refuses their results, before running anything, instead of passing with nothing saved.
  • qa auth logout only revokes the tokens qa auth login created: a token you saved with --with-token (e.g. your CI's) keeps working, unless you add --revoke.
  • qa auth status says where the server's address comes from, and lists the other servers you're signed in to. A wrong QABOARD_URL or a server that's down is reported as such, not as a bad token.
  • qa gate --server no longer sends your token for the default server to another server.
  • qa batch says when the server won't show it the batch's runs (e.g. it requires signing in), and stops instead of running them all again when you asked to skip, sync or post-process the existing ones (--action-on-existing).

Project setup and the qa CLI​

  • At SIRC, upgrading qaboard now requires also installing qaboard-site-sirc, see the installation instructions. Site packages provide a base qaboard.yaml that every project's configuration is merged on (site defaults): qa init shows the inherited values instead of overriding them.
  • qa init creates a configuration that runs locally: LSF is commented out, and storage defaults to /mnt/qaboard like the server.
  • The CLI reads your QA_TOKEN from $QA_TOKEN or ~/.qaboard/secrets.yaml, and warns if other users can read that file.
  • QABOARD_API_VERIFY controls the TLS verification of the CLI's requests to the server: true (the default), false, or the path to a CA bundle.
  • Site packages (company-specific plugins) are no longer ignored on Python 3.8 and 3.9, and the package includes the qatools compatibility package.
  • qa save-artifacts saves the subprojects' qaboard.yaml: without it, runs redone from the artifacts used the parent project's configuration and were saved in the parent project.
  • Get an email when the runs of qa --share batch end (export QA_NOTIFY=none to stop them). It's never sent in CI, and batches started by other batches, like pipeline steps, count as part of the first one. See the CLI tips.
  • See the link to your results when qa batch ends, not only when it starts.
  • Tell your users how to get storage for new projects from your site package, with QABOARD_STORAGE_HELP (docs).
  • qa tells QA-Board where it runs: from the web app, your terminal, or a CI pipeline (GitLab, GitHub Actions, Jenkins, Buildkite…, with links to the pipeline). QA-Board uses it to show you your own runs and leave out the CI's. Older versions of qa keep working: QA-Board guesses as before.
  • metadata(context) in your entrypoint gets the run's context, like run(context): filter batches on the configurations, the platform or the batch's name. metadata(absolute_input_path, database, input_path) still works. See Input metadata.

Runners: LSF, Dask and Windows​

  • LSF runs write their output live to log.lsf.txt, with LSF's job report at the end. qa batch no longer passes an LSF queue or project when none is set, and fails fast when bsub is missing.
  • With --no-wait and an existing Dask cluster (scheduler_address), the Dask runner now enforces concurrency: a driver LSF job sends the runs, as without scheduler_address. See the Dask runner.
  • qa batch --runs-file runs.json does a list of runs, each with its input, configurations, parameters and runner options. QA-Board uses it to redo runs together. See the CLI tips.
  • Read the logs of qa --share batch and qa optimize started from a terminal in QA-Board, with the batch's commands: they save what they print in the batch directory (under qa-batch-logs/), errors included. qa optimize includes the output of its iterations. In the CI, the CI keeps the logs. See the CLI tips.
  • Configure the Windows runner: PowerShell's $ErrorActionPreference per job (the batch's error_action: option) or with QA_RUNNER_ERROR_ACTION, and how long qa batch waits for its Jenkins builds, now 2 hours by default (QA_JENKINS_TIMEOUT, QA_JENKINS_STATUS_INTERVAL).
  • Batches list their inputs in the same order on every machine: the inputs found by globs are sorted by path, instead of coming in the order the filesystem lists them.
  • The Windows (Jenkins) runner sends your token. Without one, the server only starts the Jenkins jobs listed in your qaboard.yaml (docs).
  • Keep Celery passwords out of git: the URLs of runners.celery in qaboard.yaml expand environment variables, like ${QA_CELERY_PASSWORD}. Admins can give each team its own RabbitMQ account (sharing the broker).

GitHub, CI and the quality gate​

  • Fail the CI when quality regresses, with the experimental qa gate: set rules in qaboard.yaml (gate:) on the median of a metric vs a reference branch, milestone or commit, on each test, on failed runs or on bit-accuracy. It prints a table (or Markdown, or JSON), fills the GitHub Actions job summary, and can post a commit status to GitHub or GitLab that links to the Metrics Diff. See Quality gate.
  • Gate pass/fail metrics with aggregate: on qa gate rules: {metric: unsafe_actions, max: 0, aggregate: max} fails as soon as one test is unsafe, and aggregate: mean checks a pass rate, where the median said little (quality gate).
  • Compare with any milestone in qa gate: --reference '#12' (or gate.reference) uses the milestone whose link is /<project>/milestone/12-..., shared or not, and the gate compares with the milestone's batch (quality gate).
  • qa batch sets a commit status that links to the results on GitHub too (CI integration).
  • qa check-bit-accuracy waits for the GitHub checks of the reference commit too, like it does for GitLab pipelines (bit-accuracy).
  • qa check-bit-accuracy asks QA-Board where the reference's runs are, instead of guessing their folders. Offline, or for runs QA-Board doesn't know, it looks in the storage, in every user's folder when your storage settings have a {user}, and says where it looked when a run is missing (where runs are found).
  • Compare with the reference's closest runs: qa check-bit-accuracy --reference-match closest picks, on each input, the reference run the web app would show, with the closest configurations, tuning parameters and platform, and says how it differs (options).
  • In GitHub Actions, qa records the pull request's branch and last commit instead of the temporary merge commit, and the tag of tag builds (CI integration).
  • qa finds your git host, its API and its token in QABOARD_GIT_HOSTS too, and sends GITHUB_ACCESS_TOKEN only to github.com and the GitHub hosts you configured (git hosts).
  • In CI, qa's Sentry reports are tagged with its version, and QABOARD_SENTRY_VERIFY (or SENTRY_CA_CERTS) can be the CA bundle that signed your Sentry server's certificate, or system (the qa CLI).

qaboard.yaml​

  • Get completion, docs on hover and warnings for typos when you edit qaboard.yaml, batch and metrics files, in VS Code or any editor using the YAML language server: QA-Board publishes their JSON schemas, and the files qa init creates declare them (project setup).
  • Set the frame rate of a video visualization with fps in qaboard.yaml, when the web app's guess is wrong. See the video viewer.
  • Warn users who start tuning on branches whose commits have no artifacts: list the branches your CI saves them for in qaboard.yaml's tuning.branches. The tuning form offers the platforms listed in inputs.platforms (tuning from QA-Board).
  • Summarize metrics per tag of the inputs (their metadata's tags) on the History page, with dashboard_breakdown_by_tag: true in your metrics file (metrics).
  • Add recreate_artifacts to qaboard.yaml to have QA-Board rebuild a commit's deleted artifacts with a GitlabCI job, a Jenkins job or a webhook, when users redo runs or start tuning. Pass who asked (${user.email}, ${user.full_name}) and the commit (${commit.short_id}, ${commit.message}...) to your CI (docs).

Server and administration​

  • Stop a deploy when a database migration fails: the output says which migration and why, and nothing running is changed. Migrations used to be skipped silently instead, leaving a database the backend then failed on, with errors like column … does not exist. The backend also doesn't start on a failed migration. If your database misses migrations because of this, find the latest one it has, then alembic stamp <revision> && alembic upgrade head in the backend's container (deployment).
  • Upgrading saves where batches come from and who started them, and who wrote commits (their email). The migration fills in existing batches a few thousand at a time, so results keep coming in meanwhile: expect a few minutes on big databases. Results sent with a service account's token always count as the CI's.
  • Upgrading adds a project_views table (who opened which project, written at most once an hour per user and project), for the projects you look at. /api/v1/projects also says when people last used each project, its batches per day, and how many users viewed it, and to signed-in users when they last ran or committed there (my_latest_activity).
  • /api/v1/batches lists the batches of several projects (projects=a,b, skipping those you can't read), others' batches (scope=others), those with waiting or failed runs (status=pending|failed, over the last 30 days by default, 90 at most), recent ones (since_days=), and each project's default metric (metrics=default).
  • Upgrading adds an index of the failed runs by batch (small: few runs fail), built without blocking writes: the home page finds your batches with failed runs without reading all their runs.
  • Users, tokens and admins: an admin panel, settings in the database, and an audit log (user management).
    • Make admins with QABOARD_ADMINS, from the admin panel, or with qaboard-admin create-admin in the backend's container. Approve or disable accounts waiting for an admin.
    • Change settings from Admin → Settings without restarting the server: requiring sign-in, local sign-up, token lifetimes, how results are authenticated. Environment variables still win when they're set.
    • Require tokens to send results: ingest.auth (or QABOARD_INGEST_AUTH) goes from off (the default) to warn (qa asks users to sign in) to token (results sent by qa).
    • Find sign-ins, failed sign-ins, token changes, CLI logins and admin changes in the audit log, with who and from which IP.
    • Tokens are stored hashed, can have scopes, projects and an expiry date, and expire when unused for long if you want.
    • Disabled accounts stop working at once: their sessions and tokens are refused at the next request.
    • Create service accounts for the whole server in Admin → Service accounts: bots with their own qab_svc_… tokens, a role on all projects or none, that projects' owners add to their projects. Projects' owners create their own from the Access tab (service accounts). Upgrading adds two columns to users.
  • Roles on projects (viewer, developer, owner), managed by projects' owners from the web app (project access and roles). Nothing changes on upgrade: projects are public and signed-in users developers, until you set projects.default_visibility and projects.signed_in_role.
    • Create groups in Admin → Groups, filled by hand or by sign-in claims: SSO groups, LDAP memberOf, email domains.
    • Give users or groups a role on all projects, private ones included.
    • Projects in QABOARD_LOGIN_RESTRICTED_YAML stay restricted: they're private, the users its rules match their developers.
    • Files of private projects under /s/ follow their roles, and tokens limited to other projects can't read them.
    • Users' groups are updated every day, and as soon as you change a rule: LDAP accounts from the directory's memberOf (optionally with Active Directory's nested groups), single sign-on accounts with refresh tokens if you enable Refresh groups daily on the provider. Update now in Admin → Groups runs it at once.
    • Rules can read nested claims, like Keycloak's realm_access.roles, and signing in with LDAP or single sign-on keeps the groups the other gave.
  • Clean the storage from the admin panel (cleaning the storage).
    • Delete for good the runs users deleted, in Admin → Storage: choose projects with patterns (group/*, with exceptions) and the runs' dates, count them, and start a dry run first. Milestones are never deleted.
    • Purge the runs whose folder was deleted outside QA-Board (quota scripts, users...): only their metadata is deleted. Runs are never purged from a folder missing right under a storage root, in case the storage isn't mounted.
    • Delete commits' artifacts: by project and by how long commits went without runs, keeping milestones' artifacts, the reference branch's, and what you list (groups, paths, globs).
    • Follow each job's progress and log, and see every error with the file and its owner, grouped by kind, owner and project, or download them as CSV. Errors don't stop the job: the runs are kept and the next job tries again.
    • Schedule jobs with cron expressions in your time zone, instead of cron jobs on the server: hard deletes, and Delete old results (like qaboard_clean), with a default age for the projects whose qaboard.yaml says nothing.
    • Get an email when a scheduled job fails or ends with errors, with the first errors and a link to the job: admins get it by default, jobs.failure_email_to sends it to others, jobs.failure_emails turns it off (schedule jobs).
    • Get a Sentry issue when a job fails, ends with errors or is given up, one per kind of job, and when the jobs worker itself runs into errors (Monitoring with Sentry).
    • Choose who sees other people's storage usage with storage.others_visible_to: by default, admins and each project's developers and owners from a grant, a group or qaboard.yaml, not every signed-in user.
    • Users' storage usage is kept up to date by the worker, and rebuilt nightly by a Storage usage schedule it creates (QABOARD_STORAGE_USAGE_AUTO=0 to manage it yourself). A migration adds the storage_usage table, and an index on ci_commits.latest_output_datetime, built without blocking writes.
    • Scan the storage to find what QA-Board doesn't track, in the projects' folders or any other: untracked runs and exports, what deleted runs and artifacts left, other files, with their owners. Explore the results like ncdu: folders by size with what they're made of, or the biggest untracked things anywhere, by owner or age. Delete them, track the exports so their owners see them, or add the runs back to QA-Board. What QA-Board uses can't be deleted from there (scanning the storage).
    • Cancel a scan in seconds, even deep in a huge folder, and follow a running scan by the folders it has read.
    • Delete old scans' results from Admin → Storage → Scans or the scan's page (the files stay on disk). The list shows the latest scans, the older ones on demand.
    • Stop a job stuck cancelling with Stop without waiting, e.g. when its worker is stuck on a storage that doesn't answer: another job of its kind can then start.
    • See the schedules in a line (how many are enabled, the next one, those whose last run went wrong) when there are many, and their table when you open it.
    • Put the exports in one place with the storage.exports_root setting (QABOARD_EXPORTS_ROOT): <root>/<user>/<project>/<date>-<id>, instead of a share/ folder next to the commit's results. QA-Board now tracks exports made from the web app, and their size.
    • Measure what runs deleted before this release left on disk with the Measure what deleted runs left job. A migration adds the exports, storage_scans and storage_scan_entries tables.
    • Jobs run in a worker the backend starts next to the web server (QABOARD_JOBS_WORKER=0 to run it elsewhere), and resume after a restart. From the backend's container: qaboard-admin jobs list|start|worker.
  • Runs activity for admins: everyone's waiting and running runs, the load over time, and the busiest projects, users and queues (runs activity).
    • Mark the runs stuck as running (pending for over a week, or what you choose) as failed with the Fail stuck runs job, now or on a schedule; Runs activity offers it when it sees such runs (fail stuck runs).
    • Runs now record when they were queued, started and ended. A migration adds three nullable columns to outputs (no table rewrite; it gives up after 10 seconds if it can't lock the table, run it again then), and small partial indexes built without blocking writes: expect a few minutes on big databases, as each index build reads the table.
  • Single sign-on with OpenID Connect (Keycloak, Entra ID, Okta, Google...), next to LDAP, SAML and local accounts (sign-in and single sign-on).
    • Set QABOARD_OIDC_ISSUER, QABOARD_OIDC_CLIENT_ID and QABOARD_OIDC_CLIENT_SECRET; add more providers from Admin → Sign-in.
    • Find what's wrong before users do: qaboard-admin check-sign-in and Admin → Sign-in → Test check that the server reaches your identity provider, the issuer, the signing keys, and whether the redirect URI is registered.
    • Keep users' accounts when they move to single sign-on: match them by user name when your identity provider shares LDAP's directory, or by verified email.
    • Send users straight to your identity provider with auth.sso_auto_redirect; /login?local=1 keeps a way in for local admins.
  • Decide who gets an account with auth.access_policy: everyone, emails of some domains, or after an admin approves. Admins get an email and a webhook message (Slack, Teams, Mattermost) when someone waits.
  • Sessions are stored on the server: they end after sessions.idle_days without activity and sessions.max_days in any case, users and admins can end them, and disabling a user ends theirs. Nobody is signed out by the upgrade.
  • Use LDAP over TLS: ldaps:// in QABOARD_LDAP_HOST, or QABOARD_LDAP_START_TLS=true.
  • Sign-in, sign-up and qa auth login are rate-limited by the shipped nginx configuration. Behind a load balancer or an ingress, set QABOARD_TRUSTED_PROXY_HOPS=1 to record users' IPs in the audit log.
  • Keep the commit lists of projects with many runs per batch fast: list them in QABOARD_SKIP_METRICS_AGGREGATION (project ids, or their start like group/), and their commit lists skip the batches' aggregated metrics, which commit pages still show (environment variables).
  • Jenkins builds started from QA-Board send the job's token from qaboard.yaml, else the server's new JENKINS_BUILD_TOKEN, else none, instead of the token qaboard. If your Jenkins jobs expect qaboard and your integrations don't set token, set JENKINS_BUILD_TOKEN=qaboard before upgrading (Jenkins builds).
  • Free more storage: the cleanup of old results (qaboard_clean) keeps only the batches that are milestones, not all the batches of their commit (deleting old data).
  • Delete a commit's runs from the web app even when it has milestones: its milestone batches are kept.
  • Upgrade the database with a migration that moves shared milestones from the projects' data to their own table; the old data stays for rollbacks.
  • Get your shared milestones back on servers where the upgrade failed to copy them because the projects' data was in an older format: the next upgrade converts it and copies them.
  • Before you upgrade: settings to check and data to move first.
    • Records each existing project's git host, with a migration that uses the deploy's git host settings. Before upgrading, keep GITLAB_HOST set, or list your GitLab in QABOARD_GIT_HOSTS with the hostnames older projects used (upgrading).
    • Security: the server only writes, deletes or runs code in the folders listed in QABOARD_STORAGE_ROOTS (default: /mnt/qaboard), and refuses results stored elsewhere. If your projects store results in other folders, set it before upgrading (see environment variables).
    • Security: the server only loads avatars and badges for the web app from your GitLab hosts and gravatar. If your projects' avatar_url or badges (integrations[].src) come from other hosts, list them in QABOARD_PROXY_HOSTS (see environment variables).
    • Daily database backups older than 14 days are deleted after each successful backup, but never the newest 3, nor dumps with other names. Set QABOARD_BACKUP_RETENTION_DAYS to keep them longer (0 keeps them all). Before upgrading, move the old daily backups you want to keep (backups).
    • Versions from before October 2026 don't start on a database migrated by this one: to roll back to them, first undo the newer migrations (rollbacks).
    • The website image (the docs your server serves) is built from the repository's root, like the frontend image: if you build it without docker compose, run docker build -f website/Dockerfile ..
    • Local sign-up is off by default with LDAP or SAML: local accounts could take the name of directory users who never signed in. Turn on auth.local_signup in the admin panel if you need it.
    • Name your first admins in QABOARD_ADMINS (user names, or the emails of LDAP/SAML accounts), else nobody can open the admin panel. Without access to the web app: qaboard-admin create-admin <user> in the backend's container.
    • Existing API tokens are hashed by the migration and keep working, as "legacy" tokens (they can read, send results and act, not use the admin API). After this upgrade, older versions can't use any API token, even after a plain rollback of the code (and while old backends still run during the deploy), and they ignore what you set in the admin panel, like requiring sign-in, and disabled accounts. If you may roll back, set what matters with environment variables. Downgrading the database (alembic downgrade) deletes the tokens.
    • Results sent with an unknown, expired or revoked token are now refused, even when ingest.auth is off: before, the token was ignored. Check the QABOARD_TOKEN/QA_TOKEN of your CI jobs (and users' shell profiles).
    • The audit log records the address nginx sees, not X-Forwarded-For, unless you set QABOARD_TRUSTED_PROXY_HOPS: clients could choose what it says. Behind a load balancer or an ingress, set it to 1.
    • QABOARD_LOGIN_REQUIRED=true now also blocks anonymous requests to the API, so CI jobs that read results (qa gate, qa check-bit-accuracy) need a token with the read scope. QABOARD_LOGIN_REQUIRED=false now allows anonymous users (any non-empty value used to block them), and an empty value lets you set it in the admin panel.
    • At SIRC, QA-Board no longer serves the storage over WebDAV (/davs/algo/): it let anyone write and delete files. services/nginx/webdav.conf explains how to bring it back.
  • Deploying: Compose, Kubernetes, quickstart: zero-downtime deploys, published images, a Helm chart and a quickstart.
    • Zero-downtime deploys with deployments/deploy.py: images tagged per version, several backend replicas behind nginx, rolling updates, rollback, status and restore-db.
    • deploy.py seed-db fills a staging database with only production's recent data.
    • Pull QA-Board's images from ghcr.io/samsung/qaboard: docker-compose.yml uses them by default, each release has its own tags (e.g. backend-1.2.0), and QABOARD_IMAGE picks another registry.
    • Run QA-Board on Kubernetes with the Helm chart published with each release, oci://ghcr.io/samsung/charts/qaboard.
    • Try QA-Board on your laptop with kind: charts/qaboard/quickstart.yaml and charts/qaboard/kind.yaml start a small cluster with the docs' demo projects (the chart's new demoData setting), where the qa CLI can save results (see the quickstart).
    • Mount the storage where runs save results with the chart's new results setting (an existing claim, a new volume, NFS or a node folder): the chart mounts it where the backend, nginx and Cantaloupe expect it, and sets QABOARD_STORAGE_ROOTS.
    • The Helm chart's migrations Job gets the chart's config: settings, which data migrations need (Kubernetes).
    • Try QA-Board on your computer with demo projects to look at: download one Compose file, deployments/quickstart/compose.yml, and run docker compose up -d --wait (see the quickstart). It only answers on your computer.
    • Build the website image behind a proxy that blocks some of the docs' media: with QABOARD_MEDIA_MISSING=remote, readers' browsers load them from the media host or your mirror (QABOARD_MEDIA_BASE_URL), and with placeholder they render as placeholders, instead of failing the build.
    • The backend has a /api/v1/health endpoint, shuts down gracefully, and keeps working when redis is down.
    • The web app links to the docs served by your server at /docs/, which match its version, instead of the public website. Set QABOARD_DOCS_ROOT to use another docs site.
    • Add pages for your own users to the docs your server serves: put them in website/site-docs/<site>/ and build with QABOARD_SITE=<site>. They get their own menu, and the installation page links to them. At SIRC, "At SIRC" has the install commands and SIRC specifics.
  • Git hosts and webhooks: GitHub, GitLab, Gitea/Forgejo and Bitbucket, with safer tokens and webhooks.
    • Configure all your git hosts (GitHub and GitHub Enterprise, GitLab, Gitea/Forgejo, Bitbucket, other git servers), each with its own token and webhook secret, with QABOARD_GIT_HOSTS (git hosts).
    • Keeps your current git settings working: GITLAB_HOST (also without a scheme, or with credentials), GITLAB_ACCESS_TOKEN, GITHUB_ACCESS_TOKEN, QABOARD_GITHUB_HOSTS for GitHub Enterprise, and QABOARD_GITLAB_HOSTS, whose hostnames are other names of GITLAB_HOST. A GitLab and a GitHub host can share a hostname (backward compatibility).
    • Receives the push events of repositories on Gitea, Forgejo and Bitbucket Cloud, at /webhook/gitea (or /webhook/forgejo) and /webhook/bitbucket (webhooks, project setup).
    • Require a secret in push webhooks: set QABOARD_WEBHOOK_SECRET, and the same secret in your repositories' webhooks, so that only your git hosts can send push events (webhooks).
    • Stops saving git tokens in new clones' .git/config, or showing them in error messages: git gets them from a credential helper, and only for their own host (it needs git 2.31 or newer). Clones made by older versions still have the token in their remote URL: remove it.
    • Stops sending git tokens, and GitLab session cookies, to another host when git or API calls are redirected there, and reads tokens and webhook secrets only from its environment, not from the CLI's shared secrets file (QA_SECRETS).
    • Webhooks are stricter. They refuse push events: without a webhook secret once any host has one or QABOARD_WEBHOOK_SECRET is set (401), from git hosts the server doesn't know (403; pushes to /webhook/gitlab from unknown hostnames still go to your GitLab), that would move a project to another git host (409), or over 25 MB (413). They ignore other events, like GitHub's ping, and answer 503 while the git host settings are invalid, instead of the server failing to start.
  • Security and access to files: logins for actions, and who can read which files.
    • Security: actions that run commands (redo, tuning, stopping or deleting batches...) now require a login and run as the logged-in user, and their inputs are escaped. See SECURITY.md.
    • Redoing, deleting or marking a run as failed now requires access to its project, for restricted projects.
    • Each batch started from QA-Board gets its own folder with its scripts and logs, under the batch's qaboard-batches/.
    • Restrict who can read the files of restricted projects, or of the folders you list under paths in users_restrict.yml: nginx asks the backend before sending each file (see user management). If you use your own nginx configuration, include snippets/qaboard-files.conf in the /s/ locations to get the file browser.
    • In the folders listed in QABOARD_FILES_UNIX_PERMISSIONS (e.g. /home), users only read the files that their own Unix account can read (see user management).
    • Without a complete /etc/passwd, e.g. with LDAP or SSSD, QABOARD_ACCOUNTS_COMMAND (and QABOARD_GROUPS_COMMAND) gives the backend the users' accounts, refreshed daily: the file browser shows owners' names instead of their ids (see users' accounts).
  • Settings: support and login hints, simpler result uploads.
    • Configure the support link with QABOARD_SUPPORT_URL.
    • Tell users which credentials to use when they can't log in, with QABOARD_LOGIN_HINT, and where to install WebCDE, which opens runs with a cde.sh, with QABOARD_WEBCDE_INSTALLER (environment variables). The tuning form's feedback link goes to QABOARD_SUPPORT_URL.
    • Email users when the batches they start end: set QABOARD_SMTP_HOST, and optionally the sender, reply-to address and email domain (QABOARD_EMAIL_FROM, QABOARD_EMAIL_REPLY_TO, QABOARD_EMAIL_DOMAIN). A migration adds a batch_notifications table and an index of the pending runs, built without blocking writes (emails).
    • Scripts can send results that describe their commit with only commit_sha and commit_authored_datetime: the other commit_* fields (branch, message, committer, parents) are optional, and the server doesn't need access to the git repository.
  • Database: faster commit lists, safer migrations, tunable PostgreSQL.
    • Lists of commits answer many times faster on big projects, and use much less memory: they no longer load every output of every commit.
    • Adds two indexes on ci_commits, to list commits faster. They are built without blocking writes, which can take a few minutes on big databases. If building one fails, the next deploy builds it again.
    • Starts even when the database has migrations it doesn't know, made by a newer version, e.g. after a rollback: it keeps them.
    • Tune PostgreSQL with a settings file: use services/db/init.sh as the database's entrypoint and it includes services/db/qaboard.conf (or your own, QABOARD_DB_CONFIG) in PostgreSQL's configuration at every start, and in new databases right away, with the pg_stat_statements extension (database settings).
    • Size the backend's database connection pools with QABOARD_DB_POOL_SIZE and QABOARD_DB_MAX_OVERFLOW (default 4 and 2 per uwsgi process, instead of 10 and 5): PostgreSQL's max_connections can stay small. The backend checks connections before using them, so it keeps working after a restart of the database.
    • The database gets two minutes to stop cleanly (stop_grace_period in production.yml), instead of being killed after 10 seconds and replaying its log at the next start.
  • Monitoring with Sentry: traces from the browser to the database.
    • See where the backend spends its time in Sentry: traces of API requests include their database queries, and a share of the backend's processes are profiled (SENTRY_PROFILE_SESSION_SAMPLE_RATE). Celery workers report their errors and tasks too (Monitoring with Sentry).
    • See the web app's page loads, web vitals (LCP, CLS, INP) and the API requests they make in the same Sentry trace as the backend's work, with readable stack traces: set SENTRY_AUTH_TOKEN, SENTRY_ORG, SENTRY_PROJECT and SENTRY_URL when you build the frontend image to upload its source maps, that are then no longer served.
    • Sentry is enabled by SENTRY_DSN alone (it also required FLASK_ENV=production), tags events with SENTRY_ENVIRONMENT and the release, traces 20% of requests by default (SENTRY_TRACES_SAMPLE_RATE), and checks the Sentry server's certificate (CA_CERT_URL build argument or SENTRY_CA_CERTS). Users are identified by their username only, tokens are filtered out, and SENTRY_FRONTEND_DSN sends the web app's events to their own project.
  • API: for scripts, agents and the image viewer.
    • Get the numbers of the Summary and Metrics Diff views from scripts and agents, with the experimental /api/v1/commit/<sha>/compare: each metric's median vs a reference, each run vs its reference run, and how many runs failed. See Quality gate.
    • API: page through /api/v1/commits with limit and offset (the X-Has-More header tells if there are more) and search them with q, find branches with /api/v1/project/branches?q=..., and get a much smaller list of projects with /api/v1/projects?summary=true (lists of commits). Without these parameters, they answer as before.
    • The API reads the values of a region of an image, for the image viewer's pixel values: GET /api/v1/output/image/pixels with image_url, x, y, w and h (up to 256×256 pixels), as JSON, or as binary with format=binary (their type and shape in the X-Pixels-Dtype and X-Pixels-Shape headers). What the image's reader says of the sensor (CFA pattern and cell size, bit depth, black level) comes along in sensor, or the X-Pixels-Sensor header: readers with other field names can be added in backend/backend/sensor.py.
    • The API finding where two images differ (POST /api/v1/output/image/diff) returns regions as x, y, w, h with the peak's value (comparable across images) and its fraction of the largest (diff), instead of a center and radius. It takes the reference's file name as path_ref, answers 404 for a missing file and 400 for images of different sizes, and keeps its results.
    • Images whose pixel values users read are decoded once and kept on disk, up to QABOARD_IMAGE_CACHE_GB (20 GB by default, the least recently used are removed), in QABOARD_IMAGE_CACHE_DIR. See the server's settings.
    • The server reads videos with FFmpeg (PyAV, in the backend's image), for the video viewer: what's in them (GET /api/v1/output/video/info), the decoded values of a frame's pixels (GET /api/v1/output/video/pixels, like images' with a frame), and where two videos differ, frame by frame (POST /api/v1/output/video/diff). Results are kept for a month in videos/, in QABOARD_IMAGE_CACHE_DIR.
  • Before running from a commit's artifacts, the server checks they are usable (the subproject's qaboard.yaml, the files in the manifests), and never runs from an incomplete folder.
  • The automated cleanup deletes artifacts again when storage.garbage.artifacts.delete is set: it was skipping every commit. Deleting a subproject's artifacts keeps the files the other subprojects of the commit still use.
  • qaboard_clean --dryrun no longer marks outputs as deleted in the database, and prints a summary of what was (or would be) deleted.
  • QA-Board's admins can open pgAdmin without a pgAdmin password: nginx asks QA-Board who is an admin (services/nginx/snippets/pgadmin.conf) and pgAdmin trusts it. The bundled pgAdmin also gets the database's password from services/pgadmin/passfile again.
  • Safer defaults: review them before upgrading.
    • Keep RabbitMQ, Flower and pgAdmin to the server's host: Docker Compose now publishes them on 127.0.0.1 only. Set QABOARD_RABBITMQ_HOST_IP when Celery workers connect from other hosts (passwords and admin services).
    • Mount only your results storage in the image server: Docker Compose no longer gives it the host's whole filesystem. If images stop loading after the upgrade, mount your storage in cantaloupe (starting a server).
    • Set the database password with QABOARD_DB_PASSWORD in .env: the database, the backend and the backups all use it (database password).
    • Protect the task queue: set RabbitMQ's account with QABOARD_RABBITMQ_USER/QABOARD_RABBITMQ_PASSWORD, and a login for Flower with FLOWER_BASIC_AUTH. The helm chart has celery.rabbitmq.* and reads FLOWER_BASIC_AUTH from its secret.
    • The Integrations menu's link checks and webhooks only reach the hosts in your projects' qaboard.yaml, your git and Jenkins hosts, and QABOARD_PROXY_HOSTS: add the hosts you only write as templates to QABOARD_PROXY_HOSTS.
    • See a warning in the admin checklist and in the server's logs when git webhooks are accepted without a secret.
    • Only QA-Board's admins can open Flower at /flower/: it can revoke tasks and shut the workers down.
    • Set QABOARD_SANDBOX_HTML=true so that the runs' HTML and SVG files can't act as whoever opens them: in the run cards their scripts run sandboxed, in an origin of their own, and opened in a tab they show without scripts. It's off by default for now: tell your users first (HTML reports).
    • Files the server creates are no longer writable by everyone, except the run folders and manifests that users' runs write in.
    • Project names with . or .. segments are refused: they could get around project access.
    • Only verified emails (directory accounts, or single sign-on with email_verified) count for email_domain group rules, the domains access policy and the emails in QABOARD_ADMINS, and local sign-up refuses the user names listed in QABOARD_ADMINS (user management).
    • Checking a Jenkins build's status needs the same access as starting one, and only reaches the Jenkins hosts of JENKINS_AUTH, on known ports.
    • Cookies from before stored sessions stop working once the user signs out or their sessions are ended, and 180 days after the upgrade (sessions).
  • Get {"error": ...} JSON answers and the right status codes from the batch, run and export APIs, and a 400 instead of a server error for invalid metrics or dates.
  • Listing a batch's tests with the project's code, to start a batch from the web app, stops after 4 minutes with a message instead of holding a server worker.
  • QABOARD_DB_ECHO=0 (or false) turns the SQL logs off.

Documentation​

Getting started​

  • Run QA-Board on your computer with the new quickstart: one Docker Compose file, or a kind cluster and the Helm chart, with demo projects to look at, then results from an example project.
  • Put your project on QA-Board step by step with the tutorial: pick your server, code hosting, CI and runners once at the top, and every step shows the instructions for your stack. Links keep your choices (e.g. ?server=k8s&ci=gitlab-ci).
  • See what QA-Board is for in your first review and in one use case per demo project of a fictional robotics company, starting with image quality. New: odometry and mapping for robotics, evaluating an LLM agent, and tests and coverage over time: unit test suites from any runner that writes JUnit XML, with flaky and deleted tests caught. The key concepts fit in one page.
  • Find the quickstart, the review loop, each viewer (videos and 3D included) and what's coming next on the website's new home page. Links to the docs shared in chats show a card with a glimpse of the web app.

User guide​

  • New user guide for people who use the web app every day.
  • See the web app in screenshots and clips of the demo projects on every page of the user guide, each image viewer tool included, and next to the settings that produce them in the project integration docs, like Visualizations.
  • Release notes going back to 2017, on the website and in the app.
  • See every tool of the new image viewer in action: a clip of the four comparison modes, and pictures of comparing with another file of the run, the values written on the pixels, a histogram's range, profiles, statistics, raw images and full screen (image viewer).
  • See configuration diffs, stacked references, the command palette, your home page, Storage usage, Runs activity, access tokens and project access in pictures of the demo projects, on their user guide pages.
  • Follow the steps that changed in October: hover a run's name for its History, Sign in everywhere, and Ctrl+K, Runs activity and Storage usage from the user guide's front door and the FAQ.
  • Start from the user guide's front door: the few words you need, a map of the commit page, and one page per task, like reading metrics, new and reference, milestones, choosing the tests to run and logs and failed runs.
  • Learn each viewer on its own page: images and videos, each with a typical review and all its shortcuts, and plots, flame graphs, 3D and more.
  • Read user guide pages that start with the most common task, with the details in collapsed sections.

Project integration and server docs​

  • Many pages updated to match the code, and features that were never documented: GitHub webhooks, API tokens, batch options, bit-accuracy manifests, auto-optimization...
  • The project setup and CI pages cover GitHub and GitLab equally (CI integration), and admins get a new git hosts page.
  • Set up a server for your team step by step: the server page goes from what the host needs to the first results in 10 steps, with a check after each, says where each setting goes, and how to close the ports you don't need. The admin pages show the admin panel.
  • Get the quickstart, the tutorial and the runner pages working as written: where qa stores results (QA_STORAGE), what to check after qa init, and the LSF, Dask, Celery and local runners' pages in the same style as the others.
  • Set up single sign-on and decide who gets an account with the new sign-in and single sign-on page.
  • Find the server docs in one place, written for any site: starting a server, deployments and rollbacks with deploy.py, Kubernetes, host maintenance, backups and troubleshooting.
  • Write your own file viewer: see Viewers and file formats.
  • Start with how a project integrates QA-Board: the steps from installing qa to running in CI, and what you get after each.
  • Find what each viewer reads, and its options, on the new Viewers and file formats page, and matrices, YAML anchors and metadata filters on advanced batches.
  • Read the setup, batch, CI, tuning and recipe pages with examples from the demo projects, shown from their files so they stay up to date.
  • See where QA-Board is going on the new roadmap: its principles, and for each theme what's shipped, experimental, next or still being explored.
  • The CI helpers example fails the CI job when a task fails.

The website​

  • Follow old links to pages that moved or were removed: they redirect to the closest page.
  • The docs show a placeholder that describes a screenshot or clip when your browser can't load it, instead of a broken image.

Performance​

Several references​

  • Switch between stacked references without waiting: once the page is idle, the next and previous references load in the background (except very big ones), and the others load without their runs, enough for their tabs to show their branch and message.

Commit pages with 1000 runs​

  • Type in the filter box of a commit page without the page freezing: the results update in half a second, instead of after 7 seconds of a frozen page. Sorting runs, and matching them with their reference, no longer freeze the page either.
  • Open the Metrics Table in a third of a second instead of 7 seconds, with a fraction of the memory (40 MB instead of 640 MB): big tables only render the rows on screen.
  • Open the Visualizations twice as fast: cards download their files as they scroll into view, and files shared by several cards or viewers are downloaded once. Batches with more than 300 runs show more cards as you scroll, or all of them with "show all".
  • Commit pages load in 0.6 s instead of 1 s.
  • Switch between references twice as fast, without the page freezing: the Summary shows the new reference's numbers at once, then updates its plots one after the other. Pages with many plots no longer freeze while they draw.
  • Open tuning batches faster: runs that have an exact match in the reference no longer go through a name-similarity search.
  • Load a commit's Summary without the page freezing while it draws its box plots: with 1000 runs, it takes a sixth of the work it did.
  • Switch references or filter runs with the Summary updating at once, also with 5000 runs.
  • Open the Visualizations and Output Files of big batches faster: with 1000 runs, the Visualizations take half the work.
  • Resize the View options panel smoothly on big commit pages: the results make room for it when you let go of its edge.
  • Keep a commit page open while its runs are stuck in a queue: it checks for updates less and less often while nothing changes (up to every 5 minutes), and big commits less often. It checks often again as soon as something changes.

Lists of commits​

  • Scroll through long lists of commits (a branch's, a committer's) without the page freezing, with under a third of the memory: each commit's tooltips and menu are created when you reach them, and the commits already shown don't render again when more load.

Everywhere in the web app​

  • Start the web app faster: it downloads a quarter less JavaScript on first load (date picker, error reporting and number formatting load only when needed), and servers send precompressed files.
  • Come back to a page you already visited, or reload, and it shows up at once from your browser's cache, then refreshes in the background. Commits with running outputs keep updating.
  • Open commits sooner: hovering a commit in a list starts loading its page, and a project's commits no longer wait for its configuration.
  • Hover the History plot and image-heavy pages more smoothly: the page updates once you pause on a commit, and image viewers you've left no longer keep redrawing in the background.
  • Load pages with less code: the log viewer's colors come with the logs, and the release notes' details when you open What's new.
  • Keep time-of-flight point clouds open without the page redrawing them all the time: they're drawn when they change, and free their memory when you close them or change frames.

Logs​

  • Scroll through hundreds of runs in the Logs view without slowdowns.
  • Open very long logs (tens of thousands of lines) several times faster, and search them without the page freezing for seconds.
  • See the diff of big text files (logs, configurations...) with the reference sooner: the editor loads while the files download, instead of after.

Images​

  • Find where images differ faster: on a 12 MP image in under a second instead of 4 to 8 seconds, on a 96 MP image in 4 to 5 seconds instead of 6 to 18 (under a second for another way to compare), and at once the next time. Differences of a few pixels in large images are no longer missed.
  • Read the values of pixels in large 16-bit or raw images without waiting: the image viewer loads them by tiles of 64×64 pixels and keeps them, the server sends them as binary, decodes each image once even when many users open it at the same time, and no longer decodes images again after 20 others.

Fixes​

Across the web app​

  • Admin → Service accounts shows the service accounts, instead of an empty project named admin/service-accounts.
  • The Storage usage menu in a project's sidebar opens next to it, instead of off-screen on the left.
  • Long project names no longer overflow the Storage usage summary.
  • Storage scans, exports and jobs name file owners from QABOARD_ACCOUNTS_COMMAND's accounts, like the file browser, instead of uid 2041: users see their files in Storage usage → Not tracked and their exports, also from earlier scans.
  • Storage scans no longer fail when the backend can't update the users' accounts (e.g. their shared folder belongs to another user): they use the accounts they have.
  • Storage scans read each folder once: the same folder under several roots (symlinks, bind mounts) and storage snapshots are no longer scanned again, even when you set your own excluded paths.
  • The whole web app says Sign in and Sign out: the sidebar, the viewers and the file browser used different words.
  • The cheat sheet (?) lists the viewers' real shortcuts: h starts a blind test, < and > change a video's speed, and the run filter's example fail keeps the failed runs. The image viewer's own list includes full screen (f).
  • The batch menu says Move the filtered runs to another batch when you filter runs, instead of the other way around.
  • The automated tuning example names the solver's acq_func option correctly.
  • Copying the path of an output, batch or commit directory gives the right path when folder names contain characters like #, and the confirmation shows the copied path.
  • Notifications show at the bottom of the window again, instead of at the bottom of the page's first screen where they were out of sight once you scrolled down.
  • Move on quickly from the sidebar's Integrations and Milestones menus: they close once the mouse leaves them, instead of staying up to 3 seconds over the page.
  • Links to the LSF docs and to the getting-started page work again, and the retired chat link now points to your server's support page (by default, QA-Board's GitHub issues).

Commit page​

  • Run cards draw the visualizations of projects that still list them under their old name, outputs.detailed_views, instead of empty cards under the View options' switches.
  • The controls panel shows the sort actually used, the project's default metric, instead of Name, and Output Files no longer shows an empty visualization filter.
  • Run cards grow to fit a plot wider than their configured width (outputs.style.width), instead of the plot spilling out of the card.
  • Run cards with a configured width (outputs.style.width) get narrower when the page is, e.g. when the View options are open, instead of going under them and hiding the image viewer's zoom and settings.
  • Visualization cards line up from the left, instead of being pushed to the edges of the page.
  • The Tuning Analysis view's efficient frontier colors points by nested tuning parameters (like denoise.strength) instead of drawing black dots.
  • Dynamic options no longer show an empty card when a run doesn't have the selected value, sort numbers as numbers (2 before 10), and no longer split into options like frame_1234 that control nothing. Whether an option is linked is no longer remembered across projects.
  • A run's card in full screen has a button to leave it, and scrolls when it's taller than the screen, instead of its content sticking to the screen's edges.
  • Hiding a visualization shown by default takes one click instead of two.
  • The reference commit's refresh button works again: it was disabled once the commit had loaded.
  • The error page shows the error as text, instead of inserting it in the page as HTML.
  • Artifacts & Configs shows the artifacts again when the commit loads after the view, instead of "Could not find" them.

Metrics and tables​

  • A run is compared with the reference run of the same platform: when the reference batch had the same input and configuration on two platforms, the other platform's run was picked.
  • The Improvement report showed "=" for metrics that changed, and badges for those that didn't.
  • The Summary's change in % has the right sign for metrics with negative values, and runs that return no value (null) for a metric no longer count as 0 in its median.
  • The Summary only warns that it mixes runs with different parameters when the runs it aggregates really have different parameters: filter the runs to compare one configuration.
  • In the Metrics Table, the New and Ref headers stay over their columns when a filter leaves out every run that has some metric, and metrics without a target show just their unit, like [s].
  • Removing a metric from a table's metric picker no longer also drops the selected metrics that the current batch doesn't have.
  • The Summary's box plots on a log scale show a few round ticks, instead of labels that overlapped.
  • The Summary's box plots shade the target's zones right for metrics where higher is better: green above the target, red below.
  • The Summary's box plots use a linear scale when some values are 0 or negative, instead of squashing the plot.
  • Output Files tags runs Bit-accurate when their output files match the reference's: like qa check-bit-accuracy, it only checks the files that match bit_accuracy.patterns in qaboard.yaml (all files by default, except logs, run.json and metrics.json). The other files are still listed when they differ, greyed and tagged "not checked". See bit-accuracy tests.

Image viewer​

  • In the image viewer, changing a Dynamic Option (e.g. a slider over your pipeline's stages) keeps the zoom and the selected ROI, for the new and reference images, instead of zooming out to the whole image, most often when the reference's image showed up. Images with another aspect ratio fit the selected ROI again.
  • The regions where images differ most (formerly Auto ROIs) are centered on the differences, instead of starting at them.
  • Images show in the image viewer on servers that don't set QABOARD_IMAGE_SERVERS: the default is now Cantaloupe's /iiif/2/ endpoint, instead of a /iiif URL that Cantaloupe couldn't answer.
  • JPEG images show in the image viewer again, instead of an "Image Download Error": the image server only sent them as JPEG tiles, and the viewer asks for PNG ones. They now come as PNG tiles, with the exact pixel values for the pixel tooltip and the diff. Update the cantaloupe image to get the fix.
  • The image viewer's Auto ROIs finds regions again, instead of failing with an error.
  • The image viewer's ROI thumbnails show the image the viewer shows: they no longer break for runs that save a TIFF instead of a PNG, or for images served by another image server (QABOARD_IMAGE_SERVERS).
  • ROIs set on inputs of another size than the output images are placed right on non-square images, in the thumbnails and when you zoom on them.
  • Images whose proportions differ from 4:3, like 300×400 portraits, fill their pane when they open, and stay fitted when the pane resizes, instead of showing smaller with dark margins.
  • In full screen, the images make room for the analysis panel: you see the images and their histogram, profiles or statistics together, without scrolling.
  • The analysis of the image's own values no longer spins forever when the image keeps redrawing, e.g. while some of its tiles fail to load.
  • The image viewer decodes an image again when its cached pixel values are damaged, instead of failing.

Videos, plots and other viewers​

  • Point clouds take the colors and names of what you compare, like the other viewers: blue for new, the reference's color and its name, e.g. a milestone's, instead of orange for new and blue for the reference.
  • In the video viewer, n and N go to the frames that differ more than the frames around them. On videos where every frame differs, like with a new denoiser, they no longer only go back to the first frame.
  • Text diffs stay side by side in narrow run cards, as the Side-by-side button says, and inline diffs are tall enough to show every changed line.
  • Playing or pausing a video that has no reference no longer raises errors ("pause is not a function").
  • Videos your browser can't play (e.g. HEVC) say so, with the command to convert them, instead of staying black. Animated .webp and .gif files now show, side by side with the reference's, instead of a black box or their bytes as text.
  • In point clouds, WASD moves while the key is pressed, not after it's released.
  • Plots saved with side_by_side: false draw the reference run's traces on the same axes, instead of nothing when a figure has several traces. Traces keep their own dash style.
  • Differential flame graphs are colored by what changed: red for frames that take more time than in the reference, blue for less, orange for new ones. The colors no longer exaggerate changes near the root.
  • The SLAM and tuning plots no longer have Plotly's "Send to Cloud" button, so their data can't be uploaded to Plotly's public cloud.
  • Text viewers set to show only differences hide identical files, and videos play without a reference.
  • Text files in a run's output files show again when the reference run doesn't have them, and their diff with the reference no longer breaks when you press t to swap the sides.
  • HTML outputs (like a run's report) show their images and links, which are relative to their own folder, and their styles no longer change the rest of the page: they show in a frame, without running scripts.

Commits, branches and history​

  • Opening a commit from a project's page no longer crashes the page with find is not a function.
  • Clicking a run's name opens its History from 3 days before the commit, so you see the commits before it without picking dates.
  • Projects without a qaboard.yaml use master as their reference branch in the web app, like the server and qa, instead of develop.
  • Reloading a page, opening a shared link or using the back button shows what the URL says: the selected commits, batches and filters could disagree with it.
  • The back button takes you back in one click after following links inside QA-Board: some links added two entries to the browser's history.
  • The chart above a branch's commits shows the branch's metrics, the median or average over each commit's runs, instead of staying empty. Each commit row also shows the median and average of the default metric over its CI runs.
  • History's "Performance over time" chart shows your project's metrics again, instead of asking you to define them.
  • The History page's per-test tables show their metric columns again, in tabs named "Metrics Table" and "Metrics Diff" like on commit pages.
  • On the History page, picking metrics updates the metrics distribution too.
  • Selecting a milestone from qaboard.yaml selects the latest commit of its branch.
  • The batch selector no longer jumps back to the top while you scroll it, and its keyboard navigation works again. It now opens on the selected batch, marked with a check.
  • Switching to a branch you had opened in a previous visit no longer crashes the commits list ("setHours is not a function").
  • The History page no longer crashes when dashboard_metrics or dashboard_evolution_metrics in a metrics file isn't a list of metric names (e.g. dashboard_metrics: psnr): it uses the defaults instead.
  • Pages no longer crash when a link repeats filter or search (e.g. ?search=a&search=b): they search for both.
  • Pages that crash while Sentry loads, or before QA-Board knows the server's configuration, are reported to the server's Sentry too.
  • Commit pages and the branch list no longer break when results were sent without a branch.
  • The projects list shows the description from qaboard.yaml (project.description) for projects that aren't on GitHub or GitLab too.
  • The chart above a branch's commits leaves out the commits that don't have the metric, instead of drawing gaps in its line.
  • Typing a commit id in the commits list's search and pressing Enter no longer leaves "[object Object]" in the search box.
  • Pressing Enter in the commits list's search keeps filtering the commits, instead of going to the branch or commit you typed. To go there, pick it with the arrow keys and press Enter, or click it.
  • Editing your own shared milestone no longer asks whether to overwrite it.
  • The commits list shows the newest commits on servers whose time zone isn't UTC.
  • Find the commit of a tag, or of a branch you didn't run on, with the API and qa check-bit-accuracy --reference <tag>, when the server can read the git repository.

Runs, batches and tuning​

  • Compute times are right in run menus: 90 seconds said "2min30s".
  • Run qa --share batch with an edited qaboard.yaml to see its new visualizations and metrics on that batch, while the commit's other batches, CI included, keep the commit's. Each qa batch used to replace the commit's qaboard.yaml for all its batches.
  • The automated tuning form's feedback link goes to your server's support page, and is left out when the server has none.
  • Redoing runs from QA-Board respects the batch's concurrency, and uses the runner the runs were started with (e.g. Dask) and its settings. QA-Board used to start each run with its own qa batch, so they all ran at once, with the default runner.
  • Runs that start again (redo, qa batch...) no longer show as failed, and count as failed, until they finish: their status is reset when they are pending or running.
  • Finished runs whose metrics don't include is_failed count as successful, instead of neither successful nor failed.
  • A stuck run's "Mark as Finished" action is now labelled "Mark as Failed", which is what it does.
  • Runs killed while they run show as failed right away, instead of running until their batch ends, or forever: when you bkill their LSF job, when LSF kills it for using too much memory, or when you press Ctrl+C. Stopping qa batch also marks the runs it stopped as failed.
  • Moving runs to another batch closes the dialog, and saving a milestone no longer risks losing it.
  • The tuning form shows a readable error instead of the server's raw response.
  • The tuning form remembers the "Overwrite" option, and setting the number of iterations no longer crashes.
  • Run Tests / Tuning runs as you when you open it from a link: it no longer asks for a user it can't let you type, with Send disabled.
  • The Tuning Analysis view plots each iteration's metrics (those in the objective) next to the objective, for tunings run from now on: qa optimize used to get no metrics from the server.
  • Projects whose inputs are inside the project (a relative inputs.database, like inputs) see their tests in the Run Tests / Tuning form and in Available Tests: they are looked for in the commit's artifacts, where the runs started from QA-Board run, instead of in a folder of the server. In monorepos, from the root qaboard.yaml's folder, like qa does.
  • Run Tests / Tuning and Available Tests find the inputs of projects whose inputs.database uses {project} or {subproject} (e.g. /datasets/{project.name}), instead of looking for them from /.
  • When several of a commit's batch files define the same batch, Run Tests / Tuning and Available Tests use the last one, like qa batch, instead of the first.
  • Batches started from Run Tests / Tuning run like qa batch on your computer: they use the defaults set at the top of their batch file (database, configurations, platform), and when several files define a batch, the last one replaces it instead of being merged into it. QA-Board used to run a merged copy of the batch files that dropped these defaults; it now runs copies of each file, saved next to the logs.
  • Listing a batch's tests no longer fails for commits that QA-Board saw before qa sent their qaboard.yaml (e.g. opened by branch name): it uses the project's, and the commit gets its own the next time qa runs on it.
  • Run Tests / Tuning lists the tests of batches whose name contains +, & or #.
  • When a batch has an input it can't use, like an empty - or a number, qa batch says which batch, in which file and at which line, instead of "Could not understand the inputs in the batch" followed by all its inputs. Available Tests and Run Tests / Tuning show the same error.
  • Stopping a batch whose runs can't be stopped reports the error instead of failing.
  • The tuning form no longer asks for an Android device, which runs never received.
  • qa batch --runner=local (or any --runner) uses the batch's and the input type's settings for that runner, e.g. its local concurrency and timeout, also when the project's default runner is another one, like LSF.
  • Exports put runs next to the reference runs on the same inputs, even when the inputs moved.
  • Deleting some files of a run also removes the folders they leave empty.
  • With --tuning, context.configs and context.params no longer include the _configs parameter.

CLI​

  • Give a batch its own concurrency (under local:, lsf: or dask:) or local timeout: its runs now use it instead of the command's. Command-line options like --local-concurrency still win, even when they equal the value in qaboard.yaml.
  • Runs over their local timeout fail, and qa batch kills what they started and goes on with the other runs, instead of stopping with a TimeoutError and leaving them pending.
  • Inputs with their own settings can set their platform:, and no longer make qa batch fail with KeyError: 'platform' in batches that set one.
  • qa batch '*' runs your batches only, instead of also running every input as a batch named aliases.
  • qa batch --action-on-existing assert-exists leaves the runs that already succeeded as they are, instead of failing them with a command that doesn't exist.
  • qa wait exits with an error when the run it waited for failed, so qa wait ... || ... works.
  • qa optimize works without --checkpoint, instead of failing with a TypeError.
  • qa check-bit-accuracy --reference-platform finds the reference's runs on the other platform when runs have configurations, instead of saying there is no reference.
  • qa tells you to run qa init, not qatools init, when it can't find qaboard.yaml, and qa batch --help says that --local-concurrency -1 runs one run per CPU.
  • Run qa run and qa batch locally before the results storage is set up: when qa can't create the storage folder (/mnt/qaboard after qa init, or $QA_STORAGE), it warns and saves the results in output/, instead of every command failing. --share, CI runs and qa save-artifacts still stop, and say how to fix it (where is the data).
  • qa init fills the project's name and URL from your git remote even when it can't reach it (no network or credentials), with master as reference branch, instead of leaving the sample's user/sample_project.
  • The LSF settings in the qaboard.yaml that qa init creates use max_threads, which the LSF runner reads, instead of threads, which it ignored.
  • With LSF job arrays and the Dask runner, qa run no longer deletes the log file its output goes to: errors are no longer lost.
  • Runs on Dask workers are headless, like with LSF: they use matplotlib's agg backend and no DISPLAY, instead of opening X11 connections back to the display of whoever started the batch.
  • Runs started from a git worktree checkout are saved under their commit, instead of their branch name. Submodules with a relative .git pointer also work.
  • On Windows, qa works in a git worktree whose .git pointer holds an absolute Linux path: it is translated with the site's path mappings (QABOARD_PATH_MAPPINGS).
  • qa starts even when the git HEAD cannot be read, e.g. an unresolvable worktree pointer: it warns and continues with <local:user> commit placeholders instead of crashing at startup.
  • On Windows, qa batch finds existing results for projects whose name contains a "/", instead of failing with "could not find any data on commit ID": --action-on-existing works there.
  • qa optimize works with recent versions of scikit-optimize, instead of failing with "cannot import name 'Integer'".
  • qa optimize no longer fails at the end when matplotlib isn't installed: it only skips its convergence plot.
  • qa batch --tuning-search runs a grid search when the JSON doesn't say search_type, as the docs show, instead of failing with KeyError: 'search_type'.
  • qa no longer warns that it doesn't know where your server is when you set QABOARD_URL=http://localhost:5151.
  • qa no longer prints a git error when the commit it runs on isn't in the local clone (e.g. shallow CI clones): it only did without the commit's details. QA_VERBOSE=1 still shows it.
  • With Python 3.12 and later, qa no longer prints a SyntaxWarning: invalid escape sequence when it starts.
  • qa batch accepts a single input written as text (inputs: a.jpg), and says that a batch written as a list (my-batch: [a, b]) must be a mapping with inputs:, also in the web app (the server used to call exit() there).
  • qa get works with Python 3.13: it said "Could not find" for every variable.
  • bit_accuracy.on_reference_failed_ci: compare-first-parent goes back up to bit_accuracy.max_parents_depth parents, instead of two at most.
  • bit_accuracy.failed_ci_job_name works again: qa check-bit-accuracy only looks at that job (<job> <subproject> in subprojects).
  • qa check-bit-accuracy counts skipped CI jobs as successful, instead of waiting for them.
  • Windows test vectors no longer fail bit-accuracy on hex outputs whose end-of-lines differ.
  • qa --share check-bit-accuracy compares your runs with the reference's CI runs, instead of looking for reference runs with your own @user| label.
  • qa check-bit-accuracy fails when the runs differ from any of the references, e.g. any parent of a merge commit: only the last one counted.
  • qa check-bit-accuracy without --batch only compares the runs of the --label you give: with the default label, it also compared the other labels' runs.
  • qa check-bit-accuracy --reference origin/my-branch works, instead of failing with a NameError.
  • qa check-bit-accuracy ignores the same files with or without manifests: bit_accuracy.ignore patterns match the start of file paths.
  • qa check-bit-accuracy without manifests no longer reports identical files as different.
  • The update check no longer crashes on release-candidate or development versions.
  • Functions decorated with @on_branch no longer print "Called".
  • On Windows, qa no longer leaves a temporary file behind for each text file it checks for bit-accuracy.

Server and administration​

  • qa gate, the server's comparisons and exports to a folder compare each run with the reference run the web app shows: they used to rank configurations by how many characters differ, and exports preferred reference runs on another platform.
  • Cancelled jobs whose worker died stay cancelled, instead of being queued again.
  • The backend's log no longer repeats argument of type 'NoneType' is not iterable for projects without storage settings. Invalid storage settings are logged with the project and the error.
  • LDAP accounts get their email from QABOARD_LDAP_ATTRIBUTE_EMAIL when it isn't mail: only mail was read.
  • Show the demo projects' latest commits in the quickstart's commit list: the demo data's dates, and its files' modification times, move to when QA-Board first starts, instead of keeping the day the demo data was made (quickstart).
  • The project's settings (metrics, visualizations, storage...) are no longer replaced by those of whichever branch got new results last: they come from the reference branch (project.reference_branch in qaboard.yaml), or from the project's first results.
  • nginx starts on hosts without IPv6, instead of failing on its listen [::]:80 line.
  • Repositories whose clone is broken are cloned again, instead of failing until an admin deletes them. The broken clone is only replaced once the new clone succeeded.
  • Tag pushes record the tag's whole name: its first character was cut.
  • Pushes to repositories with files or folders whose name only looks like qaboard.yaml (for example an agent skill folder named qaboard-yaml) no longer fail: only files named exactly qaboard.yaml or qatools.yaml define projects.
  • Jenkins integrations show the status of the builds they trigger when Jenkins' own URL uses another hostname than the integration's build_url (for example a full domain name): checking them failed with a 403 because no credentials matched that hostname in JENKINS_AUTH.
  • Jenkins integrations trigger jobs that have no parameters, and when Jenkins refuses a build the error says why (HTTP status, Jenkins' message, a hint about JENKINS_AUTH or build_url) instead of showing a whole HTML page about a missing location header.
  • Servers outside SIRC no longer show avatars from SIRC's directory for GitLab users without one: set QABOARD_AVATAR_URL to use your own (environment variables).
  • Backend workers no longer risk crashing, or losing Sentry events, when Sentry is enabled: Sentry starts in each uWSGI worker, and uwsgi.ini sets py-call-uwsgi-fork-hooks, as Sentry's SDK requires. Keep it if you use your own uwsgi.ini.
  • The cleanup also deletes commits that never had runs, and uses each project's own storage.garbage settings instead of the first project's.
  • storage.garbage.artifacts.keep now works with artifact group names, and also accepts folders and globs.
  • Commits whose artifacts were only partly restored are no longer marked as usable.
  • qaboard_clean --dryrun no longer deletes the folders of runs without a manifest, or with a corrupted one.
  • Single sign-on with SAML builds the right URLs behind QA-Board's nginx, which now passes X-Forwarded-Proto and X-Forwarded-Host to the backend (keeping those of your own proxy), and signing out brings you back to QA-Board instead of an error page.
  • The server no longer signs in to GitLab (GITLAB_AUTH) when it starts: a slow GitLab doesn't delay it anymore.
  • Zero-downtime deploys (deploy.py up) no longer fail a few requests with a 502 when the old backend replicas are busy: they are stopped once they answered every request they received (waiting up to a minute), instead of 5 seconds after nginx switched to the new ones (deployments).

Security and data safety​

  • Requiring sign-in now blocks anonymous requests to the API itself, not only the web app's pages.
  • API tokens are no longer stored in clear, and a token in a URL (?token=) is only accepted to read. Requests made with a token no longer open a session cookie.
  • Session cookies are HttpOnly and SameSite=Lax, and Secure when QABOARD_URL is https. Changes made with cookies must come from QA-Board's own pages.
  • Signing in with SAML only sends you back to QA-Board's own pages (open redirect), and lands on the home page instead of an empty error page when there's nowhere to go back to.
  • Logging in no longer sends everything the directory knows about you to the browser.
  • Reading the values of an image's or a video's pixels, what's in a video, and finding where images or videos differ now require a login, and only read files in the storage (QABOARD_STORAGE_ROOTS) that you're allowed to see. Signed out, the viewers show the displayed values and say to sign in for the rest. These endpoints could read images anywhere on the server, without a login.
  • The unused PDF report of the regions where images differ (/api/v1/output/diff/report) is removed: it could write files anywhere on the server, without a login.
  • Marking a run as failed or deleting it now requires a login.
  • On servers that restrict projects to some users, only users allowed on a project can stop, rename, move or delete its batches, delete a commit's results, save its artifacts, or change its milestones. Exporting runs also checks you can access the reference's project.
  • QABOARD_LOGIN_RESTRICTED=false (or 0) turns project restrictions off, as you would expect: any non-empty value used to turn them on.
  • Deleting results no longer deletes the contents of folders that they link to.
  • When a restricted project saves its results in a folder per user (e.g. /algo/outputs/{user}), the project's files are now restricted in every user's folder, not only in the server's.
  • The API qa uses to update batches no longer lets anyone without a login delete runs. qa optimize still replaces its previous best iteration's runs, and only those.
  • When qa fails to update QA-Board, its warning no longer prints your API token (QA_TOKEN).
  • When signing up with a user name or email that is already used, the server answers "already taken" instead of an error that included the password's hash, and no longer writes the password to its logs.
  • Logging in with an empty password is refused, and LDAP logins escape the user name.
  • The server only deletes the git clones it made: it refuses repository paths inside a clone, or that hold other clones.
  • Error messages in run cards show as text, instead of being rendered as HTML.
  • GitLab CI jobs in the Integrations menu only show and play jobs of the project's own repository, for people who can access the project.
  • Starting batches from the web app no longer changes the server's umask for a while: other files it created meanwhile could become writable by everyone.
  • Exporting into a folder that already exists never replaces or deletes what's in it, and only the folders exports create are listed in Storage usage (exporting files).
  • Importing the milestones saved in your browser needs access to their results, and the developer role.
  • Editing a run from the web app no longer changes who sent it.