QA-Board release notes | QA-Board

Skip to main content

2023-q2

June 30, 2023

qaboard.yaml files can include another file, bit-accuracy checks can compare two labels of the same batch, and a new page documents monorepos and subprojects.

Project integration

Include other qaboard.yaml files

A qaboard.yaml can now use a top-level include key to pull in another YAML file, to share settings between subprojects. Paths are relative to the root QA-Board directory.

Users

Compare bit-accuracy across labels

qa check-bit-accuracy has a new --reference-label option to compare a batch's outputs against the outputs of another batch label.

2023-q1

March 31, 2023

The image viewer has a new regions-of-interest menu, with automatic difference detection that is faster and offers several diff metrics. Text diffs can jump from one change to the next, and the backend moved to Python 3.11.

Users

New regions-of-interest viewer

A "View ROIs" menu in the image viewer switches between the full image, the ROI presets from your input metadata, and automatically detected differences with the reference. It always offers the full image and keeps a stable layout.

Users

Faster automatic diffs, with more metrics

Finding regions of interest is faster and no longer runs out of memory on huge images. You can pick the difference metric (YIQ, SSIM, CIE 1976, CIE 1994 or CIE 2000), and the last one used is remembered.

Users

Navigate text diffs

The text viewer jumps to the first difference with the reference, has a "Next Diff" button, and says when both files have the same content.

2022-q4

December 31, 2022

A quiet quarter: safer output directory names, Jenkins runners configurable from environment variables, a cap on the size of tuning batches, and backend stability tweaks.

Project integration

Jenkins runner configuration from the environment

The Jenkins runner can now read its build URL and token from QA_RUNNERS_JENKINS_BUILD_URL and QA_RUNNERS_JENKINS_TOKEN, so they don't have to live in qaboard.yaml. The token is now optional.

2022-q3

September 30, 2022

qaboard v1.0.2

You can delete several batches at once and sort runs by storage. Auto-optimizations can resume from a checkpoint, and servers can report crashes to Sentry.

Users

Delete multiple batches

The commit page's batch menu has a new "Delete multiple batches" option to pick and delete several batches in one go. Milestones are skipped.

Users

Resumable auto-optimization

qa optimize has a new --checkpoint option to save its progress and restart interrupted optimizations. Tunings started from the web app use it automatically, and show accurate iteration counts.

Users

Storage per batch

The Output Files view shows the total storage used by the selected runs, and runs can be sorted by storage.

Admins

Crash monitoring with Sentry

The backend and the web app can report errors and performance traces to Sentry. Set SENTRY_DSN (and optionally SENTRY_SAMPLE_RATE) on the backend to enable it.

2022-q2

June 30, 2022

A maintenance quarter: free-form LSF options, commit URLs that accept a branch name, and more robust Jenkins integrations and tuning pipelines.

Project integration

Free-form LSF options

The LSF runner accepts a new options setting, for example "-W 24:00", appended to the bsub flags QA-Board already sends. The LSF docs now use the correct max_memory and max_threads keys.

2022-q1

March 31, 2022

qaboard v1.0.1

The qaboard package reached 1.0. The server now ships its own Celery task queue, string metrics show up in tables and cards, and you can delete only the output files that match a pattern.

Admins

Built-in Celery task queue

The QA-Board server now starts a RabbitMQ broker and a Flower monitoring dashboard, so projects only need to register Celery workers to run batches or tuning from the web app. Set runners.default: celery in qaboard.yaml to use it.

Project integration

String metrics

String metrics returned by your run() are now displayed in the metric tables and output cards, next to the numeric ones.

Users

Delete output files by pattern

When deleting a batch's files while keeping its metadata, you can now give patterns like *.png to delete only the matching files.

Project integration

Custom bit-accuracy comparisons

qa check-bit-accuracy can compare files with your own Python function, set with QA_BITACCURACY_CMP, instead of only comparing file hashes.

2021-q4

December 31, 2021

Tuning from the web app can now use shared or private batch definitions, milestones are protected from deletion, and bit-accuracy checks also compare the inputs. Batches files gained a default database and more flexible paths.

Users

Shared and private batches for tuning

When starting a tuning experiment, batches can come from the commit's batches files, from a form shared with all users, or from your own private form. The merged batch definitions are saved with the tuning batch.

Users

Protected milestones

Commits with milestones get a star in the commit list, and the web app disables the menus that would delete their batches, so milestone results can't be deleted by mistake.

Project integration

Bit-accuracy checks compare inputs

Runs now save a manifest of their input files in manifest.inputs.json, and qa check-bit-accuracy reports when the inputs differ from the reference's, not only the outputs.

Project integration

More flexible batches files

A batches file can set a default database for all its batches, database paths can use {project} and {subproject}, and inputs.allow_duplicate_batches lets subprojects define the same batch names.

2021-q3

September 30, 2021

The image viewer now compares images of different sizes, runs can show custom parameters and badges, and the tuning analysis lets you focus on the metrics you care about.

Users

Compare images of different sizes

The new and reference images no longer need the same dimensions: the viewer, the color tooltip and the crops all handle different sizes.

Project integration

Run parameters and badges

run() can return params, shown and filterable like configuration parameters in the web app, and badges that link from a run to another page, such as the training log of a machine learning model.

Users

Better tuning analysis

The tuning and auto-optimization plots let you select a subset of metrics, and have a clearer layout.

2021-q2

June 30, 2021

Users can now log in, and tuning from the web app runs as the logged-in user. You can compare against runs from other projects, manage runs inside batches more finely, and run batches on Windows machines through Jenkins.

Admins

Log in to QA-Board

The web app has login and logout menus, with users authenticated by LDAP or a local user list. Tuning from the web app now requires logging in, and tuning experiments run as the user who started them.

Users

Compare with other projects

The reference of a comparison can now be a commit from another project. Exports work with cross-project references too.

Users

Finer control over runs in a batch

Run menus can move runs to an existing or new batch, redo a single run with its original parameters and runner, mark a pending run as finished, or delete only its output files while keeping its metrics.

Project integration

Windows runners through Jenkins

Batches can run on Windows machines by sending jobs to a Jenkins server configured in qaboard.yaml. The CLI prints a link to each Jenkins job and waits for it to finish. Runners can also have their own default platform.

2021-q1

March 31, 2021

The image viewer's color tooltip now shows the real pixel values of your images, batches give more control over per-input configurations, and the server handles heavy load better.

Users

Real pixel values in the color tooltip

Hovering over an image now shows the actual values stored in the image file, not just the displayed colors. Images are cached on the server, so the tooltip stays responsive.

Project integration

Per-input configurations in batches

Each input of a batch can have its own extra configuration, and the same input can appear several times with different ones. Matrix batches no longer add the default configurations when they define their own.

Project integration

Run outside of a git repository

The new GIT_MESSAGE and GIT_AUTHORED_DATETIME environment variables, along with GIT_COMMIT and GIT_BRANCH, describe the commit when your code doesn't run from a git checkout. When set, they take precedence over git.