Skip to main content

2020-q2

qaboard v0.8.7

QA-Board got ready for outside users: the server now starts with docker compose, projects no longer need GitLab, and the CLI works in GitHub Actions. New flame graph viewer, matrix batches, and many image viewer improvements.

Admins

Start the server with docker compose

The server is now a set of single-purpose images (backend, web app, database, image server, reverse proxy) started with docker compose, with optional daily database backups and a simple restore script.

Learn more →
Users

Flame graph viewer

Outputs saved as *.flame.json are displayed as interactive flame graphs, with differential flame graphs to compare against the reference and pinpoint performance regressions.

Learn more →
Project integration

No more GitLab requirement

The CLI now sends the commit, branch and configuration data to the server itself, so QA-Board works without GitLab and without giving the server read access to your repository. The CLI also integrates with GitHub Actions.

Learn more →
Project integration

Matrix batches

Batches can define a matrix of configurations and platforms, and qa batch runs every combination.

Learn more →

Web app​

  • Commit pages refresh automatically while results are pending.
  • References are matched to each run by the closest configuration, so runs with slightly different settings are still compared.
  • The image viewer selects the first region of interest automatically (or the one marked focused: true), shows regions of interest from input metadata, and shows them even without a reference. Images are requested as lossless PNG when possible.
  • The text viewer lets you toggle between new and reference with the T key, and highlights large files correctly.
  • Viewers are only shown for files that exist, and outputs show more clearly whether they changed.
  • Output lists can be sorted by ID, and filtering is much faster.
  • Batches show the storage they use.
  • Each batch can have its own configuration and metrics, which helps with qa --share runs.
  • Integrations can use the commit's branch_slug, a URL-friendly version of the branch name.
  • The tuning menu is easier to discover, has clearer sample configurations, and hides LSF-only options when LSF isn't used.
  • A friendlier welcome page is shown when the database is empty.

CLI and project setup​

  • QA_BATCH_FAIL_IF_EMPTY makes qa batch fail when a batch matches no inputs.
  • Auto-optimization accepts parameters for scikit-optimize's Optimizer, and works with subprojects.
  • Tuning no longer requires users to set up SSH keys.
  • The commit tag can be read from environment variables, and qa --share run always sends the git data it needs.
  • QA_CI_HELPERS_N_JOBS sets how many jobs the CI helpers run concurrently.
  • On Windows, paths like C:// are accepted and | characters in commands are escaped.
  • The CLI no longer fails on a repository's first commit, or prints warnings when GitLab isn't used.
  • The qaboard package now requires Python 3.7 and is released to PyPI from GitHub Actions; it is now version 0.8.7.

Server and administration​

  • Docker images are published to Docker Hub and the GitHub container registry, with development and production compose overrides.
  • The image server (Cantaloupe) runs as a compose service with an upgraded image, and pgAdmin comes preconfigured.
  • Added a daily database backup service, and the backup docs explain how to restore.
  • Integrations that trigger CI jobs support a custom token and cause.
  • Added a cron job to remove old docker images.

Documentation​

  • The website is published on GitHub Pages, with screenshots, Algolia search and a clearer order.
  • New docs on starting the server with docker compose, matrix batches and tuning, and updated troubleshooting.

Fixes​

  • Fixed deleting failed results, sorting outputs, and exporting renamed batches.
  • Fixed thin lines between image tiles and colors differing between new and reference images.
  • Fixed commit_url with subprojects and custom artifact locations.
  • Fixed the tuning "override" option.