Skip to main content

2018

qaboard v0.2.0

The year QA-Board became a platform: a React web app backed by a database and a REST API, a `qa` command-line tool that wraps any project's code, tuning and automated optimization on the cluster, and the first generic viewers for images, plots and videos.

Users

A new web app, for every project

The server-rendered pages were replaced by a React single-page app, backed by a database and a REST API. It now hosts many projects, each with its commit list, per-commit results, filters and metrics over time.

Project integration

The qa command-line tool

A small Python package wraps your code so that each run on an input is tracked: qa run for one input, qa batch for groups of inputs on the cluster, and qa init to set up a project with a single YAML file.

Learn more →
Users

Tuning and automated optimization

Parameter sweeps can be started from the web app and run on the cluster, then explored with dedicated plots. An optimizer based on scikit-optimize can also search the parameter space automatically to minimize a loss you define.

Learn more →
Users

Viewers for any output

Output files are shown with generic viewers: Plotly charts, videos, 3D point clouds, and an image viewer with deep zoom, side-by-side comparison and perceptual diffs against the reference.

Learn more →

Web app​

  • The commit list can be filtered by branch, committer and date range, including negative filters, and the selected batches and filters are saved in the URL.
  • Added charts that follow the metrics over time, with boxplot summaries, log scale, medians and improvements relative to the reference.
  • Output cards show each run's status (pending, running, failed, crashed), its parameters, its logs with ANSI colors, and a link to the output folder, with a button to copy its Windows path.
  • Added tuning exploration plots: 1D and 2D sensitivity, a parallel coordinates plot of the parameters and the main metrics, and the trade-offs between metrics.
  • Tuning forms support grid and random sampling, arrays and categorical parameters, a label for each experiment, and the platform to run on.
  • Added a viewer switcher on output cards, and the selected tab is saved in the URL.
  • Large components are loaded on demand, for faster page loads.

CLI and project setup​

  • Projects are configured with qatools.yaml, and their metrics with a metrics.yaml file (labels, units, targets, whether smaller is better, which ones to show by default).
  • Groups of inputs are listed in YAML files, with globs and lists of globs; qa batch --group runs over them, locally or on LSF.
  • qa batch can cap LSF memory and threads per project, waits for its jobs, and cancels them if it is interrupted.
  • When results already exist, you can choose to skip, re-run, only re-compute the metrics or only sync them (--action-on-existing, also configurable per project).
  • Added qa optimize for automated tuning, with custom loss functions and targets.
  • Added qa check-bit-accuracy to compare a commit's outputs with its reference branch, and qa save-artifacts to save build outputs (with multiple globs).
  • Added qa get to read the CLI's variables, a module entrypoint (python -m qatools), and a sample project.
  • Added sub-projects: a qatools.yaml in a parent folder is inherited, so one repository can host several projects.
  • Runs can be tagged with a label to keep tuning experiments apart, and their parameters are hashed to keep output paths short.

Server and administration​

  • Results are stored in a SQL database, PostgreSQL in production, with Alembic migrations.
  • GitLab webhooks keep commits, branches and committers in sync with the repositories.
  • Added a REST API, used by the web app and by the CLI to report runs and their metrics.
  • The server ships as a Docker image, with staging and production environments deployed from CI, behind nginx with HTTPS.
  • Added a cleanup script that deletes old results, configurable per project.
  • Added a Cantaloupe IIIF image server to serve large images efficiently.

Notes​

  • The CLI was then distributed as the qatools Python package, configured with qatools.yaml. Version 0.1.0 came out in July 2018 and 0.2.0 in December 2018.
  • QA-Board was used internally at Samsung by a growing number of algorithm teams; it was not open-source yet.