Skip to main content

2019

qaboard v0.8.0

The web app got its name, QA-Board, and its documentation website. Bit-accuracy checks, milestones and batches defined in YAML arrived, and the image viewer became a real comparison tool.

Project integration

Bit-accuracy checks

qa check-bit-accuracy compares a commit's outputs with any reference (branch, tag or commit), with clear reports and file manifests to make it fast. In the web app, a bit-accuracy tab shows which outputs changed.

Learn more →
Users

Milestones

Star important commits or batches as milestones, with a name and a description, and compare any result against them. Milestones can be kept for yourself or shared with the whole project.

Learn more →
Users

Image and text comparison

Images are compared with the reference with synced zoom, pixel diffs, color tooltips, histograms and brightness, contrast and gamma controls. New text, diff and HTML viewers cover the other outputs.

Learn more →
Project integration

Batches of inputs and configurations

Batches files list the inputs to run on, together with their configurations and database, and support aliases, filters and custom input discovery. Run them with qa batch --batch, on LSF or locally in parallel.

Learn more →

Web app​

  • The web app is now called QA-Board.
  • Added search on the commit list, by message, author, branch or batch, and a refresh button.
  • Added favorite projects; projects are sorted by their latest results, and sub-projects show their own avatar, description and color.
  • You can pick any commit, not only a branch, as the commit or the reference to compare with.
  • Visualizations are now defined per commit in the project configuration, and the files each one shows are matched by patterns.
  • The image viewer: press t to toggle between the new and the reference image, and image settings are synced across viewers.
  • Added a configuration tab that shows and diffs all the configuration files and artifacts in a file tree, with syntax highlighting.
  • Two videos play side by side, in sync.
  • The 3D point cloud viewer can color points by intensity, show a mesh, and be moved with the WASD keys.
  • Added an integrations menu on commits, with links to build artifacts and documentation and buttons to trigger external jobs such as Jenkins builds.
  • Added live logs for running jobs, logs for whole batches, and buttons to stop a batch's jobs or delete a batch.
  • Tuning: the exploration plots are saved in the URL, the efficient frontier plot is colored by parameter, the 1D sensitivity plots are split per input and configuration, and you can choose the platform to run on.
  • Results can be exported to a shared folder.
  • Added a page that follows metrics across commits over time.
  • The web app works in Internet Explorer 11.

CLI and project setup​

  • Batches files replace the input groups: qa batch --batch (and --batches-file) run over their inputs and configurations, with only/exclude filters and merging of several files.
  • Added --runner local with multiprocessing, and LSF settings now live under runners.lsf in qatools.yaml, with per-batch overrides.
  • Added qa batch --list, --list-inputs and --list-output-dirs to see what a batch would run.
  • Projects can define their own way of finding inputs with iter_inputs, and the kind of inputs with --type.
  • --share (formerly --ci) saves local runs at the central location so they appear in the web app.
  • Configurations can set environment variables for your code with {"ENV": {...}}.
  • Bit-accuracy: qa check-bit-accuracy takes --batch, --reference-platform and ignore patterns (bit_accuracy.ignore), lets you define which files are text or binary, and normalizes line endings on Windows. Added qa check-bit-accuracy-manifest.
  • Added CI helpers (@on_branch(), run_tests()) and GitLab commit statuses, and better support for Jenkins.
  • qa save-artifacts --file saves a list of files instead of the configured artifacts.
  • qa exits with code 1 when runs fail, forwards your code's return codes, and warns when a newer version is available.
  • Improved Windows support: paths, line endings and file names.
  • qa starts much faster.

Server and administration​

  • Added daily database backups, with instructions to dump and restore.
  • Old results are garbage-collected once a day.
  • The server is deployed automatically when master changes.
  • Added API endpoints to stop pending runs, update batch metadata and export results.

Documentation​

  • Added a documentation website, with an "Edit" button on each page, guides on visualizations and on debugging runs with an IDE.

Notes​

  • The CLI was still distributed as the qatools package, configured with qatools.yaml. Versions 0.3.0 to 0.8.0 came out during the year. The --group options became --batch, and --ci became --share.