Bit accuracy tests

QA-Board was started at Samsung, within a business unit focused on hardware digital design. Because of those root, QA-Board provides a number of ways to check that results are equal from commit to commit.

"Soft" bit-accuracy checks from the UI

The web application lets you view and compare all files created by your algorithm's runs:

• Files are marked depending on their status (identical, different, added, removed...). Identical files are hidden by default.
• You can click on each file to open it with an appropriate file viewer.

Only the files that changed are listed: a rounding change in the fixed-point model shows in its register dump, then in the last two stages' buffers.

Click a file to diff it with the reference: Next Diff jumps to the register that changed. Ctrl-click another file to add its diff. On a commit that only changed docs, the run is Bit-accurate.

Like qa check-bit-accuracy, the web app only checks the files selected by bit_accuracy.patterns (see what files are checked). When all of them are identical, the run is tagged Bit-accurate. The other files are still listed when they differ, greyed and tagged not checked: for instance metrics.json, which has the run's measured run time. Unlike qa check-bit-accuracy without --strict, checked files that exist in only one of the runs also make the web app show differences.

"Hard" qa check-bit-accuracy on the CLI

qa check-bit-accuracy --batch $batch compares the results of qa batch $batch to:

• The latest results on project.reference_branch from qaboard.yaml (default: master).
• ...unlesss you're checking a merge made to that branch. In which case the commit's parents will act as references.
• You can ask to compare versus a specific git commit, branch or tag with qa check-bit-accuracy --reference $git-ref.

If the commit you compare against has not finished its CI, qa will wait. It asks GitLab for the commit's pipeline jobs, or GitHub for its statuses and check runs, so it needs a token (see CI integration). Skipped jobs, and jobs allowed to fail, count as successful. bit_accuracy.failed_ci_job_name restricts it to a given job: in a subproject, it looks for the job <failed_ci_job_name> <subproject> (with the name of the subproject's folder).

Custom Needs

You can opt-in to more complex behaviour in qaboard.yaml with bit_accuracy.on_reference_failed_ci: compare-first-parent (it goes back at most bit_accuracy.max_parents_depth first parents, default 5), in case there are not results in the reference commit. Maybe the build failed, in which case you want to compare against the previous commit... If you're interested open an issue we'll add more details to the docs.

If output files are different, qa prints a report and exits with a failure.

On the rounding change, qa check-bit-accuracy lists the files that differ and fails. On a commit that only changed the docs, it passes.

Other options:

• --strict: by default only the files that exist in both runs are compared. With --strict, files that exist in only one of the runs are errors too.
• --reference-label: compare to the reference's runs that used another label, e.g. qa --label new check-bit-accuracy --batch all --reference-label old.

Input and output manifests

Each run saves in its output directory a manifest.outputs.json, with a hash of each output file, and a manifest.inputs.json, with a hash of the input files (at most 200 files for an input folder). qa check-bit-accuracy compares both, so it also tells you if the inputs changed. To list other files as inputs, set context.obj['manifest-inputs'] to a list of paths in run().

For specific use-cases, there is also qa check-bit-accuracy-manifest --batch my-batch (it also accepts --strict), which checks the results versus manifests stored in the database next to the input files, instead of versus another commit. To save those manifests, use qa batch my-batch --save-manifests-in-database (the flag is forwarded to qa run). It only works for inputs that are folders.

What files are checked?

Files matching the patterns defined as bit_accuracy.patterns in qaboard.yaml will be checked. Without patterns, all files are checked.

qaboard.yaml

bit_accuracy:
  patterns: ['out.png', 'stages/*.png', 'regs.txt']
  ignore: ['debug/']

• Patterns are glob patterns that match the end of a file's path in the run's output directory: out.png checks out.png and some/folder/out.png. * also matches /.
• Files whose path starts with one of the bit_accuracy.ignore patterns are not checked.
• Files that change at each run are never checked: metrics.json (it has the run time), run.json, log.txt, log.lsf.txt and manifest.inputs.json (qa check-bit-accuracy compares it on its own, see above).

The web app's Output Files and the quality gate's require_bit_accuracy check the same files.

note

If you work with text files on both Linux and Windows, EOL can make things tricky... You can decide what files are plaintext or binary using bit_accuracy.plaintext or bit_accuracy.binary.

Sample CI for bit-accuracy checks

You often want to know when your algorithm's results change, especially if another team is busy implementing them in hardware!

Here is how you could get the CI to warn you with GitLab CI (it works the same with GitHub Actions):

.gitlab-ci.yml

stages:
  - tests
  - bit-accuracy

tests-all:
  stage: tests
  script:
  - qa batch all

bit-accuracy-all:
  stage: bit-accuracy
  allow_failure: true
  script:
  - qa check-bit-accuracy --batch all

Custom output comparisons

Bit accuracy checks in QA-Board are done by checking the hashes of your output files. To make it fast, those hashes are computed once, and stored in "manifest" files to make it fast, along with other file metadata.

In some cases, you may want to compare files using something else than bit-exactness between files. Maybe you want to ignore timestamps, or maybe you want to do "semantic" comparisons... To do this, you can supply your own comparison function:

1. Implement a cmp function in some file.py:

from pathlib import Path

def cmp(path1 : Path, path2 : Path) -> bool:
  """Checks if the files have the same content, like QA-Board does by default"""
  return path1.read_bytes() == path2.read_bytes()

2. Ask qa to use it:

export QA_BITACCURACY_CMP=/path/to/file.py
qa check-bit-accuracy --batch <batch_name>