Skip to main content

Bit accuracy tests

QA-Board was started at Samsung, within a business unit focused on hardware digital design. Because of those root, QA-Board provides a number of ways to check that results are equal from commit to commit.

"Soft" bit-accuracy checks from the UI​

The web application lets you view and compare all files created by your algorithm's runs:

  • Files are marked depending on their status (identical, different, added, removed...). Identical files are hidden by default.
  • You can click on each file to open it with an appropriate file viewer:
bit accuracy viewer
note

The UI doesn't care about qaboard.yaml's bit-accuracy.patterns (discussed later).

"Hard" qa check-bit-accuracy on the CLI​

qa check-bit-accuracy $batch compares the results of qa batch $batch to:

  • The latest results on project.reference_branch from qaboard.yaml (default: master).
  • ...unlesss you're checking a merge made to that branch. In which case the commit's parents will act as references.
  • You can ask to compare versus a specific git commit, branch or tag with qa check-bit-accuracy --reference $git-ref.

If the commit you compare against has not finished its CI, qa will wait.

Custom Needs

You can opt-in to more complex behaviour in qaboard.yaml with bit-accuracy.on_reference_failed_ci, in case there are not results in the reference commit. Maybe the build failed, in which case you want to compare against the previous commit... If you're interested open an issue we'll add more details to the docs.

If output files are different, qa prints a report and exists with a failure.

note

For specific use-case, there is also qa check-bit-accuracy-manifest which checks accuracy versus a manifest file stored in the database next to the input files. To generate those manifests, use qa batch --save-manifests-in-database.

What files are checked?​

Files matching the patterns defined as bit-accuracy.patterns in qaboard.yaml will be checked.

note

If you work with text files on both Linux and Windows, EOL can make things tricky... You can decide what files are plaintext or binary using bit-accuracy.plaintext or bit-accuracy.binary.

Sample CI for bit-accuracy checks​

You often want to know when your algorithm's results change, especially if another team is busy implementing them in hardware!

Here is how you could get the CI to warn you with GitlabCI:

qaboard.yaml
stages:
  - tests
  - bit-accuracy

tests-all:
  stage: tests
  script:
  - qa batch all

bit-accuracy-all:
  stage: bit-accuracy
  allowed_failure: true
  script:
  - qa check-bit-accuracy --batch all

Custom output comparisons​

Bit accuracy checks in QA-Board are done by checking the hashes of your output files. To make it fast, those hashes are computed once, and stored in "manifest" files to make it fast, along with other file metadata.

In some cases, you may want to compare files using something else than bit-exactness between files. Maybe you want to ignore timestamps, or maybe you want to do "semantic" comparaions... To to this, you can supply your own comparaison function:

  1. Implement a cmp function in some file.py:
from pathlib import Path

def cmp(path1 : Path, path2 : Path) -> bool
  """Checks if the files have the same content, like QA-Board does by default"""
  return path1.read_bytes() == path2.read_bytes()
  1. Ask qa to use it:
export QA_BITACCURACY_CMP=/path/to/file.py
qa check-bit-accuracy --batch <batch_name>