Visualizations

Your runs write files in context.output_dir. List the ones people should look at in qaboard.yaml: the commit page's Visualizations view shows them on each run's card, next to the reference run's, in a viewer picked from the file's extension.

Show an image

If your code writes out.png:

qaboard.yaml

outputs:
  visualizations:
    - name: Output
      path: out.png

path is relative to the run's output directory. Run a batch with qa --share batch <batch>, open the link it prints, then the Visualizations view: each run's card shows its out.png next to the reference's, and zooming one zooms the other.

Each run's card starts with its main metrics, then shows out.png next to the reference's, zoomed on the same window.

When the web app sees your changes

A commit keeps the qaboard.yaml that qa sent with its first results. To try new visualizations, edit qaboard.yaml and run qa --share batch <batch> again: the batch keeps your edited qaboard.yaml, and the web app shows its new visualizations for that batch. The commit's other batches, like the CI's, keep the commit's. qa --share run doesn't change which qaboard.yaml a batch uses.

Which viewer opens a file

The extension picks the viewer. Set type when it doesn't say, e.g. type: 6dof/txt for a trajectory saved as poses.tsv.

Files	type	Viewer	In the user guide
.png, .jpg, .jpeg, .bmp, .tif, .tiff, .pdf	image/*	Image: side by side, wipe, difference, pixel values	The image viewer
.raw, .dng, .hex	image/*	Image, if the image server reads these raw formats (not included)	The image viewer
.mp4, .webm, .mov, .mkv..., animated .webp and .gif	video/*	Video: in sync, frame by frame	The video viewer
.plotly.json, .plotly.html	plotly/json, plotly/html	Plot, side by side or on the same axes	Plots
.flame.json	flame/json	Flame graph, differential	Flame graphs
.html	text/html	HTML, in a frame, without scripts	HTML reports
.pcd	pointcloud/pcd	Point cloud in 3D, with the reference's and a ground truth	Point clouds
any, e.g. .tsv	6dof/txt	6DoF trajectory, with the reference's and the ground truth	Trajectories
any	pointcloud/txt	A time-of-flight camera's frames: point clouds and depth maps (legacy layout)
anything else: .txt, .json, .log...	text/plain	Text, diffed with the reference's	Text

What each viewer reads (video codecs, point cloud formats, the trajectories' columns...) and its options are in Viewers and file formats.

Show several files, hide the debug ones

Cards show the visualizations in the order of the list. Those with default_hidden: true start hidden: people turn them on in the View options. Hide the debug plots and the large files.

qaboard.yaml

outputs:
  visualizations:
    - name: Output
      path: out.png
    - name: Edge profile
      path: edge.plotly.json
    - name: Color errors
      path: color.plotly.json
      default_hidden: true

The View options list every visualization: Color errors and Noise vs signal are hidden until someone turns them on.

Give each visualization a name: the View options list it by its name (or by its label, if you give one), and the web app shows and hides visualizations by name. Visualizations without a name are shown and hidden together.

Every file your run writes

A visualization only chooses what shows on the cards: QA-Board keeps every file your run writes in context.output_dir.

• The Output Files view lists all of a run's files, and which differ from the reference's: see the commit page. qa writes their sizes and hashes in manifest.outputs.json at the end of each run, next to the run's log (log.txt) and metrics (metrics.json).
• The file browser opens any run's folder.
• qa run empties the output directory before it runs your code. To keep what's there, e.g. your own cache, use qa run --keep-previous.
• Where the files are saved and when they are deleted: Where is the data saved? and Deleting old data.

Show a file per frame, stage or step

When runs write one file per frame, pipeline stage or iteration, one visualization can offer them all. Name the part of the path that changes with :, here :stage:

qaboard.yaml

outputs:
  visualizations:
    - name: ISP stage (1 raw, 2 demosaic, 3 denoise, 4 sharpen, 5 tone map)
      path: "stages/:stage.jpg"

The View options get a Files to show section, with a slider when the values are consecutive integers, else a select. The value applies to all the runs' cards, unless people choose it Per run (see the commit page).

One visualization for the 5 ISP stages: the slider steps from the raw mosaic to the tone-mapped image, on every run's card.

• Name the parts of your paths for your users: each option is labeled after its part, :stage as Stage, :noise_level as Noise level. Visualizations that use the same name share the option.
• Runs don't need the same values. The View options offer the values of all runs, and sort numbers as numbers (2 before 10). Until people pick a value, each run shows its own default: its last value when the values are numbers (e.g. its last frame), else its first. When a run doesn't have the selected value, its card shows the closest number (or the default) and says so.
• Show all the matching files at once, one after the other, with display: all.

Files to show picks the stage: synced, the slider on 3 shows stages/3.jpg on every card. Changing it keeps the zoom.

Patterns: several parts, parts of names, regular expressions

Paths are patterns in the syntax of path-to-regexp:

qaboard.yaml

outputs:
  visualizations:
    - name: Frames
      path: ":frame/output.jpg"           # a folder per frame
    - name: Frames, by number
      path: ":frame/frame_:number.jpg"    # part of a file name
    - name: Debug images
      path: "(debug_.*\\.jpg)"            # a regular expression, in parentheses
    - name: Logs per frame
      path: ":frame/(.*\\.txt)"           # both
    - name: Numbered frames
      path: "frames/:frame(\\d+).jpg"     # a named part that only matches digits

With regular expressions:

• write them in parentheses: they are "capture groups";
• (.*)/output.jpg also matches folders inside folders: you often want ([^/]*)/output.jpg;
• their options are named after the visualization, and not shared with other visualizations: prefer named parts. Two visualizations of (.*)/debug_output.jpg and (.*)/output.jpg get two selects for the frame.

Preset regions of interest

The image viewer shows thumbnails of regions of interest above the images, to jump to the details that matter: the input's presets, and the regions where the images differ most. Give the presets in the input's metadata, or in a configuration, as a list under roi. The camera ISP example reads each photo's metadata from a YAML file next to it:

inputs/photos/facade.metadata.yaml

# Regions of interest for the image viewer (pixels of the 1224x816 photo)
roi:
- x: 100
  y: 40
  w: 160
  h: 200
  label: Window frame (halos)
- x: 520
  y: 20
  w: 220
  h: 220
  label: Leaves over a window
- x: 380
  y: 300
  w: 120
  h: 120
  label: Flat wall (noise)

Source: examples/image-quality/inputs/photos/facade.metadata.yaml

The input's preset regions, and below them the regions where the server finds that the images differ most.

Regions are in the pixels of the image shown. When the image is smaller than the input, e.g. downscaled, give the input's size in its metadata (width, height), or the size of the image a region refers to (image_width, image_height): the viewer scales the regions. A region can also have a color.

To write new presets, zoom on a region in the web app, then pick Copy the view as a region in the regions menu: you get its coordinates in this format. How people use regions: the image viewer.

Make room on the cards

Cards are as wide as the page allows, up to 1500 px. Set a width for all of them in outputs.style, or for one visualization in its style. Widths are a maximum: cards get narrower when the window is.

qaboard.yaml

outputs:
  style:
    width: 1180px    # new and reference images about 600 px wide each
  visualizations:
    - name: Report
      path: report.html
      style:
        width: 840px

Viewer options

Other keys of a visualization go to its viewer: side_by_side: false draws a plot over the reference's, fps sets a video's frame rate, point_size a point cloud's. Each viewer's options are in Viewers and file formats.

Every key of a visualization

Key	What it does
name	How the View options list it, and the key that shows or hides it.
label	Shown instead of the name.
path	The file, relative to the run's output directory, or a pattern.
type	The viewer, when the extension doesn't say it: see the table.
default_hidden	true to start hidden.
display	With a pattern: single (default) shows the file picked in the View options, all every matching file, viewer gives the pattern to the viewer.
style	CSS for the viewer, e.g. width.
Other keys	Options of the viewer: see Viewers and file formats.

outputs.controls adds switches to the Viewers section of the View options. Each viewer gets the switch's value as an option of the same name, which overrides the visualizations' own. The 6DoF viewer, for instance, reads show_debug, which adds plots of DebugExtensions.txt:

qaboard.yaml

outputs:
  controls:
    - type: toggle
      label: Debug
      name: show_debug
      default: false

When nothing shows

• Invalid visualization path on the cards: the web app can't read a path pattern. Hover the tag for the error.
• A line like 2 more without files in these runs, under the visualizations of the View options: hover it to see which visualizations found no file in the runs shown. Compare their path with the run's files in Output Files.
• A run without the file shows nothing for that visualization. A file only the new run has shows alone, without the reference's.
• You changed qaboard.yaml but the cards didn't: see when the web app sees your changes.

Examples

The demo projects show complete configurations, with what they look like in the web app:

• Image quality: images, a slider over the ISP's stages, plots, a video.
• Detection: detections drawn on the images, precision-recall curves over the reference's, an HTML report.
• Serving latency: latency plots, a flame graph, text diffs.
• Robotics: a trajectory, a video, a point cloud.
• Agent evals: transcripts and HTML reports.