Visualizing your algorithm's outputs

The run()function wrapping your code receives an output_directory where it can create all kinds of files. Usually, you only want to look at a few of those files, the rest being debug data. Visualizations help you declare pre-sets of relevant files.

Here is a simple example, assuming your code an image named output.jpg.

qaboard.yaml
outputs:
visualizations:
- path: output.jpg
note

For now, to debug your visualizations, you have to commit and push your new qaboard.yaml. We plan on letting you edit simply qaboard.yaml locally, and update the visualizations when you use qa --share.

You can provide multiple relevant files, and hide debug visualizations by default:

qaboard.yaml
outputs:
visualizations:
- path: output.jpg
- path: debug.jpg
default_hidden: true
# type: image/jpg # auto-guessed

Users will get switches to toggle debug visualizations:

toggle visualizations

Available file viewers

QA-Board tries to guess the right image viewer depending on the file extension or a type

FileViewer TypeViewer
*.jpg*, *.png*, *.bmp*, *.tif*, *.pdf*...image/*Image
*.plotly.jsonplotly/jsonPlot.ly
*.flame.jsonflame/jsonFlame Graph, diffable
*.htmlplain/htmlHTML (assumes trusted input..!)
*.mp4video/*Video (synced)
*.txt, unidentifiedtext/plain*Text (diffs, with VSCode's Monaco Editor)
pointcloud/txtpointcloud viewer (needs to be refactored, coupled to a specific internal project...)
6dof/txt6DoF viewer (needs to be refactored, coupled to a specific internal project...)

Image viewer

  • Supports all common image formats.
  • Fast and smooth zoom & pan, synced. Fast image streaming via IIIF.
  • Perceptual color difference.
  • Color tooltip.
  • Image filters (exposure, contrast, gamma...).
  • Histograms per channel.
  • Automatic regions of interest.
Image viewer
tip

If your configurations or input metadata contain roi: [{x, y, w, h, label}], those "regions of interest" will be displayed and easily selectable.

Plot.ly viewer

The Plotly library has everything you need from bar charts to 3d plots.

  • huge variety of plots
  • interactive plots
  • easy-ish to use with binding to python/JS/matlab...
  • web-based
  • open-source and popular
  • performant
plotly gallery3d plot with plotly for LSF/Calibration

All you need is to save your plot data as JSON.

import plotly.graph_objects as go
fig = go.Figure(data=go.Bar(y=[2, 3, 1]))
with open('graph.plotly.json', 'w') as f:
spec = fig.to_json() # '{"layout": {...}, "data": [{...}, {...}, ...]}'
f.write()

Text Viewer

Text/diff viewer

Flame Graphs

We love Brendan Gregg's flame charts and integrated Martin Spier's d3-flame-graph. At a glance, you can check where you code spends its CPU cycles, and use differential flame graphs to debug regressions.

flame graphs viewer

For a tutorial read our blog post.

More Viewers?

Tell us what you need! The next we'll implement is likely vega specs. It would notably allow us to display altair visualizations. We could also have a minimal integration with notebooks, visdom, webiz...

Dynamic visualizations

You can use a special syntax to create dynamic visualizations at display-time. Users will we able to choose what to display using sliders / select options:

Viewing each frame of a movie
qaboard.yaml
outputs:
visualizations:
- name: Movie Frames
# you can use the `/user/:name` syntax to match part of filenames
path: ":frame/output.jpg"
# you can match part of filenames (experimental)
path: ":frame/frame_:number.jpg"
# For more examples, the full syntax is available at:
# https://github.com/pillarjs/path-to-regexp

You can also use regular expressions (inside parentheses!) to match which output files you want to view:

outputs:
visualizations:
# A common use case is matching file extensions
path: "(.*\.jpg)"
# ... or parts of filenames
path: "(debug_.*\.jpg)"
# you can mix with the previous syntax
path: ":frame/(.*\.txt)"
# If you use regular expressions, we aware that:
# - You MUST use "( )" aka "capture groups" !
# - While you can often get away "(.*)/output.jpg", in many cases you'd want "([^/]*)/output.jpg"
# - Parts of paths matched via regular expressions are not synced with other outputs. Prefer the ":name" syntax
# Eg if you ask also to visualize "(.*)/debug_output.jpg" and "(.*)/output.jpg"
# you will get two select inputs for the frame.

By default, only one viewer/path is shown at a time, and you get sliders/select to decide what to show:

Everything is synced

If you want, you can visualize all matching files:

# --snip--
- name: KPI reports
path: "reports/:report"
type: plotly/json
display: single # (default): will list views one after the other
# all # will render all matching paths/views
# viewer # let the viewer decide what to do... (EXPERIMENTAL)

Advanced Options [EXPERIMENTAL]

Custom Styles

You can style your visualizations:

outputs:
# define global or per-view styles
style:
# use any CSS properties
width: 500px
# the style will be applied to the outer-container
# and passed down to the viewers
detailed_views:
- name: My debug visualization
style:
width: 400px

Viewer Configuration

Some viewers can read extra configuration parameters from their configuration:

outputs:
detailed_views:
- name: My SLAM plot
type: 6dof/txt
show_3d: true

You can specify those parameters at "display-time" by defining controls:

outputs:
controls:
- type: toggle
label: Debug
name: show_debug
default: false