Visualizing your algorithm's outputs

The run()function wrapping your code receives an output_directory where it can create all kinds of files. Usually, you only want to look at a few of those files, the rest being debug data. Visualizations help you declare pre-sets of relevant files.

Here is a simple example, assuming your code an image named output.jpg.

qaboard.yaml

outputs:
    visualizations:
    - path: output.jpg

note

For now, to debug your visualizations, you have to commit and push your new qaboard.yaml. We plan on letting you edit simply qaboard.yaml locally, and update the visualizations when you use qa --share.

You can provide multiple relevant files, and hide debug visualizations by default:

qaboard.yaml

outputs:
    visualizations:
    - path: output.jpg
    - path: debug.jpg
      default_hidden: true
      # type: image/jpg # auto-guessed

Users will get switches to toggle debug visualizations:

Available file viewers

QA-Board tries to guess the right image viewer depending on the file extension or a type

File	Viewer Type	Viewer
*.jpg*, *.png*, *.bmp*, *.tif*, *.pdf*...	image/*	Image
*.plotly.json	plotly/json	Plot.ly
*.flame.json	flame/json	Flame Graph, diffable
*.html	text/html	HTML (assumes trusted input..!)
*.mp4	video/*	Video (synced)
*.txt, unidentified	text/plain*	Text (diffs, with VSCode's Monaco Editor)
pointcloud/txt	pointcloud viewer (needs to be refactored, coupled to a specific internal project...)
6dof/txt	6DoF viewer (needs to be refactored, coupled to a specific internal project...)

Image viewer

• Supports all common image formats.
• Fast and smooth zoom & pan, synced. Fast image streaming via IIIF.
• Perceptual color difference.
• Color tooltip.
• Image filters (exposure, contrast, gamma...).
• Histograms per channel.
• Automatic regions of interest.

tip

If your configurations or input metadata contain roi: [{x, y, w, h, label}], those "regions of interest" will be displayed and easily selectable.

Regions of interest

The image viewer can thumbnails to navigate between Regions of Interest. To enable this, you must provide as [metadata-integration-external-databases](input metadata) some info. With YAML syntax:

width: 800   # optional, useful if your results are showed with downscaled resolution
height: 600  # optional, idem
roi:
- {x: 1408, y: 244, w: 121, h: 97, label: "Gradations and overshoots"}
- {x: 455, y: 651, w: 58, h: 58, label: "False colors"}
# you can override give the image dimension per ROI:
- {image_width: 1600, image_height: 1200, x: 351, y: 567, w: 45, h: 53, label: "Details fine line"}

If you click on the histogram button, you'll be able to select a region of interest and easily copy the format above:

Plot.ly viewer

The Plotly library has everything you need from bar charts to 3d plots.

• huge variety of plots
• interactive plots
• easy-ish to use with binding to python/JS/matlab...
• web-based
• open-source and popular
• performant

All you need is to save your plot data as JSON.

import plotly.graph_objects as go
fig = go.Figure(data=go.Bar(y=[2, 3, 1]))
with open('graph.plotly.json', 'w') as f:
  spec = fig.to_json() # '{"layout": {...}, "data": [{...}, {...}, ...]}'
  f.write()

Text Viewer

Flame Graphs

We love Brendan Gregg's flame charts and integrated Martin Spier's d3-flame-graph. At a glance, you can check where you code spends its CPU cycles, and use differential flame graphs to debug regressions.

For a tutorial read our blog post.

More Viewers?

Tell us what you need! The next we'll implement is likely vega specs. It would notably allow us to display altair visualizations. We could also have a minimal integration with notebooks, visdom, webiz...

Dynamic visualizations

You can use a special syntax to create dynamic visualizations at display-time. Users will be able to choose what to display using sliders / select options:

qaboard.yaml

outputs:
    visualizations:
    - name: Movie Frames
      # you can use the `/user/:name` syntax to match part of filenames
      path: ":frame/output.jpg"
      # you can match part of filenames (experimental)
      path: ":frame/frame_:number.jpg"

      # For more examples, the full syntax is available at:
      # https://github.com/pillarjs/path-to-regexp

You can also use regular expressions (inside parentheses!) to match which output files you want to view:

outputs:
    visualizations:
      # A common use case is matching file extensions
      path: "(.*\.jpg)"
      # ... or parts of filenames
      path: "(debug_.*\.jpg)"
      # you can mix with the previous syntax
      path: ":frame/(.*\.txt)"

      # If you use regular expressions, we aware that:
      # - You MUST use "( )" aka "capture groups" !
      # - While you can often get away "(.*)/output.jpg", in many cases you'd want "([^/]*)/output.jpg"
      # - Parts of paths matched via regular expressions are not synced with other outputs. Prefer the ":name" syntax
      #   Eg if you ask also to visualize "(.*)/debug_output.jpg" and "(.*)/output.jpg"
      #   you will get two select inputs for the frame.

By default, only one viewer/path is shown at a time, and you get sliders/select to decide what to show:

If you want, you can visualize all matching files:

   # --snip--
    - name: KPI reports
      path: "reports/:report"
      type: plotly/json
      display: single  # (default): will list views one after the other
             # all     # will render all matching paths/views
             # viewer  # let the viewer decide what to do... (EXPERIMENTAL)

Advanced Options [EXPERIMENTAL]

Custom Styles

You can style your visualizations:

outputs:
    # define global or per-view styles
    style:
        # use any CSS properties
        width: 500px
        # the style will be applied to the outer-container
        # and passed down to the viewers
    detailed_views:
    - name: My debug visualization
      style:
        width: 400px   

Viewer Configuration

Some viewers can read extra configuration parameters from their configuration:

outputs:
    detailed_views:
    - name: My SLAM plot
      type: 6dof/txt
      show_3d: true

You can specify those parameters at "display-time" by defining controls:

outputs:
  controls:
    - type: toggle
      label: Debug
      name: show_debug
      default: false