Skip to main content

Your first review

Was this commit better? This page answers that question for one real commit of the demo data, in ten short steps. On the way, you meet the parts of the web app you'll use every day.

The camera team of Kitefin Robotics, a fictional company, works on kitefin/camera-isp: the image pipeline of their robot's camera, from the raw sensor data to the displayed image. Ines replaced its denoiser with a bilateral filter. Her commit said "+1.6 dB PSNR in low light", and she merged it into main. Check it yourself.

Before you start

You need a QA-Board server with the demo projects: run one on your computer, then open http://localhost:5151. Steps 7 and 9 need you to sign in: click Sign in (top right of Explore, or at the bottom of the sidebar) and sign in as ines, password kitefin-demo. Every person of the demo world has an account.

1. Open the project​

On Explore (Explore in the top bar, or the home page when you are signed out), click camera-isp in the kitefin group. You get its commits with results, on every branch, newest first, grouped by day. Signed in, the page starts with your own batches: click Everyone above the list to see every commit.

Explore: a filter box, then the kitefin group with its runs over 30 days, and its projects, each with its avatar, name, path, description, when it was last used, a sparkline of its runs, a star and a link to its source; the camera ISP pinned on top; "1 quiet subproject" below.Explore: a filter box, then the kitefin group with its runs over 30 days, and its projects, each with its avatar, name, path, description, when it was last used, a sparkline of its runs, a star and a link to its source; the camera ISP pinned on top; "1 quiet subproject" below.
Click a project's name to see its commits. The star pins a project on top of the list.

The list shows the last 3 days at first: the demo history ends when your QA-Board first started, so you see its latest commits. If it started days ago, the list may be empty: the search in step 2 looks at every date. More in Finding results.

2. Find the commit​

  1. Type bilateral in the Search commits box at the top right. After a moment, the list shows every commit that mentions it, on any branch and at any date.
  2. Find Merge branch 'isp/bilateral-denoise', on main, and click 26 runs on the right of its row.

The commit page opens. More in Searching commits.

3. Compare it with the release​

The page has two bars at the top: the commit you opened, tagged new, and below it the reference it's compared with, tagged ref. By default, the reference is the latest commit with results on the project's reference branch, main. For a commit in the past, that's a later one. To judge the denoiser, compare it with the release before it, which the team saved as a milestone:

  1. In the reference bar, click Change.
  2. Under Shared with the project, click isp-4.1.

The reference bar now shows the release, isp: release 4.1, gaussian denoise and detail boost 1.2, with a crown: it's a milestone shared with the project. Tables and viewers now call it isp-4.1.

More in Picking a reference.

4. Read the metrics​

The page opened on the Summary: each main metric's median over the runs, new against reference, with its change, green when it gets better and red when it gets worse. Medians say that something moved. To see which runs moved:

  1. Click Metrics Diff in the sidebar: one row per run, the change of each metric.
  2. Type base linux in the Filter runs… box of the new commit's bar. The table keeps the 10 runs at ISO 100, on the float pipeline (the hw-model platform runs the camera chip's fixed-point model).

On the facade photo, PSNR is up 1.34 dB (5%). On the slanted-edge chart, PSNR is up too, but Overshoot is up 163%, in red: the sharpened edge now overshoots by 11% of its step, against about 4% in the release. That's what halos look like in numbers.

Narrowed down further to the facade and the slanted edge: PSNR is up, and so is the edge overshoot. Then the facade's images show why, as the next steps do.

More in Reading metrics.

5. Look at the images​

  1. Click Visualizations in the sidebar, and type facade before base linux in the filter box.
  2. On the card of photos/facade.png, click the preset Window frame (halos), then scroll on an image to zoom and drag it to move.

Each card is one run: its main metrics against the reference, then its images, new and reference side by side. When you zoom into one image, the other follows.

A run card of the facade photo: PSNR, SSIM and runtime with their changes against the reference, preset regions of interest, and the new and reference images side by side, zoomed on a window.A run card of the facade photo: PSNR, SSIM and runtime with their changes against the reference, preset regions of interest, and the new and reference images side by side, zoomed on a window.
The facade's card: PSNR and SSIM up against the release, the regions the team saved for this photo, then the new and reference images, zoomed in together.

More in Synced zoom and pan.

6. Show their difference​

  1. In the card's toolbar, click Difference. Their difference shows between the images: small differences, like the noise the denoisers leave, color it everywhere.
  2. In the menu next to Perceptual, pick 20%.

Small differences fade. What's left outlines the window frame: the halos that PSNR missed.

The facade window, new and reference on each side of their perceptual difference: the window frame's edges stand out, from yellow through orange and purple to black where they differ most, the flat wall fades to white. Above them, the comparison modes and the difference's settings.The facade window, new and reference on each side of their perceptual difference: the window frame's edges stand out, from yellow through orange and purple to black where they differ most, the flat wall fades to white. Above them, the comparison modes and the difference's settings.
At 20%, the flat wall fades to white: only the edges of the window frame, where the halos are, stay colored.

More in The difference.

7. See which runs changed most (optional)​

Did other images change as much?

  1. Remove facade from the filter box, so the page shows the 10 runs again.
  2. On any card, open the regions menu at the right of the toolbar (it says Presets), and click Where they differ, in all 10 runs…

The server compares each run's images with the reference's, and ranks the runs. Show scrolls to a run's card and zooms on the regions found.

The list of a commit's runs ranked by how much their output images differ from the reference's, with a bar, a percentage and the number of regions found for each; the 113 MP test wall, a TIFF, has no out.png to compare.The list of a commit's runs ranked by how much their output images differ from the reference's, with a bar, a percentage and the number of regions found for each; the 113 MP test wall, a TIFF, has no out.png to compare.
The Siemens star, the backlit window and the facade changed most; the flat gray chart barely moved.

More in Which runs changed most.

8. Find when it started​

  1. On the card of photos/facade.png, hover its name, then click History (this input over time). History opens: the facade's metrics over the commits of main, from 3 days before the merge to today.
  2. Hover the points of the PSNR chart, from right to left.
The menu of a run's name, photos/facade.png, hovered in its run card: History, this input over time; its batch, ci; its database and input path; its metadata; and Run Info: sent by ci, 1 s of compute time, created 20 days ago, and its storage.The menu of a run's name, photos/facade.png, hovered in its run card: History, this input over time; its batch, ci; its database and input path; its metadata; and Run Info: sent by ci, 1 s of compute time, created 20 days ago, and its storage.
The menu of a run's name: History is at the top.

Below the chart, the hovered commit shows as New and the commit before it as Reference, with their images. Where the PSNR jumps, the new commit is the merge, and the leaves get halos.

PSNR of the facade photo over six weeks of main, a point per commit, with a jump at the merge of the bilateral denoiser. Below, that commit and the one before, and their output images: only the new one outlines the leaves.PSNR of the facade photo over six weeks of main, a point per commit, with a jump at the merge of the bilateral denoiser. Below, that commit and the one before, and their output images: only the new one outlines the leaves.
The facade's PSNR jumps at the merge of the bilateral denoiser; hovering that point compares its images with the commit before: only the new one outlines the leaves.

Press your browser's Back button to return to the commit page. More in History.

9. Save it as a milestone​

A milestone keeps these results, so that you can compare the fix with them later.

  1. Clear the filter box (its ✕): a milestone keeps the batch and the filter you're looking at.
  2. Click the star next to the new commit's hash.
  3. Name it, like bilateral, with halos, leave Only me selected, and click Save milestone.

The star turns gold, and the name shows next to it. The milestone is saved in your account: you'll find it under My milestones, in the Change menus and in the sidebar's Milestones. More in Milestones.

10. Share what you found​

Copy the page's link from your browser, and paste it in your bug report or chat. The link keeps the commits, the reference, the filter, the view and how images are compared, but not the zoom: whoever opens it gets the same comparison, even after main moves on. To point at a spot, paste a copy of the image too (the copy button above the images), or name the region, e.g. Window frame (halos).

To share the milestone, click its star again, then Copy link. More in Sharing.

Check the fix​

Ten days later, Ines merged a fix: near strong edges, the sharpening now stays within the range of the neighboring pixels. Compare it with your milestone:

  1. Click the project's name, camera-isp, in the sidebar: its commits show again. Search for halo-clamp, and open the runs of Merge branch 'isp/halo-clamp'.
  2. In the reference bar, click Change, then your milestone under My milestones.

The edge overshoot is back to about 5%, and on the facade, the dark rims around the leaves fade.

Leaves in front of the facade, after the bilateral denoiser: a dark rim outlines every leaf against the wall.Leaves in front of the facade, after the bilateral denoiser: a dark rim outlines every leaf against the wall.
The same leaves once the sharpening is clamped: the edges stay crisp, with only faint rims.The same leaves once the sharpening is clamped: the edges stay crisp, with only faint rims.
Drag the divider over the leaves: the dark rims fade.

Try the other demo projects​

Each demo project has a story of its own. Its use case page says what to look for.

What else you can do​

  • Compare with several references at once, like isp-4.1 and the latest main: hover the reference bar, then Stack a reference. See Several references.
  • Run more tests, or tune parameters, from the web app, on any commit: see Running from the web app.
  • Find why a run failed: on the camera project, search one tile, open isp: process frames in one tile, then its Logs: the 113 MP test wall's run was killed at its 3 GB memory limit. See Logs and failed runs.
  • Compare videos frame by frame: on the merge commit, against isp-4.1, pick the video batch in both bars. See The video viewer.
  • Put your own project on QA-Board, and fail CI when a metric gets worse: see the tutorial and the quality gate (experimental).