Skip to main content

Log with your W&B code

Your training already logs with Weights & Biases (wandb.init(), wandb.log()...). Log the same runs to QA-Board without changing that code: they show in the web app like the runs of qaboard.start_run(), compared with the reference's, while W&B keeps getting everything, or instead of W&B.

Experimental

The W&B shim is new: tell us what your code does that it doesn't handle. What QA-Board doesn't keep is said once when it happens, and listed when the run ends.

Run your script as it is​

python -m qaboard.integrations.wandb train.py --lr 1e-3

It runs train.py like python train.py --lr 1e-3 would: same arguments, same exit code. Each W&B run becomes a QA-Board run, which prints where its files are and its address in the web app.

  • With wandb installed, every call still goes to W&B, and is mirrored to QA-Board. If QA-Board's side fails, it says so once, and W&B's logging goes on.
  • Without wandb, import wandb in your script gets QA-Board's module: the runs go to QA-Board only. It says so once when it starts.

For a module, python -m qaboard.integrations.wandb -m my_package.train --lr 1e-3. You need what any run started from code needs: a project (a qaboard.yaml, or QA_PROJECT=group/repo), and a server to show it in the web app (log from your code).

Or add one line​

In a notebook, or when you don't start the script yourself, mirror W&B from the code, before wandb.init():

train.py
import qaboard.integrations.wandb
qaboard.integrations.wandb.mirror()

import wandb
run = wandb.init(project="detector", group="lr-sweep", config={"lr": 1e-3})

It does the same as python -m. Call it before from wandb import ...: names imported earlier aren't mirrored.

Or use QA-Board instead of W&B​

import qaboard.integrations.wandb as wandb

with wandb.init(project="detector", group="lr-sweep", name="coco-mini", config={"lr": 1e-3}) as run:
for step, batch in enumerate(loader):
wandb.log({"train/loss": train_step(batch)})
run.summary["mAP"] = evaluate(model)

The module has W&B's API, wandb doesn't have to be installed, and nothing is sent to W&B. run.qaboard is the QA-Board run, with its own methods, and run.dir its folder.

How runs are identified​

A QA-Board run is identified by its project, commit, batch, input and configuration (what makes a run): runs with the same input and configuration are compared from a commit to the next.

In W&BIn QA-Board
group=The batch
project=The batch, without group=. Also a parameter, wandb/project. QA-Board's project is your repository's: from the qaboard.yaml, or QA_PROJECT
name=The input. Without a name, the script's name: W&B's random names would never compare
config=, wandb.config.update(), wandb.config.x = ...The configuration, one mapping. Dictionaries, argparse namespaces, absl flags and YAML files
tags=, notes=, job_type=, entity=, id=Parameters: wandb/tags... They don't identify the run
resume="allow", "must", "auto", Truestart_run(resume=True): the run continues, with its steps
mode="offline", WANDB_MODE=offlineThe run stays on this machine, like share=False
mode="disabled", WANDB_MODE=disabledNo run
W&B's environment variablesWANDB_PROJECT, WANDB_RUN_GROUP, WANDB_NAME, WANDB_TAGS... are read like W&B does

The QA-Board run starts at the first call that logs something (log, summary, an artifact, run.dir, the end), not at wandb.init(): the config you set right after init() is part of its identity. Changes after that are kept as parameters, with a warning: they don't change which run it is.

  • Steps follow W&B's: each log() is a step, commit=False adds to the current one, step= jumps ahead, and a step lower than the current one is ignored.
  • Summaries: each series' last value is a metric of the run. define_metric("val/loss", summary="min") makes it the minimum, also with patterns ("val/*"), and run.summary["mAP"] = 0.41 sets a metric.
  • Several runs in a process: like W&B, wandb.init() gives the active run again in a script, and finishes it first with reinit=True or in a notebook.
  • Distributed trainings: only rank 0 logs to QA-Board (RANK, LOCAL_RANK). Other ranks say so once.
  • The end: wandb.finish(), the end of the with block, or the end of the process finish the run; an exception, or finish(exit_code=1), fails it.

What is kept​

Supported: wandb.init() and its run (name, id, config, summary, dir, url, step, a with block), wandb.log() of numbers (also numpy and PyTorch scalars, nested dictionaries), run.summary, wandb.config, define_metric(summary="min" | "max" | "mean" | "last" | "none"), wandb.finish(), resume, the modes. System metrics (CPU, memory, GPU) are sampled by QA-Board itself.

Every W&B feature, and what to do instead
W&BIn QA-BoardInstead
wandb.Image in log()Partial: saved as files, media/<key>/<step>.png, listed in Output Files. The web app doesn't show images per stepShow files on the run's card with visualizations. From arrays, needs Pillow. Boxes and masks aren't saved
wandb.Video, Audio, Object3D, Molecule in log()Partial: saved as files when made from a file. From arrays, not saved (with QA-Board's module)Write the file in run.dir
wandb.Html, wandb.Plotly, matplotlib figures in log()Partial: saved as .html, .plotly.json, .png files, which open in their viewers
wandb.Table, wandb.plot.* in log()Partial: saved as <step>.table.json files, not shown as tables or charts
wandb.Histogram in log()Partial: saved as <step>.histogram.json files, not plottedLog a few quantiles as series
Texts in log()Partial: the last one is a metric of the run. Series are numbers
Other values in log() (lists...)Not kept
config changed after the first logPartial: kept as parameters, not in the run's identitySet it in wandb.init(config=...)
A config that isn't a mapping, namespace or absl flagsNot keptPass a dictionary
define_metric(step_metric="epoch")Not supported: series have one x axis, the stepLog the x value as its own series
define_metric(summary="best")Partial: with goal="minimize" or "maximize", the min or max
define_metric(summary="first" | "copy"), several summariesNot supported: one summary per series, last, min, max or meanrun.summary["x"] = value
define_metric(hidden=True)Not supported: the series is shownsummary="none" keeps it out of the metrics
Summary values that aren't numbers, booleans or texts; deleting themNot kept
Renaming a run after it logged (run.name = ...)Not supported: the name is the input, fixed when the run startswandb.init(name=...)
Logging is_failed or compute_time, names over 200 charactersNot kept: QA-Board sets the first twoAnother name
log_artifact(), log_model(), wandb.ArtifactPartial: the files are copied in the run's artifacts/<name>/ folder. No versions, aliases or lineage
Artifact.add_reference(), Artifact.add()Not copiedadd_file(), add_dir()
use_artifact(), use_model(), artifact.download()Not supported: no artifact storeRead the files where they are
link_model(), the model registryNot supportedLink to your registry with badges
wandb.save()Supported: the files are copied in the run, again when it endsOr write them in run.dir
wandb.restore()Not supportedRead the files from the previous run's folder
wandb.watch()Not supported: no gradients or parameters histogramsLog their norms with wandb.log()
wandb.alert()Not supported: runs started from code send no alerts
wandb.sweep(), wandb.agent()Not supported (mirrored, each trial's run is a QA-Board run)qa optimize, or a loop over configurations
wandb.Api()Not supportedqaboard.runs(url)
log_code(), init(save_code=True)Not supported: the commit identifies the code, and a run whose files differ is marked as such
mark_preempting()Not supportedStart the run again with resume=True (resume)
init(fork_from=...), init(resume_from=...)Not supported: runs aren't forked or rewoundresume=True, or a new run
init(sync_tensorboard=True), monitor_gymNot supportedLog the values with wandb.log()
Other functions of wandb and of its runsQA-Board's module: they do nothing, said once
Framework loggers (Lightning, Transformers, Keras)With wandb installed and mirrored: what they log through W&B's API

Runs started this way have the limits of runs started from code.