Skip to main content

Log from your code

Create a run from code that is already running, a training loop, a notebook, an evaluation script, and log to it as you go. The run shows in the web app like the runs of qa run: compared with the reference, with its metrics, its files, and the curves of what it logged at each step, while it runs.

Experimental

qaboard.start_run() is new: its arguments and the files it writes may change. Tell us what your training code needs. What it doesn't do yet is at the end of this page.

Use qa run when QA-Board starts your code on each input of a batch (Running your code). Use start_run() when your code starts by itself and results appear while it runs. When the results are already files, like a test report, qa upload saves them as a run from the shell.

Log a training​

In a project (a folder with a qaboard.yaml, see Start a project):

train.py
import qaboard as qb

with qb.start_run(input="train/coco-mini", configs=["resnet50", {"lr": 1e-3}], batch="lr-sweep") as run:
for step, batch in enumerate(loader):
loss = train_step(batch)
run.log({"train/loss": loss, "lr": scheduler.lr}, step=step) # a point of each series

run.log_metrics({"mAP": evaluate(model)}) # the run's metrics
torch.save(model.state_dict(), run.output_dir / "model.pt") # its files

Run it with python train.py. It prints where the run's files are and its address in the web app. Open it: in the Visualizations view, the run's card plots train/loss and lr as they are logged, with the reference run's curves dashed on the same axes. The Summary view plots all the runs of the batch together, one color per run (Logged series in the user guide). Leaving the with block ends the run; an exception ends it as failed, with the error in its log.

You need what qa --share needs: the qaboard package, a server (QABOARD_URL, or your site's package), and to be signed in when the server asks for it (qa auth login, or QABOARD_TOKEN). The run's files are saved in the shared storage (storage in qaboard.yaml), or sent to the server without one. qa doctor checks them. Without a server, the run stays in the project's output/ folder: see where the files go.

What makes a run​

A run started from code is identified like any run: a project, a commit, a batch, an input and configurations. Runs with the same input and configurations are the same test, compared from a commit to the next and with the reference. If you come from an experiment tracker:

You knowIn QA-BoardWith start_run()
ProjectThe projectThe one of the qaboard.yaml above the current folder, or project="group/repo" without one
Experiment, groupA batchbatch="lr-sweep". Outside the CI, your runs go to your own batch, shown as yours, like with qa --share
Run nameThe input and the configurationsinput="train/coco-mini": what the run works on, a dataset or any name. Default: the script's name
Config, hyperparametersConfigurationsconfigs=["resnet50", {"lr": 1e-3}]: names and mappings, like qa --config. run.params merges the mappings
Code versionThe commitThe git HEAD, or the CI's commit. A run whose tracked files differ from the commit is recorded as such (data.dirty)
Logged values, log()Seriesrun.log({...}, step=...)
SummaryMetricsrun.log_metrics({...}), and each series' last value

Two runs with other configurations are two runs, shown side by side in the batch. Starting the same run again is a new attempt of it: see run again, or resume.

Log values, metrics, parameters and files​

Series: values at each step​

run.log({"train/loss": 0.31, "lr": 1e-3}, step=120)
run.log({"val": {"loss": 0.42, "top1": 0.71}}, step=120) # nested mappings: val/loss, val/top1
  • Values are numbers, including numpy and PyTorch scalars. Without step, steps count the calls. A value that isn't a number is skipped, with a warning the first time, and None silently: a training doesn't end for that.
  • Each series is plotted on the run's card, grouped by what comes before the / of its name.
  • Each series' last value is also a metric of the run, shown in the tables and on the History page, while the run goes on and after. Choose another summary with start_run(summary={"val/loss": "min"}): last, min, max, mean, or none to keep a series out of the metrics. Names can be patterns: {"val/*": "min", "system/*": "none"}, and "*" sets the default.
  • NaN and infinities are kept in the series, where they show as gaps, and never become the metric.
  • run.log() never waits for the server: it writes to memory, and a background thread writes the files every 2 seconds.

Metrics​

from qaboard import Metric

run.log_metrics({"mAP": Metric(0.41, smaller_is_better=False, target=0.4), "verdict": "ship"})

Metrics are the run's results: numbers, booleans, short texts. Declare how to show them with Metric(...), define_metric() or your metrics file, as for qa run (Metrics). Metric(...) works in run.log() too, for the metric a series gives.

Parameters​

run.log_params({"device": "cuda:0", "weights": 25_557_032})

The web app shows them with the run's configuration. Unlike configs, they don't identify the run: use configs for what you chose, and log_params() for what the run found out.

Files​

Write the run's files in run.output_dir, or copy them there with run.log_artifact():

(run.output_dir / "checkpoints").mkdir(exist_ok=True)
torch.save(checkpoint, run.output_dir / "checkpoints" / "last.pt") # big files: write in place
run.log_artifact("predictions.json", predictions_path) # copies a file
run.log_artifact("report.html", html.encode()) # or writes bytes
run.log_artifact(confusion_matrix_path) # a file alone keeps its name

The web app lists them in Output Files and opens them with the viewer their name picks: images, plots, text... To show a file on the run's card, list it in outputs.visualizations. What the process prints is saved in the run's log.txt, shown in Logs.

Where the files go, with and without a server​

FilesWeb app
In the CI, or with a server, and a storage in qaboard.yamlIn the shared storage, where qa --share run would save themThe run shows when it starts
In the CI, or with a server, without storage (or storage: upload)In the project's output/ folder, and sent to the server: the log and the series as the run goes, the rest when it endsThe run shows when it starts
share=False, QA_OFFLINE=1, or no serverIn the project's output/ folder, like a local qa runNot shown
output_dir="..."In your folder. The server must read it to show the filesAs above

A run's folder has what qa run writes (run.json, metrics.json, log.txt, manifest.outputs.json), your files, and its series: series.jsonl with everything logged, and series.json, a version for plots that stays small however long the run (the format). Read a run's series back with run.series("train/loss"), also from the runs of qaboard.runs(url).

When the server doesn't answer, the run goes on: it warns once, keeps writing its files, and asks again later. The run tells the server what it looks like now, never a list of events: the first message that gets through is enough, and finish() sends everything once more. If the server refuses the run (no token, no role on the project), it says why when the run starts, before hours of training.

Run again, or resume​

Starting a run with the same input and configurations, on the same commit and batch, is a new attempt of that run: it gets its own folder, the web app shows the latest attempt with an attempt N tag, and keeps the earlier attempts' logs and metrics (When a run is done again). The earlier attempt's other files are removed.

To continue a run instead, after a preemption or to train longer, pass resume=True: the run keeps its folder, its files, the metrics and parameters it logged, and its steps go on from the last one logged. Without a previous run, it starts one.

run = qb.start_run(input="train/coco-mini", configs=configs, resume=True)
checkpoint = run.output_dir / "checkpoints" / "last.pt"

To keep several trainings side by side, make them different runs: put the seed or the trial number in configs.

Without a with block, in notebooks and libraries​

run = qb.start_run(project="kitefin/detector", input="explore-augmentations")
run.log({"loss": 0.3})
run.finish() # or run.fail("the loss diverged")
  • Runs nobody finished end with the process: as failed when it dies from an exception, or when the job is killed with SIGTERM, as schedulers do at a time limit.
  • Starting the same run again while it goes on, like a notebook's cell run twice, finishes the first one.
  • Logging to a finished run is skipped, with a warning the first time.
  • qb.current_run() gives the run started last, for code deep inside a library: if run := qb.current_run(): run.log(...).
  • Outside a project, give project=. Without a git repository, the run is attached to a placeholder commit with your name. In a CI job, the project is the job's repository. The files go to the storage that QA_STORAGE or your site's package sets, else they are sent to the server.
  • Inside qa run, start_run() starts another run: give it another input. It refuses to start the run that qa run is doing, which would remove its files: return metrics from run() instead.
  • With several processes (one per GPU), call start_run() in one of them only, e.g. on rank 0: each call starts a run. A forked worker that logs to its parent's run is ignored, with a warning.
  • commit= attaches the run to another commit than the code's, e.g. the commit that trained the model you evaluate.
Every argument of start_run()
ArgumentDefaultWhat it does
inputThe script's nameWhat the run works on: a path relative to the inputs' database, or any name.
configs[]The configurations: names and mappings.
batch"default"The batch. Outside the CI, runs go to your own batch of that name.
labelThe batch too, under the name qa --label gives it: give either. With both, label is the batch in the web app, and batch names the group of tests on the run's card, like qa --label nightly batch unit-tests.
projectThe qaboard.yaml'sRequired without a qaboard.yaml, except in a CI job: its repository.
commitThe git HEAD, or the CI'sA full commit id, or a branch, tag or short id of the folder's repository.
shareIn the CI, or with a serverFalse keeps the run local. QA_OFFLINE=1 does it for a whole process.
output_dirFrom QA-Board's conventionsYour own folder for the run's files. Only a previous run's series are removed from it.
resumeFalseContinue the previous run of the same identity.
summary"last"Which value of each series is the run's metric: last, min, max, mean, none, or a mapping by series.
databaseThe project'sThe inputs' database, when input is a path under another one.
platformThis machine'slinux or windows.
capture_outputTrueSave what the process prints in the run's log.txt.
server, tokenThose qa usesAnother server, or a token.

What it doesn't do yet​

  • Compare runs across batches or commits on one plot. The Summary view plots the runs of one batch together, up to 40, and each with its reference run.
  • Images per step, system metrics (GPU, memory), alerts, a model registry. Save images as files of the run; link to your registry with badges.
  • Sweeps. Loop over configurations in your script, or use automated tuning with qa run.
  • Sending a run later when the server was away for the whole run: its files are complete, but only a run that reaches the server once shows in the web app.
  • A server from before this feature shows the run and its files, and its metrics when it ends. The curves and the metrics of running runs need a server of November 2026 or later.

More on the roadmap.