Skip to main content

Logs and failed runs

When a run fails, or seems stuck, its log says why. The commit page's Logs view has the log of every run of the batch, updated live while they run, the cluster's report when it stopped a run, and the commands that started the batch.

A run failed: find out why​

Kaito changed how the inference gateway fills its batches. On his commit, the CI batch has a failed run: the bursty-high load profile.

  1. Open the commit. The sidebar's Logs item is red, with the number of failed runs, and a N failed message above the views lists them (when more than 5 failed, Show only the failed runs filters the page to them). Click Logs, or Open the logs in the message.

  2. Above the list of runs, click Failed to keep only the failed ones, then Open failed runs to show their logs.

  3. Each log opens on its last lines, where crashes usually are. Error lines are highlighted in red: click the N errors button to jump from one to the next.

  4. Search the log: type in Search…, then Enter for the next match. Here, panicked finds the panic, an index out of bounds in src/batcher.rs at line 163.

    The Logs view of the batcher change, its crashed bursty-high run open: searching the log for "panicked" highlights the panic, an index out of bounds in src/batcher.rs at line 163, then its backtrace.The Logs view of the batcher change, its crashed bursty-high run open: searching the log for "panicked" highlights the panic, an index out of bounds in src/batcher.rs at line 163, then its backtrace.
    Filtered on Failed, the failed run's log, searched for its panic: the match is highlighted, with the source line that raised it.
  5. If the cluster stopped the run, for instance at its memory limit, a red summary above the log says why and what to change: see LSF reports.

  6. Once it's fixed, redo the run from its ⋯ menu, or all the failed runs from the commit bar's menu: see Running from the web app.

Kaito's next commit, "batcher: fix off-by-one when the batch is full", brings the run back.

tip

A run is missing from the list? The Logs view only shows the runs left by the new bar's filter box: clear it.

Is it still running?​

While runs are pending or running, the Logs item says how many (for instance 1 running), and so do the counters under the filter box and a message above the views. Hover N running or N pending in the message to list the runs; Stop runs cancels them. The page refreshes by itself every 15 seconds.

In the Logs view, Running keeps the runs that are not finished (and Successful the ones that succeeded). Open one: its log has a Live tag and updates as the run writes it, following the end like a terminal. Scroll up to read in peace, Follow goes back to the end. New errors add to the count as they arrive.

A run still running: its log follows the end as lines arrive, the error count grows, and a click on it jumps to the first missed deadline.

A run stuck as pending or running whose job is gone never reports: its ⋯ menu's Mark as failed marks it as failed, so that you can redo it.

Reading a log​

Click a run to open it. Above its log, its input path and full configuration: hover a setting to copy it. In the log:

  • colors, line numbers and progress bars show as in a terminal; error lines are red, warnings orange;
  • N errors jumps to the next error;
  • Search… highlights the matches, with the previous and next ones (Enter, Shift Enter). With the log focused, Ctrl F (⌘ F on a Mac) goes to the search box;
  • the buttons on the right wrap lines, make the log taller, copy it, download it, or open the raw file;
  • when a run has several logs, switch between them at the top left: log.txt, and log.lsf.txt or log.dask.txt when the run used LSF or Dask. When the run was done again, the logs of its earlier attempts are there too, log.attempt2.txt, log.attempt1.txt, with their runners' logs, log.attempt1.lsf.txt: often the only place that says why an attempt failed, e.g. LSF's report of a job it killed because it used too much memory.

Open a run's logs to see what it used of its machine: 6.8 GiB peak memory · 1.0 core · GPU 0: 87% · 21 GiB / 40 GiB. Hover it for the details, and click Plot to see them over time (memory, CPU and GPUs).

Each run's line says how long it ran, and has buttons to copy the path of its output directory, open it in the file browser, and a ⋯ menu: its input's History, its batch, input path and database (click to copy), who ran it, its compute time, date, storage, peak memory and GPUs, and the actions to redo, mark as failed or delete the run. Hover the run's name to see the same, without the actions, like on the run cards. When the runs come from several batches or databases, their names start with them.

The Logs view's filters, All · 19, Failed · 1, Running · 0, Successful · 18, then a line per run: the 113 MP test wall first, failed, tagged "LSF: out of memory", then charts and photos, each with its configuration, how long it ran and its buttons.The Logs view's filters, All · 19, Failed · 1, Running · 0, Successful · 18, then a line per run: the 113 MP test wall first, failed, tagged "LSF: out of memory", then charts and photos, each with its configuration, how long it ran and its buttons.
Tobias's commit: the run LSF stopped comes first, tagged with why; the others say how long they ran. The filters above count each status.
Long logs, missing logs and other details
  • Very long logs show their last 50,000 lines, to keep the page fast: Show 50,000 more (or Show them) shows the rest. If the hidden lines have errors, or matches for your search, the notice says how many.
  • Large files: the web app first reads the last 512 KB of the file. Load everything reads the whole file. Until then, the lines have no numbers.
  • Live logs check for new lines every 2 seconds, and only download what was added.
  • No log yet: while a run is starting, the log says it will show as soon as the run writes it. A finished run without a log may have failed before it started, or had its files deleted.
  • log.lsf.txt shows up in the list once the LSF job has started.
  • Searches count up to 5,000 matches. Whether lines wrap is remembered for next time.
  • Big batches list their runs 100 at a time: Show 100 more below the list. Open failed runs opens the first 20.
  • The N failed message above the views links each failed run's name to its raw log, in a new tab.

When the cluster stopped the run: LSF reports​

On LSF, a job that goes over its memory or run time limit is killed by the cluster, often without a word in your own log. QA-Board reads LSF's job report, and tags the failed run with the reason, like LSF: out of memory, as soon as it's on screen.

Tobias made the camera ISP process each frame in one tile. On his commit, the run of the 113 MP test wall fails. Open it: a red summary says LSF killed the job at its memory limit (TERM_MEMLIMIT), compares the memory it used with the 3 GB it requested, and says what to change: ask for more memory in the batch's definition, with lsf: {max_memory: 8000} (in MB).

The Logs view with the 113 MP test wall's run open: tagged "LSF: out of memory", a red summary says the job was killed at its memory limit (TERM_MEMLIMIT), compares the memory it used with the 3072 MB it requested, and says how to ask for more.The Logs view with the 113 MP test wall's run open: tagged "LSF: out of memory", a red summary says the job was killed at its memory limit (TERM_MEMLIMIT), compares the memory it used with the 3072 MB it requested, and says how to ask for more.
LSF killed this 113 MP run at its memory limit: the summary compares what it used with what it asked for, and says what to change.

The summary also gives the job id, host, queue, run time, CPU time, threads, and when the job started and ended. The memory is orange when the run used 90% of what it asked for, or more. Click the tag to open log.lsf.txt: everything the job printed, including what doesn't go through Python, like subprocesses, native code and crashes, then LSF's report.

Tobias went back to 512×512 tiles: in one tile, the 113 MP wall needs more than the job's 3 GB. To change the memory, time or queue of a batch's jobs, see LSF options per batch.

What each reason means

When a job exits with an error by itself, there is no summary: its log says more than LSF.

TagLSF's reasonWhat to do
LSF: out of memoryTERM_MEMLIMITAsk for more memory in the batch: lsf: {max_memory: 8000} (MB).
LSF: time limitTERM_RUNLIMITUse a queue with a longer limit (lsf: {queue: ...}), or ask for more time (lsf: {options: -W 24:00}).
LSF: CPU time limitTERM_CPULIMITUse a queue with a longer limit.
LSF: too many threadsTERM_THREADLIMITAsk for more: lsf: {max_threads: 8}.
LSF: out of swap, too many processesTERM_SWAP, TERM_PROCESSLIMITThe run went over the swap or process limit.
LSF: killed by its ownerTERM_OWNERSomeone stopped the job, for instance by stopping the batch from QA-Board.
LSF: killed by an admin, preempted, requeuedTERM_ADMIN, TERM_PREEMPT, TERM_REQUEUE_*An administrator killed the job, a job with a higher priority preempted it, or it was requeued.
LSF: host overloaded, run window closed, deadlineTERM_LOAD, TERM_WINDOW, TERM_DEADLINEThe cluster stopped the job: because of the host's load, the queue's run window, or the job's deadline.
LSF: no working directoryTERM_CWD_NOTEXISTThe host can't see the working directory: check the shared filesystem.
LSF: job lost, killedTERM_ZOMBIE, TERM_UNKNOWN, TERM_EXTERNAL_SIGNALThe host may have gone down or rebooted, or the system's out-of-memory killer stopped the job.

Batches started from QA-Board​

When you redo runs or start tests and tuning from the web app, QA-Board runs qa batch for you: on a cluster, with Celery, or on the server. Until its runs show up, a message on every view of the commit page says how it's doing, with Show the logs:

  • Your batch is waiting for LSF: the job is queued, the cluster may be busy.
  • Starting your batch…: qa batch is running, its runs will show up.
  • The batch you started from QA-Board failed: why, e.g. qa batch exited with an error, or the cluster killed it, with the last error of its log.
  • qa batch didn't start any run: maybe there were no inputs, or all the runs already existed. To run them again, turn on Overwrite previous identical runs in the form.

They are listed with the other commands that started the batch.

The commands that started a batch​

At the top of the Logs view, Commands lists the qa batch (and qa optimize) commands that started the batch, the latest first, wherever they ran (Show N older for more than four). Each one says where it comes from: from QA-Board (with its status: Queued, Running, Failed, Done, Stopped), Terminal (qa batch) or CI, who started it and when, on which host or where it runs (e.g. an LSF job, a Kubernetes job, or the QA-Board server itself), and the batch files its batches come from. The buttons next to Commands copy the path of the batch's directory, or open it.

The Logs view of the tune-batching sweep: under "Started from QA-Board", the qa batch command kaito's tuning request ran, with its grid of max batch and max delay, and the end of its log; then the two runs left by the filter, and the Batch section with the command again.The Logs view of the tune-batching sweep: under "Started from QA-Board", the qa batch command kaito's tuning request ran, with its grid of max batch and max delay, and the end of its log; then the two runs left by the filter, and the Batch section with the command again.
A tuning sweep started from QA-Board: its qa batch command, with its own logs, then its runs.

Commands from a terminal or the CI show what was typed, to copy. Click a command to open it: the batches it ran, the aliases that reached them and the files they come from, and its log when QA-Board has it. Commands started from QA-Board also show where their batches are defined, linked to your git server at the commit or to Available Tests, the command QA-Board ran and its log, live while it runs; their folder buttons open the folder with the script QA-Board ran and its logs. On the right, where the logs of a command are:

  • Logs in the CI job: a link to the CI job that ran it.
  • Show its logs: QA-Board has them. The command was started from a terminal with qa batch (or qa optimize), which saves what it prints; or from QA-Board; or it's an older batch that logged in its directory's log.txt, shown as Batch logs.
  • Logs not saved: for instance, it was started from a terminal by an older version of qaboard, which only printed its logs there.

Whatever started the batch, each run's own logs are saved, in Runs below. With the Dask runner, Dask workers' logs opens the logs of the workers. For CI jobs you started from the Integrations menu, click their status tag to open the job's page and its logs: see Integrations.

Did the batches change?​

Results differ from the reference, and you wonder whether the batch itself changed: an input added, a configuration edited. When QA-Board saved the batches of two commands, Batches that ran, above the commands, compares them: the latest of this batch with the latest of the reference batch, else with the previous command of this batch. Pick other commands in its menus, from either batch.

It says Same batches, or what changed, like low-light changed, smoke added. Show the changes shows the two files side by side (or Inline). The comments at their top, like when each command started, are left out: they always differ. Batches and aliases compare by their content, whatever the order of their keys.

Run a batch again​

Batch files change: someone edits a batch, and the inputs or configurations a batch ran last week are gone. When a batch starts, qa batch and QA-Board save its batches as they were defined then, with only what it ran: the batches, chosen by name, pattern or alias (aliases of aliases too), and the aliases that reached them.

Click Run again… next to a command:

  • Open in Run Tests / Tuning fills in the form for this commit: the batch, the same tuning parameters and platform, and a new name (label-again). A message above the batch says it runs the batches as they ran; Use the batches as defined now goes back to the current definitions. Change what you want, then start.
  • From a terminal: download the batches as a YAML file, then copy the command that runs them, like it was started, or have QA-Board run it on the commit from its artifacts with qa batch --commit.
  • The file itself is at the bottom, to read what ran.
What is saved, and what isn't
  • The file stands alone: the defaults of each file (database, configurations, platform) are copied into its batches, YAML anchors are expanded, and a comment says which file each batch comes from. When several files define a batch, it keeps the one that ran (the last one).
  • Batches started from QA-Board read this file: it is batches.yaml in the folder with the script QA-Board ran. Commands from a terminal or the CI send it to QA-Board, up to 256 kB.
  • Names that are not batches are input paths: the file doesn't have them, the command does.
  • Your code, qaboard.yaml and the inputs aren't in it: running it on another commit runs that commit's code.
  • Commands started before 2026-12 don't have it: Run again… then uses the batches as they are defined now.
Where the log files are

Each run writes its logs in its output directory:

  • log.txt: what qa run and your code print, with every runner.
  • log.lsf.txt: on LSF, everything the job printed, then LSF's job report. See LSF integration.
  • log.dask.txt: with the Dask runner.
  • log.attempt1.txt, log.attempt2.txt...: the log.txt of the earlier attempts of a shared or CI run that was done again, the last 10.
  • log.attempt1.lsf.txt...: the runners' logs of those earlier attempts, their log.lsf.txt, log.dask.txt, log.k8s.txt or log.pending.txt.

Batches started from a terminal with qa batch save what qa batch prints in the batch directory, under qa-batch-logs/ (see shared results). In the CI, the CI job keeps them.

Where to go next​