Run batches on an LSF cluster

With the lsf runner, qa batch sends each run to your IBM Spectrum LSF cluster as a job (bsub), and waits for them to end. The cluster's nodes must mount your inputs and the results storage at the same paths as the machine where you start qa batch. For other runners, see runners.

Send your batches to LSF

1. Make lsf the default runner in qaboard.yaml, and give it your queue:

   qaboard.yaml

   runners:
     default: lsf
     lsf:
       queue: <your-queue>

   <your-queue> is one of your cluster's queues (bqueues lists them). Without queue, jobs go to the cluster's default queue.

2. Run a batch. qa sends its runs as LSF jobs, then waits for them to end:

   qa --share batch ci

   QA_BATCH_VERBOSE=1 also prints each bsub command.

Or keep the local runner by default, and choose LSF for one command: qa batch --runner lsf ci.

Jobs get your QA-Board project's name as LSF project (-P). Every setting is optional:

qaboard.yaml

runners:
  lsf:
    queue: <your-queue>     # default: the cluster's default queue ($LSB_DEFAULTQUEUE, or DEFAULT_QUEUE in lsb.params)
    # project: my/project   # default: your QA-Board project's name
    # priority: 2000        # bsub -sp. Default: 2000, or 1000 for tuning runs
    # max_threads: 8        # threads per job (0: LSF's default)
    # max_memory: 8000      # memory per job, in MB (0: LSF's default)
    # resources: "select[mem>8000]"  # bsub -R
    # options: "-W 24:00"   # other bsub options, added after all the others

tip

Your organization's site package can set defaults for all projects, e.g. runners.default: lsf and the queue. See Site defaults.

Limit how many run at the same time

By default all the runs from a qa batch are sent at once. To limit how many run at the same time:

qaboard.yaml

runners:
  lsf:
    concurrency: 20              # 0=unlimited (default)
    # concurrency_strategy: array  # default

or qa batch --lsf-concurrency 20 my-batch. A batch can have its own concurrency (and concurrency_strategy) under lsf:, see LSF options per batch: its runs get their own job arrays, or waves, with their own limit. The command line option wins.

Two strategies are available:

• array (default): runs with identical LSF options are sent as a single job array with a slot limit (-J "name[1-n]%20"). It's a sliding window, and a single bsub call. If runs need different LSF options (e.g. per input type), each group gets its own array and its own limit.
• waves: runs are sent by groups of concurrency jobs, each group waiting for the previous one to end (-w "ended(...)"). Slow runs delay the next wave. Use it only if job arrays don't work for you.

In both cases job names keep the batch prefix, so stopping a batch from QA-Board still works.

Logs

Each run has 2 log files in its output directory:

• log.txt: what qa run and your code print from Python, written live. You get it with every runner.
• log.lsf.txt: everything the LSF job printed, written live, including output that doesn't go through Python (subprocesses, native code, crashes). When the job ends, LSF appends its job report: exit code, why LSF terminated the job (e.g. TERM_MEMLIMIT), the host, run time, memory usage...

In the web app's Logs view:

• Failed runs get a tag explaining why LSF ended them (e.g. "LSF: out of memory"). Above the logs, a summary of LSF's job report says what to do about it, and shows the host, run time, and memory used vs requested.
• You can switch between log.txt and log.lsf.txt, search (Ctrl/Cmd+F, then Enter), and jump to errors.
• While runs are running, logs update live. The web app only downloads what was added since the last update, and for large logs, only the end of the file at first.

LSF killed this run at its memory limit (TERM_MEMLIMIT): the summary compares the memory it used with what it requested, and says how to ask for more.

note

With job arrays (see above), all runs share the same bsub -o file name. To get each run's job report in its own log.lsf.txt, qaboard creates symlinks to the runs' output directories in {batch_dir}/lsf/{job_array_name}/{index}, and uses -o "{batch_dir}/lsf/{job_array_name}/%I/log.lsf.txt".

Many short runs? Use a pool of workers

Each run is an LSF job, and pays LSF's scheduling overhead. If you have many short runs, run them instead on a few long-lived workers, themselves started as LSF jobs: see Dask.

Settings per batch and per input

A batch can change the LSF settings of its runs, and an input its own run's:

qa/batches.yaml

big-images:
  lsf:
    max_memory: 10000
    max_threads: 4
  configs: [base]
  inputs:
  - images/A.jpg
  - images/B.jpg

qa/batches.yaml

small-images:
  lsf:
    max_memory: 1000
  configs: [base]
  inputs:
    images/A.jpg:
    images/B.jpg:
      lsf:
        max_memory: 200

See runner settings per batch for which settings win.

On the command line

Options of qa batch override the settings of qaboard.yaml and of the batches:

qa batch --help

[...]
  --lsf-max-threads INTEGER    restrict number of lsf threads to use. 0=no
                               restriction
  --lsf-max-memory INTEGER     restrict memory (MB) to use. 0=no restriction
  --lsf-queue TEXT             LSF queue (-q)
  --lsf-resources TEXT         LSF resources restrictions (-R)
  --lsf-priority INTEGER       LSF priority (-sp)
  --lsf-options TEXT           Other LSF options (as 1 string, like '-W
                               24:00') that bsub can understand. Will be added
                               after all other CLI flags.
  --lsf-concurrency INTEGER    Max number of LSF jobs from this batch running
                               at the same time. 0=unlimited

note

--lsf-threads was renamed --lsf-max-threads, and --lsf-fast-queue was removed.

Send jobs through a bridge host

Where bsub isn't available, qa can send the jobs through a host that has it, e.g. from the QA-Board server, which runs in a container, to start tuning from the web app, or from Windows. Give the command that runs {bsub_command} there, in an environment variable:

export QA_RUNNERS_LSF_BRIDGE='ssh <lsf-host> {bsub_command}'
export QA_RUNNERS_LSF_BRIDGE='ssh <lsf-host> su {user} {bsub_command}'   # as the user who started the runs

<lsf-host> is a host that can send LSF jobs. The jobs run in the folder where you started qa batch: qa gives it to LSF, so the bridge doesn't need to cd there.