Skip to main content

Auto-Optimization

qa optimize​

EXPERIMENTAL: This feature is experimental and the API is subject to change at any time.

Specifically, we may change the solver backend to nevergrad, offer multiple choices, etc.

It uses scikit-optimize, which is installed with qaboard.

qa optimize --help
qa --label my-optim --share optimize --batch my-batch --config-file optim.yaml --checkpoint optim.pkl

The configuration file sets the objective, the number of evaluations, the search_space, and optional solver settings or fixed preset_params. The web application's tuning page shows a commented template for it ("optimize" search type), and runs qa optimize for you.

The Automated tuning tab of the Run Tests / Tuning form, with the int8 model's tuning in YAML: 20 evaluations, the solver's seed and 8 initial points, then the objective, AP small and mAP; the Send button below.
Automated tuning from the web app: the search space and the objective in YAML.

Useful options:

  • --checkpoint: the optimizer's state is saved there after each iteration, and loaded if it exists, to restart interrupted optimizations. (At the moment you need to give it.)
  • parallel_sampling in the configuration file, or --parallel-param-sampling: how many parameter sets are evaluated at the same time (default 1).
  • keep_all_best_iters: true: by default the results of the previous "best iteration" are deleted when a better one is found. Use it to keep them.
  • QA_SEED: the random seed (default 101), for reproducible optimizations.

Each iteration runs qa batch with the label my-optim|iterN. During the optimization, qa sets QA_OPTIM_DIR to the experiment's batch directory, where convergence plots are saved.

Select the experiment's batch on the commit page, then the Analysis view:

The Analysis of the tune-int8 experiment: the best parameters found at iteration 10 of 20 (percentile 99.999, nothing kept in fp32, score 0.2, NMS 0.4) with their mAP, AP small and model size; parallel coordinates of the 20 iterations; the convergence of the objective.
The Analysis of an automated tuning: the best parameters, every iteration in parallel coordinates, and the convergence.

WIP: We're working on section about "auto-optimization". Stay tuned!