The default configuration for the models will not always work on a new dataset. Additionally, to increase performance, learning rate, optimizers, activation functions, etc. need to be tuned/selected. To make it easier to run such broad experiments that isolate the right combination of hyperparameters, Anomalib supports hyperparameter optimization using Comet or weights and biases.
A Sample configuration files for hyperparameter optimization with Comet is provided at
tools/hpo/configs/comet.yaml and reproduced below:
algorithm: "bayes" spec: maxCombo: 10 metric: "image_F1Score" objective: "maximize" parameters: dataset: category: capsule image_size: type: discrete values: [128, 256] model: backbone: type: categorical values: ["resnet18", "wide_resnet50_2"]
The maxCombo defines the total number of experiments to run. The algorithm is the optimization method to be used. The metric is the metric to be used to evaluate the performance of the model. The parameters are the hyperparameters to be optimized. For details on other possible configurations with Comet’s Optimizer , refer to the Comet’s documentation.
A sample configuration file for hyperparameter optimization with Weights and Bias is provided at
tools/hpo/configs/wandb.yaml and is reproduced below:
observation_budget: 10 method: bayes metric: name: pixel_AUROC goal: maximize parameters: dataset: category: hazelnut image_size: values: [128, 256] model: backbone: values: [resnet18, wide_resnet50_2]
The observation budget defines the total number of experiments to run. The method is the optimization method to be used. The metric is the metric to be used to evaluate the performance of the model. The parameters are the hyperparameters to be optimized. For details on methods other than
bayes and parameter values apart from list, refer the Weights and Biases documentation.
Everything under the
parameters key (in both configuration formats) overrides the default values defined in the model configuration. In these examples, only the dataset and model parameters are overridden for the HPO search.
You will need to have logged into a Comet or wandb account to use HPO search and view the results.
To run the hyperparameter optimization, use the following command:
python tools/hpo/sweep.py --model padim \ --model_config ./path_to_config.yaml \ --sweep_config tools/hpo/configs/comet.yaml
model_config is not provided, the script looks at the default config location for that model.
python tools/hpo/sweep.py --sweep_config tools/hpo/configs/comet.yaml