Anomaly Detection Tutorial#
This tutorial demonstrates how to train, evaluate, and deploy a classification, detection, or segmentation model for anomaly detection in industrial or medical applications. Read Anomaly Detection for more information about the Anomaly tasks.
Note
To learn more about managing the training process of the model including additional parameters and its modification, refer to Object Detection model.
The process has been tested with the following configuration:
Ubuntu 20.04
NVIDIA GeForce RTX 3090
Intel(R) Core(TM) i9-11900
CUDA Toolkit 11.8
Setup the Virtual environment#
1. To create a universal virtual environment for OpenVINO™ Training Extensions, please follow the installation process in the quick start guide.
2. Activate your virtual environment:
.otx/bin/activate
# or by this line, if you created an environment, using tox
. venv/otx/bin/activate
Dataset Preparation#
1. For this example, we will use the MVTec dataset.
You can download the dataset from the link above. We will use the bottle category for this tutorial.
2. This is how it might look like in your file system:
datasets/MVTec/bottle
├── ground_truth
│ ├── broken_large
│ │ ├── 000_mask.png
│ │ ├── 001_mask.png
│ │ ├── 002_mask.png
│ │ ...
│ ├── broken_small
│ │ ├── 000_mask.png
│ │ ├── 001_mask.png
│ │ ...
│ └── contamination
│ ├── 000_mask.png
│ ├── 001_mask.png
│ ...
├── license.txt
├── readme.txt
├── test
│ ├── broken_large
│ │ ├── 000.png
│ │ ├── 001.png
│ │ ...
│ ├── broken_small
│ │ ├── 000.png
│ │ ├── 001.png
│ │ ...
│ ├── contamination
│ │ ├── 000.png
│ │ ├── 001.png
│ │ ...
│ └── good
│ ├── 000.png
│ ├── 001.png
│ ...
└── train
└── good
├── 000.png
├── 001.png
...
Training#
1. For this example let’s look at the anomaly detection tasks
(otx) ...$ otx find --task ANOMALY_DETECTION
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Task ┃ Model Name ┃ Recipe Path ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ANOMALY_DETECTION │ stfpm │ src/otx/recipe/anomaly_detection/stfpm.yaml │
│ ANOMALY_DETECTION │ padim │ src/otx/recipe/anomaly_detection/padim.yaml │
└───────────────────┴────────────┴─────────────────────────────────────────────┘
from otx.engine.utils.api import list_models
model_lists = list_models(task="ANOMALY_DETECTION")
print(model_lists)
'''
['stfpm', 'padim']
'''
You can see two anomaly detection models, STFPM and PADIM. For more detail on each model, refer to Anomalib’s STFPM and PADIM documentation.
2. Let’s proceed with PADIM for this example.
(otx) ...$ otx train --data_root datasets/MVTec/bottle \
--task ANOMALY_DETECTION
(otx) ...$ otx train --config src/otx/recipe/anomaly_detection/padim.yaml \
--data_root datasets/MVTec/bottle
from otx.engine import Engine
data_root = "datasets/MVTec/bottle"
recipe = "src/otx/recipe/anomaly_detection/padim.yaml"
engine = Engine.from_config(
config_path=recipe,
data_root=data_root,
work_dir="otx-workspace",
)
engine.train(...)
from otx.engine import Engine
data_root = "datasets/MVTec/bottle"
engine = Engine(
model="padim",
data_root=data_root,
task="ANOMALY_DETECTION",
work_dir="otx-workspace",
)
engine.train(...)
3. (Optional) Additionally, we can tune training parameters such as batch size, learning rate, patience epochs.
Learn more about specific parameters using otx train --help -v or otx train --help -vv.
For example, to decrease the batch size to 4, fix the number of epochs to 100, extend the command line above with the following line.
(otx) ...$ otx train ... --data.config.train_subset.batch_size 4 \
--max_epochs 100
from otx.core.config.data import DataModuleConfig, SubsetConfig
from otx.core.data.module import OTXDataModule
from otx.engine import Engine
data_config = DataModuleConfig(..., train_subset=SubsetConfig(..., batch_size=4))
datamodule = OTXDataModule(..., config=data_config)
engine = Engine(..., datamodule=datamodule)
engine.train(max_epochs=100)
4. The training result checkpoints/*.ckpt file is located in {work_dir} folder,
while training logs can be found in the {work_dir}/{timestamp} dir.
This will start training and generate artifacts for commands such as export and optimize. You will notice the otx-workspace directory in your current working directory. This is where all the artifacts are stored.
Evaluation#
Now we have trained the model, let’s see how it performs on a specific dataset. In this example, we will use the same dataset to generate evaluation metrics. To perform evaluation you need to run the following commands:
(otx) ...$ otx test --work_dir otx-workspace
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ image_AUROC │ 0.8 │
│ image_F1Score │ 0.8 │
│ pixel_AUROC │ 0.8 │
│ pixel_F1Score │ 0.8 │
│ test/data_time │ 0.6517705321311951 │
│ test/iter_time │ 0.6630784869194031 │
└───────────────────────────┴───────────────────────────┘
(otx) ...$ otx test --config src/otx/recipe/anomaly_detection/padim.yaml \
--data_root datasets/MVTec/bottle \
--checkpoint otx-workspace/20240313_042421/checkpoints/epoch_010.ckpt
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ image_AUROC │ 0.8 │
│ image_F1Score │ 0.8 │
│ pixel_AUROC │ 0.8 │
│ pixel_F1Score │ 0.8 │
│ test/data_time │ 0.6517705321311951 │
│ test/iter_time │ 0.6630784869194031 │
└───────────────────────────┴───────────────────────────┘
engine.test()
The primary metric here is the f-measure computed against the ground-truth bounding boxes. It is also called the local score. In addition, f-measure is also used to compute the global score. The global score is computed based on the global label of the image. That is, the image is anomalous if it contains at least one anomaly. This global score is stored as an additional metric.
Note
All task types report Image-level F-measure as the primary metric. In addition, both localization tasks (anomaly detection and anomaly segmentation) also report localization performance (F-measure for anomaly detection and Dice-coefficient for anomaly segmentation).
Export#
1. otx export exports a trained Pytorch .pth model to the OpenVINO™ Intermediate Representation (IR) format.
It allows running the model on the Intel hardware much more efficient, especially on the CPU. Also, the resulting IR model is required to run PTQ optimization. IR model consists of 2 files: exported_model.xml for weights and exported_model.bin for architecture.
2. We can run the below command line to export the trained model
and save the exported model to the openvino folder:
(otx) ...$ otx export --work_dir otx-workspace
...
Elapsed time: 0:00:06.588245
(otx) ...$ otx export ... --checkpoint otx-workspace/20240313_042421/checkpoints/epoch_010.ckpt
...
Elapsed time: 0:00:06.588245
engine.export()
Now that we have the exported model, let’s check its performance using otx test:
(otx) ...$ otx test --work_dir otx-workspace \
--checkpoint otx-workspace/20240313_052847/exported_model.xml \
--engine.device cpu
...
(otx) ...$ otx test --config src/otx/recipe/anomaly_detection/padim.yamll \
--data_root data/wgisd \
--checkpoint otx-workspace/20240312_052847/exported_model.xml \
--engine.device cpu
...
exported_model = engine.export()
engine.test(checkpoint=exported_model)
Optimization#
Anomaly tasks can be optimized either in PTQ or NNCF format. The model will be quantized to INT8 format.
For more information refer to the optimization explanation section.
1. Let’s start with PTQ optimization.
(otx) ...$ otx optimize --work_dir otx-workspace \
--checkpoint otx-workspace/20240312_052847/exported_model.xml
...
Statistics collection ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 30/30 • 0:00:14 • 0:00:00
Applying Fast Bias correction ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 58/58 • 0:00:02 • 0:00:00
Elapsed time: 0:00:24.958733
ckpt_path = "otx-workspace/20240312_052847/exported_model.xml"
engine.optimize(checkpoint=ckpt_path)
Please note, that PTQ will take some time without logging to optimize the model.
3. Finally, we can also evaluate the optimized model by passing
it to the otx test function.
(otx) ...$ otx test --work_dir otx-workspace \
--checkpoint otx-workspace/20240313_055042/optimized_model.xml \
--engine.device cpu
...
Elapsed time: 0:00:10.260521
ckpt_path = "otx-workspace/20240313_055042/optimized_model.xml"
engine.test(checkpoint=ckpt_path)
Segmentation and Classification#
While the above example shows Anomaly Detection, you can also train Anomaly Segmentation and Classification models.
To see what tasks are available, you can pass ANOMALY_SEGMENTATION and ANOMALY_CLASSIFICATION to otx find mentioned in the Training section. You can then use the same commands to train, evaluate, export and optimize the models.
Note
The Segmentation and Detection tasks also require that the ground_truth masks be present to ensure that the localization metrics are computed correctly.
The ground_truth masks are not required for the Classification task.