Visual Prompting (Fine-tuning)

Visual prompting is a computer vision task that uses a combination of an image and prompts, such as texts, bounding boxes, points, and so on to troubleshoot problems. Using these useful prompts, the main purpose of this task is to obtain labels from unlabeled datasets, and to use generated label information on particular domains or to develop a new model with the generated information.

This section examines the solutions for visual prompting offered by the OpenVINO Training Extensions library. Segment Anything (SAM), is one of the most famous visual prompting methods and this model will be used to adapt a new dataset domain. Because SAM was trained by using web-scale dataset and has huge backbone network, fine-tuning the whole network is difficult and lots of resources are required. Therefore, in this section, we try to fine-tune the mask decoder only for several epochs to increase performance on the new dataset domain. For fine-tuning SAM, we use following algorithms components:

• Pre-processing: Resize an image according to the longest axis and pad the rest with zero.

• Optimizer: We use Adam optimizer.

• Loss function: We use standard loss combination, 20 * focal loss + dice loss + iou loss, used in SAM as it is.

• Additional training techniques

  • Early stopping: To add adaptability to the training pipeline and prevent overfitting. Early stopping will be automatically applied.

Note

In OTX 2.0, fine-tuning using either bounding boxes or points is available. Given both bounding boxes and points prompts at the same time, bounding boxes have priority because using bounding boxes as prompts is more accurate.

Note

Currently, Post-Training Quantization (PTQ) for SAM is only supported, not Quantization Aware Training (QAT).

Dataset Format

For the dataset handling inside OpenVINO™ Training Extensions, we use Dataset Management Framework (Datumaro).

We support four dataset formats for visual prompting:

• Common Semantic Segmentation for semantic segmentation

• COCO for instance segmentation

• Pascal VOC for instance segmentation and semantic segmentation

• Datumaro for custom format dataset

Models

We support the following model recipes in experimental phase:

Recipe ID	Name	Complexity (GFLOPs)	Model size (MB)
Visual_Prompting_SAM_Tiny_ViT	SAM_Tiny_ViT	38.55	47
Visual_Prompting_SAM_ViT_B	SAM_ViT_B	454.76	363

To check feasibility of SAM, we did experiments using three public datasets with each other domains: WGISD and Trashcan, and checked F1-score. We used sampled training data from Trashcan and full training data (=110) from WGISD.

Dataset	#samples
WGISD	110
Trashcan	500

The below table shows performance improvement after fine-tuning.

Model name	WGISD	Trashcan
SAM_Tiny_ViT	90.32 → 92.10 (+1.78)	82.38 → 85.35 (+2.97)
SAM_ViT_B	92.32 → 93.16 (+0.84)	79.61 → 86.40 (+6.79)

According to datasets, learning rate and batch size can be adjusted like below:

(otx) ...$ otx train \
    --config <model_config_path> \
    --data_root <path_to_data_root> \
    --data.config.train_subset.batch_size <batch_size_to_be_updated> \
    --optimizer.lr <learning_rate_to_be_updated>