otx.core.model.visual_prompting#

Class definition for visual prompting models entity used in OTX.

Classes

`OTXVisualPromptingModel`(label_info, ...)	Base class for the visual prompting models used in OTX.
`OTXZeroShotVisualPromptingModel`(input_size, ...)	Base class for the zero-shot visual prompting models used in OTX.
`OVVisualPromptingModel`(model_name, ...)	Visual prompting model compatible for OpenVINO IR inference.
`OVZeroShotVisualPromptingModel`(model_name, ...)	Zero-shot visual prompting model compatible for OpenVINO IR inference.

class otx.core.model.visual_prompting.OTXVisualPromptingModel(label_info: LabelInfoTypes = NullLabelInfo(label_names=[], label_groups=[[]]), input_size: tuple[int, int] = (1024, 1024), optimizer: OptimizerCallable = <function _default_optimizer_callable>, scheduler: LRSchedulerCallable | LRSchedulerListCallable = <function _default_scheduler_callable>, metric: MetricCallable = <function _visual_prompting_metric_callable>, torch_compile: bool = False)[source]#

Bases: OTXModel[VisualPromptingBatchDataEntity, VisualPromptingBatchPredEntity]

Base class for the visual prompting models used in OTX.

get_dummy_input(batch_size: int = 1) → VisualPromptingBatchDataEntity[source]#: Returns a dummy input for VPT model.

test_step(inputs: VisualPromptingBatchDataEntity, batch_idx: int) → None[source]#

Perform a single test step on a batch of data from the test set.

Parameters:

inputs (VisualPromptingBatchDataEntity) – The input data for the test step.
batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type VisualPromptingBatchPredEntity.

validation_step(inputs: VisualPromptingBatchDataEntity, batch_idx: int) → None[source]#

Perform a single validation step on a batch of data from the validation set.

Parameters:

inputs (VisualPromptingBatchDataEntity) – The input data for the validation step.
batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type VisualPromptingBatchPredEntity.

Returns:

None

class otx.core.model.visual_prompting.OTXZeroShotVisualPromptingModel(input_size: tuple[int, int], label_info: LabelInfoTypes = NullLabelInfo(label_names=[], label_groups=[[]]), optimizer: OptimizerCallable = <function _default_optimizer_callable>, scheduler: LRSchedulerCallable | LRSchedulerListCallable = <function _default_scheduler_callable>, metric: MetricCallable = <function _visual_prompting_metric_callable>, torch_compile: bool = False)[source]#

Bases: OTXModel[ZeroShotVisualPromptingBatchDataEntity, ZeroShotVisualPromptingBatchPredEntity]

Base class for the zero-shot visual prompting models used in OTX.

configure_optimizers() → None[source]#: Skip configure_optimizers unused in zero-shot visual prompting.

get_dummy_input(batch_size: int = 1) → ZeroShotVisualPromptingBatchDataEntity[source]#: Returns a dummy input for ZSL VPT model.

on_predict_start() → None[source]#: Load previously saved reference info.

on_test_start() → None[source]#: Load previously saved reference info.

on_train_epoch_end() → None[source]#: Skip on_train_epoch_end unused in zero-shot visual prompting.

on_train_epoch_start() → None[source]#: Skip on_train_epoch_start unused in zero-shot visual prompting.

on_train_start() → None[source]#: Initialize reference infos before learn.

on_validation_epoch_end() → None[source]#: Skip on_validation_epoch_end unused in zero-shot visual prompting.

on_validation_epoch_start() → None[source]#: Skip on_validation_epoch_start unused in zero-shot visual prompting.

test_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) → None[source]#

Perform a single test step on a batch of data from the test set.

Parameters:

inputs (ZeroShotVisualPromptingBatchDataEntity) – The input data for the test step.
batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type ZeroShotVisualPromptingBatchDataEntity.

training_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) → Tensor[source]#: Skip training_step unused in zero-shot visual prompting.

validation_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) → None[source]#: Skip validation_step unused in zero-shot visual prompting.

class otx.core.model.visual_prompting.OVVisualPromptingModel(model_name: str, model_type: str = 'Visual_Prompting', async_inference: bool = False, max_num_requests: int | None = None, use_throughput_mode: bool = False, model_api_configuration: dict[str, Any] | None = None, metric: MetricCallable = <function _visual_prompting_metric_callable>, **kwargs)[source]#

Bases: OVModel[VisualPromptingBatchDataEntity, VisualPromptingBatchPredEntity]

Visual prompting model compatible for OpenVINO IR inference.

It can only consume OpenVINO IR model path and create the OTX visual prompting model compatible: for OTX testing pipeline.

forward(inputs: VisualPromptingBatchDataEntity) → VisualPromptingBatchPredEntity[source]#: Model forward function.

get_dummy_input(batch_size: int = 1) → VisualPromptingBatchDataEntity[source]#: Returns a dummy input for classification OV model.

optimize(output_dir: Path, data_module: OTXDataModule, ptq_config: dict[str, Any] | None = None) → dict[str, Path][source]#: Runs NNCF quantization.

test_step(inputs: VisualPromptingBatchDataEntity, batch_idx: int) → None[source]#

Perform a single test step on a batch of data from the test set.

Parameters:

inputs (VisualPromptingBatchDataEntity) – The input data for the test step.
batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type VisualPromptingBatchPredEntity.

validation_step(inputs: VisualPromptingBatchDataEntity, batch_idx: int) → None[source]#

Perform a single validation step on a batch of data from the validation set.

Parameters:

inputs (VisualPromptingBatchDataEntity) – The input data for the validation step.
batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type VisualPromptingBatchPredEntity.

Returns:

None

class otx.core.model.visual_prompting.OVZeroShotVisualPromptingModel(model_name: str, model_type: str = 'Zero_Shot_Visual_Prompting', async_inference: bool = False, max_num_requests: int | None = None, use_throughput_mode: bool = False, model_api_configuration: dict[str, Any] | None = None, metric: MetricCallable = <function _visual_prompting_metric_callable>, reference_info_dir: Path | str = 'reference_infos', infer_reference_info_root: Path | str = '../.latest/train', save_outputs: bool = True, **kwargs)[source]#

Bases: OVModel[ZeroShotVisualPromptingBatchDataEntity, ZeroShotVisualPromptingBatchPredEntity]

Zero-shot visual prompting model compatible for OpenVINO IR inference.

It can only consume OpenVINO IR model path and create the OTX zero-shot visual prompting model compatible: for OTX testing pipeline.

configure_optimizers() → None[source]#: Skip configure_optimizers unused in zero-shot visual prompting.

forward(inputs: ZeroShotVisualPromptingBatchDataEntity) → tuple[dict[str, ndarray], list[ndarray]] | ZeroShotVisualPromptingBatchPredEntity[source]#: Model forward function.

get_dummy_input(batch_size: int = 1) → ZeroShotVisualPromptingBatchDataEntity[source]#: Returns a dummy input for classification OV model.

infer(inputs: ZeroShotVisualPromptingBatchDataEntity, reference_feats: np.ndarray, used_indices: np.ndarray) → list[dict[int, PredictedMask]][source]#: Infer for target predictions.

initialize_reference_info() → None[source]#: Initialize reference information.

learn(inputs: ZeroShotVisualPromptingBatchDataEntity, reset_feat: bool = False) → tuple[dict[str, ndarray], list[ndarray]][source]#: Learn for reference features.

load_reference_info(default_root_dir: Path | str, *args, path_to_directly_load: Path | None = None, **kwargs) → bool[source]#

Load latest reference info to be used.

Parameters:

default_root_dir (Path | str) – Default root directory to be used when inappropriate infer_reference_info_root is given.
path_to_directly_load (Path | None) – Reference info path to directly be loaded. Normally, it is obtained after learn which is executed when trying to do infer without reference features in on_test_start or on_predict_start.

Returns:

Whether normally loading checkpoint or not.

Return type:

(bool)

on_predict_start() → None[source]#: Load previously saved reference info.

on_test_start() → None[source]#: Load previously saved reference info.

on_train_epoch_end() → None[source]#: Skip on_train_epoch_end unused in zero-shot visual prompting.

on_train_epoch_start() → None[source]#: Skip on_train_epoch_start unused in zero-shot visual prompting.

on_train_start() → None[source]#: Initialize reference infos before learn.

on_validation_epoch_end() → None[source]#: Skip on_validation_epoch_end unused in zero-shot visual prompting.

on_validation_epoch_start() → None[source]#: Skip on_validation_epoch_start unused in zero-shot visual prompting.

optimize(output_dir: Path, data_module: OTXDataModule, ptq_config: dict[str, Any] | None = None) → dict[str, Path][source]#: Runs NNCF quantization.

save_reference_info(default_root_dir: Path | str) → None[source]#: Save reference info.

test_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) → None[source]#

Perform a single test step on a batch of data from the test set.

Parameters:

inputs (ZeroShotVisualPromptingBatchDataEntity) – The input data for the test step.
batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type ZeroShotVisualPromptingBatchPredEntity.

training_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) → Tensor[source]#: Skip training_step unused in zero-shot visual prompting.

validation_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) → None[source]#: Skip validation_step unused in zero-shot visual prompting.