otx.core.model.visual_prompting#

Class definition for visual prompting models entity used in OTX.

Classes

OTXVisualPromptingModel(label_info, ...)

Base class for the visual prompting models used in OTX.

OTXZeroShotVisualPromptingModel(input_size, ...)

Base class for the zero-shot visual prompting models used in OTX.

OVVisualPromptingModel(model_name, ...)

Visual prompting model compatible for OpenVINO IR inference.

OVZeroShotVisualPromptingModel(model_name, ...)

Zero-shot visual prompting model compatible for OpenVINO IR inference.

class otx.core.model.visual_prompting.OTXVisualPromptingModel(label_info: LabelInfoTypes = NullLabelInfo(label_names=[], label_ids=[], label_groups=[[]]), input_size: tuple[int, int] = (1024, 1024), optimizer: OptimizerCallable = <function _default_optimizer_callable>, scheduler: LRSchedulerCallable | LRSchedulerListCallable = <function _default_scheduler_callable>, metric: MetricCallable = <function _visual_prompting_metric_callable>, torch_compile: bool = False)[source]#

Bases: OTXModel[VisualPromptingBatchDataEntity, VisualPromptingBatchPredEntity]

Base class for the visual prompting models used in OTX.

get_dummy_input(batch_size: int = 1) VisualPromptingBatchDataEntity[source]#

Returns a dummy input for VPT model.

test_step(inputs: VisualPromptingBatchDataEntity, batch_idx: int) None[source]#

Perform a single test step on a batch of data from the test set.

Parameters:
  • inputs (VisualPromptingBatchDataEntity) – The input data for the test step.

  • batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type VisualPromptingBatchPredEntity.

validation_step(inputs: VisualPromptingBatchDataEntity, batch_idx: int) None[source]#

Perform a single validation step on a batch of data from the validation set.

Parameters:
  • inputs (VisualPromptingBatchDataEntity) – The input data for the validation step.

  • batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type VisualPromptingBatchPredEntity.

Returns:

None

class otx.core.model.visual_prompting.OTXZeroShotVisualPromptingModel(input_size: tuple[int, int], label_info: LabelInfoTypes = NullLabelInfo(label_names=[], label_ids=[], label_groups=[[]]), optimizer: OptimizerCallable = <function _default_optimizer_callable>, scheduler: LRSchedulerCallable | LRSchedulerListCallable = <function _default_scheduler_callable>, metric: MetricCallable = <function _visual_prompting_metric_callable>, torch_compile: bool = False)[source]#

Bases: OTXModel[ZeroShotVisualPromptingBatchDataEntity, ZeroShotVisualPromptingBatchPredEntity]

Base class for the zero-shot visual prompting models used in OTX.

configure_optimizers() None[source]#

Skip configure_optimizers unused in zero-shot visual prompting.

get_dummy_input(batch_size: int = 1) ZeroShotVisualPromptingBatchDataEntity[source]#

Returns a dummy input for ZSL VPT model.

on_predict_start() None[source]#

Load previously saved reference info.

on_test_start() None[source]#

Load previously saved reference info.

on_train_epoch_end() None[source]#

Skip on_train_epoch_end unused in zero-shot visual prompting.

on_train_epoch_start() None[source]#

Skip on_train_epoch_start unused in zero-shot visual prompting.

on_train_start() None[source]#

Initialize reference infos before learn.

on_validation_epoch_end() None[source]#

Skip on_validation_epoch_end unused in zero-shot visual prompting.

on_validation_epoch_start() None[source]#

Skip on_validation_epoch_start unused in zero-shot visual prompting.

test_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) None[source]#

Perform a single test step on a batch of data from the test set.

Parameters:
  • inputs (ZeroShotVisualPromptingBatchDataEntity) – The input data for the test step.

  • batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type ZeroShotVisualPromptingBatchDataEntity.

training_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) Tensor[source]#

Skip training_step unused in zero-shot visual prompting.

validation_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) None[source]#

Skip validation_step unused in zero-shot visual prompting.

class otx.core.model.visual_prompting.OVVisualPromptingModel(model_name: str, model_type: str = 'Visual_Prompting', async_inference: bool = False, max_num_requests: int | None = None, use_throughput_mode: bool = False, model_api_configuration: dict[str, Any] | None = None, metric: MetricCallable = <function _visual_prompting_metric_callable>, **kwargs)[source]#

Bases: OVModel[VisualPromptingBatchDataEntity, VisualPromptingBatchPredEntity]

Visual prompting model compatible for OpenVINO IR inference.

It can only consume OpenVINO IR model path and create the OTX visual prompting model compatible

for OTX testing pipeline.

forward(inputs: VisualPromptingBatchDataEntity) VisualPromptingBatchPredEntity[source]#

Model forward function.

get_dummy_input(batch_size: int = 1) VisualPromptingBatchDataEntity[source]#

Returns a dummy input for classification OV model.

optimize(output_dir: Path, data_module: OTXDataModule, ptq_config: dict[str, Any] | None = None) dict[str, Path][source]#

Runs NNCF quantization.

test_step(inputs: VisualPromptingBatchDataEntity, batch_idx: int) None[source]#

Perform a single test step on a batch of data from the test set.

Parameters:
  • inputs (VisualPromptingBatchDataEntity) – The input data for the test step.

  • batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type VisualPromptingBatchPredEntity.

validation_step(inputs: VisualPromptingBatchDataEntity, batch_idx: int) None[source]#

Perform a single validation step on a batch of data from the validation set.

Parameters:
  • inputs (VisualPromptingBatchDataEntity) – The input data for the validation step.

  • batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type VisualPromptingBatchPredEntity.

Returns:

None

class otx.core.model.visual_prompting.OVZeroShotVisualPromptingModel(model_name: str, model_type: str = 'Zero_Shot_Visual_Prompting', async_inference: bool = False, max_num_requests: int | None = None, use_throughput_mode: bool = False, model_api_configuration: dict[str, Any] | None = None, metric: MetricCallable = <function _visual_prompting_metric_callable>, reference_info_dir: Path | str = 'reference_infos', infer_reference_info_root: Path | str = '../.latest/train', save_outputs: bool = True, **kwargs)[source]#

Bases: OVModel[ZeroShotVisualPromptingBatchDataEntity, ZeroShotVisualPromptingBatchPredEntity]

Zero-shot visual prompting model compatible for OpenVINO IR inference.

It can only consume OpenVINO IR model path and create the OTX zero-shot visual prompting model compatible

for OTX testing pipeline.

configure_optimizers() None[source]#

Skip configure_optimizers unused in zero-shot visual prompting.

forward(inputs: ZeroShotVisualPromptingBatchDataEntity) tuple[dict[str, ndarray], list[ndarray]] | ZeroShotVisualPromptingBatchPredEntity[source]#

Model forward function.

get_dummy_input(batch_size: int = 1) ZeroShotVisualPromptingBatchDataEntity[source]#

Returns a dummy input for classification OV model.

infer(inputs: ZeroShotVisualPromptingBatchDataEntity, reference_feats: np.ndarray, used_indices: np.ndarray) list[dict[int, PredictedMask]][source]#

Infer for target predictions.

initialize_reference_info() None[source]#

Initialize reference information.

learn(inputs: ZeroShotVisualPromptingBatchDataEntity, reset_feat: bool = False) tuple[dict[str, ndarray], list[ndarray]][source]#

Learn for reference features.

load_reference_info(default_root_dir: Path | str, *args, path_to_directly_load: Path | None = None, **kwargs) bool[source]#

Load latest reference info to be used.

Parameters:
  • default_root_dir (Path | str) – Default root directory to be used when inappropriate infer_reference_info_root is given.

  • path_to_directly_load (Path | None) – Reference info path to directly be loaded. Normally, it is obtained after learn which is executed when trying to do infer without reference features in on_test_start or on_predict_start.

Returns:

Whether normally loading checkpoint or not.

Return type:

(bool)

on_predict_start() None[source]#

Load previously saved reference info.

on_test_start() None[source]#

Load previously saved reference info.

on_train_epoch_end() None[source]#

Skip on_train_epoch_end unused in zero-shot visual prompting.

on_train_epoch_start() None[source]#

Skip on_train_epoch_start unused in zero-shot visual prompting.

on_train_start() None[source]#

Initialize reference infos before learn.

on_validation_epoch_end() None[source]#

Skip on_validation_epoch_end unused in zero-shot visual prompting.

on_validation_epoch_start() None[source]#

Skip on_validation_epoch_start unused in zero-shot visual prompting.

optimize(output_dir: Path, data_module: OTXDataModule, ptq_config: dict[str, Any] | None = None) dict[str, Path][source]#

Runs NNCF quantization.

save_reference_info(default_root_dir: Path | str) None[source]#

Save reference info.

test_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) None[source]#

Perform a single test step on a batch of data from the test set.

Parameters:
  • inputs (ZeroShotVisualPromptingBatchDataEntity) – The input data for the test step.

  • batch_idx (int) – The index of the current batch.

Raises:

TypeError – If the predictions are not of type ZeroShotVisualPromptingBatchPredEntity.

training_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) Tensor[source]#

Skip training_step unused in zero-shot visual prompting.

validation_step(inputs: ZeroShotVisualPromptingBatchDataEntity, batch_idx: int) None[source]#

Skip validation_step unused in zero-shot visual prompting.