Keypoint Detection#

Description#

Keypoint detection model aims to detect a set of pre-defined keypoints on a cropped object. If a crop is not tight enough, quality of keypoints degrades. Having this model and an object detector, one can organize keypoint detection for all objects of interest presented on an image (top-down approach).

Models#

Top-down keypoint detection pipeline uses detections that come from any appropriate detector, and a keypoints regression model acting on crops.

Parameters#

The following parameters can be provided via python API or RT Info embedded into OV model:

  • labels(list(str)) : a list of keypoints names.

OpenVINO Model Specifications#

Inputs#

A single NCHW tensor representing a batch of images.

Outputs#

Two vectors in Simple Coordinate Classification Perspective (SimCC) format:

  • pred_x (B, N, D1) - x coordinate representation, where N is the number of keypoints.

  • pred_y (B, N, D2) - y coordinate representation, where N is the number of keypoints.

Example#

import cv2
from model_api.models import TopDownKeypointDetectionPipeline, Detection, KeypointDetectionModel

model = KeypointDetectionModel.create_model("kp_model.xml")
# a list of detections in (x_min, y_min, x_max, y_max, score, class_id) format
detections = [Detection(0, 0, 100, 100, 1.0, 0)]
top_down_pipeline = TopDownKeypointDetectionPipeline(model)
predictions = top_down_detector.predict(image, detections)

# iterating over a list of DetectedKeypoints. Each of the items corresponds to a detection
for obj_keypoints in predictions:
    for point in obj_keypoints.keypoints.astype(np.int32):
        cv2.circle(
            image, point, radius=0, color=(0, 255, 0), thickness=5
        )
class model_api.models.keypoint_detection.KeypointDetectionModel(inference_adapter, configuration={}, preload=False)#

Bases: ImageModel

A wrapper that implements a basic keypoint regression model.

Initializes the keypoint detection model.

Parameters:

  • inference_adapter (InferenceAdapter) – inference adapter containing the underlying model.

  • configuration (dict, optional) – configuration overrides the model parameters (see parameters() method). Defaults to {}.

  • preload (bool, optional) – forces inference adapter to load the model. Defaults to False.

classmethod parameters()#

Defines the description and type of configurable data parameters for the wrapper.

See types.py to find available types of the data parameter. For each parameter the type, default value and description must be provided.

The example of possible data parameter:
‘confidence_threshold’: NumericalValue(

default_value=0.5, description=”Threshold value for detection box confidence”

)

The method must be implemented in each specific inherited wrapper.

Return type:

dict

Returns:

  • the dictionary with defined wrapper data parameters

postprocess(outputs, meta)#

Applies SCC decoded to the model outputs.

Parameters:

  • outputs (dict[str, np.ndarray]) – raw outputs of the model

  • meta (dict[str, Any]) – meta information about the input data

Returns:

detected keypoints

Return type:

DetectedKeypoints

class model_api.models.keypoint_detection.TopDownKeypointDetectionPipeline(base_model)#

Bases: object

Pipeline implementing top down keypoint detection approach.

predict(image, detections)#

Predicts keypoints for the given image and detections.

Parameters:

  • image (np.ndarray) – input full-size image

  • detections (list[Detection]) – detections located within the given image

Returns:

per detection keypoints in detection coordinates

Return type:

list[DetectedKeypoints]

predict_crops(crops)#

Predicts keypoints for the given crops.

Parameters:

crops (list[np.ndarray]) – list of cropped object images

Returns:

per crop keypoints

Return type:

list[DetectedKeypoints]