Skip to main content

Semantic Search Using Multimodal Embedding

EmbeddingPipeline computes embedding vectors for text, images, and videos with a single multimodal embedding model. It enables cross-modal use cases such as retrieving the most relevant image or video for a text query.

info

We recommend using EmbeddingPipeline instead of TextEmbeddingPipeline. It is more generic — it handles text, images, and videos — while providing all of the features that were available in TextEmbeddingPipeline.

Convert and Optimize Model​

Download and convert an embedding model (e.g. multimodal Qwen/Qwen3-VL-Embedding-2B, or a text-only model BAAI/bge-small-en-v1.5) to OpenVINO format from Hugging Face:

optimum-cli export openvino --model Qwen/Qwen3-VL-Embedding-2B --task feature-extraction Qwen3-VL-Embedding-2B

See all supported Embedding Models.

info

Refer to the Model Preparation guide for detailed instructions on how to download, convert and optimize models for OpenVINO GenAI.

Run Model Using OpenVINO GenAI​

EmbeddingPipeline generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval). Each call to embed() computes embedding vectors, The input can be:

  • Text — a single string or a batch of strings.
  • Images — each image is a tensor of shape [H, W, C] (uint8), or a batch [N, H, W, C].
  • Videos — each video is a tensor of shape [F, H, W, C] (uint8), where F is the number of frames. Accompanying video metadata (frame rate, sampled frame indices) can be provided via videos_metadata.

Text, images, and videos are mapped into the same embedding space, so embeddings produced from different modalities can be compared directly (for example, with cosine similarity).

import openvino as ov
import openvino_genai as ov_genai

pipeline = ov_genai.EmbeddingPipeline(models_path, "CPU")

# Embed a text query
query_embedding = pipeline.embed("a photo of a cat").embeddings

# Embed images  (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")
image_embedding = pipeline.embed(images=images).embeddings

# Embed a video  (video is an ov.Tensor of shape [F, H, W, 3], uint8)
video, video_metadata = read_video("path/to/video.mp4")
video_embedding = pipeline.embed(videos=[video], videos_metadata=[video_metadata]).embeddings
tip

Use CPU or GPU as devices without any other code change.

Additional Usage Options​

tip

Check out the Python RAG samples for multimodal embedding examples. For a text embedding example, check out the Python, C++, and JavaScript samples.

Embedding Prompt​

Some models support a special instruction that describes how an input should be encoded. Use the embedding_prompt parameter of embed() to set it per call.

  • If the model has a chat template, the prompt is added to the system message.
  • Otherwise, it is prepended to the text.
embedding = pipeline.embed(
    "a photo of a cat",
    embedding_prompt="Represent this text for retrieval: "
).embeddings

Pooling Strategies​

Embedding models support different pooling strategies to aggregate token embeddings into a single vector:

  • CLS: Use the first token embedding (default for many models)
  • MEAN: Average all token embeddings
  • LAST_TOKEN: Use the last token embedding

Set the pooling strategy via the pooling_type parameter.

L2 Normalization​

L2 normalization can be applied to the output embeddings for improved retrieval performance. Enable it with the normalize parameter.

Input Size and Padding​

You can control how input texts are tokenized and padded:

  • max_length: Maximum length of tokens passed to the embedding model. Longer texts will be truncated.
  • pad_to_max_length: If true, model input tensors are padded to the maximum length.
  • padding_side: Side to use for padding ("left" or "right").

Batch Size Configuration​

The batch_size parameter is useful for optimizing performance during database population:

  • When set, the pipeline fixes the model shape for inference optimization.
  • The number of documents passed to the pipeline must equal batch_size.
  • For query embeddings, set batch_size=1 or leave it unset.

Fixed Shape Optimization​

Setting batch_size, max_length, and pad_to_max_length=true together will fix the model shape for optimal inference performance.

info

Fixed shapes are required for NPU device inference.

Query and Embed Instructions​

Some models support special instructions for queries and documents. Use query_instruction and embed_instruction to provide these if needed.

info

query_instruction and embed_instruction apply only to text-only pipelines. For multimodal inputs, set a per-call instruction with the embedding_prompt parameter instead.

Pipeline Configuration​

The embedding behavior of the pipeline can be tuned by passing configuration parameters at construction time. They can be passed directly as properties or bundled into a TextEmbeddingPipeline.Config and supplied via text_embedding_config.

import openvino_genai as ov_genai

pipeline = ov_genai.EmbeddingPipeline(
    models_path,
    "CPU",
    pooling_type=ov_genai.TextEmbeddingPipeline.PoolingType.LAST_TOKEN,
    normalize=True,
    max_length=512,
    pad_to_max_length=True,
    padding_side="left",
)

# Embed the text query
query = pipeline.embed("a photo of a cat").embeddings
info

For the full list of configuration options, see EmbeddingPipeline API Reference and TextEmbeddingPipeline API Reference.