Text Reranking

Convert and Optimize Model

Download and convert a reranker model (e.g. cross-encoder/ms-marco-MiniLM-L6-v2) to OpenVINO format from Hugging Face:

optimum-cli export openvino --model cross-encoder/ms-marco-MiniLM-L6-v2 --trust-remote-code cross-encoder/ms-marco-MiniLM-L6-v2

See all supported Reranker Models.

info

Refer to the Model Preparation guide for detailed instructions on how to download, convert and optimize models for OpenVINO GenAI.

Run Model Using OpenVINO GenAI

The TextRerankPipeline enables you to reorder candidate documents or passages by semantic relevance to a query using a cross-encoder or reranker model. You can control how many top results are returned using the top_n parameter.

Python

CPU

import openvino_genai

pipeline = openvino_genai.TextRerankPipeline(model_path, "CPU", top_n=3)

rerank_result = pipeline.rerank(query, documents)

print("Reranked documents:")
for index, score in rerank_result:
  print(f"Document {index} (score: {score:.4f}): {documents[index]}")

GPU

import openvino_genai

pipeline = openvino_genai.TextRerankPipeline(model_path, "GPU", top_n=3)

rerank_result = pipeline.rerank(query, documents)

print("Reranked documents:")
for index, score in rerank_result:
  print(f"Document {index} (score: {score:.4f}): {documents[index]}")

C++

CPU

#include "openvino/genai/rag/text_rerank_pipeline.hpp"

int main(int argc, char* argv[]) {
  std::vector<std::string> documents(argv + 3, argv + argc);
  std::string models_path = argv[1], query = argv[2];
  
  ov::genai::TextRerankPipeline pipeline(models_path, "CPU", ov::genai::top_n(3));
  
  auto rerank_result = pipeline.rerank(query, documents);
  
  std::cout << "Reranked documents:\n";
  for (const auto& [index, score] : rerank_result) {
      std::cout << "Document " << index << " (score: " << score << "): " << documents[index] << '\n';
  }
}

GPU

#include "openvino/genai/rag/text_rerank_pipeline.hpp"

int main(int argc, char* argv[]) {
  std::vector<std::string> documents(argv + 3, argv + argc);
  std::string models_path = argv[1], query = argv[2];
  
  ov::genai::TextRerankPipeline pipeline(models_path, "GPU", ov::genai::top_n(3));
  
  auto rerank_result = pipeline.rerank(query, documents);
  
  std::cout << "Reranked documents:\n";
  for (const auto& [index, score] : rerank_result) {
      std::cout << "Document " << index << " (score: " << score << "): " << documents[index] << '\n';
  }
}