Skip to main content

Performance Metrics

Overview​

openvino_genai.PerfMetrics (referred as PerfMetrics for simplicity) is a structure that holds performance metrics for each generate call. PerfMetrics holds fields with mean and standard deviations for the following metrics:

  • Time To the First Token (TTFT), ms
  • Time per Output Token (TPOT), ms/token
  • Generate total duration, ms
  • Chat template application duration, ms
  • Tokenization duration, ms
  • Detokenization duration, ms
  • Throughput, tokens/s

and:

  • Load time, ms
  • Number of generated tokens
  • Number of tokens in the input prompt
  • Number of input tokens reused from the prefix cache

Performance metrics are stored either in the DecodedResults or EncodedResults in perf_metrics field. Additionally to the fields mentioned above, PerfMetrics has a member raw_metrics of type openvino_genai.RawPerfMetrics that contains raw values for the durations of each batch of new token generation, tokenization durations, detokenization durations, and more. These raw metrics are accessible if you wish to calculate your own statistical values such as median or percentiles. However, since mean and standard deviation values are usually sufficient, we will focus on PerfMetrics.

import openvino_genai as ov_genai
pipe = ov_genai.LLMPipeline(models_path, "CPU")
result = pipe.generate(["The Sun is yellow because"], max_new_tokens=20)
perf_metrics = result.perf_metrics

print(f'Generate duration: {perf_metrics.get_generate_duration().mean:.2f}')
print(f'TTFT: {perf_metrics.get_ttft().mean:.2f} ms')
print(f'TPOT: {perf_metrics.get_tpot().mean:.2f} ms/token')
print(f'Throughput: {perf_metrics.get_throughput().mean:.2f} tokens/s')
Output:
Generate duration: 702.85
TTFT: 137.58 ms
TPOT: 29.74 ms/token
Throughput: 33.62 tokens/s
Note

If the input prompt is just a string, the generate function returns only a string without perf_metrics. To obtain perf_metrics, provide the prompt as a list with at least one element or call generate with encoded inputs.

Note

TPOT (Time Per Output Token) represents the average time required to generate each output token in the final result. For beam search scenario, TPOT is calculated based on the effective output tokens delivered to users, not the tokens generated by individual beams during internal processing.

Prefix Cache Hit Tokens​

For pipelines that support prefix caching, the prefix cache hit token count reports how many input tokens were reused instead of processed during prefill. The metric is 0 when prefix caching is disabled, no matching prefix is found, or the pipeline does not support prefix-cache restoration.

For an input prompt containing N tokens and a matching cached prefix containing K tokens, the metric has the following values:

  • 0 when no prefix is reused
  • K for a partial prefix match where K < N
  • N - 1 when the full prompt is cached, because the final prompt token must still be processed to produce logits for the first generated token
prefix_cache_hit_tokens = result.perf_metrics.get_num_prefix_cache_hit_tokens()

When multiple PerfMetrics objects are added, their prefix cache hit token counts are summed, consistent with the input and generated token counters.

Speculative Decoding Metrics​

Pipelines with speculative decoding enabled expose additional metrics through SDPerModelsPerfMetrics in extended_perf_metrics. These metrics distinguish draft candidates from draft-model processing work:

  • get_num_draft_tokens() returns the number of draft candidate tokens offered to the main model for validation.
  • get_num_accepted_tokens() returns how many draft candidates were accepted.
  • get_num_rejected_tokens() returns how many draft candidates were rejected.
  • get_draft_acceptance_rate() returns accepted / draft as a value in the [0, 1] range.
  • draft_model_metrics.get_num_generated_tokens() reports draft-model processed tokens and can include non-candidate work such as cache or state alignment; do not use it as the acceptance-rate denominator.
import openvino_genai as ov_genai

draft_model = ov_genai.draft_model(draft_model_path, "CPU")
pipe = ov_genai.LLMPipeline(models_path, "CPU", draft_model=draft_model)
config = ov_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 4

result = pipe.generate([prompt], config)
metrics = result.extended_perf_metrics

print(f"Accepted draft tokens: {metrics.get_num_accepted_tokens()}")
print(f"Draft candidate tokens: {metrics.get_num_draft_tokens()}")
print(f"Rejected draft tokens: {metrics.get_num_rejected_tokens()}")
print(f"Draft acceptance rate: {100 * metrics.get_draft_acceptance_rate():.2f}%")
print(f"Draft processed tokens: {metrics.draft_model_metrics.get_num_generated_tokens()}")

Accumulating Metrics​

Several perf_metrics can be added to each other. In that case raw_metrics are concatenated and mean/std values are recalculated. This accumulates statistics from several generate() calls.

import openvino_genai as ov_genai
pipe = ov_genai.LLMPipeline(models_path, "CPU")
res_1 = pipe.generate(["The Sun is yellow because"], max_new_tokens=20)
res_2 = pipe.generate(["Why Sky is blue because"], max_new_tokens=20)
perf_metrics = res_1.perf_metrics + res_2.perf_metrics

print(f'Generate duration: {perf_metrics.get_generate_duration().mean:.2f}')
print(f'TTFT: {perf_metrics.get_ttft().mean:.2f} ms')
print(f'TPOT: {perf_metrics.get_tpot().mean:.2f} ms/token')
print(f'Throughput: {perf_metrics.get_throughput().mean:.2f} tokens/s')

Using Raw Performance Metrics​

In addition to mean and standard deviation values, the perf_metrics object has a raw_metrics field. This field stores raw data, including:

  • Timestamps for each batch of generated tokens
  • Batch sizes for each timestamp
  • Tokenization durations
  • Detokenization durations
  • Other relevant metrics

These metrics can be use for more fine grained analysis, such as calculating exact median values, percentiles, etc.

Below are a few examples of how to use raw metrics.

Getting timestamps for each generated token:​

import openvino_genai as ov_genai
pipe = ov_genai.LLMPipeline(models_path, "CPU")
result = pipe.generate(["The Sun is yellow because"], max_new_tokens=20)
perf_metrics = result.perf_metrics
raw_metrics = perf_metrics.raw_metrics

print(f'Generate duration: {perf_metrics.get_generate_duration().mean:.2f}')
print(f'Throughput: {perf_metrics.get_throughput().mean:.2f} tokens/s')
print(f'Timestamps: {" ms, ".join(f"{i:.2f}" for i in raw_metrics.m_new_token_times)}')

Getting pure inference time without tokenizatin and detokenization duration:​

import openvino_genai as ov_genai
import numpy as np
pipe = ov_genai.LLMPipeline(models_path, "CPU")
result = pipe.generate(["The Sun is yellow because"], max_new_tokens=20)
perf_metrics = result.perf_metrics
print(f'Generate duration: {perf_metrics.get_generate_duration().mean:.2f} ms')

raw_metrics = perf_metrics.raw_metrics
generate_duration = np.array(raw_metrics.generate_durations)
tok_detok_duration = np.array(raw_metrics.tokenization_durations) - np.array(raw_metrics.detokenization_durations)
pure_inference_duration = np.sum(generate_duration - tok_detok_duration) / 1000 # in milliseconds
print(f'Pure Inference duration: {pure_inference_duration:.2f} ms')

Using raw metrics to calculate median value of generate duration:​

import openvino_genai as ov_genai
import numpy as np
pipe = ov_genai.LLMPipeline(models_path, "CPU")
result = pipe.generate(["The Sun is yellow because"], max_new_tokens=20)
perf_metrics = result.perf_metrics
raw_metrics = perf_metrics.raw_metrics

print(f'Generate duration: {perf_metrics.get_generate_duration().mean:.2f}')
print(f'Throughput: {perf_metrics.get_throughput().mean:.2f} tokens/s')
durations = np.array(raw_metrics.m_new_token_times[1:]) - np.array(raw_metrics.m_new_token_times[:-1])
print(f'Median from token to token duration: {np.median(durations):.2f} ms')
tip

For more information, refer to the Python, C++ and JavaScript benchmark samples.