Skip to main content

Speculative Decoding

Overview​

Speculative decoding (also referred to as assisted-generation) is a latency-oriented optimization for autoregressive text generation. A small, cheap drafter proposes one or more candidate continuations, and the larger target (main) model validates them in a single forward pass. Tokens that the target would have produced anyway are accepted; the first divergent token is replaced by the target's own sample, and drafting resumes from there.

Because validation is parallel over the candidate length, each accepted draft token replaces a sequential decode step of the target model with (a fraction of) one batched step. When acceptance is high, this yields end-to-end speedups on memory-bandwidth-bound decoding without changing the model output distribution in theory (the standard verification scheme is statistically equivalent to sampling from the target).

OpenVINO GenAI implements three drafting strategies. They differ only in how candidates are produced — the target-side validation step is shared.

StrategyDrafterBest ForContinuous Batching (CPU/GPU)Stateful (NPU)
Speculative Decoding (Fast Draft)Smaller off-the-shelf LLMGeneral-purpose, robust speedup without retraining✓✓
Prompt Lookup DecodingN-gram match against the promptOutput copies large spans of the input (RAG, summarization, code editing)✓✗
EAGLE3Custom draft head trained on the target's hidden statesHighest acceptance rate; also supports tree drafting✓✓
info

All three strategies are available through the LLMPipeline Python and C++ APIs. Prompt Lookup Decoding and EAGLE3 are additionally supported in VLMPipeline (e.g. Qwen3-VL).

The Continuous Batching backend supports multi-request batching; the Stateful backend is single-request.

Every strategy is configured in two places: the drafter is selected when the pipeline is constructed, and the drafting behavior is tuned per request through GenerationConfig. The shared knob is num_assistant_tokens — the number of candidate tokens submitted to the target per iteration, defaulting to 5 when unset. The remaining knobs are strategy-specific and described in the corresponding sections below.

Measuring Effectiveness

Inspect result.extended_perf_metrics, which exposes main_model_metrics, draft_model_metrics and get_num_accepted_tokens().

The single most useful diagnostic is get_num_accepted_tokens() divided by the number of target iterations — the average accepted tokens per step. Aim for this to be well above 1; values close to 1 mean drafting is not paying for itself.

Speculative Decoding (Fast Draft)​

Fast Draft is the classic two-model setup: a smaller off-the-shelf LLM that shares the target's tokenizer drafts tokens autoregressively, and the target verifies them. It works with any target/draft pair you already have, without retraining. The speedup is bounded by how often the small model's distribution agrees with the large one.

Model Preparation​

Export both models with optimum-cli. The draft model must use the same tokenizer as the target:

optimum-cli export openvino --model Qwen/Qwen3-8B --weight-format int4 --task text-generation-with-past Qwen3-8B-ov-int4
optimum-cli export openvino --model Qwen/Qwen3-0.6B --weight-format int4 --task text-generation-with-past Qwen3-0.6B-ov-int4

Configuration and Usage​

In addition to num_assistant_tokens, Fast Draft supports:

  • assistant_confidence_threshold (float) — a dynamic-length stopping criterion. The drafter keeps proposing while the candidate token probability exceeds this threshold, then hands off to the target. It is mutually exclusive with num_assistant_tokens.
import openvino_genai

main_model_dir = "Qwen3-8B-ov-int4"
draft_model_dir = "Qwen3-0.6B-ov-int4"

draft_model = openvino_genai.draft_model(draft_model_dir, "CPU")
pipe = openvino_genai.LLMPipeline(main_model_dir, "CPU", draft_model=draft_model)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 5
# Or, on the Continuous Batching backend, dynamic-length drafting:
# config.assistant_confidence_threshold = 0.4

result = pipe.generate("The Sun is yellow because", config)
tip

Increase num_assistant_tokens until the accepted-tokens-per-step figure plateaus, then back off. Past the plateau, rejected draft tokens are pure overhead.

Backend Differences
  • Continuous Batching uses num_assistant_tokens as-is and runs the drafter and target as fully scheduled paged-attention pipelines, each with its own KV cache. The target and draft caches share the configured SchedulerConfig.cache_size, so increase it when the drafter is large.
  • Stateful uses num_assistant_tokens as the initial value and adapts it per step based on recent acceptance, falling back gracefully when acceptance is low.
warning

assistant_confidence_threshold is supported on the Continuous Batching backend only.

The Stateful backend requires NPU to be the execution device for at least one of the two models. For NPU deployments, the recommended configuration places both the target and the draft model on the NPU.

tip

Check out Python and C++ speculative decoding samples.

Prompt Lookup Decoding​

Prompt Lookup decoding replaces the draft model with a string match: the most recently emitted suffix is searched as an n-gram inside the prompt, and the tokens that follow each match are proposed as candidates. No second model is loaded.

This is highly effective for input-grounded generation — RAG, document QA, summarization, code editing, multi-turn chat — where the output frequently copies entity names, phrases or code chunks verbatim from the input. On these workloads it delivers speculative-decoding-class speedups at near-zero drafter cost. On workloads with low prompt/output overlap it degrades gracefully back to plain decoding, since the proposals are simply rejected.

Configuration and Usage​

Prompt Lookup is enabled at pipeline construction time with prompt_lookup=True; it cannot be switched on per request. In addition to num_assistant_tokens it supports:

  • max_ngram_size (size_t) — the maximum n-gram length matched against the prompt. Larger values produce longer, more confident continuations when they hit, but match less often.
import openvino_genai

main_model_dir = "Qwen3-8B-ov-int4"

pipe = openvino_genai.LLMPipeline(main_model_dir, "CPU", prompt_lookup=True)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 5
config.max_ngram_size = 3

pipe.generate("The Sun is yellow because", config)
warning

Prompt Lookup is available on the Continuous Batching backend only.

The draft_model and prompt_lookup constructor options are mutually exclusive — Prompt Lookup uses no draft model.

tip

Check out Python and C++ prompt lookup decoding samples.

EAGLE3​

EAGLE-3 (Extrapolation Algorithm for Greater Language-model Efficiency) replaces the generic draft model with a small head — typically one transformer layer — trained to mimic the target's distribution conditioned on the target's hidden states, not just on tokens. This much tighter coupling raises the per-token acceptance rate substantially over Fast Draft on the same target, while keeping the drafter cheap.

EAGLE3 can produce candidates either as a linear chain or as a tree.

Model Preparation​

Export both the target and the matching EAGLE3 draft head with optimum-cli. The draft repository must be an EAGLE3 head trained against the same target family — the head exposes a d2t mapping table that OpenVINO GenAI picks up automatically:

optimum-cli export openvino --model Qwen/Qwen3-8B --weight-format int4 --task text-generation-with-past --trust-remote-code Qwen3-8B-ov-int4
optimum-cli export openvino --model AngelSlim/Qwen3-8B_eagle3 --weight-format int4 --task text-generation-with-past --trust-remote-code Qwen3-8B_eagle3-ov-int4
Chain Or Tree?

Start with chain drafting: it has a single knob, and on a well-matched EAGLE3 head it already captures most of the available speedup.

Switch to tree drafting when acceptance per step has plateaued but the target still has spare compute per forward pass — typically on GPU at small batch sizes, where the target's step time grows sublinearly with the candidate count. Tree drafting trades a larger validation batch for a higher accepted-token count per step.

Chain Drafting​

Chain drafting runs the EAGLE3 head autoregressively for num_assistant_tokens iterations and submits the resulting linear chain to the target for verification. It needs no configuration beyond num_assistant_tokens.

import openvino_genai

main_model_dir = "Qwen3-8B-ov-int4"
draft_model_dir = "Qwen3-8B_eagle3-ov-int4"

draft_model = openvino_genai.draft_model(draft_model_dir, "GPU")
pipe = openvino_genai.LLMPipeline(main_model_dir, "GPU", draft_model=draft_model)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 5

pipe.generate("What is OpenVINO?", config)

Tree Drafting​

Instead of a single linear chain, the drafter expands the top branching_factor continuations at each of tree_depth layers, scores the resulting tree, and submits the highest-scoring num_assistant_tokens candidates to the target in one packed verification step. With a single target forward pass the pipeline can therefore validate multiple alternative continuations and accept the longest matching one, compounding EAGLE3's already-high acceptance rate into longer accepted runs.

Tree drafting adds two GenerationConfig fields:

  • branching_factor (size_t) — number of top-k expansions retained per tree node and per tree layer.
  • tree_depth (size_t) — number of draft iterations, i.e. tree layers. Setting tree_depth > 0 switches the request from chain drafting to tree drafting.
How Tree Drafting Works
  1. The EAGLE3 head runs tree_depth iterations. At each layer it expands the top branching_factor children of every surviving node, producing up to branching_factor^2 * (tree_depth - 1) + branching_factor candidate tokens organized as a tree.
  2. Candidates are scored cumulatively along their tree paths; the top num_assistant_tokens are flattened into a single packed input together with a tree attention mask and tree position ids.
  3. The target processes the packed input in one forward pass and verifies all paths simultaneously. The longest path whose tokens match the target's own samples is accepted.
  4. Accepted tokens become the new prefix, the draft and target KV caches are advanced to match, and the next round begins.

Two constraints follow from this, and both are enforced by GenerationConfig::validate():

  • The tree must be able to supply the requested number of candidates: branching_factor^2 * (tree_depth - 1) + branching_factor >= num_assistant_tokens.
  • num_assistant_tokens >= tree_depth, since an accepted path is at most tree_depth tokens long.
import openvino_genai

main_model_dir = "Qwen3-8B-ov-int4"
draft_model_dir = "Qwen3-8B_eagle3-ov-int4"

draft_model = openvino_genai.draft_model(draft_model_dir, "GPU")
pipe = openvino_genai.LLMPipeline(main_model_dir, "GPU", draft_model=draft_model)

config = openvino_genai.GenerationConfig()
config.max_new_tokens = 100
config.num_assistant_tokens = 15  # top-K candidates verified per step
config.branching_factor = 8       # top-k expansions per tree node
config.tree_depth = 4             # number of draft layers

pipe.generate("What is OpenVINO?", config)
tip

The trio (branching_factor, tree_depth, num_assistant_tokens) is the main tuning surface. A reasonable starting point on a high-acceptance target is branching_factor=4..8 and tree_depth=3..4, with num_assistant_tokens chosen to match how many candidate tokens the target can verify per step on your hardware.

warning

Tree drafting is an EAGLE3-only feature; Fast Draft and Prompt Lookup ignore branching_factor and tree_depth.

It cannot be combined with beam search or multinomial sampling — set tree_depth=0, do_sample=false or num_beams=1 accordingly.

EAGLE3 Streaming

Streaming is restricted to batch size 1, with either greedy decoding or tree search. Beam search and multinomial sampling are not supported.

assistant_confidence_threshold must be 0 for EAGLE3 — dynamic-length drafting is a Fast Draft feature.