Run GGUF models

For optimal performance and memory consumption, use prepared OpenVINO IR models from OpenVINO on Hugging Face whenever a suitable model is available. These models are created by exporting and compressing Hugging Face models with Optimum Intel. If no prepared IR is available, you can use the same export path to create one from the source model.

Use the GGUF frontend when your model is available only in GGUF format or you need to run an existing GGUF file directly.

OpenVINO GenAI can load a local .gguf file directly, including its tokenizer metadata. No Optimum export is required. ENABLE_GGUF is enabled by default in GenAI CMake. Use a compatible OpenVINO build that includes the GGUF frontend.

The default GGUF_READER="LEGACY" supports the llama, qwen2, and qwen3 architectures. Select GGUF_READER="FRONTEND" explicitly to use the OpenVINO GGUF frontend, which supports more architectures and is expected to provide better model quality through more complete handling of GGUF metadata.

Preview feature

The GGUF frontend is in preview. Architecture, quantization, tokenizer, and attention-backend support depend on the OpenVINO version. Generated output can differ from other runtimes. If the default legacy reader fails, GenAI includes a suggestion to try the frontend in the error message; retry by selecting it explicitly.

Prefer models quantized in Q4_0 or Q8_0 format for better performance and accuracy. Their quantization layouts are closest to those used by OpenVINO.

Generate text

Replace model.gguf with the path to your downloaded model. For a base model without a chat template, set apply_chat_template=False in Python or config.apply_chat_template = false in C++.

Python
C++

import openvino_genai as genai

pipe = genai.LLMPipeline("model.gguf", "CPU", GGUF_READER="FRONTEND")
print(pipe.generate("Explain why the sky is blue.", max_new_tokens=64))

#include <iostream>
#include <openvino/genai/gguf.hpp>
#include <openvino/genai/llm_pipeline.hpp>

int main() {
    ov::genai::LLMPipeline pipe(
        "model.gguf", "CPU", ov::genai::gguf_reader("FRONTEND"));
    ov::genai::GenerationConfig config;
    config.max_new_tokens = 64;
    std::cout << pipe.generate("Explain why the sky is blue.", config);
}

Use the tokenizer and stream a chat

The pipeline constructs its tokenizer from the frontend's model metadata. You can also construct genai.GGUFTokenizer("model.gguf") independently, without exporting tokenizer IR files. In C++, include <openvino/genai/gguf.hpp> and use ov::genai::GGUFTokenizer. The existing Tokenizer("model.gguf") path also remains supported.

import openvino_genai as genai

pipe = genai.LLMPipeline("model.gguf", "CPU", GGUF_READER="FRONTEND")
tokenizer = pipe.get_tokenizer()
encoded = tokenizer.encode("Hello", add_special_tokens=False)
print(tokenizer.decode(encoded.input_ids))

history = genai.ChatHistory()
history.append({"role": "user", "content": "Explain gravity in one sentence."})
result = pipe.generate(
    history,
    max_new_tokens=64,
    streamer=lambda text: print(text, end="", flush=True),
)
history.append({"role": "assistant", "content": result.texts[0]})
history.append({"role": "user", "content": "Now give an example."})
print(pipe.generate(history, max_new_tokens=64))

Chat requires a suitable template in the GGUF metadata. See Tokenization for configuring the tokenizer and Streaming for streamer behavior.

Generate text​

Use the tokenizer and stream a chat​

Generate text

Use the tokenizer and stream a chat