Skip to main content

OpenVINO GenAI Text Generation C Samples

Table of Contents​

  1. Download OpenVINO GenAI
  2. Build Samples
  3. Download and Convert the Model and Tokenizers
  4. Sample Descriptions
  5. Support and Contribution

Download OpenVINO GenAI​

Download and extract OpenVINO GenAI Archive Visit the OpenVINO Download Page.

Build Samples​

Set up the environment and build the samples Linux and macOS

source <INSTALL_DIR>/setupvars.sh
./<INSTALL_DIR>/samples/c/build_samples.sh

Windows Command Prompt:

<INSTALL_DIR>\setupvars.bat
<INSTALL_DIR>\samples\c\build_samples_msvc.bat

Windows PowerShell

.<INSTALL_DIR>\setupvars.ps1
.<INSTALL_DIR>\samples\c\build_samples.ps1

Download and convert the model and tokenizers​

The --upgrade-strategy eager option is needed to ensure optimum-intel is upgraded to the latest version. Install ../../export-requirements.txt if model conversion is required.

pip install --upgrade-strategy eager -r ../../export-requirements.txt
optimum-cli export openvino --model <model> <output_folder>

If a converted model in OpenVINO IR format is available in the OpenVINO optimized LLMs collection on Hugging Face, you can download it directly via huggingface-cli.

pip install huggingface-hub
huggingface-cli download <model> --local-dir <output_folder>

Sample Descriptions​

Chat Sample (chat_sample_c)​

Multi-turn conversations with an interactive chat interface powered by OpenVINO.

  • Run Command:
./chat_sample_c model_dir

LLMs benchamrking sample(benchmark_genai_c)​

The sample demonstrates how to benchmark LLMs in OpenVINO GenAI by using C language.

  • Run Command:
./benchmark_gena_c  [-m MODEL] [-p PROMPT] [-nw NUM_WARMUP] [-n NUM_ITER] [-mt MAX_NEW_TOKENS] [-d DEVICE]
  • Options:
  • -m, --model: Path to the model and tokenizers base directory.
  • -p, --prompt (default: "The Sky is blue because"): The prompt to generate text.
  • -nw, --num_warmup (default: 1): Number of warmup iterations.
  • -mt, --max_new_tokens (default: 20): Maximal number of new tokens.
  • -n, --num_iter (default: 3): Number of iterations.
  • -d, --device (default: "CPU"): Device to run the model on.

Greedy Causal LM(greedy_causal_lm)​

Basic text generation using a causal language model.

  • Run Command:
./greedy_causal_lm_c  model_dir prompt

Support and Contribution​