Image Generation Using Diffusers

Convert and Optimize Model

Download and convert model (e.g. stabilityai/stable-diffusion-xl-base-1.0) to OpenVINO format from Hugging Face:

optimum-cli export openvino --model stabilityai/stable-diffusion-xl-base-1.0 --weight-format int4 --trust-remote-code stable_diffusion_xl_base_1_0_ov

See all supported Image Generation Models.

info

Refer to the Model Preparation guide for detailed instructions on how to download, convert and optimize models for OpenVINO GenAI.

Run Model Using OpenVINO GenAI

OpenVINO GenAI supports the following diffusion model pipelines:

• Text2ImagePipeline for creating images from text prompts.
• Image2ImagePipeline for modifying existing images based on prompts.
• InpaintingPipeline for selectively replacing portions of images using masks.

Text2ImagePipeline

Python

CPU

import openvino_genai as ov_genai
from PIL import Image

pipe = ov_genai.Text2ImagePipeline(model_path, "CPU")
image_tensor = pipe.generate(prompt)

image = Image.fromarray(image_tensor.data[0])
image.save("image.bmp")

GPU

import openvino_genai as ov_genai
from PIL import Image

pipe = ov_genai.Text2ImagePipeline(model_path, "GPU")
image_tensor = pipe.generate(prompt)

image = Image.fromarray(image_tensor.data[0])
image.save("image.bmp")

C++

CPU

info

Code below requires installation of C++ compatible package. See here for additional setup details, or How to Build OpenVINO™ GenAI APP in C++ blog for full instruction.

#include "openvino/genai/image_generation/text2image_pipeline.hpp"
#include "imwrite.hpp"

int main(int argc, char* argv[]) {
  const std::string models_path = argv[1], prompt = argv[2];

  ov::genai::Text2ImagePipeline pipe(models_path, "CPU");
  ov::Tensor image = pipe.generate(prompt);

  imwrite("image.bmp", image, true);
}

GPU

#include "openvino/genai/image_generation/text2image_pipeline.hpp"
#include "imwrite.hpp"

int main(int argc, char* argv[]) {
  const std::string models_path = argv[1], prompt = argv[2];

  ov::genai::Text2ImagePipeline pipe(models_path, "GPU");
  ov::Tensor image = pipe.generate(prompt);

  imwrite("image.bmp", image, true);
}

JavaScript

CPU

import { writeFile } from 'node:fs/promises';
import bmp from 'bmp-js';
import { Text2ImagePipeline } from 'openvino-genai-node';

const pipeline = await Text2ImagePipeline(modelPath, 'CPU');
const image = await pipeline.generate(prompt);

// Save the first generated image as BMP
const [height, width] = [image.getShape()[1], image.getShape()[2]];
const rgb = Uint8Array.from(image.data);
const abgr = Buffer.allocUnsafe(width * height * 4);

for (let src = 0, dst = 0; src < rgb.length; src += 3, dst += 4) {
abgr[dst] = 255;              // A
abgr[dst + 1] = rgb[src + 2]; // B
abgr[dst + 2] = rgb[src + 1]; // G
abgr[dst + 3] = rgb[src];     // R
}

const bmpData = bmp.encode({ width, height, data: abgr });
await writeFile('image.bmp', bmpData.data);

GPU

import { writeFile } from 'node:fs/promises';
import bmp from 'bmp-js';
import { Text2ImagePipeline } from 'openvino-genai-node';

const pipeline = await Text2ImagePipeline(modelPath, 'GPU');
const image = await pipeline.generate(prompt);

// Save the first generated image as BMP
const [height, width] = [image.getShape()[1], image.getShape()[2]];
const rgb = Uint8Array.from(image.data);
const abgr = Buffer.allocUnsafe(width * height * 4);

for (let src = 0, dst = 0; src < rgb.length; src += 3, dst += 4) {
abgr[dst] = 255;              // A
abgr[dst + 1] = rgb[src + 2]; // B
abgr[dst + 2] = rgb[src + 1]; // G
abgr[dst + 3] = rgb[src];     // R
}

const bmpData = bmp.encode({ width, height, data: abgr });
await writeFile('image.bmp', bmpData.data);

tip

Use CPU or GPU as devices without any other code change.

Image2ImagePipeline

Python

CPU

import openvino_genai as ov_genai
import openvino as ov
from PIL import Image
import numpy as np

def read_image(path: str) -> ov.Tensor:
  pic = Image.open(path).convert("RGB")
  image_data = np.array(pic)[None]
  return ov.Tensor(image_data)

input_image_data = read_image("input_image.jpg")

pipe = ov_genai.Image2ImagePipeline(model_path, "CPU")
image_tensor = pipe.generate(prompt, image=input_image_data, strength=0.8)

image = Image.fromarray(image_tensor.data[0])
image.save("image.bmp")

GPU

import openvino_genai as ov_genai
import openvino as ov
from PIL import Image
import numpy as np

def read_image(path: str) -> ov.Tensor:
  pic = Image.open(path).convert("RGB")
  image_data = np.array(pic)[None]
  return ov.Tensor(image_data)

input_image_data = read_image("input_image.jpg")

pipe = ov_genai.Image2ImagePipeline(model_path, "GPU")
image_tensor = pipe.generate(prompt, image=input_image_data, strength=0.8)

image = Image.fromarray(image_tensor.data[0])
image.save("image.bmp")

C++

CPU

#include "openvino/genai/image_generation/image2image_pipeline.hpp"
#include "load_image.hpp"
#include "imwrite.hpp"

int main(int argc, char* argv[]) {
  const std::string models_path = argv[1], prompt = argv[2], image_path = argv[3];

  ov::Tensor input_image = utils::load_image(image_path);

  ov::genai::Image2ImagePipeline pipe(models_path, "CPU");
  ov::Tensor generated_image = pipe.generate(prompt, input_image, ov::genai::strength(0.8f));

  imwrite("image.bmp", generated_image, true);
}

GPU

#include "openvino/genai/image_generation/image2image_pipeline.hpp"
#include "load_image.hpp"
#include "imwrite.hpp"

int main(int argc, char* argv[]) {
  const std::string models_path = argv[1], prompt = argv[2], image_path = argv[3];

  ov::Tensor input_image = utils::load_image(image_path);

  ov::genai::Image2ImagePipeline pipe(models_path, "GPU");
  ov::Tensor generated_image = pipe.generate(prompt, input_image, ov::genai::strength(0.8f));

  imwrite("image.bmp", generated_image, true);
}

JavaScript

CPU

import { readFile, writeFile } from 'node:fs/promises';
import bmp from 'bmp-js';
import { addon as ov } from 'openvino-node';
import { Image2ImagePipeline } from 'openvino-genai-node';

async function readImage(path) {
const { width, height, data: abgr } = bmp.decode(await readFile(path));
const rgb = new Uint8Array(width * height * 3);
for (let src = 0, dst = 0; src < abgr.length; src += 4, dst += 3) {
  rgb[dst]     = abgr[src + 3]; // R
  rgb[dst + 1] = abgr[src + 2]; // G
  rgb[dst + 2] = abgr[src + 1]; // B
}
const tensor = new ov.Tensor('u8', [1, height, width, 3], rgb);
tensor._buffer = rgb; // prevent GC from collecting the backing buffer
return tensor;
}

const inputImage = await readImage('input_image.bmp');

const pipeline = await Image2ImagePipeline(modelPath, 'CPU');
const image = await pipeline.generate(prompt, inputImage, { strength: 0.8 });

// Save the generated image as BMP
const [height, width] = [image.getShape()[1], image.getShape()[2]];
const rgb = Uint8Array.from(image.data);
const abgr = Buffer.allocUnsafe(width * height * 4);

for (let src = 0, dst = 0; src < rgb.length; src += 3, dst += 4) {
abgr[dst]     = 255;          // A
abgr[dst + 1] = rgb[src + 2]; // B
abgr[dst + 2] = rgb[src + 1]; // G
abgr[dst + 3] = rgb[src];     // R
}

const bmpData = bmp.encode({ width, height, data: abgr });
await writeFile('image.bmp', bmpData.data);

GPU

import { readFile, writeFile } from 'node:fs/promises';
import bmp from 'bmp-js';
import { addon as ov } from 'openvino-node';
import { Image2ImagePipeline } from 'openvino-genai-node';

async function readImage(path) {
const { width, height, data: abgr } = bmp.decode(await readFile(path));
const rgb = new Uint8Array(width * height * 3);
for (let src = 0, dst = 0; src < abgr.length; src += 4, dst += 3) {
  rgb[dst]     = abgr[src + 3]; // R
  rgb[dst + 1] = abgr[src + 2]; // G
  rgb[dst + 2] = abgr[src + 1]; // B
}
const tensor = new ov.Tensor('u8', [1, height, width, 3], rgb);
tensor._buffer = rgb; // prevent GC from collecting the backing buffer
return tensor;
}

const inputImage = await readImage('input_image.bmp');

const pipeline = await Image2ImagePipeline(modelPath, 'GPU');
const image = await pipeline.generate(prompt, inputImage, { strength: 0.8 });

// Save the generated image as BMP
const [height, width] = [image.getShape()[1], image.getShape()[2]];
const rgb = Uint8Array.from(image.data);
const abgr = Buffer.allocUnsafe(width * height * 4);

for (let src = 0, dst = 0; src < rgb.length; src += 3, dst += 4) {
abgr[dst]     = 255;          // A
abgr[dst + 1] = rgb[src + 2]; // B
abgr[dst + 2] = rgb[src + 1]; // G
abgr[dst + 3] = rgb[src];     // R
}

const bmpData = bmp.encode({ width, height, data: abgr });
await writeFile('image.bmp', bmpData.data);

InpaintingPipeline

Python

CPU

import openvino_genai as ov_genai
import openvino as ov
from PIL import Image
import numpy as np

def read_image(path: str) -> ov.Tensor:
  pic = Image.open(path).convert("RGB")
  image_data = np.array(pic)[None]
  return ov.Tensor(image_data)

input_image_data = read_image("input_image.jpg")
mask_image = read_image("mask.jpg")

pipe = ov_genai.InpaintingPipeline(model_path, "CPU")
image_tensor = pipe.generate(prompt, image=input_image_data, mask_image=mask_image)

image = Image.fromarray(image_tensor.data[0])
image.save("image.bmp")

GPU

import openvino_genai as ov_genai
import openvino as ov
from PIL import Image
import numpy as np

def read_image(path: str) -> ov.Tensor:
  pic = Image.open(path).convert("RGB")
  image_data = np.array(pic)[None]
  return ov.Tensor(image_data)

input_image_data = read_image("input_image.jpg")
mask_image = read_image("mask.jpg")

pipe = ov_genai.InpaintingPipeline(model_path, "GPU")
image_tensor = pipe.generate(prompt, image=input_image_data, mask_image=mask_image)

image = Image.fromarray(image_tensor.data[0])
image.save("image.bmp")

C++

CPU

#include "openvino/genai/image_generation/inpainting_pipeline.hpp"
#include "load_image.hpp"
#include "imwrite.hpp"

int main(int argc, char* argv[]) {
  const std::string models_path = argv[1], prompt = argv[2];

  ov::Tensor input_image = utils::load_image(argv[3]);
  ov::Tensor mask_image = utils::load_image(argv[4]);

  ov::genai::InpaintingPipeline pipe(models_path, "CPU");
  ov::Tensor generated_image = pipe.generate(prompt, input_image, mask_image);

  imwrite("image.bmp", generated_image, true);
}

GPU

#include "openvino/genai/image_generation/inpainting_pipeline.hpp"
#include "load_image.hpp"
#include "imwrite.hpp"

int main(int argc, char* argv[]) {
  const std::string models_path = argv[1], prompt = argv[2];

  ov::Tensor input_image = utils::load_image(argv[3]);
  ov::Tensor mask_image = utils::load_image(argv[4]);

  ov::genai::InpaintingPipeline pipe(models_path, "GPU");
  ov::Tensor generated_image = pipe.generate(prompt, input_image, mask_image);

  imwrite("image.bmp", generated_image, true);
}

JavaScript

CPU

import { readFile, writeFile } from 'node:fs/promises';
import bmp from 'bmp-js';
import { addon as ov } from 'openvino-node';
import { InpaintingPipeline } from 'openvino-genai-node';

async function readImage(path) {
const { width, height, data: abgr } = bmp.decode(await readFile(path));
const rgb = new Uint8Array(width * height * 3);
for (let src = 0, dst = 0; src < abgr.length; src += 4, dst += 3) {
  rgb[dst]     = abgr[src + 3]; // R
  rgb[dst + 1] = abgr[src + 2]; // G
  rgb[dst + 2] = abgr[src + 1]; // B
}
const tensor = new ov.Tensor('u8', [1, height, width, 3], rgb);
tensor._buffer = rgb; // prevent GC from collecting the backing buffer
return tensor;
}

const inputImage = await readImage('input_image.bmp');
const maskImage = await readImage('mask_image.bmp');

const pipeline = await InpaintingPipeline(modelPath, 'CPU');
const image = await pipeline.generate(prompt, inputImage, maskImage);

// Save the generated image as BMP
const [height, width] = [image.getShape()[1], image.getShape()[2]];
const rgb = Uint8Array.from(image.data);
const abgr = Buffer.allocUnsafe(width * height * 4);

for (let src = 0, dst = 0; src < rgb.length; src += 3, dst += 4) {
abgr[dst]     = 255;          // A
abgr[dst + 1] = rgb[src + 2]; // B
abgr[dst + 2] = rgb[src + 1]; // G
abgr[dst + 3] = rgb[src];     // R
}

const bmpData = bmp.encode({ width, height, data: abgr });
await writeFile('image.bmp', bmpData.data);

GPU

import { readFile, writeFile } from 'node:fs/promises';
import bmp from 'bmp-js';
import { addon as ov } from 'openvino-node';
import { InpaintingPipeline } from 'openvino-genai-node';

async function readImage(path) {
const { width, height, data: abgr } = bmp.decode(await readFile(path));
const rgb = new Uint8Array(width * height * 3);
for (let src = 0, dst = 0; src < abgr.length; src += 4, dst += 3) {
  rgb[dst]     = abgr[src + 3]; // R
  rgb[dst + 1] = abgr[src + 2]; // G
  rgb[dst + 2] = abgr[src + 1]; // B
}
const tensor = new ov.Tensor('u8', [1, height, width, 3], rgb);
tensor._buffer = rgb; // prevent GC from collecting the backing buffer
return tensor;
}

const inputImage = await readImage('input_image.bmp');
const maskImage = await readImage('mask_image.bmp');

const pipeline = await InpaintingPipeline(modelPath, 'GPU');
const image = await pipeline.generate(prompt, inputImage, maskImage);

// Save the generated image as BMP
const [height, width] = [image.getShape()[1], image.getShape()[2]];
const rgb = Uint8Array.from(image.data);
const abgr = Buffer.allocUnsafe(width * height * 4);

for (let src = 0, dst = 0; src < rgb.length; src += 3, dst += 4) {
abgr[dst]     = 255;          // A
abgr[dst + 1] = rgb[src + 2]; // B
abgr[dst + 2] = rgb[src + 1]; // G
abgr[dst + 3] = rgb[src];     // R
}

const bmpData = bmp.encode({ width, height, data: abgr });
await writeFile('image.bmp', bmpData.data);

Additional Usage Options

tip

Check out Python, C++, and JavaScript image generation samples.

Use Different Generation Parameters

Generation Configuration Workflow

1. Get the model default config with get_generation_config()
2. Modify parameters
3. Apply the updated config using one of the following methods:
   • Use set_generation_config(config)
   • Pass config directly to generate() (e.g. generate(prompt, config))
   • Specify options as inputs in the generate() method (e.g. generate(prompt, max_new_tokens=100))

Image Generation Configuration

You can adjust several parameters to control the image generation process, including dimensions and the number of inference steps:

Python

import openvino_genai as ov_genai
from PIL import Image

pipe = ov_genai.Text2ImagePipeline(model_path, "CPU")
image_tensor = pipe.generate(
  prompt,
  width=512,
  height=512,
  num_images_per_prompt=1,
  num_inference_steps=30,
  guidance_scale=7.5
)

image = Image.fromarray(image_tensor.data[0])
image.save("image.bmp")

C++

#include "openvino/genai/image_generation/text2image_pipeline.hpp"
#include "imwrite.hpp"

int main(int argc, char* argv[]) {
    const std::string models_path = argv[1], prompt = argv[2];

    ov::genai::Text2ImagePipeline pipe(models_path, "CPU");
    ov::Tensor image = pipe.generate(
        prompt,
        ov::genai::width(512),
        ov::genai::height(512),
        ov::genai::num_images_per_prompt(1),
        ov::genai::num_inference_steps(30),
        ov::genai::guidance_scale(7.5f)
    );

    imwrite("image.bmp", image, true);
}

JavaScript

import { Text2ImagePipeline } from 'openvino-genai-node';

const pipeline = await Text2ImagePipeline(modelPath, 'CPU');
const result = await pipeline.generate(prompt, {
    width: 512,
    height: 512,
    num_images_per_prompt: 1,
    num_inference_steps: 30,
    guidance_scale: 7.5,
});

Understanding Image Generation Parameters

• width: The width of resulting image(s).
• height: The height of resulting image(s).
• num_images_per_prompt: Specifies how many image variations to generate in a single request for the same prompt.
• num_inference_steps: Defines denoising iteration count. Higher values increase quality and generation time, lower values generate faster with less detail.
• guidance_scale: Balances prompt adherence vs. creativity. Higher values follow prompt more strictly, lower values allow more creative freedom.
• rng_seed: Controls randomness for reproducible results. Same seed produces identical images across runs.

For the full list of generation parameters, refer to the Image Generation Config API.

Working with LoRA Adapters

For image generation models like Stable Diffusion, LoRA adapters can modify the generation process to produce images with specific artistic styles, content types, or quality enhancements.

Refer to the LoRA Adapters for more details on working with LoRA adapters.

Performance Metrics

Image generation pipelines (Text2ImagePipeline, Image2ImagePipeline, InpaintingPipeline) have get_performance_metrics() method that returns an ImageGenerationPerfMetrics instance, which provides:

Metric	Type	Description
get_generate_duration()	float, ms	Total duration of the generate() call
get_inference_duration()	float, ms	Total model inference time
get_load_time()	float, ms	Model load time
get_iteration_duration()	MeanStdPair, ms	Per-step denoising iteration duration
get_unet_infer_duration()	MeanStdPair, ms	UNet inference duration per step (UNet-based models)
get_transformer_infer_duration()	MeanStdPair, ms	Transformer inference duration per step (DiT-based models)
get_text_encoder_infer_duration()	dict[str, float], ms	Inference duration per text encoder
get_vae_encoder_infer_duration()	float, ms	VAE encoder inference duration
get_vae_decoder_infer_duration()	float, ms	VAE decoder inference duration
get_first_and_other_iter_duration()	tuple(float, float), ms	First iteration vs average of remaining iterations
get_first_and_other_unet_infer_duration()	tuple(float, float), ms	First UNet inference iteration vs average of remaining iterations
get_first_and_other_trans_infer_duration()	tuple(float, float), ms	First Transformer inference iteration vs average of remaining iterations

Raw values are available in the raw_metrics field.

tip

For more details, refer to the Python and C++ image generation benchmark samples.