Skip to main content

Video Generation Using Diffusers

LTX-Video Pipeline for Video Generation​

LTX-Video is the DiT-based video generation model capable of generating high-quality videos from text prompts or conditioning images.

Text-to-Video Pipeline​

The text-to-video pipeline turns a text prompt into a sequence of video frames:

  1. Prompt → tokens: the prompt is tokenized (and padded to the model’s max length).
  2. Tokens → text embeddings: a text encoder produces embeddings used to condition generation.
  3. Denoising loop: a Transformer predicts and refines video latents over multiple steps, while the scheduler controls the noise level at each step.
  4. Latents → frames: the final latents are decoded into RGB video frames.

Image-to-Video Pipeline​

The image-to-video pipeline extends the text-to-video workflow with image conditioning:

  1. Image → latents: the conditioning image is VAE-encoded into latent space and used to anchor frame 0.
  2. Prompt + image embeddings → denoising: the Transformer generates video latents conditioned on both the text prompt and the encoded image.
  3. Latents → frames: the final latents are decoded into RGB video frames, with the first frame matching the input image.

LTX Pipeline Workflow Source: arXiv:2501.00103

Convert and Optimize Model​

Download and convert LTX model Lightricks/LTX-Video to OpenVINO format from Hugging Face.

For text-to-video:

optimum-cli export openvino --model Lightricks/LTX-Video --weight-format int8 --task text-to-video --trust-remote-code LTX_Video_ov

For image-to-video (includes VAE encoder):

optimum-cli export openvino --model Lightricks/LTX-Video --weight-format fp32 --task image-to-video --trust-remote-code LTX_Video_i2v_ov
info

Refer to the Model Preparation guide for detailed instructions on how to download, convert and optimize models for OpenVINO GenAI.

Run Model Using OpenVINO GenAI​

OpenVINO GenAI supports the following video generation pipelines:

Text to Video​

import openvino_genai as ov_genai
import cv2

def save_video(filename: str, video_tensor, fps: int = 25):
  batch_size, num_frames, height, width, _ = video_tensor.shape
  video_data = video_tensor.data

  for b in range(batch_size):
      if batch_size == 1:
          output_path = filename
      else:
          base, ext = filename.rsplit(".", 1) if "." in filename else (filename, "avi")
          output_path = f"{base}_b{b}.{ext}"

      fourcc = cv2.VideoWriter_fourcc(*"MJPG")
      writer = cv2.VideoWriter(output_path, fourcc, fps, (width, height))

      for f in range(num_frames):
          frame_bgr = cv2.cvtColor(video_data[b, f], cv2.COLOR_RGB2BGR)
          writer.write(frame_bgr)

      writer.release()
      print(f"Wrote {output_path} ({num_frames} frames, {width}x{height} @ {fps} fps)")


model_path = "path/to/model"  # Path to the model directory
prompt = "your video generation prompt"

pipe = ov_genai.Text2VideoPipeline(model_path, "CPU")
video = pipe.generate(prompt).video

save_video("genai_video.avi", video)

Image to Video​

The Image2VideoPipeline requires a model exported with optimum-intel from the main branch.

import openvino_genai as ov_genai
import openvino as ov
import numpy as np
import cv2

def load_image(image_path: str) -> ov.Tensor:
  img = cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)
  return ov.Tensor(img[np.newaxis])  # [1, H, W, 3] uint8

def save_video(filename: str, video_tensor, fps: int = 25):
  batch_size, num_frames, height, width, _ = video_tensor.shape
  video_data = video_tensor.data

  for b in range(batch_size):
      if batch_size == 1:
          output_path = filename
      else:
          base, ext = filename.rsplit(".", 1) if "." in filename else (filename, "avi")
          output_path = f"{base}_b{b}.{ext}"

      fourcc = cv2.VideoWriter_fourcc(*"MJPG")
      writer = cv2.VideoWriter(output_path, fourcc, fps, (width, height))

      for f in range(num_frames):
          frame_bgr = cv2.cvtColor(video_data[b, f], cv2.COLOR_RGB2BGR)
          writer.write(frame_bgr)

      writer.release()
      print(f"Wrote {output_path} ({num_frames} frames, {width}x{height} @ {fps} fps)")


model_path = "path/to/model"   # Path to the model directory
image_path = "path/to/image.png"  # Path to the conditioning image
prompt = "your video generation prompt"

image = load_image(image_path)
pipe = ov_genai.Image2VideoPipeline(model_path, "CPU")
video = pipe.generate(image, prompt).video

save_video("genai_video.avi", video)
tip

Use CPU or GPU as devices without any other code change.

Additional Usage Options​

tip

Check out Python and C++ video generation samples.

Use Different Generation Parameters​

Generation Configuration Workflow​

  1. Get the model default config with get_generation_config()
  2. Modify parameters
  3. Apply the updated config using one of the following methods:
    • Use set_generation_config(config)
    • Pass config directly to generate() (e.g. generate(prompt, config))
    • Specify options as inputs in the generate() method (e.g. generate(prompt, max_new_tokens=100))

Video Generation Configuration​

You can adjust several parameters to control the video generation process, including dimensions and the number of inference steps:

import openvino_genai as ov_genai

pipe = ov_genai.Text2VideoPipeline(model_path, "CPU")
fps = 25
video = pipe.generate(
  prompt,
  width=512,
  height=512,
  num_videos_per_prompt=1,
  num_inference_steps=30,
  num_frames=161,
  guidance_scale=7.5,
  frame_rate=fps
).video

save_video("genai_video.avi", video, fps)

Working with LoRA Adapters​

For video generation models like LTX-Video, LoRA adapters can modify the generation process to produce videos with specific artistic styles, content types, or quality enhancements.

Refer to the LoRA Adapters for more details on working with LoRA adapters.

Understanding Video Generation Parameters
  • negative_prompt: Negative prompt for video(s) generation.
  • width: The width of resulting video(s).
  • height: The height of resulting video(s).
  • num_frames: Number of frames to generate for each video. Higher values produce longer videos but increase compute and memory.
  • num_videos_per_prompt: Specifies how many video variations to generate in a single request for the same prompt.
  • num_inference_steps: Defines denoising iteration count. Higher values increase quality and generation time, lower values generate faster with less detail.
  • guidance_scale: Balances prompt adherence vs. creativity. Higher values follow prompt more strictly, lower values allow more creative freedom.
  • generator: Controls randomness for reproducible results. Same generator seed produces identical videos across runs.
  • frame_rate: Target video FPS used by the pipeline.

For the full list of generation parameters, refer to the Video Generation Config API.

Performance Metrics​

Text2VideoPipeline returns VideoGenerationResult with a perf_metrics field (performance_stat in C++) of type VideoGenerationPerfMetrics, which inherits from ImageGenerationPerfMetrics.

Refer to the Image Generation Performance Metrics for details.