ComfyUI Video Workflows: How AI Video Pipelines Fit Together

ComfyUI video workflows make complex AI video pipelines visible. Instead of hiding generation behind one button, a graph can show exactly how text, images, keyframes, control signals, video models, samplers, decoders, upscalers and export nodes fit together.

This is especially useful for video because a production pipeline usually contains more stages than a standard image workflow — and because video generation is dramatically more sensitive to VRAM and generation time than image generation, which makes understanding the actual cost of each stage worth doing before you build a large graph.

The basic anatomy of a video workflow

Most modern ComfyUI video graphs contain some variation of these stages:

1. load the model and required encoders;
2. prepare text and/or image conditioning;
3. set target dimensions, frame count and frame rate;
4. run the video diffusion/transformer sampling stage;
5. decode latent frames;
6. optionally upscale or interpolate;
7. encode the frame sequence into a video file;
8. optionally add or generate audio.

Text-to-video

In a text-to-video graph, the prompt provides the main conditioning. The model must infer subject appearance, scene composition and motion from text alone.

Text-to-video is flexible but can be harder to control than image-to-video because there is no fixed source frame anchoring identity or composition.

Image-to-video

Image-to-video starts from a still image. The image provides a strong visual reference while the prompt or motion controls describe what should happen over time.

This often works well for advertising, character animation and cinematic shots because you can design or generate the first frame separately before asking the video model to animate it.

Keyframes and first/last-frame workflows

More advanced video models support multiple image conditions or first/last-frame generation. These workflows can make camera and scene transitions more predictable because the model has explicit visual targets across time.

Control models

Modern pipelines can include depth, edge, pose, motion or camera controls. In ComfyUI these controls usually appear as additional branches feeding conditioning into the main model.

LTX-2, for example, supports three IC-LoRA control models in its official ComfyUI workflows: Canny (edge-based structure), Depth and Pose, each usable for video-to-video structural guidance.

LoRAs

LoRAs can be used for style, subjects or specialized control behavior. Video LoRAs are more demanding to train and validate than image LoRAs because temporal consistency also matters.

Frame count, resolution and FPS — what it actually costs

These settings have a direct effect on memory and generation time, and the effect is large enough to make or break whether a graph runs at all on a given card. A GPU benchmark of Wan 2.1’s 14B text-to-video model (methodology: 33 frames, 16fps, 2-second clips, 30 sampling steps) found generation times ranging from under 90 seconds to over 18 minutes depending on GPU and resolution — and found that 24GB consumer cards couldn’t complete the 720p run at all:

The practical takeaway the benchmark draws is not “buy an H100” — it’s that 24GB consumer cards need to drop to a smaller variant (Wan’s 1.3B model, at reduced quality) or a more memory-efficient precision to reach 720p at all. A common production strategy is to generate at the model’s comfortable native resolution and frame count, then use spatial and temporal upscaling as separate stages rather than fighting for headroom in the main sampling pass.

VRAM optimization techniques ComfyUI pipelines actually use

When a graph runs out of memory, the fix usually isn’t a vague “reduce settings” — it’s one of a handful of specific, well-documented moves: lower precision (FP8 or GGUF quantization), offload the text encoder to CPU RAM, or drop to a smaller model variant. Third-party VRAM testing across Wan 2.1/2.2’s three model sizes shows how much headroom each move buys:

Offloading the T5-XXL text encoder to CPU RAM — a standard node option in Wan ComfyUI workflows — removes roughly 9GB from GPU memory for most of the generation, at a cost of an extra 10-20 seconds during the initial conditioning pass. That’s a trade most people on a single consumer GPU take without hesitation, but it’s worth knowing it’s happening rather than treating the slower first step as unexplained lag.

Upscaling and interpolation

Spatial upscaling increases image resolution. Temporal upscaling or frame interpolation increases frame rate or smoothness.

Some model families provide native latent upscalers, while other workflows rely on separate video upscaling/interpolation tools. NVIDIA’s own published figures for running LTX-2 through ComfyUI on RTX GPUs show what format choice alone does to both numbers at once: switching the model to NVFP4 format is reported as roughly 3x faster with about 60% less VRAM than the baseline, while NVFP8 gives roughly 2x faster generation with about 40% less VRAM. Those are NVIDIA’s own optimization figures rather than independent third-party testing, so treat them as a vendor-reported ceiling rather than a number you’re guaranteed to hit.

Audio

Traditional AI video workflows generate silent clips and add audio later. Newer systems such as LTX-2 can generate synchronized audio and video in one model — motion, dialogue, sound effects and music together in a single pass — changing the graph design because dialogue, effects and motion can be produced together instead of layered on afterward.

LTX-2 in ComfyUI

The official ComfyUI documentation describes LTX-2 as a 19B DiT-based audio-video foundation model supporting synchronized audio/video generation, text-to-video, image-to-video, video-to-video with the Canny/Depth/Pose IC-LoRA controls mentioned above, keyframe-driven interpolation, and native spatial (2x) and temporal (2x) upscaling. Five official checkpoints are published, including a distilled variant that runs in 8 steps at CFG=1 and fp8-quantized versions for lower-VRAM setups — the checkpoint choice matters as much as the graph design for whether a given workflow is practical on your hardware.

See our focused profile: ComfyUI-LTXVideo: Lightricks’ Extra Nodes for LTX-2 Video.

Wan workflows

Wan 2.1 and later Wan releases support multiple video tasks, and Wan 2.1 was integrated into ComfyUI after its public release. This makes Wan a useful family for studying how text-to-video, image-to-video and more specialized video tasks fit into node-based pipelines. For a deeper look at the full Wan lineup, licensing and quality benchmarks, see our dedicated Wan AI video guide.

HunyuanVideo workflows

HunyuanVideo-1.5 provides another open video family with ComfyUI and Diffusers integration. It’s an 8.3-billion-parameter DiT paired with a 3D causal VAE encoder, with a stated minimum of 14GB of GPU memory when model offloading is enabled — disabling offloading on higher-end GPUs trades that memory back for inference speed. Its standard output is 121 frames (about 5 seconds at 24fps) at 480p or 720p, with an optional super-resolution pass up to 1080p.

Tencent’s own published figures claim a 1.87x end-to-end speedup for 720p 10-second videos versus a FlashAttention-3 baseline, and a step-distilled variant (8-12 steps instead of the full schedule) that reaches roughly 75 seconds per clip on an RTX 4090 — about a 75% speedup over the non-distilled model on the same card. Those are the model authors’ own numbers, not independent verification, so they describe HunyuanVideo’s ceiling under its own test conditions rather than a guarantee across every workflow. Comparing Hunyuan, Wan and LTX inside a common ComfyUI environment is more useful than comparing showcase clips alone, precisely because the graph structure lets you hold everything but the model constant.

How to debug a large video graph

1. start from an official or known-good workflow;
2. verify the exact model files and versions;
3. run at conservative resolution/frame count;
4. disable optional upscaling and post-processing;
5. confirm the base generation works;
6. re-enable controls one stage at a time;
7. save working checkpoints of the graph.

Build reusable subgraphs

For production, separate reusable tasks into logical modules: model loading, conditioning, generation, upscaling and export. This makes it easier to swap one video model for another without rebuilding the full pipeline — and, per the tables above, makes it much easier to swap in a lower-VRAM precision or a smaller checkpoint without touching the rest of the graph when a model upgrade suddenly doesn’t fit anymore.

Related guides

• ComfyUI guide
• Install ComfyUI
• Open Source AI Video Generators
• Wan AI Video Models
• HunyuanVideo-1.5
• VideoCrafter
• Local AI hardware guide

About the figures in this article

The Signal did not run the GPU benchmarks cited above. The per-GPU Wan 2.1 timing table and the VRAM-by-precision table come from independent third-party testing, with the stated resolution, frame count and step count noted alongside each figure. The LTX-2 format-optimization numbers and the HunyuanVideo speedup figures are the respective model vendors’ own published claims (NVIDIA and Tencent) rather than independent benchmarks — we’ve labeled them as such above. All of these numbers move as software stacks, drivers and model versions change; treat them as an ordering of hardware and technique tiers rather than a promise of a specific runtime on your machine.

Sources

• ComfyUI — LTX-2 workflow documentation
• Wan 2.1 official repository
• HunyuanVideo-1.5 official repository
• InstaSD — Wan 2.1 GPU benchmark testing
• WillItRunAI — Wan 2.1/2.2 VRAM requirements by variant
• NVIDIA — RTX AI Garage: LTX-2 and ComfyUI performance