Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Caveats

• Preprint; not yet peer reviewed.

Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.

Zhenghan Chen, Zekai Shao, Xin Lin, Leting Cheng, Siming Chen

Leverage4 / 5

A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).

24% of this desk’s rubric

Magnitude2 / 5

Solid incremental gain on a meaningful problem.

18% of this desk’s rubric

Evidence3 / 5

Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.

14% of this desk’s rubric

Novelty2 / 5

A new combination of known ideas.

16% of this desk’s rubric

Trajectory3 / 5

A clear path to scale.

18% of this desk’s rubric

Stakes2 / 5

Benefits a professional community (practitioners, clinicians, engineers).

10% of this desk’s rubric

• Citations0 (reference 15, via semantic-scholar, Oct 23, 2026, 13:24 UTC)
• Influential citations0 (reference 3, via semantic-scholar, Oct 23, 2026, 13:24 UTC)
• Hugging Face upvotes18 (reference 25, via hf-daily, Oct 24, 2026, 13:49 UTC)
• GitHub stars5 (reference 250, via hf-daily, Oct 24, 2026, 13:49 UTC)

• Reviewed by heuristic-v10 on Oct 25, 2026, 13:49 UTC. Paper type: method.
• Categories: cs.CV, cs.AI
• BRIEF, No.5 in the Artificial Intelligence edition of October 25, 2026.