Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Caveats

• Preprint; not yet peer reviewed.

We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.

Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, Jian Hu, Haihan Gao, Heting Gao, Xin Li, Yinsong Liu, Deqian Jiang, Di Yin, Haoyu Cao, Xing Sun, Bing Liu, Yanqiu Qu, Yunfei Wu, Kun Yin, Zhixiang Wei, Yongxiang Hua, Qianyu Li, Xiaoxing Hu, Peiyuan Zhang, Lijiang Li, Dong Xu, Zhenchuan Yang, Yiliang Jiang

Leverage4 / 5

A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).

24% of this desk’s rubric

Magnitude3 / 5

Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.

18% of this desk’s rubric

Evidence3 / 5

Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.

14% of this desk’s rubric

Novelty2 / 5

A new combination of known ideas.

16% of this desk’s rubric

Trajectory3 / 5

A clear path to scale.

18% of this desk’s rubric

Stakes2 / 5

Benefits a professional community (practitioners, clinicians, engineers).

10% of this desk’s rubric

• Hugging Face upvotes10 (reference 25, via hf-daily, Oct 30, 2026, 13:37 UTC)

• Reviewed by heuristic-v10 on Oct 30, 2026, 13:36 UTC. Paper type: method.
• Categories: cs.CV
• TOP, No.6 in the Front page edition of October 30, 2026.
• TOP, No.7 in the Artificial Intelligence edition of October 30, 2026.