SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?

Omni models answer questions about videos by jointly processing sampled frames and audio in one turn, with progress driven largely by scaling model size and omni training data.

By Wu, Gou, Mu +6Nov 4, 2026

Score████░░░░░░4.0

Caveats

Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the original PDF Code

Abstract

Omni models answer questions about videos by jointly processing sampled frames and audio in one turn, with progress driven largely by scaling model size and omni training data. We ask whether a model needs to hear the audio directly at all, and present SleuthVLM, an agent harness built on a VLM that never hears the audio directly: speech will be transcribed into text, and the VLM will sleuth the video on its own without any predefined workflow. On four omni benchmarks with videos up to 90 minutes, SleuthVLM achieves the best average accuracy: 75.0%, 2.6 points above Gemini 3.8 Flash, and it also tops the Video-MME-v2 leaderboard with 77.8%; with an open-source 27B VLM, it beats all open-source omni models. We draw three findings: SleuthVLM beats a model that glances; hearing is not the bottleneck, agentic ability is; omni understanding needs better agentic ability, not just more omni data for modality alignment. So we provide a simple but effective starting point for doing omni video understanding. Omni understanding needs better agents, not just bigger models that hear.

Peiran Wu, Fangfei Gou, Zhaoxi Mu, Changhao Zhang, Jian Liu, Weiqiang Wang, Yuntao Wang, Yuanchun Shi, Junxiao Shen

The editor's rubric

Heuristic review

Leverage3 / 5: A method or resource many groups across the field will adopt within a year.; 24% of this desk’s rubric
Magnitude3 / 5: Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.; 18% of this desk’s rubric
Evidence3 / 5: Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.; 14% of this desk’s rubric
Novelty2 / 5: A new combination of known ideas.; 16% of this desk’s rubric
Trajectory3 / 5: A clear path to scale.; 18% of this desk’s rubric
Stakes2 / 5: Benefits a professional community (practitioners, clinicians, engineers).; 10% of this desk’s rubric

Dimension	Level	Weight	What that level means
Leverage	███░░3/5	24%	A method or resource many groups across the field will adopt within a year.
Magnitude	███░░3/5	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	███░░3/5	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	██░░░2/5	16%	A new combination of known ideas.
Trajectory	███░░3/5	18%	A clear path to scale.
Stakes	██░░░2/5	10%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: breadth (many tasks); gains (outperforms); novelty (alternative to status quo); verification (multiple benchmarks, code released); scale (scalable, larger models); stakes (general AI).

How the score was computed

rank-2026-10-18

Score████░░░░░░4.0

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Desks place stronger verdicts first, then use this score to order work within the same verdict.

Merit: 5.5 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.7 / 10; Shrunk toward the desk prior by editor confidence (44%).
Attention: 19%; Citations, upvotes, points, mentions.
Freshness: 53%; Half-life decay since publication.

Hugging Face upvotes6 (reference 25, via hf-daily, Nov 6, 2026, 13:40 UTC)
GitHub stars1 (reference 250, via hf-daily, Nov 6, 2026, 13:40 UTC)

The record

Reviewed by heuristic-v10 on Nov 5, 2026, 13:49 UTC. Paper type: empirical.
Categories: cs.CV
BRIEF, No.8 in the Artificial Intelligence edition of November 6, 2026.
BRIEF, No.2 in the Artificial Intelligence edition of November 5, 2026.