On the Limits of Scale: Inverse Scaling in Vision-Language Models

Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements.

By Saxena, Suglia, Rei +1Oct 19, 2026

Score████░░░░░░4.0

Caveats

Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the original PDF Code

Abstract

Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family. We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model-question evaluations in total. Using per-question pairwise analysis across all model pairs in each family, we identify 375 subtask-model-pair combinations exhibiting inverse scaling, spanning 4,342 unique queries where a smaller model answers correctly but a larger model fails, and we find that these patterns replicate across independently developed architectures. We identify 13 subtask-dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific. The most affected domains in our study include multi-image spatial reasoning, visual perception, and university-level STEM knowledge, with accuracy gaps exceeding 20 percentage points in several cases, while diagram understanding appears comparatively resistant. These findings challenge the assumption that increasing model size uniformly improves multimodal capabilities and suggest that model selection should be task-aware.

Rohit Saxena, Alessandro Suglia, Marek Rei, Pasquale Minervini

The editor's rubric

Heuristic review

Leverage3 / 5: A method or resource many groups across the field will adopt within a year.; 24% of this desk’s rubric
Magnitude3 / 5: Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.; 18% of this desk’s rubric
Evidence3 / 5: Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.; 14% of this desk’s rubric
Novelty3 / 5: A genuinely new approach to an open problem.; 16% of this desk’s rubric
Trajectory3 / 5: A clear path to scale.; 18% of this desk’s rubric
Stakes2 / 5: Benefits a professional community (practitioners, clinicians, engineers).; 10% of this desk’s rubric

Dimension	Level	Weight	What that level means
Leverage	███░░ 3	24%	A method or resource many groups across the field will adopt within a year.
Magnitude	███░░ 3	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	███░░ 3	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	███░░ 3	16%	A genuinely new approach to an open problem.
Trajectory	███░░ 3	18%	A clear path to scale.
Stakes	██░░░ 2	10%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose, we report); breadth (many tasks); gains (outperforms); novelty (contrary to expectation, alternative to status quo, discovery); verification (error bars, multiple benchmarks, independent replication); scale (scalable, larger models); stakes (general AI).

How the score was computed

rank-2026-10-18

Score████░░░░░░4.0

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Desks place stronger verdicts first, then use this score to order work within the same verdict.

Merit: 5.8 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.9 / 10; Shrunk toward the desk prior by editor confidence (50%).
Attention: 7%; Citations, upvotes, points, mentions.
Freshness: 64%; Half-life decay since publication.

Citations0 (reference 15, via semantic-scholar, Oct 21, 2026, 13:37 UTC)
Influential citations0 (reference 3, via semantic-scholar, Oct 21, 2026, 13:37 UTC)
Hugging Face upvotes2 (reference 25, via hf-daily, Oct 21, 2026, 13:37 UTC)
GitHub stars0 (reference 250, via hf-daily, Oct 21, 2026, 13:37 UTC)

The record

Reviewed by heuristic-v7 on Oct 21, 2026, 05:16 UTC. Paper type: method.
Categories: cs.CV, cs.AI
BRIEF, No.6 in the Artificial Intelligence edition of October 21, 2026.