AIarXiv
Ranked by formulaVerdict: NotableYour Transformer Is Secretly Elastic
Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads.
Key numbers
- 1.80x higher throughput and 1.26
- 60% sparsity and outperforms one-shot
- 50% sparsity removes 427
Caveats
- Preprint; not yet peer reviewed.
VerdictWorth a reader's time today.
Abstract
Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e.g., S, B, L), forcing users to commit to a model based on hardware and latency constraints. This rigidity is often suboptimal under varying query loads. We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model's loss under soft pruning across many sparsity targets, all at once. Across 27 vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to 60% sparsity and outperforms one-shot baselines under deep pruning. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, without the need for weight correction or retraining. At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to 1.80× higher throughput and 1.26× lower single-query latency on an A100 GPU. Elastic models are produced in minutes, with construction time scaling sub-linearly with parameter count.
The editor's rubric
- Leverage4 / 5
- A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
- Magnitude4 / 5
- A qualitative jump: a capability or regime that did not exist before.
- Evidence2 / 5
- Limited: single setting, weak baselines, or an observational association presented as causal.
- Novelty3 / 5
- A genuinely new approach to an open problem.
- Trajectory3 / 5
- A clear path to scale.
- Stakes2 / 5
- Benefits a professional community (practitioners, clinicians, engineers).
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); breadth (many tasks); gains (x-fold, relative gain, outperforms); novelty (alternative to status quo); verification (code released); scale (scalable).
How the score was computed
Desks place stronger verdicts first, then use this score to order work within the same verdict.
- Merit
- 5.2 / 10
- Adjusted merit
- 4.5 / 10
- Attention
- 17%
- Freshness
- 30%
- Citations0 (reference 15, via semantic-scholar, Oct 17, 2026, 13:49 UTC)
- Influential citations0 (reference 3, via semantic-scholar, Oct 17, 2026, 13:49 UTC)
- Hugging Face upvotes5 (reference 25, via hf-daily, Oct 17, 2026, 13:49 UTC)
- GitHub stars5 (reference 250, via hf-daily, Oct 17, 2026, 13:49 UTC)