AIarXiv
Ranked by formulaVerdict: NotableAutomating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?
Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine…
Key numbers
- 10.4% mean relative improvement and
- 10.4% to 20.1% and Codex
- 4.4% to 8.1%
Caveats
- Preprint; not yet peer reviewed.
VerdictWorth a reader's time today.
Abstract
Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce Agent4Sci, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains. Rather than ranking agents on a leaderboard, Agent4Sci asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short. Empirically, today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct. These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight. Trajectory analysis surfaces three recurring limitations in this setting: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. Lightweight plug-in skills for research ideation, experiment management, and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain-relevant interventions. Agent4Sci gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code and data are available on GitHub (https://github.com/maszhongming/agent4sci) and Hugging Face (https://huggingface.co/datasets/MingZhong/agent4sci).
The editor's rubric
- Leverage4 / 5
- A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
- Magnitude3 / 5
- Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
- Evidence3 / 5
- Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
- Novelty3 / 5
- A genuinely new approach to an open problem.
- Trajectory2 / 5
- Some room to improve with obvious engineering.
- Stakes2 / 5
- Benefits a professional community (practitioners, clinicians, engineers).
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); breadth (many tasks); gains (relative gain); novelty (alternative to status quo); verification (multiple benchmarks, code released); stakes (general AI).
How the score was computed
Desks place stronger verdicts first, then use this score to order work within the same verdict.
- Merit
- 5.9 / 10
- Adjusted merit
- 4.8 / 10
- Attention
- 32%
- Freshness
- 60%
- Hugging Face upvotes11 (reference 25, via hf-daily, Oct 22, 2026, 13:43 UTC)
- GitHub stars1 (reference 250, via hf-daily, Oct 22, 2026, 13:43 UTC)