FrontierCS 2: A Benchmark for Computer Science R&D

Can LLM agents make contributions to frontier computer science R&D?

By Mang, Chai, He +58Oct 21, 2026

Score████░░░░░░4.0

Key numbers

58% of its runs

Caveats

Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the original PDF Code

Abstract

Can LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution. The deleted code becomes the human reference that sets the bar. Earlier open-ended benchmarks give each agent a problem statement and grade one program after the run. FrontierCS 2 places the agent inside a research environment. The agent inherits a working repository and may submit a patch at any point. For each patch, a judge returns a development-suite score but keeps the evaluator, the held-out final suite and the reference private, and grades the final submission on that suite against the reference. The tasks span 13 domains, from machine-learning systems and high-performance computing to databases, security and robotics. Beyond these 48 tasks, FrontierCS 2 keeps a private set of more than 100 tasks, to be announced in the future. We evaluate 12 agents. The best one is Claude Opus 5.5, passing 58% of its runs. No run passes on 16 tasks. A higher score is not the same as a pass: most runs that beat the reference on the combined final score still fall short on some metric or workload. On a shared capability scale for agents and tasks, the median task is roughly as difficult as what the strongest agent can handle.

Qiuyang Mang, Wenhao Chai, Runyuan He, Dongruixuan Li, Zihan Zheng, Yujun He, Xuanyi Zhou, Hao Wang, Siddhant Ray, Bo Peng, Ruyi Ji, Shuo Yang, Haocheng Xi, Boru Chen, Xinyang David Han, Melissa Pan, Mayank Mishra, Boxi Yu, Sijia Liu, Kerem Akillioglu, Yi Pan, Zirui Wang, Mert Cemri, Diogo Antunes, Yichuan Wang, Aiwei Ivy Zhang, Zhenyang Chen, Qizheng Zhang, Kaiyuan Liu, Wenyuan Huang, et al.

The editor's rubric

Heuristic review

Leverage4 / 5: A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).; 24% of this desk’s rubric
Magnitude3 / 5: Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.; 18% of this desk’s rubric
Evidence3 / 5: Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.; 14% of this desk’s rubric
Novelty2 / 5: A new combination of known ideas.; 16% of this desk’s rubric
Trajectory2 / 5: Some room to improve with obvious engineering.; 18% of this desk’s rubric
Stakes2 / 5: Benefits a professional community (practitioners, clinicians, engineers).; 10% of this desk’s rubric

Dimension	Level	Weight	What that level means
Leverage	████░ 4	24%	A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
Magnitude	███░░ 3	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	███░░ 3	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	██░░░ 2	16%	A new combination of known ideas.
Trajectory	██░░░ 2	18%	Some room to improve with obvious engineering.
Stakes	██░░░ 2	10%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); breadth (many tasks); gains (outperforms); verification (multiple benchmarks, held-out test, code released); scale (scalable); stakes (general AI). Red flags: derivative (comparative study).

How the score was computed

rank-2026-10-18

Score████░░░░░░4.0

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Desks place stronger verdicts first, then use this score to order work within the same verdict.

Merit: 5.6 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.7 / 10; Shrunk toward the desk prior by editor confidence (44%).
Attention: 9%; Citations, upvotes, points, mentions.
Freshness: 67%; Half-life decay since publication.

Citations0 (reference 15, via semantic-scholar, Oct 22, 2026, 13:43 UTC)
Influential citations0 (reference 3, via semantic-scholar, Oct 22, 2026, 13:43 UTC)
Hugging Face upvotes1 (reference 25, via hf-daily, Oct 23, 2026, 01:47 UTC)
GitHub stars27 (reference 250, via hf-daily, Oct 23, 2026, 01:47 UTC)

The record

Reviewed by heuristic-v7 on Oct 22, 2026, 13:43 UTC. Paper type: method.
Categories: cs.LG
Not yet placed in an edition.