Evaluating Instrument-Centered AI Agents for Table-Top Science Experiments

By Cai, Cheng, Huxter +10Oct 30, 2026

Score████░░░░░░4.1

Key numbers

83% of measurement-related rubric criteria

Caveats

Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the original PDF Code

Abstract

Reliable evaluation of AI agents for table-top science experiments has remained elusive because live instruments are hard to reset and share at scale, while instrument statefulness and safety constraints complicate evaluation. We introduce instrMCP and q-devBench, an evaluation stack for instrument-centered AI agents in table-top experiments. instrMCP connects coding agents to scientific instruments through the Model Context Protocol, serving as the instrument-access harness. q-devBench reconstructs 37 real experimental tasks from condensed-matter physics as virtual QCoDeS instruments covering quantum electronic transport, scanning probe imaging, and calibration workflows. q-devBench exposes reproducible failure modes in laboratory-agent systems while preserving realistic instrument-control workflows. Across a range of commercial coding-agent configurations, current systems already show substantial operational competence, satisfying 70-83% of measurement-related rubric criteria. The main bottleneck is scientific analysis, where performance falls to 28-46%. In an analysis-only ablation in which agents receive clean reference datasets without running the virtual measurement session, analysis performance improves substantially, closing roughly two thirds of the difference between the measurement and analysis scores. This suggests that much of the deficit arises from the measurement-session context, including tool-use overhead, longer context, measurement-planning decisions, and possible acquisition errors, while a residual scientific-reasoning gap remains. Together, instrMCP and q-devBench provide a practical sandbox for benchmarking, diagnosing, and improving laboratory AI harnesses, and establish a route from controlled virtual evaluation to future live-hardware validation.

Jiaqi Cai, Siyu Cheng, William S. Huxter, Li-Qiao Xia, Shuwen Sun, Shuai Yuan, Yifan Zhao, Naiyuan Zhang, Weijie Li, Muxian Xu, Ilija Zeljkovic, Xiaodong Xu, Pablo Jarillo-Herrero

The editor's rubric

Heuristic review

Leverage3 / 5: A method or resource many groups across the field will adopt within a year.; 18% of this desk’s rubric
Magnitude2 / 5: Solid incremental gain on a meaningful problem.; 20% of this desk’s rubric
Evidence3 / 5: Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.; 22% of this desk’s rubric
Novelty3 / 5: A genuinely new approach to an open problem.; 22% of this desk’s rubric
Trajectory2 / 5: Some room to improve with obvious engineering.; 10% of this desk’s rubric
Stakes2 / 5: Benefits a professional community (practitioners, clinicians, engineers).; 8% of this desk’s rubric

Dimension	Level	Weight	What that level means
Leverage	███░░3/5	18%	A method or resource many groups across the field will adopt within a year.
Magnitude	██░░░2/5	20%	Solid incremental gain on a meaningful problem.
Evidence	███░░3/5	22%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	███░░3/5	22%	A genuinely new approach to an open problem.
Trajectory	██░░░2/5	10%	Some room to improve with obvious engineering.
Stakes	██░░░2/5	8%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); novelty (open problem); verification (ablations, independent replication, code released); scale (scalable).

How the score was computed

rank-2026-10-18

Score████░░░░░░4.1

Score = 10 × (75% × adjusted merit / 10 + 15% × attention + 10% × freshness)

Desks place stronger verdicts first, then use this score to order work within the same verdict.

Merit: 5.2 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.5 / 10; Shrunk toward the desk prior by editor confidence (40%).
Attention: 0%; Citations, upvotes, points, mentions.
Freshness: 76%; Half-life decay since publication.

No attention signals recorded yet.

The record

Reviewed by heuristic-v10 on Nov 2, 2026, 07:29 UTC. Paper type: method.
Categories: cond-mat.mes-hall
BRIEF, No.1 in the Physics edition of November 2, 2026.