He Wants AI Benchmarking To Be a Science, Not a Dark Art

Moritz Hardt, author of a new book on benchmarks, believes these "standardized tests" for artificial intelligence are fundamentally broken — but not unfixable.

By PavlusOct 16, 2026

Score█████░░░░░4.9

VerdictCompetent work. Briefs at most.

Read at Quanta Magazine

Summary from the source

Moritz Hardt, author of a new book on benchmarks, believes these “standardized tests” for artificial intelligence are fundamentally broken — but not unfixable.

John Pavlus

The editor's rubric

Heuristic review

Dimension	Level	Weight	What that level means
Leverage	██░░░ 2	15%	Reusable within one subfield (a technique, dataset, or protocol a few groups will adopt).
Magnitude	██░░░ 2	20%	Solid incremental gain on a meaningful problem.
Evidence	██░░░ 2	20%	Limited: single setting, weak baselines, or an observational association presented as causal.
Novelty	██░░░ 2	10%	A new combination of known ideas.
Trajectory	██░░░ 2	10%	Some room to improve with obvious engineering.
Stakes	██░░░ 2	25%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: stakes (general AI).

How the score was computed

rank-2026-10-07

Score█████░░░░░4.9

Score = 10 × (50% × adjusted merit / 10 + 25% × attention + 25% × freshness)

Merit: 3.4 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 3.8 / 10; Shrunk toward the desk prior by editor confidence (30%).
Attention: 29%; Citations, upvotes, points, mentions.
Freshness: 91%; Half-life decay since publication.

No attention signals recorded yet.

The record

Reviewed by heuristic-v6 on Oct 16, 2026, 15:58 UTC. Paper type: news.
Categories: Computer Science
BRIEF, No.4 in the Front page edition of October 16, 2026.
TOP, No.3 in the News edition of October 16, 2026.