AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt.

By Nitarach, Ngampornsukswadi, Taveekitworachai +3Oct 9, 2026

Score████░░░░░░4.3

Key numbers

271x and 611
84.5% of correct and 56.2%

Caveats

Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the original PDF Code

Abstract

Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval and https://github.com/nat-nischw/AMLworld-Compact.

Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul

The editor's rubric

Heuristic review

Dimension	Level	Weight	What that level means
Leverage	████░ 4	24%	A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
Magnitude	███░░ 3	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	███░░ 3	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	██░░░ 2	16%	A new combination of known ideas.
Trajectory	██░░░ 2	18%	Some room to improve with obvious engineering.
Stakes	███░░ 3	10%	Meaningful benefit to many people within a few years.

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); breadth (zero/few-shot); gains (x-fold); verification (held-out test, code released); stakes (global scale, general AI).

How the score was computed

rank-2026-10-07

Score████░░░░░░4.3

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Merit: 5.8 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.8 / 10; Shrunk toward the desk prior by editor confidence (42%).
Attention: 32%; Citations, upvotes, points, mentions.
Freshness: 38%; Half-life decay since publication.

Citations0 (reference 15, via semantic-scholar, Oct 12, 2026, 13:49 UTC)
Influential citations0 (reference 3, via semantic-scholar, Oct 12, 2026, 13:49 UTC)
Hugging Face upvotes11 (reference 25, via hf-daily, Oct 13, 2026, 01:15 UTC)
GitHub stars0 (reference 250, via hf-daily, Oct 13, 2026, 01:15 UTC)

The record

Reviewed by heuristic-v6 on Oct 12, 2026, 13:49 UTC. Paper type: method.
Categories: cs.CL
BRIEF, No.10 in the Artificial Intelligence edition of October 13, 2026.