OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation?

By Wang, Liu, Meng +16Oct 16, 2026

Score█████░░░░░5.2

Key numbers

88.39% strict feasibility and a
24% to 45% across models

Caveats

Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the original PDF Code

Abstract

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.

Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen, Gaoyang Dai, Xing He, Aiqian Li, Fengchi Yuan, Aochuan Chen, Chenyi Zi, Yuhan Li, Hongchuan Zeng, Yimo Yan, Nuo Chen, Jie Fu, Zuo-Jun Max Shen, Yong-Hong KUO, Jia Li

The editor's rubric

Heuristic review

Leverage3 / 5: A method or resource many groups across the field will adopt within a year.; 24% of this desk’s rubric
Magnitude3 / 5: Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.; 18% of this desk’s rubric
Evidence3 / 5: Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.; 14% of this desk’s rubric
Novelty2 / 5: A new combination of known ideas.; 16% of this desk’s rubric
Trajectory2 / 5: Some room to improve with obvious engineering.; 18% of this desk’s rubric
Stakes3 / 5: Meaningful benefit to many people within a few years.; 10% of this desk’s rubric

Dimension	Level	Weight	What that level means
Leverage	███░░ 3	24%	A method or resource many groups across the field will adopt within a year.
Magnitude	███░░ 3	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	███░░ 3	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	██░░░ 2	16%	A new combination of known ideas.
Trajectory	██░░░ 2	18%	Some room to improve with obvious engineering.
Stakes	███░░ 3	10%	Meaningful benefit to many people within a few years.

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); gains (outperforms); verification (experimental validation, independent replication, code released); stakes (global scale, general AI). Red flags: derivative (comparative study); weak evidence (preliminary).

How the score was computed

rank-2026-10-18

Score█████░░░░░5.2

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Desks place stronger verdicts first, then use this score to order work within the same verdict.

Merit: 5.3 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.5 / 10; Shrunk toward the desk prior by editor confidence (42%).
Attention: 76%; Citations, upvotes, points, mentions.
Freshness: 35%; Half-life decay since publication.

Hugging Face upvotes46 (reference 25, via hf-daily, Oct 20, 2026, 13:49 UTC)
GitHub stars2 (reference 250, via hf-daily, Oct 20, 2026, 13:49 UTC)

The record

Reviewed by heuristic-v7 on Oct 20, 2026, 01:36 UTC. Paper type: method.
Categories: cs.CL, cs.AI
TOP, No.4 in the Artificial Intelligence edition of October 20, 2026.