Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations

By TannenbaumOct 18, 2026

Score████░░░░░░3.7

Key numbers

95% CI
21.6% relative to the frontier-model
11.26% versus 11.20%

Caveats

Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the original PDF

Abstract

We test whether Javo, a sales agent fine-tuned on 4.6 million consented, de-identified commercial conversations, sells better than a frontier general-purpose model and better than human sales associates. The evaluation has four parts. Study 1 scores the agent on two public sales benchmarks, SalesLLM and SmartSalesBench, against 10 frontier and open-weight models. Study 2 is a randomized controlled trial on 100 live websites in 14 industries (e-commerce, travel and hospitality, and high-consideration services), run from 1 May to 30 September 2026, in which 14.8 million visitors were randomized to no agent, Javo, or a frontier-model agent with an identical interface, catalog tools and guardrails. Study 3 isolates the effect of the model by comparing those two agent arms. Study 4 compares Javo with each site's own human associates on 20 sites with staffed live chat. Relative to no agent, Javo changed revenue per visitor by +10.0% (95% CI +5.1 to +14.9). Among engaged visitors, it changed conversion by +21.6% relative to the frontier-model agent (95% CI +16.9 to +26.3). Against human associates, its engaged conversion was 11.26% versus 11.20% (non-inferior, margin -7%, one-sided p = 0.004; no significant superiority), at 1.8% of the cost per conversation. Return rates, unsupported product claims and complaint rates met the prespecified safety guardrails.

Benjamin Tannenbaum

The editor's rubric

Heuristic review

Leverage3 / 5: A method or resource many groups across the field will adopt within a year.; 24% of this desk’s rubric
Magnitude3 / 5: Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.; 18% of this desk’s rubric
Evidence4 / 5: Strong: large scale, preregistered, independently replicated, or a well-powered randomized trial.; 14% of this desk’s rubric
Novelty2 / 5: A new combination of known ideas.; 16% of this desk’s rubric
Trajectory3 / 5: A clear path to scale.; 18% of this desk’s rubric
Stakes3 / 5: Meaningful benefit to many people within a few years.; 10% of this desk’s rubric

Dimension	Level	Weight	What that level means
Leverage	███░░ 3	24%	A method or resource many groups across the field will adopt within a year.
Magnitude	███░░ 3	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	████░ 4	14%	Strong: large scale, preregistered, independently replicated, or a well-powered randomized trial.
Novelty	██░░░ 2	16%	A new combination of known ideas.
Trajectory	███░░ 3	18%	A clear path to scale.
Stakes	███░░ 3	10%	Meaningful benefit to many people within a few years.

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: breadth (general-purpose, many tasks); gains (versus baseline); design (randomized, multicenter); verification (confidence interval, p-value); stakes (global scale, general AI).

How the score was computed

rank-2026-10-18

Score████░░░░░░3.7

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Desks place stronger verdicts first, then use this score to order work within the same verdict.

Merit: 6.0 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.9 / 10; Shrunk toward the desk prior by editor confidence (48%).
Attention: 0%; Citations, upvotes, points, mentions.
Freshness: 51%; Half-life decay since publication.

Citations0 (reference 15, via semantic-scholar, Oct 21, 2026, 02:01 UTC)
Influential citations0 (reference 3, via semantic-scholar, Oct 21, 2026, 02:01 UTC)

The record

Reviewed by heuristic-v7 on Oct 20, 2026, 13:49 UTC. Paper type: clinical.
Categories: cs.IR, cs.CY
TOP, No.4 in the Medicine edition of October 21, 2026.