AIHugging Face
Heuristic editor, no API keyVerdict: NotableMore Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
Direct-decision models turn text into low-latency structured labels and scores, making them attractive for classification and automatic evaluation.
Key numbers
- 38.8 % of all predictions and
- 51.3 % of errors to Neutral
- 74.95 % accuracy
VerdictWorth a reader's time today.
Abstract
Direct-decision models turn text into low-latency structured labels and scores, making them attractive for classification and automatic evaluation. Yet reliability requires more than accuracy: a model must also use the ordinal decision scale supplied by the user faithfully. We analyze JEV~1.13 and three open KEV models. Our investigation begins with ANLI, where JEV assigns 38.8% of all predictions and 51.3% of errors to Neutral despite 74.95% accuracy, nearly balanced gold labels, and balanced candidate positions. Across 36 ordinal datasets, final decisions use only 67--76% of the effective gold support, versus 87--102% on four nominal tasks. Randomizing candidate order weakens but does not remove this compression. Holding items and source scores fixed while balancing gold support and positions, we refine scales from K=2 to 14; utilization falls for every model and reaches 26--75% at K=14, although candidate probabilities remain broad for most models. Targeted BA-LoRA post-training raises gold-relative utilization from roughly 47% to 86% on eight supervised scales at both KEV sizes, showing that the compression is learned and modifiable rather than an immutable architectural limit. We call this ordinal scale-utilization bias: decision-stage candidate-space compression distinct from accuracy, gold imbalance, fixed position, and candidate count alone. The code and data are available at https://github.com/Glax147/jev_ordinal_scale_bia
The editor's rubric
| Dimension | Level | Weight | What that level means |
|---|---|---|---|
| Leverage | ████░ 4 | 24% | A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing). |
| Magnitude | ███░░ 3 | 18% | Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem. |
| Evidence | ███░░ 3 | 14% | Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data. |
| Novelty | ███░░ 3 | 16% | A genuinely new approach to an open problem. |
| Trajectory | ███░░ 3 | 18% | A clear path to scale. |
| Stakes | ██░░░ 2 | 10% | Benefits a professional community (practitioners, clinicians, engineers). |
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (named contribution); breadth (many tasks); gains (relative gain); novelty (alternative to status quo); verification (multiple benchmarks, code released); scale (scalable).
How the score was computed
- Merit
- 6.3 / 10
- Adjusted merit
- 5.0 / 10
- Attention
- 78%
- Freshness
- 56%
- Hugging Face upvotes50 (reference 25, via hf-daily, Oct 2, 2026, 02:05 UTC)
- GitHub stars3 (reference 250, via hf-daily, Oct 2, 2026, 02:05 UTC)