BiologybioRxiv
Ranked by formulaVerdict: NotableBenchMHC: a reproducible framework for peptide-MHC presentation prediction
Predicting which peptides an MHC molecule presents is the first step in neoantigen and epitope discovery pipelines.
Key numbers
- 350 times faster than the original
- 99% of hits are found
- 12% of their decoys
Caveats
- Preprint; not yet peer reviewed.
VerdictWorth a reader's time today.
Abstract
Predicting which peptides an MHC molecule presents is the first step in neoantigen and epitope discovery pipelines. Most published predictors are released for inference only, and each is trained on a different dataset, so the effect of a new method cannot be separated from the effect of its training data. We present BenchMHC, an open framework that reproduces, retrains and evaluates pMHC presentation models on public data only, and that records every implementation decision made in a reproduction. Our NetMHCpan-4.1 reproduction reaches 0.794 per-allele Mean Top-K against 0.790 for the official webserver, and 0.0025 Median FRANK against 0.0022. It scores 280,000 peptide/allele pairs in 27 s, about 350 times faster than the original binary. We then examine the training data. In the public NetMHCpan-4.1 eluted-ligand sets, 86 to 99% of hits are found in the human proteome, against about 12% of their decoys, so a model can separate hits from decoys using this difference rather than features related to MHC binding. Redrawing the decoys within the source protein of each hit and matching their cysteine status, with no other change, raises per-allele Mean Top-K on the regenerated v2.0.0 evaluation set from 0.734 to 0.755, and increases Median FRANK on CD8 epitopes from 0.0025 to 0.0053. The same change improves one evaluation set and degrades the other. Code (Apache-2.0) and model checkpoints (CC BY-NC-4.0): https://github.com/instadeepai/BenchMHC-public.
The editor's rubric
- Leverage3 / 5
- A method or resource many groups across the field will adopt within a year.
- Magnitude3 / 5
- Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
- Evidence3 / 5
- Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
- Novelty3 / 5
- A genuinely new approach to an open problem.
- Trajectory2 / 5
- Some room to improve with obvious engineering.
- Stakes2 / 5
- Benefits a professional community (practitioners, clinicians, engineers).
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose, new method); gains (x-fold); novelty (alternative to status quo, discovery); verification (independent replication, code released); scale (efficient).
How the score was computed
Desks place stronger verdicts first, then use this score to order work within the same verdict.
- Merit
- 5.6 / 10
- Adjusted merit
- 4.7 / 10
- Attention
- 0%
- Freshness
- 88%