BiologybioRxiv
Ranked by formulaVerdict: NotableMutQA: A Cross-Validated Q&A Dataset for Genetic Mutations
We introduce MutQA, a dataset of 330,387 citation-grounded question-answer (QA) pairs linking genetic mutations to their functional consequences as described in the biomedical literature.
Key numbers
- 82.7% pass both directions of
- 86.4% of cross-validated QA pairs
- 95% CI 83.7-89.0
Caveats
- Preprint; not yet peer reviewed.
VerdictWorth a reader's time today.
Abstract
We introduce MutQA, a dataset of 330,387 citation-grounded question-answer (QA) pairs linking genetic mutations to their functional consequences as described in the biomedical literature. Each record pairs a protein-coding mutation with a natural-language question and an evidence-backed answer anchored to sentences in one of 27,442 open-access PubMed Central articles, spanning 6,620 genes and 125,593 gene-variant pairs. The dataset is constructed using CrossValQA, a framework that enforces information isolation between independent LLMs: one model generates QA pairs from an article, a second model answers the same question without access to the first model's response, and a third model judges whether each model's rationale supports the other's answer, so that retained QA pairs are reproducible from the source text rather than hallucinated. Of 401,406 audited QA pairs, 82.7% pass both directions of this cross-validation; those with a recoverable reference protein sequence form the released benchmark. In a 999-pair stratified audit, two expert annotators agree exactly on 86.4% of cross-validated QA pairs (95% CI 83.7-89.0; n = 638), a tier in which about 90% of ratings are "correct". Individual LLM judges agree with individual annotators at a comparable 87.0% on this tier but far less on QA pairs that fail cross-validation. Fine-tuning MutaPLM on MutQA nearly doubles its mean LLM-judge score for question-conditioned mutation-effect answering (1.45 to 2.82 on a 0-5 scale) and improves mutation-recovery accuracy from effect descriptions from 19.4% to 32.1%. Providing variant-matched MutQA retrieval context, without fine-tuning, improves strict answer accuracy across four LLM families by 20.0-28.6 percentage points, demonstrating the framework's ability to improve the reliability of LLM answers. We also apply CrossValQA to proteomics, providing a proof of concept for its use beyond mutation-specific QA. We release the dataset, construction framework, and evaluation code.
The editor's rubric
- Leverage3 / 5
- A method or resource many groups across the field will adopt within a year.
- Magnitude3 / 5
- Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
- Evidence3 / 5
- Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
- Novelty3 / 5
- A genuinely new approach to an open problem.
- Trajectory2 / 5
- Some room to improve with obvious engineering.
- Stakes2 / 5
- Benefits a professional community (practitioners, clinicians, engineers).
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); firsts (first); novelty (alternative to status quo); verification (confidence interval, experimental validation, independent replication); scale (scalable). Red flags: weak evidence (preliminary).
How the score was computed
Desks place stronger verdicts first, then use this score to order work within the same verdict.
- Merit
- 5.6 / 10
- Adjusted merit
- 4.7 / 10
- Attention
- 0%
- Freshness
- 88%