AIarXiv
Ranked by formulaVerdict: NotableHow Far Can Foundation Models Be Distilled for Efficient Jet Tagging?
Jet tagging is one of the most important applications of machine learning at the Large Hadron Collider.
Caveats
- Preprint; not yet peer reviewed.
VerdictWorth a reader's time today.
Abstract
Jet tagging is one of the most important applications of machine learning at the Large Hadron Collider. Recent models build in more physics knowledge or rely on large-scale pretraining, but the experiments place strict limits on their inference cost. Knowledge distillation can transfer their performance to small students that are cheap to run. We ask which properties of a teacher help the student, how small a student can get and how close it can come to the state of the art. On a jet tagging benchmark, we distil six teachers that differ in architecture, symmetry and pretraining, including two foundation models, into five small students. Every teacher improves every student, and the gain depends mainly on the strength of the teacher. We then shrink our two strongest distilled students, Transformer and L-GATr-slim, in width and in depth. With 20 thousand parameters, a distilled L-GATr-slim outperforms every published jet tagger trained from scratch at a fraction of the inference cost of every teacher. Even with two thousand parameters, it matches ParT, a model with a thousand times as many parameters. We also study what the students learn from their teachers. Our benchmark of inference cost shows that students of the same size can differ by almost an order of magnitude in speed. We release the code and the weights of all trained models.
The editor's rubric
- Leverage3 / 5
- A method or resource many groups across the field will adopt within a year.
- Magnitude4 / 5
- A qualitative jump: a capability or regime that did not exist before.
- Evidence3 / 5
- Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
- Novelty1 / 5
- A minor twist on a known approach.
- Trajectory3 / 5
- A clear path to scale.
- Stakes2 / 5
- Benefits a professional community (practitioners, clinicians, engineers).
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: gains (orders of magnitude, state of the art, outperforms); verification (multiple benchmarks, code released); scale (scalable, orders of magnitude, efficient). Red flags: derivative (comparative study).
How the score was computed
Desks place stronger verdicts first, then use this score to order work within the same verdict.
- Merit
- 5.5 / 10
- Adjusted merit
- 4.7 / 10
- Attention
- 0%
- Freshness
- 82%