AIarXiv
Ranked by formulaVerdict: NotableThe Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention
Setting.
Key numbers
- 3.4× between nf4 training and
Caveats
- Preprint; not yet peer reviewed.
VerdictWorth a reader's time today.
Abstract
Setting. This is an empirical characterization study; we propose no new method and run no comparisons against alternative objectives. We fine-tune Qwen3.5-4B with sparse LoRA and length-normalized DPO plus an SFT retention term (the RPO objective) on structured financial-filing tasks, and track downstream capability as the implicit reward margin grows to ~ 7 $β$-scaled units (nf4 training; ~ 3 under fp16 measurement on the unquantized base). Question. Does margin growth degrade capability uniformly across tasks, or selectively? Findings. In a pre-registered study (8 seeds, 6 unseen at freeze) with artifact-aware scoring and TOST equivalence testing, the alignment tax is graded and task-selective: extraction is preserved (Δ= -0.55 pts, equivalent at ±5 pts), json loses little (Δ= -3.52 pts, equivalence fails on a clustered item-level interval), and router degrades severely (Δ= -30 pts). Router's loss concentrates in a margin 1-3 damage window containing a transient format-break spike. The margin is accounted for almost entirely (≥ 96%) by suppression of rejected completions, with chosen reward approximately flat, consistent with the retention term anchoring it; we do not attribute the behavioral damage to any single gradient channel. Gold-logp deflation is task-selective (extraction -0.15, json -1.04, router -1.47 nats/token under fp16), with gold selectively protected relative to incorrect controls on extraction (8/8 seeds). QLoRA reward margins differ up to 3.4× between nf4 training and unquantized fp16 measurement, with an axis-dependent gap. Scope. Length-normalized rewards with SFT retention, sparse LoRA on 8/32 layers, one 4B model, easily-separable synthetic preferences, three structured tasks. A no-retention control and a measured format-alignment metric are left to future work.
The editor's rubric
- Leverage4 / 5
- A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
- Magnitude3 / 5
- Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
- Evidence4 / 5
- Strong: large scale, preregistered, independently replicated, or a well-powered randomized trial.
- Novelty2 / 5
- A new combination of known ideas.
- Trajectory3 / 5
- A clear path to scale.
- Stakes2 / 5
- Benefits a professional community (practitioners, clinicians, engineers).
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); breadth (many tasks); gains (x-fold); design (registered); verification (multiple benchmarks, held-out test, code released); scale (scalable).
How the score was computed
Desks place stronger verdicts first, then use this score to order work within the same verdict.
- Merit
- 6.2 / 10
- Adjusted merit
- 5.0 / 10
- Attention
- 0%
- Freshness
- 57%