Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories.

By Hou, Jiang, Quan +4Oct 15, 2026

Score███░░░░░░░3.3

VerdictCompetent work. Briefs at most.

Read the original PDF Code

Abstract

Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.

Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang

The editor's rubric

Heuristic review

Dimension	Level	Weight	What that level means
Leverage	███░░ 3	24%	A method or resource many groups across the field will adopt within a year.
Magnitude	██░░░ 2	18%	Solid incremental gain on a meaningful problem.
Evidence	███░░ 3	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	██░░░ 2	16%	A new combination of known ideas.
Trajectory	██░░░ 2	18%	Some room to improve with obvious engineering.
Stakes	██░░░ 2	10%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); verification (ablations); stakes (general AI).

How the score was computed

rank-2026-10-07

Score███░░░░░░░3.3

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Merit: 3.7 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 3.9 / 10; Shrunk toward the desk prior by editor confidence (36%).
Attention: 0%; Citations, upvotes, points, mentions.
Freshness: 81%; Half-life decay since publication.

Hugging Face upvotes181 (reference 25, via hf-daily, Oct 16, 2026, 13:29 UTC)
GitHub stars2 (reference 250, via hf-daily, Oct 16, 2026, 13:29 UTC)

The record

Reviewed by heuristic-v2 on Oct 16, 2026, 13:29 UTC. Paper type: method.
Categories: cs.LG, cs.CL
BRIEF, No.1 in the Artificial Intelligence edition of October 16, 2026.