FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders

Representation autoencoders (RAEs) reuse features from a pretrained visual encoder as reconstruction and diffusion latents, integrating strong visual representations into image generation.

By Du, Xie, Ye +13arXivSep 25, 2026

Score█████░░░░░5.3

Key numbers

27% with an unchanged RAEv2
29% on DiT-Base

VerdictCompetent work. Briefs at most.

Read the original PDF Code

Abstract

Representation autoencoders (RAEs) reuse features from a pretrained visual encoder as reconstruction and diffusion latents, integrating strong visual representations into image generation. However, RAEs still need to decide which encoder layers form the shared latent space for the generator and pixel decoder. This choice involves a trade-off. Shallower layers tend to preserve fine pixel details better, while deeper layers tend to yield better generation metrics. A fixed heuristic layer fusion therefore couples two stages that benefit from different information. We introduce FuseReg, which replaces heuristic feature selection with training over random subsets of encoder layers. We theoretically analyze the underlying mechanism: subset sampling explicitly penalizes sensitivity to cross-layer disagreement. On ImageNet-256 with DINOv3-L, a single FuseReg decoder reconstructs from full, sparse, and single-layer fusions without retraining, achieving higher PSNR than decoders specialized to fixed fusions. This flexibility also benefits generation: decoder replacement alone reduces unguided gFID by 27% with an unchanged RAEv2 DiT-XL generator. The same regularization principle extends to diffusion training, with joint regularization of both stages reducing unguided gFID by 29% on DiT-Base. These results show that training downstream models for layer-fusion robustness narrows the reconstruction-generation gap without modifying the pretrained encoder.

Hongyang Du, Yunfei Xie, Junjie Ye, Jiawei Yang, Xiaoyan Cong, Haodong Zhang, Yongchao Huang, Haiyu Wu, Zongxia Li, Shihang Gui, Dawei Liu, Runhao Li, Jingcheng Ni, Chen Wei, Randall Balestriero, Yue Wang

The editor's rubric

Heuristic review

Dimension	Level	Weight	What that level means
Leverage	███░░ 3	24%	A method or resource many groups across the field will adopt within a year.
Magnitude	███░░ 3	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	███░░ 3	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	██░░░ 2	16%	A new combination of known ideas.
Trajectory	██░░░ 2	18%	Some room to improve with obvious engineering.
Stakes	██░░░ 2	10%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); gains (relative gain).

How the score was computed

rank-2026-09-29

Score█████░░░░░5.3

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Merit: 5.1 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.4 / 10; Shrunk toward the desk prior by editor confidence (34%).
Attention: 90%; Citations, upvotes, points, mentions.
Freshness: 25%; Half-life decay since publication.

Citations0 (reference 15, via semantic-scholar, Sep 29, 2026, 23:53 UTC)
Influential citations0 (reference 3, via semantic-scholar, Sep 29, 2026, 23:53 UTC)
Hugging Face upvotes127 (reference 25, via hf-daily, Sep 30, 2026, 11:05 UTC)
GitHub stars15 (reference 250, via hf-daily, Sep 30, 2026, 11:05 UTC)

The record

Reviewed by heuristic-v2 on Sep 29, 2026, 23:53 UTC. Paper type: method.
BRIEF, No.1 in the Artificial Intelligence edition of September 29, 2026.