Cortex: Consolidating Context into Parametric Memory

Context can improve large language model performance, but these gains may be lost after context removal.

By Zhang, He, Liang +5Oct 14, 2026

Score█████░░░░░5.4

Key numbers

88.9% of the context-induced gain
99.8% of original non-target performance

Caveats

Preprint; not yet peer reviewed.

VerdictCompetent work. Briefs at most.

Read the original PDF

Abstract

Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.

Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang

The editor's rubric

Heuristic review

Dimension	Level	Weight	What that level means
Leverage	███░░ 3	24%	A method or resource many groups across the field will adopt within a year.
Magnitude	███░░ 3	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	██░░░ 2	14%	Limited: single setting, weak baselines, or an observational association presented as causal.
Novelty	██░░░ 2	16%	A new combination of known ideas.
Trajectory	██░░░ 2	18%	Some room to improve with obvious engineering.
Stakes	██░░░ 2	10%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); gains (relative gain); stakes (general AI).

How the score was computed

rank-2026-10-07

Score█████░░░░░5.4

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Merit: 4.0 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.0 / 10; Shrunk toward the desk prior by editor confidence (36%).
Attention: 80%; Citations, upvotes, points, mentions.
Freshness: 78%; Half-life decay since publication.

Hugging Face upvotes56 (reference 25, via hf-daily, Oct 15, 2026, 13:49 UTC)

The record

Reviewed by heuristic-v6 on Oct 15, 2026, 02:05 UTC. Paper type: method.
Categories: cs.CL, cs.AI, cs.CV, cs.LG, cs.MA
BRIEF, No.4 in the Artificial Intelligence edition of October 15, 2026.