Can Coding Agents Sustain Continual Evolution without Forgetting?

Recently, harness self-evolution has emerged as an important paradigm for improving agent performance.

By Zhao, Zhang, LiNov 2, 2026

Score████░░░░░░4.2

Key numbers

94.0% after evolution
32.0% by the fifth round
66% relative decrease

Caveats

Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the original PDF Code

Abstract

Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. We find that existing approaches largely study agent evolution on a single task. In practice, coding agents may need to handle multiple task types over time. Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. Code is available at https://github.com/THU-Agent/continual-harness-evolution.

Shangziqi Zhao, Yitong Zhang, Jia Li

The editor's rubric

Heuristic review

Leverage4 / 5: A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).; 24% of this desk’s rubric
Magnitude3 / 5: Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.; 18% of this desk’s rubric
Evidence3 / 5: Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.; 14% of this desk’s rubric
Novelty3 / 5: A genuinely new approach to an open problem.; 16% of this desk’s rubric
Trajectory2 / 5: Some room to improve with obvious engineering.; 18% of this desk’s rubric
Stakes2 / 5: Benefits a professional community (practitioners, clinicians, engineers).; 10% of this desk’s rubric

Dimension	Level	Weight	What that level means
Leverage	████░4/5	24%	A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
Magnitude	███░░3/5	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	███░░3/5	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	███░░3/5	16%	A genuinely new approach to an open problem.
Trajectory	██░░░2/5	18%	Some room to improve with obvious engineering.
Stakes	██░░░2/5	10%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we report); breadth (many tasks); gains (relative gain); novelty (open problem, discovery); verification (multiple benchmarks, code released); stakes (general AI).

How the score was computed

rank-2026-10-18

Score████░░░░░░4.2

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Desks place stronger verdicts first, then use this score to order work within the same verdict.

Merit: 5.9 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.8 / 10; Shrunk toward the desk prior by editor confidence (44%).
Attention: 17%; Citations, upvotes, points, mentions.
Freshness: 66%; Half-life decay since publication.

Hugging Face upvotes5 (reference 25, via hf-daily, Nov 4, 2026, 02:05 UTC)
GitHub stars1 (reference 250, via hf-daily, Nov 4, 2026, 02:05 UTC)

The record

Reviewed by heuristic-v10 on Nov 3, 2026, 13:38 UTC. Paper type: method.
Categories: cs.SE, cs.AI
TOP, No.4 in the Artificial Intelligence edition of November 4, 2026.