AutoGUIWorld: Image Generators as Visual World Models for GUI Agent

GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows.

By Yang, Wu, Huang +18arXivOct 1, 2026

Score█████░░░░░5.1

Key numbers

33.0% to 40.8% and the
14.0% to 32.2%

VerdictCompetent work. Briefs at most.

Read the original PDF Code

Abstract

GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows. However, the diversity of available trajectories is constrained by the applications, interface states, and workflows accessible in the underlying environments. Expanding this coverage requires deploying increasingly diverse and complex software, with specialized applications imposing additional installation, configuration, and runtime costs. We introduce AutoGUIWorld, a data generation framework that combines the visual priors of image generators with the task knowledge of a planner to synthesize GUI interaction trajectories without deploying or running the corresponding software environments. AutoGUIWorld samples initial GUI scenes from structured specifications of operating-system context, visual appearance, and interface state, and generates tasks conditioned on those scenes. A planner then specifies atomic actions and their intended visual consequences, while an image generator iteratively edits the current screenshot to produce subsequent observations. Action grounding and transition-level quality filtering yield 79,266 spatially annotated step-level training samples across Ubuntu, Windows, macOS, and Chrome. Fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories improves the mean task score on OSWorld from 33.0% to 40.8% and the task success rate on ScienceBoard from 14.0% to 32.2%. These results show that generated trajectories improve GUI-agent performance on real desktop and scientific tasks.

Cheng Yang, Yifan Wu, Yutao Huang, Zhaohua Zhang, Beiduo Chen, Muxi Chen, Chenchen Zhao, Hexuan Deng, Haolin Yang, Geyuan Zhu, Sa Zhu, Jianhuan Zhuo, Qiuyong Xiao, Jianhao Ruan, Yiran Peng, Jiayi Zhang, Tian Ye, Xinlei Yu, Tianwen Jiang, Jihong Zhang, Yuyu Luo

The editor's rubric

Heuristic review

Dimension	Level	Weight	What that level means
Leverage	███░░ 3	24%	A method or resource many groups across the field will adopt within a year.
Magnitude	███░░ 3	18%	Large gain: roughly 2x, or a clear new state of the art on a hard, unsaturated problem.
Evidence	███░░ 3	14%	Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
Novelty	██░░░ 2	16%	A new combination of known ideas.
Trajectory	██░░░ 2	18%	Some room to improve with obvious engineering.
Stakes	██░░░ 2	10%	Benefits a professional community (practitioners, clinicians, engineers).

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); gains (relative gain); stakes (general AI).

How the score was computed

rank-2026-09-29

Score█████░░░░░5.1

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Merit: 5.1 / 10; Weighted rubric, evidence-gated.
Adjusted merit: 4.4 / 10; Shrunk toward the desk prior by editor confidence (36%).
Attention: 78%; Citations, upvotes, points, mentions.
Freshness: 28%; Half-life decay since publication.

Hugging Face upvotes49 (reference 25, via hf-daily, Oct 4, 2026, 13:49 UTC)
GitHub stars6 (reference 250, via hf-daily, Oct 4, 2026, 13:49 UTC)

The record

Reviewed by heuristic-v2 on Oct 2, 2026, 02:05 UTC. Paper type: method.
Categories: cs.CV
TOP, No.8 in the Artificial Intelligence edition of October 5, 2026.