# AwesomeCUA curated catalog > All 2,170 curated, source-reviewed records of the AwesomeCUA computer-use agent catalog, one compact entry each, grouped by kind. Snapshot 2026-11-07T05:19:53Z. Metadata-only bibliography records and unreviewed candidates are excluded. Each entry links its catalog page. Append `index.md` to that URL for the full annotation with evidence fingerprints, or `record.json` for the complete record. Full annotations by kind: [Models and checkpoints](https://awesomecua.com/categories/model/index.md), [Datasets](https://awesomecua.com/categories/dataset/index.md), [Benchmarks](https://awesomecua.com/categories/benchmark/index.md), [Agents](https://awesomecua.com/categories/agent/index.md), [Frameworks](https://awesomecua.com/categories/framework/index.md), [Infrastructure](https://awesomecua.com/categories/infrastructure/index.md), [Tools](https://awesomecua.com/categories/tool/index.md), [Products](https://awesomecua.com/categories/product/index.md), [Publishers](https://awesomecua.com/categories/company/index.md), [Learning resources](https://awesomecua.com/categories/learning/index.md), [Surveys](https://awesomecua.com/categories/survey/index.md), [Research library](https://awesomecua.com/papers/index.md). Reviews are AI-assisted inspections of primary sources. Code, data, weights and document licenses are recorded separately. Related reading: [English research survey](https://awesomecua.com/survey/). This working manuscript uses its own frozen inventory; its text is separate from the catalog annotations below. ## Models and checkpoints (98) ### AgentCPM-GUI 8B on-device smartphone agent built on MiniCPM-V that operates Chinese and English apps from screenshots, trained with reinforcement fine-tuning and a compact JSON action space. - https://awesomecua.com/resources/github-openbmb-agentcpm-gui/ · source: https://github.com/OpenBMB/AgentCPM-GUI - Model · reviewed 2026-10-16 · environments: android · topics: grounding, mobile-navigation - Licenses: code Apache-2.0 ### AgentCPM-GUI 8B checkpoint An eight-billion-parameter MiniCPM-V-based model that consumes smartphone screenshots and predicts compact actions for Chinese and English Android app tasks. - https://awesomecua.com/resources/url-8ef3985dce697e5feea9/ · source: https://huggingface.co/openbmb/AgentCPM-GUI - Model · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, grounding, reinforcement-learning - Licenses: weights apache-2.0 ### Aguvis Vision-based GUI agent models with unified action representations. - https://awesomecua.com/resources/github-xlang-ai-aguvis/ · source: https://github.com/xlang-ai/aguvis - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning ### Aria-UI Mixture-of-experts GUI grounding model with 3.9B activated parameters that can use text or interleaved image history, with released training data and an AndroidWorld agent. - https://awesomecua.com/resources/github-ariaui-aria-ui/ · source: https://github.com/AriaUI/Aria-UI - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding ### Aria-UI context-aware checkpoint The context-aware Aria-UI release that uses interaction context to support grounding in dynamic Android and desktop agent tasks. - https://awesomecua.com/resources/url-085b6811ddd6f797475d/ · source: https://huggingface.co/Aria-UI/Aria-UI-context-aware - Model · reviewed 2026-10-17 · environments: android, desktop · topics: grounding, interaction-history, action-prediction ### ARPO UI-TARS-1.5-7B checkpoint UI-TARS-1.5-7B-derived checkpoint trained with ARPO on OSWorld desktop tasks. - https://awesomecua.com/resources/url-db6fc5f29f186d13991c/ · source: https://huggingface.co/Fanbin/ARPO_UITARS1.5_7B - Model · reviewed 2026-10-18 · environments: desktop, linux · topics: reinforcement-learning, action-prediction ### AutoGLM Phone 9B AutoGLM phone-agent checkpoint for Open-AutoGLM, fine-tuned from GLM-4.1V-9B-Base with about 10.3B BF16 parameters and tagged for Chinese. - https://awesomecua.com/resources/url-c918d5c828a3c6cfbc5d/ · source: https://huggingface.co/zai-org/AutoGLM-Phone-9B - Model · reviewed 2026-10-16 · environments: android, ios · topics: mobile-navigation - Licenses: weights mit ### AutoGLM Phone 9B Multilingual Multilingual variant of the AutoGLM phone-agent checkpoint for Open-AutoGLM, likewise fine-tuned from GLM-4.1V-9B-Base with about 10.3B BF16 parameters. - https://awesomecua.com/resources/url-d236e7ec7a2b0028950a/ · source: https://huggingface.co/zai-org/AutoGLM-Phone-9B-Multilingual - Model · reviewed 2026-10-16 · environments: android, ios · topics: mobile-navigation - Licenses: weights mit ### BlueLM-GUI Mobile GUI agent research release with technical report, benchmark data, and evaluation tooling from vivo AI Lab. - https://awesomecua.com/resources/github-vivo-ai-lab-bluelm-gui/ · source: https://github.com/vivo-ai-lab/BlueLM-GUI - Model · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, reinforcement-learning - Licenses: code Apache-2.0 ### Claude Computer Use Claude tool capability for screenshot-based computer interaction. - https://awesomecua.com/resources/url-57f17b7d0c18b2d7c8bf/ · source: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool - Model · reviewed 2026-10-16 · environments: cross-platform · topics: planning, grounding ### Code2World-8B checkpoint Released GUI world-model checkpoint that turns a screenshot and proposed action into HTML for rendering a predicted next screen. - https://awesomecua.com/resources/url-bc3deaf8eeecf569e73b/ · source: https://huggingface.co/GD-ML/Code2World - Model · reviewed 2026-10-18 · environments: android · topics: world-models, screen-understanding - Licenses: weights MIT ### CogAgent Visual language model for GUI understanding and interaction. - https://awesomecua.com/resources/github-thudm-cogagent/ · source: https://github.com/THUDM/CogAgent - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, screen-understanding - Licenses: code Apache-2.0 ### CogAgent 9B 20241220 A bilingual GLM-4V-based execution model that uses screenshots, task instructions and action history to predict GUI operations. - https://awesomecua.com/resources/url-c5eb975c0d66f8a2954f/ · source: https://huggingface.co/zai-org/cogagent-9b-20241220 - Model · reviewed 2026-10-17 · environments: desktop, windows, macos, cross-platform · topics: grounding, action-prediction, screen-understanding - Licenses: weights other ### DiMo-GUI GUI grounding research using modality-aware visual reasoning and test-time scaling. - https://awesomecua.com/resources/github-vivo-dimo-gui/ · source: https://github.com/vivo/DiMo-GUI - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, efficiency - Licenses: code Apache-2.0 ### Dreamer-7B WebDreamer model checkpoint that predicts the consequences of web actions from screenshots and textual descriptions. - https://awesomecua.com/resources/url-e3c1a2a2d556e4728953/ · source: https://huggingface.co/osunlp/Dreamer-7B - Model · reviewed 2026-10-16 · environments: web · topics: world-models, planning - Licenses: weights apache-2.0 ### EvoCUA Computer-use model and training resources from the EvoCUA project. - https://awesomecua.com/resources/github-meituan-evocua/ · source: https://github.com/meituan/EvoCUA - Model · reviewed 2026-10-16 · environments: desktop · topics: reinforcement-learning, planning - Licenses: code Apache-2.0 ### EvoCUA 32B 20260105 Larger EvoCUA computer-use agent checkpoint (20260105 release) with about 33B BF16 parameters in the Qwen3-VL architecture, tagged for OSWorld and for English and Chinese. - https://awesomecua.com/resources/url-60a95f34072aeb2771cc/ · source: https://huggingface.co/meituan/EvoCUA-32B-20260105 - Model · reviewed 2026-10-16 · environments: desktop · topics: reinforcement-learning, planning - Licenses: weights apache-2.0 ### EvoCUA 8B 20260105 EvoCUA computer-use agent checkpoint (20260105 release) with about 8.8B BF16 parameters in the Qwen3-VL architecture, tagged for OSWorld and for English and Chinese. - https://awesomecua.com/resources/url-2a2682f86dcb8ebc9a80/ · source: https://huggingface.co/meituan/EvoCUA-8B-20260105 - Model · reviewed 2026-10-16 · environments: desktop · topics: reinforcement-learning, planning - Licenses: weights apache-2.0 ### Fara Microsoft browser-use agent models that act from screenshots by predicting click coordinates without accessibility trees; Fara1.5 comes in 4B, 9B and 27B, succeeding Fara-7B. - https://awesomecua.com/resources/github-microsoft-fara/ · source: https://github.com/microsoft/fara - Model · reviewed 2026-10-16 · environments: web · topics: planning, web-navigation - Licenses: code MIT ### Fara 7B Microsoft's Fara checkpoint, a Qwen2.5-VL-architecture image-text-to-text model with about 8.3B BF16 parameters, English, MIT-licensed and linked to arXiv paper 2511.19663. - https://awesomecua.com/resources/url-a4da77ee63f879cc8e63/ · source: https://huggingface.co/microsoft/Fara-7b - Model · reviewed 2026-10-16 · environments: web · topics: planning, web-navigation - Licenses: weights mit ### Ferret-UI Screen understanding research within the Ferret model family. - https://awesomecua.com/resources/github-apple-ml-ferret/ · source: https://github.com/apple/ml-ferret - Model · reviewed 2026-10-16 · environments: android, ios · topics: screen-understanding, grounding ### GELab Zero Open mobile GUI agent stack pairing a 4B model that runs on consumer hardware with local multi-device infrastructure, an MCP server, and AndroidDaily benchmark tasks. - https://awesomecua.com/resources/github-stepfun-ai-gelab-zero/ · source: https://github.com/stepfun-ai/gelab-zero - Model · reviewed 2026-10-16 · environments: android · topics: mobile-navigation - Licenses: code MIT ### GELab Zero 4B Preview Preview GELab-Zero phone-use agent checkpoint fine-tuned from Qwen3-VL-4B-Instruct, with about 4.4B BF16 parameters, tagged for Android in English and Chinese, under Apache-2.0. - https://awesomecua.com/resources/url-d7ad447ecd712c6d51b8/ · source: https://huggingface.co/stepfun-ai/GELab-Zero-4B-preview - Model · reviewed 2026-10-16 · environments: android · topics: mobile-navigation - Licenses: weights apache-2.0 ### Gemini Computer Use Gemini capability for generating interface actions from screenshots. - https://awesomecua.com/resources/url-879d4e3e75e83582bde6/ · source: https://ai.google.dev/gemini-api/docs/computer-use - Model · reviewed 2026-10-16 · environments: cross-platform · topics: planning, grounding ### Go-Browse WebArena Qwen 7B A Qwen-2.5-7B-Instruct checkpoint fine-tuned on Go-Browse WebArena trajectories for website task execution. - https://awesomecua.com/resources/url-db1b7b7efe3755ae5693/ · source: https://huggingface.co/apurvaga/go-browse-wa-qwen-7B - Model · reviewed 2026-10-17 · environments: web · topics: web-navigation, imitation-learning ### GroundGUI-8B Screenshot grounding checkpoint that emits a single click in a normalized coordinate space. - https://awesomecua.com/resources/url-de31b07ff24ab15d9bdf/ · source: https://huggingface.co/PrentisAI/GroundGUI-8B - Model · reviewed 2026-10-17 · environments: desktop, web, cross-platform · topics: grounding, screen-understanding - Licenses: weights apache-2.0 ### GUI-Actor GUI grounding model with attention-based action localization. - https://awesomecua.com/resources/github-microsoft-gui-actor/ · source: https://github.com/microsoft/GUI-Actor - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: code MIT ### GUI-Actor 7B Qwen2-VL GUI-Actor checkpoint fine-tuned from Qwen2-VL-7B-Instruct, with about 8.4B BF16 parameters, whose config names a pointer variant of the Qwen2-VL class rather than the stock one, under MIT. - https://awesomecua.com/resources/url-e3e5dd82a3102689c4ea/ · source: https://huggingface.co/microsoft/GUI-Actor-7B-Qwen2-VL - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: weights mit ### GUI-G1 GUI-grounding training code that modifies reasoning templates, reward functions, and the GRPO objective, with a linked model release. - https://awesomecua.com/resources/github-yuqi-zhou-gui-g1/ · source: https://github.com/Yuqi-Zhou/GUI-G1 - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, reinforcement-learning - Licenses: code Apache-2.0 ### GUI-G1 3B v1 A Qwen2.5-VL-3B checkpoint released for R1-style visual grounding experiments, predicting interface target locations from screenshots and instructions. - https://awesomecua.com/resources/url-9dc9d8d3b9a4b32d6f4b/ · source: https://huggingface.co/Yuqi-Zhou/GUI-G1-3B-v1 - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding, reinforcement-learning - Licenses: weights mit ### GUI-G2 GUI grounding models at 3B and 7B trained with Gaussian rewards for click proximity to element centers and region coverage, with variance scaled to element size. - https://awesomecua.com/resources/github-zju-real-gui-g2/ · source: https://github.com/ZJU-REAL/GUI-G2 - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding ### GUI-G2 7B A Qwen2.5-VL grounding checkpoint trained with Gaussian point and coverage rewards whose granularity adapts to interface-element scale. - https://awesomecua.com/resources/url-90eea6f6d099a3297850/ · source: https://huggingface.co/inclusionAI/GUI-G2-7B - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding, reinforcement-learning, reward-models - Licenses: weights apache-2.0 ### GUI-Owl 7B A Qwen2.5-VL-based GUI checkpoint released with Mobile-Agent-v3 for grounded actions and specialized roles in its agent framework. - https://awesomecua.com/resources/url-aeb32973694f91ad9920/ · source: https://huggingface.co/mPLUG/GUI-Owl-7B - Model · reviewed 2026-10-17 · environments: android, desktop · topics: grounding, action-prediction, planning - Licenses: weights mit ### GUI-R1 GUI action-model training and evaluation code with verifiable rewards, plus linked reinforcement-learning data and checkpoints. - https://awesomecua.com/resources/github-ritzz-ai-gui-r1/ · source: https://github.com/ritzz-ai/GUI-R1 - Model · reviewed 2026-10-16 · environments: cross-platform · topics: reinforcement-learning, grounding - Licenses: code Apache-2.0 ### GUI-R1 checkpoint bundle A single Hub release containing GUI-R1 3B and 7B action-model checkpoints trained with reinforcement learning across interface platforms. - https://awesomecua.com/resources/url-61a5321db10fc8a312b9/ · source: https://huggingface.co/ritzzai/GUI-R1 - Model · reviewed 2026-10-17 · environments: desktop, windows, linux, macos, android, web · topics: grounding, reinforcement-learning, action-prediction - Licenses: weights mit ### Holo Model Family H computer-use model family and its release-specific access options. - https://awesomecua.com/resources/url-c0bd07d82ce25ad1d99d/ · source: https://hcompany.ai/research/models - Model · reviewed 2026-10-16 · environments: cross-platform · topics: planning, grounding ### Holo1 7B A Qwen2.5-VL-based action model released for the Surfer-H web-agent system, supporting policy, interface-localization and validation roles. - https://awesomecua.com/resources/url-f35cfcbaeb79663eabeb/ · source: https://huggingface.co/Hcompany/Holo1-7B - Model · reviewed 2026-10-17 · environments: web · topics: web-navigation, grounding, planning - Licenses: weights apache-2.0 ### Holo2 4B A Qwen3-VL-based checkpoint for navigation and multi-step GUI actions across web, desktop and mobile interfaces. - https://awesomecua.com/resources/url-16a51d9e725ad5a5c629/ · source: https://huggingface.co/Hcompany/Holo2-4B - Model · reviewed 2026-10-17 · environments: web, desktop, cross-platform · topics: grounding, planning, action-prediction - Licenses: weights apache-2.0 ### Holo3 35B-A3B A sparse Qwen3.5-based GUI-action checkpoint released for navigation and computer-use workflows. - https://awesomecua.com/resources/url-a349aace3cfde25628a8/ · source: https://huggingface.co/Hcompany/Holo3-35B-A3B - Model · reviewed 2026-10-17 · environments: web, desktop, cross-platform · topics: grounding, planning, action-prediction - Licenses: weights apache-2.0 ### Holo4 35B-A3B BF16 A Qwen3.6-based computer-use checkpoint designed to request GUI actions, code execution and tool calls through an external agent harness. - https://awesomecua.com/resources/url-16083a0225a60882c704/ · source: https://huggingface.co/Hcompany/Holo4-35B-A3B - Model · reviewed 2026-10-17 · environments: desktop, web · topics: grounding, action-prediction, tool-integration - Licenses: weights apache-2.0 ### InfiGUI-G1 InfiGUI-G1 3B and 7B grounding models trained with Adaptive Exploration Policy Optimization, which samples several candidate answers per pass under an adaptive exploration reward. - https://awesomecua.com/resources/github-infixai-infigui-g1/ · source: https://github.com/InfiXAI/InfiGUI-G1 - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: code Apache-2.0 ### InfiGUI-G1 7B A GUI-grounding checkpoint trained with adaptive exploration rewards and multiple candidate answers to improve instruction-to-element alignment. - https://awesomecua.com/resources/url-d70a755e3b2102f25f34/ · source: https://huggingface.co/InfiX-ai/InfiGUI-G1-7B - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding, reinforcement-learning - Licenses: weights apache-2.0 ### InfiGUI-R1 3B GUI agent trained in two stages: distilling spatial reasoning from teacher models, then reinforcement learning that rewards accurate sub-goals and trains error recovery. - https://awesomecua.com/resources/github-reallm-labs-infigui-r1/ · source: https://github.com/Reallm-Labs/InfiGUI-R1 - Model · reviewed 2026-10-16 · environments: cross-platform · topics: reinforcement-learning, planning - Licenses: code Apache-2.0 ### InfiGUI-R1 3B A Qwen2.5-VL checkpoint trained with the Actor2Reasoner procedure to combine interface grounding with planning and recovery reasoning. - https://awesomecua.com/resources/url-35b9879a351cc3b3d7d5/ · source: https://huggingface.co/InfiX-ai/InfiGUI-R1-3B - Model · reviewed 2026-10-17 · environments: web, desktop, android · topics: grounding, reinforcement-learning, self-reflection - Licenses: weights apache-2.0 ### Llama-3-8B-Web A Llama 3 8B instruction-model checkpoint fine-tuned on WebLINX web interactions and dialogue, predicting browser actions from a textual state representation. - https://awesomecua.com/resources/url-7c3e8c4ba52428584079/ · source: https://huggingface.co/McGill-NLP/Llama-3-8B-Web - Model · reviewed 2026-10-17 · environments: web · topics: web-navigation, imitation-learning, action-prediction - Licenses: weights Meta Llama 3 Community License Agreement ### Magma Microsoft foundation model for multimodal agents, pretrained with Set-of-Mark and Trace-of-Mark objectives on UI, robotics and video data and fine-tunable for web and mobile UI navigation. - https://awesomecua.com/resources/github-microsoft-magma/ · source: https://github.com/microsoft/Magma - Model · reviewed 2026-10-17 · environments: web · topics: grounding, planning, mobile-navigation - Licenses: code MIT ### MAI-UI 8B A released MAI-UI checkpoint combining mobile GUI actions, agent-user interaction and tool-use-oriented navigation. - https://awesomecua.com/resources/url-00b1a3abf9bcb424b264/ · source: https://huggingface.co/Tongyi-MAI/MAI-UI-8B - Model · reviewed 2026-10-17 · environments: android · topics: grounding, mobile-navigation, tool-integration - Licenses: weights apache-2.0 ### MobileRL 9B checkpoint A GLM-4.1V-based mobile action checkpoint released by the MobileRL project for Android interaction. - https://awesomecua.com/resources/url-d5667c854845c77b9f5b/ · source: https://huggingface.co/xuyifan/MobileRL-9B - Model · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, mobile-navigation - Licenses: weights apache-2.0 ### OmniParser v2 model bundle A screen-parsing weight bundle containing icon-region detectors and an icon captioner for converting screenshots into agent-readable interface elements. - https://awesomecua.com/resources/url-e21ca7c859544dbf5fb6/ · source: https://huggingface.co/microsoft/OmniParser-v2.0 - Model · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding - Licenses: weights Mixed by component: icon_detect = AGPL-3.0; icon_detect_v3 = MIT; icon_caption = MIT ### OpenAI CUA Computer-using model introduced with the Operator research preview. - https://awesomecua.com/resources/url-74eeb7c9fb20674fe6e2/ · source: https://openai.com/index/computer-using-agent/ - Model · reviewed 2026-10-16 · environments: web, desktop · topics: planning, grounding ### OpenCUA Open computer-use research stack including models and training resources. - https://awesomecua.com/resources/github-xlang-ai-opencua/ · source: https://github.com/xlang-ai/OpenCUA - Model · reviewed 2026-10-16 · environments: desktop · topics: data-generation, planning - Licenses: code MIT ### OpenCUA 32B A Qwen2.5-VL-based computer-use checkpoint that combines screenshot history with planning and grounded executable actions. - https://awesomecua.com/resources/url-ad7f176848fd84de64e2/ · source: https://huggingface.co/xlangai/OpenCUA-32B - Model · reviewed 2026-10-17 · environments: desktop, linux, windows, macos · topics: grounding, planning, self-reflection - Licenses: weights mit ### OpenCUA 72B A Qwen2.5-VL-based computer-use checkpoint that combines screenshot history and reflective reasoning to predict executable desktop actions. - https://awesomecua.com/resources/url-4a0b1692589594745a3c/ · source: https://huggingface.co/xlangai/OpenCUA-72B - Model · reviewed 2026-10-17 · environments: desktop, linux, windows, macos · topics: grounding, planning, action-prediction, self-reflection - Licenses: weights mit ### OpenMobile-2 27B Mobile-agent checkpoint based on Qwen3.6-27B for GUI and app-native tool interaction. - https://awesomecua.com/resources/url-1611de0e6d606165d12d/ · source: https://huggingface.co/OpenMobile-2/OpenMobile-2-27B - Model · reviewed 2026-10-17 · environments: android, web · topics: mobile-navigation, tool-integration - Licenses: weights apache-2.0 ### OpenMobile-2 9B Mobile-agent checkpoint trained on OpenMobile-Data-v2 for screenshot actions and app-native tool calls. - https://awesomecua.com/resources/url-d35224f787e7ec21ce36/ · source: https://huggingface.co/OpenMobile-2/OpenMobile-2-9B - Model · reviewed 2026-10-17 · environments: android, web · topics: mobile-navigation, tool-integration - Licenses: weights apache-2.0 ### OS-Atlas OS-Atlas GUI action models, including grounding bases fine-tuned from InternVL2-4B and Qwen2-VL-7B-Instruct that accept any image size and output 0-1000 relative coordinates. - https://awesomecua.com/resources/github-os-copilot-os-atlas/ · source: https://github.com/OS-Copilot/OS-Atlas - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: code Apache-2.0 ### OS-Atlas Base 4B OS-Atlas base checkpoint with about 4.1B parameters, fine-tuned from InternVL2-4B and loaded in Transformers through custom InternVL chat model code. - https://awesomecua.com/resources/url-30c831a1198b9bc2a0f5/ · source: https://huggingface.co/OS-Copilot/OS-Atlas-Base-4B - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: weights apache-2.0 ### OS-Atlas Base 7B OS-Atlas base checkpoint fine-tuned from Qwen2-VL-7B-Instruct, using the standard Qwen2-VL architecture in Transformers with about 8.3B BF16 parameters. - https://awesomecua.com/resources/url-82a30f45d9e855f423c7/ · source: https://huggingface.co/OS-Copilot/OS-Atlas-Base-7B - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: weights apache-2.0 ### OS-Genesis 7B AC The AndroidControl-oriented Qwen2-VL checkpoint trained with mobile trajectories generated through OS-Genesis reverse task synthesis. - https://awesomecua.com/resources/url-80858580c1009eba0e98/ · source: https://huggingface.co/OS-Copilot/OS-Genesis-7B-AC - Model · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, action-prediction, imitation-learning - Licenses: weights apache-2.0 ### PAE LLaVA 7B A released LLaVA checkpoint for visual browser interaction from the PAE autonomous task-proposal and policy-learning system. - https://awesomecua.com/resources/url-5d61c1b5fc8dd650518d/ · source: https://huggingface.co/yifeizhou/pae-llava-7b - Model · reviewed 2026-10-17 · environments: web · topics: web-navigation, reinforcement-learning - Licenses: weights cc-by-nc-4.0 ### PC-Agent-E checkpoint Computer-use checkpoint fine-tuned on PC-Agent-E trajectories; the model-card prose identifies Qwen2.5-VL-72B-Instruct as its starting model. - https://awesomecua.com/resources/url-04aa7569d183379d615c/ · source: https://huggingface.co/henryhe0123/PC-Agent-E - Model · reviewed 2026-10-18 · environments: desktop, windows, linux · topics: action-prediction, imitation-learning - Licenses: weights MIT ### Phi-Ground Microsoft GUI grounding models from an empirical study of grounding training, from data collection to model training, released as Phi-Ground-4B-7C with evaluation code. - https://awesomecua.com/resources/github-microsoft-phi-ground/ · source: https://github.com/microsoft/Phi-Ground - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: code MIT ### Phi-Ground checkpoint Microsoft Phi-Ground GUI grounding checkpoint fine-tuned from Phi-3.5-vision-instruct, shipped as two PyTorch weight shards with custom Phi-3-V configuration and processing code under MIT. - https://awesomecua.com/resources/url-cee45b940d5fbff3ccca/ · source: https://huggingface.co/microsoft/Phi-Ground - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: weights mit ### Plan-and-Act Planner 70B The planner component of Plan-and-Act, generating structured high-level plans that an executor turns into web actions. - https://awesomecua.com/resources/url-3932d69c6b6f9b2a7ba8/ · source: https://huggingface.co/xTRam1/plan-and-act-planner-70b - Model · reviewed 2026-10-17 · environments: web · topics: planning, web-navigation, long-horizon - Licenses: weights mit ### POINTS-GUI POINTS-GUI-G-8B GUI grounding model, trained from a POINTS-1.5 base without native grounding through a data pipeline that unifies coordinates, filters noise and adds hard cases. - https://awesomecua.com/resources/github-tencent-points-gui/ · source: https://github.com/Tencent/POINTS-GUI - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding ### POINTS-GUI-G 8B A specialized interface-grounding checkpoint that returns normalized point coordinates for a requested element in a screenshot. - https://awesomecua.com/resources/url-cc931199bda2bdf1f5f7/ · source: https://huggingface.co/tencent/POINTS-GUI-G - Model · reviewed 2026-10-17 · environments: web, desktop, cross-platform · topics: grounding, screen-understanding - Licenses: weights other ### ScaleCUA Computer-use model family trained with cross-platform interaction data. - https://awesomecua.com/resources/github-opengvlab-scalecua/ · source: https://github.com/OpenGVLab/ScaleCUA - Model · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation, planning - Licenses: code Apache-2.0 ### ScaleCUA 7B A Qwen2.5-VL-based computer-use checkpoint with direct grounding and agent action modes across desktop, mobile and web interfaces. - https://awesomecua.com/resources/url-ff7b1873a3eead89607a/ · source: https://huggingface.co/OpenGVLab/ScaleCUA-7B - Model · reviewed 2026-10-17 · environments: web, desktop, android, linux · topics: grounding, action-prediction, planning - Licenses: weights apache-2.0 ### SeeClick GUI grounding model trained to locate visual interface elements. - https://awesomecua.com/resources/github-njucckevin-seeclick/ · source: https://github.com/njucckevin/SeeClick - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: code Apache-2.0 ### SeeClick checkpoint The released SeeClick visual grounding checkpoint for mapping an interface screenshot and referring instruction to a point or bounding box. - https://awesomecua.com/resources/url-cfea8a0949472a637990/ · source: https://huggingface.co/cckevinn/SeeClick - Model · reviewed 2026-10-17 · environments: web, desktop, android, ios · topics: grounding ### Seg2Click Qwen2.5-VL 3B Seg2Click checkpoint based on Qwen2.5-VL-3B with a custom action-mask decoder. - https://awesomecua.com/resources/url-23ad240757185c6f7eac/ · source: https://huggingface.co/jcwang0602/Seg2Click-Qwen2.5-VL-3B - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding - Licenses: weights other ### Seg2Click Qwen3.5 0.8B Compact Seg2Click checkpoint with a merged Qwen3.5 backbone and an action-mask decoder. - https://awesomecua.com/resources/url-7405463ba56f9a41e79a/ · source: https://huggingface.co/jcwang0602/Seg2Click-Qwen3.5-0.8B - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding - Licenses: weights apache-2.0 ### Seg2Click Qwen3.5 2B Seg2Click checkpoint using a 2B Qwen3.5 backbone to predict a mask and select an interface click. - https://awesomecua.com/resources/url-fec1c5aa8fd93cb78923/ · source: https://huggingface.co/jcwang0602/Seg2Click-Qwen3.5-2B - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding - Licenses: weights apache-2.0 ### Seg2Click Qwen3.5 4B Seg2Click checkpoint pairing a 4B Qwen3.5 backbone with pixel-level action-mask prediction. - https://awesomecua.com/resources/url-29588aa5651b5839c045/ · source: https://huggingface.co/jcwang0602/Seg2Click-Qwen3.5-4B - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding - Licenses: weights apache-2.0 ### Seg2Click Qwen3.5 9B Seg2Click checkpoint pairing a 9B Qwen3.5 backbone with a decoder for valid GUI click regions. - https://awesomecua.com/resources/url-826f3753905c5c43bf46/ · source: https://huggingface.co/jcwang0602/Seg2Click-Qwen3.5-9B - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding - Licenses: weights apache-2.0 ### ShowUI Lightweight GUI-agent vision-language-action model from Show Lab and Microsoft, released with UI-guided token selection, training code, the ShowUI-web and ShowUI-desktop datasets and vLLM inference. - https://awesomecua.com/resources/github-showlab-showui/ · source: https://github.com/showlab/ShowUI - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, efficiency - Licenses: code Apache-2.0 ### ShowUI 2B ShowUI vision-language-action checkpoint fine-tuned from Qwen2-VL-2B-Instruct, published as a single PyTorch weight file under the MIT license and tagged for GUI agents and computer use. - https://awesomecua.com/resources/url-c0f39f7cf2cd9c18e8bd/ · source: https://huggingface.co/showlab/ShowUI-2B - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, efficiency - Licenses: weights mit ### ShowUI-π Vision-language-action model and training code for GUI dragging, with links to checkpoints and continuous-action evaluation data. - https://awesomecua.com/resources/github-showlab-showui-pi/ · source: https://github.com/showlab/showui-pi - Model · reviewed 2026-10-16 · environments: desktop · topics: grounding, automation ### ShowUI-π checkpoint A compact visual-action checkpoint that predicts continuous GUI drag trajectories from a screenshot and a natural-language instruction. - https://awesomecua.com/resources/url-f70aa7c95824a8729516/ · source: https://huggingface.co/showlab/ShowUI-pi - Model · reviewed 2026-10-17 · environments: unspecified · topics: grounding, action-prediction ### SpiritSight-Agent 8B base An InternVL2-based visual GUI checkpoint distributed for grounding and interface navigation, with the publisher recommending task-specific fine-tuning of the base release. - https://awesomecua.com/resources/url-52b7de0e43b1afc9c226/ · source: https://huggingface.co/SenseLLM/SpiritSight-Agent-8B - Model · reviewed 2026-10-17 · environments: cross-platform · topics: grounding, screen-understanding - Licenses: weights apache-2.0 ### UGround Code, training data and weights for universal visual grounding of GUI elements from screenshots, including Qwen2-VL-based UGround-V1 at 2B, 7B and 72B and offline and online evaluation suites. - https://awesomecua.com/resources/github-osu-nlp-group-uground/ · source: https://github.com/OSU-NLP-Group/UGround - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: code MIT ### UGround V1 7B UGround-V1 checkpoint fine-tuned from Qwen2-VL-7B, with about 8.3B BF16 parameters in the standard Qwen2-VL architecture, provided as safetensors and a PyTorch file under Apache-2.0. - https://awesomecua.com/resources/url-e554295a7f3f5c9ad0d6/ · source: https://huggingface.co/osunlp/UGround-V1-7B - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: weights apache-2.0 ### UI-AGILE GUI agent trained by RL with a continuous reward, a simple-thinking reward and cropping-based resampling on about 9k samples, plus a plug-in decomposed grounding-and-selection inference method. - https://awesomecua.com/resources/github-kdegroup-ui-agile/ · source: https://github.com/KDEGroup/UI-AGILE - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, reinforcement-learning - Licenses: code MIT ### UI-AGILE checkpoint The released UI-AGILE model for GUI grounding with reward shaping, crop-based resampling and decomposed selection during inference. - https://awesomecua.com/resources/url-b94e285c3f757e2d0f3a/ · source: https://huggingface.co/KDEGroup/UI-AGILE - Model · reviewed 2026-10-17 · environments: android · topics: grounding, reinforcement-learning, efficiency - Licenses: weights mit ### UI-Genie Agent 7B A Qwen2.5-VL mobile action model trained using existing and reward-guided synthetic trajectories from the UI-Genie framework. - https://awesomecua.com/resources/url-6d959b58bd4a2e98e417/ · source: https://huggingface.co/HanXiao1999/UI-Genie-Agent-7B - Model · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, imitation-learning, action-prediction - Licenses: weights mit ### UI-R1 Rule-based reinforcement learning for GUI action prediction, releasing UI-R1-3B, trained from Qwen2.5-VL-3B, the efficiency-oriented UI-R1-E-3B, and training data. - https://awesomecua.com/resources/github-lll6gg-ui-r1/ · source: https://github.com/lll6gg/UI-R1 - Model · reviewed 2026-10-16 · environments: cross-platform · topics: reinforcement-learning, grounding - Licenses: code Apache-2.0 ### UI-R1 3B A Qwen2.5-VL checkpoint trained with reinforcement learning for GUI action prediction and evaluated on mobile, desktop and web grounding tasks. - https://awesomecua.com/resources/url-f13474d7aa734a30624d/ · source: https://huggingface.co/LZXzju/Qwen2.5-VL-3B-UI-R1 - Model · reviewed 2026-10-17 · environments: web, desktop, android · topics: grounding, reinforcement-learning, action-prediction - Licenses: weights mit ### UI-S1 7B A mobile GUI checkpoint trained with semi-online reinforcement learning that uses offline trajectories to improve multi-turn action prediction. - https://awesomecua.com/resources/url-1c4ee99a9a55ae71a7ac/ · source: https://huggingface.co/mPLUG/UI-S1-7B - Model · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, mobile-navigation, action-prediction - Licenses: weights apache-2.0 ### UI-TARS Native GUI agent models; UI-TARS-1.5 adds reinforcement-learned reasoning before acting and an open 7B checkpoint, and UI-TARS-2 extends to games, code and tool use. - https://awesomecua.com/resources/github-bytedance-ui-tars/ · source: https://github.com/bytedance/UI-TARS - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning - Licenses: code Apache-2.0 ### UI-TARS 1.5 7B UI-TARS 1.5 GUI-agent checkpoint in the Qwen2.5-VL architecture with about 8.3B parameters stored in FP32, released under Apache-2.0 for English image-text-to-text use in Transformers. - https://awesomecua.com/resources/url-4d32ad3682ae2d71c178/ · source: https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning - Licenses: weights apache-2.0 ### UI-Venus GUI agent model family; UI-Venus-2, at 9B and 27B, acts on mobile, web and desktop in one end-to-end agent, trained with keypoint-based multi-judge task verification. - https://awesomecua.com/resources/github-inclusionai-ui-venus/ · source: https://github.com/inclusionAI/UI-Venus - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning ### UI-Venus 2 9B UI-Venus 2 GUI-agent checkpoint fine-tuned from Qwen3.5-9B, with about 9.4B BF16 parameters in the Qwen3.5 multimodal architecture, loaded through Transformers for image-text-to-text. - https://awesomecua.com/resources/url-611df37e5f3caa6befd7/ · source: https://huggingface.co/inclusionAI/UI-Venus-2-9b - Model · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning ### UIX-Qwen2 A MultiUI-trained visual-language checkpoint for text-rich webpage understanding and interface tasks. - https://awesomecua.com/resources/url-54997a3c67e54abdf319/ · source: https://huggingface.co/neulab/UIX-Qwen2 - Model · reviewed 2026-10-17 · environments: web · topics: screen-understanding, grounding - Licenses: weights odc-by ### WebJudge 7B A visual trajectory judge that retains selected intermediate screenshots to evaluate whether a web agent completed its task. - https://awesomecua.com/resources/url-772161f06c2b3d4978b5/ · source: https://huggingface.co/osunlp/WebJudge-7B - Model · reviewed 2026-10-17 · environments: web · topics: evaluation, reward-models, web-navigation - Licenses: weights apache-2.0 ### WebRL Llama 3.1 8B A WebRL browser-policy checkpoint for performing actions in the five website environments of WebArena. - https://awesomecua.com/resources/url-a6ee20fea67438d91e06/ · source: https://huggingface.co/zai-org/webrl-llama-3.1-8b - Model · reviewed 2026-10-17 · environments: web · topics: web-navigation, reinforcement-learning - Licenses: weights other ### WebRL outcome reward model 8B The outcome-supervision checkpoint used by WebRL to score browser trajectories during policy training. - https://awesomecua.com/resources/url-d1e85aeb6d96859c109d/ · source: https://huggingface.co/zai-org/webrl-orm-llama-3.1-8b - Model · reviewed 2026-10-17 · environments: web · topics: reward-models, reinforcement-learning, web-navigation ### WebShepherd 3B A public text-based process-reward checkpoint from the Web-Shepherd collection for scoring web-agent decisions against task checklists. - https://awesomecua.com/resources/url-72f7fe0211c6551b945f/ · source: https://huggingface.co/LangAGI-Lab/WebShepherd_3B - Model · reviewed 2026-10-17 · environments: web · topics: reward-models, web-navigation ### ZonUI-3B checkpoint Qwen2.5-VL-3B-based GUI grounding checkpoint that predicts click locations from screenshots and text instructions across desktop, mobile and web screens. - https://awesomecua.com/resources/url-c8c046847f829fea4821/ · source: https://huggingface.co/zonghanHZH/ZonUI-3B - Model · reviewed 2026-10-18 · environments: cross-platform, desktop, web · topics: grounding, screen-understanding, efficiency - Licenses: weights Apache-2.0 ## Datasets (86) ### AgentNet 22.6K human-annotated desktop computer-use tasks on Windows, macOS and Ubuntu, released with OpenCUA for training and evaluating vision-language-action models. - https://awesomecua.com/resources/url-964d70c40bfc5ca8e1ee/ · source: https://huggingface.co/datasets/xlangai/AgentNet - Dataset · reviewed 2026-10-16 · environments: desktop · topics: data-generation - Licenses: data mit ### AgentSynth tasks and trajectories Published AgentSynth task definitions and recorded GUI trajectories, with OSWorld and InSTA training configurations and packaged screenshots, thoughts and commands. - https://awesomecua.com/resources/url-58a209f2b3b970916396/ · source: https://huggingface.co/datasets/sunblaze-ucb/AgentSynth - Dataset · reviewed 2026-10-18 · environments: desktop, web · topics: data-generation, long-horizon, imitation-learning - Licenses: data Apache-2.0 ### AgentTrek Synthesizes browser-agent trajectories by replaying web tutorials as guidance, releasing the trajectory data with multimodal grounding and reasoning and an AgentTrek-32B agent. - https://awesomecua.com/resources/github-xlang-ai-agenttrek/ · source: https://github.com/xlang-ai/AgentTrek - Dataset · reviewed 2026-10-16 · environments: web · topics: data-generation, video ### Aguvis stage 1 grounding data The first-stage Aguvis training collection, packaging GUI element-grounding annotations and image archives from multiple interface-data sources. - https://awesomecua.com/resources/url-06dd5ac7917c45d48509/ · source: https://huggingface.co/datasets/xlangai/aguvis-stage1 - Dataset · reviewed 2026-10-17 · environments: cross-platform · topics: grounding, data-generation - Licenses: data apache-2.0 ### Aguvis stage 2 trajectory data The second-stage Aguvis collection of interface trajectories for learning action sequences, planning and reasoning after grounding pretraining. - https://awesomecua.com/resources/url-b5e2127e07f692d516b9/ · source: https://huggingface.co/datasets/xlangai/aguvis-stage2 - Dataset · reviewed 2026-10-17 · environments: cross-platform · topics: planning, data-generation - Licenses: data apache-2.0 ### AMEX Code for AMEX Android GUI data that converts its three annotation levels into question-answer pairs and fine-tunes a LLaMA2-Accessory model, with a released checkpoint. - https://awesomecua.com/resources/github-yuxiangchai-amex-codebase/ · source: https://github.com/YuxiangChai/AMEX-codebase - Dataset · reviewed 2026-10-16 · environments: android · topics: data-generation - Licenses: code Apache-2.0 ### Android in the Wild Human demonstrations of Android tasks with screenshots and touch actions. - https://awesomecua.com/resources/component-google-research-android-in-the-wild/ · source: https://github.com/google-research/google-research/tree/master/android_in_the_wild - Dataset · reviewed 2026-10-16 · environments: android · topics: data-generation, mobile-navigation - Licenses: code Apache-2.0 ### AndroidCode HTML and annotation release used for Code2World’s action-conditioned GUI-state prediction training, constructed from mobile GUI trajectories. - https://awesomecua.com/resources/url-cf4d2d45a7314cb16acb/ · source: https://huggingface.co/datasets/GD-ML/AndroidCode - Dataset · reviewed 2026-10-18 · environments: android · topics: world-models, data-generation, screen-understanding - Licenses: data MIT ### AndroidControl Android demonstrations with high-level goals, step instructions, screenshots, and accessibility trees. - https://awesomecua.com/resources/component-google-research-android-control/ · source: https://github.com/google-research/google-research/tree/master/android_control - Dataset · reviewed 2026-10-16 · environments: android · topics: data-generation, mobile-navigation - Licenses: code Apache-2.0 ### Aria-UI Context-aware Data Episode-based mobile GUI grounding data released with Aria-UI for incorporating interaction history into target localization. - https://awesomecua.com/resources/url-07b65dc937756ddff4a1/ · source: https://huggingface.co/datasets/Aria-UI/Aria-UI_Context-aware_Data - Dataset · reviewed 2026-10-16 · environments: android · topics: grounding, memory - Licenses: data apache-2.0 ### Aria-UI Data GUI element descriptions and grounding instructions spanning web, mobile, and Ubuntu desktop interfaces. - https://awesomecua.com/resources/url-f8566bd0394d36ed4f8e/ · source: https://huggingface.co/datasets/Aria-UI/Aria-UI_Data - Dataset · reviewed 2026-10-16 · environments: web, android, desktop, linux · topics: grounding, screen-understanding - Licenses: data apache-2.0 ### AutoDroid V2 training data Conversation-formatted supervision released for fine-tuning AutoDroid V2, with the project documenting a ShareGPT-style JSON configuration. - https://awesomecua.com/resources/url-9a3506ca6ef034adb424/ · source: https://huggingface.co/datasets/BlitherBoom/AutoDroid-V2-Dataset - Dataset · reviewed 2026-10-17 · environments: android · topics: data-generation, imitation-learning, mobile-navigation - Licenses: data mit ### AutoGUI v1 training data Screenshot-based grounding and captioning samples with UI functionality descriptions, target coordinates, and bounding boxes. - https://awesomecua.com/resources/url-79287058342289aa5d45/ · source: https://huggingface.co/datasets/AutoGUI/AutoGUI-v1-702k - Dataset · reviewed 2026-10-16 · environments: web, cross-platform · topics: grounding, screen-understanding - Licenses: data cc-by-4.0 ### CAGUI Chinese Android GUI benchmark released with AgentCPM-GUI, with a grounding track for widgets and OCR text and an agent track for multi-step tasks in Chinese apps. - https://awesomecua.com/resources/url-a06622da7688e3718a7b/ · source: https://huggingface.co/datasets/openbmb/CAGUI - Dataset · reviewed 2026-10-16 · environments: android · topics: data-generation - Licenses: data cc-by-nc-4.0 ### CMGUI 18k trajectories with 98k steps from more than 50 Chinese mobile apps in domains such as e-commerce and social media, for building GUI agents on Chinese apps. - https://awesomecua.com/resources/url-ab26f9ef56d651cee587/ · source: https://huggingface.co/datasets/alibabagroup/CMGUI - Dataset · reviewed 2026-10-16 · environments: android · topics: data-generation - Licenses: data cc-by-4.0 ### CUA-Gym Computer-use environment and task resources for agent research. - https://awesomecua.com/resources/url-6aa1aa58a7a888395189/ · source: https://huggingface.co/datasets/xlangai/CUA-Gym - Dataset · reviewed 2026-10-16 · environments: desktop · topics: evaluation, data-generation - Licenses: data cc-by-4.0 ### DragOn Drag-grounding training and evaluation data pairing each screenshot instruction with start and end boxes, across text highlighting, spreadsheet cells, slide resizing and sliders. - https://awesomecua.com/resources/url-b854ff53e497da3a5ad8/ · source: https://huggingface.co/datasets/Hcompany/DragOn - Dataset · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: data apache-2.0 ### Dreamer V1 Data Web-world-model training data with prompts, images, action descriptions, and responses, released with the WebDreamer project. - https://awesomecua.com/resources/url-eadc6cba9ea46c38dd1d/ · source: https://huggingface.co/datasets/osunlp/Dreamer-V1-Data - Dataset · reviewed 2026-10-16 · environments: web · topics: world-models, data-generation - Licenses: data cc-by-nc-sa-4.0 ### DroidCall data Instruction-to-function-call examples for learning Android intent invocation, with training, testing and API-annotation files for mobile task execution. - https://awesomecua.com/resources/url-ae8f5208eea9e7f1f606/ · source: https://huggingface.co/datasets/mllmTeam/DroidCall - Dataset · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, tool-integration, data-generation - Licenses: data apache-2.0 ### FingerTip-20K Over 20,000 human Android episodes across 500+ apps with user context and long-term usage patterns, plus tracks for proactive intent prediction and personalized execution. - https://awesomecua.com/resources/github-tsinghua-fib-lab-fingertip-20k/ · source: https://github.com/tsinghua-fib-lab/FingerTip-20K - Dataset · reviewed 2026-10-16 · environments: android · topics: data-generation ### Go-Browse-WA processed data Flattened web-interaction training records with accessibility-tree observations, assistant completions, trajectory rewards and step-link metadata. - https://awesomecua.com/resources/url-a13a187bc058fbb4f6af/ · source: https://huggingface.co/datasets/apurvaga/go-browse-wa - Dataset · reviewed 2026-10-17 · environments: web · topics: web-navigation, imitation-learning ### GroundGUI RL Dataset Desktop grounding prompts selected from GroundCUA and formatted for click-based reinforcement learning with target-box rewards. - https://awesomecua.com/resources/url-4e71dc456c4112eed727/ · source: https://huggingface.co/datasets/PrentisAI/GroundGUI_RL - Dataset · reviewed 2026-10-17 · environments: desktop · topics: grounding, reinforcement-learning, data-generation - Licenses: data mit ### Grounding-Anything-GUI The public training archive linked by the Phi-Ground project for its Phi-Ground-Any model. - https://awesomecua.com/resources/url-d1830de02226155ee16d/ · source: https://huggingface.co/datasets/Miaosen/Grounding-Anything-GUI - Dataset · reviewed 2026-10-17 · environments: unspecified · topics: grounding, data-generation - Licenses: data apache-2.0 ### GUI-360 Desktop GUI trajectory data supporting multiple interface understanding tasks. - https://awesomecua.com/resources/url-0b590d03a3d5103a79e4/ · source: https://huggingface.co/datasets/vyokky/GUI-360 - Dataset · reviewed 2026-10-16 · environments: desktop, windows · topics: grounding, data-generation - Licenses: data mit ### GUI-Action Narrator Action narration resources for GUI interaction trajectories. - https://awesomecua.com/resources/github-showlab-gui-action-narrator/ · source: https://github.com/showlab/GUI-Action-Narrator - Dataset · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation, video ### GUI-Actor data A processed screenshot and bounding-box collection released for training GUI-Actor visual grounding models. - https://awesomecua.com/resources/url-66096ad4057a3d3726f5/ · source: https://huggingface.co/datasets/cckevinn/GUI-Actor-Data - Dataset · reviewed 2026-10-17 · environments: unspecified · topics: grounding, data-generation ### GUI-CIDER causal knowledge data A published JSONL corpus of synthesized planning and causal knowledge for GUI-agent mid-training, linked directly from the official GUI-CIDER project. - https://awesomecua.com/resources/url-d0300db5ba754b1a64fa/ · source: https://huggingface.co/datasets/wuuuuuz/GUI-CIDER - Dataset · reviewed 2026-10-17 · environments: android · topics: data-generation, imitation-learning, planning ### GUI-Drag Data for learning text-dragging interactions in graphical interfaces. - https://awesomecua.com/resources/url-aa28e1329a52e341d285/ · source: https://huggingface.co/datasets/osunlp/GUI-Drag-dataset - Dataset · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Licenses: data apache-2.0 ### GUI-Lasagne L1 The published GUI-Lasagne training-data collection used for SpiritSight, distributed as dated archive shards for visual interface learning. - https://awesomecua.com/resources/url-0c5659ca9e59d942aea3/ · source: https://huggingface.co/datasets/SenseLLM/GUI-Lasagne-L1 - Dataset · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, data-generation - Licenses: data apache-2.0 ### GUI-Net-1M A TongUI-collected training corpus from multimodal web tutorials, distributed as annotation files and split screenshot archives for GUI trajectory supervision. - https://awesomecua.com/resources/url-8c9aaafb918aaf8b4e92/ · source: https://huggingface.co/datasets/Bofeee5675/GUI-Net-1M - Dataset · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, grounding, imitation-learning - Licenses: data apache-2.0 ### GUI-Odyssey Human-demonstrated Android episodes for cross-app navigation, spanning six task types, 212 apps and about 1,400 app combinations, for training and evaluating agents. - https://awesomecua.com/resources/github-opengvlab-gui-odyssey/ · source: https://github.com/OpenGVLab/GUI-Odyssey - Dataset · reviewed 2026-10-16 · environments: android · topics: cross-app, data-generation ### GUI-R1 data bundle Training and evaluation Parquet bundles accompanying GUI-R1, including GUI grounding and mobile/web action-prediction data. - https://awesomecua.com/resources/url-df680385f18b01a825be/ · source: https://huggingface.co/datasets/ritzzai/GUI-R1 - Dataset · reviewed 2026-10-17 · environments: desktop, windows, linux, macos, android, web · topics: grounding, reinforcement-learning, evaluation - Licenses: data mit ### GUI-World GUI video benchmark and training data covering six scenarios, from Android and iOS to websites and XR, with eight question types for image and video LLMs and the GUI-Vid model. - https://awesomecua.com/resources/github-dongping-chen-gui-world/ · source: https://github.com/Dongping-Chen/GUI-World - Dataset · reviewed 2026-10-17 · environments: cross-platform · topics: video, screen-understanding ### GUIAct Web and Android GUI action-instruction data, with single-step, multi-step and smartphone partitions. - https://awesomecua.com/resources/url-6cd150ec23361b4d2e84/ · source: https://huggingface.co/datasets/yiye2023/GUIAct - Dataset · reviewed 2026-10-17 · environments: web, android · topics: grounding, data-generation ### GUIChat Conversational GUI supervision with text-rich images and bounding boxes, organized as single-turn questions and multi-turn dialogues. - https://awesomecua.com/resources/url-edf38d43800be8c955ae/ · source: https://huggingface.co/datasets/yiye2023/GUIChat - Dataset · reviewed 2026-10-17 · environments: unspecified · topics: grounding, data-generation ### GUICourse Three datasets for turning general VLMs into GUI agents: GUIEnv for OCR and grounding, GUIAct with 67k single-step and 15k multi-step web and Android actions, and GUIChat dialogues. - https://awesomecua.com/resources/github-yiye3-guicourse/ · source: https://github.com/yiye3/GUICourse - Dataset · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation ### GUIEnv-local The released local-supervision portion of GUIEnv, pairing webpage regions and text for OCR and GUI grounding. - https://awesomecua.com/resources/url-0af07e00658a430cee6a/ · source: https://huggingface.co/datasets/yiye2023/GUIEnv - Dataset · reviewed 2026-10-17 · environments: web · topics: grounding, data-generation ### GUIMid training data A released GUI-agent mid-training mixture and supporting domain datasets, together with a post-training trajectory file and image archives for web and mobile interaction. - https://awesomecua.com/resources/url-422010da7a5c845a18f6/ · source: https://huggingface.co/datasets/hkust-nlp/GUIMid - Dataset · reviewed 2026-10-17 · environments: web, android · topics: imitation-learning, planning, screen-understanding - Licenses: data mit ### GUIOdyssey — current data release Cross-app Android interaction episodes with fine-grained step instructions, screen descriptions, action context, and target boxes. - https://awesomecua.com/resources/url-511903ebfac45e6751b2/ · source: https://huggingface.co/datasets/hflqf88888/GUIOdyssey - Dataset · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, cross-app, data-generation - Licenses: data cc-by-4.0 ### Jedi Synthetic GUI grounding data associated with UI decomposition research and the OSWorld-G benchmark. - https://awesomecua.com/resources/url-e062cf1445eaee0f3ade/ · source: https://huggingface.co/datasets/xlangai/Jedi - Dataset · reviewed 2026-10-17 · environments: desktop · topics: grounding, data-generation - Licenses: data apache-2.0 ### Mobile3M Pre-training corpus for Xiaomi's MobileVLM, distributed as per-app graph archives for mobile apps such as Baidu Map, QQ Music, Ctrip and Vipshop. - https://awesomecua.com/resources/url-b80bfa0a4939a493a11f/ · source: https://huggingface.co/datasets/xwk123/Mobile3M - Dataset · reviewed 2026-10-16 · environments: android · topics: screen-understanding ### MobileViews Over 600,000 Android screenshot and view-hierarchy pairs from more than 20,000 Google Play apps, collected with an optimized DroidBot, with complete per-app traces. - https://awesomecua.com/resources/url-cc4aa8caeee0cff58e73/ · source: https://huggingface.co/datasets/mllmTeam/MobileViews - Dataset · reviewed 2026-10-16 · environments: android · topics: screen-understanding - Licenses: data mit ### MONDAY Cross-platform mobile GUI trajectory dataset derived from instructional videos, with released data-processing helpers and example evaluation code for Android and iOS data. - https://awesomecua.com/resources/github-runamu-monday/ · source: https://github.com/runamu/monday - Dataset · reviewed 2026-10-18 · environments: android, ios, cross-platform · topics: data-generation, mobile-navigation, video - Licenses: code Apache-2.0 ### MONDAY annotations Video-derived mobile navigation annotations that pair scene boundaries and screen or element boxes with action labels for training and evaluating GUI agents. - https://awesomecua.com/resources/url-570654ff67ecefa2b955/ · source: https://huggingface.co/datasets/runamu/MONDAY - Dataset · reviewed 2026-10-17 · environments: android, ios · topics: data-generation, mobile-navigation, video, imitation-learning ### MoTIF Mobile task demonstrations and natural-language instructions. - https://awesomecua.com/resources/github-aburns4-motif/ · source: https://github.com/aburns4/MoTIF - Dataset · reviewed 2026-10-17 · environments: android · topics: mobile-navigation ### MT-Mind2Web Multi-turn web-navigation conversations constructed from Mind2Web interactions, with dialogue turns, HTML snapshots, element candidates and click/type/select action sequences. - https://awesomecua.com/resources/url-bdfb50330deb97893ce4/ · source: https://huggingface.co/datasets/magicgh/MT-Mind2Web - Dataset · reviewed 2026-10-17 · environments: web · topics: web-navigation, imitation-learning, memory - Licenses: data cc-by-4.0 ### MultiUI 7.3 million training samples built from webpage UIs and structured with enhanced accessibility trees, for text-rich visual understanding, released with UIX-Qwen2 models. - https://awesomecua.com/resources/github-neulab-multiui/ · source: https://github.com/neulab/multiui - Dataset · reviewed 2026-10-17 · environments: web · topics: screen-understanding, data-generation ### NNetNav Live data Browser-interaction data released with NNetNav for its live-website setting, with separate training and test files. - https://awesomecua.com/resources/url-4f47abde8cd28a1ec1e1/ · source: https://huggingface.co/datasets/stanfordnlp/nnetnav-live - Dataset · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation - Licenses: data apache-2.0 ### NNetNav WebArena data NNetNav's WebArena-oriented browser-interaction data release, with separate training and test files. - https://awesomecua.com/resources/url-25a4d0ab29b60250863a/ · source: https://huggingface.co/datasets/stanfordnlp/nnetnav-wa - Dataset · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation - Licenses: data apache-2.0 ### OpenMobile-Data-v2 Mobile-agent supervision from emulator and simulator trajectories, plus executable tasks with starting states and success checks for reinforcement learning. - https://awesomecua.com/resources/url-546e6c95825d1318c98d/ · source: https://huggingface.co/datasets/OpenMobile-2/OpenMobile-Data-v2 - Dataset · reviewed 2026-10-17 · environments: android, web · topics: mobile-navigation, data-generation, reinforcement-learning, tool-integration ### OS-Atlas Data GUI grounding pre-training data for OS-Atlas in desktop (Linux, macOS, Windows), mobile and web domains, with element boxes stored as relative left, top, right and bottom ratios. - https://awesomecua.com/resources/url-0fdeb1cf5f49f84820e6/ · source: https://huggingface.co/datasets/OS-Copilot/OS-Atlas-data - Dataset · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, data-generation - Licenses: data apache-2.0 ### OS-Genesis Interaction-driven pipeline that synthesizes mobile and web GUI agent trajectories without human supervision or predefined tasks, via reverse task synthesis and a trajectory reward model. - https://awesomecua.com/resources/github-os-copilot-os-genesis/ · source: https://github.com/OS-Copilot/OS-Genesis - Dataset · reviewed 2026-10-17 · environments: cross-platform · topics: data-generation ### OS-Genesis mobile training data Reverse-task-synthesis mobile trajectories released in separate AndroidControl and AndroidWorld training annotation files. - https://awesomecua.com/resources/url-d3ec5122a2b52fd5ae4b/ · source: https://huggingface.co/datasets/OS-Copilot/OS-Genesis-mobile-data - Dataset · reviewed 2026-10-17 · environments: android · topics: data-generation, mobile-navigation, imitation-learning - Licenses: data mit ### PC-Agent-E training data Released dataset used to train PC-Agent-E, linked by the official pipeline for refining human trajectories and synthesizing action reasoning. - https://awesomecua.com/resources/url-14449fe321e0a1ca34ea/ · source: https://huggingface.co/datasets/henryhe0123/PC-Agent-E - Dataset · reviewed 2026-10-18 · environments: desktop, windows, linux · topics: data-generation, imitation-learning - Licenses: data MIT ### Plan-and-Act training data Planner and action-executor training examples for WebArena-Lite and WebVoyager, with model-specific processed files and raw trajectory annotations. - https://awesomecua.com/resources/url-ba6453cfdae0eb049ced/ · source: https://huggingface.co/datasets/xTRam1/plan-and-act-data - Dataset · reviewed 2026-10-17 · environments: web · topics: planning, web-navigation, data-generation - Licenses: data mit ### Rico Mobile app screenshots, view hierarchies, and interaction traces for UI research. - https://awesomecua.com/resources/url-0b134b082a7bd507fc6a/ · source: https://www.interactionmining.org/archive/rico - Dataset · reviewed 2026-10-17 · environments: android · topics: screen-understanding ### RoTS Announced GUI trajectory synthesis resources for recovery from policy-induced errors; the inspected repository marks code and data releases as in preparation. - https://awesomecua.com/resources/github-alibabaresearch-rots/ · source: https://github.com/AlibabaResearch/RoTS - Dataset · reviewed 2026-10-17 · environments: cross-platform · topics: data-generation, robustness ### ScaleCUA Data Cross-platform training data released with ScaleCUA, with grounding and navigation annotations for Ubuntu, Windows, macOS, Android, iPhone and web screenshots. - https://awesomecua.com/resources/url-631e74223e198ef40202/ · source: https://huggingface.co/datasets/OpenGVLab/ScaleCUA-Data - Dataset · reviewed 2026-10-17 · environments: cross-platform · topics: data-generation ### ScienceBoard evaluation trajectories Archives of ScienceBoard agent evaluation runs, separated by agent configuration and screenshot, accessibility-tree or set-of-marks observation mode. - https://awesomecua.com/resources/url-4784f63cd7117f64a2a3/ · source: https://huggingface.co/datasets/OS-Copilot/ScienceBoard-Traj - Dataset · reviewed 2026-10-17 · environments: desktop, linux · topics: evaluation, science, workflow ### Screen2Words Mobile screen descriptions for language-based screen understanding. - https://awesomecua.com/resources/github-google-research-datasets-screen2words/ · source: https://github.com/google-research-datasets/screen2words - Dataset · reviewed 2026-10-16 · environments: android · topics: screen-understanding ### ScreenDrag GUI dragging data with screen observations, prior state, and continuous pointer actions, provided as trajectory files and Parquet representations. - https://awesomecua.com/resources/url-fe43af1801032be708cc/ · source: https://huggingface.co/datasets/h-siyuan/ScreenDrag - Dataset · reviewed 2026-10-16 · environments: desktop · topics: grounding, evaluation ### ScreenQA About 86,000 human-written question-answer pairs over 35,000 Rico mobile screenshots, plus ScreenQA Short and ComplexQA variants, for benchmarking screen content understanding. - https://awesomecua.com/resources/github-google-research-datasets-screen-qa/ · source: https://github.com/google-research-datasets/screen_qa - Dataset · reviewed 2026-10-16 · environments: android · topics: screen-understanding ### ScreenRef GUI screenshots paired with localization instructions and targets, including absent-target examples; image access is gated under publisher terms. - https://awesomecua.com/resources/url-56bda04bd8d3a438deb3/ · source: https://huggingface.co/datasets/PrentisAI/ScreenRef - Dataset · reviewed 2026-10-17 · environments: desktop, web, android, linux · topics: grounding, data-generation, screen-understanding - Licenses: data other ### ScreenRef Annotations Annotation files, task recipes, and image manifests for ScreenRef; screenshots reside in the separately gated image repository. - https://awesomecua.com/resources/url-c1db2ba79e6d6a3db2b9/ · source: https://huggingface.co/datasets/PrentisAI/ScreenRef-Annotations - Dataset · reviewed 2026-10-17 · environments: desktop, web, android, linux · topics: grounding, data-generation - Licenses: data other ### ScreenSpot-Pro data High-resolution professional interface screenshots and target annotations for evaluating instruction-based GUI localization. - https://awesomecua.com/resources/url-c79e152e6f7af1a9efed/ · source: https://huggingface.co/datasets/likaixin/ScreenSpot-Pro - Dataset · reviewed 2026-10-16 · environments: desktop, windows, macos, linux · topics: grounding, evaluation - Licenses: data mit ### ShowUI Desktop 7,496 desktop grounding examples built from OmniAct screenshots, with annotations rewritten by GPT-4o to describe element appearance, spatial relations and function. - https://awesomecua.com/resources/url-f976dec3d1c5e0bedbd7/ · source: https://huggingface.co/datasets/showlab/ShowUI-desktop-8K - Dataset · reviewed 2026-10-17 · environments: desktop · topics: data-generation ### ShowUI-web Web-interface grounding examples with instructions, element boxes and points, paired with a separately packaged screenshot archive. - https://awesomecua.com/resources/url-8933977bb0a57940ed23/ · source: https://huggingface.co/datasets/showlab/ShowUI-web - Dataset · reviewed 2026-10-17 · environments: web · topics: grounding, data-generation ### ShowUI-π training data Drag-and-click episode data for the ShowUI-π model, spanning desktop editing, presentation, handwriting and other interface tasks. - https://awesomecua.com/resources/url-3dab53c6c38c9e7217ea/ · source: https://huggingface.co/datasets/showlab/ShowUI-pi-data - Dataset · reviewed 2026-10-17 · environments: desktop · topics: grounding, action-prediction, imitation-learning ### UGround V1 Data Gated web grounding training data for UGround in Web-Hybrid and Web-Direct subsets, with a bounding-box version whose coordinates are normalized to 0-999. - https://awesomecua.com/resources/url-c4898ad1bfc049926a63/ · source: https://huggingface.co/datasets/osunlp/UGround-V1-Data - Dataset · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, data-generation - Licenses: data cc-by-nc-sa-4.0 ### UI-AGILE-Data UI-AGILE training annotations and screenshots together with converted GUI-grounding and AndroidControl evaluation files. - https://awesomecua.com/resources/url-41abb1b7206689cf5057/ · source: https://huggingface.co/datasets/KDEGroup/UI-AGILE-Data - Dataset · reviewed 2026-10-17 · environments: android · topics: grounding, reinforcement-learning, evaluation ### UI-Genie reward data Mobile GUI reward annotations released for training UI-Genie’s trajectory-assessment model. - https://awesomecua.com/resources/url-84e5aeeffaf561baa966/ · source: https://huggingface.co/datasets/HanXiao1999/UI-Genie-RM-517k - Dataset · reviewed 2026-10-17 · environments: android · topics: data-generation, mobile-navigation, reward-models - Licenses: data mit ### UI-Genie trajectory data Synthetic mobile GUI trajectory supervision and screenshots released with the UI-Genie agent-training pipeline. - https://awesomecua.com/resources/url-2e1927450da2183a5abe/ · source: https://huggingface.co/datasets/HanXiao1999/UI-Genie-Agent-16k - Dataset · reviewed 2026-10-17 · environments: android · topics: data-generation, mobile-navigation, imitation-learning - Licenses: data mit ### UI-Ins Treats grounding instructions as multiple reasoning perspectives, releasing a data pipeline, training code and UI-Ins-7B and 32B models trained by SFT on synthesized instructions, then RL. - https://awesomecua.com/resources/github-alibaba-ui-ins/ · source: https://github.com/alibaba/UI-Ins - Dataset · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation ### UI-R1 3B training data The released mobile training subset for the original UI-R1 3B model, with action annotations and individual screenshot files. - https://awesomecua.com/resources/url-7c568d88a472f914d316/ · source: https://huggingface.co/datasets/LZXzju/UI-R1-3B-Train - Dataset · reviewed 2026-10-17 · environments: android · topics: data-generation, grounding, action-prediction ### UI-S1 released training examples A public example subset of the UI-S1 mobile training data, packaged as JSONL annotations and Parquet images. - https://awesomecua.com/resources/url-16184830c0f4bdb146fa/ · source: https://huggingface.co/datasets/mPLUG/UI_S1_dataset - Dataset · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, data-generation, mobile-navigation - Licenses: data apache-2.0 ### UI-Vision Desktop GUI data for interface understanding and interaction research. - https://awesomecua.com/resources/url-2c2004baae4669f8169a/ · source: https://huggingface.co/datasets/ServiceNow/ui-vision - Dataset · reviewed 2026-10-17 · environments: desktop · topics: grounding, screen-understanding - Licenses: data mit ### UIBert AppSim and RefExp annotations Two Rico-derived UI component retrieval datasets: AppSim pairs semantically similar elements across screens, while RefExp links referring expressions to target elements. - https://awesomecua.com/resources/github-google-research-datasets-uibert/ · source: https://github.com/google-research-datasets/uibert - Dataset · reviewed 2026-10-17 · environments: android · topics: grounding, screen-understanding, evaluation - Licenses: data CC-BY-4.0 ### UICrit Mobile-interface critiques and quality ratings with region coordinates and RICO screen references, including a larger three-annotator release than the original paper dataset. - https://awesomecua.com/resources/github-google-research-datasets-uicrit/ · source: https://github.com/google-research-datasets/uicrit - Dataset · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, ui-design, evaluation ### VideoCAD data CAD screen recordings paired with action logs and target model images for studying precise interface control and long-horizon visual imitation. - https://awesomecua.com/resources/url-2b5dead3172c0b70da7e/ · source: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/WX8PCK - Dataset · reviewed 2026-10-18 · environments: web · topics: grounding, imitation-learning, video - Licenses: data CC0 1.0 ### Wave UI Screenshot-instruction records with target boxes and UI metadata such as OCR, source, element type and platform. - https://awesomecua.com/resources/url-0dbb55f127da04d95222/ · source: https://huggingface.co/datasets/agentsea/wave-ui - Dataset · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding - Licenses: data Wave UI Dataset License; per-image source terms retained ### WebEvolver SFT data Two JSONL collections for supervised fine-tuning the WebEvolver agent policy and web world model. - https://awesomecua.com/resources/url-e126bbd9729ebdaf05ab/ · source: https://huggingface.co/datasets/CognitiveKernel/WebEvolver - Dataset · reviewed 2026-10-18 · environments: web · topics: web-navigation, world-models, imitation-learning - Licenses: data MIT ### WebPRM checklist-generation data Web-task context, starting observations and action history paired with structured checklists and subgoal analyses for checklist generation. - https://awesomecua.com/resources/url-fcc98b149272bbf31b05/ · source: https://huggingface.co/datasets/LangAGI-Lab/WebPRMCollection_checklist_generation - Dataset · reviewed 2026-10-17 · environments: web · topics: reward-models, data-generation, web-navigation ### WebPRM Collection preference data Checklist-annotated web decision records with page observations, prior actions, one preferred continuation and rejected alternatives for process-reward learning. - https://awesomecua.com/resources/url-8dfc84eb4016204fe70a/ · source: https://huggingface.co/datasets/LangAGI-Lab/WebPRMCollection_preference_pair - Dataset · reviewed 2026-10-17 · environments: web · topics: web-navigation, reward-models, evaluation ### WebSRC v1.0 Webpage segments, questions, answers, HTML, screenshots and layout metadata for structural reading comprehension. - https://awesomecua.com/resources/url-e0ba328ae1a643b0a5aa/ · source: https://huggingface.co/datasets/X-LANCE/WebSRC_v1.0 - Dataset · reviewed 2026-10-18 · environments: web · topics: screen-understanding, grounding - Licenses: data CC-BY-4.0 ### WebUI Web screenshot and interface data for UI research. - https://awesomecua.com/resources/url-13b9442bcb6a8b984c4f/ · source: https://huggingface.co/datasets/biglab/webui-all - Dataset · reviewed 2026-10-16 · environments: web · topics: screen-understanding - Licenses: data other ### WorkDemo-350 Desktop demonstrations with task descriptions, videos, input logs, and action-linked screenshot evidence for studying reusable agent skills. - https://awesomecua.com/resources/url-243ab00712ba19cc2089/ · source: https://huggingface.co/datasets/yuqi777/WorkDemo-350 - Dataset · reviewed 2026-10-16 · environments: desktop · topics: recording, workflow, video - Licenses: data cc-by-4.0 ## Benchmarks (91) ### Adversarial Web Agents Evaluation resources for adversarial attacks on multimodal web agents. - https://awesomecua.com/resources/github-chenwu98-agent-attack/ · source: https://github.com/ChenWu98/agent-attack - Benchmark · reviewed 2026-10-16 · environments: web · topics: safety-security, robustness ### AgentBench Agent evaluation suite with containerized tasks and a function-calling interface, including operating-system and shopping environments. - https://awesomecua.com/resources/github-thudm-agentbench/ · source: https://github.com/THUDM/AgentBench - Benchmark · reviewed 2026-10-16 · environments: cross-platform, web · topics: evaluation, tool-integration - Licenses: code Apache-2.0 ### AgentDAM Web-agent benchmark for measuring unnecessary use of sensitive personal information during end-to-end task execution. - https://awesomecua.com/resources/github-facebookresearch-ai-agent-privacy/ · source: https://github.com/facebookresearch/ai-agent-privacy - Benchmark · reviewed 2026-10-16 · environments: web · topics: safety-security, evaluation ### AgentRewardBench A benchmark and evaluation toolkit for comparing automatic web-agent judges against expert judgments of task success, side effects and repetition. - https://awesomecua.com/resources/github-mcgill-nlp-agent-reward-bench/ · source: https://github.com/McGill-NLP/agent-reward-bench - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, reward-models, web-navigation - Licenses: data Custom AgentRewardBench Terms of Use ### AndroidArena Android environment for evaluating language-model-driven agents. - https://awesomecua.com/resources/github-androidarenaagent-androidarena/ · source: https://github.com/AndroidArenaAgent/AndroidArena - Benchmark · reviewed 2026-10-16 · environments: android · topics: evaluation, mobile-navigation ### AndroidLab Reproducible Android benchmark of 138 tasks across nine apps on predefined virtual devices, run via AVD on Mac or Docker on Linux, released with the Android Instruct training set. - https://awesomecua.com/resources/github-thudm-android-lab/ · source: https://github.com/THUDM/Android-Lab - Benchmark · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation - Licenses: code MIT ### AndroidWorld Android environment with programmatic task evaluation for mobile agents. - https://awesomecua.com/resources/github-google-research-android-world/ · source: https://github.com/google-research/android_world - Benchmark · reviewed 2026-10-16 · environments: android · topics: evaluation, mobile-navigation - Licenses: code Apache-2.0 ### AppSim-Bench Simulated mobile applications and evaluation tooling for reproducible GUI-agent tasks with controlled application state. - https://awesomecua.com/resources/github-acrab-agentic-labs-appsim/ · source: https://github.com/Acrab-Agentic-Labs/AppSim - Benchmark · reviewed 2026-10-16 · environments: android · topics: evaluation, sandbox ### AppWorld Simulated applications, user data, and state-based task checks for evaluating agents that interact through APIs and code. - https://awesomecua.com/resources/github-stonybrooknlp-appworld/ · source: https://github.com/StonyBrookNLP/appworld - Benchmark · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, tool-integration, cross-app - Licenses: code Apache-2.0 ### B-MoCA Mobile control benchmark with varied interface configurations. - https://awesomecua.com/resources/github-jylee425-b-moca/ · source: https://github.com/jylee425/b-moca - Benchmark · reviewed 2026-10-16 · environments: android · topics: evaluation, robustness - Licenses: code Apache-2.0 ### BrowserART Red-teaming benchmark and test suite from Scale AI evaluating whether refusal-trained language models can be jailbroken when deployed as browser agents. - https://awesomecua.com/resources/github-scaleapi-browser-art/ · source: https://github.com/scaleapi/browser-art - Benchmark · reviewed 2026-10-18 · environments: web · topics: safety-security, evaluation, web-navigation ### BrowserGym Unified interfaces for browser-agent tasks and benchmark environments. - https://awesomecua.com/resources/github-servicenow-browsergym/ · source: https://github.com/ServiceNow/BrowserGym - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation - Licenses: code Apache-2.0 ### ClawBench Everyday-task benchmark on live websites that runs browser agents in containerized Chromium, blocks each final submission request for scoring by URL match and an LLM judge, and records five layers of run evidence. - https://awesomecua.com/resources/github-tiger-ai-lab-clawbench/ · source: https://github.com/TIGER-AI-Lab/ClawBench - Benchmark · reviewed 2026-10-18 · environments: web · topics: evaluation, web-navigation, workflow, observability - Licenses: code Apache-2.0 ### ColorBench A graph-based mobile GUI benchmark that evaluates long tasks through screenshot states and multiple valid or erroneous transitions, including reflection, replanning and backtracking. - https://awesomecua.com/resources/github-madeagents-colorbench/ · source: https://github.com/MadeAgents/ColorBench - Benchmark · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation, long-horizon, cross-app - Licenses: code Apache-2.0; data apache-2.0 ### CRAB Cross-environment benchmark framework for multimodal agents. - https://awesomecua.com/resources/github-camel-ai-crab/ · source: https://github.com/camel-ai/crab - Benchmark · reviewed 2026-10-16 · environments: desktop, android · topics: evaluation, cross-app ### CUAVerifierBench Human-annotated browser-agent trajectories for comparing automated task-success and process-quality verifiers against reviewer judgments. - https://awesomecua.com/resources/url-d5140d438466c1e1348a/ · source: https://huggingface.co/datasets/microsoft/CUAVerifierBench - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, reward-models, web-navigation - Licenses: data mit ### EIA Evaluation suite for Environmental Injection Attacks that embed invisible malicious HTML elements into benign websites to test privacy leakage in generalist web agents. - https://awesomecua.com/resources/github-osu-nlp-group-eia-against-webagent/ · source: https://github.com/OSU-NLP-Group/EIA_against_webagent - Benchmark · reviewed 2026-10-18 · environments: web · topics: safety-security, evaluation, web-navigation - Licenses: code MIT ### FedMABench A federated mobile-agent training and evaluation suite that varies user-data heterogeneity and application distribution, with released partition annotations and inference/scoring scripts. - https://awesomecua.com/resources/github-wwh0411-fedmabench/ · source: https://github.com/wwh0411/FedMABench - Benchmark · reviewed 2026-10-17 · environments: android · topics: evaluation, imitation-learning, mobile-navigation ### GUI-Robust Evaluation scripts and linked data for comparing GUI action predictions and task execution under ordinary and anomalous interface conditions. - https://awesomecua.com/resources/github-chessbean1-gui-robust/ · source: https://github.com/chessbean1/GUI-Robust - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, robustness ### GUIHalluBench Measures hallucination in UI-element parsing by matching the boxes and descriptions a model lists for 1,000 English and 1,000 Chinese screenshots against human annotations. - https://awesomecua.com/resources/github-ant-research-guihallubench/ · source: https://github.com/ant-research/GUIHalluBench - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, robustness ### iOSWorld Native iPhone-simulator benchmark with shared user data, application workflows, agent runners, and rubric-based scoring. - https://awesomecua.com/resources/github-ljang0-iosworld/ · source: https://github.com/ljang0/iOSWorld - Benchmark · reviewed 2026-10-16 · environments: ios · topics: evaluation, cross-app, memory - Licenses: code Apache-2.0 ### LlamaTouch Mobile UI task evaluation using application state information. - https://awesomecua.com/resources/github-llamatouch-llamatouch/ · source: https://github.com/LlamaTouch/LlamaTouch - Benchmark · reviewed 2026-10-17 · environments: android · topics: evaluation ### MacAgentBench 676 macOS tasks across 25 applications, each run in its own Docker container and scored by deterministic rule-based checks with multiple checkpoints. - https://awesomecua.com/resources/github-jetastra-macagentbench/ · source: https://github.com/JetAstra/MacAgentBench - Benchmark · reviewed 2026-10-16 · environments: desktop, macos · topics: evaluation - Licenses: code MIT ### MCPWorld Evaluation environment combining computer interaction with tool access. - https://awesomecua.com/resources/github-saagent-mcpworld/ · source: https://github.com/SAAgent/MCPWorld - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, tool-integration - Licenses: code MIT ### Mind2Web Human demonstrations and tasks for generalist web navigation research. - https://awesomecua.com/resources/github-osu-nlp-group-mind2web/ · source: https://github.com/OSU-NLP-Group/Mind2Web - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation - Licenses: code MIT ### Mind2Web 2 Long-horizon agentic web search benchmark whose Agent-as-a-Judge framework scores time-varying, citation-backed answers, with evaluation scripts for its dev and test sets. - https://awesomecua.com/resources/github-osu-nlp-group-mind2web-2/ · source: https://github.com/OSU-NLP-Group/Mind2Web-2 - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, long-horizon - Licenses: code MIT ### MiniWoB++ Over 100 web-interaction environments extending OpenAI's MiniWoB, controlled from Python through a Gymnasium API that drives the browser with Selenium. - https://awesomecua.com/resources/github-farama-foundation-miniwob-plusplus/ · source: https://github.com/Farama-Foundation/miniwob-plusplus - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, reinforcement-learning - Licenses: code MIT ### MLA-Trust Unified evaluation framework measuring truthfulness, controllability, safety, and privacy of multimodal LLM agents across interactive web and mobile GUI tasks. - https://awesomecua.com/resources/github-thu-ml-mla-trust/ · source: https://github.com/thu-ml/MLA-Trust - Benchmark · reviewed 2026-10-18 · environments: web, android, cross-platform · topics: safety-security, evaluation, robustness - Licenses: code MIT ### MMBench-GUI Four-level GUI-agent benchmark across six platforms, from content understanding and element grounding to task automation and collaboration, scored with an efficiency-quality area metric. - https://awesomecua.com/resources/github-open-compass-mmbench-gui/ · source: https://github.com/open-compass/MMBench-GUI - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, grounding ### MMInA Multimodal web tasks in a WebArena-based environment that chain from 1 to 10 hops across sites such as a shopping store and Wikipedia, testing long-chain reasoning. - https://awesomecua.com/resources/github-shulin16-mmina/ · source: https://github.com/shulin16/MMInA - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation, long-horizon ### MobBench task definitions A public set of real-device mobile task definitions used to evaluate MobA, with Chinese and English task material packaged in a versioned JSON release. - https://awesomecua.com/resources/url-b53e03a6e4ca4226d126/ · source: https://huggingface.co/datasets/OpenDFM/MobA-MobBench - Benchmark · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation - Licenses: data cc-by-nc-sa-4.0 ### Mobile-Bench-v2 released data Offline mobile-agent task data spanning common tasks and released noisy, contaminated and ambiguous-instruction subsets. - https://awesomecua.com/resources/url-1d5b85321b98765141fc/ · source: https://huggingface.co/datasets/xwk123/MobileBench-v2 - Benchmark · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation, robustness ### Mobile-Eval-E task definitions Task definitions for Mobile-Agent-E’s long-horizon, multi-app Android evaluation, packaged as a 25-record test split with rubrics and human-reference operation strings. - https://awesomecua.com/resources/url-342b081c3668a8bbe155/ · source: https://huggingface.co/datasets/mikewang/mobile_eval_e - Benchmark · reviewed 2026-10-18 · environments: android · topics: evaluation, long-horizon, cross-app ### MobileAgentBench Automated benchmark for mobile LLM agents whose default tasks use ten Simple Mobile Tools Android apps, installed as a Python library that detects task completion. - https://awesomecua.com/resources/github-mobileagentbench-mobile-agent-bench/ · source: https://github.com/MobileAgentBench/mobile-agent-bench - Benchmark · reviewed 2026-10-16 · environments: android · topics: evaluation, mobile-navigation ### MobileGUI-VBench Mobile-app task definitions covering everyday scenarios, with annotated preconditions and clarification requirements for evaluating agent behavior. - https://awesomecua.com/resources/url-d7a4f34a6a5202af5f64/ · source: https://huggingface.co/datasets/vivo-ai/MobileGUI-VBench - Benchmark · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, cross-app ### MobileGym A browser-hosted mobile-app simulator with programmable state, parameterized tasks and deterministic state-checking judges for parallel GUI-agent evaluation and online reinforcement learning. - https://awesomecua.com/resources/github-purewhiter-mobilegym/ · source: https://github.com/Purewhiter/mobilegym - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, mobile-navigation, reinforcement-learning, sandbox - Licenses: code Apache-2.0; data CC-BY-NC-4.0 ### MobileGym++ / MobileGym++ Bench Browser-rendered mobile simulation with resettable app state and a shared task suite for GUI-only and GUI-plus-tool agents. - https://awesomecua.com/resources/github-openmobile2-mobilegym-plusplus/ · source: https://github.com/OpenMobile2/MobileGym-plusplus - Benchmark · reviewed 2026-10-17 · environments: web · topics: mobile-navigation, evaluation, cross-app, tool-integration - Licenses: code Apache-2.0 ### MobileSafetyBench Android emulator testbed that scores mobile device-control agents on both safety and helpfulness, using ADB and Appium for observations, actions and success checks. - https://awesomecua.com/resources/github-jylee425-mobilesafetybench/ · source: https://github.com/jylee425/mobilesafetybench - Benchmark · reviewed 2026-10-16 · environments: android · topics: safety-security, evaluation - Licenses: code Apache-2.0 ### MobileWorld (Tongyi-MAI) A containerized Android evaluation environment with GUI-only, agent-user interaction and MCP-augmented tasks, plus checks on application and backend state. - https://awesomecua.com/resources/github-tongyi-mai-mobileworld/ · source: https://github.com/Tongyi-MAI/MobileWorld - Benchmark · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation, human-in-the-loop, tool-integration - Licenses: code Apache-2.0 ### MobileWorldBench Benchmark of semantic world modeling for mobile agents, with question-answering tasks and GPT-judged generation tasks, plus training data built on AitW and AndroidControl screens. - https://awesomecua.com/resources/github-jacklishufan-mobileworld/ · source: https://github.com/jacklishufan/MobileWorld - Benchmark · reviewed 2026-10-16 · environments: android · topics: evaluation, mobile-navigation ### OmniAct Dataset and benchmark for generating computer interaction programs. - https://awesomecua.com/resources/url-849a86beaee006c6b147/ · source: https://huggingface.co/datasets/Writer/omniact - Benchmark · reviewed 2026-10-16 · environments: desktop, web · topics: evaluation - Licenses: data mit ### Online Mind2Web 300 tasks on 136 live websites for evaluating web agents online, with outdated tasks replaced over time, human-reviewed submissions and the WebJudge automatic evaluator. - https://awesomecua.com/resources/github-osu-nlp-group-online-mind2web/ · source: https://github.com/OSU-NLP-Group/Online-Mind2Web - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation - Licenses: code MIT ### OpenApps Configurable browser apps and task rewards for testing computer-use agents under changes in content, layout and appearance. - https://awesomecua.com/resources/github-facebookresearch-openapps/ · source: https://github.com/facebookresearch/OpenApps - Benchmark · reviewed 2026-10-18 · environments: web · topics: evaluation, robustness, cross-app, sandbox - Licenses: code CC-BY-NC-4.0 ### OS-Critic Bench A cross-platform GUI-critic benchmark with mobile, desktop and web JSONL splits and screenshot assets, accompanied by public inference and accuracy-scoring scripts. - https://awesomecua.com/resources/url-c73ba0618d1b3c505887/ · source: https://huggingface.co/datasets/OS-Copilot/OS-Critic-Bench - Benchmark · reviewed 2026-10-17 · environments: cross-platform, desktop, web · topics: evaluation, reward-models, screen-understanding - Licenses: data apache-2.0 ### OS-Harm Benchmark for safety of computer-use agents across misuse, prompt injection, and unintended model behavior. - https://awesomecua.com/resources/github-tml-epfl-os-harm/ · source: https://github.com/tml-epfl/os-harm - Benchmark · reviewed 2026-10-16 · environments: desktop · topics: safety-security, evaluation - Licenses: code Apache-2.0 ### OS-Map An OSWorld-derived desktop benchmark with an Ubuntu virtual-machine environment and expanded task configurations for measuring breadth and depth of computer interaction. - https://awesomecua.com/resources/github-os-copilot-os-map/ · source: https://github.com/OS-Copilot/OS-Map - Benchmark · reviewed 2026-10-17 · environments: desktop, linux · topics: evaluation, long-horizon, cross-app - Licenses: code MIT ### OSWorld Execution-based evaluation of agents performing tasks in real desktop environments. - https://awesomecua.com/resources/github-xlang-ai-osworld/ · source: https://github.com/xlang-ai/OSWorld - Benchmark · reviewed 2026-10-16 · environments: desktop, linux, windows, macos · topics: evaluation, cross-app - Licenses: code Apache-2.0 ### OSWorld-G Computer-use grounding benchmark with original and refined instructions, released with the Jedi synthetic data pipeline, a 4-million-example Jedi dataset and Jedi-3B and Jedi-7B models. - https://awesomecua.com/resources/github-xlang-ai-osworld-g/ · source: https://github.com/xlang-ai/OSWorld-G - Benchmark · reviewed 2026-10-16 · environments: desktop · topics: evaluation, grounding - Licenses: code Apache-2.0 ### OSWorld-Human Human reference trajectories and scoring code for measuring the action efficiency of computer-use agents on OSWorld tasks. - https://awesomecua.com/resources/github-wuklab-osworld-human/ · source: https://github.com/WukLab/osworld-human - Benchmark · reviewed 2026-10-17 · environments: desktop · topics: evaluation, efficiency, workflow ### PopupAttack Resources for evaluating GUI agents under malicious popup instructions. - https://awesomecua.com/resources/github-salt-nlp-popupattack/ · source: https://github.com/SALT-NLP/PopupAttack - Benchmark · reviewed 2026-10-16 · environments: web · topics: safety-security ### PPTC PowerPoint creation and editing benchmark with API-based execution and checks of the resulting presentation files. - https://awesomecua.com/resources/github-gydpku-pptc/ · source: https://github.com/gydpku/PPTC - Benchmark · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, tool-integration - Licenses: code MIT ### Precision CUA Bench Evaluation resources for pixel-precise cursor grounding in code editors. - https://awesomecua.com/resources/github-microsoft-precision-cua-bench/ · source: https://github.com/microsoft/precision-cua-bench - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, grounding - Licenses: code MIT ### RecreationBench Tasks for recreating reference applications with hybrid computer interaction. - https://awesomecua.com/resources/url-d9d8e68f19ef0956d189/ · source: https://huggingface.co/datasets/Qwen/RecreationBench - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, tool-integration ### RiOSWorld Measures risky behavior of multimodal computer-use agents in an Ubuntu virtual machine, with risk scenarios such as phishing emails and released evaluation trajectories. - https://awesomecua.com/resources/github-yjyddq-riosworld/ · source: https://github.com/yjyddq/RiOSWorld - Benchmark · reviewed 2026-10-16 · environments: desktop, linux · topics: evaluation ### SaaS-Bench 106 browser-driven business workflows across 23 self-hosted SaaS apps in six domains, scored by per-task verifiers that read each app's database and give partial credit. - https://awesomecua.com/resources/github-unipat-ai-saas-bench/ · source: https://github.com/UniPat-AI/SaaS-Bench - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation, workflow - Licenses: code Apache-2.0 ### SafeArena Web-agent benchmark pairing benign and harmful tasks to study deliberate misuse risks. - https://awesomecua.com/resources/github-mcgill-nlp-safearena/ · source: https://github.com/McGill-NLP/safearena - Benchmark · reviewed 2026-10-16 · environments: web · topics: safety-security, evaluation ### ScienceBoard Scientific-workflow tasks for multimodal agents in desktop software such as ChimeraX, KAlgebra, Celestia, Grass GIS and Lean 4, with apps modified to expose state for evaluation. - https://awesomecua.com/resources/github-os-copilot-scienceboard/ · source: https://github.com/OS-Copilot/ScienceBoard - Benchmark · reviewed 2026-10-16 · environments: desktop · topics: evaluation, science - Licenses: code MIT ### Screen Point-and-Read / Tree-of-Lens Screen-reading benchmark and Tree-of-Lens implementation for interpreting user-indicated regions and their layout context. - https://awesomecua.com/resources/github-ucsb-ai-screen-point-and-read/ · source: https://github.com/UCSB-AI/Screen-Point-and-Read - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: screen-understanding, grounding, evaluation - Licenses: code MIT ### ScreenSpot GUI grounding test set of 1,272 screenshot instructions from iOS, Android, macOS, Windows and web interfaces, each labeled with the target element's bounding box. - https://awesomecua.com/resources/url-7abb418096432f56c685/ · source: https://huggingface.co/datasets/rootsautomation/ScreenSpot - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, evaluation - Licenses: data apache-2.0 ### ScreenSpot Pro GUI grounding benchmark focused on professional high-resolution interfaces. - https://awesomecua.com/resources/github-likaixin2000-screenspot-pro-gui-grounding/ · source: https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding - Benchmark · reviewed 2026-10-16 · environments: desktop, windows, macos, linux · topics: evaluation, grounding - Licenses: code MIT ### ScreenSpot v2 Second version of the ScreenSpot grounding test set, published by OS-Copilot as separate desktop, mobile and web annotation files with a shared screenshot archive. - https://awesomecua.com/resources/url-bd7214de6d98917c2350/ · source: https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2 - Benchmark · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, evaluation - Licenses: data apache-2.0 ### ScreenSpot-v2 instruction variants Instruction variants over ScreenSpot-v2 screenshots that distinguish original commands, explicit actions, target descriptions and impossible operations. - https://awesomecua.com/resources/url-17729e1c532ba7043404/ · source: https://huggingface.co/datasets/likaixin/ScreenSpot-v2-variants - Benchmark · reviewed 2026-10-17 · environments: cross-platform · topics: grounding, evaluation, robustness ### SPA-Bench 340 single-app and cross-app tasks over 66 Chinese and English Android apps, with plug-in support for 11 smartphone agents and automatic coarse-to-fine success detection. - https://awesomecua.com/resources/github-ai-agents-2030-spa-bench/ · source: https://github.com/ai-agents-2030/SPA-Bench - Benchmark · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation - Licenses: code MIT ### Spider2-V Visual agent evaluation on data-science and engineering workflows. - https://awesomecua.com/resources/github-xlang-ai-spider2-v/ · source: https://github.com/xlang-ai/Spider2-V - Benchmark · reviewed 2026-10-17 · environments: web, desktop · topics: evaluation, workflow - Licenses: code Apache-2.0 ### ST-WebAgentBench BrowserGym-based benchmark for evaluating the safety and policy compliance of enterprise web agents across six trustworthiness risk dimensions. - https://awesomecua.com/resources/github-segev-shlomov-st-webagentbench/ · source: https://github.com/segev-shlomov/ST-WebAgentBench - Benchmark · reviewed 2026-10-18 · environments: web · topics: safety-security, evaluation, web-navigation ### TheAgentCompany Self-contained workplace environment for evaluating agents on professional tasks involving websites, files, code, and simulated coworkers. - https://awesomecua.com/resources/github-theagentcompany-theagentcompany/ · source: https://github.com/TheAgentCompany/TheAgentCompany - Benchmark · reviewed 2026-10-16 · environments: web, desktop · topics: evaluation, workflow, cross-app - Licenses: code MIT ### VeriWeb (formerly VeriGUI) Verifiable web information-seeking benchmark renamed from VeriGUI; desktop evaluation is described as future work. - https://awesomecua.com/resources/github-verigui-team-verigui/ · source: https://github.com/VeriGUI-Team/VeriGUI - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation - Licenses: code Apache-2.0 ### VideoGUI Benchmark for desktop GUI tasks derived from instructional videos. - https://awesomecua.com/resources/github-showlab-videogui/ · source: https://github.com/showlab/videogui - Benchmark · reviewed 2026-10-16 · environments: desktop · topics: evaluation, video ### VisualAgentBench Evaluation framework spanning visual GUI, embodied, and design tasks, including mobile and WebArena-based environments. - https://awesomecua.com/resources/github-thudm-visualagentbench/ · source: https://github.com/THUDM/VisualAgentBench - Benchmark · reviewed 2026-10-16 · environments: cross-platform, web, android · topics: evaluation - Licenses: code Apache-2.0 ### VisualWebArena Visually grounded web tasks extending the WebArena evaluation setting. - https://awesomecua.com/resources/github-web-arena-x-visualwebarena/ · source: https://github.com/web-arena-x/visualwebarena - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation, screen-understanding ### VisualWebBench Evaluation of multimodal models on visual web understanding tasks. - https://awesomecua.com/resources/github-visualwebbench-visualwebbench/ · source: https://github.com/VisualWebBench/VisualWebBench - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation, screen-understanding ### WeaveBench Computer-use benchmark mixing graphical interaction and code execution. - https://awesomecua.com/resources/url-6a212bc12f82c824f6b9/ · source: https://huggingface.co/datasets/wanlilll/WeaveBench - Benchmark · reviewed 2026-10-16 · environments: desktop, linux · topics: evaluation, tool-integration - Licenses: data mit ### Web Context Bench (Python) Local-page benchmark comparing six browser representations on token use, factual reading and target-element selection, with deterministic answer scoring. - https://awesomecua.com/resources/github-charlypoly-web-context-bench-python/ · source: https://github.com/charlypoly/web-context-bench-python - Benchmark · reviewed 2026-10-20 · environments: web · topics: evaluation, grounding, efficiency, screen-understanding - Licenses: code MIT ### WebArena Self-hosted web environments and tasks for evaluating autonomous web agents. - https://awesomecua.com/resources/github-web-arena-x-webarena/ · source: https://github.com/web-arena-x/webarena - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation - Licenses: code Apache-2.0 ### WebCanvas Framework and evaluation resources for real-world web agents. - https://awesomecua.com/resources/github-imeanai-webcanvas/ · source: https://github.com/iMeanAI/WebCanvas - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation - Licenses: code MIT ### WebClick A web-interface localization benchmark with screenshot instructions and target boxes, including agent browsing, human browsing and calendar cases. - https://awesomecua.com/resources/url-5e2bd1b3398974b79c9c/ · source: https://huggingface.co/datasets/Hcompany/WebClick - Benchmark · reviewed 2026-10-17 · environments: web · topics: grounding, evaluation, web-navigation ### WebGames 150 client-side web challenges in five capability domains, built to be easy for humans but hard for browsing agents, each revealing a password when solved. - https://awesomecua.com/resources/github-convergence-ai-webgames/ · source: https://github.com/convergence-ai/webgames - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation - Licenses: code Apache-2.0 ### WebGuard Codebase, annotation toolkit, and action-level risk dataset for training and evaluating guardrail models that flag unsafe browser-agent actions before execution. - https://awesomecua.com/resources/github-osu-nlp-group-webguard/ · source: https://github.com/OSU-NLP-Group/WebGuard - Benchmark · reviewed 2026-10-18 · environments: web · topics: safety-security, evaluation, web-navigation - Licenses: code MIT ### WebLINX Demonstrations and evaluation for conversational web navigation. - https://awesomecua.com/resources/github-mcgill-nlp-weblinx/ · source: https://github.com/mcgill-nlp/weblinx - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation - Licenses: code Apache-2.0 ### WebRewardBench An offline preference-evaluation set for web process reward models, pairing page and trajectory context with preferred and rejected candidate decisions. - https://awesomecua.com/resources/url-ac6d0f80765f5a32255a/ · source: https://huggingface.co/datasets/LangAGI-Lab/WebRewardBench - Benchmark · reviewed 2026-10-17 · environments: web · topics: web-navigation, reward-models, evaluation ### WebShop Simulated shopping environment for instruction-following web agents. - https://awesomecua.com/resources/github-princeton-nlp-webshop/ · source: https://github.com/princeton-nlp/WebShop - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation - Licenses: code MIT ### WebsiteBench Website reconstruction benchmark: agents explore a production-level site through a browser, rebuild a full-stack version, and are scored by hidden functional and visual test cases across 70 tasks. - https://awesomecua.com/resources/github-tiger-ai-lab-websitebench/ · source: https://github.com/TIGER-AI-Lab/WebsiteBench - Benchmark · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, long-horizon, ui-design - Licenses: code Apache-2.0 ### WebTailBench Live-web task definitions and rubrics for evaluating computer-use agents, with preserved original tasks, a refreshed V2 task suite and a separate refusal set. - https://awesomecua.com/resources/url-3b6dfd52f98bfe76bc7f/ · source: https://huggingface.co/datasets/microsoft/WebTailBench - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, safety-security - Licenses: data mit ### WebTestBench An end-to-end web-testing benchmark where agents derive test checklists from development instructions, interact with deployed applications and report functional or constraint-related defects. - https://awesomecua.com/resources/github-friedrichor-webtestbench/ · source: https://github.com/friedrichor/WebTestBench - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, tool-integration - Licenses: code Apache-2.0; data apache-2.0 ### WebVoyager Multimodal web navigation agent and evaluation tasks on real websites. - https://awesomecua.com/resources/github-minorjerry-webvoyager/ · source: https://github.com/MinorJerry/WebVoyager - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation - Licenses: code Apache-2.0 ### Windows Agent Arena Windows 11 benchmark for multimodal desktop agents in a reproducible VM that runs locally or in parallel on Azure ML, with a harder mode where agents must set tasks up themselves. - https://awesomecua.com/resources/github-microsoft-windowsagentarena/ · source: https://github.com/microsoft/WindowsAgentArena - Benchmark · reviewed 2026-10-17 · environments: windows, desktop · topics: evaluation - Licenses: code MIT ### WindowsAgentArena-V2 Windows benchmark revision with task/evaluator fixes and VM-initialization support, documented alongside PC-Agent-E. - https://awesomecua.com/resources/github-gair-nlp-windowsagentarena-v2/ · source: https://github.com/GAIR-NLP/WindowsAgentArena-V2 - Benchmark · reviewed 2026-10-18 · environments: desktop, windows · topics: evaluation, cross-app, robustness - Licenses: code MIT ### WindowsWorld Cross-application Windows benchmark with intermediate task evaluation. - https://awesomecua.com/resources/github-hitsz-tmg-windowsworld/ · source: https://github.com/HITsz-TMG/WindowsWorld - Benchmark · reviewed 2026-10-16 · environments: desktop, windows · topics: evaluation, cross-app - Licenses: code Apache-2.0 ### WONDERBREAD An offline benchmark of workflow documentation, quality assessment and knowledge transfer using human web-navigation demonstrations, action traces and written procedures. - https://awesomecua.com/resources/github-hazyresearch-wonderbread/ · source: https://github.com/HazyResearch/wonderbread - Benchmark · reviewed 2026-10-17 · environments: web · topics: evaluation, workflow, video, screen-understanding - Licenses: data cc-by-4.0 ### WorkArena Browser-based enterprise workflow tasks built around ServiceNow. - https://awesomecua.com/resources/github-servicenow-workarena/ · source: https://github.com/ServiceNow/WorkArena - Benchmark · reviewed 2026-10-16 · environments: web · topics: evaluation, workflow - Licenses: code Apache-2.0 ### WorldGUI 611 desktop GUI tasks that agents must complete from varied starting points, shipped with instructional videos and project files, plus the self-reflective WorldGUI-Agent baseline. - https://awesomecua.com/resources/github-showlab-worldgui/ · source: https://github.com/showlab/WorldGUI - Benchmark · reviewed 2026-10-16 · environments: desktop · topics: evaluation, robustness ## Agents (49) ### Agent E Browser automation agent on the AG2 framework that pairs a planner agent with a browser navigation agent to fill forms, search shops and work through sites, also callable over a local HTTP API. - https://awesomecua.com/resources/github-emergenceai-agent-e/ · source: https://github.com/EmergenceAI/Agent-E - Agent · reviewed 2026-10-16 · environments: web · topics: web-navigation - Licenses: code MIT ### Agent S Simular's open-source framework for computer-use agents that click, type and scroll through desktop and web apps on macOS, Windows and Linux; its third generation, Agent S3, reports 72.6% on OSWorld. - https://awesomecua.com/resources/github-simular-ai-agent-s/ · source: https://github.com/simular-ai/Agent-S - Agent · reviewed 2026-10-16 · environments: desktop · topics: planning, memory - Licenses: code Apache-2.0 ### Agent Zero Open-source agent with its own sandboxed Linux workspace that researches, writes code and works with documents using a terminal, a browser and desktop apps. - https://awesomecua.com/resources/github-agent0ai-agent-zero/ · source: https://github.com/agent0ai/agent-zero - Agent · reviewed 2026-11-06 · environments: desktop, linux, web · topics: automation, sandbox, human-in-the-loop - Licenses: code MIT ### AppAgent Multimodal agent that operates Android apps through taps and swipes without back-end access, building a reusable knowledge base from autonomous exploration or from watching human demonstrations. - https://awesomecua.com/resources/github-tencentqqgylab-appagent/ · source: https://github.com/TencentQQGYLab/AppAgent - Agent · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, memory - Licenses: code MIT ### AppAgent Pro Proactive Android agent built on AppAgent that uses GPT-4o to decide whether YouTube or Amazon content would help, operates those apps over ADB, and merges what it finds into its answer. - https://awesomecua.com/resources/github-laokuize-appagent-pro/ · source: https://github.com/LaoKuiZe/AppAgent-Pro - Agent · reviewed 2026-10-16 · environments: android · topics: mobile-navigation - Licenses: code MIT ### AppAgentX Mobile agent research using accumulated interaction experience. - https://awesomecua.com/resources/github-westlake-agi-lab-appagentx/ · source: https://github.com/Westlake-AGI-Lab/AppAgentX - Agent · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, memory ### AssistGUI LLM-based assistant for automating work in desktop productivity software by watching the screen and issuing mouse and keyboard commands; its repository says the code will be released later. - https://awesomecua.com/resources/github-showlab-assistgui/ · source: https://github.com/showlab/assistgui - Agent · reviewed 2026-10-16 · environments: desktop · topics: planning ### Auto-GUI / Auto-UI Screenshot-based mobile agent implementation with training and inference code for chain-of-action prediction. - https://awesomecua.com/resources/github-cooelf-auto-gui/ · source: https://github.com/cooelf/Auto-GUI - Agent · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, planning - Licenses: code Apache-2.0 ### AutoDroid LLM-powered Android task automation built on DroidBot, released with DroidTask, a dataset of per-app screenshots, view hierarchies, exploration records and ground-truth task files. - https://awesomecua.com/resources/github-mobilellm-autodroid/ · source: https://github.com/MobileLLM/AutoDroid - Agent · reviewed 2026-10-16 · environments: android · topics: mobile-navigation ### AutoDroid V2 An Android automation implementation that builds app documentation from exploration traces and evaluates language-model-generated execution code in emulator workflows. - https://awesomecua.com/resources/github-mobilellm-autodroid-v2/ · source: https://github.com/MobileLLM/AutoDroid-V2 - Agent · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, tool-integration, data-generation ### AutoWebGLM Web navigation agent built on ChatGLM3-6B that simplifies HTML for the model, is trained on hybrid human and AI data with RL and rejection sampling, and comes with the bilingual AutoWebBench. - https://awesomecua.com/resources/github-thudm-autowebglm/ · source: https://github.com/THUDM/AutoWebGLM - Agent · reviewed 2026-10-16 · environments: web · topics: web-navigation - Licenses: code Apache-2.0 ### BrowserGPT BrowserGPT is a browser automation script that uses GPT-4 to generate Playwright code for navigating and interacting with web pages from natural-language tasks. - https://awesomecua.com/resources/github-mayt-browsergpt/ · source: https://github.com/mayt/BrowserGPT - Agent · reviewed 2026-11-06 · environments: web · topics: automation, web-navigation, grounding, tool-integration - Licenses: code MIT ### Bytebot Open-source desktop agent that works inside its own Ubuntu Linux virtual desktop, with a web task interface, REST APIs for tasks and direct control, a takeover mode and password-manager logins. - https://awesomecua.com/resources/github-bytebot-ai-bytebot/ · source: https://github.com/bytebot-ai/bytebot - Agent · reviewed 2026-10-16 · environments: desktop · topics: workflow - Licenses: code Apache-2.0 ### CAAP Agent Screenshot-driven agent implementation for MiniWoB++ and WebShop, using a visual observer and context-aware prompts to issue mouse and keyboard actions. - https://awesomecua.com/resources/github-caap-agent-caap-agent/ · source: https://github.com/caap-agent/caap-agent - Agent · reviewed 2026-10-18 · environments: web · topics: planning, screen-understanding, web-navigation - Licenses: code AGPL-3.0 ### CoCo-Agent Research implementation combining screen and layout perception with conditional action prediction for smartphone automation. - https://awesomecua.com/resources/github-xbmxb-coco-agent/ · source: https://github.com/xbmxb/CoCo-Agent - Agent · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, screen-understanding ### COLA demo Research demonstration of collaborative Windows GUI automation with dynamic task scheduling and recovery mechanisms. - https://awesomecua.com/resources/github-alokia-cola-demo/ · source: https://github.com/Alokia/COLA-demo - Agent · reviewed 2026-10-16 · environments: desktop, windows · topics: automation, planning, human-in-the-loop ### Cradle Framework in which foundation models control games and software from screenshots alone, acting through keyboard and mouse, shown on RDR2, Stardew Valley, Cities: Skylines and Dealer's Life 2. - https://awesomecua.com/resources/github-baai-agents-cradle/ · source: https://github.com/BAAI-Agents/Cradle - Agent · reviewed 2026-10-16 · environments: desktop · topics: planning, memory - Licenses: code MIT ### LearnAct Released inference and evaluation code for demonstration-conditioned mobile GUI agents on LearnGUI-Offline. The repository describes a multi-agent adaptation method and links the LearnGUI demonstrations. - https://awesomecua.com/resources/github-lgy0404-learnact/ · source: https://github.com/lgy0404/LearnAct - Agent · reviewed 2026-10-18 · environments: unspecified · topics: mobile-navigation, imitation-learning, evaluation ### Lumen browser agent A screenshot-driven browser agent over Chrome DevTools Protocol, with resumable sessions, context compression, action replay and configurable completion checks. - https://awesomecua.com/resources/github-omxyz-lumen/ · source: https://github.com/omxyz/lumen - Agent · reviewed 2026-10-17 · environments: web · topics: web-navigation, memory, automation, observability - Licenses: code MIT ### MagenticLite / Magentic UI Microsoft agent application combining orchestration with browser interaction; successor to Magentic UI. - https://awesomecua.com/resources/github-microsoft-magentic-ui/ · source: https://github.com/microsoft/magentic-ui - Agent · reviewed 2026-10-16 · environments: web · topics: human-in-the-loop - Licenses: code MIT ### MobA An Android agent that combines global task planning and memory with a local action executor, using reflection to revise mobile interaction steps. - https://awesomecua.com/resources/github-opendfm-moba/ · source: https://github.com/OpenDFM/MobA - Agent · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, memory, planning, self-reflection ### Mobile Use Android GUI agent that operates real phones through ADB with hierarchical reflection and a multi-agent framework, run from a web interface or Python and evaluated on AndroidWorld and AndroidLab. - https://awesomecua.com/resources/github-madeagents-mobile-use/ · source: https://github.com/MadeAgents/mobile-use - Agent · reviewed 2026-10-16 · environments: android · topics: mobile-navigation - Licenses: code MIT ### Mobile-Agent Mobile device agent family using visual interface interaction. - https://awesomecua.com/resources/github-x-plug-mobileagent/ · source: https://github.com/X-PLUG/MobileAgent - Agent · reviewed 2026-10-16 · environments: android, cross-platform · topics: mobile-navigation, planning - Licenses: code MIT ### Mobile-Agent-E code component Android automation component with hierarchical planning, reflection and persistent tips/shortcuts learned across task sequences. - https://awesomecua.com/resources/component-x-plug-mobileagent-mobile-agent-e/ · source: https://github.com/X-PLUG/MobileAgent/tree/main/Mobile-Agent-E - Agent · reviewed 2026-10-18 · environments: android · topics: planning, memory, self-reflection, mobile-navigation - Licenses: code MIT ### MobileAgent (Alipay) Mobile-control research implementation combining operating-procedure context with human interaction for personalized task execution. - https://awesomecua.com/resources/github-alipay-mobile-agent/ · source: https://github.com/alipay/mobile-agent - Agent · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, workflow, human-in-the-loop ### MobileGPT An Android client and Python-server agent that learns reusable app subtasks through exploration, selection, derivation and memory recall. - https://awesomecua.com/resources/github-fclab-skku-mobilegpt/ · source: https://github.com/fclab-skku/MobileGPT - Agent · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, memory, planning, workflow - Licenses: code Apache-2.0 ### Mobster engine iOS agent engine organized around an observe, decide, verify and act loop, using WebDriverAgent screen state and phone actions such as taps, typing, swipes and app switches. - https://awesomecua.com/resources/github-radishsoftware-mobster/ · source: https://github.com/RadishSoftware/mobster - Agent · reviewed 2026-10-20 · environments: ios · topics: mobile-navigation, evaluation - Licenses: code MIT ### Nanobrowser A browser-extension agent that reads pages, interacts with elements and moves between tabs from a side-panel task interface. - https://awesomecua.com/resources/github-nanobrowser-nanobrowser/ · source: https://github.com/nanobrowser/nanobrowser - Agent · reviewed 2026-10-17 · environments: web · topics: web-navigation, automation, human-in-the-loop - Licenses: code Apache-2.0 ### Open Interpreter Assistant framework combining code execution and computer interaction. - https://awesomecua.com/resources/github-openinterpreter-open-interpreter/ · source: https://github.com/OpenInterpreter/open-interpreter - Agent · reviewed 2026-10-16 · environments: desktop · topics: tool-integration - Licenses: code Apache-2.0 ### Open-AutoGLM Phone agent model and framework with device interaction tooling. - https://awesomecua.com/resources/github-zai-org-open-autoglm/ · source: https://github.com/zai-org/Open-AutoGLM - Agent · reviewed 2026-10-16 · environments: android, ios · topics: mobile-navigation - Licenses: code Apache-2.0 ### OpenAdapt Tools for learning and automating workflows from desktop demonstrations. - https://awesomecua.com/resources/github-openadaptai-openadapt/ · source: https://github.com/OpenAdaptAI/OpenAdapt - Agent · reviewed 2026-10-16 · environments: desktop · topics: recording, workflow - Licenses: code MIT ### OpenAgents Web Agent The OpenAgents browser-control component, connecting a Chrome extension to a web agent and local chat platform for navigating sites and completing interface tasks. - https://awesomecua.com/resources/component-xlang-ai-openagents-web-agent/ · source: https://github.com/xlang-ai/OpenAgents/tree/main/real_agents/web_agent - Agent · reviewed 2026-10-17 · environments: web · topics: web-navigation, automation, tool-integration - Licenses: code Apache-2.0 ### OS-Copilot Open-source library for generalist agents acting on an operating system's web, terminals, files and third-party apps, with the FRIDAY assistant, which takes custom tools and self-learns Excel tasks. - https://awesomecua.com/resources/github-os-copilot-os-copilot/ · source: https://github.com/OS-Copilot/OS-Copilot - Agent · reviewed 2026-10-16 · environments: desktop · topics: planning, tool-integration - Licenses: code MIT ### OS-Kairos Android GUI-agent research implementation that uses confidence-guided interaction to choose between autonomous actions and human assistance. - https://awesomecua.com/resources/github-wuzheng02-os-kairos/ · source: https://github.com/Wuzheng02/OS-Kairos - Agent · reviewed 2026-10-16 · environments: android · topics: human-in-the-loop, safety-security, mobile-navigation ### PC-Agent Human-cognition-transfer framework for PC agents: PC Tracker logs human computer use, cognition completion turns logs into cognitive trajectories, and a planner works with a grounding agent. - https://awesomecua.com/resources/github-gair-nlp-pc-agent/ · source: https://github.com/GAIR-NLP/PC-Agent - Agent · reviewed 2026-10-16 · environments: desktop · topics: planning, cross-app - Licenses: code MIT ### PC-Agent (X-PLUG component) Multi-agent desktop automation component for Windows and macOS, combining active perception, task decomposition and reflection for workflows spanning applications. - https://awesomecua.com/resources/component-x-plug-mobileagent-pc-agent/ · source: https://github.com/X-PLUG/MobileAgent/tree/main/PC-Agent - Agent · reviewed 2026-10-18 · environments: desktop, windows, macos · topics: planning, cross-app, self-reflection - Licenses: code MIT ### Qwen Code Qwen Code is an open-source coding agent that supports desktop automation through computer-use tools alongside terminal, editor, web, and chat interfaces. - https://awesomecua.com/resources/github-qwenlm-qwen-code/ · source: https://github.com/QwenLM/qwen-code - Agent · reviewed 2026-11-06 · environments: desktop · topics: automation, tool-integration, workflow - Licenses: code Apache-2.0 ### RCI Agent A MiniWoB++ agent that uses explicit and implicit recursive criticism-and-improvement prompts to refine action plans and ground actions to browser state. - https://awesomecua.com/resources/github-posgnu-rci-agent/ · source: https://github.com/posgnu/rci-agent - Agent · reviewed 2026-10-17 · environments: web · topics: web-navigation, planning, self-reflection, training-free - Licenses: code MIT ### ScreenAgent Environment and agent in which a vision-language model controls real desktops over VNC with mouse and keyboard in plan, act and reflect stages, plus a hand-annotated dataset of daily computer tasks. - https://awesomecua.com/resources/github-niuzaisheng-screenagent/ · source: https://github.com/niuzaisheng/ScreenAgent - Agent · reviewed 2026-10-16 · environments: desktop · topics: planning ### SeeAct Generalist web agent framework for large multimodal models such as GPT-4V, with a Playwright tool that runs agents on live websites, text-choice and set-of-mark grounding, and a crawler mode. - https://awesomecua.com/resources/github-osu-nlp-group-seeact/ · source: https://github.com/OSU-NLP-Group/SeeAct - Agent · reviewed 2026-10-16 · environments: web · topics: web-navigation, grounding ### Self Operating Computer Framework, released in November 2023, in which a multimodal model views the screen and picks mouse and keyboard actions toward a goal, with backends such as GPT-4o, Claude 3, Gemini and LLaVA. - https://awesomecua.com/resources/github-othersideai-self-operating-computer/ · source: https://github.com/OthersideAI/self-operating-computer - Agent · reviewed 2026-10-16 · environments: desktop · topics: grounding - Licenses: code MIT ### SheetCopilot Spreadsheet automation agent for Windows/Excel, with benchmark assets and a separately released headless Ubuntu evaluation backend. - https://awesomecua.com/resources/github-bravegroup-sheetcopilot/ · source: https://github.com/BraveGroup/SheetCopilot - Agent · reviewed 2026-10-18 · environments: desktop, windows, linux · topics: workflow, tool-integration, evaluation ### ShowUI-Aloha Desktop agent that records demonstrations, derives semantic action traces, and uses them to plan and execute related workflows. - https://awesomecua.com/resources/github-showlab-showui-aloha/ · source: https://github.com/showlab/ShowUI-Aloha - Agent · reviewed 2026-10-16 · environments: desktop, windows, macos · topics: recording, workflow, automation - Licenses: code Apache-2.0 ### Surf New Next.js and FastAPI playground from Steel for trying web agents in a controlled browser environment, with live views of browser interaction and support for providers such as Claude and GPT-4. - https://awesomecua.com/resources/github-steel-dev-surf-new/ · source: https://github.com/steel-dev/surf.new - Agent · reviewed 2026-10-16 · environments: web · topics: web-navigation - Licenses: code MIT ### TestZeus Hercules Open-source testing agent that takes Gherkin features and executes them end to end on web applications through Playwright. - https://awesomecua.com/resources/github-test-zeus-ai-testzeus-hercules/ · source: https://github.com/test-zeus-ai/testzeus-hercules - Agent · reviewed 2026-11-06 · environments: web · topics: automation, evaluation, web-navigation - Licenses: code AGPL-3.0 ### UFO Windows agent framework for application and desktop automation. - https://awesomecua.com/resources/github-microsoft-ufo/ · source: https://github.com/microsoft/UFO - Agent · reviewed 2026-10-16 · environments: windows, desktop · topics: planning, cross-app ### UI-TARS Desktop Desktop app for a local GUI agent driven by UI-TARS and Seed-1.5-VL/1.6 models, offering local and remote operators for computers and browsers; the repository also ships the Agent TARS CLI and web UI. - https://awesomecua.com/resources/github-bytedance-ui-tars-desktop/ · source: https://github.com/bytedance/UI-TARS-desktop - Agent · reviewed 2026-10-16 · environments: desktop, web · topics: workflow - Licenses: code Apache-2.0 ### VisionTasker An Android automation implementation that translates screenshot-based UI understanding into language descriptions for step-by-step LLM planning and action execution. - https://awesomecua.com/resources/github-akimotoayako-visiontasker/ · source: https://github.com/AkimotoAyako/VisionTasker - Agent · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, screen-understanding, planning - Licenses: code GPL-3.0 ### Yunkai Android phone assistant combining model-driven tool use with accessibility-tree reading, screenshots, and tap, swipe and text-entry primitives. Its documented interaction flow presents operation plans for user confirmation. - https://awesomecua.com/resources/github-solplum-yunkai/ · source: https://github.com/solplum/yunkai - Agent · reviewed 2026-10-20 · environments: android · topics: mobile-navigation, screen-understanding, human-in-the-loop - Licenses: code MIT ## Frameworks (87) ### AdvAgent (formerly AdvWeb) Research code for constructing and evaluating adversarial inputs to web agents, with a black-box red-team pipeline using Mind2Web data and SeeAct. - https://awesomecua.com/resources/github-ai-secure-advagent/ · source: https://github.com/AI-secure/AdvAgent - Framework · reviewed 2026-10-17 · environments: web · topics: safety-security, robustness, evaluation ### Agent Workflow Memory implementation Research code that extracts reusable browser-task routines from demonstrations or prior executions, stores them as workflow memory and reuses them during subsequent navigation. - https://awesomecua.com/resources/github-zorazrw-agent-workflow-memory/ · source: https://github.com/zorazrw/agent-workflow-memory - Framework · reviewed 2026-10-17 · environments: web · topics: workflow, memory, web-navigation - Licenses: code Apache-2.0 ### Agent-R Trajectory collection and revision utilities that use tree search and model-guided critique to train agents to recover from mistakes, including WebShop tasks. - https://awesomecua.com/resources/github-bytedance-seed-agent-r/ · source: https://github.com/ByteDance-Seed/Agent-R - Framework · reviewed 2026-10-17 · environments: web · topics: self-reflection, data-generation, web-navigation - Licenses: code Apache-2.0 ### AgentLab Experiment framework for developing and evaluating browser agents. - https://awesomecua.com/resources/github-servicenow-agentlab/ · source: https://github.com/ServiceNow/AgentLab - Framework · reviewed 2026-10-16 · environments: web · topics: evaluation - Licenses: code Apache-2.0 ### AgentProg Android GUI-agent framework that keeps long-running task context in structured programs, variables and control flow, with a CLI and AndroidWorld integration. - https://awesomecua.com/resources/github-mobilellm-agentprog/ · source: https://github.com/MobileLLM/AgentProg - Framework · reviewed 2026-10-18 · environments: android · topics: long-horizon, memory, planning, state-tracking - Licenses: code Apache-2.0 ### AgentStudio Toolkit and Ubuntu-based environment for building and evaluating generalist computer-use agents through GUI and API interaction. - https://awesomecua.com/resources/github-skyworkai-agent-studio/ · source: https://github.com/skyworkai/agent-studio - Framework · reviewed 2026-10-18 · environments: desktop, linux · topics: sandbox, evaluation, grounding, recording - Licenses: code AGPL-3.0 ### AgentSynth task-generation pipeline Generates synthetic computer-use tasks and records trajectories in OSWorld desktop environments and InSTA browser environments. - https://awesomecua.com/resources/github-sunblaze-ucb-agentsynth/ · source: https://github.com/sunblaze-ucb/AgentSynth - Framework · reviewed 2026-10-18 · environments: desktop, web · topics: data-generation, long-horizon, automation - Licenses: code Apache-2.0 ### Alumnium An AI-native end-to-end testing library and MCP that lets agents interact with applications through automation drivers and verify assertions. - https://awesomecua.com/resources/github-alumnium-hq-alumnium/ · source: https://github.com/alumnium-hq/alumnium - Framework · reviewed 2026-11-06 · environments: web · topics: automation, tool-integration, workflow ### AndroTMem inference and evaluation code A public Android GUI evaluation framework with action-history, summary and anchored-memory prompt modes, plus step-action and dependency-aware task scoring. - https://awesomecua.com/resources/github-cvc2233-androtmem/ · source: https://github.com/CVC2233/AndroTMem - Framework · reviewed 2026-10-17 · environments: android · topics: memory, long-horizon, evaluation - Licenses: code MIT ### ARPO training framework Replay-based policy-optimization framework for GUI agents, with distributed OSWorld rollouts and screenshot/action histories. - https://awesomecua.com/resources/github-jia-lab-research-arpo/ · source: https://github.com/JIA-Lab-research/ARPO - Framework · reviewed 2026-10-18 · environments: desktop, linux · topics: reinforcement-learning, long-horizon, automation - Licenses: code Apache-2.0 ### Browser Use Open-source browser agent library that runs locally with any LLM against a local or cloud browser, and also serves as a CLI for coding agents and as a driver for Claude's browser toolset. - https://awesomecua.com/resources/github-browser-use-browser-use/ · source: https://github.com/browser-use/browser-use - Framework · reviewed 2026-10-16 · environments: web · topics: web-navigation - Licenses: code MIT ### Capybara Capybara is a web application acceptance testing framework that simulates user interactions through browser and headless drivers. - https://awesomecua.com/resources/github-teamcapybara-capybara/ · source: https://github.com/teamcapybara/capybara - Framework · reviewed 2026-11-06 · environments: web · topics: automation, web-navigation, evaluation ### CHOP Mobile automation framework that uses reusable basis subtasks to constrain and refine planning between user goals and executable actions. - https://awesomecua.com/resources/github-yuqi-zhou-chop/ · source: https://github.com/Yuqi-Zhou/CHOP - Framework · reviewed 2026-10-16 · environments: android, harmonyos · topics: mobile-navigation, planning, workflow - Licenses: code MIT ### ClawGUI Framework for GUI agent training, evaluation, and deployment. - https://awesomecua.com/resources/github-zju-real-clawgui/ · source: https://github.com/ZJU-REAL/ClawGUI - Framework · reviewed 2026-10-16 · environments: cross-platform · topics: reinforcement-learning, evaluation - Licenses: code Apache-2.0 ### CoAT research implementation Research scripts that generate screen descriptions, action reasoning and expected outcomes, then predict Android GUI actions using the resulting Chain-of-Action-Thought context. - https://awesomecua.com/resources/github-imnearth-coat/ · source: https://github.com/IMNearth/CoAT - Framework · reviewed 2026-10-17 · environments: android · topics: planning, mobile-navigation, action-prediction ### CodeceptJS CodeceptJS is an end-to-end testing framework that expresses browser and mobile interactions as user-facing steps through helpers such as Playwright, WebDriver, and Appium. - https://awesomecua.com/resources/github-codeceptjs-codeceptjs/ · source: https://github.com/codeceptjs/CodeceptJS - Framework · reviewed 2026-11-06 · environments: web, android · topics: automation, tool-integration, workflow ### Computer Use OOTB Gradio desktop GUI agent for Windows and macOS that needs no Docker, using Claude 3.5 Computer Use over the API or ShowUI and UI-TARS run locally, with multi-display and phone remote control. - https://awesomecua.com/resources/github-showlab-computer-use-ootb/ · source: https://github.com/showlab/computer_use_ootb - Framework · reviewed 2026-10-16 · environments: desktop · topics: workflow - Licenses: code Apache-2.0 ### ComputerRL Reinforcement-learning infrastructure for computer-use agents. - https://awesomecua.com/resources/github-thudm-computerrl/ · source: https://github.com/THUDM/ComputerRL - Framework · reviewed 2026-10-16 · environments: desktop · topics: reinforcement-learning - Licenses: code Apache-2.0 ### ConnectOnion ConnectOnion is a CLI and Python framework that gives agents commands for connected accounts, a logged-in browser, files, and chat apps. - https://awesomecua.com/resources/github-openonion-connectonion/ · source: https://github.com/openonion/connectonion - Framework · reviewed 2026-11-06 · environments: web · topics: tool-integration, automation, memory, human-in-the-loop - Licenses: code Apache-2.0 ### Cua Monorepo behind Cua Spaces, which gives agents full macOS or Linux desktops, plus Cua Driver for operating native apps, Lume for Apple Silicon VMs, Cua Bench and the CUA-S1 decision models. - https://awesomecua.com/resources/github-trycua-cua/ · source: https://github.com/trycua/cua - Framework · reviewed 2026-10-16 · environments: desktop · topics: evaluation, workflow - Licenses: code MIT ### CUAWright (formerly Webwright) Agent framework for browser and desktop tasks using automation code. - https://awesomecua.com/resources/github-microsoft-webwright/ · source: https://github.com/microsoft/Webwright - Framework · reviewed 2026-10-16 · environments: web, desktop · topics: tool-integration - Licenses: code MIT ### D-PoT evaluation code Offline prompting and evaluation code for Dynamic Planning of Thoughts on Android-in-the-Wild, including action extraction and comparison with reference actions. - https://awesomecua.com/resources/github-sqzhang-lazy-d-pot/ · source: https://github.com/sqzhang-lazy/D-PoT - Framework · reviewed 2026-10-18 · environments: android · topics: planning, action-prediction, evaluation ### Digi-Q Offline device-control training code that learns action values and uses them to select promising actions, with linked trajectories and checkpoints. - https://awesomecua.com/resources/github-digirl-agent-digiq/ · source: https://github.com/DigiRL-agent/digiq - Framework · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, efficiency ### DigiRL Reinforcement learning for autonomous mobile device control. - https://awesomecua.com/resources/github-digirl-agent-digirl/ · source: https://github.com/DigiRL-agent/digirl - Framework · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning - Licenses: code Apache-2.0 ### DistRL Distributed online reinforcement-learning framework for Android agents, with environment setup and trajectory collection tools. - https://awesomecua.com/resources/github-distrl-lab-distrl-open/ · source: https://github.com/DistRL-lab/distrl-open - Framework · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, efficiency - Licenses: code Apache-2.0 ### DroidCall generation and evaluation code Data-generation, fine-tuning and evaluation utilities for teaching small language models to invoke Android intents through predefined functions. - https://awesomecua.com/resources/github-ubiquitouslearning-droidcall/ · source: https://github.com/UbiquitousLearning/DroidCall - Framework · reviewed 2026-10-17 · environments: android · topics: tool-integration, data-generation, evaluation - Licenses: code MIT ### Go-Browse Structured website-exploration and data-processing code that revisits discovered pages, proposes tasks and collects trajectories for web-agent fine-tuning. - https://awesomecua.com/resources/github-apga-go-browse/ · source: https://github.com/ApGa/Go-Browse - Framework · reviewed 2026-10-17 · environments: web · topics: web-navigation, data-generation, imitation-learning - Licenses: code MIT ### GroundGUI training and evaluation code Code for screenshot grounding through supervised training, teacher-guided distillation, reinforcement learning, and benchmark evaluation. - https://awesomecua.com/resources/github-prentisai-groundgui/ · source: https://github.com/PrentisAI/GroundGUI - Framework · reviewed 2026-10-17 · environments: desktop, web, cross-platform · topics: grounding, reinforcement-learning, data-generation - Licenses: code Apache-2.0 ### GUI-CIDER research code Released data-synthesis and evaluation scripts that distill planning and causal GUI knowledge into text for mid-training mobile interface agents. - https://awesomecua.com/resources/github-wuzheng02-gui-cider/ · source: https://github.com/Wuzheng02/GUI-CIDER - Framework · reviewed 2026-10-17 · environments: android · topics: data-generation, imitation-learning, planning ### GUI-HARVEST Optimizes GUI-agent execution code from repeated rollout evidence and checks proposed changes against fixed validation gates. - https://awesomecua.com/resources/github-garyyang12345-gui-harvest/ · source: https://github.com/GaryYang12345/GUI-HARVEST - Framework · reviewed 2026-10-16 · environments: desktop · topics: automation, observability, robustness - Licenses: code Apache-2.0 ### GUI-RC / GUI-RCPO GUI-grounding research code for spatial voting across sampled predictions and test-time policy optimization using region-consistency rewards. - https://awesomecua.com/resources/github-zju-real-gui-rcpo/ · source: https://github.com/ZJU-REAL/GUI-RCPO - Framework · reviewed 2026-10-17 · environments: cross-platform, desktop, web · topics: grounding, reinforcement-learning, efficiency ### GUI-ReWalk A desktop trajectory-generation framework that moves from stochastic GUI exploration to inferred goals and multi-stage task completion, with retrospective reasoning annotations. - https://awesomecua.com/resources/github-bytedance-gui-rewalk/ · source: https://github.com/bytedance/GUI-ReWalk - Framework · reviewed 2026-10-17 · environments: desktop, linux · topics: data-generation, planning, cross-app - Licenses: code Apache-2.0 ### GUIPivot Query-inference training tools connecting screenshot grounding with GUI action reasoning, including preprocessing, query refinement and offline single-step evaluation on AndroidControl and AITZ. - https://awesomecua.com/resources/github-zrw00-guipivot/ · source: https://github.com/ZrW00/GUIPivot - Framework · reviewed 2026-10-18 · environments: android · topics: grounding, planning, imitation-learning - Licenses: code GPL-3.0 ### HUD SDK for RL environments and evals spanning coding, browser, computer-use and robotics agents, whose environments expose SSH, MCP, CDP or VNC capabilities behind a minimal start-and-grade protocol. - https://awesomecua.com/resources/github-hud-evals-hud-python/ · source: https://github.com/hud-evals/hud-python - Framework · reviewed 2026-10-16 · environments: web, desktop · topics: evaluation, reinforcement-learning - Licenses: code MIT ### HyperAgent Library adding natural-language commands to Playwright: accessibility-tree single actions, multi-step tasks that can use screenshots, an MCP client and action caching for replay without LLM calls. - https://awesomecua.com/resources/github-hyperbrowserai-hyperagent/ · source: https://github.com/hyperbrowserai/HyperAgent - Framework · reviewed 2026-10-16 · environments: web · topics: web-navigation ### LaVague Open-source web agent framework where a World Model turns an objective and the current page into instructions and an Action Engine compiles them into Selenium or Playwright code to run. - https://awesomecua.com/resources/github-lavague-ai-lavague/ · source: https://github.com/lavague-ai/LaVague - Framework · reviewed 2026-10-16 · environments: web · topics: web-navigation - Licenses: code Apache-2.0 ### LiteWebAgent Modular web-agent framework with browser control, planning, workflow memory, and tree-search components, plus web and extension interfaces. - https://awesomecua.com/resources/github-pathonaiorg-litewebagent/ · source: https://github.com/PathOnAIOrg/LiteWebAgent - Framework · reviewed 2026-10-16 · environments: web · topics: web-navigation, planning, memory, tool-integration - Licenses: code Apache-2.0 ### MAI-UI inference and evaluation code Published MAI-UI grounding and navigation-agent implementations with inference demos and evaluation scripts. - https://awesomecua.com/resources/github-tongyi-mai-mai-ui/ · source: https://github.com/Tongyi-MAI/MAI-UI - Framework · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, grounding, tool-integration - Licenses: code Apache-2.0 (MAI-UI subdirectory) ### MemGUI-Agent MemGUI-Agent training and offline evaluation tools provide trajectory conversion, LoRA SFT templates, and step-level checks of GUI actions, memory updates and context folding. - https://awesomecua.com/resources/github-kwai-memgui-agent/ · source: https://github.com/kwai/MemGUI-Agent - Framework · reviewed 2026-10-20 · environments: android · topics: memory, interaction-history, long-horizon, imitation-learning, evaluation ### META-GUI baseline Training and evaluation pipeline for conversational mobile GUI agents, with action prediction, response generation, and dataset processing. - https://awesomecua.com/resources/github-x-lance-meta-gui-baseline/ · source: https://github.com/X-LANCE/META-GUI-baseline - Framework · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, human-in-the-loop - Licenses: code MIT ### Midscene.js Visual automation toolkit for web, mobile, and desktop interfaces. - https://awesomecua.com/resources/github-web-infra-dev-midscene/ · source: https://github.com/web-infra-dev/midscene - Framework · reviewed 2026-10-16 · environments: cross-platform · topics: automation, grounding - Licenses: code MIT ### MobileForge Android GUI-agent adaptation framework combining app exploration, generated curricula and rollout feedback with hint-conditioned step-level GRPO; the documented public training path targets Qwen3-VL-8B. - https://awesomecua.com/resources/github-kwai-mobileforge/ · source: https://github.com/kwai/MobileForge - Framework · reviewed 2026-10-20 · environments: android · topics: mobile-navigation, data-generation, reinforcement-learning, evaluation - Licenses: code Apache-2.0 ### MobileRL Online agentic RL framework for mobile GUI agents built around the difficulty-adaptive ADAGRPO algorithm, with Dockerized AndroidWorld and AndroidLab evaluation and a MobileRL-9B checkpoint. - https://awesomecua.com/resources/github-thudm-mobilerl/ · source: https://github.com/THUDM/MobileRL - Framework · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning - Licenses: code MIT ### Mobilerun (formerly Droidrun) A framework that combines screenshots and accessibility state with agent tools for mobile-device tasks through command-line and Python interfaces. - https://awesomecua.com/resources/github-droidrun-mobilerun/ · source: https://github.com/droidrun/mobilerun - Framework · reviewed 2026-10-17 · environments: android, ios · topics: mobile-navigation, tool-integration, planning - Licenses: code MIT ### NNetNav Collects and labels browser-exploration trajectories for agent training, with code for data processing, fine-tuning, and web evaluation. - https://awesomecua.com/resources/github-murtyshikhar-nnetnav/ · source: https://github.com/MurtyShikhar/NNetnav - Framework · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation - Licenses: code Apache-2.0 ### Notte Web agent framework and cloud SDKs for managed Chrome sessions, adding a CLI, an MCP server for coding agents, serverless browser functions and an API that builds automations from English tasks. - https://awesomecua.com/resources/github-nottelabs-notte/ · source: https://github.com/nottelabs/notte - Framework · reviewed 2026-10-16 · environments: web · topics: web-navigation ### OpenManus OpenManus is an open-source agent framework that runs tasks from a terminal and integrates browser execution and screenshots through Browser Use. - https://awesomecua.com/resources/github-foundationagents-openmanus/ · source: https://github.com/FoundationAgents/OpenManus - Framework · reviewed 2026-11-06 · environments: web · topics: automation, tool-integration, web-navigation - Licenses: code MIT ### OpenMobile-2 code Evaluation adapters and training guidance for OpenMobile-2 across Android emulators and browser-rendered mobile simulation. - https://awesomecua.com/resources/github-openmobile2-openmobile2-code/ · source: https://github.com/OpenMobile2/OpenMobile2-Code - Framework · reviewed 2026-10-17 · environments: android, web · topics: mobile-navigation, tool-integration, evaluation - Licenses: code Apache-2.0 ### ORBIT framework Inspect-based multi-agent evaluation framework with role, scheduling and communication controls, BrowserART accessibility-tree actions, and desktop mouse, keyboard and screenshot tools. - https://awesomecua.com/resources/github-wlanderson0-orbit/ · source: https://github.com/wlanderson0/orbit - Framework · reviewed 2026-10-20 · environments: web, desktop · topics: evaluation, safety-security, web-navigation, observability - Licenses: code Apache-2.0 ### OS-Sentinel Safety-checking environment and benchmark for mobile agents on AndroidWorld that judges proposed actions with rule-based and VLM-based checks, evaluated on the MobileRisk trajectory dataset. - https://awesomecua.com/resources/github-os-copilot-os-sentinel/ · source: https://github.com/OS-Copilot/OS-Sentinel - Framework · reviewed 2026-10-16 · environments: desktop · topics: safety-security - Licenses: code Apache-2.0 ### PC-Agent-E training and deployment Tools for refining and augmenting human computer-use trajectories, preparing supervised training data, and deploying a native PC-Agent-E scaffold. - https://awesomecua.com/resources/github-gair-nlp-pc-agent-e/ · source: https://github.com/GAIR-NLP/PC-Agent-E - Framework · reviewed 2026-10-18 · environments: desktop, windows, linux · topics: data-generation, imitation-learning, automation - Licenses: code MIT ### Plan-and-Act A web-agent framework that separates high-level planning from action execution, with replanning and synthetic planner-data generation components. - https://awesomecua.com/resources/github-squeezeailab-plan-and-act/ · source: https://github.com/SqueezeAILab/plan-and-act - Framework · reviewed 2026-10-17 · environments: web · topics: planning, web-navigation, data-generation - Licenses: code MIT ### Proposer-Agent-Evaluator (PAE) A browser-agent training system that proposes tasks, collects visual interaction trajectories, and uses automated evaluation to drive policy improvement. - https://awesomecua.com/resources/github-amazon-science-pae/ · source: https://github.com/amazon-science/PAE - Framework · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, web-navigation, data-generation ### puppeteer-extra A modular plugin framework for Puppeteer, maintained in a monorepo with plugins and related packages. - https://awesomecua.com/resources/github-berstend-puppeteer-extra/ · source: https://github.com/berstend/puppeteer-extra - Framework · reviewed 2026-11-06 · environments: web · topics: automation, tool-integration ### RedTeamCUA A controlled testing framework that combines desktop virtual machines and replicated websites to evaluate computer-use agents against indirect prompt injection. - https://awesomecua.com/resources/github-osu-nlp-group-redteamcua/ · source: https://github.com/OSU-NLP-Group/RedTeamCUA - Framework · reviewed 2026-10-17 · environments: web, desktop · topics: safety-security, evaluation, sandbox - Licenses: code Apache-2.0 ### SCALE-CUA (THUDM) Computer-use training framework with VeriGen task synthesis, adaptive rollout sampling, and visual-context segmentation. - https://awesomecua.com/resources/github-thudm-scale-cua/ · source: https://github.com/THUDM/SCALE-CUA - Framework · reviewed 2026-10-16 · environments: desktop · topics: reinforcement-learning, data-generation, efficiency ### ScreenSuite A shared evaluation harness for GUI-model perception, offline action prediction and multi-step interaction, using simple smolagents-based agents across existing benchmarks. - https://awesomecua.com/resources/github-huggingface-screensuite/ · source: https://github.com/huggingface/screensuite - Framework · reviewed 2026-10-17 · environments: web, android, desktop · topics: evaluation, grounding, action-prediction - Licenses: code Apache-2.0 ### Seg2Click Training and inference implementation that predicts an executable-region mask and derives a GUI click from that mask. - https://awesomecua.com/resources/github-jcwang0602-seg2click/ · source: https://github.com/jcwang0602/Seg2Click - Framework · reviewed 2026-10-17 · environments: cross-platform · topics: grounding, screen-understanding - Licenses: code MIT ### SeleniumBase Python browser automation framework for web crawling, end-to-end testing and scraping, with a recorder that generates tests and stealth options. - https://awesomecua.com/resources/github-seleniumbase-seleniumbase/ · source: https://github.com/seleniumbase/SeleniumBase - Framework · reviewed 2026-11-06 · environments: web · topics: automation, recording, evaluation ### Self-MAP A conversational web-agent implementation with candidate ranking and memory-assisted action planning, including optional memory simplification, refinement and self-reflection modules. - https://awesomecua.com/resources/github-magicgh-self-map/ · source: https://github.com/magicgh/Self-MAP - Framework · reviewed 2026-10-17 · environments: web · topics: web-navigation, memory, self-reflection, planning - Licenses: code MIT ### SkillWeaver A web-agent framework that explores sites, synthesizes reusable interaction APIs and builds a skill library for later tasks. - https://awesomecua.com/resources/github-osu-nlp-group-skillweaver/ · source: https://github.com/OSU-NLP-Group/SkillWeaver - Framework · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration, data-generation - Licenses: code MIT ### Skyvern Browser workflow automation that relies on vision LLMs and a swarm of agents instead of fixed XPath selectors, offered as a Playwright-compatible SDK and a no-code workflow builder. - https://awesomecua.com/resources/github-skyvern-ai-skyvern/ · source: https://github.com/Skyvern-AI/skyvern - Framework · reviewed 2026-10-16 · environments: web · topics: workflow ### Stagehand Browser automation framework combining code and natural-language actions. - https://awesomecua.com/resources/github-browserbase-stagehand/ · source: https://github.com/browserbase/stagehand - Framework · reviewed 2026-10-16 · environments: web · topics: web-navigation - Licenses: code MIT ### StateAct State-tracking prompts and experiment runners for language agents, including a WebShop implementation alongside text-based environments. - https://awesomecua.com/resources/github-ai-nikolai-stateact/ · source: https://github.com/ai-nikolai/StateAct - Framework · reviewed 2026-10-17 · environments: web · topics: planning, state-tracking, web-navigation ### STEVE / STEVE-R1 research code Research code and linked models for step-verified computer-use training; the current repository describes STEVE-R1 and its trajectory releases. - https://awesomecua.com/resources/github-fanbinlu-steve-r1/ · source: https://github.com/FanbinLu/STEVE-R1 - Framework · reviewed 2026-10-16 · environments: desktop, windows, linux · topics: data-generation, automation, evaluation ### Surfkit Python toolkit and CLI for building, publishing and running multimodal agents that operate devices such as Ubuntu desktops, locally or on Kubernetes, with task trackers and an agent hub. - https://awesomecua.com/resources/github-agentsea-surfkit/ · source: https://github.com/agentsea/surfkit - Framework · reviewed 2026-10-16 · environments: desktop · topics: workflow - Licenses: code MIT ### SwitchMem Memory-routing implementation that chooses text or image demonstrations for a frozen GUI actor, with offline evaluation and Android transfer experiments. - https://awesomecua.com/resources/github-dmis-lab-switchmem/ · source: https://github.com/dmis-lab/SwitchMem - Framework · reviewed 2026-10-17 · environments: android · topics: memory, mobile-navigation ### Synapse A computer-control research implementation that retrieves example trajectories from memory, abstracts task-relevant state and prompts an agent to navigate browser tasks. - https://awesomecua.com/resources/github-ltzheng-synapse/ · source: https://github.com/ltzheng/Synapse - Framework · reviewed 2026-10-17 · environments: web · topics: memory, planning, web-navigation, imitation-learning ### TestDriver SDK Vitest plugin that spawns ephemeral devices and uses vision-based LLMs to build end-to-end tests of web apps, desktop apps and browser extensions. - https://awesomecua.com/resources/github-testdriverai-testdriverai/ · source: https://github.com/testdriverai/testdriverai - Framework · reviewed 2026-11-06 · environments: desktop, web · topics: automation, evaluation, sandbox ### TongUI training and inference framework Training configurations, preprocessing and inference examples for tutorial-trained GUI models, with a LLaMA-Factory fine-tuning workflow and screenshot-grounding interfaces. - https://awesomecua.com/resources/github-tongui-agent-tongui-agent/ · source: https://github.com/TongUI-agent/TongUI-agent - Framework · reviewed 2026-10-17 · environments: cross-platform, desktop, web · topics: grounding, imitation-learning, data-generation ### TreeCUA Framework for generating GUI trajectories through tree-structured exploration and verification. - https://awesomecua.com/resources/github-uitron-hub-treecua/ · source: https://github.com/UITron-hub/TreeCUA - Framework · reviewed 2026-10-16 · environments: desktop · topics: data-generation - Licenses: code Apache-2.0 ### UI-Genie Training and evaluation code for a mobile-agent framework that iteratively generates trajectories using agent and reward-model feedback. - https://awesomecua.com/resources/github-euphoria16-ui-genie/ · source: https://github.com/Euphoria16/UI-Genie - Framework · reviewed 2026-10-17 · environments: android · topics: data-generation, reward-models, imitation-learning ### VEM GUI-Agent-RL Training code that learns a value environment model from offline Android interaction data and uses its frozen value estimates to guide GUI policy optimization. - https://awesomecua.com/resources/github-microsoft-gui-agent-rl/ · source: https://github.com/microsoft/GUI-Agent-RL - Framework · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, reward-models, mobile-navigation - Licenses: code MIT ### VeriOS Research code for proactive human–GUI-agent verification, including offline interaction-loop and dual-agent evaluation scripts plus LLaMA-Factory fine-tuning configurations. - https://awesomecua.com/resources/github-wuzheng02-verios/ · source: https://github.com/Wuzheng02/VeriOS - Framework · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, human-in-the-loop, mobile-navigation ### VeriSafe Agent Runtime verification framework for Android GUI agents that checks proposed actions against logical rules derived from user instructions and developer-defined app states. - https://awesomecua.com/resources/github-verisafeagent-verisafeagent/ · source: https://github.com/VeriSafeAgent/VeriSafeAgent - Framework · reviewed 2026-10-18 · environments: android · topics: safety-security, mobile-navigation, state-tracking ### VideoCAD training and evaluation code VideoCADFormer training, action-log preprocessing and evaluation code for learning CAD interface action sequences from screen recordings of browser-based Onshape. - https://awesomecua.com/resources/github-ghadinehme-videocad/ · source: https://github.com/ghadinehme/VideoCAD - Framework · reviewed 2026-10-18 · environments: web · topics: grounding, imitation-learning, evaluation - Licenses: code MIT ### WebAgent-R1 Environment setup, multi-turn reinforcement-learning and evaluation tooling for browser agents interacting with WebArena. - https://awesomecua.com/resources/github-weizhepei-webagent-r1/ · source: https://github.com/weizhepei/WebAgent-R1 - Framework · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, web-navigation, evaluation - Licenses: code Apache-2.0 ### WebDreamer World-model and planning components that simulate web-action outcomes, with linked training data and model checkpoints. - https://awesomecua.com/resources/github-osu-nlp-group-webdreamer/ · source: https://github.com/OSU-NLP-Group/WebDreamer - Framework · reviewed 2026-10-16 · environments: web · topics: world-models, planning ### WebEvolver code component Component for web-agent trajectory collection, supervised-data preparation and world-model-based trajectory synthesis. - https://awesomecua.com/resources/component-tencent-selfevolvingagent-webevolver/ · source: https://github.com/Tencent/SelfEvolvingAgent/tree/main/WebEvolver - Framework · reviewed 2026-10-18 · environments: web · topics: web-navigation, world-models, data-generation, self-reflection - Licenses: code MIT ### WebLlama Training, evaluation and integration tooling for dialogue-guided web agents, including Llama fine-tuning workflows and BrowserGym examples built around WebLINX. - https://awesomecua.com/resources/github-mcgill-nlp-webllama/ · source: https://github.com/McGill-NLP/webllama - Framework · reviewed 2026-10-17 · environments: web · topics: web-navigation, imitation-learning, evaluation, tool-integration - Licenses: code MIT ### WebRL Training pipeline that evolves web tasks from prior attempts and uses an outcome reward model to refine browser policies. - https://awesomecua.com/resources/github-thudm-webrl/ · source: https://github.com/THUDM/WebRL - Framework · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, web-navigation, data-generation ### WebVLN research implementation Training and evaluation code for question-guided website navigation that combines visual observations with web-specific HTML information. - https://awesomecua.com/resources/github-webvln-webvln/ · source: https://github.com/WebVLN/WebVLN - Framework · reviewed 2026-10-17 · environments: web · topics: web-navigation, grounding, evaluation ### WinDOM code GUI-grounding training and evaluation code with same-family distillation, GRPO recipes and point-in-box scoring, using a Windows-style web environment. - https://awesomecua.com/resources/github-chenghengli-windom/ · source: https://github.com/ChenghengLi/WinDOM - Framework · reviewed 2026-10-20 · environments: web · topics: grounding, imitation-learning, reinforcement-learning, evaluation - Licenses: code MIT ### Work2Skill Collects human GUI demonstrations and inspects multimodal evidence to derive reusable agent workflow skills. - https://awesomecua.com/resources/github-china-qijizhifeng-work2skill/ · source: https://github.com/china-qijizhifeng/Work2Skill - Framework · reviewed 2026-10-16 · environments: desktop, macos, linux · topics: recording, workflow, data-generation - Licenses: code MIT ### Workflow-Guided Exploration research code Research code that learns browser-control policies by constraining exploration with workflows induced from demonstrations, with a Selenium-based MiniWoB environment and experiment harness. - https://awesomecua.com/resources/github-stanfordnlp-wge/ · source: https://github.com/stanfordnlp/wge - Framework · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, web-navigation, workflow, history - Licenses: code Apache-2.0 ### WorkForceAgent-R1 Web-task reward functions, WorkArena data processing and GRPO training configurations for reasoning-oriented language agents. - https://awesomecua.com/resources/github-night-chen-workforceagent-r1/ · source: https://github.com/night-chen/WorkForceAgent-R1 - Framework · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, web-navigation, planning ### ZeroGUI Online RL framework for GUI agents in which VLMs generate the training tasks and judge success from trajectory screenshots, with OSWorld and AndroidLab checkpoints trained from UI-TARS-7B-DPO. - https://awesomecua.com/resources/github-opengvlab-zerogui/ · source: https://github.com/OpenGVLab/ZeroGUI - Framework · reviewed 2026-10-16 · environments: cross-platform · topics: reinforcement-learning, data-generation - Licenses: code Apache-2.0 ## Infrastructure (16) ### Agent Infra Sandbox Sandbox environments supporting computer and browser interaction. - https://awesomecua.com/resources/github-agent-infra-sandbox/ · source: https://github.com/agent-infra/sandbox - Infrastructure · reviewed 2026-10-16 · environments: desktop, web · topics: sandbox - Licenses: code Apache-2.0 ### AgentDesk Desktop environments that AI agents control programmatically, run locally with Docker or remotely on Kubernetes, built on the agentd REST daemon and offering a coordinate-grid overlay for LMM input. - https://awesomecua.com/resources/github-agentsea-agentdesk/ · source: https://github.com/agentsea/agentdesk - Infrastructure · reviewed 2026-10-16 · environments: desktop · topics: sandbox - Licenses: code MIT ### AIOS AIOS is an agent operating system whose computer-use architecture provides a virtual machine controller and MCP server for agents interacting with graphical computer environments. - https://awesomecua.com/resources/github-agiresearch-aios/ · source: https://github.com/agiresearch/AIOS - Infrastructure · reviewed 2026-11-06 · environments: desktop · topics: sandbox, tool-integration, memory ### AndroidEnv Android device-control platform with screenshots, accessibility telemetry, input actions, and an optional reinforcement-learning task interface. - https://awesomecua.com/resources/github-google-deepmind-android-env/ · source: https://github.com/google-deepmind/android_env - Infrastructure · reviewed 2026-10-16 · environments: android · topics: sandbox, reinforcement-learning, automation - Licenses: code Apache-2.0 ### Appium Open-source WebDriver-based automation server for mobile, desktop and IoT apps, extended by per-platform drivers for iOS, Android, macOS and Windows, plugins, and clients in Java, Python, Ruby and C#. - https://awesomecua.com/resources/github-appium-appium/ · source: https://github.com/appium/appium - Infrastructure · reviewed 2026-10-16 · environments: android, ios · topics: automation - Licenses: code Apache-2.0 ### BrowserOS / BrowserOS neo Chromium-based browser implementations for external agent control and built-in assistant workflows, with session visibility and replay facilities. - https://awesomecua.com/resources/github-browseros-ai-browseros/ · source: https://github.com/browseros-ai/BrowserOS - Infrastructure · reviewed 2026-10-17 · environments: web, desktop, windows, macos, linux · topics: web-navigation, tool-integration, recording - Licenses: code AGPL-3.0 ### E2B Desktop Open-source, isolated Linux Xfce desktop sandboxes for computer-use agents, driven from Python or JavaScript SDKs with screen streaming, mouse, keyboard and window control, screenshots and shell. - https://awesomecua.com/resources/github-e2b-dev-desktop/ · source: https://github.com/e2b-dev/desktop - Infrastructure · reviewed 2026-10-16 · environments: desktop, linux · topics: sandbox - Licenses: code Apache-2.0 ### Gobii Platform Self-hostable platform for always-on agents based on browser-use, adding headed browsers, persistent profiles, proxy-aware routing, schedules and event triggers. - https://awesomecua.com/resources/github-gobii-ai-gobii-platform/ · source: https://github.com/gobii-ai/gobii-platform - Infrastructure · reviewed 2026-11-06 · environments: web · topics: automation, web-navigation, long-horizon - Licenses: code MIT ### Lume Virtual machine management for local computer-use environments. - https://awesomecua.com/resources/github-trycua-lume/ · source: https://github.com/trycua/lume - Infrastructure · reviewed 2026-10-16 · environments: desktop, macos, linux · topics: sandbox - Licenses: code MIT ### Mobile Env AndroidEnv-based platform for building GUI agent benchmarks and training agents, where task definition files emit instructions, rewards and episode ends from screen text, view hierarchy or log events. - https://awesomecua.com/resources/github-x-lance-mobile-env/ · source: https://github.com/X-LANCE/Mobile-Env - Infrastructure · reviewed 2026-10-16 · environments: android · topics: evaluation - Licenses: code Apache-2.0 ### Nut.js Desktop automation library for keyboard, mouse, and screen interaction. - https://awesomecua.com/resources/github-nut-tree-nut-js/ · source: https://github.com/nut-tree/nut.js - Infrastructure · reviewed 2026-10-16 · environments: desktop · topics: automation ### Playwright Browser automation library used as an execution layer by agents. - https://awesomecua.com/resources/github-microsoft-playwright/ · source: https://github.com/microsoft/playwright - Infrastructure · reviewed 2026-10-16 · environments: web · topics: automation - Licenses: code Apache-2.0 ### PyAutoGUI Cross-platform Python module for scripted mouse and keyboard control, message boxes and screenshots with on-screen image search, working on the primary monitor under Windows, macOS and Linux. - https://awesomecua.com/resources/github-asweigart-pyautogui/ · source: https://github.com/asweigart/pyautogui - Infrastructure · reviewed 2026-10-16 · environments: desktop · topics: automation ### ScienceBoard VMware environment A prebuilt Ubuntu virtual-machine archive containing the software environment used for ScienceBoard scientific computer-use tasks. - https://awesomecua.com/resources/url-75917f2c2b2802aafd3e/ · source: https://huggingface.co/OS-Copilot/ScienceBoard-Env - Infrastructure · reviewed 2026-10-17 · environments: linux, desktop · topics: sandbox, science, evaluation ### Steel Browser Open-source browser API managing Chrome sessions for agents, reachable through Puppeteer, Playwright or Selenium over CDP, with persisted cookies, proxy rotation, stealth and a session debugging UI. - https://awesomecua.com/resources/github-steel-dev-steel-browser/ · source: https://github.com/steel-dev/steel-browser - Infrastructure · reviewed 2026-10-16 · environments: web · topics: sandbox, web-navigation - Licenses: code Apache-2.0 ### Universe legacy GUI interaction client An archived Gym-compatible client that exposes screen observations and keyboard/mouse actions over VNC for remote application and browser environments, including the historical World of Bits integration. - https://awesomecua.com/resources/github-openai-universe/ · source: https://github.com/openai/universe - Infrastructure · reviewed 2026-10-17 · environments: web, desktop · topics: automation, reinforcement-learning, history - Licenses: code MIT ## Tools (32) ### Activity Recorder Windows tray utility that records primary-display screenshots, keyboard key codes and mouse events as local JPEG and JSON files for training-data collection. - https://awesomecua.com/resources/github-anikasomaia-local-activity-recorder/ · source: https://github.com/anikasomaia/local-activity-recorder - Tool · reviewed 2026-10-20 · environments: desktop, windows · topics: recording, data-generation ### Agent Browser Shield Browser extension for filtering potentially unsafe agent input. - https://awesomecua.com/resources/github-pixiebrix-agent-browser-shield/ · source: https://github.com/pixiebrix/agent-browser-shield - Tool · reviewed 2026-10-16 · environments: web · topics: safety-security ### agent-browser A browser-automation command-line interface that exposes browser state and interaction commands for AI agents. - https://awesomecua.com/resources/github-vercel-labs-agent-browser/ · source: https://github.com/vercel-labs/agent-browser - Tool · reviewed 2026-10-17 · environments: web · topics: web-navigation, automation, tool-integration - Licenses: code Apache-2.0 ### AUITestAgent Generates mobile GUI interactions from natural-language test requirements and checks application behavior against those requirements. - https://awesomecua.com/resources/github-bz-lab-auitestagent/ · source: https://github.com/bz-lab/AUITestAgent - Tool · reviewed 2026-10-16 · environments: unspecified · topics: automation, evaluation ### Browser Control MCP An MCP server and Firefox extension that let AI assistants manage tabs, search browser history, read webpage text, and capture screenshots with user consent. - https://awesomecua.com/resources/github-eyalzh-browser-control-mcp/ · source: https://github.com/eyalzh/browser-control-mcp - Tool · reviewed 2026-11-06 · environments: web · topics: tool-integration, web-navigation, human-in-the-loop, safety-security - Licenses: code MIT ### Browser Use WebUI A Gradio interface for configuring and running Browser Use agents, including persistent browser sessions and interaction recordings. - https://awesomecua.com/resources/github-browser-use-web-ui/ · source: https://github.com/browser-use/web-ui - Tool · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration, recording - Licenses: code MIT ### BrowserTrace Local, cloud-free replay debugger that records each Browser Use step's screenshot, URL, action and model I/O as a timeline and reports the step at which a failed run diverges from a known-good one. - https://awesomecua.com/resources/github-aaronlab-browsertrace/ · source: https://github.com/aaronlab/browsertrace - Tool · reviewed 2026-10-16 · environments: web · topics: observability - Licenses: code MIT ### Chrome DevTools MCP MCP tools for browser control and inspection through Chrome DevTools. - https://awesomecua.com/resources/github-chromedevtools-chrome-devtools-mcp/ · source: https://github.com/ChromeDevTools/chrome-devtools-mcp - Tool · reviewed 2026-10-16 · environments: web · topics: automation, tool-integration - Licenses: code Apache-2.0 ### chromedp A Go package for driving browsers through the Chrome DevTools Protocol, with actions for navigating pages, interacting with elements, and reading page content. - https://awesomecua.com/resources/github-chromedp-chromedp/ · source: https://github.com/chromedp/chromedp - Tool · reviewed 2026-11-06 · environments: web · topics: automation, web-navigation ### Firecrawl Firecrawl is an API that lets agents search and scrape web pages, then interact with pages through prompts or code. - https://awesomecua.com/resources/github-firecrawl-firecrawl/ · source: https://github.com/firecrawl/firecrawl - Tool · reviewed 2026-11-06 · environments: web · topics: web-navigation, automation, tool-integration - Licenses: code AGPL-3.0 ### GUI Grounding Evaluation Harness ScreenSpot-Pro evaluation harness with explicit coordinate transforms, prompt and crop-based inference comparisons, and per-sample grounding accuracy and cost records. - https://awesomecua.com/resources/github-patrickstarl-gui-grounding-eval/ · source: https://github.com/PatrickStarL/gui-grounding-eval - Tool · reviewed 2026-10-20 · environments: unspecified · topics: grounding, evaluation, efficiency ### Libretto Libretto is a toolkit that gives coding agents a live browser and CLI to inspect pages, record user actions, build scripts, and debug web integrations. - https://awesomecua.com/resources/github-cashew-labs-libretto/ · source: https://github.com/cashew-labs/libretto - Tool · reviewed 2026-11-06 · environments: web · topics: automation, recording, tool-integration, web-navigation - Licenses: code MIT ### mcp-server-browser-use MCP server that wraps browser-use so that an MCP client can automate a real browser, with a CLI, a web dashboard and a deep research mode. - https://awesomecua.com/resources/github-saik0s-mcp-browser-use/ · source: https://github.com/Saik0s/mcp-browser-use - Tool · reviewed 2026-11-06 · environments: web · topics: automation, web-navigation, tool-integration - Licenses: code MIT ### Mobile Harness Portable agent instructions and integration tooling for local or cloud phone control through mobilerun-core. - https://awesomecua.com/resources/github-droidrun-mobile-harness/ · source: https://github.com/droidrun/mobile-harness - Tool · reviewed 2026-10-16 · environments: android, ios · topics: automation, tool-integration - Licenses: code MIT ### Mobile Next MCP An MCP server exposing mobile accessibility snapshots, screenshots and device interactions across real devices, simulators and emulators. - https://awesomecua.com/resources/github-mobile-next-mobile-mcp/ · source: https://github.com/mobile-next/mobile-mcp - Tool · reviewed 2026-10-17 · environments: android, ios · topics: mobile-navigation, tool-integration, automation - Licenses: code Apache-2.0 ### Mobilerun Portal (formerly Droidrun Portal) An Android accessibility service that exposes UI-element inspection, device-control interfaces and screen feedback to mobile automation agents. - https://awesomecua.com/resources/github-droidrun-mobilerun-portal/ · source: https://github.com/droidrun/mobilerun-portal - Tool · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, screen-understanding, tool-integration - Licenses: code AGPL-3.0-or-later ### OculiX (SikuliX continuation) A maintained continuation of SikuliX that locates on-screen targets with image matching and OCR, drives desktop or Android interfaces, and exposes scripting and optional MCP interfaces. - https://awesomecua.com/resources/github-oculix-org-oculix/ · source: https://github.com/oculix-org/Oculix - Tool · reviewed 2026-10-17 · environments: desktop, windows, macos, linux, android · topics: automation, grounding, tool-integration - Licenses: code MIT ### OmniParser Screenshot parser that detects interactable regions and captions icons so vision-language models can ground actions, released as V1.5 and V2 weights alongside OmniTool for driving a Windows 11 VM. - https://awesomecua.com/resources/github-microsoft-omniparser/ · source: https://github.com/microsoft/OmniParser - Tool · reviewed 2026-10-16 · environments: cross-platform · topics: screen-understanding, grounding ### PhantomJS PhantomJS is a scriptable headless WebKit tool for automating web pages, capturing page contents, testing, and monitoring network activity. - https://awesomecua.com/resources/github-ariya-phantomjs/ · source: https://github.com/ariya/phantomjs - Tool · reviewed 2026-11-06 · environments: web · topics: automation, evaluation, observability ### pi-computer-use Pi extension that gives agents tools to find app windows, observe and search visible controls, click, type, scroll and wait for UI changes in desktop software. - https://awesomecua.com/resources/github-injaneity-pi-computer-use/ · source: https://github.com/injaneity/pi-computer-use - Tool · reviewed 2026-11-06 · environments: desktop, macos, windows, linux · topics: automation, screen-understanding, tool-integration - Licenses: code MIT ### Playwright MCP MCP server that lets LLM clients drive Playwright browsers through structured accessibility snapshots instead of screenshots, with opt-in vision, PDF and devtools capabilities. - https://awesomecua.com/resources/github-microsoft-playwright-mcp/ · source: https://github.com/microsoft/playwright-mcp - Tool · reviewed 2026-10-16 · environments: web · topics: automation, tool-integration - Licenses: code Apache-2.0 ### Playwright MCP Server (ExecuteAutomation) Community MCP server that gives LLMs Playwright browser automation for interacting with web pages, taking screenshots, generating test code and running JavaScript. - https://awesomecua.com/resources/github-executeautomation-mcp-playwright/ · source: https://github.com/executeautomation/mcp-playwright - Tool · reviewed 2026-11-06 · environments: web · topics: automation, web-navigation, tool-integration - Licenses: code MIT ### Set of Mark Visual-prompting toolbox that labels image regions for multimodal reference and grounding, including GUI applications. - https://awesomecua.com/resources/github-microsoft-som/ · source: https://github.com/microsoft/SoM - Tool · reviewed 2026-10-16 · environments: unspecified · topics: grounding - Licenses: code MIT ### Sikuli legacy research implementation The original Sikuli source distribution combines screenshot targets with a Jython scripting library and IDE for visual GUI automation and testing. - https://awesomecua.com/resources/github-sikuli-sikuli/ · source: https://github.com/sikuli/sikuli - Tool · reviewed 2026-10-17 · environments: desktop, macos, linux, windows · topics: automation, grounding, screen-understanding, history - Licenses: code MIT ### Tactile Skill, protocol and tool layer that has agents operate software through accessibility semantics first, reading roles, names and states before reaching for screenshots. - https://awesomecua.com/resources/github-yliust-tactile/ · source: https://github.com/yliust/Tactile - Tool · reviewed 2026-11-06 · environments: desktop, macos · topics: screen-understanding, grounding, tool-integration ### TAG attention-driven GUI grounding Grounding and evaluation code that aggregates a pretrained vision-language model’s attention maps to localize requested interface elements without fine-tuning. - https://awesomecua.com/resources/github-heimingx-tag/ · source: https://github.com/HeimingX/TAG - Tool · reviewed 2026-10-17 · environments: web · topics: grounding, evaluation, training-free ### Tarsier Perception utilities for web agents that tag interactable elements with typed IDs an LLM can act on, and use OCR to render page screenshots as whitespace-structured text for models without vision. - https://awesomecua.com/resources/github-reworkd-tarsier/ · source: https://github.com/reworkd/tarsier - Tool · reviewed 2026-10-16 · environments: web · topics: screen-understanding - Licenses: code MIT ### UI Defect Mocker Synthesizes visual UI defects from mobile screenshots and Android layout metadata to support GUI-testing experiments. - https://awesomecua.com/resources/github-zju-aces-ise-uidefectmocker/ · source: https://github.com/ZJU-ACES-ISE/UIDefectMocker - Tool · reviewed 2026-10-16 · environments: android · topics: data-generation, evaluation, screen-understanding - Licenses: code MIT ### web-eval-agent MCP server that launches a browser-use agent to run a web app, capture screenshots, console and network logs, and return a UX report to the coding agent. - https://awesomecua.com/resources/github-refreshdotdev-web-eval-agent/ · source: https://github.com/refreshdotdev/web-eval-agent - Tool · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, observability - Licenses: code Apache-2.0 ### Web-Shepherd inference and evaluation Process-reward inference and benchmark-evaluation utilities that use task checklists to assess individual web-agent decisions. - https://awesomecua.com/resources/github-kyle8581-web-shepherd/ · source: https://github.com/kyle8581/Web-Shepherd - Tool · reviewed 2026-10-17 · environments: web · topics: reward-models, evaluation, web-navigation ### WebMarker A browser library that labels interactive elements and their boxes for visual prompting, preserving element mappings that automation code can use to execute selected actions. - https://awesomecua.com/resources/github-reidbarber-webmarker/ · source: https://github.com/reidbarber/webmarker - Tool · reviewed 2026-10-17 · environments: web · topics: grounding, screen-understanding, tool-integration - Licenses: code MIT ### Windows-MCP A Windows MCP bridge exposing application, window, keyboard and mouse operations plus native UI-state inspection for agents. - https://awesomecua.com/resources/github-cursortouch-windows-mcp/ · source: https://github.com/CursorTouch/Windows-MCP - Tool · reviewed 2026-10-17 · environments: windows, desktop, web · topics: tool-integration, automation, screen-understanding - Licenses: code MIT ## Products (20) ### AgentQL Natural-language querying and interaction tools for web interfaces. - https://awesomecua.com/resources/url-8896d44370711136fd54/ · source: https://www.agentql.com/ - Product · reviewed 2026-10-16 · environments: web · topics: grounding, web-navigation ### Amazon Nova Act AWS service for building and deploying agents that automate browser-based UI workflows from natural language and Python, powered by a custom Nova 2 Lite model. - https://awesomecua.com/resources/url-dec156cbf0cb1181f7b6/ · source: https://aws.amazon.com/nova/act/ - Product · reviewed 2026-10-17 · environments: web · topics: automation, web-navigation ### AskUI Agentic test-automation platform running plain-language test cases from pixels rather than selectors on desktop apps, Citrix, in-car head units and mobile devices, deployable on-premise or air-gapped. - https://awesomecua.com/resources/url-69d6afe38a9a128ba9f7/ · source: https://www.askui.com/ - Product · reviewed 2026-10-16 · environments: desktop, web · topics: workflow ### Browser Use Cloud Hosted browser-agent platform from Browser Use. - https://awesomecua.com/resources/url-ef244017b1460269fc68/ · source: https://browser-use.com/ - Product · reviewed 2026-10-16 · environments: web · topics: web-navigation ### BrowserAct Managed web scraping platform with reusable browser interaction workflows. - https://awesomecua.com/resources/url-7ae2670859e3796e8607/ · source: https://www.browseract.com/ - Product · reviewed 2026-10-16 · environments: web · topics: workflow ### Browserbase Cloud Managed browser infrastructure for agent applications. - https://awesomecua.com/resources/url-978d1dcd70fde8661b9e/ · source: https://www.browserbase.com/ - Product · reviewed 2026-10-16 · environments: web · topics: sandbox ### Browserless Hosted browser infrastructure for automation applications. - https://awesomecua.com/resources/url-1e6b8682e0f1ddc3d36f/ · source: https://www.browserless.io/ - Product · reviewed 2026-10-16 · environments: web · topics: sandbox ### ChatGPT Agent Hosted agent combining browser interaction, tools, and computer-based tasks. - https://awesomecua.com/resources/url-b074ccdb2d9ec0277b65/ · source: https://openai.com/index/introducing-chatgpt-agent/ - Product · reviewed 2026-10-16 · environments: web, desktop · topics: workflow ### ChatGPT Atlas OpenAI browser product integrating ChatGPT and agent capabilities. - https://awesomecua.com/resources/url-240a05c02f85d4da60dc/ · source: https://openai.com/index/introducing-chatgpt-atlas/ - Product · reviewed 2026-10-16 · environments: web · topics: web-navigation ### Claude for Chrome Claude browser extension for interacting with websites. - https://awesomecua.com/resources/url-07b6bbce7ce26a847f79/ · source: https://claude.com/chrome - Product · reviewed 2026-10-16 · environments: web · topics: web-navigation ### E2B Sandbox infrastructure for executing agent workloads. - https://awesomecua.com/resources/url-fda52d641c6b482bda7b/ · source: https://e2b.dev/ - Product · reviewed 2026-10-16 · environments: desktop · topics: sandbox ### HoloTab H browser extension for executing tasks through website interfaces. - https://awesomecua.com/resources/url-4a47a6df845735984bf1/ · source: https://hcompany.ai/newsroom/meet-holotab - Product · reviewed 2026-10-16 · environments: web · topics: web-navigation ### Hyperbrowser Browser infrastructure and automation services for AI agents. - https://awesomecua.com/resources/url-053ebfe0afac66c86222/ · source: https://www.hyperbrowser.ai/ - Product · reviewed 2026-10-16 · environments: web · topics: sandbox ### Manus General agent platform with browser-based task execution. - https://awesomecua.com/resources/url-d3414794302313051ff0/ · source: https://manus.im/ - Product · reviewed 2026-10-16 · environments: web · topics: workflow ### Operator Historical OpenAI browser-agent research preview. - https://awesomecua.com/resources/url-f0584842138a52158f51/ · source: https://openai.com/index/introducing-operator/ - Product · reviewed 2026-10-16 · environments: web · topics: web-navigation, history ### Scribe Workflow recording and documentation product adjacent to agent data collection. - https://awesomecua.com/resources/url-63ef8c7e455951a20281/ · source: https://scribehow.com/ - Product · reviewed 2026-10-16 · environments: desktop, web · topics: recording, workflow ### Simular Computer-use products and research from Simular. - https://awesomecua.com/resources/url-0bdd6b5ebf2e1f248828/ · source: https://www.simular.ai/ - Product · reviewed 2026-10-16 · environments: desktop · topics: workflow ### Skyvern Cloud Hosted Skyvern service for AI browser workflows such as invoice downloads and form filling, with built-in CAPTCHA and 2FA handling, proxy geotargeting, run explanations and JSON or CSV output. - https://awesomecua.com/resources/url-15ed5fd3cd950d837f98/ · source: https://www.skyvern.com/ - Product · reviewed 2026-10-16 · environments: web · topics: workflow ### Steel Cloud Managed browser infrastructure associated with Steel Browser. - https://awesomecua.com/resources/url-b940a884f200a916a61b/ · source: https://steel.dev/ - Product · reviewed 2026-10-16 · environments: web · topics: sandbox ### TinyFish Browser-agent API for executing web tasks and extracting structured results. - https://awesomecua.com/resources/url-d92596cfe618e5322ec2/ · source: https://www.tinyfish.ai/ - Product · reviewed 2026-10-16 · environments: web · topics: web-navigation ## Publishers (16) ### Anthropic Publisher or product brand associated with Anthropic computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-anthropic/ · source: https://claude.com/chrome - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### AskUI Publisher or product brand associated with AskUI computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-askui/ · source: https://www.askui.com/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Browser Use Publisher or product brand associated with Browser Use computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-browser-use/ · source: https://browser-use.com/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### BrowserAct Publisher or product brand associated with BrowserAct computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-browseract/ · source: https://www.browseract.com/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Browserbase Publisher or product brand associated with Browserbase computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-browserbase/ · source: https://www.browserbase.com/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Browserless Publisher or product brand associated with Browserless computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-browserless/ · source: https://www.browserless.io/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### E2B Publisher or product brand associated with E2B computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-e2b/ · source: https://e2b.dev/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Google Publisher or product brand associated with Google computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-google/ · source: https://ai.google.dev/gemini-api/docs/computer-use - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### H Publisher or product brand associated with H computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-h/ · source: https://hcompany.ai/research/models - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Hyperbrowser Publisher or product brand associated with Hyperbrowser computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-hyperbrowser/ · source: https://www.hyperbrowser.ai/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Manus Publisher or product brand associated with Manus computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-manus/ · source: https://manus.im/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### OpenAI Publisher or product brand associated with OpenAI computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-openai/ · source: https://openai.com/index/computer-using-agent/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Simular Publisher or product brand associated with Simular computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-simular/ · source: https://www.simular.ai/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Skyvern Publisher or product brand associated with Skyvern computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-skyvern/ · source: https://www.skyvern.com/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### Steel Publisher or product brand associated with Steel computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-steel/ · source: https://steel.dev/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ### TinyFish Publisher or product brand associated with TinyFish computer-use or browser automation offerings. - https://awesomecua.com/resources/organization-tinyfish/ · source: https://www.tinyfish.ai/ - Publisher · reviewed 2026-10-16 · environments: unspecified · topics: industry ## Learning resources (2) ### Building AI Browser Agents Browser-agent course with worked lessons on structured web extraction and multi-step browser interaction, followed by AgentQ search and self-correction concepts. - https://awesomecua.com/resources/url-aa693ca3e8948623f31e/ · source: https://www.deeplearning.ai/courses/building-ai-browser-agents - Learning resource · reviewed 2026-10-20 · environments: web · topics: web-navigation, planning ### Web Browser Automation with Agents (smolagents tutorial) Versioned tutorial building a vision-guided CodeAgent with Selenium/Helium browser tools, screenshot callbacks and example web-navigation tasks. - https://awesomecua.com/resources/url-5857c840372512cfe3fc/ · source: https://huggingface.co/docs/smolagents/examples/web_browser - Learning resource · reviewed 2026-10-20 · environments: web · topics: web-navigation, screen-understanding ## Surveys (28) ### A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions Reviews 87 computer-use agents and 33 datasets through a taxonomy of operating domain, observation and action modalities, and agent design. - https://awesomecua.com/resources/arxiv-2501-16150/ · source: https://arxiv.org/abs/2501.16150v3 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey - Published: 2025-01-27 · Authors: Pascal J. Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F. Grewe, Thilo Stadelmann ### A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models Surveys foundation-model web agents through their architectures, training methods and trustworthiness. It organizes research on completing browser tasks from user instructions and identifies directions for further development. - https://awesomecua.com/resources/arxiv-2503-23350/ · source: https://arxiv.org/abs/2503.23350v4 - Survey · reviewed 2026-10-18 · environments: web · topics: survey, web-navigation, safety-security - Published: 2025-03-30 · Authors: Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S. Yu, Qing Li ### A Survey on (M)LLM-Based GUI Agents Analyzes LLM-based GUI agents through four components (perception, exploration, planning, and interaction) and the methods used to evaluate them. - https://awesomecua.com/resources/arxiv-2504-13865/ · source: https://arxiv.org/abs/2504.13865v2 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey - Published: 2025-03-27 · Authors: Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang ### A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning Formalizes GUI tasks as Markov decision processes and traces agent training from prompting through supervised fine-tuning to reinforcement learning. - https://awesomecua.com/resources/arxiv-2504-20464/ · source: https://arxiv.org/abs/2504.20464v2 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey, reinforcement-learning - Published: 2025-04-29 · Authors: Jiahao Li, Kaer Huang ### A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle Organizes security risks and defenses for persistent agent memory into six lifecycle phases and proposes five verifiable governance primitives. - https://awesomecua.com/resources/arxiv-2604-16548/ · source: https://arxiv.org/abs/2604.16548v3 - Survey · reviewed 2026-10-18 · environments: unspecified · topics: memory, safety-security, survey - Published: 2026-04-17 · Authors: Zehao Lin, Xixuan Hao, Renyu Fu, Shaobo Cui, Kai Chen, Chunyu Li, Zhiyu Li, Feiyu Xiong ### A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron? Systematizes safety and security threats to computer-using agents, with taxonomies of threats and defensive strategies. - https://awesomecua.com/resources/arxiv-2505-10924/ · source: https://arxiv.org/abs/2505.10924v4 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey, safety-security - Published: 2025-05-16 · Authors: Ada Chen, Yongjiang Wu, Junyuan Zhang, Jingyu Xiao, Shu Yang, Jen-tse Huang, Kun Wang, Wenxuan Wang, Shuai Wang ### A Systematic Literature Review of Automated Techniques for Functional GUI Testing of Mobile Applications Surveys automated functional GUI testing techniques for mobile apps, comparing the effectiveness, efficiency and practicality of approaches reported in 25 studies. - https://awesomecua.com/resources/arxiv-1812-11470/ · source: https://arxiv.org/abs/1812.11470v1 - Survey · reviewed 2026-11-06 · environments: unspecified · topics: survey, automation, evaluation - Published: 2018-12-30 · Authors: Yauhen Leanidavich Arnatovich, Lipo Wang ### A Systematic Survey of Agentic Skills: Architecture, Lifecycle, and Security Surveys modular agentic skill architectures, skill lifecycles, and security risks in autonomous tool- and interface-using agents. - https://awesomecua.com/resources/arxiv-2608-29596/ · source: https://arxiv.org/abs/2608.29596v2 - Survey · reviewed 2026-10-18 · environments: cross-platform · topics: survey, workflow, memory, safety-security - Published: 2026-08-30 · Authors: Sanket Badhe, Deep Shah, Priyanka Tiwari, Nehal Kathrotia ### A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents Presents a conceptual perception–reasoning–action framework and 12 potential attack scenarios for language agents, including hypothetical web-navigation attacks. - https://awesomecua.com/resources/arxiv-2402-10196/ · source: https://arxiv.org/abs/2402.10196v1 - Survey · reviewed 2026-10-18 · environments: web · topics: safety-security, survey, robustness - Published: 2024-02-15 · Authors: Lingbo Mo, Zeyi Liao, Boyuan Zheng, Yu Su, Chaowei Xiao, Huan Sun ### Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward This survey organizes agent skills by architecture, acquisition, deployment and security, including their role in computer-use systems and GUI grounding. It also proposes a permission and lifecycle governance framework for skill deployment. - https://awesomecua.com/resources/arxiv-2602-12430/ · source: https://arxiv.org/abs/2602.12430v4 - Survey · reviewed 2026-10-17 · environments: unspecified · topics: survey, tool-integration, safety-security, planning - Published: 2026-02-12 · Authors: Renjun Xu, Yang Yan ### Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents Surveys LLM agent architectures, operating environments and evaluation, covering web navigation and native computer use alongside other agent domains. - https://awesomecua.com/resources/arxiv-2601-12560/ · source: https://arxiv.org/abs/2601.12560v1 - Survey · reviewed 2026-11-06 · environments: web · topics: survey, evaluation, planning, tool-integration - Published: 2026-01-18 · Authors: Arunkumar V, Gangadharan G. R., Rajkumar Buyya ### Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond Survey of agentic world models that classifies predictive capabilities and governing laws, including the digital setting of web and GUI agents. - https://awesomecua.com/resources/arxiv-2604-22748/ · source: https://arxiv.org/abs/2604.22748v3 - Survey · reviewed 2026-11-06 · environments: web · topics: survey, world-models, evaluation - Published: 2026-04-24 · Authors: Meng Chu, Xuan Billy Zhang, Kevin Qinghong Lin, Lingdong Kong, Jize Zhang, Teng Tu, Weijian Ma, Ziqi Huang, Senqiao Yang, Wei Huang, Yeying Jin, Zhefan Rao, Jinhui Ye, Xinyu Lin, Xichen Zhang, Qisheng Hu, Shuai Yang, Leyang Shen, Wei Chow, Yifei Dong, Fengyi Wu, Quanyu Long, Bin Xia, Shaozuo Yu, Mingkang Zhu, Wenhu Zhang, Jiehui Huang, Haokun Gui, Runyi Li, Chenyu Tang, Dong Huang, Xuhang Chen, Rui Liu, Chengzu Li, Shiyi Du, Xu Huang, Haoxuan Che, Long Chen, Qifeng Chen, Wenya Wang, Wenxuan Zhang, Xiaojuan Qi, Yang Deng, Yanwei Li, Mike Zheng Shou, Zhi-Qi Cheng, See-Kiong Ng, Ziwei Liu, Philip Torr, Jiaya Jia ### Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI Surveys model-native agentic AI and the role of RL in learned planning, tool use and memory, covering GUI agents as one application alongside deep-research agents. - https://awesomecua.com/resources/arxiv-2510-16720/ · source: https://arxiv.org/abs/2510.16720v2 - Survey · reviewed 2026-11-06 · environments: unspecified · topics: survey, reinforcement-learning, planning - Published: 2025-10-19 · Authors: Jitao Sang, Jinlin Xiao, Jiarun Han, Jilin Chen, Xiaoyi Chen, Shuyu Wei, Yongjie Sun, Yuhang Wang ### Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization Surveys GUI-agent efficiency beyond task success: observation, context and memory, action, and planner or runtime cost. - https://awesomecua.com/resources/arxiv-2609-02309/ · source: https://arxiv.org/abs/2609.02309v1 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey - Published: 2026-09-02 · Authors: Bizhe Bai, Jiakang Yuan, Hongming Wu, Xinyue Wang, Jie Ren, Siyao Chen, Yuchen Ya, Fan Bai, Pai Peng, Huafeng Qin, Tao Chen ### Foundations and Recent Trends in Multimodal Mobile Agents: A Survey Surveys multimodal mobile agents, organizing prompt-based and training-based approaches alongside evaluation benchmarks, supporting technologies, and open challenges in mobile task execution. - https://awesomecua.com/resources/arxiv-2411-02006/ · source: https://arxiv.org/abs/2411.02006v3 - Survey · reviewed 2026-10-17 · environments: unspecified · topics: survey, mobile-navigation, evaluation - Published: 2024-11-04 · Authors: Biao Wu, Yanda Li, Zhiwei Zhang, Yunchao Wei, Meng Fang, Ling Chen ### Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms Surveys virtual agents for browsers, applications and operating systems, organizing prior work by task type, observation and action spaces, model strategies and evaluation methods. It examines screen-based interaction and keyboard, mouse and touch control, alongside transferability and long-sequence decision-making challenges. - https://awesomecua.com/resources/arxiv-2411-10943/ · source: https://arxiv.org/abs/2411.10943v1 - Survey · reviewed 2026-10-18 · environments: web, desktop, android, windows · topics: survey, web-navigation, mobile-navigation, evaluation, planning - Published: 2024-11-17 · Authors: Minghe Gao, Wendong Bu, Bingchen Miao, Yang Wu, Yunfei Li, Juncheng Li, Siliang Tang, Qi Wu, Yueting Zhuang, Meng Wang ### GUI Agents with Foundation Models: A Comprehensive Survey Survey of (M)LLM-based GUI agents organized around their datasets and benchmarks, a unified agent framework with a taxonomy, and commercial applications. - https://awesomecua.com/resources/arxiv-2411-04890/ · source: https://arxiv.org/abs/2411.04890v2 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey - Published: 2024-11-07 · Authors: Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang, Chuhan Wu, Yasheng Wang, Ruiming Tang, Jianye Hao ### GUI Agents: A Survey Categorizes GUI-agent benchmarks, evaluation metrics, architectures, and training methods, and proposes a unified perception, reasoning, planning, and acting framework. - https://awesomecua.com/resources/arxiv-2412-13501/ · source: https://arxiv.org/abs/2412.13501v3 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey - Published: 2024-12-18 · Authors: Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Jihyung Kil, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, Franck Dernoncourt ### Large Language Model-Brained GUI Agents: A Survey Survey of LLM-brained GUI agents that carry out natural-language instructions across web, mobile, and desktop interfaces, organizing research and industry progress. - https://awesomecua.com/resources/arxiv-2411-18279/ · source: https://arxiv.org/abs/2411.18279v12 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey - Published: 2024-11-27 · Authors: Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang ### LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects Reviews LLM-driven phone GUI agents with a taxonomy of agent frameworks, prompt- and training-based modeling, datasets, and benchmarks. - https://awesomecua.com/resources/arxiv-2504-19838/ · source: https://arxiv.org/abs/2504.19838v3 - Survey · reviewed 2026-10-16 · environments: android, ios · topics: survey - Published: 2025-04-28 · Authors: Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, Hao Wang, Xiaoyu Liang, WenHao Wang, Tianze Wu, Zhengxi Lu, Siheng Chen, LiLinghao, Hao Wang, Guanjing Xiong, Yong Liu, Hongsheng Li ### Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use PAPAV is a capability framework and survey comparing computer-use agents with physical agents through perception, anticipation, planning, action and verification. - https://awesomecua.com/resources/arxiv-2610-30108/ · source: https://arxiv.org/abs/2610.30108v1 - Survey · reviewed 2026-11-06 · environments: unspecified · topics: survey, evaluation, planning - Published: 2026-10-21 · Authors: Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Kevin Qinghong Lin, Mike Zheng Shou ### OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use Surveys agents that operate computers and phones through operating-system interfaces, covering components, capabilities, foundation models, frameworks, and benchmarks. - https://awesomecua.com/resources/arxiv-2508-04482/ · source: https://arxiv.org/abs/2508.04482v1 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey - Published: 2025-08-06 · Authors: Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, Yuhuai Li, Shengze Xu, Shenzhi Wang, Xinchen Xu, Shuofei Qiao, Zhaokai Wang, Kun Kuang, Tieyong Zeng, Liang Wang, Jiwei Li, Yuchen Eleanor Jiang, Wangchunshu Zhou, Guoyin Wang, Keting Yin, Zhou Zhao, Hongxia Yang, Fan Wu, Shengyu Zhang, Fei Wu ### Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability Survey of computer-use agent reliability and security across browsers, desktops, and mobile apps, using architecture and lifecycle views to map failures and controls. - https://awesomecua.com/resources/arxiv-2605-07110/ · source: https://arxiv.org/abs/2605.07110v1 - Survey · reviewed 2026-11-06 · environments: web, desktop, cross-platform · topics: survey, safety-security, robustness, observability - Published: 2026-05-08 · Authors: Zejian Chen, Zhanyuan Liu, Chaozhuo Li, Mengxiang Han, Songyang Liu, Litian Zhang, Feng Gao, Yiming Hei, Xi Zhang ### Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory Surveys multimodal memory using representation and management axes, distinguishing fixed-policy updates from policy evolution and comparing GUI memory evaluation with other modalities. - https://awesomecua.com/resources/arxiv-2610-19366/ · source: https://arxiv.org/abs/2610.19366v1 - Survey · reviewed 2026-10-20 · environments: unspecified · topics: survey, memory, evaluation, interaction-history - Published: 2026-10-14 · Authors: Minghao Guo, Qingyue Jiao, Xiujin Liu, Jianhong Pang, Ruimeng Yang, Naihao Xu, Liwei Che, Haochen Xue, Ziqi Wang, Xinying Cai, Jiaxuan Luo, Jiaxin Deng, Xuesong Ye, Jintao Qu, Wenyao Gao, Jingwen Gu, Chenyang Zhao, Shichun Liu, Guibin Zhang, Zefang Liu, Siyin Wang, Wujiang Xu, Danrui Li, Zeru Shi, Woongyeong Yeo, Shuang Liu, Wei-Chieh Huang, Weizhi Zhang, Tianyu Yang, Kangsan Kim, Yebowen Hu, Bowen Wen, Bowen Jiang, Haohao Su, Ming Yin, Sheldon Yu, Mian Zhang, Yuyan Chen, Hanrong Zhang, Kai Mei, Zihao Zhang, Mingyu Jin, Xi Zhu, Hengyi Wang, Tianxin Wei, Linyong Nan, Zhenwen Liang, Zhendong Chu, Ziqi Huang, Yongye Su, Sixun Dong, Shujian Liu, Kaiqu Liang, Jiaojiao Han, Zhaoye Fei, Kun Wang, Shihan Dou, Zhichun Guo, Xingjian Diao, Zeqi Gu, Heng Wang, Tao Feng, Cheng Qian, Hamid Palangi, Silei Xu, Xi Yun, Yi Zhu, Shilong Liu, Qingsong Wen, Sung Ju Hwang, Zuxuan Wu, Yu-Gang Jiang, Yiyu Shi, Dimitris N. Metaxas, Jiaxuan You, Julian McAuley, Mengdi Wang, Jiawei Han, Philip S. Yu, Ziyi Ye - Licenses: document CC-BY-4.0 ### Software Engineering for and with GUI Agent Surveys 336 GUI-agent papers through a software engineering lens, covering architectures, evaluation, and lifecycle concerns in mobile and web settings. - https://awesomecua.com/resources/arxiv-2608-09278/ · source: https://arxiv.org/abs/2608.09278v1 - Survey · reviewed 2026-11-06 · environments: web · topics: survey, evaluation, observability - Published: 2026-08-10 · Authors: Shengcheng Yu, Yuchen Ling, Junyang Xing, Quan Zhou, Chunrong Fang, Zhenyu Chen ### The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration Surveys multi-tool LLM agents across planning, training, safety, efficiency, and evaluation, including GUI and mobile systems among its application areas. - https://awesomecua.com/resources/arxiv-2603-22862/ · source: https://arxiv.org/abs/2603.22862v2 - Survey · reviewed 2026-11-06 · environments: unspecified · topics: survey, tool-integration, planning, evaluation - Published: 2026-03-24 · Authors: Haoyuan Xu, Chang Li, Xinyan Ma, Xianhao Ou, Zihan Zhang, Tao He, Xiangyu Liu, Zixiang Wang, Jiafeng Liang, Zheng Chu, Runxuan Liu, Rongchuan Mu, Dandan Tu, Ming Liu, Bing Qin ### Towards Trustworthy GUI Agents: A Survey Frames GUI-agent trustworthiness as an execution gap and reviews failures, attacks, and defenses across perception, reasoning, and interaction. - https://awesomecua.com/resources/arxiv-2503-23434/ · source: https://arxiv.org/abs/2503.23434v2 - Survey · reviewed 2026-10-16 · environments: cross-platform · topics: survey, safety-security - Published: 2025-03-30 · Authors: Yucheng Shi, Wenhao Yu, Jingyuan Huang, Wenlin Yao, Wenhu Chen, Ninghao Liu ### Vision-Based Mobile App GUI Testing: A Survey Surveys vision-based mobile app GUI testing across test generation, record-and-replay and testing frameworks, drawing on 271 papers including 92 vision-based studies. - https://awesomecua.com/resources/arxiv-2310-13518/ · source: https://arxiv.org/abs/2310.13518v3 - Survey · reviewed 2026-11-06 · environments: unspecified · topics: survey, screen-understanding, recording, automation - Published: 2023-10-20 · Authors: Shengcheng Yu, Chunrong Fang, Ziyuan Tuo, Quanjun Zhang, Chunyang Chen, Zhenyu Chen, Zhendong Su ## Research papers (1,645) ### "Allow" to Achieve, Over-Privileged Inadvertently: The Unintended Cost of Task-Completion-Driven Pop-up Decisions in Mobile GUI Agents This permission-literacy study tests whether mobile GUI agents grant only task-necessary permissions. Controlled changes to task context and requester identity expose authorization biases and the trade-off between suppressing unnecessary and legitimate grants. - https://awesomecua.com/resources/arxiv-2608-04755/ · source: https://arxiv.org/abs/2608.04755v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, evaluation, robustness - Published: 2026-08-05 · Authors: Dongsheng Chen, Yuxuan Li, Guanhua Chen, Jiaxin Zhang, Xiangyu Zhao, Lei Ma, Xin Yao, Xuetao Wei ### "Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents This study tests vision-language models as task-completion judges using screenshots and instructions from 42 built-in macOS applications. It evaluates judgment accuracy on human-labeled tasks and separately studies how judge feedback affects agent task success. - https://awesomecua.com/resources/arxiv-2511-20067/ · source: https://arxiv.org/abs/2511.20067v1 - Paper · reviewed 2026-10-17 · environments: desktop, macos · topics: evaluation, self-reflection - Published: 2025-11-25 · Authors: Marta Sumyk, Oleksandr Kosovan ### "What Did It Actually Do?": Understanding Risk Awareness and Traceability for Computer-Use Agents Presents an interview study of computer-use agent risks and AgentTrace, a prototype interface showing agent actions, resource access, permission history, and lasting changes. - https://awesomecua.com/resources/arxiv-2603-28551/ · source: https://arxiv.org/abs/2603.28551v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, observability, ui-design - Published: 2026-03-30 · Authors: Zifan Peng, Mingchen Li ### (A)I Sees What You Don't: Exploiting New Attack Surfaces in Third-Party Mobile Agents Analyzes cross-app privilege escalation and visual attack surfaces introduced by third-party VLM-powered mobile agents. - https://awesomecua.com/resources/arxiv-2607-00333/ · source: https://arxiv.org/abs/2607.00333v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, mobile-navigation - Published: 2026-07-01 · Authors: Zidong Zhang, Zhentao Xie, Wenrui Diao, Jianliang Wu ### 3Dify: a Framework for Procedural 3D-CG Generation Assisted by LLMs Using MCP and RAG Introduces 3Dify, a natural-language 3D graphics framework that operates creation tools through MCP and uses a computer-using agent for GUI operations when MCP is unavailable. - https://awesomecua.com/resources/arxiv-2510-04536/ · source: https://arxiv.org/abs/2510.04536v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, tool-integration, science - Published: 2025-10-06 · Authors: Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Satoshi Ohshima, Takahiro Katagiri ### \textsc{GUI-Spotlight}: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding GUI-Spotlight trains a grounding model to reason over images and call tools that iteratively narrow its visual focus. The paper evaluates this refinement method on ScreenSpot-Pro after training on 18,500 samples. - https://awesomecua.com/resources/arxiv-2510-04039/ · source: https://arxiv.org/abs/2510.04039v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, planning, tool-integration - Published: 2025-10-05 · Authors: Bin Lei, Nuo Xu, Ali Payani, Mingyi Hong, Chunhua Liao, Yu Cao, Caiwen Ding ### A Baseline Assessment for the Agentic Web Technology Stack Presents MASTIF, a framework for comparing combinations of agent models, protocols, and implementation frameworks on web tasks, with an initial assessment using 300 Mind2Web tasks. - https://awesomecua.com/resources/arxiv-2610-13703/ · source: https://arxiv.org/abs/2610.13703v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation - Published: 2026-10-09 · Authors: Vagner Figueredo de Santana, Diogo Cortiz ### A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets Introduces a benchmark and online evaluation method for predicting the next spreadsheet action from a user's action history, using curated action sequences. - https://awesomecua.com/resources/arxiv-2606-13802/ · source: https://arxiv.org/abs/2606.13802v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: action-prediction, interaction-history, evaluation, data-generation - Published: 2026-06-11 · Authors: Tejas Agrawal, Vu Le, Sumit Gulwani, Gust Verbruggen ### A data-driven approach for learning to control computers Combines reinforcement learning with behavioral priors from human interactions to learn computer control through keyboard and mouse actions. - https://awesomecua.com/resources/arxiv-2202-08137/ · source: https://arxiv.org/abs/2202.08137v2 - Paper · reviewed 2026-10-16 · environments: web · topics: reinforcement-learning, automation - Published: 2022-02-16 · Authors: Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, Timothy Lillicrap ### A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility Introduces MoTIF mobile-app interactions with feasible and infeasible commands, feasibility explanations, and follow-up questions for ambiguous requests. - https://awesomecua.com/resources/arxiv-2202-02312/ · source: https://arxiv.org/abs/2202.02312v3 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, human-in-the-loop, evaluation - Published: 2022-02-04 · Authors: Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A. Plummer ### A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolution Introduces DeskIF-Bench to study how desktop application interfaces affect agents, and ANIS to synthesize interfaces from installed application capabilities. - https://awesomecua.com/resources/arxiv-2610-15765/ · source: https://arxiv.org/abs/2610.15765v1 - Paper · reviewed 2026-11-06 · environments: desktop · topics: evaluation, ui-design, tool-integration - Published: 2026-10-11 · Authors: Alan Wu, Wei Tao, Yiwen Guo ### A Dual-Process Perspective on Nudge Susceptibility in LLM-Based GUI Agents Studies how automatic defaults and reflective social-influence cues affect GUI-agent shopping decisions under different reasoning configurations. A randomized simulated-shopping experiment finds that reasoning changes the type of susceptibility rather than uniformly removing interface influence. - https://awesomecua.com/resources/arxiv-2609-19843/ · source: https://arxiv.org/abs/2609.19843v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, evaluation, ui-design, robustness - Published: 2026-09-17 · Authors: Haya Halimeh, Sascha Kaltenpoth, Kevin Bösch, Oliver Müller ### A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains Amazon-Bench generates functionality-specific web tasks from page content and interactive controls, covering operations beyond product search. Its evaluator checks task completion and unintended changes to account state, such as address or purchase settings. - https://awesomecua.com/resources/arxiv-2508-15832/ · source: https://arxiv.org/abs/2508.15832v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, safety-security, data-generation, web-navigation - Published: 2025-08-18 · Authors: Xianren Zhang, Shreyas Prasad, Di Wang, Qiuhai Zeng, Suhang Wang, Wenbo Yan, Mat Hans ### A History-Aware Visually Grounded Critic for Computer Use Agents HiViG combines a compact history of completed actions with a visually grounded critic that checks proposed coordinates before execution. Its evaluations separate history-related planning failures from errors caused by targeting the wrong screen element. - https://awesomecua.com/resources/arxiv-2606-11078/ · source: https://arxiv.org/abs/2606.11078v1 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: interaction-history, memory, grounding, reward-models, long-horizon, evaluation - Published: 2026-06-09 · Authors: Jaewoo Lee, Zaid Khan, Archiki Prasad, Justin Chih-Yao Chen, Supriyo Chakraborty, Kartik Balasubramaniam, Sambit Sahu, Elias Stengel-Eskin, Hyunji Lee, Mohit Bansal ### A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism Studies a controlled setting where GRPO fails to improve a strong small web agent and can reduce text-track performance. Matched reachable-reward tasks and component grafting distinguish limited optimization headroom from learning-rate-dependent failure mechanisms. - https://awesomecua.com/resources/arxiv-2607-12640/ · source: https://arxiv.org/abs/2607.12640v1 - Paper · reviewed 2026-10-18 · environments: web · topics: reinforcement-learning, evaluation, robustness - Published: 2026-07-14 · Authors: Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, Shiwen Ni, Qinghao Zhang ### A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants Presents an MCP architecture that exposes an application's GUI navigation graph and view-specific tools to speech-enabled LLM assistants for voice interaction. - https://awesomecua.com/resources/arxiv-2510-06223/ · source: https://arxiv.org/abs/2510.06223v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: tool-integration, ui-design - Published: 2025-08-31 · Authors: Hans G. W. van Dam ### A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis Combines task decomposition, HTML summarization, and generated Python programs to carry out instructions on websites. - https://awesomecua.com/resources/arxiv-2307-12856/ · source: https://arxiv.org/abs/2307.12856v4 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, planning - Published: 2023-07-24 · Authors: Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, Aleksandra Faust ### A Subgoal-driven Framework for Improving Long-Horizon LLM Agents The framework separates inference-time subgoal planning from MiRA, a reinforcement-learning method that supplies milestone rewards. WebArena-Lite experiments evaluate these mechanisms for long-horizon web navigation. - https://awesomecua.com/resources/arxiv-2603-19685/ · source: https://arxiv.org/abs/2603.19685v1 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, reinforcement-learning, reward-models, long-horizon, evaluation - Published: 2026-03-20 · Authors: Taiyi Wang, Sian Gooding, Florian Hartmann, Oriana Riva, Edward Grefenstette ### A Systematization of Security Vulnerabilities in Computer Use Agents Analyzes security boundaries in computer-use agents through a risk taxonomy and concrete adversarial case studies. It links observed failures to input provenance, interface-action binding, memory, and delegation, and proposes CUA-specific evaluation principles. - https://awesomecua.com/resources/arxiv-2507-05445/ · source: https://arxiv.org/abs/2507.05445v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation - Published: 2025-07-07 · Authors: Daniel Jones, Giorgio Severi, Martin Pouliot, Gary Lopez, Joris de Gruyter, Santiago Zanella-Beguelin, Justin Song, Blake Bullwinkel, Pamela Cortez, Amanda Minnich ### A Task-State Representation for Long-Horizon Mobile GUI Agents Introduces TSR, a training-free wrapper for long-horizon mobile GUI agents that tracks instructions, subgoal progress, and action outcomes separately from screen observations. - https://awesomecua.com/resources/arxiv-2607-00502/ · source: https://arxiv.org/abs/2607.00502v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: state-tracking, training-free, long-horizon, mobile-navigation - Published: 2026-07-01 · Authors: Yujie Zheng, Zikang Liu, Xin Zhao, Ji-Rong Wen ### A Training-Free, Task-Agnostic Framework for Enhancing MLLM Performance on High-Resolution Images Introduces Extract Candidate then Predict (ECP), a training-free two-stage method for high-resolution images, evaluated partly on 4K GUI grounding. - https://awesomecua.com/resources/arxiv-2507-10202/ · source: https://arxiv.org/abs/2507.10202v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: training-free, grounding, screen-understanding - Published: 2025-07-14 · Authors: Jaeseong Lee, Yeeun Choi, Heechan Choi, Hanjung Kim, Seonjoo Kim ### A Trajectory-Based Safety Audit of Clawdbot (OpenClaw) Audits Clawdbot safety through logged action and tool-call trajectories across local execution and web-mediated workflows, using automated and human review. - https://awesomecua.com/resources/arxiv-2602-14364/ · source: https://arxiv.org/abs/2602.14364v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, observability, workflow - Published: 2026-02-16 · Authors: Tianyu Chen, Dongrui Liu, Xia Hu, Jingyi Yu, Wenjie Wang ### A Zero-Shot Language Agent for Computer Control with Structured Reflection Studies a zero-shot computer-control agent that plans executable actions and organizes its own reflections after mistakes. MiniWoB++ experiments examine learning to complete tasks without supplied expert trajectories. - https://awesomecua.com/resources/arxiv-2310-08740/ · source: https://arxiv.org/abs/2310.08740v3 - Paper · reviewed 2026-10-17 · environments: web · topics: self-reflection, planning, web-navigation - Published: 2023-10-12 · Authors: Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, Yang Li ### A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction A11y-Compressor transforms linearized accessibility trees through modal detection, redundancy reduction and semantic structuring. Its OSWorld evaluation measures both observation-token reduction and task success. - https://awesomecua.com/resources/arxiv-2605-00551/ · source: https://arxiv.org/abs/2605.00551v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: screen-understanding, efficiency, grounding, evaluation - Published: 2026-05-01 · Authors: Michito Takeshita, Takuro Kawada, Takumi Ohashi, Shunsuke Kitada, Hitoshi Iyatomi ### A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents Introduces A11y-CUA, a dataset comparing how blind or low-vision users and sighted users perform everyday computer tasks. It contrasts those traces with agent behavior under default, keyboard-only, and magnifier conditions to identify barriers to shared control. - https://awesomecua.com/resources/arxiv-2602-09310/ · source: https://arxiv.org/abs/2602.09310v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, evaluation, data-generation - Published: 2026-02-10 · Authors: Ananya Gubbi Mohanbabu, Rosiana Natalie, Brandon Kim, Anhong Guo, Amy Pavel ### A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation Introduces Android Agent Arena with online application tasks and procedural evaluation based on reaching essential intermediate states. - https://awesomecua.com/resources/arxiv-2501-01149/ · source: https://arxiv.org/abs/2501.01149v3 - Paper · reviewed 2026-10-16 · environments: android · topics: evaluation, mobile-navigation - Published: 2025-01-02 · Authors: Yuxiang Chai, Shunye Tang, Han Xiao, Weifeng Lin, Hanhao Li, Jiayu Zhang, Liang Liu, Pengxiang Zhao, Guangyi Liu, Guozhi Wang, Shuai Ren, Rongduo Han, Haining Zhang, Siyuan Huang, Hongsheng Li ### Absorbed in Inertia: Activation Analysis for Computer-Use Agents Studies repetitive, unproductive actions in desktop agents through their internal activation states. It proposes Reset, Reroute, Restore, a context intervention that temporarily resets an agent’s trajectory and later restores it to help escape repeated ineffective behavior. - https://awesomecua.com/resources/arxiv-2609-37176/ · source: https://arxiv.org/abs/2609.37176v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: robustness, observability, memory - Published: 2026-09-29 · Authors: Giulio Segalini, Zhi Wen Soi, Jérémie Decouchant, Lydia Chen ### AcademiClaw: When Students Set Challenges for AI Agents AcademiClaw benchmarks OpenClaw agents on long academic workflows, scoring task completion in isolated Docker sandboxes with rubrics and a safety audit. - https://awesomecua.com/resources/arxiv-2605-02661/ · source: https://arxiv.org/abs/2605.02661v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon, sandbox, safety-security - Published: 2026-05-04 · Authors: Junjie Yu, Pengrui Lu, Weiye Si, Hongliang Lu, Jiabao Wu, Kaiwen Tao, Kun Wang, Lingyu Yang, Qiran Zhang, Xiuting Guo, Xuanyu Wang, Yang Wang, Yanjie Wang, Yi Yang, Zijian Hu, Ziyi Yang, Zonghan Zhou, Binghao Qiang, Borui Zhang, Chenning Li, Enchang Zhang, Feifan Chen, Feng Jian, Fengyin Sun, Hao Qiu, Hao Zheng, Haoran Zhu, Hongyu Liu, Jianbin Deng, Jiaxin Song, Jiaying Chi, Jiayou Shi, Jie Fang, Jinghui Zhong, Jingyu Zhou, Jinze Li, Junfeng Yi, Junyan Yu, Junzhi Xue, Ni Song, Pengyi Chen, Qi Chen, Quansheng Li, Rui Tao, Shenghai Gong, Shenhang Lu, Tianqi Shen, Tianxiang Zhu, Tiehan Kang, Tingyu Li, Wendi Wu, Xiao Shen, Xiao Zhou, Xiaotao Zhang, Xinrong Li, Xuankun Yang, Xun Zhang, Yan Li, Ye Lu, Yi Wang, Yibo Zhou, Yichi Zhang, Yihao Sun, Yijun Huang, Yixin Zhu, Yixuan Wu, Yuchen Sun, Yue Wu, Yuheng Sun, Yukun Li, Yutian Tu, Yuxuan Qin, Yuzhuo Wu, Zeyu Li, Zhengyu Lou, Zhenning Ran, Zizhu He, Pengfei Liu ### Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions Introduces intention understanding, prediction, and confirmed execution tasks over mobile GUI trajectories, with a proactive agent combining personalized intention models and experience-guided actions. - https://awesomecua.com/resources/arxiv-2608-14132/ · source: https://arxiv.org/abs/2608.14132v1 - Paper · reviewed 2026-10-18 · environments: android · topics: human-in-the-loop, planning, mobile-navigation, evaluation - Published: 2026-08-14 · Authors: Xiaokai Yan, Jingtao Ding, Yong Li, Zhiwen Yu ### Action Conditioned Bisimulation For GUI Agent Memory Defines a GUI-memory merge rule that treats two states as equivalent only when shared actions lead to matching outcomes and successor states. The training-free method replaces observation-similarity merging and is tested in closed-loop MiniWoB++ interaction. - https://awesomecua.com/resources/arxiv-2609-38778/ · source: https://arxiv.org/abs/2609.38778v1 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, web-navigation - Published: 2026-09-30 · Authors: Hongbo Zhang, Liuyang Song, Quanquan Li, Daqian Yang, Yan Wen, Zhengtao Yao ### ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces Pretrains ActionBert on user interaction traces to combine visual, linguistic, and UI-structure features. It evaluates the resulting UI representations on tasks including icon classification and description-based component retrieval. - https://awesomecua.com/resources/arxiv-2012-12350/ · source: https://arxiv.org/abs/2012.12350v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding, action-prediction - Published: 2020-12-22 · Authors: Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, Jindong Chen, Blaise Agüera y Arcas ### ActionEngine: From Reactive to Programmatic Web Agents via State Machine Memory ActionEngine explores web applications to build an updateable state-machine memory of GUI states, operations and transitions. It generates complete task programs from that memory, using reactive repair to handle changed interfaces or missing knowledge. - https://awesomecua.com/resources/arxiv-2602-20502/ · source: https://arxiv.org/abs/2602.20502v2 - Paper · reviewed 2026-10-18 · environments: web · topics: memory, planning, efficiency, web-navigation - Published: 2026-02-24 · Authors: Hongbin Zhong, Fazle Faisal, Luis França, Tanakorn Leesatapornwongsa, Adriana Szekeres, Kexin Rong, Suman Nath ### Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay Introduces Activity Frames, a deterministic pipeline that turns captured screen activity into typed episodes for computer-use agent memory and routine replay. - https://awesomecua.com/resources/arxiv-2608-05784/ · source: https://arxiv.org/abs/2608.05784v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: recording, memory, automation, efficiency - Published: 2026-08-06 · Authors: Nossa Iyamu ### ActReal: System-Level Mobile Agents Challenge Mobile Automation Detection Introduces ActReal, which translates semantic mobile-agent actions into coordinated touch and sensor signals to study automation-detection weaknesses. It evaluates detectors that observe touch events alone or together with inertial measurements under a privileged executor threat model. - https://awesomecua.com/resources/arxiv-2608-30038/ · source: https://arxiv.org/abs/2608.30038v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, mobile-navigation, evaluation - Published: 2026-08-30 · Authors: Mingshuo Wang, Hanqing Guo, Huining Li, Yuliang Fu, Jing Xu, Chenhan Xu ### AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding Introduces AdaGround, a training-free GUI grounding method that adjusts screenshot zoom to target size and uses a click verifier to refine or reject predictions. - https://awesomecua.com/resources/arxiv-2610-36874/ · source: https://arxiv.org/abs/2610.36874v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, training-free, self-reflection - Published: 2026-10-27 · Authors: Bingchao Wang, Yongcheng Jing, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao ### AdaPlanner: Adaptive Planning from Feedback with Language Models Introduces AdaPlanner, which revises code-structured plans in response to environmental feedback and reuses successful plans as exemplars. Experiments include MiniWoB++ web tasks alongside ALFWorld. - https://awesomecua.com/resources/arxiv-2305-16653/ · source: https://arxiv.org/abs/2305.16653v1 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, self-reflection, evaluation - Published: 2023-05-26 · Authors: Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, Chao Zhang ### AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations Adapts multimodal web agents to unfamiliar websites using a small number of human demonstrations through prompting or meta-learning. - https://awesomecua.com/resources/arxiv-2411-13451/ · source: https://arxiv.org/abs/2411.13451v1 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, data-generation - Published: 2024-11-20 · Authors: Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, Manuela Veloso ### AdaptArena: Evaluating Test-Time Personalization of Web Agents Introduces AdaptArena to test whether web agents can infer unstated user preferences from relevant past trajectories and carry those preferences into new tasks. Its AdaptiveAgent baseline separates preference-retrieval challenges from failures in downstream web execution. - https://awesomecua.com/resources/arxiv-2609-36488/ · source: https://arxiv.org/abs/2609.36488v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, memory, web-navigation - Published: 2026-09-29 · Authors: Dongchan Shin, Xing Han Lù, Jiaqi Deng, Jay Gala, Tomás Vergara Browne, Jaewon Moon, Fengyuan Liu, Alexandre Drouin, Siva Reddy, Alexandre Lacoste ### Adaptation of Computer Use Agents to New Software with User Manuals MAGE retrieves software-manual sections into fixed execution plans and generates setup/main task-plan pairs whose GUI rollouts support application-specific policy adaptation with round-based GRPO. - https://awesomecua.com/resources/arxiv-2610-23012/ · source: https://arxiv.org/abs/2610.23012v1 - Paper · reviewed 2026-10-20 · environments: desktop · topics: planning, data-generation, reinforcement-learning - Published: 2026-10-16 · Authors: Chulhoon Yoon, Kyungri Park, Jinyeong Yun, Daeyun Jeong, Sukmin Yun, Woohwan Jung ### Adaptive Milestone Reward for GUI Agents ADMIRE extracts milestones from successful GUI explorations to provide adaptive intermediate rewards. It assigns credit differently to successful and failed trajectories and evaluates training on AndroidWorld, with additional transfer studies in web and embodied settings. - https://awesomecua.com/resources/arxiv-2602-11524/ · source: https://arxiv.org/abs/2602.11524v1 - Paper · reviewed 2026-10-17 · environments: android, web · topics: reinforcement-learning, reward-models, mobile-navigation - Published: 2026-02-12 · Authors: Congmin Zheng, Xiaoyun Mo, Xinbei Ma, Qiqiang Lin, Yin Zhao, Jiachen Zhu, Xingyu Lou, Jun Wang, Zhaoxiang Wang, Weiwen Liu, Zhuosheng Zhang, Yong Yu, Weinan Zhang ### Adaptive Vision-Language Model Routing for Computer Use Agents Adaptive VLM Routing estimates GUI-action difficulty and small-model confidence to select a model under a predicted reliability threshold. Grounding and agent-routing benchmarks assess the cost–accuracy trade-off, with prior interaction context used for warm agents. - https://awesomecua.com/resources/arxiv-2603-12823/ · source: https://arxiv.org/abs/2603.12823v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: efficiency, grounding, memory, evaluation - Published: 2026-03-13 · Authors: Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen ### Adaptive-Group Process Rewards for GUI Agents Groups GUI trajectories by subgoal and combines action-level and group-level assessments into intermediate rewards for training agents with sparse task outcomes. - https://awesomecua.com/resources/arxiv-2610-17710/ · source: https://arxiv.org/abs/2610.17710v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reinforcement-learning, long-horizon, planning - Published: 2026-10-13 · Authors: Zirui Shang, Yang Liu, Xin Shu, Zhi Gao, Xinxiao Wu, Lifeng Fan ### AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement Combines instruction rewriting with conditional second-stage zooming for small GUI targets. A grounding dataset and GRPO training support joint prediction of click coordinates and element bounding boxes. - https://awesomecua.com/resources/arxiv-2603-17441/ · source: https://arxiv.org/abs/2603.17441v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, screen-understanding - Published: 2026-03-18 · Authors: Siqi Pei, Liang Tang, Tiaonan Duan, Long Chen, Shuxian Li, Kaer Huang, Yanzhe Jing, Yiqiang Yan, Bo Zhang, Chenghao Jiang, Borui Zhang, Jiwen Lu ### ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices Evaluates interface-embedded threats and whether computer-use agents seek clarification when mobile or desktop instructions are ambiguous. - https://awesomecua.com/resources/arxiv-2608-26204/ · source: https://arxiv.org/abs/2608.26204v2 - Paper · reviewed 2026-10-16 · environments: desktop, cross-platform · topics: safety-security, evaluation, human-in-the-loop - Published: 2026-08-25 · Authors: Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro, Yuxuan Sun, Chloe Evans, Joseph Tighe ### AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery Studies advertising delivery as an indirect-injection channel for web agents under a black-box threat model. AdInject evaluates how static ad content shaped by website context can redirect browser actions. - https://awesomecua.com/resources/arxiv-2505-21499/ · source: https://arxiv.org/abs/2505.21499v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2025-05-27 · Authors: Haowei Wang, Junjie Wang, Xiaojun Jia, Rupeng Zhang, Mingyang Li, Zhe Liu, Yang Liu, Qing Wang ### AdvAgent: Controllable Blackbox Red-teaming on Web Agents Introduces a black-box web-agent red-teaming framework that learns adversarial prompts from agent feedback to study security weaknesses. - https://awesomecua.com/resources/arxiv-2410-17401/ · source: https://arxiv.org/abs/2410.17401v4 - Paper · reviewed 2026-10-16 · environments: web · topics: safety-security, evaluation - Published: 2024-10-22 · Authors: Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, Bo Li ### Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learning Uses variational subgoal-conditioned reinforcement learning to improve learning efficiency on sparse-reward mobile and web-control tasks. - https://awesomecua.com/resources/arxiv-2502-07949/ · source: https://arxiv.org/abs/2502.07949v2 - Paper · reviewed 2026-10-16 · environments: web, cross-platform · topics: reinforcement-learning, planning - Published: 2025-02-11 · Authors: Qingyuan Wu, Jianheng Liu, Jianye Hao, Jun Wang, Kun Shao ### Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization Introduces CollabUIAgents, a multi-agent RL framework that assigns LLM-based process rewards to help agents generalize across interactive settings including mobile operations and web browsing. - https://awesomecua.com/resources/arxiv-2502-14496/ · source: https://arxiv.org/abs/2502.14496v3 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, reward-models, robustness - Published: 2025-02-20 · Authors: Zhitao He, Zijun Liu, Peng Li, Yi R. Fung, Ming Yan, Ji Zhang, Fei Huang, Yang Liu ### Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment Introduces V-Droid, which verifies candidate actions in a discrete action space rather than directly generating each mobile operation. - https://awesomecua.com/resources/arxiv-2503-15937/ · source: https://arxiv.org/abs/2503.15937v5 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, efficiency, planning - Published: 2025-03-20 · Authors: Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, Lili Qiu ### Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution Introduces WebMirage, a visual red-teaming framework that models webpage element competition and action post-processing to test whether manipulated grounding decisions lead to executed browser actions. - https://awesomecua.com/resources/arxiv-2610-09240/ · source: https://arxiv.org/abs/2610.09240v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, grounding, evaluation - Published: 2026-10-07 · Authors: Wanjing Han, Levi Taiji Li, Mu Zhang, Yue Jiang, Guanhong Tao ### AdvSim2Real : Training Web Agents Against Adaptive Prompt Injection in a Web World Model Presents AdvSim2Real, which jointly adapts web-task curricula, prompt-injection adversaries, and agent policies inside a frozen web world model. It evaluates robustness against an unseen adversary and transfer from simulated interaction to a real browser. - https://awesomecua.com/resources/arxiv-2610-08773/ · source: https://arxiv.org/abs/2610.08773v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, world-models, reinforcement-learning - Published: 2026-10-06 · Authors: Sarim Hashmi, Mukul Ranjan, Kshitij Mishra, Mikhail Kuznetsov, Praneeth Vepakomma, Nils Lukas ### Affora: A Design System for Agent-Friendly Interfaces Presents Affora, a design system with reusable components and executable checks for interfaces used by computer-use agents, informed by studies of components and visual variation. - https://awesomecua.com/resources/arxiv-2609-19125/ · source: https://arxiv.org/abs/2609.19125v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: ui-design, evaluation - Published: 2026-09-16 · Authors: Jin Gao ### AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent Introduces AFRAgent, which uses adaptive feature renormalization to combine low-resolution visual embeddings with high-resolution details. The architecture is evaluated for smartphone GUI automation on Meta-GUI and AITW. - https://awesomecua.com/resources/arxiv-2512-00846/ · source: https://arxiv.org/abs/2512.00846v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, mobile-navigation, efficiency - Published: 2025-11-30 · Authors: Neeraj Anand, Rishabh Jain, Sohan Patnaik, Balaji Krishnamurthy, Mausoom Sarkar ### Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents Agent Alpha uses step-level Monte Carlo tree search to reuse successful prefixes and recover from early GUI mistakes. Comparison-based evaluation and diversity-constrained expansion guide the search, with experiments on OSWorld. - https://awesomecua.com/resources/arxiv-2602-02995/ · source: https://arxiv.org/abs/2602.02995v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: planning, efficiency, evaluation - Published: 2026-02-03 · Authors: Sizhe Tang, Rongqian Chen, Tian Lan ### Agent Data Injection Attacks are Realistic Threats to AI Agents This study examines agent data injection, where untrusted content impersonates trusted metadata or tool-context data. It reports arbitrary-click vulnerabilities in web agents alongside separate coding-agent attacks, highlighting trust-boundary failures beyond instruction injection. - https://awesomecua.com/resources/arxiv-2607-05120/ · source: https://arxiv.org/abs/2607.05120v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, robustness - Published: 2026-07-06 · Authors: Woohyuk Choi, Juhee Kim, Taehyun Kang, Jihyeon Jeong, Luyi Xing, Byoungyoung Lee ### Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents The agent data protocol (ADP) provides a shared format for agent training data across browsing, tool use, coding, and other tasks, including web agent workflows. - https://awesomecua.com/resources/arxiv-2510-24702/ · source: https://arxiv.org/abs/2510.24702v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, workflow - Published: 2025-10-28 · Authors: Yueqi Song, Ketan Ramaneti, Zaid Sheikh, Ziru Chen, Boyu Gou, Tianbao Xie, Yiheng Xu, Danyang Zhang, Apurva Gandhi, Fan Yang, Joseph Liu, Tianyue Ou, Zhihao Yuan, Frank Xu, Shuyan Zhou, Xingyao Wang, Xiang Yue, Tao Yu, Huan Sun, Yu Su, Graham Neubig ### Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling Agent JIT compilation turns browser task descriptions into validated executable plans containing model calls, tools and parallel work. A cost-based scheduler and pre/postcondition protocol target latency and incorrect tool use in web-agent execution. - https://awesomecua.com/resources/arxiv-2605-21470/ · source: https://arxiv.org/abs/2605.21470v2 - Paper · reviewed 2026-10-18 · environments: web · topics: planning, efficiency, tool-integration, web-navigation - Published: 2026-05-20 · Authors: Caleb Winston, Ron Yifeng Wang, Azalia Mirhoseini, Christos Kozyrakis ### Agent MechSuits: Mechanistic Subspace Safety Steering for Multi-Turn CLI Agents Introduces Agent MechSuits, a framework that detects unsafe states and steers multi-turn CLI agents through hidden-representation interventions, alongside the MAS trajectory benchmark. - https://awesomecua.com/resources/arxiv-2606-22673/ · source: https://arxiv.org/abs/2606.22673v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, observability - Published: 2026-06-21 · Authors: Weidi Luo, Qiming Zhang, Yihao Quan, Mingyu Jin, Jie Cai, Chaowei Xiao, Jingcheng Niu, Zhen Xiang ### Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents Agent Q combines guided tree search, self-critique and preference-based fine-tuning to train web agents from successful and unsuccessful interactions. - https://awesomecua.com/resources/arxiv-2408-07199/ · source: https://arxiv.org/abs/2408.07199v1 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, self-reflection, imitation-learning, web-navigation - Published: 2024-08-13 · Authors: Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, Rafael Rafailov ### Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents Combines generalist and specialist models with hierarchical planning and multiple grounding methods for computer-use tasks. - https://awesomecua.com/resources/arxiv-2504-00906/ · source: https://arxiv.org/abs/2504.00906v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning - Published: 2025-04-01 · Authors: Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, Xin Eric Wang ### Agent S: An Open Agentic Framework that Uses Computers Like a Human Combines hierarchical planning, retrieved experience, and a computer interaction interface to execute multistep GUI tasks. - https://awesomecua.com/resources/arxiv-2410-08164/ · source: https://arxiv.org/abs/2410.08164v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: planning, memory - Published: 2024-10-10 · Authors: Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang ### Agent Workflow Memory Agent Workflow Memory extracts reusable task routines and selectively supplies them to guide later web actions. It studies both offline workflow induction from training examples and online induction from test experience on Mind2Web and WebArena. - https://awesomecua.com/resources/arxiv-2409-07429/ · source: https://arxiv.org/abs/2409.07429v1 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, workflow, web-navigation - Published: 2024-09-11 · Authors: Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, Graham Neubig ### Agent World Model in the Wild: Computer-Use Agent Environment Synthesis WildAWM synthesizes resettable GUI/MCP/CLI environments and cross-interface tasks, using execution-based checks of solvability, difficulty and verifier consistency before supervised and reinforcement learning. - https://awesomecua.com/resources/arxiv-2610-26828/ · source: https://arxiv.org/abs/2610.26828v1 - Paper · reviewed 2026-10-20 · environments: web, desktop, linux · topics: data-generation, sandbox, tool-integration, long-horizon, imitation-learning, reinforcement-learning - Published: 2026-10-19 · Authors: Zhaoyang Wang, Qianhui Wu, Haorui Wang, Simon Yu, Baolin Peng, Canwen Xu, Hao Cheng, Wenlin Yao, Boyi Liu, Yite Wang, Jianfeng Gao, Huaxiu Yao ### Agent+P: Guiding UI Agents via Symbolic Planning Guides UI agents with symbolic planning over a UI transition graph initialized from app-code analysis and updated through online exploration, with evaluation on AndroidWorld. - https://awesomecua.com/resources/arxiv-2510-06042/ · source: https://arxiv.org/abs/2510.06042v3 - Paper · reviewed 2026-10-18 · environments: android · topics: planning, long-horizon, mobile-navigation - Published: 2025-10-07 · Authors: Shang Ma, Xusheng Xiao, Yanfang Ye ### Agent-Computer Observation Interfaces Enable Dynamic Computer Use The Agent-Computer Observation Interface separates continuous perception from discrete computer actions using captured frames, audio transcription and persistent visual narration. DynaCU-Bench tests these components on dynamic browser tasks and static controls, with model-specific ablations showing that additional observations can also hurt. - https://awesomecua.com/resources/arxiv-2606-29472/ · source: https://arxiv.org/abs/2606.29472v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, video, screen-understanding, memory, training-free - Published: 2026-06-28 · Authors: Bojie Li, Noah Shi ### Agent-Dice: Disentangling Knowledge Updates via Geometric Consensus for Agent Continual Learning Agent-Dice fuses parameter updates to support continual learning in LLM agents, with GUI agent tasks evaluated alongside tool-use domains. - https://awesomecua.com/resources/arxiv-2601-03641/ · source: https://arxiv.org/abs/2601.03641v4 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: robustness, efficiency - Published: 2026-01-07 · Authors: Zheng Wu, Xingyu Lou, Xinbei Ma, Yansi Li, Weiwen Liu, Weinan Zhang, Jun Wang, Zhuosheng Zhang ### Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems Agent-E combines hierarchical web-task control with DOM distillation and observations of interface changes. WebVoyager experiments inform a broader discussion of primitive skills, observation reduction, and self-improvement in agent architecture. - https://awesomecua.com/resources/arxiv-2407-13032/ · source: https://arxiv.org/abs/2407.13032v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, planning, efficiency, state-tracking - Published: 2024-07-17 · Authors: Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, Ravi Kokku ### Agent-Initiated Interaction in Phone UI Automation Defines when phone agents should request user interaction and introduces AndroidInteraction annotations for deciding whether and how to engage the user. - https://awesomecua.com/resources/arxiv-2503-19537/ · source: https://arxiv.org/abs/2503.19537v1 - Paper · reviewed 2026-10-16 · environments: android · topics: human-in-the-loop, mobile-navigation, evaluation - Published: 2025-03-25 · Authors: Noam Kahlon, Guy Rom, Anatoly Efros, Filippo Galgani, Omri Berkovitch, Sapir Caduri, William E. Bishop, Oriana Riva, Ido Dagan ### Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training Builds self-critique training examples from searched recovery paths to improve agents' ability to detect and correct errors during interaction. - https://awesomecua.com/resources/arxiv-2501-11425/ · source: https://arxiv.org/abs/2501.11425v3 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, planning, robustness - Published: 2025-01-20 · Authors: Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, Jiecao Chen ### Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents Introduces Agent-RewardBench to evaluate step-level reward models for multimodal agents across perception, planning, and safety; web navigation is among its scenarios. - https://awesomecua.com/resources/arxiv-2506-21252/ · source: https://arxiv.org/abs/2506.21252v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, reward-models, planning, safety-security - Published: 2025-06-26 · Authors: Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao ### Agent-SAMA: State-Aware Mobile Assistant Models mobile application execution as a finite-state machine to support planning, verification, and recovery across interface transitions. - https://awesomecua.com/resources/arxiv-2505-23596/ · source: https://arxiv.org/abs/2505.23596v3 - Paper · reviewed 2026-10-16 · environments: android · topics: planning, memory, mobile-navigation - Published: 2025-05-29 · Authors: Linqiang Guo, Wei Liu, Yi Wen Heng, Tse-Hsun, Chen, Yang Wang ### Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks Introduces Agent-X, a benchmark of multi-step visual reasoning and tool use with step-level grading across six settings, including web browsing. - https://awesomecua.com/resources/arxiv-2505-24876/ · source: https://arxiv.org/abs/2505.24876v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, planning, tool-integration - Published: 2025-05-30 · Authors: Tajamul Ashraf, Amal Saqib, Hanan Ghani, Muhra AlMahri, Yuhao Li, Noor Ahsan, Umair Nawaz, Jean Lahoud, Hisham Cholakkal, Mubarak Shah, Philip Torr, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan ### AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents Introduces AgentA/B, which uses persona-based LLM agents to navigate real webpages and perform shopping interactions for simulated web A/B tests. - https://awesomecua.com/resources/arxiv-2504-09723/ · source: https://arxiv.org/abs/2504.09723v4 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, web-navigation, evaluation - Published: 2025-04-13 · Authors: Yuxuan Lu, Ting-Yao Hsu, Hansu Gu, Limeng Cui, Yaochen Xie, William Headden, Bingsheng Yao, Akash Veeragouni, Jiapeng Liu, Sreyashi Nag, Jessie Wang, Dakuo Wang ### AgentBench: Evaluating LLMs as Agents Evaluates language agents across eight interactive environments and analyzes failures in reasoning, decisions, and instruction following. - https://awesomecua.com/resources/arxiv-2308-03688/ · source: https://arxiv.org/abs/2308.03688v3 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, planning - Published: 2023-08-07 · Authors: Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, Jie Tang ### AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents Introduces AgentBoard, a multiturn agent evaluation framework with intermediate progress metrics and analysis tools. Its broader task suite includes WebShop and WebArena interactions with stepwise progress measurements. - https://awesomecua.com/resources/arxiv-2401-13178/ · source: https://arxiv.org/abs/2401.13178v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, observability, web-navigation - Published: 2024-01-24 · Authors: Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, Junxian He ### AgentClick: A Skill-Based Human-in-the-Loop Review Layer for Terminal AI Agents AgentClick gives terminal-based agents a browser review interface for plans, drafts, execution traces and editable memory. A local server and skill plugin let users inspect proposed actions and provide structured feedback. - https://awesomecua.com/resources/arxiv-2604-16520/ · source: https://arxiv.org/abs/2604.16520v1 - Paper · reviewed 2026-10-17 · environments: web · topics: human-in-the-loop, observability, workflow, memory, tool-integration - Published: 2026-04-15 · Authors: Haomin Zhuang, Hanwen Xing, Xiangliang Zhang ### AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning Combines grounding pretraining, bilingual interaction trajectories, and reinforcement fine-tuning for mobile GUI actions. - https://awesomecua.com/resources/arxiv-2506-01391/ · source: https://arxiv.org/abs/2506.01391v2 - Paper · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, mobile-navigation - Published: 2025-06-02 · Authors: Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Jie Xie, Wei Zhou, Wang Xu, Yuanheng Zhang, Zhou Su, Zhongwu Zhai, Xiaoming Liu, Yudong Mei, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, Maosong Sun ### AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents Evaluates whether autonomous web agents minimize use of sensitive personal information when completing tasks that expose unnecessary private data. - https://awesomecua.com/resources/arxiv-2503-09780/ · source: https://arxiv.org/abs/2503.09780v3 - Paper · reviewed 2026-10-16 · environments: web · topics: safety-security, evaluation - Published: 2025-03-12 · Authors: Arman Zharmagambetov, Chuan Guo, Ivan Evtimov, Maya Pavlova, Ruslan Salakhutdinov, Kamalika Chaudhuri ### AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents Introduces AgentHazard, a benchmark of harmful multi-step behavior by computer-use agents acting through tools, files, and execution environments. - https://awesomecua.com/resources/arxiv-2604-02947/ · source: https://arxiv.org/abs/2604.02947v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, long-horizon - Published: 2026-04-03 · Authors: Yunhao Feng, Yifan Ding, Yingshui Tan, Xingjun Ma, Yige Li, Yutao Wu, Yifeng Gao, Kun Zhai, Yanming Guo ### AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access Barriers Introduces AgentHeal, a benchmark of whether web agents detect access barriers, find legitimate alternative sources, and preserve the user's original goal. - https://awesomecua.com/resources/arxiv-2610-39234/ · source: https://arxiv.org/abs/2610.39234v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, robustness, web-navigation, safety-security - Published: 2026-10-29 · Authors: Zhiyi Liu, Haohua Du, Haoqi Sun, Haisheng Tan ### AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions Introduces AgentHijack, a benchmark for desktop computer-use agents facing pop-ups, resolution changes and other common corruptions, alongside an agent that checks its environment. - https://awesomecua.com/resources/arxiv-2605-25707/ · source: https://arxiv.org/abs/2605.25707v1 - Paper · reviewed 2026-11-06 · environments: desktop · topics: evaluation, robustness, observability - Published: 2026-05-25 · Authors: Jingwei Sun, Jianing Zhu, Yuanyi Li, Tongliang Liu, Xia HU, Bo Han ### AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents AgentHijack evaluates whether visual patch effects propagate from screenshots through model outputs and action parsing into real environment changes. Its controlled web-page experiments distinguish intermediate attack signals from verified end-to-end consequences. - https://awesomecua.com/resources/arxiv-2609-09212/ · source: https://arxiv.org/abs/2609.09212v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, robustness - Published: 2026-09-06 · Authors: Zhihao Liu, Hongyu Sun, Zhiyuan Fu, Xiaonan Duan, Jice Wang, Shangru Zhao, Weizhi Meng, Wuxin Yang, Yangfan Zhou, Yuqing Zhang ### AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks Introduces AgentHorizon, a benchmark for judging long computer-use trajectories using closely related instructions paired with recorded actions and screenshots. It tests whether judges can detect constraint violations and distinguish successful tasks from similar but incompatible requests. - https://awesomecua.com/resources/arxiv-2610-11050/ · source: https://arxiv.org/abs/2610.11050v1 - Paper · reviewed 2026-10-17 · environments: cross-platform · topics: evaluation, long-horizon, cross-app - Published: 2026-10-08 · Authors: Xing Han Lù, Dheeraj Vattikonda, Sina Hajimiri, Fatemeh Pesaran Zadeh, Parishad BehnamGhader, Ghazwa Darwiche, Amirhossein Kazemnejad, Christopher Pal, Alexandre Drouin, Siva Reddy ### Agentic Compilation: Mitigating the LLM Rerun Crisis for Minimized-Inference-Cost Web Automation Proposes Compile-and-Execute, which turns a sanitized DOM into a reusable JSON workflow so repetitive browser tasks can run without further model queries. - https://awesomecua.com/resources/arxiv-2604-09718/ · source: https://arxiv.org/abs/2604.09718v2 - Paper · reviewed 2026-11-06 · environments: web · topics: automation, efficiency, workflow, human-in-the-loop - Published: 2026-04-08 · Authors: Jagadeesh Chundru ### Agentic Critical Training Introduces Agentic Critical Training (ACT), a reinforcement-learning warm-up that teaches general agents to choose expert actions over alternatives, with WebShop among its evaluations. - https://awesomecua.com/resources/arxiv-2603-08706/ · source: https://arxiv.org/abs/2603.08706v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, imitation-learning, action-prediction - Published: 2026-03-09 · Authors: Weize Liu, Minghui Liu, Sy-Tuyen Ho, Yongkyun Lee, Andrew Adams Schoen, Souradip Chakraborty, Xiyao Wang, Furong Huang ### Agentic Entropy-Balanced Policy Optimization Introduces Agentic Entropy-Balanced Policy Optimization (AEPO), an RL method that adjusts rollout sampling and policy updates for tool-using web agents. - https://awesomecua.com/resources/arxiv-2510-14545/ · source: https://arxiv.org/abs/2510.14545v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, efficiency - Published: 2025-10-16 · Authors: Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jinghan Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, Zhicheng Dou ### Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements Introduces Agentic ESOpt, an evolution-strategies fine-tuning framework for long-horizon agents, evaluated on WebArena-Lite web tasks and automatic heuristic design. - https://awesomecua.com/resources/arxiv-2608-17310/ · source: https://arxiv.org/abs/2608.17310v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: long-horizon, efficiency, reinforcement-learning, evaluation - Published: 2026-08-18 · Authors: Zhi Zheng, Rongsheng Chen, Yunpeng Ba, Zhenkun Wang, Yee Whye Teh, Wee Sun Lee ### Agentic Lybic: Multi-Agent Execution System with Tiered Reasoning and Orchestration Introduces Agentic Lybic, a desktop automation system that routes tasks among GUI, code, and decision-support workers using a finite-state machine and evaluator. - https://awesomecua.com/resources/arxiv-2509-11067/ · source: https://arxiv.org/abs/2509.11067v2 - Paper · reviewed 2026-11-06 · environments: desktop · topics: automation, planning, tool-integration, workflow - Published: 2025-09-14 · Authors: Liangxuan Guo, Bin Zhu, Qingqian Tao, Kangning Liu, Xun Zhao, Xianzhe Qin, Jin Gao, Guangfu Hao ### Agentic Reinforcement Learning with Implicit Step Rewards Introduces iStar, a general agentic RL method that learns implicit step rewards; WebShop is one of several interactive benchmarks used to evaluate it. - https://awesomecua.com/resources/arxiv-2509-19199/ · source: https://arxiv.org/abs/2509.19199v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, reward-models, evaluation - Published: 2025-09-23 · Authors: Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, Jianbin Jiao ### Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction VAGEN verifies GUI trajectories by first extracting readily available evidence and escalating to tool-based environment probes when needed. Evaluation on OSWorld-Verified and AndroidWorld studies verification accuracy and its cost. - https://awesomecua.com/resources/arxiv-2602-00575/ · source: https://arxiv.org/abs/2602.00575v2 - Paper · reviewed 2026-10-17 · environments: desktop, android · topics: reward-models, tool-integration, evaluation, efficiency - Published: 2026-01-31 · Authors: Chaoqun Cui, Jing Huang, Shijing Wang, Liming Zheng, Qingchao Kong, Zhixiong Zeng ### Agentic Test-Time Scaling for WebAgents CATTS allocates additional inference computation when an agent’s action votes show uncertainty. The paper compares voting and arbiter strategies and evaluates this selective scaling on WebArena-Lite, Online-Mind2Web and GoBrowse. - https://awesomecua.com/resources/arxiv-2602-12276/ · source: https://arxiv.org/abs/2602.12276v2 - Paper · reviewed 2026-10-17 · environments: web · topics: efficiency, planning, evaluation, web-navigation - Published: 2026-02-12 · Authors: Nicholas Lee, Lutfi Eren Erdogan, Chris Joseph John, Surya Krishnapillai, Michael W. Mahoney, Kurt Keutzer, Amir Gholami ### AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents Introduces AgentLens, a mobile GUI agent that shows full, partial, or generated interface views during background task execution to support user interaction. - https://awesomecua.com/resources/arxiv-2604-20279/ · source: https://arxiv.org/abs/2604.20279v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: mobile-navigation, human-in-the-loop, ui-design - Published: 2026-04-22 · Authors: Jeonghyeon Kim, Byeongjun Joung, Junwon Lee, Joohyung Lee, Taehoon Min, Sunjae Lee ### AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents Studies how simplifying and aligning a web agent's observations and actions with language-model capabilities affects task completion. - https://awesomecua.com/resources/arxiv-2410-13825/ · source: https://arxiv.org/abs/2410.13825v2 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, planning - Published: 2024-10-17 · Authors: Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, Huzefa Rangwala ### AgentOS: From Application Silos to a Natural Language-Driven Data Ecosystem Proposes Personal Agent Operating System (AgentOS), with a natural language or voice portal and an Agent Kernel that coordinates agents and tasks in place of a GUI desktop. - https://awesomecua.com/resources/arxiv-2603-08938/ · source: https://arxiv.org/abs/2603.08938v2 - Paper · reviewed 2026-11-06 · environments: desktop · topics: planning, workflow, tool-integration - Published: 2026-03-09 · Authors: Rui Liu, Tao Zhe, Dongjie Wang, Zijun Yao, Kunpeng Liu, Yanjie Fu, Huan Liu, Jian Pei ### AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress AgentPRM scores sequential agent decisions by their promise and progress toward a goal, with web shopping and browser navigation among the motivating tasks. - https://awesomecua.com/resources/arxiv-2511-08325/ · source: https://arxiv.org/abs/2511.08325v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reward-models, planning, web-navigation - Published: 2025-11-11 · Authors: Zhiheng Xi, Chenyang Liao, Guanyu Li, Yajie Yang, Wenxiang Chen, Zhihao Zhang, Binghai Wang, Senjie Jin, Yuhao Zhou, Jian Guan, Wei Wu, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang ### AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Management Introduces AgentProg, organizing mobile-agent interaction history as a program with variables and control flow to decide what context to retain. A global belief state handles partial observability, with evaluation on AndroidWorld and extended long-horizon tasks. - https://awesomecua.com/resources/arxiv-2512-10371/ · source: https://arxiv.org/abs/2512.10371v2 - Paper · reviewed 2026-10-17 · environments: android · topics: interaction-history, memory, long-horizon - Published: 2025-12-11 · Authors: Shizuo Tian, Hao Wen, Yuxuan Chen, Jiacheng Liu, Shanhui Zhao, Guohong Liu, Ju Ren, Yunxin Liu, Yuanchun Li ### AgentProofBench: Benchmarking Reusable Verifiers for Web Agents Introduces AgentProofBench, a benchmark linking tests of reusable web-agent verification rules to subsequent execution from saved web states. - https://awesomecua.com/resources/arxiv-2610-34634/ · source: https://arxiv.org/abs/2610.34634v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, workflow - Published: 2026-10-25 · Authors: Jingjie Ning, Xueqi Li, Yi Lu, Yibo Kong ### AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents AgentRAE studies notification-based visual backdoors that redirect mobile GUI-agent actions. Its training separates sensitivity to icon differences from trigger/action association, with evaluations of clean behavior and targeted redirection. - https://awesomecua.com/resources/arxiv-2603-23007/ · source: https://arxiv.org/abs/2603.23007v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, robustness, evaluation - Published: 2026-03-24 · Authors: Yutao Luo, Haotian Zhu, Shuchao Pang, Zhigang Lu, Tian Dong, Yongbin Zhou, Minhui Xue ### AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories AgentRewardBench evaluates automated web-agent judges against expert reviews of 1,302 trajectories from five benchmarks. The annotations cover completion, side effects, and repetition, enabling comparison of LLM judges and rule-based task evaluators. - https://awesomecua.com/resources/arxiv-2504-08942/ · source: https://arxiv.org/abs/2504.08942v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation - Published: 2025-04-11 · Authors: Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, Siva Reddy ### Agents in the Sandbox: End-to-End Crash Bug Reproduction for Minecraft Introduces BugCraft, which turns Minecraft crash reports into steps and has a vision-based agent execute them through a macro API; BugCraft-Bench supports evaluation. - https://awesomecua.com/resources/arxiv-2503-20036/ · source: https://arxiv.org/abs/2503.20036v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, planning, evaluation - Published: 2025-03-25 · Authors: Eray Yapağcı, Yavuz Alp Sencer Öztürk, Eray Tüzün ### AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents Introduces AgentSentinel, which intercepts sensitive computer-agent operations and audits them using task context together with system traces. BadComputerUse evaluates this enforcement approach across attack scenarios involving tool and computer control. - https://awesomecua.com/resources/arxiv-2509-07764/ · source: https://arxiv.org/abs/2509.07764v1 - Paper · reviewed 2026-10-17 · environments: desktop, linux, web · topics: safety-security, observability, evaluation - Published: 2025-09-09 · Authors: Haitao Hu, Peng Chen, Yanpeng Zhao, Yuqi Chen ### AgentShield: Fortifying Web Applications Against AI Agents and Agentic Browsers Evaluates agentic browsers against web anti-bot defenses and introduces AgentShield, a behavioral fingerprinting framework for distinguishing agents from people. - https://awesomecua.com/resources/arxiv-2610-41181/ · source: https://arxiv.org/abs/2610.41181v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2026-10-30 · Authors: Moiz Ali, Nafis Karim, Rownak Islam, Jyotirmay Chauhan, Dhruv Kuchhal, Jason Polakis ### AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant Introduces a platform that integrates specialized agents under a coordinating MetaAgent for varied computer tasks. - https://awesomecua.com/resources/arxiv-2410-18603/ · source: https://arxiv.org/abs/2410.18603v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: automation, tool-integration - Published: 2024-10-24 · Authors: Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, Zhiyong Wu ### AgentStudio: A Toolkit for Building General Virtual Agents AgentStudio combines an interactive GUI/API environment with tools for creating tasks, annotating interface elements, and labeling actions in videos. Its online suite and GroundUI, IDMBench, and CriticBench datasets examine interaction, grounding, learning from video, and success detection. - https://awesomecua.com/resources/arxiv-2403-17918/ · source: https://arxiv.org/abs/2403.17918v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, sandbox, grounding, video - Published: 2024-03-26 · Authors: Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, Shuicheng Yan ### AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents AgentSynth composes easily generated subtasks into longer computer-use tasks and trajectories with adjustable difficulty. Its evaluation uses the number of subtasks to probe how agent success changes as task complexity grows. - https://awesomecua.com/resources/arxiv-2506-14205/ · source: https://arxiv.org/abs/2506.14205v2 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: data-generation, evaluation, long-horizon - Published: 2025-06-17 · Authors: Jingxu Xie, Dylan Xu, Xuandong Zhao, Dawn Song ### AgentTell: Behavioural Side-Channel Leakage in Browser-Use Agents Introduces AgentTell, a benchmark for privacy leakage through browser actions after an agent learns a private fact on another website. It examines whether agents choose revealing options despite nondisclosure instructions and whether their final reports accurately describe that behavior. - https://awesomecua.com/resources/arxiv-2609-32915/ · source: https://arxiv.org/abs/2609.32915v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, memory - Published: 2026-09-26 · Authors: Asif Shahriar, Md Nafiu Rahman, Sadif Ahmed, Farig Sadeque, Md Rizwan Parvez ### AgentTime: Can Agents Estimate and Control Their Own Runtime? Evaluates duration following, runtime forecasts and elapsed-time estimates in native agent harnesses, including OSWorld desktop tasks, with external timers, native graders and work-versus-wait analysis. - https://awesomecua.com/resources/arxiv-2610-09944/ · source: https://arxiv.org/abs/2610.09944v2 - Paper · reviewed 2026-10-20 · environments: desktop · topics: evaluation, observability - Published: 2026-10-07 · Authors: Michael Ofengenden, Maksym Andriushchenko ### AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials Converts public tutorials into task specifications and executes them with an agent to synthesize multimodal interaction trajectories. - https://awesomecua.com/resources/arxiv-2412-09605/ · source: https://arxiv.org/abs/2412.09605v2 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation - Published: 2024-12-12 · Authors: Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu ### AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents Introduces AgentVigil, a black-box fuzzing framework for finding indirect prompt injections against LLM agents, including attacks that redirect web navigation. - https://awesomecua.com/resources/arxiv-2505-05849/ · source: https://arxiv.org/abs/2505.05849v4 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2025-05-09 · Authors: Zhun Wang, Vincent Siu, Zhe Ye, Tianneng Shi, Yuzhou Nie, Xuandong Zhao, Chenguang Wang, Wenbo Guo, Dawn Song ### AgentVQA: A Unified Benchmark for Agentic Visual Understanding AgentVQA is an offline visual question-answering benchmark across five agent domains, including web agents and questions involving UI element location. - https://awesomecua.com/resources/arxiv-2610-31494/ · source: https://arxiv.org/abs/2610.31494v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, screen-understanding, grounding - Published: 2026-10-22 · Authors: Advait Gupta, Sachin Konan, Tom Yin, Gabriel Sarch, Zhuang Liu ### Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction Uses screenshots, a unified action space, and staged grounding and reasoning training for cross-platform GUI interaction. - https://awesomecua.com/resources/arxiv-2412-04454/ · source: https://arxiv.org/abs/2412.04454v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning, data-generation - Published: 2024-12-05 · Authors: Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, Caiming Xiong ### AI Planning Framework for LLM-Based Web Agents Introduces a planning taxonomy and trajectory-quality metrics for web agents, supported by 794 human-labeled WebArena trajectories and a comparison of two planning styles. - https://awesomecua.com/resources/arxiv-2603-12710/ · source: https://arxiv.org/abs/2603.12710v1 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, evaluation, data-generation, observability - Published: 2026-03-13 · Authors: Orit Shahnovsky, Rotem Dror ### AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation AJ-Bench evaluates judge agents that actively gather evidence from search, data systems and graphical interfaces. Its annotated tasks and trajectories test information acquisition, state verification and process verification. - https://awesomecua.com/resources/arxiv-2604-18240/ · source: https://arxiv.org/abs/2604.18240v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, reward-models, state-tracking - Published: 2026-04-20 · Authors: Wentao Shi, Yu Wang, Yuyang Zhao, Yuxin Chen, Fuli Feng, Xueyuan Hao, Xi Su, Qi Gu, Hui Su, Xunliang Cai, Xiangnan He ### Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion Presents a paired diagnostic of prompt guardrails in GUI agents under screen-grounded user persuasion, comparing single-turn probes with four-turn escalation. - https://awesomecua.com/resources/arxiv-2607-29199/ · source: https://arxiv.org/abs/2607.29199v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, long-horizon - Published: 2026-07-31 · Authors: Haoxin An, Yunpeng Song, Zihao Bai, Zhongmin Cai, Guojun Xiong, Chenhao Lin, Wentao Chen, Feng Wei, Chao Shen ### AliyunConsoleAgent: Training Web Agents in Real-World Cloud Environments via Distillation and Reinforcement Learning AliyunConsoleAgent trains browser agents to verify documented procedures against changing cloud-console UIs through trajectory distillation and online GRPO. Provisioned rollout environments and backend audit-log checks support training and outcome evaluation. - https://awesomecua.com/resources/arxiv-2606-09447/ · source: https://arxiv.org/abs/2606.09447v1 - Paper · reviewed 2026-10-18 · environments: web · topics: web-navigation, imitation-learning, reinforcement-learning, evaluation - Published: 2026-06-08 · Authors: Bojie Rong, Zheyu Shen, Qiaoping Wang, Pengfei Kang, Yang Xu, Yawen Wei, Hanyu Wu, Zhi Zhao, Leihao Pei, Linquan Jiang ### ALLOY: Generating Reusable Agent Workflows from User Demonstration Introduces ALLOY, which turns user demonstrations into editable, reusable workflows for LLM-based web agents carrying out preference-dependent tasks. - https://awesomecua.com/resources/arxiv-2510-10049/ · source: https://arxiv.org/abs/2510.10049v1 - Paper · reviewed 2026-11-06 · environments: web · topics: human-in-the-loop, workflow, imitation-learning - Published: 2025-10-11 · Authors: Jiawen Li, Zheng Ning, Yuan Tian, Toby Jia-jun Li ### AllTogether: Investigating the Efficacy of Spliced Prompt for Web Navigation using Large Language Models AllTogether standardizes prompts for HTML-based web navigation and compares prompting with instruction fine-tuning. Its analysis examines how HTML context, interaction history, and current environmental feedback affect task execution. - https://awesomecua.com/resources/arxiv-2310-18331/ · source: https://arxiv.org/abs/2310.18331v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, interaction-history, evaluation - Published: 2023-10-20 · Authors: Jiarun Liu, Wentao Hu, Chunhong Zhang ### AMBER: Training Long-Horizon Web Agents through Append-Only Memory Introduces AMBER, an append-only memory framework in which a web agent learns to reason, act, and retain evidence using outcome-reward reinforcement learning. It studies whether preserving entries avoids the loss of facts and corrective feedback seen with rewritten memory. - https://awesomecua.com/resources/arxiv-2610-07118/ · source: https://arxiv.org/abs/2610.07118v1 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, long-horizon, reinforcement-learning - Published: 2026-10-05 · Authors: Chinmay Savadikar, Zhaoyu Zhang, Mingyu Zhao, Shuang Xie, Han Li, Tianfu Wu, Lingyun Wang ### AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild AmbiBench evaluates mobile GUI agents under four levels of instruction clarity, including tasks requiring active clarification. Its MUSE evaluator separately audits outcomes, execution quality and interaction quality to study alignment with user intent. - https://awesomecua.com/resources/arxiv-2602-11750/ · source: https://arxiv.org/abs/2602.11750v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, human-in-the-loop, mobile-navigation - Published: 2026-02-12 · Authors: Jiazheng Sun, Mingxuan Li, Yingying Zhang, Jiayang Niu, Yachen Wu, Ruihan Jin, Shuyu Lei, Pengrongrui Tan, Zongyu Zhang, Ruoyi Wang, Jiachen Yang, Boyu Yang, Jiacheng Liu, Xin Peng ### AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents Introduces Android screenshots with element locations, functionality descriptions, and instructions linked to sequences of GUI actions. - https://awesomecua.com/resources/arxiv-2407-17490/ · source: https://arxiv.org/abs/2407.17490v2 - Paper · reviewed 2026-10-16 · environments: android · topics: grounding, screen-understanding, mobile-navigation - Published: 2024-07-03 · Authors: Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Shuai Ren, Hongsheng Li ### An Illusion of Progress? Assessing the Current State of Web Agents Introduces Online-Mind2Web, with 300 tasks across 136 live websites, and examines how benchmark design affects assessments of web agents. It also develops an LLM judge for task completion and compares its judgments with human evaluation. - https://awesomecua.com/resources/arxiv-2504-01382/ · source: https://arxiv.org/abs/2504.01382v4 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation - Published: 2025-04-02 · Authors: Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, Yu Su ### ANCHOR: Branch-Point Data Generation for GUI Agents ANCHOR expands verified desktop demonstrations by generating new tasks at meaningful branch points and executing their instructions. State-aware verification and action filtering refine the resulting trajectories, which are evaluated through fine-tuned agents on OSWorld and WindowsAgentArena. - https://awesomecua.com/resources/arxiv-2602-07153/ · source: https://arxiv.org/abs/2602.07153v2 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: data-generation, planning, evaluation - Published: 2026-02-06 · Authors: Jinbiao Wei, Yilun Zhao, Kangqi Ni, Arman Cohan ### Anchor: Preventing Artifact Drift for Reliable Agent Verification Anchor builds ERP-Bench by generating procurement and manufacturing tasks with shared state verifiers for evaluating browser and computer-use agents. - https://awesomecua.com/resources/arxiv-2605-26321/ · source: https://arxiv.org/abs/2605.26321v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, workflow, long-horizon, industry - Published: 2026-05-25 · Authors: Abhijay Rana, Maksim Ivanov ### AnchorGUI: Asymmetric Memory for Dual-Scale Learning in GUI Navigation AnchorGUI compares expected and observed UI transitions and selectively retains screenshots when they disagree. This asymmetric memory supports immediate correction and focuses cross-attempt experience distillation on likely failure steps. - https://awesomecua.com/resources/arxiv-2609-15457/ · source: https://arxiv.org/abs/2609.15457v1 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, interaction-history, state-tracking, self-reflection, efficiency - Published: 2026-09-14 · Authors: Shengjie Jin, Zelong Sun, Hengbo Xu, Yanbiao Ma, Zhiwu Lu ### Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions Introduces Android Coach, which uses a learned critic to evaluate multiple candidate actions at each observed emulator state. Process rewards and a group-wise advantage estimator support online training, evaluated on AndroidLab and AndroidWorld. - https://awesomecua.com/resources/arxiv-2604-07277/ · source: https://arxiv.org/abs/2604.07277v2 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, reward-models, efficiency - Published: 2026-04-08 · Authors: Guo Gan, Yuxuan Ding, Cong Chen, Yuwei Ren, Yin Huang, Hong Zhou ### Android in the Wild: A Large-Scale Dataset for Android Device Control Collects Android interaction demonstrations with screens, actions, and instructions across device and platform variations for robustness evaluation. - https://awesomecua.com/resources/arxiv-2307-10088/ · source: https://arxiv.org/abs/2307.10088v2 - Paper · reviewed 2026-10-16 · environments: android · topics: data-generation, mobile-navigation, evaluation - Published: 2023-07-19 · Authors: Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, Timothy Lillicrap ### Android in the Zoo: Chain-of-Action-Thought for GUI Agents Introduces Chain-of-Action-Thought and Android-In-The-Zoo annotations that connect prior actions, screen observations, intended actions, and expected outcomes. - https://awesomecua.com/resources/arxiv-2403-02713/ · source: https://arxiv.org/abs/2403.02713v2 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, planning - Published: 2024-03-05 · Authors: Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, Duyu Tang ### AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification AndroidControl-Curated audits ambiguities and factual errors in AndroidControl and produces a refined evaluation dataset. The paper also studies post-training a small Magma-R1 model on a curated subset to examine the effect of data quality on GUI action assessment. - https://awesomecua.com/resources/arxiv-2510-18488/ · source: https://arxiv.org/abs/2510.18488v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, data-generation, grounding - Published: 2025-10-21 · Authors: Ho Fai Leung, Xiaoyan Xi, Fei Zuo ### AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications AndroidDaily evaluates mobile GUI agents on daily workflows in closed-source Android apps. Its GRADE evaluator follows visual trajectories against operational obligations, output-quality requirements and negative constraints, producing process-level diagnostics without hidden app state. - https://awesomecua.com/resources/arxiv-2605-27761/ · source: https://arxiv.org/abs/2605.27761v1 - Paper · reviewed 2026-10-18 · environments: android · topics: mobile-navigation, evaluation, long-horizon, robustness - Published: 2026-05-26 · Authors: Yifan Sui, Xin Huang, Hongbing Li, Fang Xu, Jiahe Lv, Haolong Yan, Yeqing Shen, Litao Liu, Zhimin Fan, Ziyang Meng, Jia Wang, Junbo Qi, Kaijun Tan, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Osamu Yoshie ### AndroidEnv: A Reinforcement Learning Platform for Android Presents an Android environment for training and evaluating reinforcement-learning agents through touchscreen interaction. - https://awesomecua.com/resources/arxiv-2105-13231/ · source: https://arxiv.org/abs/2105.13231v1 - Paper · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, sandbox - Published: 2021-05-27 · Authors: Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, Doina Precup ### AndroidFlux: Evaluating and Reward Modeling Failure Recovery Capabilities in Mobile Use Multimodal Agents Introduces AndroidFlux, a mobile-agent failure-recovery benchmark that restores trajectories to error checkpoints, plus a reward-modeling benchmark and offline RL dataset. - https://awesomecua.com/resources/arxiv-2610-34384/ · source: https://arxiv.org/abs/2610.34384v1 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, evaluation, reward-models, reinforcement-learning - Published: 2026-10-25 · Authors: Gyubeum Lim, Yemo Koo, Ghunwoo Jang, Zsolt Kira ### AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents Provides a shared Android action environment, reproducible tasks, and instruction data for training and comparing language and multimodal agents. - https://awesomecua.com/resources/arxiv-2410-24024/ · source: https://arxiv.org/abs/2410.24024v2 - Paper · reviewed 2026-10-16 · environments: android · topics: evaluation, data-generation, mobile-navigation - Published: 2024-10-31 · Authors: Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, Yuxiao Dong ### AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents AndroidLens provides 571 extended mobile tasks in Chinese and English, with multiple valid paths and preserved environmental anomalies. Its dynamic evaluation uses nested milestones and Average Task Progress to measure partial completion of long workflows. - https://awesomecua.com/resources/arxiv-2512-21302/ · source: https://arxiv.org/abs/2512.21302v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, long-horizon, mobile-navigation, robustness - Published: 2025-12-24 · Authors: Yue Cao, Yingyao Wang, Pi Bu, Jingxuan Xing, Wei Jiang, Zekun Zhu, Junpeng Ma, Sashuai Zhou, Tong Lu, Jun Song, Yu Cheng, Yuning Jiang, Bo Zheng ### AndroidReality: How Far Are Mobile Agents from the Real World? Extends AndroidWorld with controlled state, transition, and action perturbations to test mobile-agent robustness, and studies a training-free introspective recovery mechanism. - https://awesomecua.com/resources/arxiv-2608-07775/ · source: https://arxiv.org/abs/2608.07775v1 - Paper · reviewed 2026-10-18 · environments: android · topics: evaluation, robustness, self-reflection, mobile-navigation - Published: 2026-08-07 · Authors: Xiaoou Liu, Longchao Da, Hanyang Chen, Yuan Ling, Hua Wei ### AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents Introduces parameterized Android tasks with initialization, success checks, and cleanup for reproducible interactive evaluation. - https://awesomecua.com/resources/arxiv-2405-14573/ · source: https://arxiv.org/abs/2405.14573v5 - Paper · reviewed 2026-10-16 · environments: android · topics: evaluation, mobile-navigation - Published: 2024-05-23 · Authors: Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva ### AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents Introduces AndroTMem-Bench for Android tasks whose later actions depend on earlier UI states. Anchored State Memory retains causally linked intermediate states for subgoal-specific retrieval and evaluates them against full-history and summary-based memory. - https://awesomecua.com/resources/arxiv-2603-18429/ · source: https://arxiv.org/abs/2603.18429v1 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, long-horizon, evaluation - Published: 2026-03-19 · Authors: Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, Hanqian Li, Yu Huang, Song Dai, Yonghua Hei, Yue Ding, Xiang Li, Shikang Wang, Chengdong Xu, Jingqi Liu, Xueying Ma, Zhiwen Zheng, Xiaofei Zhang, Bincheng Wang, Nichen Yang, Jie Wu, Lihua Tian, Chen Li, Xuming Hu ### ANGLER: Verifier-Grounded Benchmarking with an Adaptive Attacker for Web Agents Introduces REEF, a website benchmark with state-based attack verification, and ANGLER, which adapts attack lures using outcomes from web-agent rollouts. - https://awesomecua.com/resources/arxiv-2610-41272/ · source: https://arxiv.org/abs/2610.41272v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2026-10-30 · Authors: Yini Huang, Aiwen LU, Zhen Sun, Yi Han, Yuzhi Zhao, Jiaheng Wei ### Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible An anonymization framework replaces sensitive screen content and instructions with consistent, type-preserving placeholders while mediating actions through a local interaction proxy. Experiments on AndroidLab and PrivScreen examine the privacy–utility trade-off when mobile GUI agents operate over these transformed interfaces. - https://awesomecua.com/resources/arxiv-2602-10139/ · source: https://arxiv.org/abs/2602.10139v3 - Paper · reviewed 2026-10-17 · environments: android · topics: safety-security, screen-understanding, mobile-navigation - Published: 2026-02-08 · Authors: Lepeng Zhao, Zhenhua Zou, Shuo Li, Zhuotao Liu ### AOHP: An Open-Source OS-Level Agent Harness for Personalized, Efficient and Secure Interaction Introduces AOHP, an Android OS-level harness that treats agents as system actors and provides adaptive interfaces, agent interfaces, and information-flow controls. - https://awesomecua.com/resources/arxiv-2606-23449/ · source: https://arxiv.org/abs/2606.23449v1 - Paper · reviewed 2026-11-06 · environments: android · topics: sandbox, ui-design, efficiency, safety-security - Published: 2026-06-22 · Authors: Shanhui Zhao, Jiacheng Liu, Guohong Liu, Jichao Yan, Jialei Ye, Yuhao Yang, Hao Wen, Shizuo Tian, Yizhen Yuan, Yuxuan Chen, Yunxin Liu, Ju Ren, Ya-Qin Zhang, Chao Huang, Yao Guo, Yuanchun Li ### API Agents vs. GUI Agents: Divergence and Convergence Compares API-driven and GUI-driven agents across architecture, development, and interaction considerations, discussing when hybrid approaches are useful. - https://awesomecua.com/resources/arxiv-2503-11069/ · source: https://arxiv.org/abs/2503.11069v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: tool-integration, survey, automation - Published: 2025-03-14 · Authors: Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang ### AppAgent v2: Advanced Agent for Flexible Mobile Interactions Separates exploration and deployment while using a structured application knowledge base and retrieval to guide mobile workflows. - https://awesomecua.com/resources/arxiv-2408-11824/ · source: https://arxiv.org/abs/2408.11824v4 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, memory - Published: 2024-08-05 · Authors: Yanda Li, Chi Zhang, Wenjia Jiang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, Yunchao Wei ### AppAgent-Claw: CLI Is All You Need for GUI Automation Introduces AppAgent-Claw, which records demonstrated GUI workflows as reusable OpenClaw skills, locates visual targets during replay and checks intended on-screen effects. - https://awesomecua.com/resources/arxiv-2606-05171/ · source: https://arxiv.org/abs/2606.05171v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, recording, workflow - Published: 2026-04-15 · Authors: Zhixue Song, Zhiheng Zhang, Yi Song, Chi Zhang ### AppAgent-Pro: A Proactive GUI Agent System for Multidomain Information Integration and User Assistance Proposes a proactive GUI assistant that gathers information across domains in response to user requests and anticipated information needs. - https://awesomecua.com/resources/arxiv-2508-18689/ · source: https://arxiv.org/abs/2508.18689v2 - Paper · reviewed 2026-10-16 · environments: android · topics: planning, workflow - Published: 2025-08-26 · Authors: Yuyang Zhao, Wentao Shi, Fuli Feng, Xiangnan He ### AppAgent: Multimodal Agents as Smartphone Users Builds app knowledge through exploration or human demonstrations and uses it to guide multimodal smartphone interactions. - https://awesomecua.com/resources/arxiv-2312-13771/ · source: https://arxiv.org/abs/2312.13771v3 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, memory - Published: 2023-12-21 · Authors: Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, Gang Yu ### AppAgentX: Evolving GUI Agents as Proficient Smartphone Users Records interaction history and converts repeated action sequences into reusable high-level shortcuts for mobile GUI tasks. - https://awesomecua.com/resources/arxiv-2503-02268/ · source: https://arxiv.org/abs/2503.02268v3 - Paper · reviewed 2026-10-16 · environments: android · topics: memory, efficiency - Published: 2025-03-04 · Authors: Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, Joey Tianyi Zhou, Chi Zhang ### AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learning Introduces AppBuddy, an RL training platform and task suite for mobile apps where agents use on-screen element states and tap or type to complete multi-step tasks. - https://awesomecua.com/resources/arxiv-2106-00133/ · source: https://arxiv.org/abs/2106.00133v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, mobile-navigation, sandbox, long-horizon - Published: 2021-05-31 · Authors: Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan Jepson, Sheila A. McIlraith ### AppCopilot: Toward General, Accurate, Long-Horizon, and Efficient Mobile Agent Describes a mobile-agent stack combining region-based coordinate correction, hierarchical task planning, multi-agent voting, and reuse of successful interaction traces. - https://awesomecua.com/resources/arxiv-2509-02444/ · source: https://arxiv.org/abs/2509.02444v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: planning, grounding, workflow, mobile-navigation, long-horizon - Published: 2025-09-02 · Authors: Jingru Fan, Yufan Dang, Jingyao Wu, Huatao Li, Runde Yang, Xiyuan Yang, Yuheng Wang, Chen Qian ### AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents AppDeltaWorld predicts executable code updates for the next mobile UI state, using transition-constrained retrieval and rendered visual assets. It evaluates reconstruction fidelity separately from policy training and adaptation using generated environments. - https://awesomecua.com/resources/arxiv-2608-05891/ · source: https://arxiv.org/abs/2608.05891v1 - Paper · reviewed 2026-10-17 · environments: android · topics: world-models, data-generation, imitation-learning, reinforcement-learning, mobile-navigation - Published: 2026-08-06 · Authors: Weikai Xu, Yunren Feng, Haoxiang Lei, Kun Huang, Yuxuan Liu, Kang Zhao, Xiaolin Hu, Shuo Shang, Bo An ### APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions Introduces APPINITE, a conversational interface for specifying the intended targets of demonstrated Android actions. It constructs semantic UI representations and uses an on-screen overlay to preview and refine executable data descriptions. - https://awesomecua.com/resources/url-ba2c664c1b3c6f1b172f/ · source: https://ieeexplore.ieee.org/document/8506506 - Paper · reviewed 2026-10-17 · environments: android · topics: grounding, human-in-the-loop, mobile-navigation - Published: 2018 · Authors: Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, Brad A. Myers ### AppSelectBench: Application-Level Tool Selection Benchmark AppSelectBench evaluates whether computer-using agents choose the right desktop application for a task, using generated user intents and multiple evaluation protocols. - https://awesomecua.com/resources/arxiv-2511-19957/ · source: https://arxiv.org/abs/2511.19957v2 - Paper · reviewed 2026-11-06 · environments: desktop · topics: evaluation, planning, data-generation - Published: 2025-11-25 · Authors: Tianyi Chen, Michael Solodko, Sen Wang, Jongwoo Ko, Junheng Hao, Colby Banbury, Sara Abdali, Saeed Amizadeh, Qing Xiao, Yinheng Li, Tianyu Ding, Kamran Ghasedi Dizaji, Suzhen Zheng, Hao Fan, Justin Wagle, Pashmina Cameron, Kazuhito Koishida ### APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents Introduces controlled simulated mobile apps with deterministic backend data and outcome checks for reproducible evaluation of realistic interactions. - https://awesomecua.com/resources/arxiv-2609-07712/ · source: https://arxiv.org/abs/2609.07712v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, sandbox, mobile-navigation - Published: 2026-09-07 · Authors: Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, Ziyang Wang, Yichen Gong ### AppVLM: A Lightweight Vision Language Model for Online App Control AppVLM first learns app actions from AndroidControl and then refines its policy through data collected in AndroidWorld. The study compares offline action prediction and online task completion for a lightweight vision-language smartphone agent. - https://awesomecua.com/resources/arxiv-2502-06395/ · source: https://arxiv.org/abs/2502.06395v1 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, imitation-learning, data-generation, efficiency - Published: 2025-02-10 · Authors: Georgios Papoudakis, Thomas Coste, Zhihao Wu, Jianye Hao, Jun Wang, Kun Shao ### AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents Introduces simulated application APIs and state-based task evaluation for interactive coding agents performing everyday digital workflows. - https://awesomecua.com/resources/arxiv-2407-18901/ · source: https://arxiv.org/abs/2407.18901v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, tool-integration, cross-app - Published: 2024-07-26 · Authors: Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, Niranjan Balasubramanian ### AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees AQuaUI reduces screenshot tokens for GUI agents at inference time with adaptive quadtrees that retain spatial positions and reuse prior tree structure across steps. - https://awesomecua.com/resources/arxiv-2605-19260/ · source: https://arxiv.org/abs/2605.19260v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, screen-understanding, training-free - Published: 2026-05-19 · Authors: Yuankai Li, Tinghui Zhu, Ha Min Son, Zhe Zhao, Xin Liu, Muhao Chen ### Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments Introduces AnTrap, which injects task-solvable runtime anomalies into Android GUI-agent trajectories using a layered failure taxonomy. It compares model robustness and adversarial reinforcement learning to distinguish recoverable interface traps from persistent contextual failures. - https://awesomecua.com/resources/arxiv-2608-24099/ · source: https://arxiv.org/abs/2608.24099v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, robustness, reinforcement-learning - Published: 2026-08-25 · Authors: Guo Gan, Yilun Zhao, Cong Chen, Jinbiao Wei, Tingyu Song, Zheyuan Yang, Lin Fu, Hong Zhou ### Are Autonomous Web Agents Good Testers? Adapts web agents to execute natural-language test steps, verify assertions, and issue test verdicts. The study introduces 113 passing and failing test cases across three offline web applications and compares two agent implementations with qualitative failure analysis. - https://awesomecua.com/resources/arxiv-2504-01495/ · source: https://arxiv.org/abs/2504.01495v1 - Paper · reviewed 2026-10-17 · environments: web · topics: automation, evaluation, web-navigation - Published: 2025-04-02 · Authors: Antoine Chevrot, Alexandre Vernotte, Jean-Rémy Falleri, Xavier Blanc, Bruno Legeard ### Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection This study tests black-box GUI-grounding failures caused by strategically added, innocuous-looking UI elements. Its iterative overlay search is compared with random injection and repeated independent grounding trials to distinguish persistent misdirection from incidental clutter. - https://awesomecua.com/resources/arxiv-2604-07831/ · source: https://arxiv.org/abs/2604.07831v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, grounding, robustness, evaluation - Published: 2026-04-09 · Authors: Wenkui Yang, Chao Jin, Haisu Zhu, Weilin Luo, Derek Yuen, Kun Shao, Junxian Duan, Huaibo Huang, Jie Cao, Ran He ### Are LLM Agents the New RPA? A Comparative Study with RPA Across Enterprise Workflows Compares an agent using computer interfaces with UiPath RPA on enterprise data entry, monitoring, and document extraction, measuring speed, reliability, and development effort. - https://awesomecua.com/resources/arxiv-2509-04198/ · source: https://arxiv.org/abs/2509.04198v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, automation, industry, workflow - Published: 2025-09-04 · Authors: Petr Průcha, Michaela Matoušková, Jan Strnad ### Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents Compares online skill and memory augmentations with a vanilla web agent given the same total inference-token budget. Across WebArena and WorkArena settings, it studies when module overhead removes aggregate gains and how rerun variability changes conclusions. - https://awesomecua.com/resources/arxiv-2606-15017/ · source: https://arxiv.org/abs/2606.15017v2 - Paper · reviewed 2026-10-18 · environments: web · topics: memory, efficiency, evaluation, robustness - Published: 2026-06-12 · Authors: Sina Hajimiri, Masih Aminbeidokhti, Jose Dolz, Ismail Ben Ayed, Issam H. Laradji, Spandana Gella, Nicolas Gontier ### Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications A diary study deploys the screen-reader-accessible OLLA computer-use prototype with blind users across desktop applications. Recorded commands and interaction traces support model comparisons and analysis of grounding, planning, constraint tracking and termination failures. - https://awesomecua.com/resources/arxiv-2609-00524/ · source: https://arxiv.org/abs/2609.00524v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, human-in-the-loop, grounding - Published: 2026-09-01 · Authors: Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan, Vikas Ashok ### Ares: Adaptive Reasoning Effort Selection for Efficient LLM Agents Introduces Ares, a router that selects a reasoning level for each step of a general LLM agent from its interaction history, with web agents among the evaluation settings. - https://awesomecua.com/resources/arxiv-2603-07915/ · source: https://arxiv.org/abs/2603.07915v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, interaction-history, data-generation - Published: 2026-03-09 · Authors: Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, Shiyu Chang ### Aria-UI: Visual Grounding for GUI Instructions Introduces a vision-based grounding model trained with diverse synthetic instructions and action histories for context-sensitive GUI targeting. - https://awesomecua.com/resources/arxiv-2412-16256/ · source: https://arxiv.org/abs/2412.16256v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, memory - Published: 2024-12-20 · Authors: Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, Junnan Li ### ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay ARPO extends group-relative policy optimization with a replay buffer that reuses successful GUI trajectories across training iterations. It filters tasks using baseline performance and compares end-to-end reinforcement learning with offline preference optimization on OSWorld. - https://awesomecua.com/resources/arxiv-2505-16282/ · source: https://arxiv.org/abs/2505.16282v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: reinforcement-learning, long-horizon - Published: 2025-05-22 · Authors: Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, Jiaya Jia ### ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions ASIL exposes application state as structured JSON and provides code-executable semantic actions. Its application benchmark compares this interface with screenshot control and application-native APIs, and studies supervised and reinforcement learning on the structured interface. - https://awesomecua.com/resources/arxiv-2608-26991/ · source: https://arxiv.org/abs/2608.26991v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, tool-integration, evaluation, reinforcement-learning, imitation-learning - Published: 2026-08-27 · Authors: Rui Xie, Lu Chen ### AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? Introduces realistic open-web tasks and the SeePlanAct agent, analyzing the limitations of browsing and retrieval in time-consuming assistant work. - https://awesomecua.com/resources/arxiv-2407-15711/ · source: https://arxiv.org/abs/2407.15711v2 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation, planning - Published: 2024-07-22 · Authors: Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant ### ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation Introduces Windows application tasks and an actor-critic agent framework for mouse-and-keyboard automation of desktop workflows. - https://awesomecua.com/resources/arxiv-2312-13108/ · source: https://arxiv.org/abs/2312.13108v2 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: evaluation, planning - Published: 2023-12-20 · Authors: Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, Mike Zheng Shou ### AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agents AsyncWebRL overlaps web-agent rollouts, updates and policy refresh while managing screenshot handling and a persistent rollout pool. It also replaces trajectory-length normalization in multi-step GRPO to reduce verbose failed trajectories, with throughput and out-of-distribution evaluation. - https://awesomecua.com/resources/arxiv-2606-05597/ · source: https://arxiv.org/abs/2606.05597v3 - Paper · reviewed 2026-10-18 · environments: web · topics: reinforcement-learning, efficiency, web-navigation - Published: 2026-06-04 · Authors: Hao Bai, Rui Yang, Chenlu Ye, Spencer Whitehead, Aviral Kumar, Tong Zhang ### ATLaS: Agent Tuning via Learning Critical Steps Selects critical steps from expert interaction trajectories for targeted agent fine-tuning, studying efficiency and generalization across tasks. - https://awesomecua.com/resources/arxiv-2503-02197/ · source: https://arxiv.org/abs/2503.02197v2 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, planning, efficiency - Published: 2025-03-04 · Authors: Zhixun Chen, Ming Li, Yuxuan Huang, Yali Du, Meng Fang, Tianyi Zhou ### Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System UI-NEXUS evaluates compositional mobile tasks in local utility apps and online Chinese and English service apps. AGENT-NEXUS decomposes those long tasks into self-contained atomic subtasks to study generalization beyond isolated screens or short action chains. - https://awesomecua.com/resources/arxiv-2506-08972/ · source: https://arxiv.org/abs/2506.08972v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, long-horizon, planning, mobile-navigation - Published: 2025-06-10 · Authors: Yuan Guo, Tingjia Miao, Zheng Wu, Pengzhou Cheng, Ming Zhou, Zhuosheng Zhang ### Atomicity for Agents: Exposing, Exploiting, and Mitigating TOCTOU Vulnerabilities in Browser-Use Agents Identifies time-of-check to time-of-use (TOCTOU) race vulnerabilities when web pages mutate between agent planning and action execution. - https://awesomecua.com/resources/arxiv-2603-00476/ · source: https://arxiv.org/abs/2603.00476v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, web-navigation, state-tracking - Published: 2026-02-28 · Authors: Linxi Jiang, Zhijie Liu, Haotian Luo, Zhiqiang Lin ### Attacking Vision-Language Computer Agents via Pop-ups Tests whether adversarial visual pop-ups divert computer agents from intended tasks in desktop and web benchmark environments. - https://awesomecua.com/resources/arxiv-2411-02391/ · source: https://arxiv.org/abs/2411.02391v2 - Paper · reviewed 2026-10-16 · environments: desktop, web · topics: safety-security, evaluation - Published: 2024-11-04 · Authors: Yanzhe Zhang, Tao Yu, Diyi Yang ### Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning Locates GUI targets by identifying and aggregating attention maps in pretrained multimodal models without task-specific fine-tuning. - https://awesomecua.com/resources/arxiv-2412-10840/ · source: https://arxiv.org/abs/2412.10840v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Published: 2024-12-14 · Authors: Hai-Ming Xu, Qi Chen, Lei Wang, Lingqiao Liu ### Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories Audits WebArena-Lite outcomes and trajectories with human review to identify automatic-evaluator false negatives and the first consequential execution errors. It also studies explicit execution-state memory and procedural guidance, showing why final task scores can miss substantial differences in agent behavior. - https://awesomecua.com/resources/arxiv-2610-01491/ · source: https://arxiv.org/abs/2610.01491v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, observability, memory - Published: 2026-10-01 · Authors: Chengguang Gan, Zimeng He, Yoshihiro Tsujii, Ken-ichiro Kobayashi, Hiroki Itoh, Kotaro Funakoshi ### Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents A study adapts interface usability heuristics for computer-use agents and evaluates augmented designs in UI-Verse, a suite of controlled interface environments. - https://awesomecua.com/resources/arxiv-2605-02729/ · source: https://arxiv.org/abs/2605.02729v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: ui-design, evaluation, robustness - Published: 2026-05-04 · Authors: Jiateng Liu, Rushi Wang, Bingxuan Li, Kunlun Zhu, Yifan Shen, Qingyun Wang, Ahmed Abbasi, Denghui Zhang, Heng Ji ### AUITestAgent: Automatic Requirements Oriented GUI Function Testing Automates mobile GUI tests from natural-language requirements, combining action generation with verification based on interaction traces. - https://awesomecua.com/resources/arxiv-2407-09018/ · source: https://arxiv.org/abs/2407.09018v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: automation, evaluation - Published: 2024-07-12 · Authors: Yongxiang Hu, Xuan Wang, Yingchuan Wang, Yu Zhang, Shiyu Guo, Chaoyi Chen, Xin Wang, Yangfan Zhou ### AUTO-Explorer: Automated Data Collection for GUI Agent Introduces Auto-Explorer, an automated GUI exploration method that records software states for training-data collection. UIXplore evaluates exploration coverage, and collected data is used to fine-tune and test GUI element grounding. - https://awesomecua.com/resources/arxiv-2511-06417/ · source: https://arxiv.org/abs/2511.06417v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, grounding, evaluation - Published: 2025-11-09 · Authors: Xiangwu Guo, Difei Gao, Mike Zheng Shou ### Auto-Formula: Recommend Formulas in Spreadsheets using Contrastive Learning for Table Representations Auto-Formula recommends formulas for target spreadsheet cells by adapting formulas from similar spreadsheets within an organization. - https://awesomecua.com/resources/arxiv-2404-12608/ · source: https://arxiv.org/abs/2404.12608v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, action-prediction - Published: 2024-04-19 · Authors: Sibei Chen, Yeye He, Weiwei Cui, Ju Fan, Song Ge, Haidong Zhang, Dongmei Zhang, Surajit Chaudhuri ### Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents Discovers compact intents from demonstrations and uses intent predictions to guide a language-model web agent's decisions and exploration. - https://awesomecua.com/resources/arxiv-2410-22552/ · source: https://arxiv.org/abs/2410.22552v1 - Paper · reviewed 2026-10-16 · environments: web · topics: planning, web-navigation - Published: 2024-10-29 · Authors: Jaekyeom Kim, Dong-Ki Kim, Lajanugen Logeswaran, Sungryull Sohn, Honglak Lee ### Auto-scaling Continuous Memory for GUI Agent Encodes GUI trajectories into compact continuous embeddings that can be retrieved directly into an agent’s input. An automated environment-discovery and rollout pipeline expands this memory for long tasks and unfamiliar interfaces. - https://awesomecua.com/resources/arxiv-2510-09038/ · source: https://arxiv.org/abs/2510.09038v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, interaction-history, efficiency - Published: 2025-10-10 · Authors: Wenyi Wu, Kun Zhou, Ruoxin Yuan, Vivian Yu, Stephen Wang, Zhiting Hu, Biwei Huang ### AutoData: A Multi-Agent System for Open Web Data Collection Introduces AutoData, a multi-agent system that collects web datasets from a natural-language request, and Instruct2DS, a benchmark for live collection across three domains. - https://awesomecua.com/resources/arxiv-2505-15859/ · source: https://arxiv.org/abs/2505.15859v1 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, automation, evaluation, efficiency - Published: 2025-05-21 · Authors: Tianyi Ma, Yiyue Qian, Zheyuan Zhang, Zehong Wang, Xiaoye Qian, Feifan Bai, Yifan Ding, Xuwei Luo, Shinan Zhang, Keerthiram Murugesan, Chuxu Zhang, Yanfang Ye ### AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation Recasts mobile UI automation as on-device code generation, using app-specific documentation and synthetic tasks to guide smaller models. - https://awesomecua.com/resources/arxiv-2412-18116/ · source: https://arxiv.org/abs/2412.18116v3 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, tool-integration - Published: 2024-12-24 · Authors: Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, Yuanchun Li ### AutoDroid: LLM-powered Task Automation in Android Combines language-model reasoning with app analysis, UI representations, and exploration-derived memory for Android task automation. - https://awesomecua.com/resources/arxiv-2308-15272/ · source: https://arxiv.org/abs/2308.15272v4 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, memory - Published: 2023-08-29 · Authors: Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, Yunxin Liu ### AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents Generates reward signals from UI-state changes and uses a judging system to evaluate mobile agents with reduced manual evaluator construction. - https://awesomecua.com/resources/arxiv-2503-02403/ · source: https://arxiv.org/abs/2503.02403v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, mobile-navigation - Published: 2025-03-04 · Authors: Jiahui Sun, Zhichao Hua, Yubin Xia ### AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding Presents AutoFocus, a training-free grounding method that turns sampled coordinates and token uncertainty into spatial region proposals. Shape-aware crops and visual comparison select a target while retaining surrounding interface context. - https://awesomecua.com/resources/arxiv-2605-02630/ · source: https://arxiv.org/abs/2605.02630v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, screen-understanding - Published: 2026-05-04 · Authors: Ruilin Yao, Shegnwu Xiong, Tianyu Zou, Shili Xiong, Yi Rong ### AutoGLM: Autonomous Foundation Agents for GUIs Presents AutoGLM agents for browser and phone control, separating planning from grounding and training through an evolving interaction curriculum. - https://awesomecua.com/resources/arxiv-2411-00820/ · source: https://arxiv.org/abs/2411.00820v1 - Paper · reviewed 2026-10-16 · environments: web, android · topics: automation, reinforcement-learning, grounding - Published: 2024-10-28 · Authors: Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, Junjie Gao, Junjun Shan, Kangning Liu, Shudan Zhang, Shuntian Yao, Siyi Cheng, Wentao Yao, Wenyi Zhao, Xinghan Liu, Xinyi Liu, Xinying Chen, Xinyue Yang, Yang Yang, Yifan Xu, Yu Yang, Yujia Wang, Yulin Xu, Zehan Qi, Yuxiao Dong, Jie Tang ### AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark AutoGUI-v2 evaluates interface functionality, grounding and predicted interaction outcomes using hierarchical regions derived through VLM-human annotation. Its tasks span multiple operating systems and expose differences between functional localization, captioning and transition-logic understanding. - https://awesomecua.com/resources/arxiv-2604-24441/ · source: https://arxiv.org/abs/2604.24441v1 - Paper · reviewed 2026-10-18 · environments: cross-platform · topics: screen-understanding, grounding, state-tracking, evaluation, data-generation - Published: 2026-04-27 · Authors: Hongxin Li, Xiping Wang, Jingran Su, Zheng Ju, Yuntao Chen, Qing Li, Zhaoxiang Zhang ### AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs Infers element functionality from UI changes before and after interaction to produce verified annotations for grounding-model training. - https://awesomecua.com/resources/arxiv-2502-01977/ · source: https://arxiv.org/abs/2502.01977v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, data-generation - Published: 2025-02-04 · Authors: Hongxin Li, Jingfan Chen, Jingran Su, Yuntao Chen, Qing Li, Zhaoxiang Zhang ### AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents Introduces AutoGuide, which generates conditional, context-aware guidelines from past agent experience to support decisions in sequential tasks, including web navigation. - https://awesomecua.com/resources/arxiv-2403-08978/ · source: https://arxiv.org/abs/2403.08978v2 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, planning, web-navigation - Published: 2024-03-13 · Authors: Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, Honglak Lee ### AutoGUIWorld: Image Generators as Visual World Models for GUI Agent Introduces AutoGUIWorld, which combines task planning and image editing to synthesize action-grounded GUI trajectories without running the depicted software. It filters generated transitions and tests whether training on them improves agents operating real desktop and scientific applications. - https://awesomecua.com/resources/arxiv-2610-01215/ · source: https://arxiv.org/abs/2610.01215v1 - Paper · reviewed 2026-10-17 · environments: desktop, linux, windows, macos, web · topics: world-models, data-generation, grounding - Published: 2026-10-01 · Authors: Cheng Yang, Yifan Wu, Yutao Huang, Zhaohua Zhang, Beiduo Chen, Muxi Chen, Chenchen Zhao, Hexuan Deng, Haolin Yang, Geyuan Zhu, Sa Zhu, Jianhuan Zhuo, Qiuyong Xiao, Jianhao Ruan, Yiran Peng, Jiayi Zhang, Tian Ye, Xinlei Yu, Tianwen Jiang, Jihong Zhang, Yuyu Luo ### Automated Test Transfer Across Android Apps Using Large Language Models LLMigrate uses LLMs to transfer usage-based UI tests between mobile apps with similar functions, reducing the need to write tests for each app. - https://awesomecua.com/resources/arxiv-2411-17933/ · source: https://arxiv.org/abs/2411.17933v2 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, automation, cross-app - Published: 2024-11-26 · Authors: Benyamin Beyzaei, Saghar Talebipour, Ghazal Rafiei, Nenad Medvidovic, Sam Malek ### Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation Introduces CRATE, a step-level visual judge for mobile-agent trajectories, and CRATE-S for assessing operational safety from action-conditioned state changes. - https://awesomecua.com/resources/arxiv-2608-20797/ · source: https://arxiv.org/abs/2608.20797v1 - Paper · reviewed 2026-11-06 · environments: android · topics: evaluation, state-tracking, safety-security, mobile-navigation - Published: 2026-08-21 · Authors: Pengshuai Yang, Zijing Gao, Xue Yu, Benhui Zhuang, Bo Yuan, Junlan Feng ### Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining Segments and clusters GUI trajectories into inspectable skills, then trains a skill-aware policy from those annotations. Its diagnostic results distinguish readable source-domain clusters from reliable transfer and report limited or absent downstream gains. - https://awesomecua.com/resources/arxiv-2606-20363/ · source: https://arxiv.org/abs/2606.20363v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: memory, data-generation, reinforcement-learning, evaluation - Published: 2026-06-18 · Authors: Yuexing Hao, Xiaomin Li ### Automating the Enterprise with Foundation Models ECLAIR uses multimodal foundation models to automate enterprise workflows from natural-language descriptions, with initial studies of workflow understanding and completion. - https://awesomecua.com/resources/arxiv-2405-03710/ · source: https://arxiv.org/abs/2405.03710v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: workflow, automation, industry - Published: 2024-05-03 · Authors: Michael Wornow, Avanika Narayan, Krista Opsahl-Ong, Quinn McIntyre, Nigam H. Shah, Christopher Re ### AutomationBench Introduces AutomationBench, a benchmark for agents coordinating business workflows across applications through REST APIs, including endpoint discovery and policy adherence. - https://awesomecua.com/resources/arxiv-2604-18934/ · source: https://arxiv.org/abs/2604.18934v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, cross-app, workflow, tool-integration - Published: 2026-04-21 · Authors: Daniel Shepard, Robin Salimans ### Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents ACuRL uses autonomous interface exploration and an adaptive task curriculum to continually train computer-use policies with reinforcement learning. CUAJudge supplies outcome feedback, and experiments study adaptation across desktop applications and environment changes. - https://awesomecua.com/resources/arxiv-2602-10356/ · source: https://arxiv.org/abs/2602.10356v2 - Paper · reviewed 2026-10-17 · environments: desktop, linux, windows · topics: reinforcement-learning, data-generation, reward-models, robustness - Published: 2026-02-10 · Authors: Tianci Xue, Zeyi Liao, Tianneng Shi, Zilu Wang, Kai Zhang, Dawn Song, Yu Su, Huan Sun ### Autonomous Evaluation and Refinement of Digital Agents Compares automatic trajectory evaluators for web navigation and device control against oracle judgments. The resulting feedback is used for agent fine-tuning and inference-time guidance, including experiments on WebArena. - https://awesomecua.com/resources/arxiv-2404-06474/ · source: https://arxiv.org/abs/2404.06474v3 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, reward-models - Published: 2024-04-09 · Authors: Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, Alane Suhr ### AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation Introduces AutoScraper, a two-stage framework that uses HTML structure and similarities across pages to generate scrapers for data collection from websites. - https://awesomecua.com/resources/arxiv-2404-12753/ · source: https://arxiv.org/abs/2404.12753v2 - Paper · reviewed 2026-11-06 · environments: web · topics: automation, data-generation, evaluation - Published: 2024-04-19 · Authors: Wenhao Huang, Zhouhong Gu, Chenghao Peng, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Liqian Wen, Zulong Chen ### AutoSurfer -- Teaching Web Agents through Comprehensive Surfing, Learning, and Modeling AutoSurfer explores website actions breadth-first, retaining page and trajectory knowledge to guide task synthesis and trajectory refinement. It evaluates the resulting training data by fine-tuning a web agent on WebArena and examining synthesized-task diversity. - https://awesomecua.com/resources/arxiv-2604-27253/ · source: https://arxiv.org/abs/2604.27253v1 - Paper · reviewed 2026-10-18 · environments: web · topics: web-navigation, data-generation, imitation-learning, evaluation - Published: 2026-04-29 · Authors: Fazle Elahi Faisal, Qianhui Wu, Baolin Peng, Jianfeng Gao ### AutoTailor: Automatic, User-Aligned Capability Selection and Adaptation for Web Agents AutoTailor compiles web trajectories into reusable browser-automation APIs, then filters and dynamically revises the capability inventory. WebArena experiments distinguish API-only operation from an agent that can fall back to ordinary reasoning and browser actions. - https://awesomecua.com/resources/arxiv-2609-13548/ · source: https://arxiv.org/abs/2609.13548v1 - Paper · reviewed 2026-10-17 · environments: web · topics: tool-integration, web-navigation, workflow, efficiency - Published: 2026-09-11 · Authors: Xinyun Cao, Adriana Szekeres, Fazle Elahi Faisal ### AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI Executes voice-requested mobile tasks through GUI exploration, backtracking, and accumulated interaction experience. - https://awesomecua.com/resources/arxiv-2312-16062/ · source: https://arxiv.org/abs/2312.16062v1 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, memory - Published: 2023-12-26 · Authors: Lihang Pan, Bowen Wang, Chun Yu, Yuxuan Chen, Xiangyu Zhang, Yuanchun Shi ### AutoWebGLM: A Large Language Model-based Web Navigating Agent Builds a web agent through simplified HTML observations, human-and-AI browsing data, curriculum training, and reinforcement-learning refinement. - https://awesomecua.com/resources/arxiv-2404-03648/ · source: https://arxiv.org/abs/2404.03648v2 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, reinforcement-learning, data-generation - Published: 2024-04-04 · Authors: Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, Jie Tang ### AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machines AutoWebWorld turns finite-state task models into interactive websites whose actions and goal states can be checked programmatically. It generates verified trajectories for training and evaluates transfer to WebVoyager and Online-Mind2Web. - https://awesomecua.com/resources/arxiv-2602-14296/ · source: https://arxiv.org/abs/2602.14296v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, sandbox, web-navigation, evaluation - Published: 2026-02-15 · Authors: Yifan Wu, Yiran Peng, Yiyu Chen, Jianhao Ruan, Zijie Zhuang, Cheng Yang, Jiayi Zhang, Man Chen, Yenchi Tseng, Zhaoyang Yu, Liang Chen, Yuyao Zhai, Bang Liu, Chenglin Wu, Yuyu Luo ### Avenir-UX: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding Avenir-UX combines grounded web interaction with simulated user profiles and structured usability measures to produce evaluations of website journeys. The paper describes its architecture and use of SUS, step-wise ease questions and think-aloud traces for automated UX reporting. - https://awesomecua.com/resources/arxiv-2604-09581/ · source: https://arxiv.org/abs/2604.09581v2 - Paper · reviewed 2026-10-17 · environments: web · topics: automation, evaluation, ui-design, grounding - Published: 2026-02-25 · Authors: Wee Joe Tan, Zi Rui Lucas Lim, Shashank Durgad, Karim Obegi, Aiden Yiliu Li ### Avenir-Web: Human-Experience-Imitating Multimodal Web Agents with Mixture of Grounding Experts Avenir-Web combines multiple grounding specialists with procedural experience, a task checklist and adaptive memory. It evaluates this planning and execution framework on live Online-Mind2Web tasks. - https://awesomecua.com/resources/arxiv-2602-02468/ · source: https://arxiv.org/abs/2602.02468v1 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, planning, memory, web-navigation - Published: 2026-02-02 · Authors: Aiden Yiliu Li, Xinyue Hao, Shilong Liu, Mengdi Wang ### AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents Introduces AXIS, an API-first application agent that can use UI actions and expand its available APIs through automated application exploration. The work evaluates the approach on Microsoft Word tasks. - https://awesomecua.com/resources/arxiv-2409-17140/ · source: https://arxiv.org/abs/2409.17140v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: tool-integration, automation, efficiency - Published: 2024-09-25 · Authors: Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang ### AXNav: Replaying Accessibility Tests from Natural Language Introduces an LLM and pixel-based UI system that executes natural-language accessibility tests, records navigable videos and flags potential interface issues. - https://awesomecua.com/resources/arxiv-2310-02424/ · source: https://arxiv.org/abs/2310.02424v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, screen-understanding, video, evaluation - Published: 2023-10-03 · Authors: Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, Jeffrey Nichols ### Back to Basics: A Skill-Centric Approach to Generalist Web Agents Introduces MiniWeb for skill-labeled browser tasks, MiniWebGen for generating training trajectories, and MiniWebAgent for composing learned skills into web workflows. - https://awesomecua.com/resources/arxiv-2610-35823/ · source: https://arxiv.org/abs/2610.35823v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, data-generation, planning, web-navigation - Published: 2026-10-26 · Authors: Minh Pham-Dinh, Md Farhan Ishmam, Hernan Yupanqui, Reaz Hassan Joarder, Kenneth Marino ### Back to the Future: A workbook time machine for spread sheet creation benchmarks Introduces the workbook time machine to create spreadsheet-object tasks and wtmbench to evaluate spreadsheet manipulation agents using different control interfaces. - https://awesomecua.com/resources/arxiv-2608-07873/ · source: https://arxiv.org/abs/2608.07873v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, evaluation, automation - Published: 2026-08-08 · Authors: Mansi Uniyal, Agamdeep Singh, Ananya Singha, Priyanshu Gupta, Mukul Singh, Gust Verbruggen, Vu Le, Sumit Gulwani ### BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism BacktrackAgent adds verification, judgment, and reflection modules to detect erroneous GUI actions and guide recovery. It develops backtracking-oriented training data that includes post-action pages and evaluates task completion and step accuracy on Mobile3M and Auto-UI. - https://awesomecua.com/resources/arxiv-2505-20660/ · source: https://arxiv.org/abs/2505.20660v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: robustness, planning, data-generation, mobile-navigation - Published: 2025-05-27 · Authors: Qinzhuo Wu, Pengzhi Gao, Wei Liu, Jian Luan ### BAMI: Training-Free Bias Mitigation in GUI Grounding BAMI uses coarse-to-fine focus and candidate selection at inference time to address precision and ambiguity errors in GUI grounding. - https://awesomecua.com/resources/arxiv-2605-06664/ · source: https://arxiv.org/abs/2605.06664v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, training-free, screen-understanding - Published: 2026-05-07 · Authors: Borui Zhang, Bo Zhang, Bo Wang, Wenzhao Zheng, Yuhao Cheng, Liang Tang, Yiqiang Yan, Jie Zhou, Jiwen Lu ### BaRA: Budget-constrained and Reliable Web Data Collection Agent Introduces BaRA, a web data-collection agent that discovers live site links, checks extracted text, images, and videos, and uses history to recover from failures. - https://awesomecua.com/resources/arxiv-2607-00007/ · source: https://arxiv.org/abs/2607.00007v2 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, data-generation, self-reflection, interaction-history - Published: 2026-05-02 · Authors: Soojeong Lee, Joseph Lee, Yongseong Cho, Sunjae Kim, Youngwoo Moon, Kyungwoo Song ### BEAP-Agent: Backtrackable Execution and Adaptive Planning for GUI Agents BEAP-Agent treats GUI execution as depth-first search with coordinated planning, execution and task-state tracking. It supports backtracking across multiple levels of earlier states and evaluates recovery on OSWorld. - https://awesomecua.com/resources/arxiv-2601-21352/ · source: https://arxiv.org/abs/2601.21352v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: planning, state-tracking, robustness, long-horizon - Published: 2026-01-29 · Authors: Ziyu Lu, Tengjin Weng, Yiying Yang, Yuhang Zhao, Xinxin Huang, Wenhao Jiang ### BEARCUBS: A benchmark for computer-using web agents BEARCUBS provides 111 live-web information-seeking questions with short answers and human-validated browsing trajectories. Its tasks require multimodal computer interaction, including video understanding and navigation of interactive 3D content. - https://awesomecua.com/resources/arxiv-2503-07919/ · source: https://arxiv.org/abs/2503.07919v3 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, video - Published: 2025-03-10 · Authors: Yixiao Song, Katherine Thai, Chau Minh Pham, Yapei Chang, Mazin Nadaf, Mohit Iyyer ### Before Acting, Change the State: Prospective State Intervention for Web Agents under Deceptive Interfaces Introduces Veer, a runtime defense that changes task-relevant web state when an otherwise valid proposed action would produce an unauthorized consequence. It plans a prospective intervention and grounds and verifies its execution before the base agent continues. - https://awesomecua.com/resources/arxiv-2609-34974/ · source: https://arxiv.org/abs/2609.34974v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, planning, robustness - Published: 2026-09-28 · Authors: Ruozhao Yang, Mingfei Cheng, Xiaofei Xie ### Benchmarking and Improving GUI Agents in High-Dynamic Environments Introduces DynamicGUIBench for applications whose interfaces change between actions, together with DynamicUI, an agent that processes screen recordings. The agent selects informative frames, filters inconsistent action reasoning and uses trajectory reflection for subsequent decisions. - https://awesomecua.com/resources/arxiv-2604-25380/ · source: https://arxiv.org/abs/2604.25380v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: video, state-tracking, evaluation - Published: 2026-04-28 · Authors: Enqi Liu, Liyuan Pan, Zhi Gao, Yan Yang, Chenrui Shi, Yang Liu, Jingrong Wu, Qing Li ### Benchmarking General Mobile Assistants in Challenging Real-World Scenarios Introduces GMA, a benchmark of 300 mobile-assistant tasks across seven applications, and studies how agent context retention and state tracking affect complex workflows. - https://awesomecua.com/resources/arxiv-2608-27477/ · source: https://arxiv.org/abs/2608.27477v1 - Paper · reviewed 2026-11-06 · environments: android · topics: evaluation, mobile-navigation, workflow, state-tracking - Published: 2026-08-21 · Authors: Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu ### Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms LivingScreen evaluates agents in a browser-based short-video environment where observing content consumes time and affects subsequent actions. Its task suite scores accuracy together with information efficiency to diagnose over- and under-observation on continuously changing screens. - https://awesomecua.com/resources/arxiv-2606-04701/ · source: https://arxiv.org/abs/2606.04701v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, video, efficiency, screen-understanding - Published: 2026-06-03 · Authors: Jiashu Yao, Heyan Huang, Daiqing Wu, Wangke Chen, Huaxi Ai, Haoyu Wen, Zeming Liu, Yuhang Guo ### Benchmarking LLM Judges for Mobile Agent Evaluation MobileJudgeBench compares completion judges against human labels on mobile-agent trajectories. It studies screenshot-sampling baselines, model-dependent error profiles and how judge quality relates to agent rankings and reinforcement-learning rewards. - https://awesomecua.com/resources/arxiv-2608-11434/ · source: https://arxiv.org/abs/2608.11434v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, reward-models, robustness - Published: 2026-08-11 · Authors: Ziqiang Wang, Li Gu, Zhixiang Chi, Zhi Liu, Seyed Mehdi Ayyoubzadeh, Yuanhao Yu, Yang Wang ### Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety Evaluates multimodal LLMs on web interface reasoning, visual perturbation robustness, and safety-critical web understanding tasks. - https://awesomecua.com/resources/arxiv-2509-21782/ · source: https://arxiv.org/abs/2509.21782v2 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, screen-understanding, safety-security, robustness - Published: 2025-09-26 · Authors: Junliang Liu, Jingyu Xiao, Wenxin Tang, Zhixian Wang, Zipeng Xie, Wenxuan Wang, Minrui Zhang, Shuanghe Yu ### Benchmarking Mobile Device Control Agents across Diverse Configurations Introduces B-MoCA, an interactive mobile-control benchmark that varies device settings, layouts, and languages to test generalization. - https://awesomecua.com/resources/arxiv-2404-16660/ · source: https://arxiv.org/abs/2404.16660v3 - Paper · reviewed 2026-10-16 · environments: android · topics: evaluation, robustness - Published: 2024-04-25 · Authors: Juyong Lee, Taywon Min, Minyong An, Dongyoon Hahm, Haeone Lee, Changyeon Kim, Kimin Lee ### Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces Introduces WebDecept, a plugin for injecting deceptive e-commerce interface patterns during web-agent tasks and evaluating how multimodal agents respond. - https://awesomecua.com/resources/arxiv-2606-13686/ · source: https://arxiv.org/abs/2606.13686v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2026-04-26 · Authors: Zijing Shi, Meng Fang, Ling Chen ### Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment BBCritic ranks candidate GUI-agent actions through instruction–action alignment, while BBBench evaluates fine-grained ranking with a hierarchical action taxonomy. - https://awesomecua.com/resources/arxiv-2605-14311/ · source: https://arxiv.org/abs/2605.14311v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: action-prediction, evaluation, reward-models - Published: 2026-05-14 · Authors: Yuchen Sun, Pei Fu, Shaojie Zhang, Anan Du, Xiuwen Xi, Ruoceng Zhang, Zhenbo Luo, Jian Luan, Chongyang Zhang ### Beyond Browsing: API-Based Web Agents Compares browsing-only, API-only, and hybrid web agents on WebArena, examining how access to application APIs changes task execution alongside browser interaction. - https://awesomecua.com/resources/arxiv-2410-16464/ · source: https://arxiv.org/abs/2410.16464v3 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration, evaluation - Published: 2024-10-21 · Authors: Yueqi Song, Frank Xu, Shuyan Zhou, Graham Neubig ### Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation DOMSteer translates help requests and live page context into reversible DOM changes such as control highlighting, contextual tooltips and layout reorganization. The work evaluates in-place GUI assistance with interface tests, use cases and a comparative user study. - https://awesomecua.com/resources/arxiv-2604-14668/ · source: https://arxiv.org/abs/2604.14668v1 - Paper · reviewed 2026-10-18 · environments: web · topics: human-in-the-loop, ui-design, grounding, evaluation - Published: 2026-04-16 · Authors: Pan Hao, Rishi Selvakumaran, Jacob Sun, Qianwen Wang ### Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Dragging GUI-Drag provides synthesized text-dragging examples, while ScreenDrag benchmarks selecting text by dragging across varied GUI contexts. - https://awesomecua.com/resources/arxiv-2601-06031/ · source: https://arxiv.org/abs/2601.06031v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, data-generation, evaluation - Published: 2025-11-07 · Authors: Zeyi Liao, Yadong Lu, Boyu Gou, Huan Sun, Ahmed Awadallah ### Beyond Domains: Reusing Web Skills via Transferable Interaction Patterns Introduces SkillMigrator, a web agent that turns successful interactions into reusable skills and transfers them between sites by matching page layouts and live element references. - https://awesomecua.com/resources/arxiv-2606-17645/ · source: https://arxiv.org/abs/2606.17645v1 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, efficiency, memory, workflow - Published: 2026-06-16 · Authors: Shiqi He, Yue Cui, Feijie Wu, Xinyu Ma, Jiaheng Lu, Yaliang Li, Bolin Ding, Mosharaf Chowdhury ### Beyond Next-Screen Prediction: The Decision Value of GUI World Models Separates next-screen prediction from action evaluation in GUI lookahead and trains Direct Action Ranking on reference actions, comparing prediction-augmented scoring under matched offline supervision. - https://awesomecua.com/resources/arxiv-2610-24372/ · source: https://arxiv.org/abs/2610.24372v1 - Paper · reviewed 2026-10-20 · environments: android, web · topics: evaluation, world-models, action-prediction, imitation-learning - Published: 2026-10-17 · Authors: Sizhe Tang, Hanhan Zhou, Jeffrey Svajlenko, Rashmi Gangadharaiah ### Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation Sphinx benchmarks foundation models on goal-based mobile UI navigation, using app test cases and a toolkit that identifies different failure modes. - https://awesomecua.com/resources/arxiv-2501-02863/ · source: https://arxiv.org/abs/2501.02863v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, mobile-navigation, industry - Published: 2025-01-06 · Authors: Dezhi Ran, Mengzhou Wu, Hao Yu, Yuetong Li, Jun Ren, Yuan Cao, Xia Zeng, Haochuan Lu, Zexin Xu, Mengqian Xu, Ting Su, Liangchao Yao, Ting Xiong, Wei Yang, Yuetang Deng, Assaf Marron, David Harel, Tao Xie ### Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents D2Snap reduces serialized DOM observations while preserving actionable elements and features that distinguish their uses. The study compares reduced DOM inputs with screenshot-based observations for a web agent on tasks sampled from Online-Mind2Web. - https://awesomecua.com/resources/arxiv-2508-04412/ · source: https://arxiv.org/abs/2508.04412v4 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, efficiency, screen-understanding - Published: 2025-08-06 · Authors: Thassilo M. Schiepanski, Nicholas Piël ### Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations AniMINT evaluates how VLMs perceive, explain and interpret UI animations using densely annotated videos. The study distinguishes basic motion recognition from higher-level animation meaning and probes motion, context and perceptual cues. - https://awesomecua.com/resources/arxiv-2604-26148/ · source: https://arxiv.org/abs/2604.26148v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, screen-understanding, video, ui-design - Published: 2026-04-28 · Authors: Chen Liang, Xirui Jiang, Naihao Deng, Eytan Adar, Anhong Guo ### Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents Introduces ParaGUIBench, with tasks and efficiency measurements for coordinated GUI agents operating separate desktop instances over a shared file system. Its ParaGUI planner-worker system decomposes tasks and dispatches concurrent subtasks for comparison with serial execution. - https://awesomecua.com/resources/arxiv-2607-22689/ · source: https://arxiv.org/abs/2607.22689v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, planning, efficiency, long-horizon - Published: 2026-07-17 · Authors: Zedong Yu, Qianxing Li, Zhi Gao, Liuyu Xiang, Chenrui Shi, Yang Liu, Huiming Wu, Yujie Wei, Yuhao Fei, Yubo Fu, Zhaofeng He ### Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents Introduces LACL-GUI, a contrastive learning objective that distinguishes trajectory quality within both successful and failed GUI executions. It favors concise successes and uses divergence from successful trajectories to differentiate failures during policy optimization. - https://awesomecua.com/resources/arxiv-2608-21830/ · source: https://arxiv.org/abs/2608.21830v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reinforcement-learning, efficiency, action-prediction - Published: 2026-08-22 · Authors: Chengyang Gu, Le Zhang, Jingbo Zhou, Yize Chen, Yu Shi, Siqi Bao, Zheng-Fan Wu, Hua Wu, Hui Xiong ### Beyond Task Completion: Training Capable and Safe Computer-Use Agents Introduces SCOPE, which trains computer-use agents on ordinary task demonstrations, safe continuations around hazards, and explicit refusals. Its SCOPE-Gen pipeline pairs verifiable tasks with environmental-risk variants to support joint capability and safety training. - https://awesomecua.com/resources/arxiv-2609-22178/ · source: https://arxiv.org/abs/2609.22178v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: safety-security, reinforcement-learning, data-generation - Published: 2026-08-27 · Authors: Zeyu Kang, Zhenyun Yin, Yang Zhang, Shan He, Shanzhe Lei, Yanjiu Zhong, Xinquan Chen, Xuhong Wang ### Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen? This study compares general coding agents using mobile command-line access with reproducible GUI agents on AndroidWorld and MobileWorld. It also supplies oracle CLI analyses and a separate task suite for operations such as bulk changes, filtering and hidden device state. - https://awesomecua.com/resources/arxiv-2606-19388/ · source: https://arxiv.org/abs/2606.19388v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, automation, cross-app, tool-integration - Published: 2026-06-16 · Authors: Li Gu, Zihuan Jiang, Linqiang Guo, Zhixiang Chi, Ziqiang Wang, Huan Liu, Yuanhao Yu, Tse-Hsun Chen, Yang Wang ### Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails Introduces Mind2Web-Injection, a benchmark for evaluating web-agent guardrail decisions on instruction-screenshot pairs using evidence boxes and image-side counterfactuals. - https://awesomecua.com/resources/arxiv-2609-05535/ · source: https://arxiv.org/abs/2609.05535v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, safety-security, screen-understanding - Published: 2026-09-02 · Authors: Suyoung Lee, Myungsub Choi ### BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks BigDocs-7.5M provides multimodal document data, while BigDocs-Bench includes GUI reasoning and image-to-code tasks relevant to screen understanding. - https://awesomecua.com/resources/arxiv-2412-04626/ · source: https://arxiv.org/abs/2412.04626v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, evaluation, screen-understanding - Published: 2024-12-05 · Authors: Juan Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-André Noël, Mats Leon Richter, Saverio Vadacchino, Shubham Agarwal, Sanket Biswas, Sara Shanian, Ying Zhang, Noah Bolger, Kurt MacDonald, Simon Fauvel, Sathwik Tejaswi, Srinivas Sunkara, Joao Monteiro, Krishnamurthy DJ Dvijotham, Torsten Scholak, Nicolas Chapados, Sepideh Kharagani, Sean Hughes, M. Özsu, Siva Reddy, Marco Pedersoli, Yoshua Bengio, Christopher Pal, Issam Laradji, Spandana Gella, Perouz Taslakian, David Vazquez, Sai Rajeswar ### BIMgent: Towards Autonomous Building Modeling via Computer-use Agents BIMgent plans software-specific workflows and executes GUI actions for building-information modeling. It evaluates multimodal building design and reconstruction tasks in professional authoring software. - https://awesomecua.com/resources/arxiv-2506-07217/ · source: https://arxiv.org/abs/2506.07217v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: industry, planning, workflow, evaluation - Published: 2025-06-08 · Authors: Zihan Deng, Changyu Du, Stavros Nousias, André Borrmann ### BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References Benchmarks agents creating editable Blender projects through shell, Python and MCP tools, judging delivered scenes with multi-view renders, task-specific rubrics and visual-quality ratings. - https://awesomecua.com/resources/arxiv-2610-23986/ · source: https://arxiv.org/abs/2610.23986v1 - Paper · reviewed 2026-10-20 · environments: desktop, linux · topics: evaluation, tool-integration, sandbox - Published: 2026-10-14 · Authors: Xinyu Fang, Yuchi Wang, Weihao Yu, Rongyao Fang, Zhaohai Li, Zhixun Li, Tiancheng Gu, Xilin Wei, Yuhang Zang, Hongsheng Li, Dahua Lin, Guofeng Zhang, Shuai Bai, Zhibo Yang, Dayiheng Liu ### Blind Gods and Broken Screens: Architecting a Secure, Intent-Centric Mobile Agent Operating System Analyzes security risks in screen-based mobile agents and proposes Aura, an agent runtime with structured interactions, app-agent isolation and mediated communication. - https://awesomecua.com/resources/arxiv-2602-10915/ · source: https://arxiv.org/abs/2602.10915v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, mobile-navigation, sandbox - Published: 2026-02-11 · Authors: Zhenhua Zou, Sheng Guo, Qiuyang Zhan, Lepeng Zhao, Shuo Li, Qi Li, Ke Xu, Mingwei Xu, Zhuotao Liu ### BlueFin: Benchmarking LLM Agents on Financial Spreadsheets Introduces BlueFin, a benchmark and evaluation framework for agents handling professional finance spreadsheet workbooks through synthesis, manipulation, and comprehension tasks. - https://awesomecua.com/resources/arxiv-2605-30907/ · source: https://arxiv.org/abs/2605.30907v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, industry, workflow - Published: 2026-05-29 · Authors: Srivatsa Kundurthy, Clara Na, Colton Moraine, Anoushka Mohta, Case Winter, George Fang, John Ling, Emma Strubell, Zach Kirshner ### BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents Describes a mobile-agent training pipeline centered on real-device rollouts, recovery of useful supervision from failures, and evolving evaluation tasks. - https://awesomecua.com/resources/arxiv-2609-12394/ · source: https://arxiv.org/abs/2609.12394v3 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, reinforcement-learning, data-generation - Published: 2026-09-11 · Authors: see the paper ### Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory Branch-and-Browse is a web agent framework that uses tree-structured subtasks, web state replay, and page action memory to explore and complete tasks. - https://awesomecua.com/resources/arxiv-2510-19838/ · source: https://arxiv.org/abs/2510.19838v2 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, memory, efficiency, web-navigation - Published: 2025-10-18 · Authors: Shiqi He, Yue Cui, Xinyu Ma, Yaliang Li, Bolin Ding, Mosharaf Chowdhury ### BraveGuard: From Open-World Threats to Safer Computer-Use Agents Introduces BraveGuard, a framework that turns emerging threats and computer-use agent trajectories into training supervision for guards that monitor multi-step safety risks. - https://awesomecua.com/resources/arxiv-2606-01166/ · source: https://arxiv.org/abs/2606.01166v2 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, observability, data-generation - Published: 2026-05-31 · Authors: Yunhao Feng, Xiaohu Du, Xinhao Deng, Yifan Ding, Ming Wen, Yixu Wang, Yuxiang Xie, Baihui Zheng, Yingshui Tan, Yige Li, Yutao Wu, Kerui Cao, Wenke Huang, Yanming Guo, Xingjun Ma, Yu-Gang Jiang ### Breaking the Data Barrier -- Building GUI Agents Through Task Generalization Studies whether mid-training on GUI perception, multimodal reasoning, and text reasoning transfers to GUI-agent planning. Experiments compare 11 task sources and curated mixtures using WebArena and AndroidWorld to investigate training-data choices when GUI trajectories are scarce. - https://awesomecua.com/resources/arxiv-2504-10127/ · source: https://arxiv.org/abs/2504.10127v2 - Paper · reviewed 2026-10-17 · environments: web, android · topics: data-generation, planning, evaluation - Published: 2025-04-14 · Authors: Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, Junxian He ### Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction Introduces VG-GUIBench to evaluate GUI agents following video tutorials, and TASKER to select task-relevant keyframes for video-guided tasks and VideoQA. - https://awesomecua.com/resources/arxiv-2606-29445/ · source: https://arxiv.org/abs/2606.29445v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, video, long-horizon - Published: 2026-06-28 · Authors: Sunqi Fan, Qingle Liu, Runqi Yin, Meng-Hao Guo, Shuojin Yang ### Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents Measures how autonomous LLM browser agents interact with commercial bot-management and anti-automation defenses on the web. - https://awesomecua.com/resources/arxiv-2607-18659/ · source: https://arxiv.org/abs/2607.18659v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2026-07-21 · Authors: Behzad Ousat, Nikita Turkmen, Lalchandra Rampersaud, Dillan Bailey, Amin Kharraz ### BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions Trains BrowserAgent to use Playwright-based scrolling, clicking and typing directly on webpages for information-seeking tasks. Supervised and rejection fine-tuning are paired with explicit memory for conclusions carried across steps. - https://awesomecua.com/resources/arxiv-2510-10666/ · source: https://arxiv.org/abs/2510.10666v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, imitation-learning, memory - Published: 2025-10-12 · Authors: Tao Yu, Zhengbo Zhang, Zhiheng Lyu, Junhao Gong, Hongzhu Yi, Xinming Wang, Yuxuan Zhou, Jiabing Yang, Ping Nie, Yan Huang, Wenhu Chen ### BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks Introduces BrowserArena, a live-web platform for head-to-head agent comparisons on user-submitted tasks. Step-level human annotations support targeted analysis of navigation, pop-up handling and CAPTCHA-related failures. - https://awesomecua.com/resources/arxiv-2510-02418/ · source: https://arxiv.org/abs/2510.02418v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, human-in-the-loop - Published: 2025-10-02 · Authors: Sagnik Anupam, Davis Brown, Shuo Li, Eric Wong, Hamed Hassani, Osbert Bastani ### BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes Introduces BrowserForge, which combines broad website sourcing, parallel browser sandboxes, and task-proposer/solver interaction to collect verified web trajectories. The pipeline uses page structure during synthesis but trains a policy that acts from screenshots. - https://awesomecua.com/resources/arxiv-2608-24848/ · source: https://arxiv.org/abs/2608.24848v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, sandbox, web-navigation - Published: 2026-08-25 · Authors: Fei Tang, Huawen Shen, Zhiqiong Lu, Zhengxi Lu, Pengyuan Lyu, Chengquan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen ### BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser Views Extends BrowseSafe with paired attacked and clean webpages in several representations, and studies text and screenshot detectors plus feedback-adapted prompt injections. - https://awesomecua.com/resources/arxiv-2610-17457/ · source: https://arxiv.org/abs/2610.17457v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2026-10-12 · Authors: Yuetian Chen, Kaiyuan Zhang, Yuntao Du, Thomas Li, Mark Tenenholtz, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li ### BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents BrowseSafe builds a benchmark of prompt injections in realistic HTML for browser agents and evaluates defenses against attacks that can affect agent actions. - https://awesomecua.com/resources/arxiv-2511-20597/ · source: https://arxiv.org/abs/2511.20597v2 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2025-11-25 · Authors: Kaiyuan Zhang, Mark Tenenholtz, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li ### BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent BTL-UI separates GUI interaction into screen-region attention, planning and executable-command generation. It adds an automated annotation pipeline and rule-based process/outcome rewards for training, with evaluation on both static understanding and dynamic interaction. - https://awesomecua.com/resources/arxiv-2509-15566/ · source: https://arxiv.org/abs/2509.15566v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, planning, data-generation, reinforcement-learning - Published: 2025-09-19 · Authors: Shaojie Zhang, Ruoceng Zhang, Pei Fu, Shaokang Wang, Jiahui Yang, Xin Du, Shiqi Cui, Bin Qin, Ying Huang, Zhenbo Luo, Jian Luan ### Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing Compares a custom harness with general-purpose LLM agent harnesses for GUI testing of Android-based vehicle interfaces and tasks in public apps. - https://awesomecua.com/resources/arxiv-2610-33636/ · source: https://arxiv.org/abs/2610.33636v1 - Paper · reviewed 2026-11-06 · environments: android · topics: evaluation, industry, observability - Published: 2026-10-24 · Authors: Arsham Gholamzadeh Khoee, Ruyu Fu, Jiaqi Wei, Ruikun Deng, Robert Feldt, Dhasarathy Parthasarathy, Yinan Yu ### Build the web for agents, not agents for the web Proposes Agentic Web Interface (AWI), a website interface intended for agent navigation, and sets out six design principles for web-agent use. - https://awesomecua.com/resources/arxiv-2506-10953/ · source: https://arxiv.org/abs/2506.10953v1 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, web-navigation, safety-security, efficiency - Published: 2025-06-12 · Authors: Xing Han Lù, Gaurav Kamath, Marius Mosbach, Siva Reddy ### Building a Stable Planner: An Extended Finite State Machine Based Planning Module for Mobile GUI Agent SPlanner models mobile-app functions and configurations with extended finite-state machines, traverses them to construct execution paths, and converts those paths into natural-language plans. A VLM follows these plans to perform GUI actions, with evaluation on AndroidWorld. - https://awesomecua.com/resources/arxiv-2505-14141/ · source: https://arxiv.org/abs/2505.14141v1 - Paper · reviewed 2026-10-17 · environments: android · topics: planning, mobile-navigation - Published: 2025-05-20 · Authors: Fanglin Mo, Junzhe Chen, Haoxuan Zhu, Xuming Hu ### Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization This framework learns an agentic-Q model to estimate individual actions’ contributions to GUI task completion. It then optimizes a policy with step-level samples from its own trajectories, decoupling policy updates from environment interaction. - https://awesomecua.com/resources/arxiv-2602-13653/ · source: https://arxiv.org/abs/2602.13653v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reinforcement-learning, reward-models, grounding - Published: 2026-02-14 · Authors: Yibo Wang, Guangda Huzhang, Yuwei Hu, Yu Xia, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang ### Building Browser Agents: Architecture, Security, and Practical Solutions Reports architectural and security lessons from building a browser agent, including hybrid accessibility-tree and visual context, browser tooling and programmatic constraints. A WebGames evaluation accompanies the discussion of reliability and specialization. - https://awesomecua.com/resources/arxiv-2511-19477/ · source: https://arxiv.org/abs/2511.19477v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, safety-security, tool-integration - Published: 2025-11-22 · Authors: Aram Vardanyan ### CA-OPD: Confidence-Aware On-Policy Distillation for Structured Visual Prediction Introduces CA-OPD, a confidence-guided distillation method for visual prediction, evaluated on GUI grounding and optical character recognition benchmarks. - https://awesomecua.com/resources/arxiv-2609-02401/ · source: https://arxiv.org/abs/2609.02401v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: imitation-learning, grounding, evaluation - Published: 2026-09-02 · Authors: Menghao Li, Linjie Mu, Haotian Hu, Yannian Gu, Lujiayi Xue, Liujian Tang, Yu Zhang, Fanyi Wang ### CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only CAAP converts screenshots into textual observations and plans keyboard-and-mouse actions using a prompt that examines task context from multiple perspectives. The agent is evaluated on MiniWoB++ and WebShop using only front-end visual observations. - https://awesomecua.com/resources/arxiv-2406-06947/ · source: https://arxiv.org/abs/2406.06947v3 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, screen-understanding, web-navigation - Published: 2024-06-11 · Authors: Junhee Cho, Jihoon Kim, Daseul Bae, Jinho Choo, Youngjune Gwon, Yeong-Dae Kwon ### CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers CAD-Assistant plans CAD tasks and executes actions through FreeCAD's Python API, using the changing design state to guide later commands. - https://awesomecua.com/resources/arxiv-2412-13810/ · source: https://arxiv.org/abs/2412.13810v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: science, planning, tool-integration, state-tracking - Published: 2024-12-18 · Authors: Dimitrios Mallis, Ahmet Serdar Karadeniz, Sebastian Cavada, Danila Rukhovich, Niki Foteinopoulou, Kseniya Cherenkova, Anis Kacem, Djamila Aouada ### CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design CADWorld evaluates screenshot-and-GUI agents on long FreeCAD workflows, from modeling and assembly to manufacturing and simulation. Executable checks inspect saved engineering artifacts for geometry, constraints, construction structure and downstream state. - https://awesomecua.com/resources/arxiv-2609-16251/ · source: https://arxiv.org/abs/2609.16251v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, long-horizon, workflow, science - Published: 2026-09-14 · Authors: Zihan Dong, Yuanzhe Liu, Zhiyuan Ma, Qishi Zhan, Dehan Kong, Guohao Li, Kaixin Li ### CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents NOVA separates trusted task planning from screen-derived observations by producing a branching plan before execution and resolving runtime values through perception. Its OSWorld study examines security and utility while identifying remaining risks from perception-driven branch steering. - https://awesomecua.com/resources/arxiv-2601-09923/ · source: https://arxiv.org/abs/2601.09923v3 - Paper · reviewed 2026-10-17 · environments: desktop · topics: safety-security, planning, robustness - Published: 2026-01-14 · Authors: Hanna Foerster, Tom Blanchard, Kristina Nikolić, Ilia Shumailov, Cheng Zhang, Robert Mullins, Nicolas Papernot, Florian Tramèr, Yiren Zhao ### Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces Introduces PAIR for controlled evaluation of mobile GUI agents across user-conditioned application states and RePAIR for training from cross-user subgoal outcomes. The study examines failures when personalized interfaces change which items an agent must identify. - https://awesomecua.com/resources/arxiv-2610-07972/ · source: https://arxiv.org/abs/2610.07972v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, robustness, mobile-navigation, reinforcement-learning - Published: 2026-10-06 · Authors: Yeji Park, Jaeyun Shim, Taesik Gong ### Can LLM Agents Learn to Test? Maintaining Scoped Experience as Testing Evidence for Mobile GUI Testing AdaptiTest specifies scoped, revalidated memory for an LLM-based mobile GUI tester to reuse routes and locators across read-only Android test workflows. - https://awesomecua.com/resources/arxiv-2611-05873/ · source: https://arxiv.org/abs/2611.05873v1 - Paper · reviewed 2026-11-06 · environments: android · topics: memory, interaction-history, mobile-navigation, workflow - Published: 2026-11-04 · Authors: Zimu Liu, Xiaoxi Guo, Xiaolu Tang, Liang Yu ### Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes Tests natural-language no-code fixes through GUI execution in Linux Brave. Scripted fixed/buggy-state gates and issue-specific checkers distinguish fix outcomes from the executor's completion claims. - https://awesomecua.com/resources/arxiv-2610-11963/ · source: https://arxiv.org/abs/2610.11963v1 - Paper · reviewed 2026-10-20 · environments: web, desktop, linux · topics: evaluation, web-navigation - Published: 2026-10-08 · Authors: Utku Boran Torun, Veli Karakaya, Eray Tüzün ### Can We Stop Malicious AI? KILLBENCH: A Benchmark for External AI Kill Switch Feasibility KillBench evaluates external prompt-style signals intended to halt maliciously operating web agents, across harmful scenarios and agent configurations. - https://awesomecua.com/resources/arxiv-2511-13725/ · source: https://arxiv.org/abs/2511.13725v5 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, safety-security, web-navigation - Published: 2025-09-26 · Authors: Sechan Lee, Hyounghun Kim, Sangdon Park ### CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception CAP builds cross-site browser workflows by decomposing websites into operation and perception requirements and recomposing them into tasks. Its judge-based evaluation diagnoses complex interactions and visual-perception failures across real websites. - https://awesomecua.com/resources/arxiv-2608-08392/ · source: https://arxiv.org/abs/2608.08392v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, web-navigation, cross-app, screen-understanding - Published: 2026-08-09 · Authors: Zejun Xu, Taiyi Chen, Jin Li, Yongtong Gu, Qi Cheng, Aixuan Lv, Shuai Zhu, Pengfei Zhu, Kaichen Yang, Boyu Sun, Yixian Yang, Mulong Xie, Xin Liu, Dagang Li, Xiaoteng Ma, Hongru Wang ### Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity? AgentCIBench evaluates whether computer-use agents disclose information in ways that violate its original context. Deterministically scored scenarios test nearby sensitive UI content, ambiguous requests and recipient mismatches, including end-to-end agent execution. - https://awesomecua.com/resources/arxiv-2606-23189/ · source: https://arxiv.org/abs/2606.23189v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, safety-security, cross-app - Published: 2026-06-22 · Authors: Anmol Goel, Iryna Gurevych ### CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents CAPED filters mobile screenshots before upload by combining task requirements, screen context and parsed UI elements to mask incidental private content. AndroidWorld and a controlled seeded privacy evaluation examine leakage reduction alongside task utility and prototype costs. - https://awesomecua.com/resources/arxiv-2606-12666/ · source: https://arxiv.org/abs/2606.12666v2 - Paper · reviewed 2026-10-17 · environments: android · topics: safety-security, screen-understanding, mobile-navigation - Published: 2026-06-10 · Authors: Siyu Shen, Fenghao Xu, Wenrui Diao, Kehuan Zhang ### CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training Introduces ReCAP, a native GUI agent for interactive CAPTCHA challenges, trained with generated interaction trajectories, reasoning traces and self-correction data. - https://awesomecua.com/resources/arxiv-2603-23559/ · source: https://arxiv.org/abs/2603.23559v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, self-reflection, automation - Published: 2026-03-23 · Authors: Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang ### CaptchaAgent: Verifiable Visual Reasoning for CAPTCHA Solving Presents CaptchaAgent, which builds an explicit visual representation of objects and relations, refines ambiguous regions, and checks counterfactual consistency before acting on interactive CAPTCHAs. - https://awesomecua.com/resources/arxiv-2610-12866/ · source: https://arxiv.org/abs/2610.12866v1 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, screen-understanding, evaluation - Published: 2026-10-09 · Authors: Shangquan Sun, Ruoyu Chen, Zhirui Liu, Xiang Liu, Chenghao Qian, Wenqi Ren, Xiaochun Cao ### CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs Introduces CaptchaArena, a dataset of execution-verified interactive CAPTCHA solutions with screenshot–action trajectories, reasoning annotations, and pixel masks for irregular targets. The work trains a single policy across CAPTCHA types using supervised learning followed by verifier-reward reinforcement learning. - https://awesomecua.com/resources/arxiv-2609-31957/ · source: https://arxiv.org/abs/2609.31957v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, grounding, reinforcement-learning - Published: 2026-09-25 · Authors: Zhenhao Zhang, Zhaoyu Fan, Haohan Ying, Jingwen Hu, Hancen Fan, Junhao Zhou, Zitian Chen, Linchao Zhu ### CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare CareFlow benchmarks long healthcare-software workflows, while CarePilot coordinates action proposals, execution checks and corrective feedback using short- and long-term memory. Its actor consumes visual interfaces and system state to select semantic actions. - https://awesomecua.com/resources/arxiv-2603-24157/ · source: https://arxiv.org/abs/2603.24157v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, planning, memory, evaluation, long-horizon, workflow - Published: 2026-03-25 · Authors: Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh, Numan Saeed, Sriparna Saha, Xiuying Chen, Salman Khan ### CausalCache: Conditional High-Fidelity Restoration for Long-Horizon GUI Agents Introduces CausalCache, which selects archived screenshots to restore alongside text summaries under a fixed visual-memory budget. A history-gated adapter processes restored images, with evaluation distinguishing standard step limits from longer-horizon and mobile-transfer settings. - https://awesomecua.com/resources/arxiv-2608-22577/ · source: https://arxiv.org/abs/2608.22577v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: memory, long-horizon, efficiency - Published: 2026-08-23 · Authors: Jiaxuan Luo, Zhanfeng Liao, Jiayao Teng, Yuan Wang ### Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions Measures how irrelevant GUI content affects multimodal agents using a simulated dataset and three perception patterns. The study also examines adversarial environment injection and possible ways to improve instruction faithfulness. - https://awesomecua.com/resources/arxiv-2408-02544/ · source: https://arxiv.org/abs/2408.02544v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, robustness, evaluation - Published: 2024-08-05 · Authors: Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, Hai Zhao ### CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments Introduces CAVEAT, a controlled marketplace benchmark for testing whether delegated agents preserve user objectives when interfaces favor other outcomes. It diagnoses decision failures and studies harness interventions and targeted training to improve resistance to marketplace steering. - https://awesomecua.com/resources/arxiv-2609-27273/ · source: https://arxiv.org/abs/2609.27273v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, robustness, safety-security - Published: 2026-09-23 · Authors: Yuxuan Li, Will Epperson, Wesley Deng, Zezhou Huang ### ceLLMate: Sandboxing Browser AI Agents ceLLMate constrains browser-agent authority through an extension that applies policies at the HTTP layer. It evaluates this sandbox against prompt-injection attacks in WASP, avoiding dependence on brittle low-level click and keystroke policies. - https://awesomecua.com/resources/arxiv-2512-12594/ · source: https://arxiv.org/abs/2512.12594v2 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, sandbox, web-navigation, robustness, evaluation - Published: 2025-12-14 · Authors: Luoxi Meng, Henry Feng, Ilia Shumailov, Earlence Fernandes ### CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning Combines supervised and reinforcement fine-tuning for continual GUI learning, using policy entropy to adjust their proportions and gradient projection to limit conflicting updates. AndroidControl-CL groups applications into sequential tasks to evaluate adaptation and retention. - https://awesomecua.com/resources/arxiv-2603-02951/ · source: https://arxiv.org/abs/2603.02951v2 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, robustness, evaluation - Published: 2026-03-03 · Authors: Zhenquan Yao, Zitong Huang, Yihan Zeng, Jianhua Han, Hang Xu, Chun-Mei Feng, Jianwei Ma, Wangmeng Zuo ### Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback Chain-of-Ground uses iterative visual reasoning and reference feedback to refine GUI localization without additional training. It evaluates on ScreenSpot-Pro and introduces TPanel-UI to probe grounding on industrial control panels with visual distortions. - https://awesomecua.com/resources/arxiv-2512-01979/ · source: https://arxiv.org/abs/2512.01979v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, robustness, evaluation - Published: 2025-12-01 · Authors: Aiden Yiliu Li, Bizhi Yu, Daoan Lei, Tianhe Ren, Shilong Liu ### Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation Chain-of-Memory explicitly records action descriptions and task-relevant screen information in short- and long-term memory for GUI navigation. GUI Odyssey-CoM provides 111,000 screen-action pairs with memory annotations for learning and evaluating information retention across applications. - https://awesomecua.com/resources/arxiv-2506-18158/ · source: https://arxiv.org/abs/2506.18158v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, state-tracking, cross-app, data-generation - Published: 2025-06-22 · Authors: Xinzge Gao, Chuanrui Hu, Bin Chen, Teng Li ### ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks ChainWorld composes compatible atomic OSWorld tasks into multi-objective desktop chains while preserving their evaluators. It compares giving the full sequence at once with revealing goals turn by turn, exposing different state and session-management failures. - https://awesomecua.com/resources/arxiv-2606-21654/ · source: https://arxiv.org/abs/2606.21654v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, long-horizon, state-tracking, workflow - Published: 2026-06-19 · Authors: Vincent Siu, Manasi Sharma, Dawn Song, Daniel Yue Zhang, Chenguang Wang, Ying Liu ### Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsing Studies unintended consequences of collaborative web browsing with GUI agents through social-media analysis and interviews. It organizes observed failures, user impacts, and mitigation practices around planning, interface interaction, feedback, privacy, and trust. - https://awesomecua.com/resources/arxiv-2505-09875/ · source: https://arxiv.org/abs/2505.09875v2 - Paper · reviewed 2026-10-17 · environments: web · topics: human-in-the-loop, evaluation, safety-security - Published: 2025-05-15 · Authors: Shuning Zhang, Jingruo Chen, Zhiqi Gao, Jiajing Gao, Xin Yi, Hewu Li ### Chart the Invisible: Interactive Perception for Hidden-Target Exploration in GUI Agents Introduces Hagasu, a scrollable-region perception model that helps GUI agents decide where to swipe to reveal hidden targets, especially on mobile screens. - https://awesomecua.com/resources/arxiv-2610-35237/ · source: https://arxiv.org/abs/2610.35237v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, mobile-navigation, data-generation - Published: 2026-10-26 · Authors: Yongxiang Hu, Ruiyang Liu, Qining He, Xuan Wang, Yangfan Zhou ### ChartAct: A Benchmark for Dynamic Chart Understanding ChartAct benchmarks multimodal models and GUI agents on interactive charts from websites, requiring actions such as hovering or zooming to answer questions. - https://awesomecua.com/resources/arxiv-2605-26994/ · source: https://arxiv.org/abs/2605.26994v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, screen-understanding, state-tracking - Published: 2026-05-26 · Authors: Muye Huang, Lin Wu, Lingling Zhang, Hang Yan, Zhiyuan Wang, Yumeng Fu, Zesheng Yang, Jun Liu ### CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planning Uses human-planned subtasks to constrain and refine the intermediate plans connecting user goals with executable mobile operations. - https://awesomecua.com/resources/arxiv-2503-03743/ · source: https://arxiv.org/abs/2503.03743v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, planning - Published: 2025-03-05 · Authors: Yuqi Zhou, Shuai Wang, Sunhao Dai, Qinglin Jia, Zhaocheng Du, Zhenhua Dong, Jun Xu ### CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline Introduces CLASHarness, generating verifiable computer-use tasks with conflicting agent reports, shared-state constraints, and coordination requirements to evaluate conflict detection and resolution. - https://awesomecua.com/resources/arxiv-2610-19128/ · source: https://arxiv.org/abs/2610.19128v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, robustness, workflow - Published: 2026-10-13 · Authors: Yuxiang Lai, Jike Zhong, Chengzhi Liu, Ming Li, Haonan Wang, Xiaofeng Yang, Liang Zhao, Youbao Tang, Ruixi Fan ### Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World Claw-Anything benchmarks persistent personal assistance across activity histories, backend services and integrated GUI/CLI interaction. Its simulated event histories introduce irrelevant or conflicting context, and a separate generation pipeline supplies training environments. - https://awesomecua.com/resources/arxiv-2605-26086/ · source: https://arxiv.org/abs/2605.26086v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, interaction-history, long-horizon, robustness, data-generation, workflow - Published: 2026-05-25 · Authors: Yusong Lin, Xinyuan Liang, Haiyang Wang, Qipeng Gu, Siqi Cheng, Jiangui Chen, Shuzhe Wu, Feiyang Pan, Lue Fan, Sanyuan Zhao, Dandan Tu ### ClawBench: Can AI Agents Complete Everyday Online Tasks? Introduces ClawBench, 153 everyday tasks on production websites in 15 categories; an interception layer blocks each final submission so bookings, purchases and job applications are scored without side effects. - https://awesomecua.com/resources/arxiv-2604-08523/ · source: https://arxiv.org/abs/2604.08523v2 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, web-navigation, workflow - Published: 2026-04-09 · Authors: see the paper ### ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents Presents ClawGUI, a shared harness for GUI-agent reinforcement learning, benchmark evaluation and deployment on mobile devices. It combines parallel virtual and physical environments, process rewards, and hybrid CLI-GUI execution with persistent memory. - https://awesomecua.com/resources/arxiv-2604-11784/ · source: https://arxiv.org/abs/2604.11784v1 - Paper · reviewed 2026-10-17 · environments: android, harmonyos, ios · topics: reinforcement-learning, evaluation, tool-integration, mobile-navigation - Published: 2026-04-13 · Authors: Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen ### ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts Introduces ClawProBench, a trace-aware benchmark for agents on OpenClaw that scores workspace and native-runtime tasks using execution traces, safety checks, and process quality. - https://awesomecua.com/resources/arxiv-2608-22510/ · source: https://arxiv.org/abs/2608.22510v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, observability, safety-security - Published: 2026-08-23 · Authors: YuanHang Xiao ### ClawTrap: A MITM-Based Red-Teaming Framework for Real-World OpenClaw Security Evaluation Presents a man-in-the-middle network red-teaming framework for evaluating autonomous OpenClaw agents under live traffic manipulation. - https://awesomecua.com/resources/arxiv-2603-18762/ · source: https://arxiv.org/abs/2603.18762v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2026-03-19 · Authors: Haochen Zhao, Shaoyang Cui ### CLI-Anything: Towards Agent-Native Computer Use CLI-Anything argues for application interfaces built around structured commands, explicit state and execution feedback. It describes transforming existing software into command-line harnesses and organizing that approach through CLI-Hub. - https://awesomecua.com/resources/arxiv-2606-03854/ · source: https://arxiv.org/abs/2606.03854v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, tool-integration, workflow - Published: 2026-06-02 · Authors: Yuhao Yang, Tianyu Fan, Chao Huang ### ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents Separates multimodal reasoning and action planning from a dedicated UI localization model, with evaluation on Android devices. - https://awesomecua.com/resources/arxiv-2410-11872/ · source: https://arxiv.org/abs/2410.11872v2 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, grounding - Published: 2024-10-09 · Authors: Jakub Hoscilowicz, Bartosz Maj, Bartosz Kozakiewicz, Oleksii Tymoshchuk, Artur Janicki ### CLIFT: Conformal Self-Verification for Web Agent Training and Test-Time Scaling Introduces CLIFT, which certifies agent-generated verification signals against training-time judges and reuses them for per-step rewards and trajectory selection. At inference, a frozen question bank helps compare candidate web rollouts without querying an external judge. - https://awesomecua.com/resources/arxiv-2610-06829/ · source: https://arxiv.org/abs/2610.06829v1 - Paper · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, evaluation, web-navigation - Published: 2026-10-05 · Authors: Yifan Zhang, Yutong Dai, Viraj Prabhu, Zhiyuan Hu, Ran Xu, Zeyuan Chen ### Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models Introduces Co-Design Bench to evaluate coding agents that revise applications from multimodal user requests, including UI changes assessed with browser evidence. - https://awesomecua.com/resources/arxiv-2610-35138/ · source: https://arxiv.org/abs/2610.35138v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, ui-design - Published: 2026-10-26 · Authors: Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang, Weixiang Yu, Cheng Lu, Chonggao Sun, Tong Zhou, Dahua Lin, Hang Zhang, Shuai Bai, Dayiheng Liu ### Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents Co-EPG alternates reinforcement learning for a planner with updates to a grounding model using generated training examples. The evolving grounder provides feedback to the planner, and evaluation uses Multimodal Mind2Web and AndroidControl. - https://awesomecua.com/resources/arxiv-2511-10705/ · source: https://arxiv.org/abs/2511.10705v1 - Paper · reviewed 2026-10-17 · environments: web, android · topics: grounding, planning, reinforcement-learning, data-generation - Published: 2025-11-13 · Authors: Yuan Zhao, Hualei Zhu, Tingyu Jiang, Shen Li, Xiaohang Xu, Hao Henry Wang ### Co-Evolving Acting and Forecasting for Language Agents Introduces CoPE, which trains language agents to forecast later actions from earlier histories during RL; WebShop is one of five evaluation benchmarks. - https://awesomecua.com/resources/arxiv-2610-20597/ · source: https://arxiv.org/abs/2610.20597v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, action-prediction, long-horizon - Published: 2026-10-14 · Authors: Haofei Yu, Yixing Chen, Yining Zhao, Xuyan Ye, Yuxiang Liu, Zhigang Hua, Yan Xie, Shuang Yang, Jiaxuan You ### CoAct-1: Computer-using Multi-Agent System with Coding Actions CoAct-1 delegates subtasks between a GUI operator and a programmer that executes Python or Bash. Its OSWorld evaluation studies a hybrid action space in which scripts handle suitable file and data operations while visual interaction handles other steps. - https://awesomecua.com/resources/arxiv-2508-03923/ · source: https://arxiv.org/abs/2508.03923v3 - Paper · reviewed 2026-10-17 · environments: desktop · topics: tool-integration, automation, efficiency, planning - Published: 2025-08-05 · Authors: Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, Caiming Xiong ### CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications Introduces CoAdapt-GUI, which adapts workflow context and policy from a mobile GUI agent's own rollouts and rewards when it encounters unseen applications. - https://awesomecua.com/resources/arxiv-2608-11588/ · source: https://arxiv.org/abs/2608.11588v1 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, workflow, reinforcement-learning, robustness - Published: 2026-08-12 · Authors: Linqiang Guo, Li Gu, Zihuan Jiang, Zhixiang Chi, Siobhan Reid, Ziqiang Wang, Yuanhao Yu, Wei Liu, Yang Wang, Tse-Hsun, Chen ### CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time Introduces CoArena, an evaluation method where user tasks run concurrently in sandboxed desktops and blind pairwise judgments update a public leaderboard. - https://awesomecua.com/resources/arxiv-2609-14239/ · source: https://arxiv.org/abs/2609.14239v1 - Paper · reviewed 2026-11-06 · environments: desktop · topics: evaluation, sandbox, human-in-the-loop - Published: 2026-09-13 · Authors: Nitish Kovuru, Prateek Jannu ### Coati: Host-Grounded Execution for Installable Mobile GUI Agents Introduces Coati, an installable Android GUI agent that checks action outcomes with phone-side evidence, stores per-app procedures, and routes simpler steps to a fast executor. - https://awesomecua.com/resources/arxiv-2611-02644/ · source: https://arxiv.org/abs/2611.02644v1 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, state-tracking, memory, efficiency - Published: 2026-11-02 · Authors: Yudou Ying, Pu Shi, Hanlei Shi, Yuhua Zheng ### CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation Combines screenshots, layout information, and action history with a factored action predictor for smartphone automation. - https://awesomecua.com/resources/arxiv-2402-11941/ · source: https://arxiv.org/abs/2402.11941v3 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, screen-understanding - Published: 2024-02-19 · Authors: Xinbei Ma, Zhuosheng Zhang, Hai Zhao ### CocoaBench: Evaluating Unified Digital Agents in the Wild Introduces CocoaBench, a benchmark for unified digital agents on long-horizon tasks combining vision, search, and coding, including capabilities related to GUI automation. - https://awesomecua.com/resources/arxiv-2604-11201/ · source: https://arxiv.org/abs/2604.11201v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon, tool-integration - Published: 2026-04-13 · Authors: CocoaBench Team, Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, Haoxiang Zhang, Junli Wang, Hexi Jin, Boyuan Zheng, Kun Zhou, Yu Wang, Feng Yao, Licheng Liu, Yijiang Li, Zhifei Li, Zhengtao Han, Pracha Promthaw, Tommaso Cerruti, Xiaohan Fu, Ziqiao Ma, Jingbo Shang, Lianhui Qin, Julian McAuley, Eric P. Xing, Zhengzhong Liu, Rupesh Kumar Srivastava, Zhiting Hu ### CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning CODA combines a generalist planner with a specialist GUI executor and trains application-specific planners through decoupled GRPO. Successful specialist trajectories then support supervised training of a consolidated planner, evaluated across four ScienceBoard applications. - https://awesomecua.com/resources/arxiv-2508-20096/ · source: https://arxiv.org/abs/2508.20096v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: reinforcement-learning, planning, science, imitation-learning - Published: 2025-08-27 · Authors: Zeyi Sun, Yuhang Cao, Jianze Liang, Qiushi Sun, Ziyu Liu, Zhixiong Zhang, Yuhang Zang, Xiaoyi Dong, Kai Chen, Dahua Lin, Jiaqi Wang ### Code2World: A GUI World Model via Renderable Code Generation Code2World predicts future GUI screens through renderable code, trained on AndroidCode trajectories with supervised learning and render-aware reinforcement learning. It evaluates visual prediction and downstream AndroidWorld navigation with the learned world model. - https://awesomecua.com/resources/arxiv-2602-09856/ · source: https://arxiv.org/abs/2602.09856v1 - Paper · reviewed 2026-10-17 · environments: android · topics: world-models, data-generation, reinforcement-learning, mobile-navigation - Published: 2026-02-10 · Authors: Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin ### CogAgent: A Visual Language Model for GUI Agents Introduces a visual-language model with low- and high-resolution image encoders for screenshot understanding and GUI navigation. - https://awesomecua.com/resources/arxiv-2312-08914/ · source: https://arxiv.org/abs/2312.08914v3 - Paper · reviewed 2026-10-16 · environments: desktop, android, web · topics: screen-understanding, grounding - Published: 2023-12-14 · Authors: see the paper ### Cognitive Duality for Adaptive Web Agents Introduces CogniWeb, which switches between fast reactive processing and slower deliberative reasoning according to task complexity. WebArena experiments examine this combination of learned behaviors and online planning. - https://awesomecua.com/resources/arxiv-2508-05081/ · source: https://arxiv.org/abs/2508.05081v1 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, efficiency, web-navigation - Published: 2025-08-07 · Authors: Jiarun Liu, Chunhong Zhang, Zheng Hu ### COLA: A Scalable Multi-Agent Framework For Windows UI Task Automation Coordinates specialized Windows UI agents with dynamic task scheduling, shared memory, and human-triggered rollback for recovering from errors. - https://awesomecua.com/resources/arxiv-2503-09263/ · source: https://arxiv.org/abs/2503.09263v1 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: planning, memory, human-in-the-loop - Published: 2025-03-12 · Authors: Di Zhao, Longhui Ma, Siwei Wang, Miao Wang, Zhao Lv ### ColorAgent: Building A Robust, Personalized, and Interactive OS Agent Presents ColorAgent, an operating system agent for long-horizon user instructions, trained with step-wise RL and self-evolving training and evaluated on AndroidWorld and AndroidLab. - https://awesomecua.com/resources/arxiv-2510-19386/ · source: https://arxiv.org/abs/2510.19386v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: long-horizon, reinforcement-learning, human-in-the-loop - Published: 2025-10-22 · Authors: Ning Li, Qiqiang Lin, Zheng Wu, Xiaoyun Mo, Weiming Zhang, Yin Zhao, Xiangmou Qu, Jiamu Zhou, Jun Wang, Congmin Zheng, Yuanyi Song, Hongjiang Chen, Heyuan Huang, Jihong Wang, Jiaxin Yin, Jingwei Yu, Junwei Liao, Qiuying Peng, Xingyu Lou, Jun Wang, Weiwen Liu, Zhuosheng Zhang, Weinan Zhang ### ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Tasks ColorBench represents mobile tasks as graphs of states collected from real devices, providing a quasi-dynamic environment with multiple valid paths and error paths. It evaluates single-app and cross-app tasks using completion, subtask progress, and atomic diagnostics. - https://awesomecua.com/resources/arxiv-2510-14621/ · source: https://arxiv.org/abs/2510.14621v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, long-horizon, state-tracking - Published: 2025-10-16 · Authors: Yuanyi Song, Heyuan Huang, Qiqiang Lin, Yin Zhao, Xiangmou Qu, Jun Wang, Xingyu Lou, Weiwen Liu, Zhuosheng Zhang, Jun Wang, Yong Yu, Weinan Zhang, Zhaoxiang Wang ### ColorBrowserAgent: Complex Long-Horizon Browser Agent with Adaptive Knowledge Evolution ColorBrowserAgent converts human feedback into reusable website knowledge and uses that knowledge to guide progressive context summaries. It evaluates the resulting memory and adaptation mechanisms on long browser workflows. - https://awesomecua.com/resources/arxiv-2601-07262/ · source: https://arxiv.org/abs/2601.07262v3 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, human-in-the-loop, long-horizon, web-navigation - Published: 2026-01-12 · Authors: Jihong Wang, Jiamu Zhou, Weiming Zhang, Teng Wang, Weiwen Liu, Zhuosheng Zhang, Xingyu Lou, Weinan Zhang, Huarong Deng, Jun Wang ### ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm ComAct proposes COM program synthesis as an alternative interface for manipulating professional software and introduces ComCADBench with GUI-based comparison agents. ComActor adds a staged training and correction approach, supported by a Windows-container training platform. - https://awesomecua.com/resources/arxiv-2606-13239/ · source: https://arxiv.org/abs/2606.13239v2 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: automation, evaluation, tool-integration, science - Published: 2026-06-11 · Authors: Jiaxin Ai, Tao Hu, Xuemeng Yang, Shu Zou, Hairong Zhang, Daocheng Fu, Yu Yang, Hongbin Zhou, Nianchen Deng, Pinlong Cai, Zhongyuan Wang, Botian Shi, Kaipeng Zhang, Licheng Wen ### CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoning Introduces Channel-of-Mobile-Experts (CoME), a mobile agent with experts for screen summary, subtask planning, action decisions and action functions, trained in stages with InfoGain-Driven DPO (Info-DPO). - https://awesomecua.com/resources/arxiv-2602-24142/ · source: https://arxiv.org/abs/2602.24142v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, planning, action-prediction - Published: 2026-02-27 · Authors: Yuxuan Liu, Weikai Xu, Kun Huang, Changyu Chen, Jiankun Zhao, Pengzhi Gao, Wei Liu, Jian Luan, Shuo Shang, Bo Du, Ji-Rong Wen, Rui Yan ### Comparing Computer Use Behavior Across Multiple Levels of Abstraction MAP annotates computer-use traces at linked operation, function/consequence, method and goal levels, then compares human and agent workflows through sequence, pattern and effort metrics. - https://awesomecua.com/resources/arxiv-2610-26703/ · source: https://arxiv.org/abs/2610.26703v1 - Paper · reviewed 2026-10-20 · environments: desktop, web · topics: evaluation, observability - Published: 2026-10-19 · Authors: Ananya Gubbi Mohanbabu, Rosiana Natalie, Anhong Guo, Amy Pavel ### Comparing Human Oversight Strategies for Computer-Use Agents Compares four human oversight strategies during live web-agent sessions, examining exposure to problematic actions and opportunities to intervene. The analysis identifies timing and attention problems in control handoffs and upfront plan approval. - https://awesomecua.com/resources/arxiv-2604-04918/ · source: https://arxiv.org/abs/2604.04918v2 - Paper · reviewed 2026-10-17 · environments: web · topics: human-in-the-loop, safety-security, evaluation - Published: 2026-04-06 · Authors: Chaoran Chen, Zhiping Zhang, Zeya Chen, Eryue Xu, Yinuo Yang, Ibrahim Khalilov, Simret A Gebreegziabher, Yanfang Ye, Ziang Xiao, Yaxing Yao, Tianshi Li, Toby Jia-Jun Li ### ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents ComponentBench evaluates short interactions with canonical web UI components using verified tasks and matched human reference trajectories. Its diagnostic pipeline compares observation/action representations and analyzes structural difficulty, failure patterns and interaction efficiency. - https://awesomecua.com/resources/arxiv-2608-18307/ · source: https://arxiv.org/abs/2608.18307v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, screen-understanding, efficiency, robustness - Published: 2026-08-18 · Authors: Tianchen Guan, Xinlei Lin, Royce Cheng-Yue, Xiangjun Wang, Shuyan Zhou ### Compress to Focus: Efficient Coordinate Compression for Policy Optimization in Multi-Turn GUI Agents CCPO couples GUI policy optimization with coordinate-guided compression of past visual observations. It aggregates targets across rollouts to focus retained context and uses distance-based advantages to train grounding and compression together. - https://awesomecua.com/resources/arxiv-2601-11631/ · source: https://arxiv.org/abs/2601.11631v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, efficiency, interaction-history - Published: 2026-01-14 · Authors: Yurun Song, Jiong Yin, Rongjunchen Zhang, Ian G. Harris ### Computer Environments Elicit General Agentic Intelligence in LLMs LLM-in-Sandbox gives agents a minimal computer environment for resource access, file management and code execution; LLM-in-Sandbox-RL trains models to use it. - https://awesomecua.com/resources/arxiv-2601-16206/ · source: https://arxiv.org/abs/2601.16206v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: sandbox, tool-integration, reinforcement-learning, efficiency - Published: 2026-01-22 · Authors: Daixuan Cheng, Shaohan Huang, Yuxian Gu, Huatong Song, Guoxin Chen, Li Dong, Wayne Xin Zhao, Ji-Rong Wen, Furu Wei ### Computer Use at the Edge of the Statistical Precipice DigiWorld is a sandboxed mobile-app benchmark for computer-use agents, paired with PRISM environment principles and a confidence-interval aggregation method. - https://awesomecua.com/resources/arxiv-2605-08261/ · source: https://arxiv.org/abs/2605.08261v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, sandbox, mobile-navigation - Published: 2026-05-07 · Authors: Pierluca D'Oro, Sneha Silwal, William Wong, Yuxuan Sun, Fanyi Xiao, Manchen Wang, Eric Gan, Allen Bolourchi, Joseph Tighe ### Computer User Interface Understanding. A New Dataset and a Learning Framework Introduces desktop interaction videos and a framework combining synthetic examples with hierarchical contrastive learning for interface classification. - https://awesomecua.com/resources/arxiv-2403-10170/ · source: https://arxiv.org/abs/2403.10170v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: screen-understanding, video, data-generation - Published: 2024-03-15 · Authors: Andrés Muñoz, Daniel Borrajo ### Computer-Use Agents as Judges for Generative User Interface AUI-Gym evaluates generated websites by having computer-use agents attempt tasks whose executability is checked programmatically. A feedback dashboard condenses navigation histories into visual summaries that a coding agent can use to revise an interface. - https://awesomecua.com/resources/arxiv-2511-15567/ · source: https://arxiv.org/abs/2511.15567v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, ui-design, observability - Published: 2025-11-19 · Authors: Kevin Qinghong Lin, Siyuan Hu, Linjie Li, Zhengyuan Yang, Lijuan Wang, Philip Torr, Mike Zheng Shou ### Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials RAG-PolicyJudge selects computer-use actions proposed with task-specific tutorials; the study compares its decisions with and without world-model rollouts. - https://awesomecua.com/resources/arxiv-2611-04073/ · source: https://arxiv.org/abs/2611.04073v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: world-models, evaluation, planning, science - Published: 2026-11-02 · Authors: Anh Nguyen, Tianyi Xu, Phat Tran, Harry Yu, Huazheng Wang, Stefan Lee ### Computer-Using World Model Presents a desktop world model that predicts textual UI changes before synthesizing the next screenshot. It learns from Microsoft Office transitions and is evaluated by letting a frozen agent compare simulated actions before execution. - https://awesomecua.com/resources/arxiv-2602-17365/ · source: https://arxiv.org/abs/2602.17365v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: world-models, planning, action-prediction - Published: 2026-02-19 · Authors: Yiming Guan, Rui Yu, John Zhang, Lu Wang, Chaoyun Zhang, Liqun Li, Bo Qiao, Si Qin, He Huang, Fangkai Yang, Pu Zhao, Lukas Wutschitz, Samuel Kessler, Huseyin A Inan, Robert Sim, Saravan Rajmohan, Qingwei Lin, Dongmei Zhang ### ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents Combines API and GUI actions with distributed desktop reinforcement learning and alternating supervised training to address training instability. - https://awesomecua.com/resources/arxiv-2508-14040/ · source: https://arxiv.org/abs/2508.14040v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: reinforcement-learning, tool-integration - Published: 2025-08-19 · Authors: Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, Jie Tang ### ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents Introduces ComputerSD, which turns feedback from executed GUI transitions into online self-distillation guidance and step-level scores. A GUI analyzer supplies these signals, which are combined with trajectory-level reinforcement learning in asynchronous training. - https://awesomecua.com/resources/arxiv-2609-40253/ · source: https://arxiv.org/abs/2609.40253v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: reinforcement-learning, evaluation - Published: 2026-09-30 · Authors: Yong Du, Tongbo Chen, Zhengxi Lu, Yizhou Liu, Bofan Chen, Tao Jiang, Wenhao Xu, Yongliang Shen ### ConClick: Element-Wise Contrastive Learning Improves GUI Grounding Analyzes recurring distractor clicks in GUI grounding and introduces ConClick, a contrastive training method built from target–distractor pairs. Its ReClick inference procedure revisits candidate regions and compares competing predictions. - https://awesomecua.com/resources/arxiv-2610-13268/ · source: https://arxiv.org/abs/2610.13268v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, data-generation, robustness - Published: 2026-10-09 · Authors: Ruotian Zhang, Yuhui Zhang, Junzhe Huang, Yu Xin, Hehe Fan, Xiaoxiao Sun ### Constraining Host-Level Abuse in Self-Hosted Computer-Use Agents via TEE-Backed Isolation A TEE-backed design confines host operations by self-hosted computer-use agents, classifying and authorizing actions before constrained local execution. - https://awesomecua.com/resources/arxiv-2605-06393/ · source: https://arxiv.org/abs/2605.06393v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, sandbox, tool-integration - Published: 2026-05-07 · Authors: Di Lu, Bo Zhang, Xiyuan Li, Yongzhi Liao, Xuewen Dong, Yulong Shen, Zhiquan Liu, Jianfeng Ma ### Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions BreakingWeb pairs solvable browser tasks with controlled environment interventions that preserve the goal, target and backend success criterion. The paired evaluations isolate robustness failures, including agents declaring completion without the required state change. - https://awesomecua.com/resources/arxiv-2609-35814/ · source: https://arxiv.org/abs/2609.35814v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, robustness, data-generation, state-tracking - Published: 2026-09-20 · Authors: Xunjian Yin, Tianchen Guan, Jinao Wang, Weili Cao, Daisy Xinlei Lin, Royce Cheng-Yue, Keagan Long, Kyle Wong, Bhuwan Dhingra, Xiangjun Wang, Shuyan Zhou ### Context manipulation attacks : Web agents are susceptible to corrupted memory Studies context manipulation in web agents by altering stored task representations rather than webpage prompts alone. Evaluation on Browser-use and Agent-E examines memory-directed attacks and the implications for secure context handling. - https://awesomecua.com/resources/arxiv-2506-17318/ · source: https://arxiv.org/abs/2506.17318v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, memory, evaluation - Published: 2025-06-18 · Authors: Atharv Singh Patlan, Ashwin Hebbar, Pramod Viswanath, Prateek Mittal ### Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models Studies staged prompting for automated mobile UI element annotation, comparing workflow depths and element groupings on expert-annotated MUIAnno screens. - https://awesomecua.com/resources/arxiv-2606-02208/ · source: https://arxiv.org/abs/2606.02208v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, screen-understanding, workflow, evaluation - Published: 2026-06-01 · Authors: Athar Parvez, Muhammad Jawad Mufti, Muqaddas Gull, Omar Hammad ### Continual GUI Agents Continual GUI Agents defines grounding under changing application domains and screen resolutions. Its GUI-AiF training framework uses separate point- and region-alignment rewards to adapt grounding as these distributions shift. - https://awesomecua.com/resources/arxiv-2601-20732/ · source: https://arxiv.org/abs/2601.20732v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, robustness, evaluation - Published: 2026-01-28 · Authors: Ziwei Liu, Borui Kang, Hangjie Yuan, Zixiang Zhao, Wei Li, Yifan Zhu, Tao Feng ### ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agents Turns draft web-agent skills into structured executable artifacts that support deterministic checks, fault localization and local repair. VisualWebArena and MiniWoB experiments examine execution and reuse within the same benchmark family. - https://awesomecua.com/resources/arxiv-2603-20340/ · source: https://arxiv.org/abs/2603.20340v3 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, planning, robustness - Published: 2026-03-20 · Authors: Zijian Lu, Yiping Zuo, Yupeng Nie, Xin He, Weibei Fan, Lianyong Qi, Shi Jin ### Control OSWorld: An AI Control Environment for GUI Computer Use Agents Pairs OSWorld tasks with harmful side objectives to evaluate monitors that detect malicious behavior in computer-use agent trajectories. - https://awesomecua.com/resources/arxiv-2610-03818/ · source: https://arxiv.org/abs/2610.03818v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: safety-security, evaluation - Published: 2026-10-02 · Authors: Maria Sanchez del Rio, Mikhail Terekhov ### CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation Introduces CORA, which calibrates a risk model to choose between executing and abstaining from mobile GUI actions, then diagnoses rejected actions to suggest interventions. Phone-Harm provides step-level safety labels for evaluating this safeguarding process. - https://awesomecua.com/resources/arxiv-2604-09155/ · source: https://arxiv.org/abs/2604.09155v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, mobile-navigation, evaluation - Published: 2026-04-10 · Authors: Yushi Feng, Junye Du, Qifan Wang, Zizhan Ma, Qian Niu, Yutaka Matsuo, Long Feng, Lequan Yu ### CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs Introduces CORE, which divides mobile UI hierarchies into semantic blocks and shares planning between local and cloud language models. Local ranking limits the screen information sent for cloud-side element selection, with a multi-round mechanism for correcting inadequate context. - https://awesomecua.com/resources/arxiv-2510-15455/ · source: https://arxiv.org/abs/2510.15455v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, mobile-navigation, efficiency - Published: 2025-10-17 · Authors: Gucongcong Fan, Chaoyue Niu, Chengfei Lyu, Fan Wu, Guihai Chen ### CORNET: Learning Table Formatting Rules By Example Introduces CORNET, a system that learns spreadsheet conditional-formatting rules from cells formatted by users, using symbolic rule enumeration and a neural ranker. - https://awesomecua.com/resources/arxiv-2208-06032/ · source: https://arxiv.org/abs/2208.06032v4 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, imitation-learning - Published: 2022-08-11 · Authors: Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, Mohammad Raza, Gust Verbruggen ### Correct Answers, Closed Gates: SolveBind for GUI Agents at the Verification Barrier Introduces SolveBind, a GUI-agent framework that decomposes verification tasks and uses runtime evidence to guide interactions with protected websites and applications. - https://awesomecua.com/resources/arxiv-2610-32223/ · source: https://arxiv.org/abs/2610.32223v1 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, grounding, state-tracking, safety-security - Published: 2026-10-23 · Authors: Xinhao Song, Haoxuan Peng, Su Su, Hongliang Wu, Sirui Song, Gongshen Liu, Linfeng Zhang, Dongrui Liu ### cotomi Act: Learning to Automate Work by Watching You cotomi Act is a browser agent that executes multi-step tasks and turns observed user browsing into persistent knowledge shared with users. - https://awesomecua.com/resources/arxiv-2605-03231/ · source: https://arxiv.org/abs/2605.03231v1 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, memory, recording, long-horizon - Published: 2026-05-04 · Authors: Masafumi Oyamada, Kunihiro Takeoka, Kosuke Akimoto, Ryoma Obara, Masafumi Enomoto, Haochen Zhang, Daichi Haraguchi, Takuya Tamura ### CovAgent: Overcoming the 30% Curse of Mobile Application Coverage with Agentic AI and Dynamic Instrumentation CovAgent augments Android GUI fuzzers by analyzing decompiled app code and component transitions to identify unsatisfied activation conditions. A second agent generates instrumentation that enables otherwise difficult-to-reach activities, with coverage-based evaluation. - https://awesomecua.com/resources/arxiv-2601-21253/ · source: https://arxiv.org/abs/2601.21253v1 - Paper · reviewed 2026-10-17 · environments: android · topics: automation, mobile-navigation, evaluation - Published: 2026-01-29 · Authors: Wei Minn, Biniam Fisseha Demissie, Yan Naing Tun, Jiakun Liu, Mariano Ceccato, Lwin Khin Shar, David Lo ### Covering Human Action Space for Computer Use: Data Synthesis and Benchmark Introduces CUActSpot to evaluate clicks, drags, drawing and other screen interactions across GUI, text, tables, canvases and natural images. A renderer-based pipeline records geometry and screenshots, then generates matching instructions and action traces for training a grounding model. - https://awesomecua.com/resources/arxiv-2605-12501/ · source: https://arxiv.org/abs/2605.12501v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: grounding, data-generation, evaluation, action-prediction - Published: 2026-05-12 · Authors: Miaosen Zhang, Xiaohan Zhao, Zhihong Tan, Zhou Huoshen, Yijia Fan, Yifan Yang, Kai Qiu, Bei Liu, Justin Wagle, Chenzhong Yin, Mingxi Cheng, Ji Li, Qi Dai, Chong Luo, Xu Yang, Xin Geng, Baining Guo ### CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation Introduces CowPilot, a web-navigation framework where agents suggest actions and users can pause, reject, override, or resume them. It evaluates autonomous and collaborative use through task success and human effort. - https://awesomecua.com/resources/arxiv-2501-16609/ · source: https://arxiv.org/abs/2501.16609v4 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, human-in-the-loop, evaluation - Published: 2025-01-28 · Authors: Faria Huq, Zora Zhiruo Wang, Frank F. Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P. Bigham, Graham Neubig ### CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents Introduces CRAB, a framework for cross-environment agent tasks with graph-based evaluation and a benchmark spanning desktop and phone interactions. - https://awesomecua.com/resources/arxiv-2407-01511/ · source: https://arxiv.org/abs/2407.01511v4 - Paper · reviewed 2026-10-16 · environments: desktop, cross-platform · topics: evaluation, cross-app - Published: 2024-07-01 · Authors: Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin, Jianbo Deng, Philip Torr, Bernard Ghanem, Guohao Li ### Cradle: Empowering Foundation Agents Towards General Computer Control Uses screenshot observations, planning modules, and generated low-level keyboard and mouse control for general computer interaction. - https://awesomecua.com/resources/arxiv-2403-03186/ · source: https://arxiv.org/abs/2403.03186v3 - Paper · reviewed 2026-10-16 · environments: desktop · topics: planning, screen-understanding - Published: 2024-03-05 · Authors: Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, Zongqing Lu ### CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks CRAFT-GUI organizes GUI trajectories by difficulty for curriculum-based GRPO training. Its reward combines rule-based signals with model judgments, with evaluation on Android Control and an internal online benchmark. - https://awesomecua.com/resources/arxiv-2508-11360/ · source: https://arxiv.org/abs/2508.11360v2 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, reward-models, mobile-navigation - Published: 2025-08-15 · Authors: Songqin Nong, Xiaoxuan Tang, Jingxuan Xu, Sheng Zhou, Jianfeng Chen, Tao Jiang, Wenhao Xu ### CrashTranslator: Automatically Reproducing Mobile Application Crashes Directly from Stack Trace Introduces CrashTranslator, which predicts GUI exploration steps from stack traces with an LLM and uses RL-guided search to reproduce crashes in Android apps. - https://awesomecua.com/resources/arxiv-2310-07128/ · source: https://arxiv.org/abs/2310.07128v1 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, reinforcement-learning, planning, automation - Published: 2023-10-11 · Authors: Yuchao Huang, Junjie Wang, Zhe Liu, Yawen Wang, Song Wang, Chunyang Chen, Yuanzhe Hu, Qing Wang ### Creating General User Models from Computer Use Proposes general user models that turn computer-use observations into confidence-weighted propositions and revise them as new context arrives. Demonstrations use this memory to contextualize assistants, manage notifications, and support proactive actions across apps. - https://awesomecua.com/resources/arxiv-2505-10831/ · source: https://arxiv.org/abs/2505.10831v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, human-in-the-loop, cross-app - Published: 2025-05-16 · Authors: Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, Michael S. Bernstein ### Cross-Interface Authorization for AI Computer-Use Agents Presents a profile-bounded authorization protocol for computer-use agents working through GUIs, APIs, scripts, and other routes, with route-bound tickets and commit-time checks. - https://awesomecua.com/resources/arxiv-2610-41397/ · source: https://arxiv.org/abs/2610.41397v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, tool-integration - Published: 2026-10-30 · Authors: Genliang Zhu, Chu Wang ### Cross-Modal Content Optimization for Steering Web Agent Preferences Introduces Cross-Modal Preference Steering (CPS), a black-box attack that edits a listing's images and text to steer VLM-based web agents' selections. - https://awesomecua.com/resources/arxiv-2510-03612/ · source: https://arxiv.org/abs/2510.03612v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation - Published: 2025-10-04 · Authors: Tanqiu Jiang, Min Bai, Nikolaos Pappas, Yanjun Qi, Sandesh Swamy ### CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents Presents CUA-Gym, which co-generates instructions, initial and target states, and executable reward checks through generator-discriminator iterations. Mock web applications expand training environments, with learned agents evaluated on OSWorld-Verified and WebArena. - https://awesomecua.com/resources/arxiv-2605-25624/ · source: https://arxiv.org/abs/2605.25624v2 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: data-generation, reinforcement-learning, sandbox - Published: 2026-05-25 · Authors: Bowen Wang, Dunjie Lu, Junli Wang, Tianyi Bai, Shixuan Liu, Zhipeng Zhang, Haiquan Wang, Hao Hu, Tianbao Xie, Shuai Bai, Dayiheng Liu, Que Shen, Junyang Lin, Tao Yu ### CUA-Lite: An Open Platform for Computer Use Agents CUA-Lite unifies environment interfaces, supervised data formats and per-model harnesses for evaluation, reinforcement learning and supervised fine-tuning across desktop, browser and mobile tasks. - https://awesomecua.com/resources/arxiv-2610-26071/ · source: https://arxiv.org/abs/2610.26071v1 - Paper · reviewed 2026-10-20 · environments: desktop, linux, windows, web, android · topics: tool-integration, data-generation, evaluation, imitation-learning, reinforcement-learning, sandbox - Published: 2026-10-19 · Authors: Zhanhui Zhou, Weichen Zhang, Haoran Liu, Lingjie Chen, Tianneng Shi, Kevin Lin, Zhengyuan Yang, Lijuan Wang, Dawn Song ### CUA-Master: Scaling Computer-use Agents through Open-World Recursive Improvements Introduces CUA-Master, a computer-use training framework that mines workflows from instructional videos and turns them into synthesized tasks and trajectories. - https://awesomecua.com/resources/arxiv-2611-03126/ · source: https://arxiv.org/abs/2611.03126v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, video, workflow - Published: 2026-11-02 · Authors: Xinlei Yu, Gen Li, Ji Zhao, Zhangquan Chen, Congcong Wang, Shuai Dong, Zelin Peng, Yuqi Xu, Hongrui Wu, Kaiwen Tuo, Cheng Yang, Kaituo Feng, Yifan Wu, Jiayi Zhang, Dongming Wu, Wenqi Shao, Jiaqi Wang, Xiangyu Yue ### CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents CUA-Recovery benchmarks error detection and recovery across desktop applications, using agent failures to create test states and training trajectories. - https://awesomecua.com/resources/arxiv-2610-27645/ · source: https://arxiv.org/abs/2610.27645v1 - Paper · reviewed 2026-11-06 · environments: desktop · topics: evaluation, robustness, long-horizon, data-generation - Published: 2026-10-20 · Authors: Xu Hu, Haoming Li, Yuxiang Wang, Jinxin Yang, Kangshuo Li, Licheng Wang, Dingyi Kang, Jiayu Song, Tianhao Wang, Zhanhua Hu, Xu Yang, Qiannan Li, Bingzhe Li, Feng Chen ### CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning Introduces CUA-Sandbox, which separates each rollout’s mutable state from shared initialized software runtimes. State-scoped execution and transactional resets or branches aim to reduce training-environment costs while retaining original interfaces and evaluators. - https://awesomecua.com/resources/arxiv-2609-32750/ · source: https://arxiv.org/abs/2609.32750v1 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: sandbox, reinforcement-learning, efficiency - Published: 2026-09-26 · Authors: Xin Yan, Zhengbo Jiao, Jiaqi Liu, Zhenglin Wan, SiYuan Ma, Xuliang Yu, Tianyi Jiang, Chubin Zhang, Pengfei Zhou, Wangbo Zhao, Xingrui Yu, Bo An, Yang You, Ivor Tsang ### CUA-Skill: Develop Skills for Computer Using Agent CUA-Skill encodes Windows application knowledge in parameterized skills and execution/composition graphs. Its agent retrieves and instantiates these skills and uses memory for failure recovery, with end-to-end evaluation on WindowsAgentArena. - https://awesomecua.com/resources/arxiv-2601-21123/ · source: https://arxiv.org/abs/2601.21123v2 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: tool-integration, planning, memory, automation - Published: 2026-01-28 · Authors: Tianyi Chen, Yinheng Li, Michael Solodko, Sen Wang, Nan Jiang, Tingyuan Cui, Junheng Hao, Jongwoo Ko, Sara Abdali, Leon Xu, Suzhen Zheng, Hao Fan, Pashmina Cameron, Justin Wagle, Kazuhito Koishida ### cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents Standardizes virtual machines, interaction pipelines, and agent interfaces to compare computer-use speed, cost, and task performance. - https://awesomecua.com/resources/arxiv-2609-40284/ · source: https://arxiv.org/abs/2609.40284v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: evaluation, efficiency - Published: 2026-09-30 · Authors: Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, Jing Yu Koh ### CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents CUA-Suite combines continuous expert desktop demonstrations in VideoCUA with the UI-Vision evaluation benchmark and GroundCUA grounding data. Screen recordings, cursor traces and layered annotations support study of temporal behavior and professional software workflows. - https://awesomecua.com/resources/arxiv-2603-24440/ · source: https://arxiv.org/abs/2603.24440v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: data-generation, video, evaluation, grounding, workflow - Published: 2026-03-25 · Authors: Xiangru Jian, Shravan Nayak, Kevin Qinghong Lin, Aarash Feizi, Kaixin Li, Patrice Bechard, Spandana Gella, Sai Rajeswar ### CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering Introduces CUA-SWE, a benchmark and execution environment that combines code changes with graphical interaction and visual inspection of running software. Task-specific tests verify repairs and changes whose required information may be available only through the application interface. - https://awesomecua.com/resources/arxiv-2609-32600/ · source: https://arxiv.org/abs/2609.32600v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, workflow, tool-integration - Published: 2026-09-26 · Authors: Prince Zizhuang Wang, Chenhao Liang, Zelong Xu, Aojie Yuan, Xiaolin Zhou, Haiyue Zhang, Yue Zhao, Xiyang Hu, Shuli Jiang ### CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents Turns desktop applications into reproducible GUI-and-CLI environments and generates verified hybrid trajectories for agent post-training. - https://awesomecua.com/resources/arxiv-2609-05374/ · source: https://arxiv.org/abs/2609.05374v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: tool-integration, data-generation, sandbox - Published: 2026-09-04 · Authors: Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang, Tian Jin, Pengxiang Zhao, Guangyi Liu, Siheng Chen, Yanfeng Wang ### CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents CUAAudit examines vision-language models that judge computer-use task completion from an instruction and final environment state. Across macOS, Windows and Linux benchmarks, it measures accuracy, confidence calibration and disagreement between auditors. - https://awesomecua.com/resources/arxiv-2603-10577/ · source: https://arxiv.org/abs/2603.10577v2 - Paper · reviewed 2026-10-17 · environments: desktop, macos, windows, linux · topics: evaluation, observability, robustness - Published: 2026-03-11 · Authors: Marta Sumyk, Oleksandr Kosovan ### CUADebug: Diagnosing and Repairing Computer-Use Agent Failures CUADebug combines a failure taxonomy and human-annotated OSWorld failure trajectories with an agent that locates causal steps and proposes corrections from screenshots and action traces. It evaluates root-cause diagnosis separately from controlled task re-execution and failure recovery. - https://awesomecua.com/resources/arxiv-2608-02643/ · source: https://arxiv.org/abs/2608.02643v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, robustness, self-reflection, state-tracking - Published: 2026-07-31 · Authors: Weijia Zhang, Kunlun Zhu, Zeyi Liu, Yinting Chen, Tianyi Ma, Jiateng Liu, Jiaxun Zhang, Bingxuan Li, Xiangru Tang, Heng Ji, Jiaxuan You ### CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent Introduces CUARewardBench for evaluating outcome and process reward models on annotated computer-use trajectories, plus Unanimous Prompt Ensemble (UPE) for reward judgments. - https://awesomecua.com/resources/arxiv-2510-18596/ · source: https://arxiv.org/abs/2510.18596v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, reward-models - Published: 2025-10-21 · Authors: Haojia Lin, Xiaoyu Tan, Yulei Qin, Zihan Xu, Yuchen Shi, Zongyi Li, Gang Li, Shaofei Cai, Siqi Cai, Chaoyou Fu, Ke Li, Xing Sun ### CUAWright: A Minimal Unified Interface for Digital Agents Studies a minimal terminal-based execution framework that lets digital agents construct tools and manage context across browser, desktop, and CAD tasks. - https://awesomecua.com/resources/arxiv-2610-04116/ · source: https://arxiv.org/abs/2610.04116v1 - Paper · reviewed 2026-10-16 · environments: desktop, web · topics: tool-integration, efficiency - Published: 2026-10-02 · Authors: Yadong Lu, Theodore Lee, Yifei Li, Lawrence Keunho Jang, Tianci Xue, Yu Su, Huan Sun, Ahmed Hassan Awadallah ### CURA: Certified Runtime Alarms for Computer-Use Agents Introduces CURA, a telemetry-based runtime monitor that turns computer-use trajectories into sequential failure alarms and can trigger mid-task oversight. Its certificate controls false alarms, while the study separately reports detection performance and limits of retrospective monitoring. - https://awesomecua.com/resources/arxiv-2608-27808/ · source: https://arxiv.org/abs/2608.27808v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: observability, safety-security, evaluation - Published: 2026-08-28 · Authors: Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi ### Curiosity Driven Knowledge Retrieval for Mobile Agents Uses execution uncertainty to trigger retrieval from application documentation, code, and past trajectories, organizing the results into structured AppCards for Android agent planning. - https://awesomecua.com/resources/arxiv-2601-19306/ · source: https://arxiv.org/abs/2601.19306v1 - Paper · reviewed 2026-10-18 · environments: android · topics: memory, planning, mobile-navigation - Published: 2026-01-27 · Authors: Sijia Li, Xiaoyu Tan, Shahir Ali, Niels Schmidt, Gengchen Ma, Xihe Qiu ### CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing Cutverse benchmarks GUI agents on long editing workflows across professional media applications and parses recordings and input logs into grounded action trajectories. - https://awesomecua.com/resources/arxiv-2605-19484/ · source: https://arxiv.org/abs/2605.19484v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon, recording, grounding - Published: 2026-05-19 · Authors: Haobo Hu, Xiangwu Guo, Zhiheng Chen, Difei Gao, Haotian Liu, Libiao Jin, Qi Mao ### D-Artemis: A Deliberative Cognitive Framework for Mobile GUI Multi-Agents Introduces D-Artemis, a mobile GUI agent framework that retrieves app-specific tips, checks actions before execution, and reflects on their outcomes without trajectory training. - https://awesomecua.com/resources/arxiv-2509-21799/ · source: https://arxiv.org/abs/2509.21799v3 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, memory, self-reflection, training-free - Published: 2025-09-26 · Authors: Hongze Mi, Yibo Feng, Wenjie Lu, Yuqi Wang, Jinyuan Li, Song Cao, He Cui, Tengfei Tian, Xuelin Zhang, Haotian Luo, Di Sun, Jun Fang, Hua Chai, Naiqiang Tan, Gang Pan ### D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies D-GARA evaluates Android GUI agents under runtime anomalies such as permission dialogs, battery warnings, and update prompts. Its modular environment embeds these interruptions in Android applications to study recovery and robustness. - https://awesomecua.com/resources/arxiv-2511-16590/ · source: https://arxiv.org/abs/2511.16590v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, robustness, mobile-navigation - Published: 2025-11-20 · Authors: Sen Chen, Tong Zhao, Yi Bin, Fei Ma, Wenqi Shao, Zheng Wang ### Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight Studies how GUI agents, people, and human-agent teams respond to manipulative interface designs. The two-phase evaluation examines failure modes and the benefits and costs of human oversight. - https://awesomecua.com/resources/arxiv-2509-10723/ · source: https://arxiv.org/abs/2509.10723v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, safety-security, evaluation - Published: 2025-09-12 · Authors: Jingyu Tang, Chaoran Chen, Jiawen Li, Zhiping Zhang, Bingcan Guo, Ibrahim Khalilov, Simret Araya Gebreegziabher, Bingsheng Yao, Dakuo Wang, Yanfang Ye, Tianshi Li, Ziang Xiao, Yaxing Yao, Toby Jia-Jun Li ### Darwin Mobile Agent: A Roadmap for Self-Evolution Describes asynchronous reinforcement-learning infrastructure using parallel cloud-hosted Android instances, explicit task lifecycles, and persistent-state management for mobile GUI interaction. - https://awesomecua.com/resources/arxiv-2606-20622/ · source: https://arxiv.org/abs/2606.20622v1 - Paper · reviewed 2026-10-18 · environments: android · topics: reinforcement-learning, mobile-navigation, evaluation - Published: 2026-05-26 · Authors: Daniel Beechey, Derek Yuen, Jianheng Liu, Dezhao Luo, Tiantian He, Weilin Luo, Jun Wang, Kun Shao ### Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution Darwinian Memory splits GUI trajectories into reusable units and updates the memory by retaining useful paths and pruning poor ones. It evaluates this training-free memory mechanism on long multi-application tasks. - https://awesomecua.com/resources/arxiv-2601-22528/ · source: https://arxiv.org/abs/2601.22528v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, training-free, cross-app, long-horizon - Published: 2026-01-30 · Authors: Hongze Mi, Yibo Feng, WenJie Lu, Song Cao, Jinyuan Li, Yanming Li, Xuelin Zhang, Haotian Luo, Songyang Peng, He Cui, Tengfei Tian, Jun Fang, Hua Chai, Naiqiang Tan ### DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis Diagnoses dashboard-agent failures by progressively restoring interaction context, next-action support, and visual target information within the same task. - https://awesomecua.com/resources/arxiv-2609-32385/ · source: https://arxiv.org/abs/2609.32385v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, grounding, planning - Published: 2026-09-26 · Authors: Chuhan Zhang, Qi Xie, Ziyue Wang, Jianing Yin, Yunfan Zhou, Dazhen Deng, Yingcai Wu ### DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards Introduces DashboardQA to evaluate how vision-language GUI agents answer questions by interacting with dashboards, grounding elements, and planning actions. - https://awesomecua.com/resources/arxiv-2508-17398/ · source: https://arxiv.org/abs/2508.17398v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, grounding, planning, screen-understanding - Published: 2025-08-24 · Authors: Aaryaman Kartha, Ahmed Masry, Mohammed Saidul Islam, Thinh Lang, Shadikur Rahman, Ridwan Mahbub, Mizanur Rahman, Mahir Ahmed, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty ### DAVE: A VLM Vision Encoder for Document Understanding and Web Agents DAVE trains a vision encoder for document images and web agents using self-supervised pretraining followed by supervised parsing and localization. It combines decoder-specific encoder merging with feature fusion and evaluates web localization and agent benchmarks alongside document tasks. - https://awesomecua.com/resources/arxiv-2512-17221/ · source: https://arxiv.org/abs/2512.17221v3 - Paper · reviewed 2026-10-17 · environments: web · topics: screen-understanding, grounding, evaluation - Published: 2025-12-19 · Authors: Brandon Huang, Hang Hua, Zhuoran Yu, Trevor Darrell, Rogerio Feris, Roei Herzig ### DECEPTICON: How Dark Patterns Manipulate Web Agents DECEPTICON is an environment of 700 web navigation tasks that isolates dark patterns to measure how deceptive interface designs steer agent actions. - https://awesomecua.com/resources/arxiv-2512-22894/ · source: https://arxiv.org/abs/2512.22894v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, ui-design, safety-security, web-navigation - Published: 2025-12-28 · Authors: Phil Cuvin, Hao Zhu, Diyi Yang ### Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type? Compares ways to condition a small GUI grounding model on action type under matched training conditions on Android in the Wild. The study traces apparent gains to a coordinate-serialization choice and reports limited evidence that the gains persist with predicted action types. - https://awesomecua.com/resources/arxiv-2610-07444/ · source: https://arxiv.org/abs/2610.07444v1 - Paper · reviewed 2026-10-17 · environments: android · topics: grounding, evaluation, robustness - Published: 2026-10-05 · Authors: Aadi Chauhan, Arthur Ilyasov ### Deep Research Bench: Evaluating AI Web Research Agents Introduces Deep Research Bench, 89 multi-step web research tasks and a frozen collection of scraped pages for evaluating search agents and their traces over time. - https://awesomecua.com/resources/arxiv-2506-06287/ · source: https://arxiv.org/abs/2506.06287v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, long-horizon, observability - Published: 2025-05-06 · Authors: FutureSearch, :, Nikos I. Bosse, Jon Evans, Robert G. Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, Jack Wildman ### Deep Search in Real-World Web Environments OpenDS-World benchmarks multi-step research on the live open Web, requiring agents to search, interact with browsers and combine evidence across sources. - https://awesomecua.com/resources/arxiv-2610-30181/ · source: https://arxiv.org/abs/2610.30181v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, long-horizon, tool-integration - Published: 2026-10-19 · Authors: Xiaomin Yu, Hongliang Chen, Zhe Huang, Shilin Yan, Shao-Rong Su, Xuqiao Ma, Feiyang Wang, Jiayu Chen, Jiwei Fang, Yu Yan, Yini Huang, Chen Chen, Ruoxi Cheng, Yuzhi He, Yuhui Zhang, Xiaoxing Hu, Chen Liu, Jianbo Lin, Yijiang Li, Xiaobin Hu, Menglong Li, Zhuoran Lu, Weishi Wang, Zhongqi Yue, Chengwei Qin ### DeepShop: A Benchmark for Deep Research Shopping Agents DeepShop evaluates shopping agents with evolving query complexity across product attributes, filters and sorting preferences. Its evaluation combines checks of these individual requirements with overall task success. - https://awesomecua.com/resources/arxiv-2506-02839/ · source: https://arxiv.org/abs/2506.02839v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, web-navigation - Published: 2025-06-03 · Authors: Yougang Lyu, Xiaoyu Zhang, Lingyong Yan, Maarten de Rijke, Zhaochun Ren, Xiuying Chen ### DeltaReplay: Task-Relative Memory Reuse for Mobile GUI Agents Introduces DeltaReplay, which stores mobile GUI trajectories as transition graphs and decides at each step whether to reuse an action, substitute task-specific parameters, or defer to the base agent. This supports reuse of partially matching experiences without rewriting stored memory. - https://awesomecua.com/resources/arxiv-2610-11707/ · source: https://arxiv.org/abs/2610.11707v1 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, mobile-navigation, planning - Published: 2026-10-08 · Authors: Yudong Bai, Yihong Chen, Quanming Yao, Yaqing Wang ### Demo2Tutorial: From Human Experience to Multimodal Software Tutorials Introduces Demo2Tutorial, which turns recorded software interactions into multimodal tutorials with task graphs and reports benefits for GUI-agent planning. - https://awesomecua.com/resources/arxiv-2606-03951/ · source: https://arxiv.org/abs/2606.03951v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: recording, video, planning, workflow - Published: 2026-06-02 · Authors: Zechen Bai, Zhiheng Chen, Yiqi Lin, Kevin Qinghong Lin, Difei Gao, Xiangwu Guo, Xin Wang, Mike Zheng Shou ### Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics Introduces WebDetective, a deep-search benchmark with a traceable Wikipedia sandbox, and EvidenceLoop, an evidence-tracking workflow for web-agent search and synthesis. - https://awesomecua.com/resources/arxiv-2510-05137/ · source: https://arxiv.org/abs/2510.05137v3 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, observability, planning - Published: 2025-10-01 · Authors: Maojia Song, Renhang Liu, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou, Dorien Herremans, Soujanya Poria ### Designing Agent-Ready Websites for AI Web Agents: A Framework for Machine Readability, Actionability, and Decision Reliability Proposes an agent-ready website design framework for e-commerce browser agents and compares it with a human-oriented site on shopping tasks, outcomes, steps, and token use. - https://awesomecua.com/resources/arxiv-2607-12056/ · source: https://arxiv.org/abs/2607.12056v1 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, web-navigation, evaluation - Published: 2026-07-13 · Authors: Said Elnaffar, Farzad Rashidi ### DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration DeskCraft evaluates professional desktop workflows and human-agent collaboration through mid-execution clarification, interruption and post-completion feedback. Its tasks cover creative applications and expose difficulties in long-horizon execution and proactive clarification. - https://awesomecua.com/resources/arxiv-2606-03103/ · source: https://arxiv.org/abs/2606.03103v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, human-in-the-loop, long-horizon, workflow - Published: 2026-06-02 · Authors: Wenkai Wang, Tao Xiong, Jingchen Ni, Yunpeng Bao, Xiyun Li, Tianqi Liu, Hongcan Guo, Zilong Huang, Shengyu Zhang ### DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents Generates dense desktop supervision by varying real application scenes and combining screenshots, accessibility trees, and window geometry. - https://awesomecua.com/resources/arxiv-2610-02320/ · source: https://arxiv.org/abs/2610.02320v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: data-generation, grounding - Published: 2026-10-01 · Authors: A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong, Marc Pollefeys, Peter W. J. Staar ### Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions? Introduces Desktop-Delta Bench, an offline diagnostic of GUI transition understanding using temporal-ordering tasks and before/after action inference. It includes decoy observations to test whether models track the source and task relevance of screen changes. - https://awesomecua.com/resources/arxiv-2607-26041/ · source: https://arxiv.org/abs/2607.26041v2 - Paper · reviewed 2026-10-17 · environments: desktop, linux · topics: evaluation, screen-understanding, state-tracking - Published: 2026-07-28 · Authors: Abhishek Pillai, Samir Kumar Nayak, Yuan Chen ### DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents Introduces AutoCaptioner, DeskVision, and GUIExplorer for generating desktop region descriptions and training interface-understanding models. - https://awesomecua.com/resources/arxiv-2503-11170/ · source: https://arxiv.org/abs/2503.11170v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: data-generation, grounding, screen-understanding - Published: 2025-03-14 · Authors: Yibin Xu, Liang Yang, Hao Chen, Hua Wang, Zhi Chen, Yaohua Tang ### Detect Anything via Next Point Prediction Introduces Rex-Omni, a general visual perception model trained for coordinate prediction; GUI grounding is one capability evaluated alongside detection and OCR. - https://awesomecua.com/resources/arxiv-2510-12798/ · source: https://arxiv.org/abs/2510.12798v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, evaluation, data-generation - Published: 2025-10-14 · Authors: Qing Jiang, Junan Huo, Xingyu Chen, Yuda Xiong, Zhaoyang Zeng, Yihao Chen, Tianhe Ren, Junzhi Yu, Lei Zhang ### Detecting Pipeline Failures through Fine-Grained Analysis of Web Agents Introduces a modular framework for identifying errors at intermediate stages of a web-agent pipeline. A SeeAct and Mind2Web case study examines weaknesses that are obscured by final task-success metrics. - https://awesomecua.com/resources/arxiv-2509-14382/ · source: https://arxiv.org/abs/2509.14382v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, observability, web-navigation - Published: 2025-09-17 · Authors: Daniel Röder, Akhil Juneja, Roland Roller, Sven Schmeier ### DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents DiagEval diagnoses failed GUI-agent evaluations of interactive software by reusing failed trajectories for targeted probes that distinguish execution errors from defects. - https://awesomecua.com/resources/arxiv-2605-17439/ · source: https://arxiv.org/abs/2605.17439v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, observability, interaction-history - Published: 2026-05-17 · Authors: Sirui Hong, Zhijie Liu, Tengfei Li, Wei Tao, Yifan Wu, Chenglin Wu ### Diagnosing Personalization Failures in Mobile GUI Agents PrefDiag-Bench diagnoses when and how user preferences should affect sequential mobile GUI agent actions, using staged probes and user context. - https://awesomecua.com/resources/arxiv-2610-30573/ · source: https://arxiv.org/abs/2610.30573v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, mobile-navigation, memory, state-tracking - Published: 2026-10-22 · Authors: Zhixin Lin, Dongliang Xu, Jungang Li, Shidong Pan, Jun Feng, Yanbin Sun, Jing Li, Liang Liu, Yue Yao ### DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections Evaluates whether deceptive UI content steers mobile agents away from instruction-consistent choices, using paired benign and injected decision states and decision-integrity metrics. - https://awesomecua.com/resources/arxiv-2610-06898/ · source: https://arxiv.org/abs/2610.06898v1 - Paper · reviewed 2026-10-18 · environments: android · topics: evaluation, safety-security, mobile-navigation - Published: 2026-09-29 · Authors: Li Hu, Kanghua Mo, Yingbin Jin, Qingqing Ye, Haibo Hu ### DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces? DiffSpot benchmarks detection of fine visual changes in rendered web interfaces using image pairs created by single-property CSS mutations. - https://awesomecua.com/resources/arxiv-2605-29615/ · source: https://arxiv.org/abs/2605.29615v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, screen-understanding - Published: 2026-05-28 · Authors: Linhao Zhang, Aiwei Liu, Yuan Liu, Xiao Zhou ### Digi-Q: Learning Q-Value Functions for Training Device-Control Agents Learns action-value functions from offline device-control experience and uses them to rank candidate actions for policy improvement. - https://awesomecua.com/resources/arxiv-2502-15760/ · source: https://arxiv.org/abs/2502.15760v1 - Paper · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, efficiency - Published: 2025-02-13 · Authors: Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, Aviral Kumar ### DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning Combines offline initialization and online reinforcement learning in Android environments, with automated evaluation and curriculum-based sampling. - https://awesomecua.com/resources/arxiv-2406-11896/ · source: https://arxiv.org/abs/2406.11896v1 - Paper · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, mobile-navigation - Published: 2024-06-14 · Authors: Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar ### DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coaching DigitalCoach records expert–novice software coaching conversations aligned with screen and input events. Automated and interactive evaluations examine how model coaches differ from humans in explanation, error diagnosis and visual grounding when helping a learner operate software. - https://awesomecua.com/resources/arxiv-2606-31980/ · source: https://arxiv.org/abs/2606.31980v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, video, evaluation, grounding - Published: 2026-06-30 · Authors: Meng Chen, Anya Ji, Tsung-Han Wu, Tobias Maringgele, David M. Chan, Alane Suhr, Amy Pavel ### DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning Separates textual and iconic UI elements and iteratively focuses visual regions to refine grounding without additional model training. - https://awesomecua.com/resources/arxiv-2507-00008/ · source: https://arxiv.org/abs/2507.00008v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, screen-understanding - Published: 2025-06-12 · Authors: Hang Wu, Hongkai Chen, Yujun Cai, Chang Liu, Qingwen Ye, Ming-Hsuan Yang, Yiwei Wang ### Discriminative World Models for Web Agents Trains discriminative web world models to distinguish the true next state from outcomes of alternative actions. Branching web trajectories support training, followed by tests of predicted-state matching, process-reward action ranking and end-to-end WebArena-Lite selection. - https://awesomecua.com/resources/arxiv-2609-02885/ · source: https://arxiv.org/abs/2609.02885v1 - Paper · reviewed 2026-10-18 · environments: web · topics: world-models, reward-models, planning, data-generation, evaluation - Published: 2026-09-02 · Authors: Kelvin Li, Dhruv Pendharkar, Anish Pahilajani, Chuyi Shang, Leon Oks, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Roei Herzig ### Dissecting Adversarial Robustness of Multimodal LM Agents Introduces adversarial tasks on VisualWebArena and an evaluation framework that tracks how manipulated information passes between agent components. The analysis examines vulnerabilities introduced by reflection and tree-search evaluators as well as direct action generation. - https://awesomecua.com/resources/arxiv-2406-12814/ · source: https://arxiv.org/abs/2406.12814v3 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, robustness, evaluation - Published: 2024-06-18 · Authors: Chen Henry Wu, Rishi Shah, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, Aditi Raghunathan ### DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents Introduces asynchronous distributed reinforcement learning with centralized updates and decentralized interaction collection for device-control agents. - https://awesomecua.com/resources/arxiv-2410-14803/ · source: https://arxiv.org/abs/2410.14803v5 - Paper · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, efficiency - Published: 2024-10-18 · Authors: Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, Kun Shao ### Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure Defines visual state reliance and probes whether GUI agents resolve screenshot/structure conflicts using pixels, serialized interface structure or prior beliefs. Paired interventions and live multi-step tests connect misattributed state beliefs to failures and compare consistency-based mitigations. - https://awesomecua.com/resources/arxiv-2607-04334/ · source: https://arxiv.org/abs/2607.04334v2 - Paper · reviewed 2026-10-17 · environments: cross-platform · topics: screen-understanding, state-tracking, evaluation, robustness, safety-security - Published: 2026-07-05 · Authors: Guijia Zhang, Yuxun Chen, Yuheng Qi, Harry Yang ### Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents CONFLICTGUI evaluates termination under instruction conflicts and mismatches between instructions and interface context. CONFLICTGUARD couples feasibility checks with conditional action modulation to reduce unnecessary execution while tracking normal-task behavior. - https://awesomecua.com/resources/arxiv-2609-03438/ · source: https://arxiv.org/abs/2609.03438v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, evaluation, robustness, planning - Published: 2026-09-03 · Authors: Zhaoyuan Huang, Tianjie Ju, Pengzhou Cheng, Zheng Wu, Yansi Li, Chuanbiao Song, Jun Lan, Huijia Zhu, Weiqiang Wang, Zhuosheng Zhang ### Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots DailyDroid compares text-only and text-plus-screenshot agents on everyday Android tasks across multiple apps and difficulty levels. Its failure analysis examines accessibility, observation modality and agent/application behavior. - https://awesomecua.com/resources/arxiv-2604-17817/ · source: https://arxiv.org/abs/2604.17817v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation, screen-understanding, robustness - Published: 2026-04-20 · Authors: Shiquan Zhang, Tianyi Zhang, Le Fang, Simon D'Alfonso, Hong Jia, Vassilis Kostakos ### Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding Introduces WiserUI-Bench, pairing real UI variants from industry A/B tests with observed behavioral winners and expert explanations. It evaluates models’ ability to predict the more effective interface and explain its effect on user behavior. - https://awesomecua.com/resources/arxiv-2505-05026/ · source: https://arxiv.org/abs/2505.05026v5 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, ui-design, evaluation - Published: 2025-05-08 · Authors: Jaehyun Jeon, Min Soo Kim, Jang Han Yoon, Sumin Shim, Yejin Choi, Hanbin Kim, Dae Hyun Kim, Youngjae Yu ### Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decomposition Minitap coordinates six specialized agents for AndroidWorld tasks, checking text input against device state and detecting action loops to revise its strategy. - https://awesomecua.com/resources/arxiv-2602-07787/ · source: https://arxiv.org/abs/2602.07787v1 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, planning, state-tracking, self-reflection - Published: 2026-02-08 · Authors: Pierre-Louis Favreau, Jean-Pierre Lo, Clement Guiguet, Charles Simon-Meunier, Nicolas Dehandschoewercker, Allen G. Roush, Judah Goldfeder, Ravid Shwartz-Ziv ### Do Phone-Use Agents Respect Your Privacy? Introduces MyPhoneBench to verify whether phone-use agents minimize personal data disclosure when filling forms during benign mobile tasks. - https://awesomecua.com/resources/arxiv-2604-00986/ · source: https://arxiv.org/abs/2604.00986v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, mobile-navigation, evaluation - Published: 2026-04-01 · Authors: Zhengyang Tang, Ke Ji, Xidong Wang, Zihan Ye, Xinyuan Wang, Yiduo Guo, Ziniu Li, Chenxin Li, Jingyuan Hu, Shunian Chen, Tongxu Luo, Jiaxi Bi, Zeyu Qin, Shaobo Wang, Xin Lai, Pengyuan Lyu, Junyi Li, Can Xu, Chengquan Zhang, Han Hu, Ming Yan, Benyou Wang ### Do Web Agents Investigate Before They Decide? Introduces MIRAGE, a web decision benchmark where agents must actively uncover evidence that can contradict the visible context. Separate investigation, reasoning, decision, and hallucination measures diagnose failures beyond reaching the right page. - https://awesomecua.com/resources/arxiv-2602-05354/ · source: https://arxiv.org/abs/2602.05354v3 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, planning - Published: 2026-02-05 · Authors: Syed Nazmus Sakib, Nafiul Haque, Tapodhir Karmakar Taton, Shahrear Bin Amin, Shifat E. Arman ### Doc2Agent: Scalable Generation of Tool-Using Agents from API Documentation Introduces Doc2Agent, a pipeline that generates and refines Python tools from API documentation for tool-using agents, with WebArena APIs among its evaluations. - https://awesomecua.com/resources/arxiv-2506-19998/ · source: https://arxiv.org/abs/2506.19998v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: tool-integration, automation - Published: 2025-06-24 · Authors: Xinyi Ni, Haonan Jian, Qiuyang Wang, Vedanshi Chetan Shah, Pengyu Hong ### DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations Introduces DocOps, a verifiable benchmark for agents handling complex digital-document workflows, with analysis of state tracking, semantic checks, and structural edits. - https://awesomecua.com/resources/arxiv-2607-19865/ · source: https://arxiv.org/abs/2607.19865v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, workflow, state-tracking - Published: 2026-07-22 · Authors: Jiazhen Jiang, Boxi Cao, Lingyong Yan, Yaojie Lu, Hongyu Lin, Shuaiqiang Wang, Dawei Yin, Xianpei Han, Le Sun ### DocOS: Towards Proactive Document-Guided Actions in GUI Agents DocOS benchmarks agents that search online documentation and translate procedural instructions into GUI actions in interactive web environments. It separates failures to retrieve useful procedures from failures to ground retrieved instructions into executable actions. - https://awesomecua.com/resources/arxiv-2605-18048/ · source: https://arxiv.org/abs/2605.18048v1 - Paper · reviewed 2026-10-18 · environments: web · topics: web-navigation, grounding, evaluation, planning - Published: 2026-05-18 · Authors: Jingjing Liu, Ziye Huang, Zihao Cheng, Zeming Liu, Jiahong Wu, Yuhang Guo, Kehai Chen, Yunhong Wang, Haifeng Wang ### Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Study This study compares matched reasoning and non-reasoning vision-language models on static mobile GUI tasks and interactive AndroidWorld navigation. It finds that chain-of-thought effects vary across tasks and model families. - https://awesomecua.com/resources/arxiv-2503-16788/ · source: https://arxiv.org/abs/2503.16788v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation, planning - Published: 2025-03-21 · Authors: Li Zhang, Longxi Gao, Mengwei Xu ### DOM-Q-NET: Grounded RL on Structured Language Introduces DOM-Q-NET, an RL architecture for web navigation that represents HTML as a tree and uses separate Q-function networks for clicking elements and typing text. - https://awesomecua.com/resources/arxiv-1902-07257/ · source: https://arxiv.org/abs/1902.07257v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, web-navigation, screen-understanding - Published: 2019-02-19 · Authors: Sheng Jia, Jamie Kiros, Jimmy Ba ### Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming Introduces CUA-HandCrafted, a browser-agent red-teaming benchmark that varies attack templates, tasks and system prompts. Its domain comparison and reproducibility audit caution against transferring browser-surface safety conclusions to coding agents or unreleased optimized injections. - https://awesomecua.com/resources/arxiv-2606-05233/ · source: https://arxiv.org/abs/2606.05233v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, evaluation, robustness - Published: 2026-06-03 · Authors: Nicholas Saban ### Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction Introduces VeriGUI, which checks expected action effects and reasons about recovery before subsequent GUI actions. Training combines synthetic failure trajectories with verification-weighted GRPO, evaluated on an AndroidControl-based robustness benchmark. - https://awesomecua.com/resources/arxiv-2604-05477/ · source: https://arxiv.org/abs/2604.05477v1 - Paper · reviewed 2026-10-17 · environments: android · topics: robustness, reinforcement-learning, state-tracking - Published: 2026-04-07 · Authors: Yuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen, Chen Liu, Xinran He, Run Shao, Feiran Liu, Huanmin Xu, Qiutong Pan, Haiwei Wang ### Don't Click That: Teaching Web Agents to Resist Deceptive Interfaces DUDE combines asymmetric hybrid-reward learning with summaries of deceptive-interface failures to guide safer web actions. Its RUC benchmark evaluates deception susceptibility alongside ordinary task performance across interface scenarios. - https://awesomecua.com/resources/arxiv-2605-09497/ · source: https://arxiv.org/abs/2605.09497v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, reinforcement-learning, evaluation, robustness - Published: 2026-05-10 · Authors: Yilin Zhang, Yingkai Hua, Chunyu Wei, Xin Wang, Yueguo Chen ### Don't Retrain, Remix: Adaptation of Web Agents using Mixture Hypernetworks WebMix uses a scout-produced interface manual to route mixtures of source-trained LoRA adapters for web agents, adapting to target interfaces without target-time gradient updates. - https://awesomecua.com/resources/arxiv-2610-22539/ · source: https://arxiv.org/abs/2610.22539v1 - Paper · reviewed 2026-10-20 · environments: web · topics: web-navigation, imitation-learning, robustness - Published: 2026-10-15 · Authors: Md Farhan Ishmam, Minh Pham Dinh, Kenneth Marino ### DoomArena: A framework for Testing AI Agents Against Evolving Security Threats Presents a modular security testing framework that plugs into BrowserGym and tool-use benchmarks to evaluate agents under configurable threat models. - https://awesomecua.com/resources/arxiv-2504-14064/ · source: https://arxiv.org/abs/2504.14064v3 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, evaluation - Published: 2025-04-18 · Authors: Leo Boisvert, Mihir Bansal, Chandra Kiran Reddy Evuru, Gabriel Huang, Abhay Puri, Avinandan Bose, Maryam Fazel, Quentin Cappart, Jason Stanley, Alexandre Lacoste, Alexandre Drouin, Krishnamurthy Dvijotham ### DPO Learning with LLMs-Judge Signal for Computer Use Agents Uses an LLM judge to evaluate and filter synthetic GUI interaction trajectories for training a compact vision-language agent. OSWorld experiments assess the policy designed for local execution. - https://awesomecua.com/resources/arxiv-2506-03095/ · source: https://arxiv.org/abs/2506.03095v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: data-generation, evaluation, efficiency - Published: 2025-06-03 · Authors: Man Luo, David Cobbley, Xin Su, Shachar Rosenman, Vasudev Lal, Shao-Yen Tseng, Phillip Howard ### DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions DragOn supplies drag-grounding training data and a held-out benchmark for highlighting text, selecting cells, resizing elements and moving sliders. It evaluates existing vision-language models and a model fine-tuned on the drag tasks. - https://awesomecua.com/resources/arxiv-2606-06322/ · source: https://arxiv.org/abs/2606.06322v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, data-generation, evaluation, screen-understanding - Published: 2026-06-04 · Authors: Nathan Bout, Maxime Langevin, Ronan Riochet ### Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want Introduces Draw-and-Understand and MDVP-Bench for understanding visual prompts such as points and boxes; its multi-domain data includes mobile and web screenshots. - https://awesomecua.com/resources/arxiv-2403-20271/ · source: https://arxiv.org/abs/2403.20271v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, data-generation, evaluation - Published: 2024-03-29 · Authors: Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, Hongsheng Li ### DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks DrawingBench evaluates sequences of low-level mouse-based GUI actions through 250 drawing prompts and rule-based criteria. Action inspection and multi-turn external feedback support analysis of spatial reasoning, tool-state management, and refinement. - https://awesomecua.com/resources/arxiv-2512-01174/ · source: https://arxiv.org/abs/2512.01174v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, grounding, human-in-the-loop, state-tracking - Published: 2025-12-01 · Authors: Hyunjun Kim, Sooyoung Ryu ### DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation Generates labeled slides and interface visuals through code to support visual element detection, description, and classification with less manual annotation. - https://awesomecua.com/resources/arxiv-2410-00201/ · source: https://arxiv.org/abs/2410.00201v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation, screen-understanding - Published: 2024-09-30 · Authors: Yi-Hao Peng, Faria Huq, Yue Jiang, Jason Wu, Amanda Xin Yue Li, Jeffrey Bigham, Amy Pavel ### DRIVE: Modeling Skills at the Reasoning and Interaction Levels for Web Agents under Continual Learning DRIVE separates reusable reasoning skills from page-specific interaction skills for web agents, retrieving both to plan tasks and execute actions on websites. - https://awesomecua.com/resources/arxiv-2605-23939/ · source: https://arxiv.org/abs/2605.23939v1 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, planning, web-navigation, self-reflection - Published: 2026-04-28 · Authors: Xirui Liu, Sihang Zhou, Yanning Hou, Rong Zhou, Haoyuan Chen, Maolin He, Siwei Wang, Hao Chen, Jian Huang ### DroidBot-GPT: GPT-powered UI Automation for Android Transforms Android UI states and available actions into language prompts for an LLM to choose and execute task steps. - https://awesomecua.com/resources/arxiv-2304-07061/ · source: https://arxiv.org/abs/2304.07061v5 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, automation - Published: 2023-04-14 · Authors: Hao Wen, Hongming Wang, Jiaxuan Liu, Yuanchun Li ### DroidCall: A Dataset for LLM-powered Android Intent Invocation Introduces data and a generation pipeline for translating natural-language requests into Android intent invocations using compact language models. - https://awesomecua.com/resources/arxiv-2412-00402/ · source: https://arxiv.org/abs/2412.00402v1 - Paper · reviewed 2026-10-16 · environments: android · topics: tool-integration, data-generation - Published: 2024-11-30 · Authors: Weikai Xie, Li Zhang, Shihe Wang, Rongjie Yi, Mengwei Xu ### DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding DRS-GUI searches screenshot regions through focus, shift and scatter operations selected by a tree-search planner. A region-quality reward guides this training-free perceptual search before final GUI grounding. - https://awesomecua.com/resources/arxiv-2605-15542/ · source: https://arxiv.org/abs/2605.15542v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, planning, screen-understanding - Published: 2026-05-15 · Authors: Yichao Liu, Huawen Shen, Liu Yu, Shiyu Liu, Zeyu Chen, Yu Zhou ### DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? DSAgentBench evaluates end-to-end data-science workflows spanning analysis, modeling and visualization inside a real desktop environment. Task evaluators check produced analytical and visual artifacts, while the released interface supports screenshot or accessibility observations and GUI actions. - https://awesomecua.com/resources/arxiv-2608-10366/ · source: https://arxiv.org/abs/2608.10366v1 - Paper · reviewed 2026-10-17 · environments: desktop, linux · topics: evaluation, workflow, science, long-horizon - Published: 2026-08-11 · Authors: Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub, Md Tahmid Rahman Laskar, Shafiq Joty, Enamul Hoque Prince ### Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks Studies webpage injections that affect both screenshots and accessibility trees, then trains web agents through teacher imitation, supervised adversarial examples, and adversarial reinforcement learning. The evaluation tests defenses on MiniWoB++ and held-out tasks. - https://awesomecua.com/resources/arxiv-2603-04364/ · source: https://arxiv.org/abs/2603.04364v2 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, reinforcement-learning, evaluation - Published: 2026-03-04 · Authors: Haoyu Liu, Dingcheng Li, Lukas Rutishauser, Zeyu Zheng ### Dual-View Visual Contextualization for Web Navigation Uses screenshot geometry and visual content to enrich HTML element representations for web navigation. Nearby elements contribute textual and visual context, with evaluation on cross-task, cross-website, and cross-domain Mind2Web splits. - https://awesomecua.com/resources/arxiv-2402-04476/ · source: https://arxiv.org/abs/2402.04476v2 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, screen-understanding, web-navigation - Published: 2024-02-06 · Authors: Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, Wei-Lun Chao ### DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agents Applies dual-task adversarial perturbations to mobile screenshots to conceal personally identifiable information from untrusted MLLM routers while preserving agent task utility. - https://awesomecua.com/resources/arxiv-2511-13248/ · source: https://arxiv.org/abs/2511.13248v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, mobile-navigation - Published: 2025-11-17 · Authors: Fuyao Zhang, Jiaming Zhang, Che Wang, Xiongtao Sun, Yurong Hao, Guowei Guan, Wenjie Li, Longtao Huang, Wei Yang Bryan Lim ### Dynamic Planning for LLM-based Graphical User Interface Automation Dynamic Planning of Thoughts revises a mobile GUI plan using current environmental feedback and execution history. The study compares this planning strategy with long-dialogue prompting and examines its behavior on unfamiliar tasks and different language-model backbones. - https://awesomecua.com/resources/arxiv-2410-00467/ · source: https://arxiv.org/abs/2410.00467v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: planning, interaction-history, mobile-navigation, robustness - Published: 2024-10-01 · Authors: Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen, Xinbei Ma, Muyun Yang, Tiejun Zhao, Min Zhang ### DynaRev: Benchmarking Web Agents under Dynamic Goal Revision Introduces DynaRev, a benchmark of 181 web-agent tasks whose goals change during execution, with trajectory-based measures of correctness, responsiveness, and efficiency. - https://awesomecua.com/resources/arxiv-2610-40133/ · source: https://arxiv.org/abs/2610.40133v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, state-tracking - Published: 2026-10-29 · Authors: Shiqi Zhang, Guochen Zhou, Zesheng Wei, Hai Zhao, Zhisong Zhang ### DynaWeb: Model-Based Reinforcement Learning of Web Agents DynaWeb learns a world model that predicts web-page representations after actions and uses it to generate policy-training rollouts. It interleaves these rollouts with expert trajectories and evaluates the trained web agents on WebArena and WebVoyager. - https://awesomecua.com/resources/arxiv-2601-22149/ · source: https://arxiv.org/abs/2601.22149v2 - Paper · reviewed 2026-10-17 · environments: web · topics: world-models, reinforcement-learning, data-generation, web-navigation - Published: 2026-01-29 · Authors: Hang Ding, Peidong Liu, Junqiao Wang, Ziwei Ji, Meng Cao, Rongzhao Zhang, Lynn Ai, Eric Yang, Tianyu Shi, Lei Yu ### E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion Introduces Chinese-language GUI navigation traces and annotated screenshots collected from human interactions with mini-applications. - https://awesomecua.com/resources/arxiv-2406-14250/ · source: https://arxiv.org/abs/2406.14250v3 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, evaluation - Published: 2024-06-20 · Authors: Ke Wang, Tianyu Xia, Zhangxuan Gu, Yi Zhao, Shuheng Shen, Changhua Meng, Weiqiang Wang, Ke Xu ### Eager: Programming Repetitive Tasks by Example Describes Eager, which detects repeated HyperCard interactions, generalizes them into loops, and previews predicted actions by highlighting interface objects. Users can then ask it to execute the inferred repetitive task. - https://awesomecua.com/resources/url-bf0c11018e49dff6aeec/ · source: https://acypher.com/Publications/CHI91/EagerCHI.html - Paper · reviewed 2026-10-17 · environments: desktop · topics: automation, human-in-the-loop, interaction-history - Published: 1991 · Authors: Allen Cypher ### ECHOES: Seeing Beyond the Current Image via Recursive History Compression and Privileged On-Policy Distillation ECHOES learns recursive textual task memory through privileged screenshot-history distillation and GRPO, then predicts GUI actions from the current screenshot and memory. - https://awesomecua.com/resources/arxiv-2610-22408/ · source: https://arxiv.org/abs/2610.22408v1 - Paper · reviewed 2026-10-20 · environments: android · topics: memory, interaction-history, state-tracking, long-horizon, imitation-learning, reinforcement-learning, efficiency, evaluation - Published: 2026-10-14 · Authors: Tengda Guo, Jiawang Hu, Yifei Tao, Yin Wang, Haotian Hu, Fanyi Wang, Liujuan Tang ### EchoPath: Execution-Level Replayable Memory for GUI Agents EchoPath turns validated GUI trajectories into parameterized replay procedures with explicit runtime preconditions. Image-based target matching adjusts stored coordinates, while ambiguous steps fall back to bounded repair or fresh planning. - https://awesomecua.com/resources/arxiv-2609-16635/ · source: https://arxiv.org/abs/2609.16635v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, workflow, automation, grounding, efficiency, robustness - Published: 2026-09-15 · Authors: Yao Zhao, Aditya Shanmugham, Swastik Roy, Yanxun Xu ### EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration EchoTrail-GUI builds a memory of successful GUI trajectories through autonomous exploration filtered by a reward model. It retrieves relevant trajectories as in-context guidance for new tasks and evaluates the approach on AndroidWorld and AndroidLab. - https://awesomecua.com/resources/arxiv-2512-19396/ · source: https://arxiv.org/abs/2512.19396v3 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, data-generation, reward-models, mobile-navigation - Published: 2025-12-22 · Authors: Runze Li, Yuwen Zhai, Bo Xu, LiWu Xu, Nian Shi, Wei Zhang, Ran Lin, Liang Wang ### Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale Introduces Echoverse, which compiles specifications into stateful applications with database-grounded graders and uses rollouts to improve both the environments and agent policies. It examines behavioral depth, targeted interface practice, and environment repair as sources of training value. - https://awesomecua.com/resources/arxiv-2607-28074/ · source: https://arxiv.org/abs/2607.28074v1 - Paper · reviewed 2026-10-17 · environments: web · topics: sandbox, data-generation, reinforcement-learning - Published: 2026-07-30 · Authors: Yash Pandya, Sahil Gupta, Sarthak Harne, Archana Yadav, Kavyansh Chourasia, Hussein Mozannar, Vibhav Vineet, Sara Abdali, Corby Rosset, Yash Lara, Ahmed Awadallah, Ece Kamar, Akshay Nambi ### EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automation EcoAgent splits mobile automation between cloud planning and on-device observation and verification. Compact screen descriptions and feedback reduce communication costs, with evaluation on AndroidWorld. - https://awesomecua.com/resources/arxiv-2505-05440/ · source: https://arxiv.org/abs/2505.05440v3 - Paper · reviewed 2026-10-18 · environments: android · topics: mobile-navigation, planning, efficiency - Published: 2025-05-08 · Authors: Biao Yi, Xavier Hu, Yurun Chen, Shengyu Zhang, Hongxia Yang, Fan Wu ### EconSkills: Studying Skill Transfer and Retrieval for Web Agents on Live Economic Data EconSkills converts verified economic-data browsing traces into parameterized procedures with navigation, verification and recovery guidance. Its evaluation separates known-skill transfer from library retrieval, exposing the limits of approximate matches to new tasks. - https://awesomecua.com/resources/arxiv-2609-19523/ · source: https://arxiv.org/abs/2609.19523v2 - Paper · reviewed 2026-10-18 · environments: web · topics: memory, web-navigation, evaluation, workflow - Published: 2026-09-17 · Authors: Yinzhu Quan, Zefang Liu ### EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environments Introduces EconWebArena, a curated benchmark of economic-data tasks requiring live website navigation, multimodal interpretation, and precise extraction. It evaluates visual grounding, planning, and interaction design within multistep workflows. - https://awesomecua.com/resources/arxiv-2506-08136/ · source: https://arxiv.org/abs/2506.08136v3 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, planning - Published: 2025-06-09 · Authors: Zefang Liu, Yinzhu Quan ### EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data Synthesizes GUI training examples at multiple levels of detail from webpages and studies transfer of learned understanding to other interface environments. - https://awesomecua.com/resources/arxiv-2410-19461/ · source: https://arxiv.org/abs/2410.19461v2 - Paper · reviewed 2026-10-16 · environments: web, cross-platform · topics: data-generation, grounding, screen-understanding - Published: 2024-10-25 · Authors: Xuetian Chen, Hangcheng Li, Jiaqing Liang, Sihang Jiang, Deqing Yang ### EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning EE-MCP studies application-dependent mixtures of GUI actions and MCP calls using generated environments, trajectory collection and filtered training. An experience bank extracts rules from trajectory comparisons for inference-time improvement, distinct from model distillation. - https://awesomecua.com/resources/arxiv-2604-09815/ · source: https://arxiv.org/abs/2604.09815v1 - Paper · reviewed 2026-10-18 · environments: desktop · topics: tool-integration, data-generation, imitation-learning, memory - Published: 2026-04-10 · Authors: Tiantian He, Yihang Chen, Keyue Jiang, Ka Yiu Lee, Kaiwen Zhou, Kun Shao, Shuai Wang ### Efficient Agent Training for Computer Use PC Agent-E augments 312 human computer-use trajectories with alternative action decisions synthesized by Claude 3.7 Sonnet. It studies training with this combination and introduces WindowsAgentArena-V2 as an updated evaluation benchmark. - https://awesomecua.com/resources/arxiv-2505-13909/ · source: https://arxiv.org/abs/2505.13909v2 - Paper · reviewed 2026-10-17 · environments: windows, desktop · topics: data-generation, evaluation - Published: 2025-05-20 · Authors: Yanheng He, Jiahe Jin, Pengfei Liu ### Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation DART separates GUI reinforcement learning into asynchronous environment, rollout, data-management, and training services. It curates experience by task difficulty, successful trajectories, and step uncertainty while correcting rollout-policy mismatch, with evaluation on OSWorld. - https://awesomecua.com/resources/arxiv-2509-23866/ · source: https://arxiv.org/abs/2509.23866v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: reinforcement-learning, efficiency, data-generation, sandbox - Published: 2025-09-28 · Authors: Pengxiang Li, Zechen Hu, Zirui Shang, Jingrong Wu, Yang Liu, Hui Liu, Zhi Gao, Chenrui Shi, Bofei Zhang, Zihao Zhang, Xiaochuan Shi, Zedong YU, Yuwei Wu, Xinxiao Wu, Yunde Jia, Liuyu Xiang, Zhaofeng He, Qing Li ### Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos Ego2Web pairs first-person videos with online tasks that require agents to connect the user’s physical context to web actions. It also evaluates an automatic task judge against human judgments and analyzes failures in video understanding and web execution. - https://awesomecua.com/resources/arxiv-2603-22529/ · source: https://arxiv.org/abs/2603.22529v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, video, web-navigation, planning - Published: 2026-03-23 · Authors: Shoubin Yu, Lei Shu, Antoine Yang, Yao Fu, Srinivas Sunkara, Maria Wang, Jindong Chen, Mohit Bansal, Boqing Gong ### EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage Evaluates privacy-directed webpage instruction injection against generalist web agents using PII-related interaction steps from Mind2Web, including attempts to expose specific user information or entire requests. - https://awesomecua.com/resources/arxiv-2409-11295/ · source: https://arxiv.org/abs/2409.11295v5 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2024-09-17 · Authors: Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, Huan Sun ### ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults ElderBench evaluates mobile GUI agents on smartphone requests elicited from older adults, including indirect and underspecified language. Online and offline tests, instruction normalization and linguistic analyses examine how these request patterns contribute to failures. - https://awesomecua.com/resources/arxiv-2609-04850/ · source: https://arxiv.org/abs/2609.04850v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, human-in-the-loop, mobile-navigation, robustness - Published: 2026-09-04 · Authors: Weide Zhan, Qumu Shaqu, Yuanqing Liu, Peng Zhang, Jiahao Liu, Kam Him Lam, Ning Gu, Zhan Hu, Tun Lu ### EmbeWebAgent: Embedding Web Agents into Any Customized UI EmbeWebAgent embeds agent access into existing web applications through frontend observations and a page-specific function registry. Its workflow combines GUI primitives, composite actions and MCP tools, demonstrated in a live interface. - https://awesomecua.com/resources/arxiv-2602-14865/ · source: https://arxiv.org/abs/2602.14865v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration, automation, workflow - Published: 2026-02-16 · Authors: Chenyang Ma, Clyde Fare, Matthew Wilson, Dave Braines ### Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence Introduces environments that couple 3D physical-world simulations with functional web interfaces. The accompanying benchmark evaluates tasks requiring agents to coordinate online information and physical actions across cooking, navigation and shopping scenarios. - https://awesomecua.com/resources/arxiv-2506-15677/ · source: https://arxiv.org/abs/2506.15677v3 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, planning - Published: 2025-06-18 · Authors: Yining Hong, Rui Sun, Bingxuan Li, Xingcheng Yao, Maxine Wu, Alexander Chien, Da Yin, Ying Nian Wu, Zhecan James Wang, Kai-Wei Chang ### Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild Audits sources of ambiguity and operational variation in WebVoyager evaluation and introduces standardized guidance for task setup, failure handling, annotation and reporting. Emergence WebVoyager applies this protocol to a web-agent evaluation. - https://awesomecua.com/resources/arxiv-2603-29020/ · source: https://arxiv.org/abs/2603.29020v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation - Published: 2026-03-30 · Authors: Deepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku ### Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning PEEU explores web environments and turns hindsight experience into aligned high-level planning supervision. A hierarchical task-decomposition analysis examines why atomic-skill competence alone does not ensure planning generalization across websites. - https://awesomecua.com/resources/arxiv-2606-27330/ · source: https://arxiv.org/abs/2606.27330v1 - Paper · reviewed 2026-10-18 · environments: web · topics: planning, data-generation, imitation-learning, web-navigation - Published: 2026-06-25 · Authors: Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao ### Enabling Conversational Interaction with Mobile UI using Large Language Models Studies few-shot prompting of a single language model for four mobile-UI interaction tasks, aiming to support conversational interface use without a separate trained model for each task. - https://awesomecua.com/resources/arxiv-2209-08655/ · source: https://arxiv.org/abs/2209.08655v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, screen-understanding, training-free - Published: 2022-09-18 · Authors: Bryan Wang, Gang Li, Yang Li ### End-to-End Goal-Driven Web Navigation Introduces WebNav for converting websites into goal-directed hyperlink navigation tasks and WikiNav for evaluating language-conditioned navigation and question answering on Wikipedia. - https://awesomecua.com/resources/arxiv-1602-02261/ · source: https://arxiv.org/abs/1602.02261v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, evaluation, planning - Published: 2016-02-06 · Authors: Rodrigo Nogueira, Kyunghyun Cho ### EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments? Introduces professional engineering tasks across software domains, with GUI and CLI access and artifact checks for geometric, physical, and design validity. - https://awesomecua.com/resources/arxiv-2609-37686/ · source: https://arxiv.org/abs/2609.37686v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: evaluation, science, cross-app - Published: 2026-09-29 · Authors: Hongcheng Gao, Hailong Qu, Yu Lei, Henghui Sun, Haoyang Li, Yipeng Wei, Naihao Xue, Xiaohan Yu, Zhuo Tao, Yihe Zang, Yajiao Wang, Jingyi Tang, Yi Li, Jingjing Zhou, Jie Luo, Bohan Zeng, Chengyu Shen, Hao Jiang, Chong Chen, Bowen Qu, Olive Huang, Zeqiang Wang ### Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking Executes instructions from mobile how-to search results inside a controlled Android environment, then uses the observed success of those instructions to verify and rerank technical-support results. - https://awesomecua.com/resources/arxiv-2404-08860/ · source: https://arxiv.org/abs/2404.08860v3 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, evaluation, automation - Published: 2024-04-13 · Authors: Lei Ding, Jeshwanth Bheemanpally, Yi Zhang ### Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning HyperClick trains GUI grounders to produce a click and an accompanying confidence estimate using rewards for correctness and confidence alignment. Its evaluation examines localization, calibration, and abstention behavior. - https://awesomecua.com/resources/arxiv-2510-27266/ · source: https://arxiv.org/abs/2510.27266v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, robustness, evaluation - Published: 2025-10-31 · Authors: Shaojie Zhang, Pei Fu, Ruoceng Zhang, Jiahui Yang, Anan Du, Xiuwen Xi, Shaokang Wang, Ying Huang, Bin Qin, Zhenbo Luo, Jian Luan ### Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning Proposes GUI grounding training that combines curated seed examples, continuous location-dependent policy feedback, and iterative refinement guided by attention maps. The study evaluates the resulting reinforcement-learning procedure on three grounding benchmarks, including ScreenSpot-Pro. - https://awesomecua.com/resources/arxiv-2505-12370/ · source: https://arxiv.org/abs/2505.12370v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning - Published: 2025-05-18 · Authors: Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, Bo Li ### Enhancing Web Agents with a Hierarchical Memory Tree Introduces Hierarchical Memory Tree (HMT), which organizes web-agent trajectories by intent, stage and action, then uses the hierarchy to plan and act on websites. - https://awesomecua.com/resources/arxiv-2603-07024/ · source: https://arxiv.org/abs/2603.07024v1 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, planning, web-navigation, interaction-history - Published: 2026-03-07 · Authors: Yunteng Tan, Zhi Gao, Xinxiao Wu ### EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents Introduces EntWorld, an enterprise GUI benchmark built by deriving task logic from database schemas. Long workflows across enterprise domains are assessed with deterministic SQL-based state checks rather than screenshot similarity alone. - https://awesomecua.com/resources/arxiv-2601-17722/ · source: https://arxiv.org/abs/2601.17722v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, industry, workflow - Published: 2026-01-25 · Authors: Ying Mo, Yu Bai, Dapeng Sun, Yuqian Shi, Yukai Miao, Li Chen, Dan Li ### Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environments Introduces Chameleon, an environmental-injection evaluation framework that varies webpage content and trigger context. It combines simulated page variation with attention-based optimization and tests GUI agents across realistic websites. - https://awesomecua.com/resources/arxiv-2509-11250/ · source: https://arxiv.org/abs/2509.11250v3 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, robustness, evaluation - Published: 2025-09-14 · Authors: Yitong Zhang, Ximo Li, Liyi Cai, Jia Li ### EnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation Introduces EnvOS, a framework for constructing and auditing changing computer-use environments, including checks of task changes, solvability, grading, and run evidence. - https://awesomecua.com/resources/arxiv-2610-40246/ · source: https://arxiv.org/abs/2610.40246v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, robustness, observability - Published: 2026-10-29 · Authors: Ankush Kadu, Athira Gopal, Niharika Singh, Arpit Jain, Aswanth Krishnan ### ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents ENVS branches over GUI actions in live desktop VMs, verifies successful leaves, and converts them into balanced step-level training supervision. It also introduces OSWorld-Noisy to test recovery from dynamic interruptions while retaining the underlying tasks. - https://awesomecua.com/resources/arxiv-2606-22948/ · source: https://arxiv.org/abs/2606.22948v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: data-generation, reinforcement-learning, long-horizon, robustness, evaluation - Published: 2026-06-22 · Authors: Yincheng Zhou, Athena Zhuoming Zhong, Shijie Zhang, Kevin Zhang, Teresa Xiaotao Shang, Shanghang Zhang ### ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software ERPBench evaluates screenshot-only agents in enterprise software using database values to check the business records produced by their actions. It pairs these state-based checks with a harness for human approval and analyzes cases where visible completion masks incorrect persisted data. - https://awesomecua.com/resources/arxiv-2609-17885/ · source: https://arxiv.org/abs/2609.17885v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, workflow, human-in-the-loop, robustness - Published: 2026-09-15 · Authors: Kratika Bhagtani, Kusha Sridhar, Maziyar Baran Pouyan, Yuying Zhao, Eugene Siow ### EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks Introduces EVA, a red-teaming framework that evolves semantic payloads for GUI environmental-injection tests. It studies the contribution of semantic versus visual deception and distills discovered patterns into reusable evaluation rules. - https://awesomecua.com/resources/arxiv-2505-14289/ · source: https://arxiv.org/abs/2505.14289v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, evaluation - Published: 2025-05-20 · Authors: Yijie Lu, Manman Zhao, Tianjie Ju, Zihe Yan, Xinbei Ma, Yuan Guo, Daizong Ding, Gongshen Liu, Zhuosheng Zhang ### Evaluating Cultural and Social Awareness of LLM Web Agents Introduces CASA to assess web agents' responses to cultural and social norms in shopping and forum tasks, including misleading observations. - https://awesomecua.com/resources/arxiv-2410-23252/ · source: https://arxiv.org/abs/2410.23252v3 - Paper · reviewed 2026-10-16 · environments: web · topics: safety-security, evaluation - Published: 2024-10-30 · Authors: Haoyi Qiu, Alexander R. Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, Chien-Sheng Wu ### Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks Examines active environmental injection against mobile agents, testing whether adversarial interface content redirects decisions during task execution. - https://awesomecua.com/resources/arxiv-2502-13053/ · source: https://arxiv.org/abs/2502.13053v3 - Paper · reviewed 2026-10-16 · environments: android · topics: safety-security, robustness, evaluation - Published: 2025-02-18 · Authors: Yurun Chen, Xavier Hu, Keting Yin, Juncheng Li, Shengyu Zhang ### EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents EvoCUA-1.5 trains multi-turn desktop agents through online reinforcement learning in executable sandboxes. It combines step-level policy optimization, task filtering and calibration, an adaptive curriculum, and asynchronous rollout handling with staleness control. - https://awesomecua.com/resources/arxiv-2607-09773/ · source: https://arxiv.org/abs/2607.09773v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: reinforcement-learning, long-horizon, sandbox, data-generation - Published: 2026-07-07 · Authors: Mianqiu Huang, Taofeng Xue, Chong Peng, Jinrui Ding, Jie Yang, Sicheng Fan, Jiale Hong, Yufei Gao, Xiaocheng Zhang, Linsen Guo, Xin Yang, Dengchang Zhao, Yuchen Xie, Peng Pei, Xunliang Xie, Xipeng Qiu ### EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience EvoCUA combines task synthesis with executable validators, asynchronous sandbox rollouts and iterative policy updates. It turns successful routines and analyzed failures into supervision and evaluates the evolving agent on OSWorld. - https://awesomecua.com/resources/arxiv-2601-15876/ · source: https://arxiv.org/abs/2601.15876v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: data-generation, sandbox, long-horizon, self-reflection - Published: 2026-01-22 · Authors: see the paper ### EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding Introduces EvoGUI, which derives temporal ordering, inverse action/value prediction, and successor-discrimination probes from recorded GUI trajectories. Its web-derived benchmark isolates transition understanding from the other abilities involved in end-to-end task success. - https://awesomecua.com/resources/arxiv-2607-17050/ · source: https://arxiv.org/abs/2607.17050v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, world-models, action-prediction - Published: 2026-07-19 · Authors: Yaohan Yang, Minglei Shi, Borui Zhang, Jie Zhou, Jiwen Lu ### Evolving Agent Interfaces Agent S4 uses task trajectories to evolve interfaces and usage guidance for a computer-use agent, selecting changes through validation tasks. - https://awesomecua.com/resources/arxiv-2610-28601/ · source: https://arxiv.org/abs/2610.28601v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: tool-integration, workflow, evaluation - Published: 2026-10-20 · Authors: Saaket Agashe, Yuzhe Yang, Gonzalo Gonzalez-Pumariega, Bhuvan Chandra Koduru, Chih-Lun Li, Jiachen Yang, Ang Li, Xin Eric Wang ### Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agent The Self-Evolution Agent combines automatic generation of verifiable computer-use trajectories with step-wise reinforcement learning for long-horizon training. It also describes a model-enhancement procedure that integrates grounding and planning capabilities without an additional training stage. - https://awesomecua.com/resources/arxiv-2508-04037/ · source: https://arxiv.org/abs/2508.04037v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, reinforcement-learning, grounding, planning - Published: 2025-08-06 · Authors: Yuhao Cheng, Liang Tang, Shuxian Li, Yukang Huo, Tiaonan Duan, Kaer Huang, Yanzhe Jing, Yiqiang Yan ### EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications Introduces EvoWeb, a benchmark for coding agents that extend web applications through dependent requests; a multimodal web agent checks each version in a live browser. - https://awesomecua.com/resources/arxiv-2610-16402/ · source: https://arxiv.org/abs/2610.16402v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, long-horizon, workflow - Published: 2026-10-12 · Authors: David Nukrai, Bar Alon, Nativ Levy, Ido Andrew Atad, Ido Cohen, Itamar Zimerman, Lior Wolf, Eli David, Ilan Naiman ### ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning ExACT combines reflective tree search with learning from search experience to help web agents explore, assess states and backtrack on VisualWebArena. - https://awesomecua.com/resources/arxiv-2410-02052/ · source: https://arxiv.org/abs/2410.02052v5 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, self-reflection, interaction-history, web-navigation - Published: 2024-10-02 · Authors: Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, Zhou Yu ### Executable Agentic Memory for GUI Agent Executable Agentic Memory stores GUI routines in a knowledge graph built through state-aware exploration and action-group mining. A learned value model guides tree search over this executable memory for retrieval-based planning, with AndroidWorld and efficiency evaluation. - https://awesomecua.com/resources/arxiv-2605-12294/ · source: https://arxiv.org/abs/2605.12294v1 - Paper · reviewed 2026-10-18 · environments: android · topics: memory, planning, long-horizon, efficiency - Published: 2026-05-12 · Authors: Zerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu, Ju Ren ### Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation Introduces MobileGPT, which organizes mobile-app experience into reusable subtasks using an explore, select, derive, and recall cycle. Its memory supports learning task procedures and adapting them to new contexts. - https://awesomecua.com/resources/arxiv-2312-03003/ · source: https://arxiv.org/abs/2312.03003v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, memory, planning - Published: 2023-12-04 · Authors: Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y. Ko, Sangeun Oh, Insik Shin ### Explorer: Robust Collection of Interactable GUI Elements Explorer collects interactable GUI elements from a live target application and records user sessions for application-specific model training. It links repeated interface states into a navigation map that supports path planning and voice-directed interaction. - https://awesomecua.com/resources/arxiv-2504-09352/ · source: https://arxiv.org/abs/2504.09352v1 - Paper · reviewed 2026-10-17 · environments: android, web · topics: data-generation, recording, grounding, planning - Published: 2025-04-12 · Authors: Iason Chaimalas, Arnas Vyšniauskas, Gabriel Brostow ### Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents Synthesizes multimodal web trajectories through exploration and refinement, then trains a browser agent to study the effect of scaling interaction data. - https://awesomecua.com/resources/arxiv-2502-11357/ · source: https://arxiv.org/abs/2502.11357v4 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation - Published: 2025-02-17 · Authors: Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, Ahmed Awadallah ### Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualization This study compares specialized, computer-use and coding agents on scientific visualization workflows. It examines GUI, CLI, code and tool-call interaction modes, including how persistent memory changes performance across repeated trials. - https://awesomecua.com/resources/arxiv-2604-27996/ · source: https://arxiv.org/abs/2604.27996v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: science, evaluation, tool-integration, memory - Published: 2026-04-30 · Authors: Jackson Vonderhorst, Kuangshi Ai, Haichao Miao, Shusen Liu, Chaoli Wang ### Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web CompWoB adds 50 composed web-automation tasks to study generalization from simpler interactions. The paper compares prompted and fine-tuned agents, balances task training data, and probes sensitivity to changes in instruction order. - https://awesomecua.com/resources/arxiv-2311-18751/ · source: https://arxiv.org/abs/2311.18751v3 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, long-horizon, robustness, web-navigation - Published: 2023-11-30 · Authors: Hiroki Furuta, Yutaka Matsuo, Aleksandra Faust, Izzeddin Gur ### ExpSeek: Self-Triggered Experience Seeking for Web Agents ExpSeek uses step-level entropy signals to decide when a web agent should request experience-based guidance. It supplies guidance tailored to the current step and evaluates the intervention across web-agent benchmarks. - https://awesomecua.com/resources/arxiv-2601-08605/ · source: https://arxiv.org/abs/2601.08605v2 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, planning, web-navigation, evaluation - Published: 2026-01-13 · Authors: Wenyuan Zhang, Xinghua Zhang, Haiyang Yu, Shuaiyi Nie, Bingli Wu, Juwei Yue, Tingwen Liu, Yongbin Li ### Faithful Mobile GUI Agents with Guided Advantage Estimator Introduces Faithful-Agent, which first teaches abstention under perturbed evidence and then refines GUI policies with an anchor-based, variance-adaptive advantage estimator. A thought-action consistency reward encourages decisions grounded in displayed evidence and instructions. - https://awesomecua.com/resources/arxiv-2605-01208/ · source: https://arxiv.org/abs/2605.01208v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reinforcement-learning, robustness, mobile-navigation - Published: 2026-05-02 · Authors: Haowen Hu, Pengzhou Cheng, Zheng Wu, Lingzhong Dong, Gongshen Liu, Zhuosheng Zhang ### Falcon-UI: Understanding GUI Before Following User Instructions Pretrains GUI-context understanding with instruction-free web interactions before fine-tuning Falcon-UI on instructed GUI tasks. - https://awesomecua.com/resources/arxiv-2412-09362/ · source: https://arxiv.org/abs/2412.09362v1 - Paper · reviewed 2026-10-16 · environments: web, android, cross-platform · topics: screen-understanding, data-generation - Published: 2024-12-12 · Authors: Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, Xiangyang Ji ### Falsifiable Commitment Planning for Self-Correcting Web Agents Introduces FCPAgent, which attaches confirming and falsifying evidence to each skill-grounded plan step. A plan–test–repair loop checks actions and observations, then localizes contradictions so execution, skills, or plans can be revised at the appropriate level. - https://awesomecua.com/resources/arxiv-2607-24167/ · source: https://arxiv.org/abs/2607.24167v1 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, robustness, self-reflection - Published: 2026-07-27 · Authors: Guangyi Liu, Huan Zhao, Quanming Yao ### Fara-1.5: Scalable Learning Environments for Computer Use Agents Builds browser-agent training trajectories with live and synthetic environments, solvers, user simulation, and complementary correctness and efficiency verifiers. - https://awesomecua.com/resources/arxiv-2606-20785/ · source: https://arxiv.org/abs/2606.20785v2 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation, evaluation - Published: 2026-06-18 · Authors: Ahmed Awadallah, Sahil Gupta, Yash Lara, Yadong Lu, Hussein Mozannar, Akshay Nambi, Zach Nussbaum, Yash Pandya, Aravind Rajeswaran, Corby Rosset, Alexey Taymanov, Luiz do Valle, Vibhav Vineet, Spencer Whitehead, Andrew Zhao ### Fara-7B: An Efficient Agentic Model for Computer Use Introduces FaraGen synthetic web trajectories, a compact screenshot-based action model, and WebTailBench for underrepresented browser tasks. - https://awesomecua.com/resources/arxiv-2511-19663/ · source: https://arxiv.org/abs/2511.19663v1 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, efficiency, web-navigation - Published: 2025-11-24 · Authors: Ahmed Awadallah, Yash Lara, Raghav Magazine, Hussein Mozannar, Akshay Nambi, Yash Pandya, Aravind Rajeswaran, Corby Rosset, Alexey Taymanov, Vibhav Vineet, Spencer Whitehead, Andrew Zhao ### FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems Introduces FedGUI, a federated GUI-learning benchmark built from six datasets spanning mobile, web and desktop interfaces. Its experiments separate platform, device, operating-system and data-source heterogeneity when evaluating collaborative training. - https://awesomecua.com/resources/arxiv-2604-14956/ · source: https://arxiv.org/abs/2604.14956v1 - Paper · reviewed 2026-10-17 · environments: web, desktop, cross-platform · topics: evaluation, robustness - Published: 2026-04-16 · Authors: Wenhao Wang, Haoting Shi, Mengying Yuan, Yiquan Lin, Panrong Tong, Hanzhang Zhou, Guangyi Liu, Pengxiang Zhao, Yue Wang, Siheng Chen ### FedMABench: Benchmarking Mobile Agents on Decentralized Heterogeneous User Data Introduces a benchmark for federated mobile-agent training under heterogeneous user and application distributions, combining datasets, algorithms, and base models. - https://awesomecua.com/resources/arxiv-2503-05143/ · source: https://arxiv.org/abs/2503.05143v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, mobile-navigation - Published: 2025-03-07 · Authors: Wenhao Wang, Zijie Yu, Rui Ye, Jianqing Zhang, Siheng Chen, Yanfeng Wang ### Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms Extends Ferret-UI across device platforms with adaptive visual scaling and synthetic data for referring, grounding, and user-centered UI tasks. - https://awesomecua.com/resources/arxiv-2410-18967/ · source: https://arxiv.org/abs/2410.18967v2 - Paper · reviewed 2026-10-16 · environments: ios, android, web, cross-platform · topics: screen-understanding, grounding - Published: 2024-10-24 · Authors: Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, Zhe Gan ### Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents Ferret-UI Lite develops a 3B GUI agent using mixed real and synthetic data, reasoning with visual tools at inference, and reinforcement learning. It evaluates grounding and navigation across mobile, web, and desktop settings, including AndroidWorld and OSWorld. - https://awesomecua.com/resources/arxiv-2509-26539/ · source: https://arxiv.org/abs/2509.26539v1 - Paper · reviewed 2026-10-17 · environments: android, web, desktop, cross-platform · topics: grounding, reinforcement-learning, tool-integration, efficiency - Published: 2025-09-30 · Authors: Zhen Yang, Zi-Yi Dou, Di Feng, Forrest Huang, Anh Nguyen, Keen You, Omar Attia, Yuhao Yang, Michael Feng, Haotian Zhang, Ram Ramrakhya, Chao Jia, Jeffrey Nichols, Alexander Toshev, Yinfei Yang, Zhe Gan ### Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs Adapts multimodal modeling to mobile interfaces through detailed screen crops, region annotations, and UI reasoning tasks. - https://awesomecua.com/resources/arxiv-2404-05719/ · source: https://arxiv.org/abs/2404.05719v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: screen-understanding, grounding - Published: 2024-04-08 · Authors: Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, Zhe Gan ### Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing Introduces QTypist, an LLM-based method that generates text inputs from mobile GUI context and integrates with automated testing tools to help them reach further app pages. - https://awesomecua.com/resources/arxiv-2212-04732/ · source: https://arxiv.org/abs/2212.04732v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, automation, screen-understanding - Published: 2022-12-09 · Authors: Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, Qing Wang ### FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use? Introduces FinCUABuildBench for assessing financial computer-use task construction and FinCUABuildAgent for generating tasks, environments, and validators. - https://awesomecua.com/resources/arxiv-2609-07603/ · source: https://arxiv.org/abs/2609.07603v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, data-generation, industry - Published: 2026-09-07 · Authors: Jingpu Yang, Fengxian Ji, Jinri Guo, Tianhao Li, Qian Jiang, Fan Zhang, Min Peng, Qianqian Xie, Preslav Nakov, Zhuohan Xie ### FineState-Bench: A Comprehensive Benchmark for Fine-Grained State Control in GUI Agents FineState-Bench evaluates fine-grained GUI perception and control across desktop, web, and mobile interfaces. Its Visual Diagnostic Assistant separates perception and localization contributions in controlled interaction experiments. - https://awesomecua.com/resources/arxiv-2508-09241/ · source: https://arxiv.org/abs/2508.09241v1 - Paper · reviewed 2026-10-17 · environments: desktop, web, cross-platform · topics: evaluation, grounding, observability, screen-understanding - Published: 2025-08-12 · Authors: Fengxian Ji, Jingpu Yang, Zirui Song, Yuanxi Wang, Zhexuan Cui, Yuke Li, Qian Jiang, Miao Fang, Xiuying Chen ### FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting FineState-Bench tests whether GUI agents locate the intended control and set its exact state across desktop, web, and mobile interfaces, with stage-wise diagnostic metrics. - https://awesomecua.com/resources/arxiv-2604-27974/ · source: https://arxiv.org/abs/2604.27974v1 - Paper · reviewed 2026-11-06 · environments: desktop, web, cross-platform · topics: evaluation, grounding, state-tracking - Published: 2026-04-30 · Authors: Fengxian Ji, Jingpu Yang, Zirui Song, Yuanxi Wang, Zhexuan Cui, Yuke Li, Qian Jiang, Xiuying Chen ### FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents FingerTip 20K gathers 20K human Android demonstrations from users' long-term everyday app use, with user context, to evaluate proactive task suggestion and preference-aware task execution. - https://awesomecua.com/resources/arxiv-2507-21071/ · source: https://arxiv.org/abs/2507.21071v2 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, data-generation, mobile-navigation, interaction-history - Published: 2025-06-09 · Authors: Qinglong Yang, Haoming Li, Haotian Zhao, Xiaokai Yan, Jingtao Ding, Fengli Xu, Yong Li ### FLAME: A small language model for spreadsheet formulas Introduces FLAME, a model trained on Excel formulas for spreadsheet formula repair, completion and retrieval, supporting formula authoring in spreadsheet software. - https://awesomecua.com/resources/arxiv-2301-13779/ · source: https://arxiv.org/abs/2301.13779v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, data-generation - Published: 2023-01-31 · Authors: Harshit Joshi, Abishai Ebenezer, José Cambronero, Sumit Gulwani, Aditya Kanade, Vu Le, Ivan Radiček, Gust Verbruggen ### Flash-Searcher: Fast and Effective Web Agents via DAG-Based Parallel Execution Introduces Flash-Searcher, a parallel agent framework that organizes dependent subtasks as a graph and runs independent paths concurrently; it is evaluated on deep-search benchmarks. - https://awesomecua.com/resources/arxiv-2509-25301/ · source: https://arxiv.org/abs/2509.25301v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, planning, workflow - Published: 2025-09-29 · Authors: Tianrui Qin, Qianben Chen, Sinuo Wang, He Xing, King Zhu, He Zhu, Dingfeng Shi, Xinxin Liu, Ge Zhang, Jiaheng Liu, Yuchen Eleanor Jiang, Xitong Gao, Wangchunshu Zhou ### FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games FlashAdventure evaluates GUI agents on completing story arcs in 34 adventure games. It pairs an automated gameplay judge with COAST, a framework that keeps long-term clue memory to plan and complete sequential objectives. - https://awesomecua.com/resources/arxiv-2509-01052/ · source: https://arxiv.org/abs/2509.01052v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, long-horizon, memory, planning - Published: 2025-09-01 · Authors: Jaewoo Ahn, Junseo Kim, Heeseung Yun, Jaehyeon Son, Dongmin Park, Jaewoong Cho, Gunhee Kim ### FlowMind: Automatic Workflow Generation with LLMs FlowMind generates API-based workflows from user requests for process automation and presents workflow descriptions that users can inspect and comment on. - https://awesomecua.com/resources/arxiv-2404-13050/ · source: https://arxiv.org/abs/2404.13050v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: workflow, automation, human-in-the-loop - Published: 2024-03-17 · Authors: Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, Manuela Veloso ### FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agents Introduces FocusAgent, which uses a lightweight LLM to select task-relevant lines from web agents' accessibility-tree observations, reducing context size and exposure to prompt injection. - https://awesomecua.com/resources/arxiv-2510-03204/ · source: https://arxiv.org/abs/2510.03204v2 - Paper · reviewed 2026-11-06 · environments: web · topics: efficiency, safety-security, screen-understanding - Published: 2025-10-03 · Authors: Imene Kerboua, Sahar Omidi Shayegan, Megh Thakkar, Xing Han Lù, Léo Boisvert, Massimo Caccia, Jérémy Espinas, Alexandre Aussem, Véronique Eglin, Alexandre Lacoste ### Focused Omni-Modal GUI Agents FocusGUI learns step-specific weights over screenshot, audio, video and interaction-context channels, injecting a compact residual into a frozen decoder without pruning its inputs. - https://awesomecua.com/resources/arxiv-2610-26203/ · source: https://arxiv.org/abs/2610.26203v1 - Paper · reviewed 2026-10-20 · environments: android · topics: screen-understanding, grounding, imitation-learning, interaction-history, mobile-navigation - Published: 2026-10-19 · Authors: Fuyao Zhang, Wenwen He, Jiaming Zhang, Yurong Hao, Xuan Wang, Guowei Guan, Che Wang, Longtao Huang, Wei Yang Bryan Lim ### FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory Introduces FocusMem, a latent memory interface for GUI agents that separates stored content, state-conditioned readout, and a gate for irrelevant retrieved experience. - https://awesomecua.com/resources/arxiv-2608-04530/ · source: https://arxiv.org/abs/2608.04530v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, interaction-history, state-tracking - Published: 2026-08-05 · Authors: Zhuoran Zhang, Bowen Li, Jingcheng Ju, Yang Shi, Qixun Wang, Haotian Wang, Wei Chen, Tengjiao Wang ### FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection FocusUI selects instruction-relevant screenshot patches while preserving their positional structure for GUI grounding. Its PosPad scheme replaces runs of dropped tokens with positional markers and is evaluated under reduced token budgets. - https://awesomecua.com/resources/arxiv-2601-03928/ · source: https://arxiv.org/abs/2601.03928v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, efficiency, screen-understanding - Published: 2026-01-07 · Authors: Mingyu Ouyang, Kevin Qinghong Lin, Mike Zheng Shou, Hwee Tou Ng ### FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents Introduces FormFactory, an interactive web-based benchmark for agents filling online forms, with varied field formats and evaluation of instruction-to-field alignment. - https://awesomecua.com/resources/arxiv-2506-01520/ · source: https://arxiv.org/abs/2506.01520v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, screen-understanding - Published: 2025-06-02 · Authors: Bobo Li, Yuheng Wang, Hao Fei, Juncheng Li, Wei Ji, Mong-Li Lee, Wynne Hsu ### FormGym: Doing Paperwork with Agents Introduces FormGym, a benchmark for filling image-only forms, and FieldFinder, a tool that helps LLMs locate where to place text on those forms. - https://awesomecua.com/resources/arxiv-2506-14079/ · source: https://arxiv.org/abs/2506.14079v4 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, screen-understanding - Published: 2025-06-17 · Authors: Matthew Toles, Rattandeep Singh, Isaac Song, Zhou Yu ### FraudSMSWalker: Benchmarking Agentic Large Language Models for SMS-to-Webpage Fraud Detection FraudSMSWalker tests browser agents on bilingual SMS-to-webpage fraud judgments while hiding URL and reputation shortcuts. Its controlled chains include difficult benign login/payment cases and evaluate whether classifications remain supported by the visible message and page evidence. - https://awesomecua.com/resources/arxiv-2606-16659/ · source: https://arxiv.org/abs/2606.16659v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, robustness - Published: 2026-06-15 · Authors: Y. H. Zhou, Z. M. Ma, Y. J. Zhou, Y. T. Li, H. X. Xiang, Y. M. Cheng, T. L. Chen, K. J. Zhang, Z. H. Nan, J. H. Ni, Z. Wu, Q. Y. Pan, S. Zhang, S. Cheng, M. Y. Luo ### From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents Analyzes security vulnerabilities across OEM system-level, third-party, and research mobile LLM agents, covering GUI and cross-application attack surfaces. - https://awesomecua.com/resources/arxiv-2505-12981/ · source: https://arxiv.org/abs/2505.12981v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, mobile-navigation, evaluation - Published: 2025-05-19 · Authors: Liangxuan Wu, Chao Wang, Tianming Liu, Yanjie Zhao, Haoyu Wang ### From Blind Edits to Verified Repair: Building Trustworthy User-Side LLM Agents for Web Accessibility Builds a local browser-extension agent that generates and verifies additive CSS repairs for WCAG web accessibility violations. - https://awesomecua.com/resources/arxiv-2608-24913/ · source: https://arxiv.org/abs/2608.24913v1 - Paper · reviewed 2026-10-18 · environments: web · topics: ui-design, web-navigation, safety-security - Published: 2026-07-26 · Authors: Lily Bundgaard Wanscher, Markus Heidemann Lorensen, Mohammed Ammad Shafiq, Mahyar Tourchi Moghaddam, Mina Alipour ### From Context to Action: Analysis of the Impact of State Representation and Context on the Generalization of Multi-Turn Web Navigation Agents Analyzes how interaction history and webpage representation affect multiturn web-agent generalization across unfamiliar websites, task categories, and geographic settings. - https://awesomecua.com/resources/arxiv-2410-23555/ · source: https://arxiv.org/abs/2410.23555v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, interaction-history, state-tracking - Published: 2024-10-31 · Authors: Nalin Tiwary, Vardhan Dongre, Sanil Arun Chawla, Ashwin Lamani, Dilek Hakkani-Tür ### From Grounding to Planning: Benchmarking Bottlenecks in Web Agents Separates web-agent planning and grounding evaluations on Mind2Web to analyze which components limit task performance. - https://awesomecua.com/resources/arxiv-2409-01927/ · source: https://arxiv.org/abs/2409.01927v1 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, grounding, planning - Published: 2024-09-03 · Authors: Segev Shlomov, Ben wiesel, Aviad Sela, Ido Levy, Liane Galanti, Roy Abitbol ### From Imitation to Discrimination: Progressive Curriculum Learning for Robust Web Navigation Introduces the Triton dataset and a staged training curriculum for text-based web navigation agents, using hard-negative HTML examples to train element discrimination and longer task execution. - https://awesomecua.com/resources/arxiv-2604-12666/ · source: https://arxiv.org/abs/2604.12666v1 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, web-navigation, imitation-learning, reinforcement-learning - Published: 2026-04-14 · Authors: Chuang Peng, Wei Zhang, Renshuai Tao, Xinhao Zhang, Jian Yang ### From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents Introduces a Declarative Model Interface that maps existing GUIs to access, state and observation primitives while handling low-level interactions. Evaluation uses Word, PowerPoint and Excel on Windows with a GUI-agent baseline. - https://awesomecua.com/resources/arxiv-2510-04607/ · source: https://arxiv.org/abs/2510.04607v2 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: automation, tool-integration, planning - Published: 2025-10-06 · Authors: Yuan Wang, Mingyu Li, Haibo Chen ### From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents Introduces Task Advancement Memory (TAM) for long GUI workflows, tracking task progress across applications and learning reusable operations from interactions with local results. - https://awesomecua.com/resources/arxiv-2610-37482/ · source: https://arxiv.org/abs/2610.37482v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, state-tracking, cross-app, long-horizon - Published: 2026-10-27 · Authors: Ziyang Chen, Qian Li, Chengwei Qin ### From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts Develops a taxonomy of mobile-action consequences and annotated interaction data to evaluate whether models recognize impactful or risky UI operations. - https://awesomecua.com/resources/arxiv-2410-09006/ · source: https://arxiv.org/abs/2410.09006v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: safety-security, mobile-navigation, evaluation - Published: 2024-10-11 · Authors: Zhuohao Jerry Zhang, Eldon Schoop, Jeffrey Nichols, Anuj Mahajan, Amanda Swearngin ### From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents An online skill-evolution framework turns GUI interaction traces and evaluator feedback into a versioned library of reusable procedures without updating model parameters. Controlled OSWorld comparisons examine both the benefits of shared procedural memory and its domain-dependent instability under repeated revision. - https://awesomecua.com/resources/arxiv-2609-04869/ · source: https://arxiv.org/abs/2609.04869v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: memory, workflow, evaluation, training-free - Published: 2026-09-04 · Authors: Longtao Hu, Xiao Liang, Linchao Zhu ### From Local Skills to Long-Horizon Web Tasks: Progressive Skill Exploration for LLM Web Agents Introduces SCOPE, a training method for long web-agent tasks that discovers local skills from rollouts, composes them into chains, and expands them using failure experience. - https://awesomecua.com/resources/arxiv-2611-03116/ · source: https://arxiv.org/abs/2611.03116v1 - Paper · reviewed 2026-11-06 · environments: web · topics: long-horizon, planning, interaction-history, web-navigation - Published: 2026-11-02 · Authors: Xueyang Feng, Xiaohe Bo, Xu Chen, Quanyu Dai, Feng Wen ### From Monoliths to Swarms: A Study of Attack Surface Evolution in the Transition to Multi-Agent Web Systems Categorizes structural attack surfaces that emerge when single-agent browser systems are decomposed into multi-agent web workflows. - https://awesomecua.com/resources/arxiv-2608-00202/ · source: https://arxiv.org/abs/2608.00202v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, web-navigation, planning - Published: 2026-07-31 · Authors: Yashaswi Malla, Sandra Siby ### From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning Introduces StepAgent, which compares agent and expert actions to supply intermediate rewards for policy learning, with implicit-reward and inverse-RL variants. Evaluation includes WebShop and Mind2Web alongside non-web tasks. - https://awesomecua.com/resources/arxiv-2411-03817/ · source: https://arxiv.org/abs/2411.03817v3 - Paper · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, reward-models, self-reflection - Published: 2024-11-06 · Authors: Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, Weipeng Chen ### From Observations to Authorized Actions: Measuring Source Information in Android Agents Measures source-attribution limits in Android accessibility and separately tests application-history and selected-WebView-object checks for authorizing AppFunctions actions. - https://awesomecua.com/resources/arxiv-2610-15502/ · source: https://arxiv.org/abs/2610.15502v1 - Paper · reviewed 2026-10-20 · environments: android, web · topics: safety-security, tool-integration, observability - Published: 2026-10-11 · Authors: Hyobin Ahn, Dongjae Lee ### From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation BEPA adapts a small set of expert GUI traces into policy-aligned guidance through trajectories reachable by the learner and a dynamically updated task cache. It evaluates this two-level approach to reinforcement learning on desktop and web benchmarks. - https://awesomecua.com/resources/arxiv-2601-05787/ · source: https://arxiv.org/abs/2601.05787v2 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: reinforcement-learning, data-generation, evaluation - Published: 2026-01-09 · Authors: Zezhou Wang, Ziyun Zhang, Xiaoyi Zhang, Zhuzhong Qian, Yan Lu ### From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces Studies instruction-following agents that observe screenshots and act through a generic mouse-and-keyboard action space, evaluating pixel-based pretraining for GUI tasks on MiniWoB++. - https://awesomecua.com/resources/arxiv-2306-00245/ · source: https://arxiv.org/abs/2306.00245v2 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, action-prediction, web-navigation - Published: 2023-05-31 · Authors: Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, Kristina Toutanova ### From Seeing to Reporting: A Theory of Reliable Visual Set Generation in Vision-Language Models FSR models visual reporting and gives inference-time procedures for frozen vision-language models, including screen parsing and GUI target selection. - https://awesomecua.com/resources/arxiv-2610-25503/ · source: https://arxiv.org/abs/2610.25503v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, grounding, training-free - Published: 2026-10-18 · Authors: Junzhe Shi, Yuan Gan, Shida Jiang ### From Struggle to Success: Context-Aware Guidance for Screen Reader Users in Computer Use Introduces AskEase, an on-demand assistant that combines contextual information to provide screen-reader-friendly computer-use guidance. A study with screen-reader users evaluates task completion and perceived workload while preserving human control of the interaction. - https://awesomecua.com/resources/arxiv-2601-18092/ · source: https://arxiv.org/abs/2601.18092v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, evaluation, workflow - Published: 2026-01-26 · Authors: Nan Chen, Jing Lu, Zilong Wang, Luna K. Qiu, Siming Chen, Yuqing Yang ### From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks GUI-Hopper combines verified Android deeplinks with GUI actions, using landing-screen descriptions, hybrid trajectory training and reinforcement learning to select navigation shortcuts and recover through GUI control. - https://awesomecua.com/resources/arxiv-2609-30887/ · source: https://arxiv.org/abs/2609.30887v1 - Paper · reviewed 2026-10-20 · environments: android · topics: mobile-navigation, tool-integration, data-generation, imitation-learning, reinforcement-learning, efficiency - Published: 2026-09-25 · Authors: Yuchen Sun, Chenglin Cai, Gongjie Zhang, Tianyu Xia, Quyu Kong, Panrong Tong, Zhengwen Zeng, Long Chen, Steven Hoi, Chongyang Zhang, Yue Wang ### From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs Introduces SURFPRM, which turns web demonstrations into an interaction-element graph for synthesizing grounded negative actions. Its contrastive preference data trains process reward models to distinguish plausible on-page alternatives and guide downstream trajectory search. - https://awesomecua.com/resources/arxiv-2609-32351/ · source: https://arxiv.org/abs/2609.32351v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, reward-models, web-navigation - Published: 2026-09-26 · Authors: Yangzhe Peng, Xiaoyang Wang, Yiyang Zhao, Lijun Wu, Kun He ### FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback FronTalk benchmarks multi-turn front-end code generation from text and visual feedback; a web agent explores generated sites for evaluation and in AceCoder. - https://awesomecua.com/resources/arxiv-2601-04203/ · source: https://arxiv.org/abs/2601.04203v3 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, evaluation, web-navigation - Published: 2025-12-05 · Authors: Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen ### fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets Introduces fxBench, a benchmark of spreadsheet formula suggestions, and examines context selection, intent understanding and formula writing for editing assistance. - https://awesomecua.com/resources/arxiv-2610-24729/ · source: https://arxiv.org/abs/2610.24729v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, screen-understanding, efficiency - Published: 2026-10-17 · Authors: Sanket Mhatre, Sumit Gulwani, Vu Le, Yasharth Bajpai, Gust Verbruggen ### GAIA: a benchmark for General AI Assistants Introduces questions that test assistant reasoning, multimodal understanding, browsing, and tool use beyond screen-action execution alone. - https://awesomecua.com/resources/arxiv-2311-12983/ · source: https://arxiv.org/abs/2311.12983v1 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, tool-integration - Published: 2023-11-21 · Authors: Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom ### GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models GAIA trains a GUI action critic on positive and negative examples, then uses that critic to collect improved examples for another training round. It studies how recycling these data affects the test-time action choices of different GUI agents. - https://awesomecua.com/resources/arxiv-2601-18197/ · source: https://arxiv.org/abs/2601.18197v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reward-models, data-generation, evaluation - Published: 2026-01-26 · Authors: Shaokang Wang, Pei Fu, Ruoceng Zhang, Shaojie Zhang, Xiuwen Xi, Jiahui Yang, Bin Qin, Ying Huang, Zhenbo Luo, Jian Luan ### GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection Introduces GAIR, which combines information from specialized GUI models through a general-purpose coordinating model. When the available information is insufficient, group reflection sends targeted requests back to the specialists before selecting an operation. - https://awesomecua.com/resources/arxiv-2512-09396/ · source: https://arxiv.org/abs/2512.09396v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: planning, self-reflection, automation - Published: 2025-12-10 · Authors: Zishu Wei, Qixiang Ma, Xavier Hu, Yuhang Liu, Hui Zang, Yudong Zhao, Tao Wang, Shengyu Zhang, Fei Wu ### GameWAM: A World Action Model for Video Games Introduces GameWAM, a model for native game play and GUI control that predicts future visuals and keyboard-mouse actions from synchronized gameplay and GUI trajectories. - https://awesomecua.com/resources/arxiv-2608-26200/ · source: https://arxiv.org/abs/2608.26200v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: world-models, action-prediction, video, long-horizon - Published: 2026-08-25 · Authors: Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Weijia Li ### GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents Introduces GameWorld, a browser-game benchmark that evaluates multimodal agents through keyboard and mouse controls or semantic actions using outcome checks. - https://awesomecua.com/resources/arxiv-2604-07429/ · source: https://arxiv.org/abs/2604.07429v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, long-horizon, action-prediction - Published: 2026-04-08 · Authors: Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, Mike Zheng Shou ### GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents GEM detects instructions outside a GUI agent’s capabilities by modeling distances in its input-embedding space with a Gaussian mixture. Experiments across mobile, computer and browser data also examine selective escalation to a cloud model when an input is detected as unfamiliar. - https://awesomecua.com/resources/arxiv-2505-12842/ · source: https://arxiv.org/abs/2505.12842v4 - Paper · reviewed 2026-10-17 · environments: desktop, web, cross-platform · topics: robustness, safety-security, evaluation, efficiency - Published: 2025-05-19 · Authors: Zheng Wu, Pengzhou Cheng, Zongru Wu, Lingzhong Dong, Zhuosheng Zhang ### General Agentic Planning Through Simulative Reasoning with World Models Introduces SiRA, an agent architecture that uses a language-based world model to simulate action outcomes for planning in web-browser tasks. - https://awesomecua.com/resources/arxiv-2507-23773/ · source: https://arxiv.org/abs/2507.23773v3 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, world-models, web-navigation - Published: 2025-07-31 · Authors: Mingkai Deng, Jinyu Hou, Zhiting Hu, Eric Xing ### Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding GMS pairs a general vision-language scanner that proposes interface regions with a specialist locator that predicts coordinates within them. Hierarchical coarse-to-fine search coordinates the two components for GUI grounding on ScreenSpot-Pro. - https://awesomecua.com/resources/arxiv-2509-24133/ · source: https://arxiv.org/abs/2509.24133v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, planning, evaluation - Published: 2025-09-29 · Authors: Zhecheng Li, Guoxian Song, Yiwei Wang, Zhen Xiong, Junsong Yuan, Yujun Cai ### Generalization in Online Reinforcement Learning for Mobile Agents Introduces AndroidWorld-Generalization with separate tests for unseen task instances, templates, and applications. A GRPO training system combines containerized environments with asynchronous trajectory collection to study these generalization regimes. - https://awesomecua.com/resources/arxiv-2603-07432/ · source: https://arxiv.org/abs/2603.07432v1 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, evaluation, robustness - Published: 2026-03-08 · Authors: Li Gu, Zihuan Jiang, Zhixiang Chi, Huan Liu, Ziqiang Wang, Yuanhao Yu, Glen Berseth, Yang Wang ### Generative Visual Code Mobile World Models gWorld predicts future mobile GUI states as renderable web code and supplies an automated pipeline for code-based training data. The study evaluates visual state prediction and how stronger world models affect downstream mobile policies. - https://awesomecua.com/resources/arxiv-2602-01576/ · source: https://arxiv.org/abs/2602.01576v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: world-models, data-generation, screen-understanding, mobile-navigation - Published: 2026-02-02 · Authors: Woosung Koh, Sungjun Han, Segyu Lee, Se-Young Yun, Jamin Shin ### Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teaming Introduces Genesis, a red-teaming framework that generates adversarial injections for web agents and revises attack strategies using responses and interaction logs. - https://awesomecua.com/resources/arxiv-2510-18314/ · source: https://arxiv.org/abs/2510.18314v2 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, interaction-history - Published: 2025-10-21 · Authors: Zheng Zhang, Jiarui He, Yuchen Cai, Deheng Ye, Peilin Zhao, Ruili Feng, Hao Wang ### GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments? Introduces GhostEI-Bench, a benchmark that injects deceptive UI events into Android emulator workflows to assess mobile agents and analyze failures from actions and screenshots. - https://awesomecua.com/resources/arxiv-2510-20333/ · source: https://arxiv.org/abs/2510.20333v3 - Paper · reviewed 2026-11-06 · environments: android · topics: evaluation, safety-security, sandbox, observability - Published: 2025-10-23 · Authors: Chiyu Chen, Xinhao Song, Yunkai Chai, Yang Yao, Haodong Zhao, Lijun Li, Jie Li, Yan Teng, Gongshen Liu, Yingchun Wang ### Glasseek: A Transparent Recipe for Training and Trustworthy Evaluation of DeepSearch Agents Introduces Glasseek, a framework for deep-search agents that explore the open web, with trajectory construction, training across context compression, and a test-set leakage audit. - https://awesomecua.com/resources/arxiv-2610-16998/ · source: https://arxiv.org/abs/2610.16998v1 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, data-generation, reinforcement-learning, evaluation - Published: 2026-10-12 · Authors: Cong Pang, Jiaqi Su, Chengrui Wu, Kuihao Yan, Tianrui Zhang, Zixuan Qiu, Chao Lin, Zixiu Lin, Zixuan Chen, Zelin Xiao, Tiankuo Yao, Jiawei Hong, Zixuan Chen, Dan Xu, Lewei Lu, Wenjie Wang, Xin Lou ### GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning Reports the GLM-4.1V-Thinking, GLM-4.5V, and GLM-4.6V vision-language model family and curriculum-based RL training, with GUI agents among its evaluated tasks. - https://awesomecua.com/resources/arxiv-2507-01006/ · source: https://arxiv.org/abs/2507.01006v6 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, evaluation - Published: 2025-07-01 · Authors: see the paper ### Go-Browse: Training Web Agents with Structured Exploration Go-Browse frames web-environment exploration as graph search so that information can be reused across data-collection episodes. It collects 10,000 successful trajectories and 40,000 interaction steps on WebArena and uses them to fine-tune a web agent. - https://awesomecua.com/resources/arxiv-2506-03533/ · source: https://arxiv.org/abs/2506.03533v2 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, web-navigation - Published: 2025-06-04 · Authors: Apurva Gandhi, Graham Neubig ### GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction Studies a compact encoder-decoder grounding model and progressive training-data refinement for resource-constrained GUI interaction. - https://awesomecua.com/resources/arxiv-2604-23941/ · source: https://arxiv.org/abs/2604.23941v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, efficiency - Published: 2026-04-27 · Authors: Hongxin Li, Yuntao Chen, Zhaoxiang Zhang ### GPA: Learning GUI Process Automation from Demonstrations Presents demonstration-based GUI process replay with Sequential Monte Carlo localization and readiness checks. GPA executes the learned workflow locally and can serve as the GUI execution component of an agent using MCP or CLI tools. - https://awesomecua.com/resources/arxiv-2604-01676/ · source: https://arxiv.org/abs/2604.01676v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, workflow, robustness - Published: 2026-04-02 · Authors: Zirui Zhao, Jun Hao Liew, Yan Yang, Wenzhuo Yang, Ziyang Luo, Doyen Sahoo, Silvio Savarese, Junnan Li ### GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation Studies MM-Navigator, a multimodal agent that interprets smartphone screenshots and predicts localized actions without task-specific training. - https://awesomecua.com/resources/arxiv-2311-07562/ · source: https://arxiv.org/abs/2311.07562v1 - Paper · reviewed 2026-10-16 · environments: ios, android · topics: mobile-navigation, grounding - Published: 2023-11-13 · Authors: An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, Lijuan Wang ### GPT-4V(ision) is a Generalist Web Agent, if Grounded Introduces SeeAct and live-website evaluation, examining how visual planning and element grounding affect web-agent task completion. - https://awesomecua.com/resources/arxiv-2401-01614/ · source: https://arxiv.org/abs/2401.01614v2 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, grounding - Published: 2024-01-03 · Authors: Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su ### GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning Introduces GPTVoiceTasker, a mobile assistant that interprets user commands, executes device interactions, and learns from prior commands to streamline later task execution. The work includes a user study of task efficiency. - https://awesomecua.com/resources/arxiv-2401-14268/ · source: https://arxiv.org/abs/2401.14268v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, human-in-the-loop, memory - Published: 2024-01-25 · Authors: Minh Duc Vu, Han Wang, Zhuang Li, Jieshan Chen, Shengdong Zhao, Zhenchang Xing, Chunyang Chen ### Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs Introduces Graph2Eval, which generates multimodal agent evaluation tasks from knowledge graphs, including multi-step web interactions in Graph2Eval-Bench. - https://awesomecua.com/resources/arxiv-2510-00507/ · source: https://arxiv.org/abs/2510.00507v3 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, evaluation, web-navigation - Published: 2025-10-01 · Authors: Yurun Chen, Xavier Hu, Yuhan Liu, Ziqi Wang, Zeyi Liao, Lin Chen, Feng Wei, Yuxi Qian, Bo Zheng, Keting Yin, Shengyu Zhang ### GraphDroid: Asynchronous LLM-Based Mobile App GUI Testing via History-Aware Exploration and Hybrid Intent Fulfillment GraphDroid combines clustered memory of visited GUI states with asynchronous test-intent generation. A hybrid executor reserves LLM calls for complex intents and uses heuristics for simpler ones, evaluated through Android app coverage, bug discovery and crash detection. - https://awesomecua.com/resources/arxiv-2609-10031/ · source: https://arxiv.org/abs/2609.10031v1 - Paper · reviewed 2026-10-18 · environments: android · topics: evaluation, mobile-navigation, memory, interaction-history, efficiency - Published: 2026-09-09 · Authors: Xiaolei Li, Jialun Cao, Zhijian Hou, Yuzhi Zhao, Yepang Liu, Shing-Chi Cheung ### GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph GraphPilot builds app-specific knowledge graphs from explored pages, elements and transitions, then uses them to plan action sequences. It requests additional interface observations when uncertain and checks plans against graph transition rules before correction and execution. - https://awesomecua.com/resources/arxiv-2601-17418/ · source: https://arxiv.org/abs/2601.17418v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, planning, memory, efficiency - Published: 2026-01-24 · Authors: Mingxian Yu, Siqi Luo, Xu Chen ### Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps Orienter detects interactable GUI elements in XR applications by interpreting scene context and revisiting candidate elements through a validation and reflection loop. It addresses open-vocabulary element types and context-dependent interactability. - https://awesomecua.com/resources/arxiv-2409-10811/ · source: https://arxiv.org/abs/2409.10811v4 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: screen-understanding, grounding - Published: 2024-09-17 · Authors: Shuqing Li, Binchang Li, Yepang Liu, Cuiyun Gao, Jianping Zhang, Shing-Chi Cheung, Michael R. Lyu ### Grounded Language Agent for Product Search via Intelligent Web Interactions Introduces GLAINTEL, a Flan-T5-based product-search web agent studied under unsupervised learning, demonstration-based training, and unsupervised domain adaptation. It compares ways to combine demonstrations with reinforcement learning. - https://awesomecua.com/resources/arxiv-2404-10887/ · source: https://arxiv.org/abs/2404.10887v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, reinforcement-learning, imitation-learning - Published: 2024-04-16 · Authors: Moghis Fereidouni, Adib Mosharrof, A. B. Siddique ### Grounded Reinforcement Learning for Visual Reasoning Introduces ViGoRL, a visual reasoning model trained to reference image coordinates and zoom during reasoning; its evaluations include GUI element localization. - https://awesomecua.com/resources/arxiv-2505-23678/ · source: https://arxiv.org/abs/2505.23678v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, grounding, screen-understanding - Published: 2025-05-29 · Authors: Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J. Tarr, Aviral Kumar, Katerina Fragkiadaki ### GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning Introduces ScreenRef screenshot grounding data and a training sequence from supervised learning through teacher-guided distillation to reinforcement learning for GroundGUI-8B. - https://awesomecua.com/resources/arxiv-2610-19100/ · source: https://arxiv.org/abs/2610.19100v1 - Paper · reviewed 2026-10-17 · environments: desktop, web, cross-platform · topics: grounding, data-generation, reinforcement-learning - Published: 2026-10-13 · Authors: Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das ### Grounding Computer Use Agents on Human Demonstrations GroundCUA turns expert desktop demonstrations into densely annotated screenshots and grounding instructions. The GroundNext model family uses this supervision and subsequent reinforcement learning, with both grounding benchmarks and an OSWorld agent configuration used for evaluation. - https://awesomecua.com/resources/arxiv-2511-07332/ · source: https://arxiv.org/abs/2511.07332v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: grounding, data-generation, evaluation, reinforcement-learning - Published: 2025-11-10 · Authors: Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Kaixin Li, Rabiul Awal, Xing Han Lù, Johan Obando-Ceron, Juan A. Rodriguez, Nicolas Chapados, David Vazquez, Adriana Romero-Soriano, Reihaneh Rabbany, Perouz Taslakian, Christopher Pal, Spandana Gella, Sai Rajeswar ### Grounding Open-Domain Instructions to Automate Web Support Tasks Introduces RUSS, which translates support instructions into a domain-specific language and grounds them in actionable web elements. - https://awesomecua.com/resources/arxiv-2103-16057/ · source: https://arxiv.org/abs/2103.16057v2 - Paper · reviewed 2026-10-16 · environments: web · topics: grounding, web-navigation - Published: 2021-03-30 · Authors: Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, Monica S Lam ### Group-in-Group Policy Optimization for LLM Agent Training Introduces Group-in-Group Policy Optimization (GiGPO), an RL method for assigning step-level credit in multi-turn agents, evaluated on WebShop among other tasks. - https://awesomecua.com/resources/arxiv-2505-10978/ · source: https://arxiv.org/abs/2505.10978v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, reward-models - Published: 2025-05-16 · Authors: Lang Feng, Zhenghai Xue, Tingcong Liu, Bo An ### GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis Introduces GSAR, which synthesizes GUI tasks and successful goal states, then annotates task-relevant interface elements as reward anchors. These anchors support offline trajectory verification and online policy training. - https://awesomecua.com/resources/arxiv-2608-22847/ · source: https://arxiv.org/abs/2608.22847v1 - Paper · reviewed 2026-10-17 · environments: android · topics: data-generation, reward-models, reinforcement-learning - Published: 2026-08-24 · Authors: Long Zhang, Yuhan Chen, Chaoran Zhang, Wanxia Cao, Kun Huang, Pengzhi Gao, Wei Liu, Jian Luan, Chenliang Li, Lixin Zou ### GTA1: GUI Test-time Scaling Agent GTA1 samples multiple action proposals at each GUI step and uses a judge to select among them. A separate grounding model is trained with reinforcement learning to map selected proposals to visual targets, linking planning-time search with click localization. - https://awesomecua.com/resources/arxiv-2507-05791/ · source: https://arxiv.org/abs/2507.05791v5 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: planning, grounding, reinforcement-learning - Published: 2025-07-08 · Authors: Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Silvio Savarese, Caiming Xiong, Junnan Li ### GTA: Generating Long-Horizon Tasks for Web Agents at Scale GTA generates multi-hop web-agent tasks and executable trajectories through crawling, retrieval, and validation, releasing a benchmark built from over 50 websites. - https://awesomecua.com/resources/arxiv-2605-29218/ · source: https://arxiv.org/abs/2605.29218v1 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, evaluation, web-navigation, long-horizon - Published: 2026-05-28 · Authors: Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu ### GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning Introduces GuardAgent, which translates safety requests into executable guardrail code using plans and retrieved demonstrations. Mind2Web-SC supplies a web-agent safety-policy evaluation alongside a separate healthcare access-control benchmark. - https://awesomecua.com/resources/arxiv-2406-09187/ · source: https://arxiv.org/abs/2406.09187v3 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, tool-integration - Published: 2024-06-13 · Authors: Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, Bo Li ### GUI Action Narrator: Where and When Did That Action Take Place? Introduces Act2Cap and a cursor-guided method for describing GUI actions from video, with temporal and spatial localization challenges. - https://awesomecua.com/resources/arxiv-2406-13719/ · source: https://arxiv.org/abs/2406.13719v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: video, screen-understanding - Published: 2024-06-19 · Authors: Qinchen Wu, Difei Gao, Kevin Qinghong Lin, Zhuoyu Wu, Xiangwu Guo, Peiran Li, Weichen Zhang, Hengxu Wang, Mike Zheng Shou ### GUI agent: Guided Exploration of User-Sensitive Screens This short paper develops an explorer agent that expands from a demonstrated GUI task to discover requests likely to reach user-sensitive screens. The proposed exploration supports identifying situations where task execution should be handed back to the user. - https://awesomecua.com/resources/arxiv-2606-25705/ · source: https://arxiv.org/abs/2606.25705v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, human-in-the-loop, data-generation - Published: 2026-06-24 · Authors: Aradhana Nayak, Mussadiq Nazeer, Wang Peng, Feng Liu ### GUI Agents for Continual Game Generation PlaytestArena evaluates generated browser games using expected in-play behaviors adjudicated by GUI agents. Play2Code pairs a coding agent with a rubric-blind GUI playtester whose interaction traces and feedback guide iterative refinement, with a separate final judge. - https://awesomecua.com/resources/arxiv-2605-28258/ · source: https://arxiv.org/abs/2605.28258v2 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, automation, workflow - Published: 2026-05-27 · Authors: Yixu Huang, Bo Li, Na Li, Zhe Wang, Kaijie Chen, Haonan Ge, Qingyi Si, Yuanzhe Shen, Ruihan Yang, Guangjing Wang, Hongcheng Guo ### GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning Introduces GUI Exploration Lab, a simulation engine for composing screens, interface elements, and navigation graphs. It studies how supervised learning, single-turn RL, and multiturn RL affect GUI navigation and exploration. - https://awesomecua.com/resources/arxiv-2512-02423/ · source: https://arxiv.org/abs/2512.02423v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reinforcement-learning, evaluation, data-generation - Published: 2025-12-02 · Authors: Haolong Yan, Yeqing Shen, Xin Huang, Jia Wang, Kaijun Tan, Zhixuan Liang, Hongxin Li, Zheng Ge, Osamu Yoshie, Si Li, Xiangyu Zhang, Daxin Jiang ### GUI Knowledge Bench: Revealing the Knowledge Gap of VLMs in GUI Tasks GUI Knowledge Bench tests knowledge of widgets, interactions, and procedures across six named platforms and 292 applications to assess VLMs used for GUI tasks. - https://awesomecua.com/resources/arxiv-2510-26098/ · source: https://arxiv.org/abs/2510.26098v2 - Paper · reviewed 2026-11-06 · environments: web, desktop, windows, macos, linux, android, ios, cross-platform · topics: evaluation, screen-understanding, state-tracking - Published: 2025-10-30 · Authors: Chenrui Shi, Zedong Yu, Zhi Gao, Ruining Feng, Enqi Liu, Yuwei Wu, Yunde Jia, Liuyu Xiang, Zhaofeng He, Qing Li ### GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent Evaluates GUI testing through test-intent generation, execution, and defect detection using real, modified, and synthetic mobile application data. - https://awesomecua.com/resources/arxiv-2412-18426/ · source: https://arxiv.org/abs/2412.18426v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, automation, mobile-navigation - Published: 2024-12-24 · Authors: Kangjia Zhao, Jiahui Song, Leigang Sha, Haozhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, Jianwei Yin ### GUI Testing Using Computer Vision Presents visual GUI test scripts that use images to specify actions and expected screen feedback. Recording both input events and screenshots can generate tests from demonstrations, supporting unit, regression, and test-driven workflows. - https://awesomecua.com/resources/url-6262c757080f8838e62c/ · source: https://doi.org/10.1145/1753326.1753555 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, grounding, automation - Published: 2010-04 · Authors: Tsung-Hsiang Chang, Tom Yeh, Robert C. Miller ### GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents This benchmark matches desktop goals, initial states and final-state verifiers across screen-only GUI and skill-mediated CLI agents. Its experiments separate graphical execution failures from missing CLI skill coverage, including a verifier-guided skill-augmentation condition. - https://awesomecua.com/resources/arxiv-2606-24551/ · source: https://arxiv.org/abs/2606.24551v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, tool-integration, robustness, workflow - Published: 2026-06-22 · Authors: Xiao Zhou, Siyue Zhang, Yilun Zhao, Jinbiao Wei, Tingyu Song, Arman Cohan, Chen Zhao ### GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents Collects Windows application trajectories with screenshots and action traces for grounding, screen parsing, and action prediction. - https://awesomecua.com/resources/arxiv-2511-04307/ · source: https://arxiv.org/abs/2511.04307v2 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: data-generation, grounding, evaluation - Published: 2025-11-06 · Authors: Jian Mu, Chaoyun Zhang, Chiming Ni, Lu Wang, Bo Qiao, Kartik Mathur, Qianhui Wu, Yuhang Xie, Xiaojun Ma, Mengyu Zhou, Si Qin, Liqun Li, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang ### GUI-AC: Enhancing Continual Learning in GUI Agents GUI-AC uses grounding certainty to adjust advantage weighting and policy-clipping bounds during continual reinforcement fine-tuning. Its mechanisms aim to suppress noisy confidence updates while retaining exploration under shifting GUI domains and resolutions. - https://awesomecua.com/resources/arxiv-2606-10522/ · source: https://arxiv.org/abs/2606.10522v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: reinforcement-learning, grounding, robustness - Published: 2026-06-09 · Authors: Can Lin, Tao Feng, Hangjie Yuan, Dan Zhang, Yifan Zhu, Zhonghong Ou ### GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents Uses an attention-based action head and a verifier to select relevant screen regions without generating coordinates as text. - https://awesomecua.com/resources/arxiv-2506-03143/ · source: https://arxiv.org/abs/2506.03143v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Published: 2025-06-03 · Authors: Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, Si Qin, Lars Liden, Qingwei Lin, Huan Zhang, Tong Zhang, Jianbing Zhang, Dongmei Zhang, Jianfeng Gao ### GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding Introduces GUI-AIMA, which supervises instruction-relevant attention over visual patches instead of directly training coordinate text generation. Patch-level grounding can be combined with an optional zoom stage and is evaluated across GUI localization benchmarks. - https://awesomecua.com/resources/arxiv-2511-00810/ · source: https://arxiv.org/abs/2511.00810v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, efficiency - Published: 2025-11-02 · Authors: Shijie Zhou, Viet Dac Lai, Hao Tan, Jihyung Kil, Wanrong Zhu, Changyou Chen, Ruiyi Zhang ### GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agents GUI-ARP uses visual attention to crop relevant screenshot regions and decides whether a target needs single-stage or multi-stage analysis. Supervised fine-tuning followed by GRPO trains this adaptive procedure for high-resolution GUI grounding. - https://awesomecua.com/resources/arxiv-2509-15532/ · source: https://arxiv.org/abs/2509.15532v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, screen-understanding - Published: 2025-09-19 · Authors: Xianhang Ye, Yiqing Li, Wei Dai, Miancan Liu, Ziyuan Chen, Zhangye Han, Hongbo Min, Jinkui Ren, Xiantao Zhang, Wen Yang, Zhi Jin ### GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration Collects environment-specific interaction data through autonomous exploration to adapt grounding models to previously unseen GUIs. - https://awesomecua.com/resources/arxiv-2501-13896/ · source: https://arxiv.org/abs/2501.13896v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, data-generation - Published: 2025-01-23 · Authors: Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, Gang Wu ### GUI-C$^2$: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning Introduces GUI-D to score training samples by difficulty and GUI-C² to refine screen-element locations through area-gated cropping and reinforcement learning. - https://awesomecua.com/resources/arxiv-2605-30884/ · source: https://arxiv.org/abs/2605.30884v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, reinforcement-learning, data-generation, efficiency - Published: 2026-05-29 · Authors: Junlong Li, Chao Hao, Lap-Pui Chau, Yi Wang ### GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments Introduces GUI-CC, a benchmark for assessing GUI world models as multi-step agent environments. Separate reference-action and agent-loop tracks test whether generated screens preserve context, plausible transitions, and task progress throughout interaction. - https://awesomecua.com/resources/arxiv-2609-00048/ · source: https://arxiv.org/abs/2609.00048v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: world-models, evaluation, long-horizon - Published: 2026-08-30 · Authors: Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong ### GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents GUI-CEval evaluates Chinese mobile GUI agents through a hierarchy of atomic abilities and application-level tasks collected on physical devices. Its diagnostics cover perception, planning, reflection, execution and post-action evaluation across a broad set of apps. - https://awesomecua.com/resources/arxiv-2603-15039/ · source: https://arxiv.org/abs/2603.15039v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, self-reflection, screen-understanding - Published: 2026-03-16 · Authors: Yang Li, Yuchen Liu, Haoyu Lu, Zhiqiang Xia, Hongzhen Wang, Kaiyang Han, Changpeng Yang, Jinyang Wu, Jiaming Xu, Runyu Shi, Ying Huang ### GUI-CIDER: Mid-training GUI Agents via Causal Internalization and Density-aware Exemplar Reselection GUI-CIDER turns GUI trajectories into textual planning and causal interface knowledge, reselects exemplars for causal structure and low redundancy, then mid-trains GUI agents on that corpus. - https://awesomecua.com/resources/arxiv-2605-28534/ · source: https://arxiv.org/abs/2605.28534v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, planning - Published: 2026-05-27 · Authors: Zheng Wu, Chengcheng Han, Zhengxi Lu, Tianjie Ju, Yanyu Chen, Qi Gu, Xunliang Cai, Zhuosheng Zhang ### GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent GUI-explorer generates exploration goals from screenshots and interface structure, then mines observation-action-outcome triples for app-specific transition knowledge. The resulting training-free agent uses this knowledge during mobile task execution, evaluated on SPA-Bench and AndroidWorld. - https://awesomecua.com/resources/arxiv-2505-16827/ · source: https://arxiv.org/abs/2505.16827v1 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, data-generation, memory, planning - Published: 2025-05-22 · Authors: Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, Liqiang Nie ### GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents GUI-Eyes trains GUI grounders to decide when and how to use visual tools such as crops and zooms. Its coarse-to-fine policy uses dense spatial rewards based on location proximity and region overlap, with evaluation on ScreenSpot-Pro. - https://awesomecua.com/resources/arxiv-2601-09770/ · source: https://arxiv.org/abs/2601.09770v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, tool-integration, screen-understanding - Published: 2026-01-14 · Authors: Chen Chen, Jiawei Shao, Dakuan Lu, Haoyi Hu, Xiangcheng Liu, Hantao Yao, Wu Liu ### GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding GUI-G2 replaces binary grounding rewards with Gaussian point rewards and spatial-coverage rewards. An adaptive variance tied to element dimensions supplies continuous training feedback for targets of different sizes, evaluated on the ScreenSpot benchmark family. - https://awesomecua.com/resources/arxiv-2507-15846/ · source: https://arxiv.org/abs/2507.15846v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, reward-models - Published: 2025-07-21 · Authors: Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, Yueting Zhuang ### GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents Analyzes reasoning templates, reward design, and policy updates in GUI grounding, then proposes targeted adjustments to the training process. - https://awesomecua.com/resources/arxiv-2505-15810/ · source: https://arxiv.org/abs/2505.15810v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, reinforcement-learning - Published: 2025-05-21 · Authors: Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, Jun Xu ### GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments GUI-GenBench tests image-generation models as action-conditioned GUI environments across five suites, including transitions, trajectories and grounding. Its VLM-based GUI-Score separates goal completion, interaction logic, consistency and visual properties. - https://awesomecua.com/resources/arxiv-2602-09007/ · source: https://arxiv.org/abs/2602.09007v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, world-models, grounding, screen-understanding - Published: 2026-02-09 · Authors: Haodong Li, Jingwei Wu, Quan Sun, Guopeng Li, Juanxi Tian, Ziyu Guo, Donghao Zhou, Tianfei Ren, Huanyu Zhang, Yanlin Lai, Ruichuan An, Hongbo Peng, Yuhong Dai, Jia Wang, Ziyang Meng, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Pheng-Ann Heng ### GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training GUI-GENESIS reconstructs applications as lightweight web environments with executable assertions for reward computation. It evaluates the cost of these training environments and transfer from synthetic interaction training to held-out real tasks. - https://awesomecua.com/resources/arxiv-2602-14093/ · source: https://arxiv.org/abs/2602.14093v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, sandbox, reinforcement-learning, evaluation - Published: 2026-02-15 · Authors: Yuan Cao, Dezhi Ran, Mengzhou Wu, Yuzhe Guo, Xin Chen, Ang Li, Gang Cao, Gong Zhi, Hao Yu, Linyi Li, Wei Yang, Tao Xie ### GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution Improves GUI-agent execution code by analyzing action effects and repeated-task failures, then validating bounded changes with frozen model weights. - https://awesomecua.com/resources/arxiv-2610-00948/ · source: https://arxiv.org/abs/2610.00948v1 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: automation, observability, robustness - Published: 2026-10-01 · Authors: Geyi Yang, Zikun Qu, Xiang Li, Zhiyong Wang, Min Zhang, Shipei Zeng, Zhongxiang Dai ### GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents Introduces GUI-JEPA, a mobile GUI policy that scores candidate actions from screenshot latents in a batched pass without decoding action text. - https://awesomecua.com/resources/arxiv-2610-36877/ · source: https://arxiv.org/abs/2610.36877v1 - Paper · reviewed 2026-11-06 · environments: android · topics: action-prediction, efficiency, mobile-navigation - Published: 2026-10-27 · Authors: Bingchao Wang, Yongcheng Jing, Shunyu Liu, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao ### GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness GUI-KV compresses vision-language agents’ attention caches using visual-token saliency and redundancy between screenshot frames. The method requires no retraining and evaluates how cache budgets affect GUI action accuracy and computational cost. - https://awesomecua.com/resources/arxiv-2510-00536/ · source: https://arxiv.org/abs/2510.00536v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: efficiency, interaction-history, training-free, screen-understanding - Published: 2025-10-01 · Authors: Kung-Hsiang Huang, Haoyi Qiu, Yutong Dai, Caiming Xiong, Chien-Sheng Wu ### GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs GUI-Lens uses OCR and detected interface components as coordinate references while a general-purpose VLM selects progressively enlarged screenshot crops. Instruction-conditioned checks constrain the crop and click proposals before mapping the final location back to the original screen. - https://awesomecua.com/resources/arxiv-2608-03270/ · source: https://arxiv.org/abs/2608.03270v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, evaluation - Published: 2026-08-04 · Authors: Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, Wanyu Wang, Xian Wu, Xiangyu Zhao ### GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL GUI-Libra trains GUI agents with action-aware supervision, a curated reasoning dataset and reinforcement learning adapted to partially verifiable actions across web and mobile tasks. - https://awesomecua.com/resources/arxiv-2602-22190/ · source: https://arxiv.org/abs/2602.22190v2 - Paper · reviewed 2026-11-06 · environments: web, cross-platform · topics: reinforcement-learning, data-generation, grounding, action-prediction - Published: 2026-02-25 · Authors: Rui Yang, Qianhui Wu, Zhaoyang Wang, Hanyang Chen, Ke Yang, Hao Cheng, Huaxiu Yao, Baolin Peng, Huan Zhang, Jianfeng Gao, Tong Zhang ### GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models Introduces GUI-Perturbed, which varies screenshot appearance and instruction wording independently to diagnose grounding failures. Experiments on three related model architectures examine relational instructions, browser zoom and augmentation-based fine-tuning. - https://awesomecua.com/resources/arxiv-2604-14262/ · source: https://arxiv.org/abs/2604.14262v2 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, robustness, evaluation - Published: 2026-04-15 · Authors: Yangyue Wang, Harshvardhan Sikka, Yash Mathur, Tony Zhou, Jinu Nyachhyon, Pranav Guruprasad ### GUI-PRA: Process Reward Agent for GUI Tasks GUI-PRA evaluates GUI progress by generating task-specific criteria and actively inspecting visual evidence with tools. It tests offline judgment on OS-Critic Bench and the effects of this feedback on online mobile policies in AndroidWorld. - https://awesomecua.com/resources/arxiv-2509-23263/ · source: https://arxiv.org/abs/2509.23263v3 - Paper · reviewed 2026-10-17 · environments: android · topics: reward-models, evaluation, screen-understanding - Published: 2025-09-27 · Authors: Tao Xiong, Xavier Hu, Yurun Chen, Yuhang Liu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, Shengyu Zhang ### GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding Introduces GUI-Primitives, a paired-instruction benchmark that changes a spatial relation while holding the screenshot and anchor fixed. Its diagnostics separate failure to localize either candidate from failure to choose the candidate satisfying the relation. - https://awesomecua.com/resources/arxiv-2608-21832/ · source: https://arxiv.org/abs/2608.21832v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, evaluation, screen-understanding - Published: 2026-08-22 · Authors: Md Abrar Jahin, Md Rizwan Parvez ### GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents Applies reinforcement fine-tuning with unified action rules and curated cross-platform examples to vision-language GUI agents. - https://awesomecua.com/resources/arxiv-2504-10458/ · source: https://arxiv.org/abs/2504.10458v4 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: reinforcement-learning, grounding - Published: 2025-04-14 · Authors: Run Luo, Lu Wang, Wanwei He, Longze Chen, Jiaming Li, Xiaobo Xia ### GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior GUI-Reflection synthesizes reflection and error-correction examples from successful trajectories and adds dedicated learning stages for these behaviors. It introduces a reflection task suite, a mobile environment for online data collection, and iterative reflection tuning. - https://awesomecua.com/resources/arxiv-2506-08012/ · source: https://arxiv.org/abs/2506.08012v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: robustness, data-generation, evaluation, mobile-navigation - Published: 2025-06-09 · Authors: Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu ### GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning GUI-ReWalk begins with stochastic interface exploration and then infers goals to generate coherent action trajectories. It supports long-horizon workflows across applications and studies training on the resulting data using grounding and navigation benchmarks. - https://awesomecua.com/resources/arxiv-2509-15738/ · source: https://arxiv.org/abs/2509.15738v1 - Paper · reviewed 2026-10-17 · environments: android · topics: data-generation, cross-app, long-horizon, planning - Published: 2025-09-19 · Authors: Musen Lin, Minghao Liu, Taoran Lu, Lichen Yuan, Yiwei Liu, Haonan Xu, Yu Miao, Yuhao Chao, Zhaojian Li ### GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation Combines structured progress-and-decision reasoning with history summarization to improve cross-domain generalization in GUI navigation agents. - https://awesomecua.com/resources/arxiv-2510-27210/ · source: https://arxiv.org/abs/2510.27210v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: planning, interaction-history, reinforcement-learning - Published: 2025-10-31 · Authors: Tao Liu, Chongyu Wang, Rongjie Li, Yingchen Yu, Xuming He, Bai Song ### GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies Builds GUI task data with real-world interaction anomalies and evaluates how agent behavior changes under those conditions. - https://awesomecua.com/resources/arxiv-2506-14477/ · source: https://arxiv.org/abs/2506.14477v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: robustness, evaluation, data-generation - Published: 2025-06-17 · Authors: Jingqi Yang, Zhilong Song, Jiawei Chen, Mingli Song, Sheng Zhou, linjun sun, Xiaogang Ouyang, Chun Chen, Can Wang ### GUI-SAGE: Enhancing GUI Automation with Self-Explanatory Learning GUI-SAGE generates reasoning conditioned on known actions and weights policy updates using confidence and reward signals. Its experiments use AndroidControl and GUIOdyssey data and separately evaluate interactive AndroidWorld tasks. - https://awesomecua.com/resources/url-beb8f1fd6b945ffc9d93/ · source: https://openaccess.thecvf.com/content/CVPR2026/html/Tang_GUI-SAGE_Enhancing_GUI_Automation_with_Self-Explanatory_Learning_CVPR_2026_paper.html - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, data-generation, evaluation, robustness - Published: 2026-06 · Authors: Fei Tang, Zhangxuan Gu, Zhengxi Lu, Shangzhan Zhang, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Yuchen Yan, Wenqi Zhang, Yongliang Shen, Weiming Lu, Yueting Zhuang ### GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks Introduces GUI-Shepherd, a process reward model trained on annotated GUI interactions to give step-level feedback for agent training and verification on AndroidWorld and AndroidControl. - https://awesomecua.com/resources/arxiv-2509-23738/ · source: https://arxiv.org/abs/2509.23738v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reward-models, reinforcement-learning, evaluation - Published: 2025-09-28 · Authors: Cong Chen, Kaixiang Ji, Hao Zhong, Muzhi Zhu, Anzhou Li, Guo Gan, Ziyuan Huang, Cheng Zou, Jiajia Liu, Jingdong Chen, Hao Chen, Chunhua Shen ### GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning GUI-Shift trains an inverse-dynamics task that predicts the initial action connecting two GUI states, using trajectories without natural-language instructions. It combines rule-based reinforcement learning with data filtering and evaluates transfer to Android automation and screenshot grounding. - https://awesomecua.com/resources/arxiv-2505-12493/ · source: https://arxiv.org/abs/2505.12493v3 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, data-generation, grounding, mobile-navigation - Published: 2025-05-18 · Authors: Longxi Gao, Li Zhang, Pengzhi Gao, Wei Liu, Jian Luan, Mengwei Xu ### GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding Introduces video-based GUI questions and annotations for studying temporal interface understanding across varied interaction settings. - https://awesomecua.com/resources/arxiv-2406-10819/ · source: https://arxiv.org/abs/2406.10819v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, video, screen-understanding - Published: 2024-06-16 · Authors: Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, Lichao Sun ### GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration Introduces GUI-Xplore exploration videos and downstream tasks, with an agent that combines action-aware modeling and graph-based environment reasoning. - https://awesomecua.com/resources/arxiv-2503-17709/ · source: https://arxiv.org/abs/2503.17709v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation, planning, video - Published: 2025-03-22 · Authors: Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, Chongyang Zhang ### GUICourse: From General Vision Language Models to Versatile GUI Agents Introduces GUIEnv, GUIAct, and GUIChat datasets to develop OCR, grounding, component knowledge, and interaction abilities in visual-language models. - https://awesomecua.com/resources/arxiv-2406-11317/ · source: https://arxiv.org/abs/2406.11317v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, screen-understanding - Published: 2024-06-17 · Authors: Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, Maosong Sun ### GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots GUICrafter first learns grounding from large collections of unannotated screenshots and webpages, then uses a smaller high-quality dataset for reinforcement-learning calibration. The curriculum targets reduced dependence on manually annotated GUI training data. - https://awesomecua.com/resources/arxiv-2606-29705/ · source: https://arxiv.org/abs/2606.29705v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, data-generation, reinforcement-learning, screen-understanding - Published: 2026-06-29 · Authors: Sunqi Fan, Lingshan Chen, Runqi Yin, Qingle Liu, Yongming Rao, Meng-Hao Guo, Shi-Min Hu ### GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks GUIDE evaluates whether models can recognize a user’s behavior, infer intent and decide when to offer help during open-ended software tasks. It uses narrated screen recordings of novice users and studies the effect of additional user context on these assistance decisions. - https://awesomecua.com/resources/arxiv-2603-25864/ · source: https://arxiv.org/abs/2603.25864v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, human-in-the-loop, video, screen-understanding - Published: 2026-03-26 · Authors: Saelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin, Jae Won Cho, Yale Song, Juho Kim ### GUIDE: Graphical User Interface Data for Execution Introduces annotated website task trajectories with screenshots, prior actions, reasoning, and grounded next actions for process automation. - https://awesomecua.com/resources/arxiv-2404-16048/ · source: https://arxiv.org/abs/2404.16048v2 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, grounding - Published: 2024-04-09 · Authors: Rajat Chawla, Adarsh Jha, Muskaan Kumar, Mukunda NS, Ishaan Bhola ### GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis GUIDE evaluates GUI trajectories by segmenting them into subtasks, diagnosing each segment, and aggregating structured findings into a task verdict. It tests diagnostic judgments on e-commerce, web-agent and Android-control traces rather than relying on one holistic trajectory score. - https://awesomecua.com/resources/arxiv-2604-04399/ · source: https://arxiv.org/abs/2604.04399v1 - Paper · reviewed 2026-10-18 · environments: web, android · topics: evaluation, long-horizon, state-tracking - Published: 2026-04-06 · Authors: Yuwen Zhai, Runze Li, Liang Wang, Nian Shi, Liwu Xu, Wei Zhang, Ran Lin, Bo Xu, Benlei Cui ### GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation GUIDE retrieves relevant software tutorials through subtitle-based video search, then infers planning and grounding knowledge from consecutive annotated keyframes. It inserts that knowledge into existing GUI-agent modules without parameter updates and evaluates the plug-in on OSWorld. - https://awesomecua.com/resources/arxiv-2603-26266/ · source: https://arxiv.org/abs/2603.26266v3 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: video, planning, grounding, training-free - Published: 2026-03-27 · Authors: Rui Xie, Zhi Gao, Chenrui Shi, Zirui Shang, Lu Chen, Qing Li ### Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents WebMRE pairs guidance sentences with grounded actions in an offline benchmark derived from successful WebArena traces. Controlled decoding-prefix interventions test how guide content affects action selection, while separating offline prediction metrics from live browser task success. - https://awesomecua.com/resources/arxiv-2609-27353/ · source: https://arxiv.org/abs/2609.27353v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, action-prediction, imitation-learning, grounding - Published: 2026-09-23 · Authors: Chengguang Gan, Yunhao Liang, QingHao Zhang, Shiwen Ni ### Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation Guides visual GUI agents at inference time with a process-reward model that evaluates intermediate actions. It combines stepwise guidance with trajectory reflection and retry mechanisms in static and interactive navigation tasks. - https://awesomecua.com/resources/arxiv-2504-16073/ · source: https://arxiv.org/abs/2504.16073v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reward-models, self-reflection, grounding - Published: 2025-04-22 · Authors: Zhiyuan Hu, Shiyun Xiong, Yifan Zhang, See-Kiong Ng, Anh Tuan Luu, Bo An, Shuicheng Yan, Bryan Hooi ### GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents Introduces GUIFlow, an incremental inference framework that reuses visual, language, and action computation across successive GUI-agent steps. - https://awesomecua.com/resources/arxiv-2610-34959/ · source: https://arxiv.org/abs/2610.34959v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, interaction-history, training-free - Published: 2026-10-26 · Authors: Ye Li, Haoran Ding, Huanan Liu, Yuan Meng, Yutian Shen, Yuansong Wang, Kangye Ji, Hanyun Cui, Hongbo Duan, Chen Tang, Jiawei Li, Shu-Tao Xia, Zhi Wang ### GUIGuard-Bench: Toward a General Evaluation for Privacy-Preserving GUI Agents Introduces GUIGuard-Bench, which annotates private regions in GUI trajectories with categories, risk levels, and relevance to the user’s task. It evaluates privacy recognition and the effect of screenshot protection on offline planning and task utility. - https://awesomecua.com/resources/arxiv-2601-18842/ · source: https://arxiv.org/abs/2601.18842v3 - Paper · reviewed 2026-10-17 · environments: android, desktop · topics: safety-security, evaluation, grounding - Published: 2026-01-26 · Authors: Yanxi Wang, Zhiling Zhang, Wenbo Zhou, Weiming Zhang, Jie Zhang, Qiannan Zhu, Yu Shi, Shuxin Zheng, Jiyan He ### GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices Introduces annotated cross-application mobile trajectories and studies history-aware navigation over long task sequences. - https://awesomecua.com/resources/arxiv-2406-08451/ · source: https://arxiv.org/abs/2406.08451v2 - Paper · reviewed 2026-10-16 · environments: android · topics: cross-app, data-generation - Published: 2024-06-12 · Authors: Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Ping Luo ### GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs Introduces GUIPilot, which compares mobile design mock-ups with implemented screens through widget alignment and tests expected transitions by executing inferred UI actions. It evaluates both visual inconsistencies and workflow-specific bugs. - https://awesomecua.com/resources/arxiv-2506-07385/ · source: https://arxiv.org/abs/2506.07385v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, robustness - Published: 2025-06-09 · Authors: Ruofan Liu, Xiwen Teoh, Yun Lin, Guanjie Chen, Ruofei Ren, Denys Poshyvanyk, Jin Song Dong ### GUIrilla: A Scalable Framework for Automated Desktop UI Exploration GUIrilla systematically explores macOS interfaces to collect interaction traces and accessibility metadata for downstream GUI research. It organizes these observations into hierarchical MacApp Trees and describes an accessibility-driven collection library. - https://awesomecua.com/resources/arxiv-2510-16051/ · source: https://arxiv.org/abs/2510.16051v2 - Paper · reviewed 2026-10-17 · environments: macos, desktop · topics: data-generation, screen-understanding, recording - Published: 2025-10-16 · Authors: Sofiya Garkot, Maksym Shamrai, Ivan Synytsia, Mariya Hirna ### GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning Introduces GuirlVG, an empirical study of reinforcement fine-tuning for GUI visual grounding. It analyzes training components and adds an adaptive KL-based mechanism to limit reward over-optimization. - https://awesomecua.com/resources/arxiv-2508-04389/ · source: https://arxiv.org/abs/2508.04389v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning - Published: 2025-08-06 · Authors: Weitai Kang, Bin Lei, Gaowen Liu, Caiwen Ding, Yan Yan ### GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Prototypes Introduces GUISpector, an MLLM agent framework that operates GUI prototypes to check natural-language requirements and produces feedback for developers. - https://awesomecua.com/resources/arxiv-2510-04791/ · source: https://arxiv.org/abs/2510.04791v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, planning, observability - Published: 2025-10-06 · Authors: Kristian Kolthoff, Felix Kretzer, Simone Paolo Ponzetto, Alexander Maedche, Christian Bartelt ### GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions Introduces GUITAR, which groups visually different screens into functional states and analyzes agent failures over a state-transition graph. It uses the resulting bottlenecks to diagnose recurring mobile and web errors and evaluate targeted guidance. - https://awesomecua.com/resources/arxiv-2609-34113/ · source: https://arxiv.org/abs/2609.34113v2 - Paper · reviewed 2026-10-17 · environments: android, web · topics: evaluation, observability, robustness - Published: 2026-09-28 · Authors: Shaoqing Zhang, Kehai Chen, Xuefeng Bai, Zhuosheng Zhang, Pengfei Zhang, Yang Xiang, Min Zhang ### GUITester: Enabling GUI Agents for Exploratory Defect Discovery Introduces GUITestBench for interactive defect discovery and GUITester, which separates task navigation from anomaly verification. Testing-oriented plans and history-based reflection help distinguish software defects from agent execution mistakes. - https://awesomecua.com/resources/arxiv-2601-04500/ · source: https://arxiv.org/abs/2601.04500v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, evaluation, self-reflection - Published: 2026-01-08 · Authors: Yifei Gao, Jiang Wu, Xiaoyi Chen, Yifan Yang, Zhe Cui, Tianyi Ma, Jiaming Zhang, Jitao Sang ### Gym-Anything: Turn any Software into an Agent Environment Introduces Gym-Anything, where a setup agent configures software environments and an independent auditor checks the resulting evidence. CUA-World supplies long-horizon tasks across professional applications, with experiments on trajectory distillation and test-time auditing. - https://awesomecua.com/resources/arxiv-2604-06126/ · source: https://arxiv.org/abs/2604.06126v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, sandbox, long-horizon, evaluation - Published: 2026-04-07 · Authors: Pranjal Aggarwal, Graham Neubig, Sean Welleck ### HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities Introduces HackWorld for evaluating visual computer-use agents in a capture-the-flag collection of web applications. The benchmark examines interface navigation and multistep planning during vulnerability discovery and exploitation tasks. - https://awesomecua.com/resources/arxiv-2510-12200/ · source: https://arxiv.org/abs/2510.12200v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2025-10-14 · Authors: Xiaoxue Ren, Penghao Jiang, Kaixin Li, Zhiyong Huang, Xiaoning Du, Jiaojiao Jiang, Zhenchang Xing, Jiamou Sun, Terry Yue Zhuo ### Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching This framework separates frozen-model instruction expansion from a trained layout-aware matcher that localizes GUI elements without coordinate regression. Binary text/icon supervision supports the matcher, with evaluations on GUI grounding and Mind2Web element-selection tasks. - https://awesomecua.com/resources/arxiv-2608-09654/ · source: https://arxiv.org/abs/2608.09654v1 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, screen-understanding, evaluation - Published: 2026-08-10 · Authors: Yuke Li, Xuehan Hou ### HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents Introduces HalluClear with a GUI hallucination taxonomy, expert-calibrated evaluation workflow and structured-reasoning post-training scheme. It evaluates how these components affect hallucinations, grounding and action fidelity across GUI agents. - https://awesomecua.com/resources/arxiv-2604-17284/ · source: https://arxiv.org/abs/2604.17284v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: robustness, grounding, evaluation - Published: 2026-04-19 · Authors: Chao Jin, Wenkui Yang, Hao Sun, Yuqi Liao, Qianyi Jiang, Kai Zhou, Jie Cao, Ran He, Huaibo Huang ### HANSEL: Extracting Breadcrumbs from Web Agent Trajectories for Interactive Verification HANSEL extracts evidence pages and snippets from browser-agent traces and preserves relevant page state in interactive verification views. It flags untraceable answers and evaluates evidence selection and human verification effort separately. - https://awesomecua.com/resources/arxiv-2606-18671/ · source: https://arxiv.org/abs/2606.18671v1 - Paper · reviewed 2026-10-18 · environments: web · topics: human-in-the-loop, observability, recording, evaluation - Published: 2026-06-17 · Authors: Yujin Zhang, Daye Nam ### HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimization Introduces HarmonyGuard, a multi-agent framework that updates security policies and weighs safety against task completion for agents operating on the open web. - https://awesomecua.com/resources/arxiv-2508-04010/ · source: https://arxiv.org/abs/2508.04010v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation, planning - Published: 2025-08-06 · Authors: Yurun Chen, Xavier Hu, Yuhan Liu, Keting Yin, Juncheng Li, Zhuosheng Zhang, Shengyu Zhang ### Harness Evolution Hits a Ceiling: When Weight Training Should Begin Studies when to improve an LLM agent's runtime harness or train its weights by classifying failed trajectories; WebArena-Lite supplies a web-task transfer test. - https://awesomecua.com/resources/arxiv-2610-11655/ · source: https://arxiv.org/abs/2610.11655v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: observability, training-free, imitation-learning - Published: 2026-10-08 · Authors: Yuan Tian, Bing Hu, Hao Wang, Binghang Lu, Fang Wu ### Harness-Model Collaboration for Long-Horizon Continual Learning Introduces HaMo, a continual-learning framework for web agents that retrieves strategy cards from past workflows and updates model parameters to reuse them. - https://awesomecua.com/resources/arxiv-2610-34724/ · source: https://arxiv.org/abs/2610.34724v1 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, workflow, long-horizon, web-navigation - Published: 2026-10-26 · Authors: Zhen Bi, Zhizhi Peng, Yan Wang, Xueshu Chen, Haofei Zhang, Yaobo Zhang, Zeyu Yang, Qing Shen, Jungang Lou ### Harnessing Webpage UIs for Text-Rich Visual Understanding Builds MultiUI by generating instructions from webpage accessibility trees and pairing them with screenshots for multimodal training. Evaluation examines web-interface understanding, including VisualWebBench and Mind2Web, and transfer to other text-rich visual tasks. - https://awesomecua.com/resources/arxiv-2410-13824/ · source: https://arxiv.org/abs/2410.13824v3 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, screen-understanding, grounding - Published: 2024-10-17 · Authors: Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, Xiang Yue ### HATS: Hardness-Aware Trajectory Synthesis for GUI Agents Introduces HATS, which directs GUI trajectory exploration toward semantically ambiguous actions and iteratively checks instruction-execution alignment. Feedback from trajectory refinement updates the exploration signal used to collect subsequent training examples. - https://awesomecua.com/resources/arxiv-2603-12138/ · source: https://arxiv.org/abs/2603.12138v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, robustness, action-prediction - Published: 2026-03-12 · Authors: Rui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen ### HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks Introduces HealthAdminBench with GUI environments for an electronic health record, payer portals and fax handling. Expert-defined administrative tasks are decomposed into verifiable subtasks to compare local progress with end-to-end workflow completion. - https://awesomecua.com/resources/arxiv-2604-09937/ · source: https://arxiv.org/abs/2604.09937v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, workflow, industry - Published: 2026-04-10 · Authors: Suhana Bedi, Ryan Welch, Ethan Steinberg, Michael Wornow, Taeil Matthew Kim, Haroun Ahmed, Peter Sterling, Bravim Purohit, Qurat Akram, Angelic Acosta, Esther Nubla, Pritika Sharma, Michael A. Pfeffer, Sanmi Koyejo, Nigam H. Shah ### Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control Introduces Hi-Agent, jointly training a high-level reasoning model and a low-level mobile action model. Execution feedback supplies a foresight advantage for optimizing subgoal decisions, with evaluations on Android control benchmarks. - https://awesomecua.com/resources/arxiv-2510-14388/ · source: https://arxiv.org/abs/2510.14388v2 - Paper · reviewed 2026-10-17 · environments: android · topics: planning, reinforcement-learning, mobile-navigation - Published: 2025-10-16 · Authors: see the paper ### HiconAgent: History Context-aware Policy Optimization for GUI Agents HiconAgent trains GUI policies to use history through variable-length context sampling and paired policy updates with compressed and complete histories. Its compression retains past actions while removing historical observations, and an alignment loss couples the two branches. - https://awesomecua.com/resources/arxiv-2512-01763/ · source: https://arxiv.org/abs/2512.01763v2 - Paper · reviewed 2026-10-17 · environments: android · topics: interaction-history, memory, efficiency - Published: 2025-12-01 · Authors: Xurui Zhou, Gongwei Chen, Yuquan Xie, Zaijing Li, Kaiwen Zhou, Shuai Wang, Shuo Yang, Zhuotao Tian, Rui Shao ### Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents Studies backdoor risks in mobile GUI agents through AgentGhost, which combines goal-level and interaction-history triggers during model training. The paper evaluates multiple mobile agents and a tailored defense. - https://awesomecua.com/resources/arxiv-2505-14418/ · source: https://arxiv.org/abs/2505.14418v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, mobile-navigation, interaction-history - Published: 2025-05-20 · Authors: Pengzhou Cheng, Haowen Hu, Zheng Wu, Zongru Wu, Tianjie Ju, Zhuosheng Zhang, Gongshen Liu ### Hierarchical Prompting Assists Large Language Model on Web Navigation Separates web navigation prompting into a SUMMARIZER that extracts action-relevant observations and an ACTOR that chooses the next action from the condensed context. - https://awesomecua.com/resources/arxiv-2305-14257/ · source: https://arxiv.org/abs/2305.14257v3 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, planning, efficiency - Published: 2023-05-23 · Authors: Abishek Sridhar, Robert Lo, Frank F. Xu, Hao Zhu, Shuyan Zhou ### HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents HiPER trains interactive LLM agents with separate subgoal planning and multi-step execution, assigning reinforcement-learning credit at both levels. - https://awesomecua.com/resources/arxiv-2602-16165/ · source: https://arxiv.org/abs/2602.16165v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, planning, long-horizon - Published: 2026-02-18 · Authors: Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou, Charles Fleming, Zhaoran Wang, Alfredo Garcia, Mingyi Hong ### HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games HiSTEVE uses a hierarchical planner and a vision-only keyboard-and-mouse controller for Minecraft tasks that include GUI and non-GUI interactions. - https://awesomecua.com/resources/arxiv-2610-31162/ · source: https://arxiv.org/abs/2610.31162v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: planning, long-horizon, state-tracking, imitation-learning - Published: 2026-10-22 · Authors: Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai ### History-Aware Reasoning for GUI Agents HAR-GUI combines episodic reasoning and short-term memory to use earlier interactions when selecting GUI actions. Its training constructs reflective scenarios and correction guidance and applies a hybrid reinforcement-learning reward to the 3B policy. - https://awesomecua.com/resources/arxiv-2511-09127/ · source: https://arxiv.org/abs/2511.09127v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: interaction-history, memory, self-reflection, reinforcement-learning - Published: 2025-11-12 · Authors: Ziwei Wang, Leyang Yang, Xiaoxuan Tang, Sheng Zhou, Dajun Chen, Wei Jiang, Yong Li ### HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget Describes HobbyLM, a small model family whose post-trained variants include a computer-use agent and an on-device macOS computer-use adapter. - https://awesomecua.com/resources/arxiv-2610-34582/ · source: https://arxiv.org/abs/2610.34582v1 - Paper · reviewed 2026-11-06 · environments: desktop, macos · topics: action-prediction, tool-integration - Published: 2026-10-25 · Authors: Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan R ### How Auxiliary Reasoning Unleashes GUI Grounding in VLMs Studies the gap between latent visual grounding and explicit coordinate output in general vision-language models. Three zero-shot methods add axes, grids, or labeled intersections to GUI screenshots so models can express spatial locations without fine-tuning. - https://awesomecua.com/resources/arxiv-2509-11548/ · source: https://arxiv.org/abs/2509.11548v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, screen-understanding - Published: 2025-09-15 · Authors: Weiming Li, Yan Shao, Jing Yang, Yujing Lu, Ling Zhong, Yuhan Wang, Min Yu, Tongxiao Ruan, Manni Duan ### How Benchmarks Mis-Score Computer-Use Agents Audits failure-scored computer-use trajectories to distinguish agent errors from broken tasks and evaluator false negatives. It organizes reliability issues across task construction, trajectory evidence, scoring, and reporting, then derives corresponding evaluation design practices. - https://awesomecua.com/resources/arxiv-2607-28367/ · source: https://arxiv.org/abs/2607.28367v1 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: evaluation, observability, robustness - Published: 2026-07-30 · Authors: Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li, Zirou Liu, Ruixuan Deng, Qishi Zhan, Rui Qian ### How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors Introduces VAF, a controlled pipeline that varies webpage appearance while retaining item semantics. It measures both agent clicks and reasoning references to study which visual design factors change browsing decisions. - https://awesomecua.com/resources/arxiv-2601-21961/ · source: https://arxiv.org/abs/2601.21961v2 - Paper · reviewed 2026-10-17 · environments: web · topics: ui-design, evaluation, web-navigation - Published: 2026-01-29 · Authors: Kuai Yu, Naicheng Yu, Han Wang, Rui Yang, Huan Zhang ### How GUI Agents Fail: GraceBench for Execution Stability Diagnosis Introduces GraceBench, using application transition graphs and recorded interaction evidence to diagnose deviations, loops, and recovery that final task-success scores can hide. - https://awesomecua.com/resources/arxiv-2610-19900/ · source: https://arxiv.org/abs/2610.19900v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, mobile-navigation, robustness - Published: 2026-10-14 · Authors: Ridong Zhang, Shuxian Liang, Chaoqun Wan, Zhen Huang, Xian-Sheng Hua ### How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation PACE weights coordinate-digit entropy by place value to estimate confidence in GUI clicks from one forward pass. It evaluates correctness ranking and selective accuracy against repeated-sampling alternatives on ScreenSpot-Pro and ScreenSpot-v2. - https://awesomecua.com/resources/arxiv-2609-24277/ · source: https://arxiv.org/abs/2609.24277v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: grounding, efficiency, evaluation, robustness - Published: 2026-09-21 · Authors: Yunxiang Li, Xixin Wu, Helen Meng ### How Mobile World Model Guides GUI Agents? Compares mobile world models that predict state changes as delta text, full text, images or renderable code. Downstream Android benchmarks examine synthetic training trajectories and test-time guidance, including limits of post-action reflection. - https://awesomecua.com/resources/arxiv-2605-10347/ · source: https://arxiv.org/abs/2605.10347v2 - Paper · reviewed 2026-10-17 · environments: android · topics: world-models, data-generation, evaluation - Published: 2026-05-11 · Authors: Weikai Xu, Kun Huang, Yunren Feng, Jiaxing Li, Yuhan Chen, Yuxuan Liu, Zhizheng Jiang, Heng Qu, Pengzhi Gao, Wei Liu, Jian Luan, Xiaolin Hu, Bo An ### How Smart Is Your GUI Agent? A Framework for the Future of Software Interaction Proposes GUI Agent Autonomy Levels (GAL), a six-level framework for describing autonomy and benchmarking GUI agents that navigate web, desktop, and mobile software. - https://awesomecua.com/resources/arxiv-2602-11514/ · source: https://arxiv.org/abs/2602.11514v1 - Paper · reviewed 2026-11-06 · environments: web, desktop · topics: evaluation, safety-security - Published: 2026-02-12 · Authors: Sidong Feng, Chunyang Chen ### How to Train Your LLM Web Agent: A Statistical Diagnosis Studies compute allocation for web-agent post-training with teacher-generated supervised fine-tuning followed by on-policy reinforcement learning. It samples 1,370 training configurations and uses bootstrap analysis to examine hyperparameter sensitivity on WorkArena and MiniWoB++. - https://awesomecua.com/resources/arxiv-2507-04103/ · source: https://arxiv.org/abs/2507.04103v4 - Paper · reviewed 2026-10-17 · environments: web · topics: imitation-learning, reinforcement-learning, efficiency, evaluation - Published: 2025-07-05 · Authors: Dheeraj Vattikonda, Santhoshi Ravichandran, Emiliano Penaloza, Hadi Nekoei, Megh Thakkar, Thibault Le Sellier de Chezelles, Nicolas Gontier, Miguel Muñoz-Mármol, Sahar Omidi Shayegan, Stefania Raimondo, Xue Liu, Alexandre Drouin, Laurent Charlin, Alexandre Piché, Alexandre Lacoste, Massimo Caccia ### How To Train Your World Model: Fine-tuning vs RAG for LM-based World Modeling Compares fine-tuned, retrieval-based and hybrid language-model world models for next-observation prediction and action selection in text environments including WebShop. - https://awesomecua.com/resources/arxiv-2610-02542/ · source: https://arxiv.org/abs/2610.02542v1 - Paper · reviewed 2026-10-20 · environments: web · topics: world-models, planning, evaluation, web-navigation - Published: 2026-10-01 · Authors: Dhananjay Ashok, Shantanu Agarwal, Vivek Datla, Jonathan May, Alfy Samuel - Licenses: document CC-BY-4.0 ### How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition This public red-teaming study evaluates indirect prompt injection that both changes agent behavior and hides the compromise from the final response. Its scenarios include computer use alongside tool calling and coding, with analyses of transfer and model-dependent vulnerability. - https://awesomecua.com/resources/arxiv-2603-15714/ · source: https://arxiv.org/abs/2603.15714v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, evaluation, robustness - Published: 2026-03-16 · Authors: Mateusz Dziemian, Maxwell Lin, Xiaohan Fu, Micha Nowak, Nick Winter, Eliot Jones, Andy Zou, Lama Ahmad, Kamalika Chaudhuri, Sahana Chennabasappa, Xander Davies, Lauren Deason, Benjamin L. Edelman, Tanner Emek, Ivan Evtimov, Jim Gust, Maia Hamin, Kat He, Klaudia Krawiecka, Riccardo Patana, Neil Perry, Troy Peterson, Xiangyu Qi, Javier Rando, Zifan Wang, Zihan Wang, Spencer Whitman, Eric Winsor, Arman Zharmagambetov, Matt Fredrikson, Zico Kolter ### Human-Guided Harm Recovery for Computer Use Agents Introduces a preference-trained reward model for guiding computer-use agents out of harmful states and BackBench, a benchmark of recovery tasks. - https://awesomecua.com/resources/arxiv-2604-18847/ · source: https://arxiv.org/abs/2604.18847v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, reward-models, human-in-the-loop, evaluation - Published: 2026-04-20 · Authors: Christy Li, Sky CH-Wang, Andi Peng, Andreea Bobu ### Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents Introduces Hybrid Computer Use (HCU), a desktop agent interface combining code, accessibility data, and GUI actions, with AX diffs reporting action effects. - https://awesomecua.com/resources/arxiv-2610-41432/ · source: https://arxiv.org/abs/2610.41432v1 - Paper · reviewed 2026-11-06 · environments: desktop · topics: tool-integration, efficiency, state-tracking - Published: 2026-10-30 · Authors: Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu ### Hybrid Self-evolving Structured Memory for GUI Agents Introduces HyMEM, a graph-based memory for GUI agents that links symbolic task knowledge with trajectory embeddings and refreshes working memory during computer-use tasks. - https://awesomecua.com/resources/arxiv-2603-10291/ · source: https://arxiv.org/abs/2603.10291v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, interaction-history, long-horizon - Published: 2026-03-11 · Authors: Sibo Zhu, Wenyi Wu, Kun Zhou, Stephen Wang, Biwei Huang ### HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents Trains computer agents on verified GUI, command-line, and interleaved trajectories to learn when and how to combine interaction modes. - https://awesomecua.com/resources/arxiv-2609-38008/ · source: https://arxiv.org/abs/2609.38008v1 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: tool-integration, reinforcement-learning, data-generation - Published: 2026-09-29 · Authors: Tongbo Chen, Junbo Niu, Zhengxi Lu, Niu Lian, Fei Tang, Yuchen Yan, Yike Hong, Yong Du, Yizhou Liu, Bofan Chen, Yongliang Shen ### HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents Combines GUI perception data, instructional videos, real and simulated devices, and planning with reflection to develop a mobile agent. - https://awesomecua.com/resources/arxiv-2607-14548/ · source: https://arxiv.org/abs/2607.14548v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, data-generation, planning - Published: 2026-07-16 · Authors: Hy Vision Team, Huawen Shen, Zhengyang Tang, Shangpin Peng, Liang Wu, Anran Zhang, Weinong Wang, Yiduo Guo, Chenxin Li, Zhengyao Fang, Yang Ding, Junyi Li, Fei Tang, Zheng Ruan, Yi Zhang, Xingran Zhou, Dingchen Yang, Sunqi Fan, Zhiyi Wan, Han Hu, Xin Lai, Pengyuan Lyu, Chengquan Zhang ### HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents Introduces HyperBrowseComp, a multilingual benchmark for browsing agents that must connect evidence across open-web sources, including videos, scans, images, and maps. - https://awesomecua.com/resources/arxiv-2610-03574/ · source: https://arxiv.org/abs/2610.03574v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, long-horizon - Published: 2026-10-02 · Authors: Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri, Seung Hun Eddie Han, Ryandito Diandaru, Qinrong Cui, Jan Christian Blaise Cruz, Badrinath Chandana, Peerawat Chomphooyod, Ahmed Attia, Jonibek Mansurov, Emilio Villa-Cueva, Canh Duong Nguyen, Imran Turganov, Minghao Wu, Peerat Limkonchotiwat, Irina Nikishina ### HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks Introduces HyWe, which switches between information search and browser operation within one trajectory, alongside a demonstration pipeline and a benchmark of hybrid web tasks. - https://awesomecua.com/resources/arxiv-2610-16319/ · source: https://arxiv.org/abs/2610.16319v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, planning, data-generation - Published: 2026-10-12 · Authors: Zijian Cai, Haowei Liu, Kailai Chen, Lin Peng, Cong Wan, Ziqi Zhang, Chunfeng Yuan, Bing Li, Weiming Hu, Haiyang Xu ### ICA: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agents Information-Aware Credit Assignment scores intermediate information gathered by web agents, using rendered webpage snapshots to compare evidence across trajectories. - https://awesomecua.com/resources/arxiv-2602-10863/ · source: https://arxiv.org/abs/2602.10863v2 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, reward-models, web-navigation, long-horizon - Published: 2026-02-11 · Authors: Cong Pang, Xuyu Feng, Yujie Yi, Jiaqi Su, Zixuan Chen, Jiawei Hong, Tiankuo Yao, Nang Yuan, Jiapeng Luo, Lewei Lu, Xin Lou ### IDA: Breaking Barriers in No-code UI Automation Through Large Language Models and Human-Centric Design IDA helps business users create web UI automation through guided demonstrations, a semantic programming model, and a teaching-oriented interface. A prototype user study examines automation creation with enterprise applications by users without programming backgrounds. - https://awesomecua.com/resources/arxiv-2407-15673/ · source: https://arxiv.org/abs/2407.15673v3 - Paper · reviewed 2026-10-17 · environments: web · topics: human-in-the-loop, automation, workflow, industry - Published: 2024-07-22 · Authors: Segev Shlomov, Avi Yaeli, Sami Marreed, Sivan Schwartz, Netanel Eder, Offer Akrabi, Sergey Zeltyn ### Identifying User Goals from UI Trajectories Defines goal identification from observed UI trajectories and an evaluation method for comparing intent descriptions within the same interface context. Experiments reuse Android and web automation data to compare human and model inference of users’ detailed goals. - https://awesomecua.com/resources/arxiv-2406-14314/ · source: https://arxiv.org/abs/2406.14314v3 - Paper · reviewed 2026-10-17 · environments: android, web, cross-platform · topics: evaluation, interaction-history, screen-understanding - Published: 2024-06-20 · Authors: Omri Berkovitch, Sapir Caduri, Noam Kahlon, Anatoly Efros, Avi Caciularu, Ido Dagan ### IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints Introduces Interactive GUI Grounding (IGG), a benchmark that tests whether agents can reveal hidden interface targets through interaction before clicking them. - https://awesomecua.com/resources/arxiv-2610-32536/ · source: https://arxiv.org/abs/2610.32536v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, grounding, screen-understanding - Published: 2026-10-23 · Authors: Kyeongseon Kim, Jiyeon Son, Tae-Hyun Oh ### ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations ILuvUI generates 335,000 conversational training examples paired with UI screenshots, covering questions, descriptions, and plans. The resulting vision-language model is assessed on element detection, response quality, and examples of multi-step UI navigation. - https://awesomecua.com/resources/arxiv-2310-04869/ · source: https://arxiv.org/abs/2310.04869v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, screen-understanding, grounding, planning - Published: 2023-10-07 · Authors: Yue Jiang, Eldon Schoop, Amanda Swearngin, Jeffrey Nichols ### Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training Introduces Infinite-Dreamer, which learns action-conditioned screenshot editing from structured descriptions of GUI changes and uses it to synthesize robustness examples and multi-step trajectories. The synthetic data trains a mobile GUI agent evaluated on AndroidWorld, MobileWorld, and AndroidControl-Curated. - https://awesomecua.com/resources/arxiv-2610-05861/ · source: https://arxiv.org/abs/2610.05861v1 - Paper · reviewed 2026-10-17 · environments: android · topics: world-models, data-generation, mobile-navigation - Published: 2026-10-05 · Authors: Yongxin Ning, Runliang Niu, Qianli Xing, Zhiyi Duan, Qingzu He, Pan Wang, Qi Wang ### Improved GUI Grounding via Iterative Narrowing Uses iterative visual narrowing to improve localization of requested GUI elements with general or specially trained vision-language models. - https://awesomecua.com/resources/arxiv-2411-13591/ · source: https://arxiv.org/abs/2411.13591v7 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding - Published: 2024-11-18 · Authors: Anthony Nguyen ### Improving GUI Grounding with Explicit Position-to-Coordinate Mapping This work addresses GUI localization errors caused by mismatches between visual patch positions and output pixel coordinates. RULER tokens provide explicit coordinate references, while Interleaved MRoPE balances width and height encoding; evaluation uses the ScreenSpot benchmark family. - https://awesomecua.com/resources/arxiv-2510-03230/ · source: https://arxiv.org/abs/2510.03230v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, robustness - Published: 2025-10-03 · Authors: Suyuchen Wang, Tianyu Zhang, Ahmed Masry, Christopher Pal, Spandana Gella, Bang Liu, Perouz Taslakian ### Improving Language Understanding from Screenshots Studies language understanding from text-rendered screenshots using joint image-patch and text prediction; a perception foundation relevant to screen-reading models. - https://awesomecua.com/resources/arxiv-2402-14073/ · source: https://arxiv.org/abs/2402.14073v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: screen-understanding - Published: 2024-02-21 · Authors: Tianyu Gao, Zirui Wang, Adithya Bhaskar, Danqi Chen ### Improving LLM-based Autonomous Web Agents with Filtering Trains HTML-element relevance filters on Mind2Web trajectories and transfers them to WebArena to reduce irrelevant browser context. It also evaluates a zero-shot ColBERT retriever and documents baseline web-agent failure modes. - https://awesomecua.com/resources/arxiv-2609-27770/ · source: https://arxiv.org/abs/2609.27770v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, screen-understanding, efficiency, evaluation - Published: 2026-08-16 · Authors: Zhitong Guo, Jing Yu Koh, Ruiyu Li ### Improving Proficiency and Efficiency of Android GUI Agents via Self-Generating Tool Actions DroidTool generates application-state tools through proposal, implementation, testing and repair. Relational tests connect the tools’ preconditions and effects, and the resulting tools are evaluated as additions to Android GUI agents. - https://awesomecua.com/resources/arxiv-2609-06792/ · source: https://arxiv.org/abs/2609.06792v1 - Paper · reviewed 2026-10-17 · environments: android · topics: tool-integration, automation, mobile-navigation, efficiency - Published: 2026-09-06 · Authors: Juyong Lee, Woogyeol Jin, Kimin Lee ### In-Browser LLM-Guided Fuzzing for Real-Time Prompt Injection Testing in Agentic AI Browsers Proposes an in-browser fuzzing framework that uses language-model guidance to test indirect prompt-injection vulnerabilities in agentic browsers. The work focuses on unwanted actions triggered by webpage instructions under the browser agent's user privileges. - https://awesomecua.com/resources/arxiv-2510-13543/ · source: https://arxiv.org/abs/2510.13543v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, automation - Published: 2025-10-15 · Authors: Avihay Cohen ### In-Context Defense in Computer Agents: An Empirical Study Studies in-context defenses for computer agents exposed to misleading pop-ups and other environmental instructions. It examines defensive reasoning before action planning and how a small set of examples changes resistance to these attacks. - https://awesomecua.com/resources/arxiv-2503-09241/ · source: https://arxiv.org/abs/2503.09241v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, evaluation - Published: 2025-03-12 · Authors: Pei Yang, Hai Ci, Mike Zheng Shou ### Inducing Programmatic Skills for Agentic Tasks Introduces agent skill induction (ASI), which creates, checks, and reuses program-based skills for web tasks by combining primitive actions into higher-level routines. - https://awesomecua.com/resources/arxiv-2504-06821/ · source: https://arxiv.org/abs/2504.06821v2 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, automation, efficiency, memory - Published: 2025-04-09 · Authors: Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, Daniel Fried ### Inducing Task Models from Computer-Use Traces Task Model Induction disentangles concurrent activities in screenshot and mouse/keyboard traces, then constructs hierarchical goals and procedural control flow for each task. Evaluation tests reconstruction of observed execution and reuse of derived skills on held-out tasks. - https://awesomecua.com/resources/arxiv-2608-20319/ · source: https://arxiv.org/abs/2608.20319v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: recording, workflow, memory, interaction-history - Published: 2026-08-20 · Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang ### InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction Introduces InfantAgent-Next, a modular multimodal computer agent that combines tool-based and vision agents for stepwise tasks, with evaluation on OSWorld and other benchmarks. - https://awesomecua.com/resources/arxiv-2505-10887/ · source: https://arxiv.org/abs/2505.10887v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: tool-integration, planning - Published: 2025-05-16 · Authors: Bin Lei, Weitai Kang, Zijian Zhang, Winson Chen, Xi Xie, Shan Zuo, Mimi Xie, Ali Payani, Mingyi Hong, Yan Yan, Caiwen Ding ### Inferring Alt-text For UI Icons With Large Language Models During App Development Introduces IconDesc, which generates mobile icon alt-text from partial UI context such as element identifiers, bounds, nearby text and hierarchy. Model evaluation and a user study examine the resulting accessibility descriptions. - https://awesomecua.com/resources/arxiv-2409-18060/ · source: https://arxiv.org/abs/2409.18060v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, ui-design, evaluation - Published: 2024-09-26 · Authors: Sabrina Haque, Christoph Csallner ### InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization Uses multiple candidate answers and adaptive exploration rewards to improve spatial and semantic alignment in GUI grounding. - https://awesomecua.com/resources/arxiv-2508-05731/ · source: https://arxiv.org/abs/2508.05731v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, reinforcement-learning - Published: 2025-08-07 · Authors: Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xavier Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, Shengyu Zhang, Hongxia Yang, Fei Wu ### InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners Uses a staged reasoning-focused training framework to move GUI agents from reactive actions toward deliberative planning and recovery. - https://awesomecua.com/resources/arxiv-2504-14239/ · source: https://arxiv.org/abs/2504.14239v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: planning, reinforcement-learning - Published: 2025-04-19 · Authors: Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, Fei Wu ### InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection Uses staged supervised training to combine GUI understanding and grounding with hierarchical reasoning and expectation-based reflection. - https://awesomecua.com/resources/arxiv-2501-04575/ · source: https://arxiv.org/abs/2501.04575v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning - Published: 2025-01-08 · Authors: Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, Fei Wu ### InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training InfiniteWeb generates functional, interconnected websites and task evaluators using shared specifications and task-driven development. It evaluates website construction and the transfer of GUI policies trained in these synthetic web environments to OSWorld and Online-Mind2Web. - https://awesomecua.com/resources/arxiv-2601-04126/ · source: https://arxiv.org/abs/2601.04126v3 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: data-generation, reinforcement-learning, evaluation, sandbox - Published: 2026-01-07 · Authors: Ziyun Zhang, Zezhou Wang, Xiaoyi Zhang, Zongyu Guo, Jiahao Li, Bin Li, Yan Lu ### Infogent: An Agent-Based Framework for Web Information Aggregation Infogent separates navigation, extraction and aggregation for web information gathering, including browser interaction through webpage screenshots. - https://awesomecua.com/resources/arxiv-2410-19054/ · source: https://arxiv.org/abs/2410.19054v1 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, tool-integration, workflow - Published: 2024-10-24 · Authors: Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, Heng Ji ### InfraMind: A Novel Exploration-based GUI Agentic Framework for Mission-critical Industrial Management InfraMind explores industrial management GUIs using virtual-machine snapshots and builds memory for planning and hierarchical state recognition. Its framework also studies knowledge distillation and safety checks on open-source and commercial data-center management platforms. - https://awesomecua.com/resources/arxiv-2509-13704/ · source: https://arxiv.org/abs/2509.13704v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: industry, planning, memory, safety-security - Published: 2025-09-17 · Authors: Liangtao Lin, Zhaomeng Zhu, Tianwei Zhang, Yonggang Wen ### InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning InquireMobile trains mobile agents to seek human confirmation at critical decisions using staged reinforcement fine-tuning and pre-action reasoning. InquireBench evaluates safe interaction and proactive inquiry, distinguishing asking for help from completing the underlying task. - https://awesomecua.com/resources/arxiv-2508-19679/ · source: https://arxiv.org/abs/2508.19679v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, reinforcement-learning, evaluation, mobile-navigation - Published: 2025-08-27 · Authors: Qihang Ai, Pi Bu, Yue Cao, Yingyao Wang, Jihao Gu, Jingxuan Xing, Zekun Zhu, Wei Jiang, Zhicheng Zheng, Jun Song, Yuning Jiang ### InSTA: Towards Internet-Scale Training For Agents InSTA builds a pipeline that creates website tasks, collects web-agent trajectories and filters them by judged success to train smaller web navigation agents. - https://awesomecua.com/resources/arxiv-2502-06776/ · source: https://arxiv.org/abs/2502.06776v2 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, web-navigation, imitation-learning, evaluation - Published: 2025-02-10 · Authors: Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, Ruslan Salakhutdinov ### InstructExcel: A Benchmark for Natural Language Instruction in Excel Introduces InstructExcel, a benchmark for generating Excel OfficeScripts from natural-language requests, built using scripts recorded from user actions and a collection of existing spreadsheets. - https://awesomecua.com/resources/arxiv-2310-14495/ · source: https://arxiv.org/abs/2310.14495v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, workflow, automation - Published: 2023-10-23 · Authors: Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, Elnaz Nouri ### Instruction Agent: Enhancing Agent with Expert Demonstration Introduces Instruction Agent, a GUI agent that follows instructions extracted from one expert demonstration and uses verification and backtracking to handle interruptions. - https://awesomecua.com/resources/arxiv-2509-07098/ · source: https://arxiv.org/abs/2509.07098v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: imitation-learning, long-horizon, state-tracking, self-reflection - Published: 2025-09-08 · Authors: Yinheng Li, Hailey Hultquist, Justin Wagle, Kazuhito Koishida ### Intelligent Virtual Assistants with LLM-based Process Automation Presents LLMPA, a mobile process-automation architecture that decomposes requests, identifies interface elements, predicts actions and checks errors. Demonstrations evaluate multistep operations in Alipay. - https://awesomecua.com/resources/arxiv-2312-06677/ · source: https://arxiv.org/abs/2312.06677v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, planning, automation - Published: 2023-12-04 · Authors: Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua Song, Longfei Li, Jinjie Gu, Chenyi Zhuang ### IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents IntentCUA coordinates planner and critic agents through shared intent-aligned memory, retrieving reusable skills for long-horizon automation across desktop applications. - https://awesomecua.com/resources/arxiv-2602-17049/ · source: https://arxiv.org/abs/2602.17049v2 - Paper · reviewed 2026-11-06 · environments: desktop · topics: memory, planning, cross-app, long-horizon - Published: 2026-02-19 · Authors: Seoyoung Lee, Seobin Yoon, Seongbeen Lee, Yoojung Chun, Dayoung Park, Doyeon Kim, Joo Yong Sim ### IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents Trains IntentScore on offline desktop GUI interactions to assess candidate actions using both their state relevance and planning intent. The reward model is evaluated as an action reranker for Agent S3 on OSWorld. - https://awesomecua.com/resources/arxiv-2604-05157/ · source: https://arxiv.org/abs/2604.05157v3 - Paper · reviewed 2026-10-17 · environments: desktop · topics: reward-models, planning, evaluation - Published: 2026-04-06 · Authors: Rongqian Chen, Yu Li, Zeyu Fang, Sizhe Tang, Weidong Cao, Tian Lan ### IntentWeave: A Progressive Entry Ladder for Multi-Surface Browser Agents in Cloud Portals Introduces IntentWeave, a design space and browser-extension prototype for placing web-agent assistance across browser surfaces, evaluated through a study of three entry strategies. - https://awesomecua.com/resources/arxiv-2603-22917/ · source: https://arxiv.org/abs/2603.22917v1 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, human-in-the-loop, web-navigation, evaluation - Published: 2026-03-24 · Authors: Wanying Mo, Jijia Lai, Xiaoming Wang ### InterA11y: Interactive Web Accessibility Error Detection with LLM Agents Introduces InterA11y, a web accessibility evaluation system that guides LLM agents with rubrics and tools for inspecting and operating pages. - https://awesomecua.com/resources/arxiv-2610-39488/ · source: https://arxiv.org/abs/2610.39488v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, tool-integration - Published: 2026-10-29 · Authors: Mingyuan Zhong, Mengqi Shi, Davin Win Kyi, Ajit Mallavarapu, Xia Chen, James Fogarty, Jacob O. Wobbrock ### Interaction-as-a-Judge Uses computer-use agents to interact with generated applications, studying whether interaction improves preference judging, requirement checks, and feedback for application generation. - https://awesomecua.com/resources/arxiv-2610-17895/ · source: https://arxiv.org/abs/2610.17895v2 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: evaluation, data-generation, automation - Published: 2026-10-13 · Authors: Wayne Chi, I-Hung Hsu, Li Chen, Wei-Lin Chiang, Ion Stoica, Anastasios Nikolas Angelopoulos, Ameet Talwalkar, Chris Donahue, Cho-Jui Hsieh ### Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification Introduces an Interactive Reward Agent that proposes completion conditions and verifies them using system, application, and GUI tools after execution. GUI-RewardBench provides annotated desktop trajectories for evaluating these evidence-based judgments and their use in reinforcement learning. - https://awesomecua.com/resources/arxiv-2607-25904/ · source: https://arxiv.org/abs/2607.25904v2 - Paper · reviewed 2026-10-17 · environments: desktop, linux · topics: reward-models, evaluation, tool-integration - Published: 2026-07-28 · Authors: Chenrui Shi, Yuwei Wu, Yang Liu, Ruining Feng, Zirui Shang, Zhi Gao, Lifeng Fan, Che Sun ### Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interface Studies agent-system and interface co-design that incorporates user interruptions and intermediate feedback to reduce planning latency. - https://awesomecua.com/resources/arxiv-2410-00079/ · source: https://arxiv.org/abs/2410.00079v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: planning, efficiency, human-in-the-loop - Published: 2024-09-30 · Authors: Wenyue Hua, Mengting Wan, Shashank Vadrevu, Ryan Nadel, Yongfeng Zhang, Chi Wang ### Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations Introduces PUMICE, an agent that learns mobile-app procedures and conditional concepts through natural-language conversation and GUI demonstrations, recursively resolving ambiguous instructions with users. - https://awesomecua.com/resources/arxiv-1909-00031/ · source: https://arxiv.org/abs/1909.00031v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, mobile-navigation, imitation-learning - Published: 2019-08-30 · Authors: Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, Brad A. Myers ### Interface Quilt: Modifying Applications with Computer Use Agents and Generative UI Introduces Interface Quilt, which uses computer-use agents and generated UI overlays to modify how existing applications look and behave without source access. - https://awesomecua.com/resources/arxiv-2611-01891/ · source: https://arxiv.org/abs/2611.01891v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: ui-design, automation - Published: 2026-11-01 · Authors: Leping Qiu, Jo Vermeulen, Tim Zindulka, George Fitzmaurice, Fraser Anderson ### InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency Introduces InternVL3.5, a multimodal model family trained with Cascade RL and designed for more efficient inference; GUI interaction is among its stated capabilities. - https://awesomecua.com/resources/arxiv-2508-18265/ · source: https://arxiv.org/abs/2508.18265v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, efficiency - Published: 2025-08-25 · Authors: Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Zhi Hou, Haoran Hao, Tianyi Zhang, Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjun He, Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang, Huipeng Deng, Biqing Qi, Jiaye Ge, Qipeng Guo, Wenwei Zhang, Songyang Zhang, Maosong Cao, Junyao Lin, Kexian Tang, Jianfei Gao, Haian Huang, Yuzhe Gu, Chengqi Lyu, Huanze Tang, Rui Wang, Haijun Lv, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, Gen Luo ### Interpretable Early Termination of Web Navigation Agents via Closed Sequential Pattern Mining Applies sequential pattern mining to MiniWoB++ web-agent traces, producing reviewable failure signatures and an early-termination policy to reduce token use. - https://awesomecua.com/resources/arxiv-2611-04142/ · source: https://arxiv.org/abs/2611.04142v1 - Paper · reviewed 2026-11-06 · environments: web · topics: observability, efficiency, evaluation - Published: 2026-11-03 · Authors: Sergio Talavera, Magdalini Eirinaki ### InterWebCoder: Learning Interactive Webpage Generation from Executable Tests via Reinforcement Learning InterWebCoder trains models to generate interactive webpage code using recorded interactions, executable tests and repair feedback, an adjacent web interface task. - https://awesomecua.com/resources/arxiv-2610-28610/ · source: https://arxiv.org/abs/2610.28610v1 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, data-generation, reinforcement-learning, recording - Published: 2026-10-20 · Authors: Jingyu Xiao, Dasen Dai, Yuxuan Wan, Jen-tse Huang, Yintong Huo, Hongyu Zhang, Michael R. Lyu ### Investigating the Impact of Dark Patterns on LLM-Based Web Agents Presents LiteAgent for recording web-agent interactions and TrickyArena for enabling or disabling dark patterns in controlled websites. Experiments compare how individual patterns, combined patterns and visual or HTML changes affect agent decisions. - https://awesomecua.com/resources/arxiv-2510-18113/ · source: https://arxiv.org/abs/2510.18113v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, ui-design, evaluation - Published: 2025-10-20 · Authors: Devin Ersoy, Brandon Lee, Ananth Shreekumar, Arjun Arunasalam, Muhammad Ibrahim, Antonio Bianchi, Z. Berkay Celik ### Invisible in Space, Visible in Time: Motion Vision CAPTCHA against GUI Agents MVCAP-Bench tests live browser CAPTCHAs whose foreground meaning emerges through motion against dynamic backgrounds. A foreground-only control separates temporal camouflage difficulty from answer format and browser interaction. - https://awesomecua.com/resources/arxiv-2609-27461/ · source: https://arxiv.org/abs/2609.27461v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, screen-understanding, video, safety-security - Published: 2026-09-23 · Authors: Zeyu Zhang, Dingyi Rong, Zijian Chen, Zicheng Zhang, Xiongkuo Min, Guangtao Zhai ### Invisible Ink Threats: Adversarial Goals Behind Legitimate Tasks in Computer-Use Agents Introduces II-Bench for low-harm indirect prompt injections against computer-use agents and HITLCUA for testing them in virtual-machine and web environments with simulated user review. - https://awesomecua.com/resources/arxiv-2608-02018/ · source: https://arxiv.org/abs/2608.02018v2 - Paper · reviewed 2026-11-06 · environments: web, desktop, cross-platform · topics: safety-security, evaluation, human-in-the-loop, sandbox - Published: 2026-08-03 · Authors: Jia-Chen Zhang, Ze-Yu Zhang, Kai-Wei Zhang ### Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agents Uses the near-zero contact-touch signals of automated mobile agents to expose malicious visual content selectively during agent interactions. The paper combines this agent-only perceptual injection with HG-IDA* prompt optimization to study unauthorized cross-app actions. - https://awesomecua.com/resources/arxiv-2510-07809/ · source: https://arxiv.org/abs/2510.07809v4 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, mobile-navigation, robustness - Published: 2025-10-09 · Authors: Renhua Ding, Xiao Yang, Zhengwei Fang, Jun Luo, Kun He, Jun Zhu ### iOSWorld: A Benchmark for Personally Intelligent Phone Agents Introduces a native iOS simulator benchmark with persistent user data, multi-app tasks, and personalization-dependent phone workflows. - https://awesomecua.com/resources/arxiv-2606-09764/ · source: https://arxiv.org/abs/2606.09764v1 - Paper · reviewed 2026-10-16 · environments: ios · topics: evaluation, cross-app, memory - Published: 2026-06-08 · Authors: Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov ### iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents Introduces iPrefBench, an iOS benchmark for mobile agents that tests when they infer missing user preferences from activity history and when they ask the user. - https://awesomecua.com/resources/arxiv-2610-34735/ · source: https://arxiv.org/abs/2610.34735v1 - Paper · reviewed 2026-11-06 · environments: ios · topics: evaluation, human-in-the-loop, interaction-history, mobile-navigation - Published: 2026-10-26 · Authors: Yuting Ning, Ruijia Cheng, Sushobhan Ghosh, Peitong Duan, HungShi Lin, Abhishek Sundararajan, Shreyash Pandey, Huan Sun, Jeffrey Nichols, Eldon Schoop ### Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining Combines information-sensitive image cropping with a learning loop between element description and grounding for complex interface understanding. - https://awesomecua.com/resources/arxiv-2412-10342/ · source: https://arxiv.org/abs/2412.10342v2 - Paper · reviewed 2026-10-16 · environments: web, desktop · topics: grounding, screen-understanding, efficiency - Published: 2024-12-13 · Authors: Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, Yueting Zhuang ### Iron: Intent-Aligned and Retrospective Dual Learning Framework for Enhancing Generalist Virtual Agents Introduces Iron, a GUI-agent training framework that aligns low-level actions with high-level intent using cycle-consistency rewards. It also repurposes failed trajectories through hindsight reconstruction and evaluates transfer to other digital tasks. - https://awesomecua.com/resources/arxiv-2608-27866/ · source: https://arxiv.org/abs/2608.27866v1 - Paper · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, data-generation, grounding - Published: 2026-08-28 · Authors: Jiahe Ying, Wendong Bu, Kaihang Pan, Bingchen Miao, Siyu Chen, Wen Wang, Xueming Jiang, Juncheng Li, Siliang Tang ### Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents Introduces WebDreamer, which simulates candidate web actions with language-model world models and value estimates before acting. - https://awesomecua.com/resources/arxiv-2411-06559/ · source: https://arxiv.org/abs/2411.06559v2 - Paper · reviewed 2026-10-16 · environments: web · topics: world-models, planning - Published: 2024-11-10 · Authors: Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, Yu Su ### iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception Introduces iSHIFT, which combines latent reasoning with perception-control tokens to adapt GUI inference. The agent switches between detailed regional grounding and faster use of global visual cues. - https://awesomecua.com/resources/arxiv-2512-22009/ · source: https://arxiv.org/abs/2512.22009v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: efficiency, grounding, screen-understanding - Published: 2025-12-26 · Authors: Sarthak Mehrotra, Sairam V C Rebbapragada, Mani Hemanth Reddy Bonthu, Vineeth N Balasubramanian ### It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents Introduces TRAP, which evaluates task-redirection attacks embedded in realistic web interfaces. Controlled website clones and a modular injection framework support comparisons of persuasion techniques and contextual changes across agents. - https://awesomecua.com/resources/arxiv-2512-23128/ · source: https://arxiv.org/abs/2512.23128v3 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, ui-design - Published: 2025-12-29 · Authors: Karolina Korgul, Yushi Yang, Arkadiusz Drohomirecki, Piotr Błaszczyk, Will Howard, Lukas Aichberger, Chris Russell, Philip H. S. Torr, Adam Mahdi, Adel Bibi ### JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse Introduces Act from Visual Language Post-Training, a method for agents that follow instructions in Minecraft through visual input and actions, adjacent to interface operation. - https://awesomecua.com/resources/arxiv-2503-16365/ · source: https://arxiv.org/abs/2503.16365v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: action-prediction, imitation-learning - Published: 2025-03-20 · Authors: Muyao Li, Zihao Wang, Kaichen He, Xiaojian Ma, Yitao Liang ### JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition JarvisGUI composes typed input-output task transformations into dynamic workflows spanning Android, Windows and Ubuntu devices. Its virtual-environment benchmark tests transfer of intermediate results, shared state and cross-device dependencies. - https://awesomecua.com/resources/arxiv-2609-10451/ · source: https://arxiv.org/abs/2609.10451v1 - Paper · reviewed 2026-10-18 · environments: desktop, android, windows, linux · topics: evaluation, cross-app, planning, long-horizon, state-tracking - Published: 2026-09-09 · Authors: Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, Yuhang Guo, Yunhong Wang, Haifeng Wang ### JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation Introduces JEF-Hinter, which distills successful and failed trajectories into compact episodic hints and retrieves them for the current agent state. Web benchmark experiments examine guidance from decisive steps rather than full demonstration replay. - https://awesomecua.com/resources/arxiv-2510-04373/ · source: https://arxiv.org/abs/2510.04373v2 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, interaction-history, web-navigation - Published: 2025-10-05 · Authors: Hadi Nekoei, Aman Jaiswal, Patrice Bechard, Oleh Shliazhko, Orlando Marquez Ayala, Mathieu Reymond, Massimo Caccia, Alexandre Drouin, Sarath Chandar, Alexandre Lacoste ### Jev-Mobile: Jev as an Executor for Mobile GUI Agents Separates occasional visual-language planning from frequent lightweight action selection over an accessibility-tree-based action space. - https://awesomecua.com/resources/arxiv-2609-30186/ · source: https://arxiv.org/abs/2609.30186v1 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, efficiency, planning - Published: 2026-09-24 · Authors: Linghua Zhang ### JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks Introduces JHiMA-WebRL, which jointly trains a task decomposer and GUI actor with hierarchical RL for long-horizon web tasks in WebArena. - https://awesomecua.com/resources/arxiv-2611-05263/ · source: https://arxiv.org/abs/2611.05263v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, planning, long-horizon, web-navigation - Published: 2026-11-03 · Authors: Shanka Subhra Mondal, Jiayun Wang ### Just A Rather Very Intelligent Spoken Agent Introduces JarvisBench, a benchmark for spoken mediation during long-horizon agent workflows, measuring task collaboration and how users follow ongoing execution. - https://awesomecua.com/resources/arxiv-2607-16610/ · source: https://arxiv.org/abs/2607.16610v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, human-in-the-loop, long-horizon, observability - Published: 2026-07-18 · Authors: Chen Chen, Zhehuai Chen ### Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness BLIND-ACT tests whether computer-use agents persist with ambiguous, contradictory, or infeasible goals without adequate contextual reasoning. Its OSWorld-based tasks and judged trajectories examine these behaviors and the effect of mitigation prompts. - https://awesomecua.com/resources/arxiv-2510-01670/ · source: https://arxiv.org/abs/2510.01670v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, safety-security, robustness - Published: 2025-10-02 · Authors: Erfan Shayegani, Keegan Hines, Yue Dong, Nael Abu-Ghazaleh, Roman Lutz, Spencer Whitehead, Vidhisha Balachandran, Besmira Nushi, Vibhav Vineet ### K^2-Agent: Co-Evolving Know-What and Know-How for Hierarchical Mobile Device Control Introduces K2-Agent, a hierarchical mobile-control framework that refines task knowledge through a reasoning loop and trains an executor with curriculum-guided reinforcement learning. - https://awesomecua.com/resources/arxiv-2603-00676/ · source: https://arxiv.org/abs/2603.00676v1 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, planning, memory, reinforcement-learning - Published: 2026-02-28 · Authors: Zhe Wu, Donglin Mo, Hongjin Lu, Junliang Xing, Jianheng Liu, Yuheng Jing, Kai Li, Kun Shao, Jianye Hao, Yuanchun Shi ### KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation KG-RAG converts UI transition graphs into a searchable vector store and uses intent-guided retrieval to construct actionable navigation paths. It introduces KG-Android-Bench and KG-Harmony-Bench and evaluates transfer from mobile apps to web and desktop tasks. - https://awesomecua.com/resources/arxiv-2509-00366/ · source: https://arxiv.org/abs/2509.00366v1 - Paper · reviewed 2026-10-17 · environments: android, web, desktop, cross-platform · topics: memory, planning, mobile-navigation, evaluation - Published: 2025-08-30 · Authors: Ziyi Guan, Jason Chun Lok Li, Zhijian Hou, Pingping Zhang, Donglai Xu, Yuzhi Zhao, Mengyang Wu, Jinpeng Chen, Thanh-Toan Nguyen, Pengfei Xian, Wenao Ma, Shengchao Qin, Graziano Chesi, Ngai Wong ### Kimi-VL Technical Report Presents Kimi-VL, a general vision-language model evaluated on multi-turn agent tasks including OSWorld and on ScreenSpot-Pro alongside other vision-language tasks. - https://awesomecua.com/resources/arxiv-2504-07491/ · source: https://arxiv.org/abs/2504.07491v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, screen-understanding - Published: 2025-04-10 · Authors: Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen, Zongyu Lin ### Kite: Building Conversational Bots from Mobile Apps Derives task graphs from demonstrated Android app interactions and layers a conversational question-answer interface over those graphs. The evaluation examines task-model extraction and developer use of the resulting bot templates. - https://awesomecua.com/resources/url-901ca855640457dfe2f1/ · source: https://doi.org/10.1145/3210240.3210339 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, workflow, human-in-the-loop - Published: 2018-06 · Authors: Toby Jia-Jun Li, Oriana Riva ### KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill KnowAct-GUIClaw combines a host planner with pluggable GUI subagents, attributable experience memory and an evolving skill library. It uses user profiles and feedback for task decomposition and evaluates transfer across Android, iOS, HarmonyOS and Windows. - https://awesomecua.com/resources/arxiv-2607-12625/ · source: https://arxiv.org/abs/2607.12625v2 - Paper · reviewed 2026-10-18 · environments: desktop, android, ios, harmonyos, windows · topics: memory, planning, interaction-history, mobile-navigation - Published: 2026-07-14 · Authors: Yunxin Li, Jinchao Li, Shibo Su, Zhenran Xu, Chenrui Zhao, Tongshu Bian, Xiaoman Liang, Meishan Zhang, Baotian Hu, Min Zhang ### Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces A study shows that websites can infer the model behind a browser agent from passively recorded UI actions and timing traces across web tasks. - https://awesomecua.com/resources/arxiv-2605-14786/ · source: https://arxiv.org/abs/2605.14786v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, recording, interaction-history, web-navigation - Published: 2026-05-14 · Authors: William Lugoloobi, Samuelle Marro, Jabez Magomere, Joss Wright, Chris Russell ### KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation Introduces KnowU-Bench, an Android emulator benchmark for mobile agents that evaluates preference elicitation, proactive decisions, consent and GUI task execution. - https://awesomecua.com/resources/arxiv-2604-08455/ · source: https://arxiv.org/abs/2604.08455v1 - Paper · reviewed 2026-11-06 · environments: android · topics: evaluation, mobile-navigation, human-in-the-loop, sandbox - Published: 2026-04-09 · Authors: Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, Wenqi Zhang, Xu Tan, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen ### Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents Introduces Ko-WideSearch, a Korean web-agent benchmark that asks agents to find every member of a set and fill a table of attributes for each member. - https://awesomecua.com/resources/arxiv-2606-27595/ · source: https://arxiv.org/abs/2606.27595v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation - Published: 2026-06-25 · Authors: Minbyul Jeong ### Koala: Capture, Share, Automate, Personalize Business Processes on the Web Introduces Koala, a collaborative web automation system that records browser interactions as editable pseudo-natural-language scripts. Scripts are shared through a wiki and instantiated with personal or corporate data for reuse by other users. - https://awesomecua.com/resources/url-cccd967aa993404d74d0/ · source: https://doi.org/10.1145/1240624.1240767 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, workflow, recording - Published: 2007 · Authors: Greg Little, Tessa A. Lau, Allen Cypher, James Lin, Eben M. Haber, Eser Kandogan ### LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control LabOSBench evaluates GUI agents on browser-based scientific instrument simulators with configurable tasks and execution-based checks. Its subtask and end-to-end tests examine feedback-driven adjustment and long workflows without controlling physical instruments. - https://awesomecua.com/resources/arxiv-2606-16802/ · source: https://arxiv.org/abs/2606.16802v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, science, long-horizon - Published: 2026-06-15 · Authors: Anqi Zou, Han Deng, Chengyu Zhang, Junquan Hu, Yu Wang, Yuxiang Xing, Aokai Zhang, Hanling Zhang, Zhaoyang Liu, Ben Fei, Zhihui Wang, Wanli Ouyang ### LADS: Lease-Aware Delivery Scheduling of Tool Results for Language Model Agents Introduces LADS, a scheduler for returning browser-action results to web agents, using cache leases and estimated resumption delay to reduce workflow latency. - https://awesomecua.com/resources/arxiv-2611-00881/ · source: https://arxiv.org/abs/2611.00881v1 - Paper · reviewed 2026-11-06 · environments: web · topics: efficiency, workflow - Published: 2026-10-31 · Authors: Yuxuan Hou, Linshan Jiang ### Language Models can Self-Improve at State-Value Estimation for Better Search Introduces Self-Taught Lookahead (STL), which trains a value model to predict actions and resulting states for more efficient search, with evaluation on web-agent tasks. - https://awesomecua.com/resources/arxiv-2503-02878/ · source: https://arxiv.org/abs/2503.02878v3 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, efficiency, world-models - Published: 2025-03-04 · Authors: Ethan Mendes, Alan Ritter ### Language Models can Solve Computer Tasks Introduces recursive criticism and improvement prompting for language-model agents performing computer tasks, evaluated on MiniWoB++ with a small number of demonstrations per task. - https://awesomecua.com/resources/arxiv-2303-17491/ · source: https://arxiv.org/abs/2303.17491v3 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, self-reflection, training-free - Published: 2023-03-30 · Authors: Geunwoo Kim, Pierre Baldi, Stephen McAleer ### Large Action Models: From Inception to Implementation Uses a Windows-agent case study to describe a development workflow for large action models, covering data collection, training, grounding, environment integration, and evaluation. - https://awesomecua.com/resources/arxiv-2412-10047/ · source: https://arxiv.org/abs/2412.10047v2 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: automation, data-generation, evaluation - Published: 2024-12-13 · Authors: Lu Wang, Fangkai Yang, Chaoyun Zhang, Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang, Si Qin, Qisheng Su, Jiayi Ye, Yudi Zhang, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang ### Large Language Models Can Self-Improve At Web Agent Tasks Examines self-improvement by fine-tuning web agents on three mixtures of model-generated training data. WebArena experiments and trajectory-level evaluation measures study task performance, robustness, and behavior beyond aggregate success rates. - https://awesomecua.com/resources/arxiv-2405-20309/ · source: https://arxiv.org/abs/2405.20309v2 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, web-navigation, evaluation - Published: 2024-05-30 · Authors: Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condrei, Marius-Constantin Dinu, Chris Callison-Burch, Sepp Hochreiter ### Large Language Models Empowered Personalized Web Agents PersonalWAB benchmarks personalized web tasks, while PUMA retrieves relevant past web behavior to adapt agents' instruction handling and actions. - https://awesomecua.com/resources/arxiv-2410-17236/ · source: https://arxiv.org/abs/2410.17236v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, memory, interaction-history, web-navigation - Published: 2024-10-22 · Authors: Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, Tat-Seng Chua ### Large Language Models for Mobile GUI Text Input Generation: An Empirical Study Studies language-model generation of mobile text inputs, comparing UI representations, execution feedback, and human assistance for app testing. - https://awesomecua.com/resources/arxiv-2404-08948/ · source: https://arxiv.org/abs/2404.08948v4 - Paper · reviewed 2026-10-16 · environments: android · topics: automation, evaluation - Published: 2024-04-13 · Authors: Chenhui Cui, Tao Li, Junjie Wang, Chunyang Chen, Dave Towey, Rubing Huang ### LASER: LLM Agent with State-Space Exploration for Web Navigation LASER represents web navigation as exploration over a defined set of states, allowing an agent to revisit earlier states after mistakes. The study evaluates this backtracking approach on WebShop and Amazon tasks. - https://awesomecua.com/resources/arxiv-2309-08172/ · source: https://arxiv.org/abs/2309.08172v2 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, robustness, web-navigation - Published: 2023-09-15 · Authors: Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, Dong Yu ### LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agents Introduces LaSM, a training-free defense that selectively scales attention and MLP components to improve alignment with task-relevant screen regions. It evaluates the approach against pop-up-based GUI injection attacks. - https://awesomecua.com/resources/arxiv-2507-10610/ · source: https://arxiv.org/abs/2507.10610v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, grounding, training-free - Published: 2025-07-13 · Authors: Zihe Yan, Jiaping Gui, Zhuosheng Zhang, Gongshen Liu ### Latent State Estimation Helps UI Agents to Reason Studies explicit textual estimates of latent UI state, including whether commanded actions occurred and how far a task has progressed. The experiments compare agents that reason with these estimates against agents that omit them across public and internal task benchmarks. - https://awesomecua.com/resources/arxiv-2405-11120/ · source: https://arxiv.org/abs/2405.11120v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: state-tracking, planning, evaluation - Published: 2024-05-17 · Authors: William E Bishop, Alice Li, Christopher Rawles, Oriana Riva ### Latent World Action Model for GUI Agents Replaces future-screenshot generation with a shared latent representation for GUI action selection and action-conditioned transition prediction. An action-aware contrastive objective encourages the transition representation to retain discriminative action-induced changes. - https://awesomecua.com/resources/arxiv-2610-14228/ · source: https://arxiv.org/abs/2610.14228v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: world-models, planning, efficiency - Published: 2026-10-09 · Authors: Seungjun Yu, Hojun Choi, Seojeong Park, Jaeyo Shin, Hyunjung Shim ### Lean4AgenticRL: Advancing Agentic RL with Verifiable Multi-level Process Rewards Lean4AgenticRL formalizes workflow stages and process rewards for reinforcement learning on software engineering and terminal-use tasks, an adjacent setting for interface agents. - https://awesomecua.com/resources/arxiv-2610-22448/ · source: https://arxiv.org/abs/2610.22448v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, reward-models, workflow - Published: 2026-10-15 · Authors: Ruida Wang, Gerald Friedland, Tong Zhang, Chris Kong ### Learn from Move: the Next Step for GUI Agents Introduces LatentGUIWorld to test learning interface semantics and dynamics through interaction, and studies finer movement actions with intermediate visual feedback and exploration demonstrations. - https://awesomecua.com/resources/arxiv-2610-16720/ · source: https://arxiv.org/abs/2610.16720v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, grounding, reinforcement-learning - Published: 2026-10-12 · Authors: Jiahao Lu, Yize Shen, Chuanyuan Tan, Mianqiu Huang, Shicheng Fang, Xinchi Chen, Yining Zheng, Yuxin Wang, Xipeng Qiu ### Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents LearnWeak specializes small computer-use agents by finding domain-specific weaknesses, generating targeted tasks, and training separately for planning and execution errors. - https://awesomecua.com/resources/arxiv-2605-28775/ · source: https://arxiv.org/abs/2605.28775v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, planning, action-prediction - Published: 2026-05-27 · Authors: Suji Kim, Kangsan Kim, Sung Ju Hwang ### Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents GUI-SD-v2 extends on-policy self-distillation from grounding to multi-turn GUI interaction. It first strengthens responses to privileged guidance, then selectively distills step-specific reasoning and memory support for later actions. - https://awesomecua.com/resources/arxiv-2609-27307/ · source: https://arxiv.org/abs/2609.27307v1 - Paper · reviewed 2026-10-17 · environments: android · topics: imitation-learning, memory, interaction-history, mobile-navigation - Published: 2026-09-23 · Authors: Yan Zhang, Daiqing Wu, Huawen Shen, Liang Li, Gang Cao, Zhi Gong, Wei Dai, Xiaode Zhang, Can Ma, Yu Zhou ### Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding Introduces GUI-SD, an on-policy self-distillation method for locating GUI elements from instructions, using a visually enriched teacher context and entropy-guided token weighting. - https://awesomecua.com/resources/arxiv-2605-00642/ · source: https://arxiv.org/abs/2605.00642v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, efficiency - Published: 2026-05-01 · Authors: Yan Zhang, Daiqing Wu, Huawen Shen, Can Ma, Yu Zhou ### Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments Learn-by-interact synthesizes agent interaction trajectories from documentation and derives instructions from their histories for training or in-context use across web and desktop tasks. - https://awesomecua.com/resources/arxiv-2501-10893/ · source: https://arxiv.org/abs/2501.10893v1 - Paper · reviewed 2026-11-06 · environments: web, desktop, cross-platform · topics: data-generation, interaction-history, training-free - Published: 2025-01-18 · Authors: Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, Sercan Ö. Arık ### LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark LearnAct parses human demonstrations into reusable knowledge, retrieves relevant examples, and uses them during mobile GUI task execution. The accompanying LearnGUI benchmark includes 2,252 offline tasks and 101 online tasks for studying adaptation from demonstrations. - https://awesomecua.com/resources/arxiv-2504-13805/ · source: https://arxiv.org/abs/2504.13805v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, memory, human-in-the-loop, evaluation - Published: 2025-04-18 · Authors: Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Shuai Ren, Hao Wang, Shibo He, Wenchao Meng ### Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding LASER builds preference data for multi-step GUI perception by combining Monte Carlo quality estimates with region-overlap evaluation. Its self-evolving training encourages relevant region selection and adapts the amount of reasoning before predicting screen coordinates. - https://awesomecua.com/resources/arxiv-2509-04243/ · source: https://arxiv.org/abs/2509.04243v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, data-generation, screen-understanding - Published: 2025-09-04 · Authors: Wanfu Wang, Qipeng Huang, Guangquan Xue, Xiaobo Liang, Juntao Li ### Learning Efficient Guardrails for Compliance Introduces PolicyGuardBench for detecting policy violations in full web-agent trajectories and their prefixes, and trains PolicyGuard as an efficient guardrail model. - https://awesomecua.com/resources/arxiv-2510-03485/ · source: https://arxiv.org/abs/2510.03485v2 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, efficiency - Published: 2025-10-03 · Authors: Xiaofei Wen, Wenjie Jacky Mo, Yanan Xie, Peng Qi, Muhao Chen ### Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents Uses failed computer-use trajectories to diagnose failure modes and generate inference-time agent code patches, with light human verification. An OSWorld study evaluates this failure-driven improvement loop without additional model training. - https://awesomecua.com/resources/arxiv-2606-31270/ · source: https://arxiv.org/abs/2606.31270v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: self-reflection, robustness, training-free, evaluation - Published: 2026-06-30 · Authors: Xueqiao Sun, Xiaohan Wang, Ludwig Schmidt, Serena Yeung-Levy, Yuhui Zhang ### Learning from Online Videos at Inference Time for Computer-Use Agents This work retrieves online video tutorials and converts them into structured GUI action demonstrations with short objectives. A two-stage selection process supplies a relevant local trajectory as in-context guidance at each execution step. - https://awesomecua.com/resources/arxiv-2511-04137/ · source: https://arxiv.org/abs/2511.04137v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: video, memory, planning, data-generation - Published: 2025-11-06 · Authors: Yujian Liu, Ze Wang, Hao Chen, Ximeng Sun, Xiaodong Yu, Jialian Wu, Jiang Liu, Emad Barsoum, Zicheng Liu, Shiyu Chang ### Learning from Others, Acting for You: Cross-User Memory Sharing for LLM Agents ShareMem consolidates reusable experience across users while keeping preference retrieval user-bound; its web evaluation predicts target elements and actions on recorded Mind2Web states. - https://awesomecua.com/resources/arxiv-2609-32511/ · source: https://arxiv.org/abs/2609.32511v1 - Paper · reviewed 2026-10-20 · environments: web · topics: memory, web-navigation, action-prediction, evaluation - Published: 2026-09-26 · Authors: Jinming Hu, Haodong Zhao, Qi Jia, Die Chen, Tianhang Zhao, Sufeng Duan, Gongshen Liu ### Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding Confidence-Anchored Learning uses coordinate-token confidence to filter pseudo-supervision for GUI grounding. Its negative-only variant performs label-free test-time updates to reduce reliance on potentially incorrect positive examples. - https://awesomecua.com/resources/arxiv-2609-15307/ · source: https://arxiv.org/abs/2609.15307v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, robustness, evaluation - Published: 2026-09-14 · Authors: Yizhou Liu, Fei Tang, Yuchen Yan, Zhengxi Lu, Songqin Nong, Tao Jiang, Wenhao Xu, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen ### Learning GUI Grounding with Spatial Reasoning from Visual Feedback GUI-Cursor treats GUI grounding as a sequence of cursor movements guided by visible feedback and movement history. Online reinforcement learning with trajectory rewards trains the model to refine the target location across multiple steps. - https://awesomecua.com/resources/arxiv-2509-21552/ · source: https://arxiv.org/abs/2509.21552v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, action-prediction, robustness - Published: 2025-09-25 · Authors: Yu Zhao, Wei-Ning Chen, Huseyin Atahan Inan, Samuel Kessler, Lu Wang, Lukas Wutschitz, Fangkai Yang, Chaoyun Zhang, Pasquale Minervini, Saravan Rajmohan, Robert Sim ### Learning How to Remember: Memory Modality Selection for GUI Agents Introduces SwitchMem, which chooses a textual or visual representation of retrieved experience at each GUI interaction step instead of fixing one memory modality for a task. - https://awesomecua.com/resources/arxiv-2610-17928/ · source: https://arxiv.org/abs/2610.17928v1 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, mobile-navigation - Published: 2026-10-13 · Authors: Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang, Jaewoo Kang ### Learning Reliable GUI Agents under Imperfect Priors Combines structured GUI exploration with training designed to tolerate imperfect retrieved experience. It builds state-transition graphs and synthetic task trajectories, then injects realistic forms of interface drift so agents learn when to rely on or reject prior guidance. - https://awesomecua.com/resources/arxiv-2609-39547/ · source: https://arxiv.org/abs/2609.39547v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, memory, robustness - Published: 2026-09-30 · Authors: Bo Han, Qianyi Wang, Shuai Liu, Xiong Zifan, Changqiao Wu, Yuanfa Li, Pengzhi Gao, Wei Liu, Jian Luan, Heng Qu, Yunpeng Song, Zhongmin Cai ### Learning Simple Test-Time Environments for LLM Web Agents Introduces Test-Time Environment Decomposition, which uses trial interaction to break complex web observations into simpler modules. A label-free adaptation procedure learns from these sub-environments and studies whether their gains compose in full web tasks. - https://awesomecua.com/resources/arxiv-2608-29305/ · source: https://arxiv.org/abs/2608.29305v1 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, robustness, web-navigation - Published: 2026-08-29 · Authors: Junxuan Li, Zijun Liu, Ziyi Huang, Peng Li, Yuzhou Liu, Ming Yan, Yang Liu ### Learning to Adapt: Self-Improving Web Agent via Cognitive-Aware Exploration Introduces SCALE, which uses exploration roles and graph planning to train web agents, and SCALE-20k, a dataset of tasks and demonstrations from real websites. - https://awesomecua.com/resources/arxiv-2605-31365/ · source: https://arxiv.org/abs/2605.31365v1 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, planning, data-generation, self-reflection - Published: 2026-05-29 · Authors: Weile Chen, Bingchen Miao, Qifan Yu, Wendong Bu, Guoming Wang, Wenqiao Zhang, Shengyu Zhang, Juncheng Li, Siliang Tang ### Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale Introduces CLAY, a pipeline that repairs noisy mobile UI layouts by removing nodes without valid visual counterparts and assigning semantic element types, supported by human-annotated layouts for evaluation. - https://awesomecua.com/resources/arxiv-2201-04100/ · source: https://arxiv.org/abs/2201.04100v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, data-generation - Published: 2022-01-11 · Authors: Gang Li, Gilles Baechler, Manuel Tragut, Yang Li ### Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding Introduces LBS, a vision-language model trained with GUI grounding trajectories and crop-and-inspect decisions to study transfer of active visual observation. - https://awesomecua.com/resources/arxiv-2610-37093/ · source: https://arxiv.org/abs/2610.37093v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, reinforcement-learning, screen-understanding - Published: 2026-10-27 · Authors: Ruilin Yao, Xinyu Qiu, Tianyu Zou, Yi Rong, Shengwu Xiong, Shuheng Shen, Changhua Meng ### Learning to Navigate the Web Introduces guided RL for QWeb, using a curriculum of smaller instructions and generated tasks to train an agent that follows instructions and fills forms on web pages. - https://awesomecua.com/resources/arxiv-1812-09195/ · source: https://arxiv.org/abs/1812.09195v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, planning, web-navigation - Published: 2018-12-21 · Authors: Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, Dilek Hakkani-Tur ### Learning with Challenges: Adaptive Difficulty-Aware Data Generation for Mobile GUI Agent Training MobileGen estimates a mobile agent’s strengths across structural and semantic task difficulty, then samples new training targets around that profile. A controllable multi-agent generator creates the corresponding instructions and GUI trajectories. - https://awesomecua.com/resources/arxiv-2601-22781/ · source: https://arxiv.org/abs/2601.22781v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, mobile-navigation, evaluation - Published: 2026-01-30 · Authors: Linjia Kang, Zhimin Wang, Yongkang Zhang, Duo Wu, Jinghe Wang, Ming Ma, Haopeng Yan, Zhi Wang ### Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents CogniGUI combines hierarchical visual parsing of interface elements with a GRPO-based grounding agent that compares alternative interaction paths. Its ScreenSeek benchmark evaluates navigation across applications, changing interface states, and consistency between interfaces. - https://awesomecua.com/resources/arxiv-2506-17913/ · source: https://arxiv.org/abs/2506.17913v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, planning, reinforcement-learning, evaluation - Published: 2025-06-22 · Authors: Jinjie Wei, Jiyao Liu, Lihao Liu, Ming Hu, Junzhi Ning, Mingcheng Li, Weijie Yin, Junjun He, Xiao Liang, Chao Feng, Dingkang Yang ### LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use Agents Uses weak-agent uncertainty to decide when to escalate to a stronger model and monitor its subsequent task execution. - https://awesomecua.com/resources/arxiv-2610-22280/ · source: https://arxiv.org/abs/2610.22280v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: efficiency, planning - Published: 2026-10-15 · Authors: Qingni Wang, Changdae Oh, Yuzhe Yang, Sharon Li, Jiachen Yang, Ang Li, Xin Eric Wang ### LegacyWorld: Atomicity-Aware Evaluation of GUI Agents for Legacy Workflows LegacyWorld evaluates Windows GUI workflows under an atomicity criterion: finish correctly or fail without persistent unintended changes. It compares expert prompts with prompts derived from recorded golden paths and separates useful completion, safe failure and non-atomic side effects. - https://awesomecua.com/resources/arxiv-2608-14131/ · source: https://arxiv.org/abs/2608.14131v1 - Paper · reviewed 2026-10-18 · environments: desktop, windows · topics: evaluation, safety-security, workflow, recording - Published: 2026-08-14 · Authors: Thilo Reintjes, Sivajeet Chand, Derui Zhu, Sushant Kumar Pandey, Alexander Pretschner ### LegalWebAgent: Empowering Access to Justice via LLM-Based Web Agents LegalWebAgent separates user-needs clarification, webpage browsing, and actions such as form completion or scheduling. A 15-task evaluation studies legal-service workflows relevant to Québec civil-law users, using HTML and screenshot observations. - https://awesomecua.com/resources/arxiv-2512-04105/ · source: https://arxiv.org/abs/2512.04105v1 - Paper · reviewed 2026-10-17 · environments: web · topics: industry, web-navigation, human-in-the-loop, evaluation - Published: 2025-11-28 · Authors: Jinzhe Tan, Karim Benyekhlef ### Less is More: Empowering GUI Agent with Context-Aware Simplification SimpAgent prunes distracting interface elements with visual masks and compresses interaction history using consistency-based guidance. The study evaluates these two forms of context reduction during GUI navigation in web and mobile environments. - https://awesomecua.com/resources/arxiv-2507-03730/ · source: https://arxiv.org/abs/2507.03730v1 - Paper · reviewed 2026-10-17 · environments: web, cross-platform · topics: efficiency, interaction-history, web-navigation, mobile-navigation - Published: 2025-07-04 · Authors: Gongwei Chen, Xurui Zhou, Rui Shao, Yibo Lyu, Kaiwen Zhou, Shuai Wang, Wentao Li, Yinchuan Li, Zhongang Qi, Liqiang Nie ### Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification Studies agreement bias in multimodal agent verifiers across web navigation and computer use, and introduces SGV to assess trajectories using self-generated priors. - https://awesomecua.com/resources/arxiv-2507-11662/ · source: https://arxiv.org/abs/2507.11662v3 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, reward-models, self-reflection, web-navigation - Published: 2025-07-15 · Authors: Moises Andrade, Joonhyuk Cha, Brandon Ho, Vriksha Srihari, Karmesh Yadav, Zsolt Kira ### Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI Introduces AutomotiveUI-Bench-4K and a fine-tuned vision-language model for locating elements and evaluating automotive infotainment interfaces. - https://awesomecua.com/resources/arxiv-2505-05895/ · source: https://arxiv.org/abs/2505.05895v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, ui-design, data-generation - Published: 2025-05-09 · Authors: Benjamin Raphael Ernhofer, Daniil Prokhorov, Jannica Langner, Dominik Bollmann ### Lexi: Self-Supervised Learning of the UI Language Introduces Lexi and UICaption to learn UI image-text representations from descriptions of interface functionality. Evaluation covers UI action entailment, instruction-based retrieval, referring-expression grounding, and UI entity recognition. - https://awesomecua.com/resources/arxiv-2301-10165/ · source: https://arxiv.org/abs/2301.10165v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding, data-generation - Published: 2023-01-23 · Authors: Pratyay Banerjee, Shweti Mahajan, Kushal Arora, Chitta Baral, Oriana Riva ### Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web Studies whether text-embedding GUI grounding scores reflect semantic matching or recovery of visible labels. Same-screen ranking comparisons with lexical baselines expose differences between deployable fusion, oracle selection, and performance on label-poor targets. - https://awesomecua.com/resources/arxiv-2608-21794/ · source: https://arxiv.org/abs/2608.21794v1 - Paper · reviewed 2026-10-17 · environments: web, cross-platform · topics: grounding, evaluation, screen-understanding - Published: 2026-08-22 · Authors: Qijia Chen, Giulio Jacucci ### LHAW: Controllable Underspecification for Long-Horizon Tasks LHAW generates underspecified variants of long-horizon workflow tasks and uses agent trials to evaluate when missing information requires clarification. - https://awesomecua.com/resources/arxiv-2602-10525/ · source: https://arxiv.org/abs/2602.10525v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon, human-in-the-loop, data-generation - Published: 2026-02-11 · Authors: George Pu, Michael S. Lee, Udari Madhushani Sehwag, David J. Lee, Bryan Zhu, Yash Maurya, Mohit Raghavendra, Yuan Xue, Samuel Marc Denton ### Lightweight Neural App Control LiMAC combines a compact Action Transformer with a fine-tuned vision-language model for Android app control. It uses textual goals and histories of screenshots and UI trees to predict actions, with evaluation on two mobile-control datasets. - https://awesomecua.com/resources/arxiv-2410-17883/ · source: https://arxiv.org/abs/2410.17883v2 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, action-prediction, efficiency, interaction-history - Published: 2024-10-23 · Authors: Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, Kun Shao ### LineRetriever: Planning-Aware Observation Reduction for Web Agents LineRetriever uses a language model to select observation lines that matter for upcoming web-navigation actions. It reduces DOM or accessibility-tree context by considering the planning horizon instead of relying solely on semantic similarity. - https://awesomecua.com/resources/arxiv-2507-00210/ · source: https://arxiv.org/abs/2507.00210v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, efficiency, planning - Published: 2025-06-30 · Authors: Imene Kerboua, Sahar Omidi Shayegan, Megh Thakkar, Xing Han Lù, Massimo Caccia, Véronique Eglin, Alexandre Aussem, Jérémy Espinas, Alexandre Lacoste ### LiteCUA: Computer as MCP Server for Computer-Use Agent on AIOS AIOS 1.0 exposes computer states and actions through an MCP server to provide structured environmental context for agents. LiteCUA is a lightweight agent built on this interface, evaluated on OSWorld to study the effect of computer-environment contextualization. - https://awesomecua.com/resources/arxiv-2505-18829/ · source: https://arxiv.org/abs/2505.18829v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: tool-integration, automation - Published: 2025-05-24 · Authors: Kai Mei, Xi Zhu, Hang Gao, Shuhang Lin, Yongfeng Zhang ### LiteGUI: Lightweight GUI Agents via Multi-Solution Guided Distillation and Dual-Level Reinforcement Learning LiteGUI trains lightweight GUI agents on logged states with guided on-policy distillation and dual-level RL, supported by its multi-solution Lite-Dataset. - https://awesomecua.com/resources/arxiv-2605-07505/ · source: https://arxiv.org/abs/2605.07505v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: imitation-learning, reinforcement-learning, data-generation, planning - Published: 2026-05-08 · Authors: Yubin Wu, Zicheng Cai, Liping Ning, Hua Wang, Zhi Chen, Yaohua Tang, Hao Chen ### LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applications Introduces a modular visual web-agent stack with planning and memory components, exposed through a remote-browser application and Chrome extension. - https://awesomecua.com/resources/arxiv-2503-02950/ · source: https://arxiv.org/abs/2503.02950v2 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, tool-integration, planning - Published: 2025-03-04 · Authors: Danqing Zhang, Balaji Rama, Jingyi Ni, Shiying He, Fu Zhao, Kunyu Chen, Arnold Chen, Junyu Cao ### LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents LLaDA-UI adapts block-wise diffusion generation to a vision-language GUI agent through multimodal pretraining and GUI supervised fine-tuning. It evaluates grounding and navigation across multiple platform settings. - https://awesomecua.com/resources/arxiv-2609-13287/ · source: https://arxiv.org/abs/2609.13287v1 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: grounding, imitation-learning, efficiency, screen-understanding - Published: 2026-09-09 · Authors: Zhangxuan Gu, Haoxing Chen, Qi Qin, Yi Xin, Kai Gan, Lin Liu, Long Cui, Xiaomei Wang, Beitong Zhou, Yunzhu Zhang, Zhengwen Zeng, Changlong Gao, Weizhi Chen, Rongchao Zhang, Haoyuan Wu, Shuheng Shen, Changhua Meng, Weiqiang Wang, Jianguo Li, Zhenzhong Lan ### LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation LlamaTouch executes mobile tasks on devices and evaluates whether an agent visits manually annotated essential application or system states. Screenshot and hierarchy annotations, combined with exact and fuzzy state matching, support evaluation of 496 tasks across multiple agents. - https://awesomecua.com/resources/arxiv-2404-16054/ · source: https://arxiv.org/abs/2404.16054v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, state-tracking - Published: 2024-04-12 · Authors: Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, Mengwei Xu ### LLM-as-Code: Agentic Programming for Agent Harness Proposes Agentic Programming, where code controls agent loops and invokes LLM-as-Code for reasoning; a computer-use case study examines long visual action sequences. - https://awesomecua.com/resources/arxiv-2606-15874/ · source: https://arxiv.org/abs/2606.15874v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: workflow, efficiency, long-horizon - Published: 2026-06-14 · Authors: Junjia Qi, Zichuan Fu, Jingtong Gao, Wenlin Zhang, Hanyu Yan, Xian Wu, Xiangyu Zhao ### LLM-Explorer: Towards Efficient and Affordable LLM-based Exploration for Mobile Apps LLM-Explorer uses language models to maintain compact knowledge for mobile app exploration, then generates actions without invoking the model at every step. The study compares exploration coverage and cost across twenty apps. - https://awesomecua.com/resources/arxiv-2505-10593/ · source: https://arxiv.org/abs/2505.10593v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: mobile-navigation, efficiency - Published: 2025-05-15 · Authors: Shanhui Zhao, Hao Wen, Wenjie Du, Cheng Liang, Yunxin Liu, Xiaozhou Ye, Ye Ouyang, Yuanchun Li ### LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training Introduces UI-Simulator, which generates UI states and transitions for agent-training trajectories, and UI-Simulator-Grow, which prioritizes tasks and trajectory variants. - https://awesomecua.com/resources/arxiv-2510-14969/ · source: https://arxiv.org/abs/2510.14969v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: world-models, data-generation - Published: 2025-10-16 · Authors: Yiming Wang, Da Yin, Yuedong Cui, Ruichen Zheng, Zhiqian Li, Zongyu Lin, Di Wu, Xueqing Wu, Chenchen Ye, Yu Zhou, Kai-Wei Chang ### LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents Introduces LocalLSTC, a training-free control architecture that maintains persistent task state and uses it to form short execution commitments. Outcomes feed back into that state for progress checks, recovery, and termination in locally deployed desktop agents. - https://awesomecua.com/resources/arxiv-2608-25777/ · source: https://arxiv.org/abs/2608.25777v2 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: state-tracking, planning, training-free - Published: 2026-08-26 · Authors: Weiming Li, Helen Paik, Yulei Sui ### LoginTrap: Uncovering Task-Agnostic Phishing-Style Indirect Prompt Injection Attacks against LLM-based Web Agents Introduces LoginTrap, a black-box indirect-injection evaluation in which attacker-controlled page context and login flows induce unauthorized authentication. It measures end-to-end private-data leakage across web-agent backbones, architectures and defenses. - https://awesomecua.com/resources/arxiv-2608-04741/ · source: https://arxiv.org/abs/2608.04741v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, robustness, evaluation - Published: 2026-08-05 · Authors: Longtao Guo, Zelin Zhang, Kaifeng Huang, Yang Shi ### LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks Introduces LongHorizon-Harness, which maintains verified task state outside an agent's execution context and audits subtasks; its evaluations include OSWorld~2.0. - https://awesomecua.com/resources/arxiv-2608-01964/ · source: https://arxiv.org/abs/2608.01964v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: long-horizon, state-tracking, planning, evaluation - Published: 2026-08-03 · Authors: Ziyu Ma, Hailang Huang, Shun Zou, Yong Wang, Shidong Yang, Yiming Hu, Fei Wei, XiangXiang Chu ### LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues LongMemEval-V2 asks memory systems to recover environment knowledge from long web-agent histories for question answering. Its AgentRunbook variants contrast indexed knowledge pools with coding-agent retrieval from stored trajectory files. - https://awesomecua.com/resources/arxiv-2605-12493/ · source: https://arxiv.org/abs/2605.12493v1 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, interaction-history, state-tracking, workflow, evaluation - Published: 2026-05-12 · Authors: Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang ### LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles LongPuzzleBench evaluates coherent long-horizon planning on persistent visual puzzle boards through native GUI actions. It compares GUI-only and code-assisted agents and uses controlled hints and memory diagnostics to examine decisions that preserve or eliminate future solutions. - https://awesomecua.com/resources/arxiv-2609-34769/ · source: https://arxiv.org/abs/2609.34769v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: evaluation, planning, long-horizon, state-tracking - Published: 2026-09-28 · Authors: Bingo Zhang, Haochuan Lu, Zongjie Li, Genjian Li, Ari Yu Zhang, Chaozheng Wang ### Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation Uses a pre-execution critic and reinforcement training to evaluate proposed GUI actions and provide corrective suggestions before execution. - https://awesomecua.com/resources/arxiv-2506-04614/ · source: https://arxiv.org/abs/2506.04614v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: planning, robustness - Published: 2025-06-05 · Authors: Yuyang Wanyan, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Jiabo Ye, Yutong Kou, Ming Yan, Fei Huang, Xiaoshan Yang, Weiming Dong, Changsheng Xu ### LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection LookAgain treats a GUI coordinate as a revisable hypothesis, rendering its marker and a local screenshot patch before confirmation. Reflective supervised trajectories provide a cold start, followed by GRPO using terminal grounding correctness. - https://awesomecua.com/resources/arxiv-2608-09723/ · source: https://arxiv.org/abs/2608.09723v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: grounding, self-reflection, reinforcement-learning, imitation-learning - Published: 2026-08-10 · Authors: Renshan Zhang, Haoyang Meng, Yixiao He, Rui Shao, April Hua Liu, Liqiang Nie ### Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay ReTrace evaluates how keyboard-and-mouse computer-use agents recover from injected errors by replaying actions and comparing downstream outcomes. - https://awesomecua.com/resources/arxiv-2610-27954/ · source: https://arxiv.org/abs/2610.27954v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, robustness, long-horizon - Published: 2026-10-20 · Authors: Byeonghun Bang, Sangwoo Park, Jinheon Baek, Kangsan Kim, Youngwan Lee, Sung Ju Hwang ### LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization Location Preference Optimization uses an information-entropy signal to select interaction regions and a distance-based reward to assess click locations. It trains GUI localization through preference optimization and evaluates both offline grounding and online interaction. - https://awesomecua.com/resources/arxiv-2506-09373/ · source: https://arxiv.org/abs/2506.09373v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning - Published: 2025-06-11 · Authors: Jiaqi Tang, Yu Xia, Yi-Feng Wu, Yuwei Hu, Yuhui Chen, Qing-Guo Chen, Xiaogang Xu, Xiangyu Wu, Hao Lu, Yanqing Ma, Shiyin Lu, Qifeng Chen ### LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents LUMOS converts native accessibility metadata and browser UI structure into semantic interface blueprints with stable element identifiers and available actions. It combines live pointer-to-element grounding with an observe-act loop constrained to visible-UI primitives. - https://awesomecua.com/resources/arxiv-2606-30697/ · source: https://arxiv.org/abs/2606.30697v1 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, screen-understanding, tool-integration, automation - Published: 2026-06-29 · Authors: Yogeswar Reddy Thota ### M$^2$-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Mining M2-Miner uses multi-agent-guided Monte Carlo tree search to collect mobile GUI intent–trajectory pairs. Intent recycling and progressive collector training aim to increase the diversity and usefulness of the mined interaction data. - https://awesomecua.com/resources/arxiv-2602-05429/ · source: https://arxiv.org/abs/2602.05429v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, mobile-navigation, planning - Published: 2026-02-05 · Authors: Rui Lv, Juncheng Mo, Tianyi Chu, Chen Rao, Hongyi Jing, Jiajie Teng, Jiafu Chen, Shiqi Zhang, Liangzi Ding, Shuo Fang, Huaizhong Lin, Ziqiang Dang, Chenguang Ma, Lei Zhao ### M$^2$: Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrieval Introduces M², a training-free memory framework for long-horizon web agents that summarizes interaction history and retrieves task guidance from an offline insight bank. - https://awesomecua.com/resources/arxiv-2603-00503/ · source: https://arxiv.org/abs/2603.00503v1 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, interaction-history, long-horizon, training-free - Published: 2026-02-28 · Authors: Dawei Yan, Haokui Zhang, Guangda Huzhang, Yang Li, Yibo Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Ying Li, Wei Dong, Chunhua Shen ### MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop Introduces macOS tasks spanning GUI and command-line interaction, with deterministic checks and intermediate-progress scoring for application workflows. - https://awesomecua.com/resources/arxiv-2606-22557/ · source: https://arxiv.org/abs/2606.22557v1 - Paper · reviewed 2026-10-16 · environments: desktop, macos · topics: evaluation, cross-app, tool-integration - Published: 2026-06-21 · Authors: Yikun Fu, Bowen Fu, Zhenyu Wu, Shuang Cheng, Xiaowei Sun, Bowen Yang, Zehao Li, Yibo Zhao, Zichen Ding, Zhoumianze Liu, Shijie Wang, Biqing Qi, Bowen Zhou ### MacArena: Benchmarking Computer Use Agents on an Online macOS Environment MacArena runs computer-use tasks through Apple’s virtualization framework on Apple Silicon, combining task ports with macOS-native workloads. It distinguishes the ported and native subsets when examining agent performance and cross-platform transfer. - https://awesomecua.com/resources/arxiv-2606-06560/ · source: https://arxiv.org/abs/2606.06560v1 - Paper · reviewed 2026-10-17 · environments: desktop, macos · topics: evaluation, robustness - Published: 2026-06-04 · Authors: Victor Muryn, Maksym Shamrai, Sofiia Mazepa, Yehor Khodysko ### Machine-Readable Ads: Accessibility and Trust Patterns for AI Web Agents interacting with Online Advertisements Compares DOM-based and pixel-based web agents on a cloned news site containing advertisements, consent dialogs and paywalls. The study examines how interface presentation affects ad interaction and consequential subscription choices. - https://awesomecua.com/resources/arxiv-2507-12844/ · source: https://arxiv.org/abs/2507.12844v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, ui-design, safety-security - Published: 2025-07-17 · Authors: Joel Nitu, Heidrun Mühle, Andreas Stöckl ### macOSWorld: A Multilingual Interactive Benchmark for GUI Agents macOSWorld provides 202 interactive tasks across 30 applications, with instructions and operating-system interfaces in five languages. It evaluates GUI agents on macOS workflows and includes a separate subset for deception-attack testing. - https://awesomecua.com/resources/arxiv-2506-04135/ · source: https://arxiv.org/abs/2506.04135v4 - Paper · reviewed 2026-10-17 · environments: macos, desktop · topics: evaluation, workflow, safety-security - Published: 2025-06-04 · Authors: Pei Yang, Hai Ci, Mike Zheng Shou ### MAESTRO: Adapting GUIs and Guiding Navigation with User Preferences in Conversational Agents with GUIs Introduces MAESTRO, a conversational GUI system that remembers user preferences, adapts interface options, and guides backtracking in multistep booking decisions. - https://awesomecua.com/resources/arxiv-2604-06134/ · source: https://arxiv.org/abs/2604.06134v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: ui-design, memory, workflow, human-in-the-loop - Published: 2026-04-07 · Authors: Sangwook Lee, Adnan Abbas, Sang Won Lee, Young-Ho Kim, Yan Chen ### MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation MAG jointly evaluates browser task actions and user-facing guides grounded in screenshots through element marks or pixel coordinates. Its harness covers annotation, live evaluation and joint metrics, with an expert-trajectory-augmented GRPO training study. - https://awesomecua.com/resources/arxiv-2607-10079/ · source: https://arxiv.org/abs/2607.10079v3 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, grounding, reinforcement-learning, action-prediction - Published: 2026-07-11 · Authors: Chengguang Gan, Hanjun Wei, Yunhao Liang, Zhixi Cai, Qinghao Zhang, Shiwen Ni ### MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation MAGA distills specialized GUI agents into one policy by allocating supervision according to structured-action correctness rather than treating all response tokens equally. A training-only teacher hint supports fusion across mobile, web and desktop settings without changing student inputs. - https://awesomecua.com/resources/arxiv-2607-29320/ · source: https://arxiv.org/abs/2607.29320v1 - Paper · reviewed 2026-10-17 · environments: cross-platform · topics: imitation-learning, grounding, automation - Published: 2026-07-31 · Authors: Hang Yan, Zhangxuan GU, Beitong Zhou, Jiaxuan Chen, Runze Li, Yusong Hu, Shuheng Shen, Changhua Meng ### MAGAR: Multimodal Agents for GUI-Based 3D Mechanical Assembly and Minimal Rework Introduces MAGAR-Bench for GUI-based 3D mechanical assembly decisions and recovery, and MAGAR, a selective correction method for assembly outputs and actions. - https://awesomecua.com/resources/arxiv-2611-02472/ · source: https://arxiv.org/abs/2611.02472v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: science, evaluation, self-reflection, action-prediction - Published: 2026-11-02 · Authors: Jingxuan Wei, Jiahao Fu, Fuqiang Wang, Xiaoming Wu, Yuwen Liu, Kaiwen Wei, Ming Yang, Junnan Zhu ### Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses Introduces Mage-VL-Cowork, a computer-use agent trained for GUI, code, and terminal actions, with Mage-CUA-Forge generating desktop and web tasks for training. - https://awesomecua.com/resources/arxiv-2610-40747/ · source: https://arxiv.org/abs/2610.40747v1 - Paper · reviewed 2026-11-06 · environments: desktop, web, cross-platform · topics: tool-integration, data-generation, reinforcement-learning - Published: 2026-10-29 · Authors: Xiaoyi Zhang, Haoqing Wang, Xinjie Zhang, Xiang Ming, Xing Wang, Senqiao Yang, Kaichen Zhang, Zhaoyang Jia, Andong Tan, Xinjiang Wang, Jinghao Guo, Yifei Shen, Xiao Li, Peng Zhang, Xun Guo, Jiahao Li, Shicheng Zheng, Jinglu Wang, Zongyu Guo, Wenxuan Xie, Bin Li, Yan Lu ### Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks Introduces an orchestrated team of browsing, file, and coding agents with progress tracking and error recovery for general digital tasks. - https://awesomecua.com/resources/arxiv-2411-04468/ · source: https://arxiv.org/abs/2411.04468v1 - Paper · reviewed 2026-10-16 · environments: web, desktop · topics: planning, tool-integration - Published: 2024-11-07 · Authors: Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang, Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, Saleema Amershi ### Magentic-UI: Towards Human-in-the-loop Agentic Systems Magentic-UI combines browser interaction, code execution, file operations, and MCP tools in an interface for human-agent collaboration. It studies collaborative planning and task execution, action guards, and persistent memory through benchmark tasks, simulated users, user studies, and safety assessments. - https://awesomecua.com/resources/arxiv-2507-22358/ · source: https://arxiv.org/abs/2507.22358v1 - Paper · reviewed 2026-10-17 · environments: web · topics: human-in-the-loop, tool-integration, web-navigation, safety-security - Published: 2025-07-30 · Authors: Hussein Mozannar, Gagan Bansal, Cheng Tan, Adam Fourney, Victor Dibia, Jingya Chen, Jack Gerrits, Tyler Payne, Matheus Kunzler Maldaner, Madeleine Grunde-McLaughlin, Eric Zhu, Griffin Bassman, Jacob Alber, Peter Chang, Ricky Loynd, Friederike Niedtner, Ece Kamar, Maya Murad, Rafah Hosn, Saleema Amershi ### MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux MagicGUI-RMS combines domain-specific and general reward models to assess GUI actions and propose corrections. A structured data pipeline and feedback-recycling process support repeated reward-model and agent improvement. - https://awesomecua.com/resources/arxiv-2601-13060/ · source: https://arxiv.org/abs/2601.13060v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reward-models, data-generation, evaluation, robustness - Published: 2026-01-19 · Authors: Zecheng Li, Zhihui Cao, Wenke Huang, Yudong Zhang, Keying Qi, Rui Wang, Zeyu Zheng, Jian Zhao, Hao Zhu, Hengxin Wu, Yuran Wang, Guitao Fan, Guokun Wu, Yicong Liu, Zhilin Gao, Haikun Xu, He Yang, Minqi Xiang, Xingyu Liu, Zuojian Wang ### MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning Combines a mobile GUI data pipeline, perception and planning capabilities, and reinforcement fine-tuning for interactive task execution. - https://awesomecua.com/resources/arxiv-2508-03700/ · source: https://arxiv.org/abs/2508.03700v5 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, data-generation, reinforcement-learning - Published: 2025-07-19 · Authors: Liujian Tang, Shaokang Dong, Yijia Huang, Minqi Xiang, Hongtao Ruan, Bin Wang, Shuo Li, Zhiheng Xi, Zhihui Cao, Hailiang Pang, Heng Kong, He Yang, Mingxu Chai, Zhilin Gao, Xingyu Liu, Yingnan Fu, Jiaming Liu, Xuanjing Huang, Yu-Gang Jiang, Tao Gui, Qi Zhang, Kang Wang, Yunke Zhang, Yuran Wang ### Magma: A Foundation Model for Multimodal AI Agents Introduces Magma, a multimodal agent model trained with labels for clickable GUI elements and video motion traces, and evaluated on UI navigation and robot tasks. - https://awesomecua.com/resources/arxiv-2502-13130/ · source: https://arxiv.org/abs/2502.13130v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, planning, video - Published: 2025-02-18 · Authors: Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao ### MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution MAGNET maintains separate memories for stable interface functions and task procedures across changing layouts and workflows. It updates frequently used knowledge and evaluates adaptation in AndroidWorld and offline distribution-shift settings. - https://awesomecua.com/resources/arxiv-2601-19199/ · source: https://arxiv.org/abs/2601.19199v2 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, grounding, robustness, evaluation - Published: 2026-01-27 · Authors: Libo Sun, Jiwen Zhang, Siyuan Wang, Zhongyu Wei ### MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments Unifies GUI evaluation, trajectory collection, environment synthesis, and online reinforcement learning across mobile, computer, and web tasks. - https://awesomecua.com/resources/arxiv-2610-20939/ · source: https://arxiv.org/abs/2610.20939v1 - Paper · reviewed 2026-10-16 · environments: cross-platform, desktop, web · topics: sandbox, evaluation, reinforcement-learning - Published: 2026-10-14 · Authors: Panrong Tong, Quyu Kong, Pengxiang Li, Tianyu Xia, Hanzhang Zhou, Lei Wang, Chenglin Cai, Long Li, Long Chen, Jianan Zhang, Zhengwen Zeng, Xu Zhang, Gongjie Zhang, Yue Wang, Steven Hoi ### MAI-UI Technical Report: Real-World Centric Foundation GUI Agents MAI-UI combines GUI navigation, native user interaction, and MCP tool calls in a family of policies trained through generated data and online reinforcement learning. The report also studies device-cloud routing and scaling of parallel environments and interaction budgets, with grounding and Android navigation evaluation. - https://awesomecua.com/resources/arxiv-2512-22047/ · source: https://arxiv.org/abs/2512.22047v1 - Paper · reviewed 2026-10-17 · environments: android · topics: human-in-the-loop, tool-integration, reinforcement-learning, data-generation, efficiency - Published: 2025-12-26 · Authors: Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyu Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, Steven Hoi ### Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions Introduces GPTDroid for mobile GUI testing: it feeds app page information to an LLM, runs generated test scripts and uses feedback and memory to guide further exploration. - https://awesomecua.com/resources/arxiv-2310-15780/ · source: https://arxiv.org/abs/2310.15780v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, memory, screen-understanding, mobile-navigation - Published: 2023-10-24 · Authors: Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, Qing Wang ### MalURLBench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web URLs MalURLBench evaluates how LLM-based web agents handle disguised malicious URLs across attack scenarios; the paper also analyzes attack factors and proposes URLGuard. - https://awesomecua.com/resources/arxiv-2601-18113/ · source: https://arxiv.org/abs/2601.18113v3 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2026-01-26 · Authors: Dezhang Kong, Zhuxi Wu, Shiqi Liu, Zhicheng Tan, Kuichen Lu, Minghao Li, Qichen Liu, Shengyu Chu, Zhenhua Xu, Xuan Liu, Meng Han ### MANA: Towards Efficient Mobile Ad Detection via Multimodal Agentic UI Navigation MANA combines visual, temporal and exploration signals to guide navigation toward mobile advertisements. It evaluates ad detection and interaction cost on commercial smartphones, including obfuscated-ad cases. - https://awesomecua.com/resources/arxiv-2603-20351/ · source: https://arxiv.org/abs/2603.20351v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: mobile-navigation, screen-understanding, efficiency, safety-security - Published: 2026-03-20 · Authors: Yizhe Zhao, Yongjian Fu, Zihao Feng, Hao Pan, Yongheng Deng, Yaoxue Zhang, Ju Ren ### Mango: Multi-Agent Web Navigation via Global-View Optimization Proposes Mango, a multi-agent web-navigation method that selects starting URLs using Thompson Sampling and stores navigation history to inform later attempts. - https://awesomecua.com/resources/arxiv-2604-18779/ · source: https://arxiv.org/abs/2604.18779v1 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, planning, memory, efficiency - Published: 2026-04-20 · Authors: Weixi Tong, Yifeng Di, Tianyi Zhang ### Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree Studies indirect prompt injection through HTML content consumed as accessibility trees. The evaluation tests how manipulated page observations redirect browser-agent actions on real websites. - https://awesomecua.com/resources/arxiv-2507-14799/ · source: https://arxiv.org/abs/2507.14799v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2025-07-20 · Authors: Sam Johnson, Viet Pham, Thai Le ### Mano Technical Report Introduces Mano, a GUI agent trained with simulated data, supervised fine-tuning, and offline and online RL, with a verification module for error recovery. - https://awesomecua.com/resources/arxiv-2509-17336/ · source: https://arxiv.org/abs/2509.17336v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, reinforcement-learning, self-reflection - Published: 2025-09-22 · Authors: Tianyu Fu, Anyang Su, Chenxu Zhao, Hanning Wang, Minghui Wu, Zhe Yu, Fei Hu, Mingjia Shi, Wei Dong, Jiayao Wang, Yuyang Chen, Ruiyang Yu, Siran Peng, Menglin Li, Nan Huang, Haitian Wei, Jiawei Yu, Yi Xin, Xilin Zhao, Kai Gu, Ping Jiang, Sifan Zhou, Shuo Wang ### MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automation Builds reusable page memories from past mobile trajectories and retrieves them to support task planning and execution. - https://awesomecua.com/resources/arxiv-2507-21953/ · source: https://arxiv.org/abs/2507.21953v1 - Paper · reviewed 2026-10-16 · environments: android · topics: memory, planning, mobile-navigation - Published: 2025-07-29 · Authors: Yi Kong, Dianxi Shi, Guoli Yang, Zhang ke-di, Chenlin Huang, Xiaopeng Li, Songchang Jin ### MAPLE: A Multimodal Mobile Agent with Playbook-Learned Experience Introduces MAPLE, a training-free multi-agent framework that shares step-level experience while navigating mobile GUIs and updates prompts during execution. - https://awesomecua.com/resources/arxiv-2611-01351/ · source: https://arxiv.org/abs/2611.01351v1 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, interaction-history, training-free - Published: 2026-10-31 · Authors: Jun Wang, Aurora Wei, Lucas Spangher, Ming Zhang, Eugene Ie, Yiannis Kantaros ### Mapping Natural Language Commands to Web Elements Defines language grounding on webpages as choosing the UI element targeted by a natural-language command. A collection of more than 50,000 commands and three baseline models study functional references, relationships between elements, and visual position cues. - https://awesomecua.com/resources/arxiv-1808-09132/ · source: https://arxiv.org/abs/1808.09132v2 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, data-generation, evaluation - Published: 2018-08-28 · Authors: Panupong Pasupat, Tian-Shun Jiang, Evan Zheran Liu, Kelvin Guu, Percy Liang ### Mapping Natural Language Instructions to Mobile UI Action Sequences Introduces PIXELHELP for predicting mobile UI action sequences from instructions, together with action-phrase annotations and synthetic UI action descriptions for training. Separate Transformers extract instruction steps and ground object descriptions using interface content and screen position. - https://awesomecua.com/resources/arxiv-2005-03776/ · source: https://arxiv.org/abs/2005.03776v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, grounding, data-generation, action-prediction - Published: 2020-05-07 · Authors: Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge ### Mapping the Design Space of User Experience for Computer Use Agents Develops a taxonomy of computer-use-agent interaction design through system review and practitioner interviews. A Wizard-of-Oz study then examines how users respond to normal, erroneous, and risky web-based execution, focusing on control, explanation, prompts, and mental models. - https://awesomecua.com/resources/arxiv-2602-07283/ · source: https://arxiv.org/abs/2602.07283v1 - Paper · reviewed 2026-10-17 · environments: web · topics: human-in-the-loop, ui-design, evaluation - Published: 2026-02-07 · Authors: Ruijia Cheng, Jenny T. Liang, Eldon Schoop, Jeffrey Nichols ### MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents MAS-Bench evaluates mobile agents that combine GUI actions with APIs, deep links, and reusable scripts. Its 139 tasks across 11 apps test use of 88 predefined shortcuts as well as the ability to generate new reusable workflows. - https://awesomecua.com/resources/arxiv-2509-06477/ · source: https://arxiv.org/abs/2509.06477v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, tool-integration, workflow - Published: 2025-09-08 · Authors: Pengxiang Zhao, Guangyi Liu, YaoZhen Liang, Weiqing He, Zhengxi Lu, WenHao Wang, Yuehao Huang, Yuxiang Chai, Zhaolu Kang, Yaxuan Guo, Hao Wang, Kexin Zhang, Liang Liu, Yong Liu ### MaskClaw: Edge-Side Personalized Privacy Arbitration for GUI Agents with Behavior-Driven Skill Evolution MaskClaw makes local Allow, Mask or Ask decisions before GUI screenshots leave a trusted environment, using visual evidence and personalized policy memory. Its benchmark and designed evolution scenarios test how corrections and edits can become privacy skills checked by a sandbox gate. - https://awesomecua.com/resources/arxiv-2605-28646/ · source: https://arxiv.org/abs/2605.28646v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, memory, human-in-the-loop, evaluation - Published: 2026-05-27 · Authors: Yanqiu Zhao, Dongying Zheng, Kaibo Huang, Yukun Wei, Zhongliang Yang, Linna Zhou ### Masking Stale Observations Helps Search Agents -- Until It Doesn't: A Regime Map and Its Mechanism Studies when masking older retrieved pages helps long-horizon search agents manage context, using offline and live-web search benchmarks and analysis of reopened pages. - https://awesomecua.com/resources/arxiv-2606-00408/ · source: https://arxiv.org/abs/2606.00408v1 - Paper · reviewed 2026-11-06 · environments: web · topics: efficiency, interaction-history, evaluation, web-navigation - Published: 2026-05-29 · Authors: Haoxiang Zhang, Qixin Xu, Zhuofeng Li, Lei Zhang, Pengcheng Jiang, Yu Zhang, Julian McAuley ### MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning Introduces MATE, a policy-conditioned auditor for mobile-agent trajectories, and MATEBench for evaluating whether those trajectories violate app security policies. - https://awesomecua.com/resources/arxiv-2609-22724/ · source: https://arxiv.org/abs/2609.22724v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, data-generation, mobile-navigation - Published: 2026-09-19 · Authors: Changyue Jiang, Jiayi Wang, Xin Wen, Jiarun Dai, Geng Hong, Xudong Pan ### MatToolBench: Benchmarking Multimodal Agents in Real-World Materials Science Workflows Introduces MatToolBench, a Windows-based materials-science benchmark combining GUI operation, scientific-software scripting, and database queries. Expert-defined subcriteria support partial-credit scoring, while the evaluation examines domain knowledge gaps and cross-tool handoff failures. - https://awesomecua.com/resources/arxiv-2609-37053/ · source: https://arxiv.org/abs/2609.37053v1 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: science, evaluation, cross-app - Published: 2026-09-29 · Authors: Mei Wu, Rui Xie, Runyu Zhang, Yuqiang Li, Tianfan Fu, Bo Chen, Kai Yu, Xin Chen, Lu Chen ### MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web (Technical Report) Compares HTML browsing, RAG, MCP, and NLWeb interfaces for web agents performing product search, comparison, and checkout tasks in four simulated e-shops. - https://awesomecua.com/resources/arxiv-2511-23281/ · source: https://arxiv.org/abs/2511.23281v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, tool-integration, efficiency, web-navigation - Published: 2025-11-28 · Authors: Aaron Steiner, Ralph Peeters, Christian Bizer ### MCP-Driven Accessibility Tree Standardization for AI-Powered Screen Reader Agents Proposes an MCP architecture that presents accessibility trees from Windows, macOS, Android, and web frameworks in a common form for LLM assistive agents interacting with GUIs. - https://awesomecua.com/resources/arxiv-2608-24898/ · source: https://arxiv.org/abs/2608.24898v1 - Paper · reviewed 2026-11-06 · environments: desktop, windows, macos, android, web · topics: screen-understanding, tool-integration - Published: 2026-07-13 · Authors: Vishnu Ramineni, Nitin Saksena, Akash Kumar Agarwal, Darshan Mohan Bidkar, Balakrishna Pothineni, Durgaraman Maruthavanan, Lokesh Butra, Siva Kumar Chintham ### MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents MCPWorld evaluates API, GUI, and hybrid computer-use agents with 201 tasks in applications whose source can be instrumented and extended with MCP support. It verifies completion from application behavior and packages the testbed in containers with GPU support. - https://awesomecua.com/resources/arxiv-2506-07672/ · source: https://arxiv.org/abs/2506.07672v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, tool-integration, sandbox - Published: 2025-06-09 · Authors: Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, Mengwei Xu ### Me-Agent: A Personalized Mobile Agent with Two-Level User Habit Learning for Enhanced Interaction Me-Agent personalizes mobile agent behavior through a preference reward model and hierarchical memory for long-term and app-specific habits, and introduces User FingerTip. - https://awesomecua.com/resources/arxiv-2601-20162/ · source: https://arxiv.org/abs/2601.20162v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: mobile-navigation, memory, reward-models, interaction-history - Published: 2026-01-28 · Authors: Shuoxin Wang, Chang Liu, Gowen Loo, Lifan Zheng, Kaiwen Wei, Xinyi Zeng, Jingyuan Zhang, Yu Tian ### Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding Trains a GUI grounding proposer together with a visual critic that evaluates candidate locations rendered on screenshots. An adaptive reinforcement-learning schedule balances their development, using proposal diversity and critic feedback during training. - https://awesomecua.com/resources/arxiv-2604-21268/ · source: https://arxiv.org/abs/2604.21268v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, self-reflection - Published: 2026-04-23 · Authors: Wenkai Wang, Xiyun Li, Hongcan Guo, Wenhao Yu, Tianqing Fang, Haitao Mi, Dong Yu, Shengyu Zhang ### Measuring Harmfulness of Computer-Using Agents Introduces CUAHarm, 104 computer-use misuse tasks in a sandbox with verifiable rewards, and studies whether language models can monitor unsafe agent actions. - https://awesomecua.com/resources/arxiv-2508-00935/ · source: https://arxiv.org/abs/2508.00935v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, sandbox, observability - Published: 2025-07-31 · Authors: Aaron Xuxiang Tian, Ruofan Zhang, Janet Tang, Ji Wang, Tianyu Shi, Jiaxin Wen ### Measuring the Security of Mobile LLM Agents under Adversarial Prompts from Untrusted Third-Party Channels Measures how untrusted third-party notifications, messages, and app content can hijack mobile LLM agents into executing unauthorized cross-app actions. - https://awesomecua.com/resources/arxiv-2510-27140/ · source: https://arxiv.org/abs/2510.27140v3 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, mobile-navigation, evaluation - Published: 2025-10-31 · Authors: Chenghao Du, Quanfeng Huang, Tingxuan Tang, Zihao Wang, Adwait Nadkarni, Yue Xiao ### MedBrowseComp: Benchmarking Medical Deep Research and Computer Use Introduces MedBrowseComp, a benchmark of human-curated, multi-hop medical questions for agents retrieving and synthesizing facts from live knowledge bases in a computer-use setting. - https://awesomecua.com/resources/arxiv-2505-14963/ · source: https://arxiv.org/abs/2505.14963v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon - Published: 2025-05-20 · Authors: Shan Chen, Pedro Moreira, Yuxin Xiao, Sam Schmidgall, Jeremy Warner, Hugo Aerts, Thomas Hartvigsen, Jack Gallifant, Danielle S. Bitterman ### MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents MedCUA-Bench evaluates screenshot-only clinical interface workflows using paired intent- and step-level goals. It combines reconstructed medical interfaces with completion and clinical-safety checks, and separately reports results on a real OpenEMR system. - https://awesomecua.com/resources/arxiv-2606-03203/ · source: https://arxiv.org/abs/2606.03203v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, safety-security, science, workflow - Published: 2026-06-02 · Authors: Jia Yu, Zilong Wang, Xinyang Jiang, Dongsheng Li, Shuo Wang ### MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles Evaluates GUI agents in simulated medical-imaging consoles, separately checking workflow completion and acquisition outcomes against hidden anatomical targets. - https://awesomecua.com/resources/arxiv-2610-04800/ · source: https://arxiv.org/abs/2610.04800v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: evaluation, science, planning - Published: 2026-10-03 · Authors: Ziyang Long, Xinqi Li, Lujing Xing, Hsin-Jung Yang ### MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI Benchmarks sequential target localization in clinical software workflows, emphasizing how early mistakes affect subsequent interaction steps. - https://awesomecua.com/resources/arxiv-2603-19993/ · source: https://arxiv.org/abs/2603.19993v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: grounding, evaluation - Published: 2026-03-20 · Authors: Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq, Tausifa Jan Saleem, Tajamul Ashraf ### MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements Introduces MEGA-GUI, which separates coarse region selection from fine-grained interface-element grounding. Bidirectional zooming and context-aware instruction rewriting coordinate specialized vision-language agents across visual scales. - https://awesomecua.com/resources/arxiv-2511-13087/ · source: https://arxiv.org/abs/2511.13087v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding - Published: 2025-11-17 · Authors: SeokJoo Kwak, Jihoon Kim, Boyoun Kim, Jung Jae Yoon, Wooseok Jang, Jeonghoon Hong, Jaeho Yang, Yeong-Dae Kwon ### Mem-W: Latent Memory-Native GUI Agents Introduces Mem-W, which compresses previous trajectories and current-session segments into latent memory tokens used directly by the GUI policy. Self-distillation and outcome supervision train this memory representation for web and mobile navigation. - https://awesomecua.com/resources/arxiv-2605-09317/ · source: https://arxiv.org/abs/2605.09317v1 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, long-horizon, imitation-learning - Published: 2026-05-10 · Authors: Guibin Zhang, Yaohui Ling, Fanci Meng, Kun Wang, Shuicheng Yan ### MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents MementoGUI learns a controller for selecting, compressing and retrieving textual and region-level visual memories while leaving the GUI-agent backbone fixed. It also converts computer-use traces into controller training data and evaluates semantic actions, progress and memory consistency on long-horizon tasks. - https://awesomecua.com/resources/arxiv-2605-18652/ · source: https://arxiv.org/abs/2605.18652v1 - Paper · reviewed 2026-10-18 · environments: web · topics: memory, interaction-history, long-horizon, data-generation, evaluation - Published: 2026-05-18 · Authors: Ziyun Zeng, Hang Hua, Bocheng Zou, Mu Cai, Rogerio Feris, Jiebo Luo ### MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management MemGUI-Agent makes context management an action chosen by the same policy that controls the mobile UI. Structured history, UI-state and recent-step fields are trained from annotated trajectories to retain cross-app facts while limiting context growth. - https://awesomecua.com/resources/arxiv-2606-19926/ · source: https://arxiv.org/abs/2606.19926v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: memory, interaction-history, long-horizon, imitation-learning, data-generation - Published: 2026-06-18 · Authors: Guangyi Liu, Gao Wu, Congxiao Liu, Pengxiang Zhao, Liang Liu, Mading Li, Qi Zhang, Mengyan Wang, Liang Guo, Yong Liu ### MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments MemGUI-Bench evaluates short-term retention and cross-session experience reuse in mobile GUI agents using paired tasks and repeated-attempt protocols. Its staged evaluator measures memory fidelity, learning and execution efficiency. - https://awesomecua.com/resources/arxiv-2602-06075/ · source: https://arxiv.org/abs/2602.06075v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, evaluation, mobile-navigation, cross-app - Published: 2026-02-03 · Authors: Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Qinyi Luo, Shunye Tang, Yuxiang Chai, Weifeng Lin, Han Xiao, WenHao Wang, Siheng Chen, Zhengxi Lu, Gao Wu, Hao Wang, Liang Liu, Yong Liu ### MemGym: a Long-Horizon Memory Environment for LLM Agents MemGym benchmarks agent memory across several task regimes, including computer use on WebArena-Infinity, with scores designed to isolate memory performance. - https://awesomecua.com/resources/arxiv-2605-20833/ · source: https://arxiv.org/abs/2605.20833v1 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, evaluation, long-horizon - Published: 2026-05-20 · Authors: Wujiang Xu, Yu Wang, Kai Mei, Kaiqu Liang, Zhenting Wang, Mingyu Jin, Han Zhang, Shi-Xiong Zhang, Wenyue Hua, Sambit Sahu, Dimitris N. Metaxas ### MemHarness: Memory Is Reconstructed, Not Replayed Introduces MemHarness, which trains agents to critique and reconstruct retrieved experience before acting; WebShop is one of two evaluation environments. - https://awesomecua.com/resources/arxiv-2607-28272/ · source: https://arxiv.org/abs/2607.28272v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, reinforcement-learning, self-reflection - Published: 2026-07-30 · Authors: Rong Wu, Daocheng Fu, Licheng Wen, Xuemeng Yang, Shu Zou, Jianbiao Mei, Yuxin Wang, Hairong Zhang, Yu Yang, Tao Hu, Cong Zhang, Botian Shi, Pinlong Cai ### MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks MemoryArena links tasks across sessions so that agents must distill earlier interaction experience and use it in later decisions. Its evaluation includes web navigation alongside planning, search and formal reasoning, testing memory through subsequent actions rather than recall alone. - https://awesomecua.com/resources/arxiv-2602-16313/ · source: https://arxiv.org/abs/2602.16313v2 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, interaction-history, planning, evaluation, long-horizon - Published: 2026-02-18 · Authors: Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, Alex Pentland ### MemPrism: Task-Conditioned Relational Memory Views for Long-Horizon Agents Introduces MemPrism, which records prior interactions and builds task-conditioned relational memory views for decisions in long-horizon web-agent tasks. - https://awesomecua.com/resources/arxiv-2608-06745/ · source: https://arxiv.org/abs/2608.06745v2 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, interaction-history, long-horizon, efficiency - Published: 2026-08-07 · Authors: Bingfan Zeng, Zhisheng Chen, Zhengwei Xie, Bangde Cao, Yuxuan Li, Jinhan Li, Zheng Lu, Xiangchen Guan, Zikai Xiao, Rui Qian, Jingwei Song ### MemU-Bench: Benchmarking GUI Agents on Long-Horizon User Interactions Introduces MemU-Bench for mobile GUI tasks that depend on past user interactions, alongside AppMem, a training-free memory framework for retrieving app-specific context. - https://awesomecua.com/resources/arxiv-2611-00191/ · source: https://arxiv.org/abs/2611.00191v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, interaction-history, memory, mobile-navigation - Published: 2026-10-30 · Authors: Yizhou Liu, Zhengxi Lu, Fei Tang, Yiwen Qiu, Yong Du, Tongbo Chen, Bofan Chen, Tao Jiang, Wenhao Xu, Yunfeng Shu, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen ### MemVenom: Triggered Poisoning of Multimodal Memories in Web Agents Introduces MemVenom, a black-box attack that plants coordinated text and image content in web-agent memory to trigger retrieval and redirect later actions. - https://awesomecua.com/resources/arxiv-2606-10742/ · source: https://arxiv.org/abs/2606.10742v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, memory, web-navigation - Published: 2026-06-09 · Authors: Yv Zhang, Hao Sun, Hao Fang, Kuofeng Gao, Fan Mo, Bin Chen, Shu-Tao Xia, Yaowei Wang ### META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI Introduces GUI-based task-oriented dialogue and a dataset pairing mobile-interface operations with conversational responses. - https://awesomecua.com/resources/arxiv-2205-11029/ · source: https://arxiv.org/abs/2205.11029v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, human-in-the-loop - Published: 2022-05-23 · Authors: Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, Kai Yu ### MGA: Memory-Driven GUI Agent for Observation-Centric Interaction Introduces MGA, which separates task-agnostic screen-state reading from decision making and connects steps through structured state memory. Validated state deltas replace full trajectory replay, evaluated on OSWorld and application tasks. - https://awesomecua.com/resources/arxiv-2510-24168/ · source: https://arxiv.org/abs/2510.24168v3 - Paper · reviewed 2026-10-17 · environments: desktop · topics: memory, state-tracking, long-horizon - Published: 2025-10-28 · Authors: Weihua Cheng, Junming Liu, Yifei Sun, Botian Shi, Yirong Chen, Ding Wang ### MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search Introduces MHVF, a training-free grounding method that explores several candidate screen regions, verifies them in local views, and uses those checks to guide further search. - https://awesomecua.com/resources/arxiv-2610-17792/ · source: https://arxiv.org/abs/2610.17792v1 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, screen-understanding, robustness - Published: 2026-10-13 · Authors: Minghua Chen, Xingming Liao, Weiqiang Zhong, Zhuowei Wang ### MiMo-VL Technical Report Reports MiMo-VL-7B-SFT and MiMo-VL-7B-RL, general vision-language models trained with staged pre-training and mixed RL, with GUI grounding evaluated on OSWorld-G. - https://awesomecua.com/resources/arxiv-2506-03569/ · source: https://arxiv.org/abs/2506.03569v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, reinforcement-learning, evaluation - Published: 2025-06-04 · Authors: Xiaomi LLM-Core Team, :, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhixian Zheng, Zhichao Song, Zhenbo Luo, Yue Yu, Yudong Wang, Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huang, Xu Wang, Xinzhe Xu, Xingchen Song, Xing Zhang, Xing Yong, Xin Zhang, Xiangwei Deng, Wenyu Yang, Wenhan Ma, Weiwei Lv, Weiji Zhuang, Wei Liu, Sirui Deng, Shuo Liu, Shimao Chen, Shihua Yu, Shaohui Liu, Shande Wang, Rui Ma, Qiantong Wang, Peng Wang, Nuo Chen, Menghang Zhu, Kangyang Zhou, Kang Zhou, Kai Fang, Jun Shi, Jinhao Dong, Jiebao Xiao, Jiaming Xu, Huaqiu Liu, Hongshen Xu, Heng Qu, Haochen Zhao, Hanglong Lv, Guoan Wang, Duo Zhang, Dong Zhang, Di Zhang, Chong Ma, Chang Liu, Can Cai, Bingquan Xia ### Mind the Gap: Action Rebinding Attacks against Android GUI Agents Studies action-rebinding attacks in which an Android GUI agent’s planned input is redirected to a different application during the observation–action delay. The evaluation examines cross-application execution and the consequences of treating an earlier screen as the current action target. - https://awesomecua.com/resources/arxiv-2601-12349/ · source: https://arxiv.org/abs/2601.12349v3 - Paper · reviewed 2026-10-17 · environments: android · topics: safety-security, cross-app, evaluation - Published: 2026-01-18 · Authors: Yi Qian, Kunwei Qian, Xingbang He, Ligeng Chen, Jikang Zhang, Tiantai Zhang, Haiyang Wei, Linzhang Wang, Hao Wu, Bing Mao ### Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agents Introduces SAPA-Bench with 7,138 smartphone scenarios to evaluate whether MLLM-powered mobile agents detect and protect sensitive personal information on screen. - https://awesomecua.com/resources/arxiv-2508-19493/ · source: https://arxiv.org/abs/2508.19493v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, mobile-navigation, evaluation - Published: 2025-08-27 · Authors: Zhixin Lin, Jungang Li, Shidong Pan, Yibo Shi, Yue Yao, Dongliang Xu ### Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge Introduces Mind2Web 2, 130 long-horizon tasks requiring live web browsing and information synthesis, plus an Agent-as-a-Judge framework for grading answers and sources. - https://awesomecua.com/resources/arxiv-2506-21506/ · source: https://arxiv.org/abs/2506.21506v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, long-horizon - Published: 2025-06-26 · Authors: Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song, Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, Yu Su ### Mind2Web: Towards a Generalist Agent for the Web Introduces a web-task dataset spanning diverse websites and evaluates generalization to unseen sites and domains. - https://awesomecua.com/resources/arxiv-2306-06070/ · source: https://arxiv.org/abs/2306.06070v3 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation, evaluation - Published: 2023-06-09 · Authors: Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, Yu Su ### MindSearch: Mimicking Human Minds Elicits Deep AI Searcher MindSearch uses a planner and searcher agents to split information requests into subquestions and gather answers across web pages. - https://awesomecua.com/resources/arxiv-2407-20183/ · source: https://arxiv.org/abs/2407.20183v2 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, web-navigation, cross-app - Published: 2024-07-29 · Authors: Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, Feng Zhao ### MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control Introduces MineDreamer, a Minecraft agent that imagines stages of an instruction and turns them into keyboard-and-mouse actions in a simulated world. - https://awesomecua.com/resources/arxiv-2403-12037/ · source: https://arxiv.org/abs/2403.12037v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: planning, world-models, action-prediction - Published: 2024-03-18 · Authors: Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, Jing Shao ### MintAct: A Unified Visual Agent for Digital Environments MintAct trains a visual-agent family for grounding, navigation and tool use across mobile, desktop and web environments. Its shared infrastructure supports trajectory collection and asynchronous online RL while controlling cross-domain sampling and off-policy drift. - https://awesomecua.com/resources/arxiv-2609-22083/ · source: https://arxiv.org/abs/2609.22083v1 - Paper · reviewed 2026-10-18 · environments: cross-platform · topics: grounding, reinforcement-learning, data-generation, tool-integration - Published: 2026-09-18 · Authors: Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege Özsoy, Kaixin Ma, Vishwesh Kirthivasan, Oğuzhan Fatih Kar, Roman Bachmann, Anders Boesen Lindbo Larsen, Afshin Dehghan ### MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents Studies Malicious Image Patches (MIPs), altered screen regions that can steer screenshot-based OS agents into harmful mouse and keyboard actions. - https://awesomecua.com/resources/arxiv-2503-10809/ · source: https://arxiv.org/abs/2503.10809v3 - Paper · reviewed 2026-11-06 · environments: desktop · topics: safety-security, screen-understanding - Published: 2025-03-13 · Authors: Lukas Aichberger, Alasdair Paren, Guohao Li, Philip Torr, Yarin Gal, Adel Bibi ### Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills Mirage-1 organizes interaction trajectories into execution, core, and meta-skills, then uses them to guide Monte Carlo tree search during online GUI tasks. It introduces AndroidLH for long-horizon evaluation and tests the approach on Android and web-navigation benchmarks. - https://awesomecua.com/resources/arxiv-2506-10387/ · source: https://arxiv.org/abs/2506.10387v1 - Paper · reviewed 2026-10-17 · environments: android, web, cross-platform · topics: planning, memory, long-horizon, mobile-navigation, web-navigation - Published: 2025-06-12 · Authors: Yuquan Xie, Zaijing Li, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Dongmei Jiang, Liqiang Nie ### MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content MIRAGE constructs mobile screenshot injections in ordinary user-content regions through localization, contextual payload generation and realism curation. Its evaluation separates visual realism from attack effectiveness against pixel-observing GUI agents. - https://awesomecua.com/resources/arxiv-2605-28116/ · source: https://arxiv.org/abs/2605.28116v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, data-generation, evaluation, robustness - Published: 2026-05-27 · Authors: Ruoqi Guo, Yi Liu, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Ji Jie, Yuxiao Lu ### MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models Introduces MIRAGE, which trains mobile app agents to reason in compact latent states and anticipate future screenshots, reducing generated tokens during action selection. - https://awesomecua.com/resources/arxiv-2606-04627/ · source: https://arxiv.org/abs/2606.04627v2 - Paper · reviewed 2026-11-06 · environments: android · topics: world-models, efficiency, mobile-navigation, action-prediction - Published: 2026-06-03 · Authors: Zhichao Yang, Yuanze Hu, Haojie Hao, Longkun Hao, Dongshuo Huang, Hongyu Lin, Gen Li, Lanqing Hong, Yihang Lou, Yan Bai ### MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agents MIRAGE evaluates targeted next-action hijacking from adversarial images restricted to legitimate third-party webpage regions. It combines diffusion-based image generation with execution-aware optimization and tests the constrained visual-injection setting on web-agent frameworks. - https://awesomecua.com/resources/arxiv-2606-20717/ · source: https://arxiv.org/abs/2606.20717v2 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, robustness, evaluation - Published: 2026-06-16 · Authors: Xuelong Dai, Jianyu Ma, Boyang Ma, Biwei Yan, Yijun Yang, Yue Zhang ### MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction MirrorGuard trains a reasoning-correction defense using GUI risk trajectories generated in a text-based neural-symbolic simulator. It evaluates whether corrections transfer to computer-use agents operating real interfaces, keeping simulated training separate from deployment tests. - https://awesomecua.com/resources/arxiv-2601-12822/ · source: https://arxiv.org/abs/2601.12822v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, data-generation, robustness - Published: 2026-01-19 · Authors: Wenqi Zhang, Yulin Shen, Changyue Jiang, Jiarun Dai, Geng Hong, Xudong Pan ### MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models MissClick designs white-box attacks around the numerical effect of coordinate digits on executable GUI clicks. Separate untargeted displacement and targeted place-weighted objectives are evaluated across desktop, web and mobile grounding settings. - https://awesomecua.com/resources/arxiv-2608-03740/ · source: https://arxiv.org/abs/2608.03740v2 - Paper · reviewed 2026-10-18 · environments: cross-platform · topics: safety-security, grounding, robustness, evaluation - Published: 2026-08-04 · Authors: Yu Ran, Wentao Zhao ### Mitigating Coordinate Prediction Bias from Positional Encoding Failures Introduces Vision-PE Shuffle Guidance (VPSG), a training-free correction for coordinate drift in visual models, evaluated for localization on ScreenSpot-Pro. - https://awesomecua.com/resources/arxiv-2510-22102/ · source: https://arxiv.org/abs/2510.22102v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: training-free, evaluation - Published: 2025-10-25 · Authors: Xingjian Tao, Yiwei Wang, Yujun Cai, Yihong Luo, Kai Han, Jing Tang ### Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models MixKV compresses vision-language model KV caches by balancing importance and diversity; its evaluations include GUI grounding alongside broader multimodal tasks. - https://awesomecua.com/resources/arxiv-2510-20707/ · source: https://arxiv.org/abs/2510.20707v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, grounding - Published: 2025-10-23 · Authors: Xuyang Liu, Xiyan Gui, Yuchao Zhang, Linfeng Zhang ### MKD2World: A Typed Spatial MarkDown World Model For Web Agents Introduces MKD2World, a desktop web world model that represents interface elements in Typed Spatial Markdown and predicts post-action states, supported by AtomBlock-WebUI annotations. - https://awesomecua.com/resources/arxiv-2611-06544/ · source: https://arxiv.org/abs/2611.06544v1 - Paper · reviewed 2026-11-06 · environments: web, desktop · topics: world-models, screen-understanding, data-generation, planning - Published: 2026-11-04 · Authors: Zhihao Nan, Sifan Zhou, Yiming Cheng, Jinkai Zheng, Zheng Zhu, Si Shi, Ming Li, Fei Yu ### MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments Benchmarks truthfulness, controllability, safety, and privacy of multimodal LLM agents across interactive web and mobile GUI tasks. - https://awesomecua.com/resources/arxiv-2506-01616/ · source: https://arxiv.org/abs/2506.01616v1 - Paper · reviewed 2026-10-18 · environments: web, android, cross-platform · topics: safety-security, evaluation, robustness - Published: 2025-06-02 · Authors: Xiao Yang, Jiawei Chen, Jun Luo, Zhengwei Fang, Yinpeng Dong, Hang Su, Jun Zhu ### MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents Introduces MM-BrowseComp, a benchmark for browsing agents that must use evidence from web images and videos to answer multimodal search questions. - https://awesomecua.com/resources/arxiv-2508-13186/ · source: https://arxiv.org/abs/2508.13186v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, video - Published: 2025-08-14 · Authors: Shilong Li, Xingyuan Bu, Wenjie Wang, Jiaheng Liu, Jun Dong, Haoyang He, Hao Lu, Haozhe Zhang, Chenchen Jing, Zhen Li, Chuanhao Li, Jiayi Tian, Chenchen Zhang, Tianhao Peng, Yancheng He, Jihao Gu, Hui Huang, Donghao Zhou, Yuanxing Zhang, Jian Yang, Ge Zhang, Wenhao Huang, Zhaoxiang Zhang, Qiangpeng Yang, Shilei Wen ### MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning MM1.5 is a multimodal model family trained with curated data mixtures; its MM1.5-UI variant targets mobile UI understanding. - https://awesomecua.com/resources/arxiv-2409-20566/ · source: https://arxiv.org/abs/2409.20566v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, data-generation - Published: 2024-09-30 · Authors: Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afshin Dehghan, Peter Grasch, Yinfei Yang ### MMA-SafetyBench: A Benchmark for Multimodal Agent Safety Evaluation Builds trajectory-guided visual-injection tests for web, desktop and mobile observations, separating target-matching action predictions from broader semantic compromise. - https://awesomecua.com/resources/arxiv-2610-19384/ · source: https://arxiv.org/abs/2610.19384v1 - Paper · reviewed 2026-10-20 · environments: web, desktop · topics: evaluation, safety-security, grounding - Published: 2026-10-14 · Authors: Yuke Wang, Benlei Cui, Shen Pang, Xuemei Dong, Longtao Huang, Hui Xue, Yuwen Zhai, Junjie Li, Jingqun Tang, Haiwen Hong - Licenses: document CC-BY-4.0 ### MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot Introduces MMAC-Copilot, which coordinates agents with different expertise to operate PC applications. VIBench evaluates visual interaction with applications that lack usable APIs, spanning office, recreation and game tasks. - https://awesomecua.com/resources/arxiv-2404-18074/ · source: https://arxiv.org/abs/2404.18074v3 - Paper · reviewed 2026-10-17 · environments: desktop · topics: automation, evaluation, planning - Published: 2024-04-28 · Authors: Zirui Song, Yaohang Li, Meng Fang, Yanda Li, Zhenhao Chen, Zecheng Shi, Yuan Huang, Xiuying Chen, Ling Chen ### MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents MMBench-GUI evaluates interface understanding, element grounding, task automation, and collaboration across six named desktop, mobile, and web platforms. It introduces the Efficiency-Quality Area metric for examining execution efficiency alongside task quality in online GUI automation. - https://awesomecua.com/resources/arxiv-2507-19478/ · source: https://arxiv.org/abs/2507.19478v1 - Paper · reviewed 2026-10-17 · environments: web, desktop, windows, macos, linux, ios, android, cross-platform · topics: evaluation, grounding, efficiency, planning - Published: 2025-07-25 · Authors: Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, Weiyun Wang, Xiangyu Zhao, Jixuan Chen, Haodong Duan, Tianbao Xie, Chenyu Yang, Shiqian Su, Yue Yu, Yuan Huang, Yiqian Liu, Xiao Zhang, Yanting Zhang, Xiangyu Yue, Weijie Su, Xizhou Zhu, Wei Shen, Jifeng Dai, Wenhai Wang ### MMInA: Benchmarking Multihop Multimodal Internet Agents Evaluates multimodal agents on compositional tasks across changing websites and studies replaying prior trajectories as memory support. - https://awesomecua.com/resources/arxiv-2404-09992/ · source: https://arxiv.org/abs/2404.09992v2 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, long-horizon, memory - Published: 2024-04-15 · Authors: Shulin Tian, Ziniu Zhang, Liangyu Chen, Ziwei Liu ### MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents Introduces MMSearch-Plus, a multimodal browsing benchmark with iterative image-text search, and an agent framework that marks and crops images for targeted retrieval. - https://awesomecua.com/resources/arxiv-2508-21475/ · source: https://arxiv.org/abs/2508.21475v3 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, tool-integration, long-horizon - Published: 2025-08-29 · Authors: Xijia Tao, Yihua Teng, Xinxing Su, Xinyu Fu, Jihao Wu, Chaofan Tao, Ziru Liu, Haoli Bai, Rui Liu, Lingpeng Kong ### MMTB: Evaluating Terminal Agents on Multimedia-File Tasks MultiMedia-TerminalBench evaluates agents that use terminal tools to work with audio and video files. The Terminus-MM harness adds multimedia perception to study how accessible evidence changes executable workflows and task outcomes. - https://awesomecua.com/resources/arxiv-2605-10966/ · source: https://arxiv.org/abs/2605.10966v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, automation, video, tool-integration - Published: 2026-05-08 · Authors: Chiyeong Heo, Jaechang Kim, Junhyuk Kwon, Hoyoung Kim, Dongmin Park, Jonghyun Lee, Jungseul Ok ### MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation Combines adaptive mobile-task planning, reflection-based recovery, and multiple memory components, evaluated with MobBench and AndroidArena. - https://awesomecua.com/resources/arxiv-2410-13757/ · source: https://arxiv.org/abs/2410.13757v3 - Paper · reviewed 2026-10-16 · environments: android · topics: memory, planning, mobile-navigation - Published: 2024-10-17 · Authors: Zichen Zhu, Hao Tang, Yansi Li, Dingye Liu, Hongshen Xu, Kunyao Lan, Danyang Zhang, Yixuan Jiang, Hao Zhou, Chenrun Wang, Situo Zhang, Liangtai Sun, Yixiao Wang, Yuheng Sun, Lu Chen, Kai Yu ### MobiAgent: A Systematic Framework for Customizable Mobile Agents Presents MobiAgent, combining MobiMind agent models, the AgentRR acceleration framework, and MobiFlow evaluation. It also introduces an AI-assisted data-collection pipeline for mobile GUI tasks. - https://awesomecua.com/resources/arxiv-2509-00531/ · source: https://arxiv.org/abs/2509.00531v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, efficiency, data-generation - Published: 2025-08-30 · Authors: Cheng Zhang, Erhu Feng, Xi Zhao, Yisheng Zhao, Wangbo Gong, Jiahui Sun, Dong Du, Zhichao Hua, Yubin Xia, Haibo Chen ### MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents MobiBench evaluates mobile GUI agents offline while allowing multiple valid action paths. It decomposes agent pipelines into modules to study component contributions, configurations, and performance bottlenecks. - https://awesomecua.com/resources/arxiv-2512-12634/ · source: https://arxiv.org/abs/2512.12634v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, observability - Published: 2025-12-14 · Authors: Youngmin Im, Byeongung Jo, Jaeyoung Wi, Seungwoo Baek, Tae Hoon Min, Joo Hyung Lee, Sangeun Oh, Insik Shin, Sunjae Lee ### MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion Introduces MobiFlow, a mobile-agent evaluation framework that fuses task trajectories into a state graph for dynamic interaction with third-party apps and measures task outcomes. - https://awesomecua.com/resources/arxiv-2604-09587/ · source: https://arxiv.org/abs/2604.09587v1 - Paper · reviewed 2026-11-06 · environments: android · topics: evaluation, mobile-navigation, workflow - Published: 2026-02-28 · Authors: Yunfei Feng, Xi Zhao, Cheng Zhang, Dahu Feng, Daolin Cheng, Jianqi Yu, Yubin Xia, Erhu Feng ### Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments Introduces MoTIF, a dataset of natural-language requests for interactive mobile apps, including infeasible requests and follow-up questions for studying task feasibility. - https://awesomecua.com/resources/arxiv-2104-08560/ · source: https://arxiv.org/abs/2104.08560v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, evaluation, human-in-the-loop - Published: 2021-04-17 · Authors: Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A. Plummer ### Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization Proposes Trajectory Induced Preference Optimization (TIPO) to train mobile GUI agents for differing user privacy preferences across structurally different task trajectories. - https://awesomecua.com/resources/arxiv-2604-11259/ · source: https://arxiv.org/abs/2604.11259v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: mobile-navigation, safety-security, workflow - Published: 2026-04-13 · Authors: Zhixin Lin, Jungang Li, Dongliang Xu, Shidong Pan, Yibo Shi, Yuchi Liu, Yuecong Min, Yue Yao ### Mobile GUI Agents under Real-world Threats: Are We There Yet? Introduces an app-content instrumentation framework and complementary interactive and static test suites for mobile GUI agents. It studies how untrusted third-party content changes task execution and state understanding. - https://awesomecua.com/resources/arxiv-2507-04227/ · source: https://arxiv.org/abs/2507.04227v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, safety-security, evaluation - Published: 2025-07-06 · Authors: Guohong Liu, Jialei Ye, Jiacheng Liu, Yuanchun Li, Wei Liu, Pengzhi Gao, Jian Luan, Yunxin Liu ### Mobile User Interface Element Detection Via Adaptively Prompt Tuning Introduces MUI-zh and APT for detecting mobile UI elements, combining visual features with OCR descriptions to adjust category prompts in CLIP-based detectors. - https://awesomecua.com/resources/arxiv-2305-09699/ · source: https://arxiv.org/abs/2305.09699v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, data-generation, evaluation - Published: 2023-05-16 · Authors: Zhangxuan Gu, Zhuoer Xu, Haoxing Chen, Jun Lan, Changhua Meng, Weiqiang Wang ### Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks Uses hierarchical mobile agents and persistent tips and reusable action shortcuts to improve complex tasks from accumulated experience. - https://awesomecua.com/resources/arxiv-2501-11733/ · source: https://arxiv.org/abs/2501.11733v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, memory, long-horizon - Published: 2025-01-20 · Authors: Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, Heng Ji ### Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automation Uses separate retrieval systems for high-level mobile-task planning and precise interface operations, with a benchmark for extended cross-app tasks. - https://awesomecua.com/resources/arxiv-2511-12254/ · source: https://arxiv.org/abs/2511.12254v3 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, memory, long-horizon - Published: 2025-11-15 · Authors: Yuxiang Zhou, Jichang Li, Yanhao Zhang, Haonan Lu, Guanbin Li ### Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration Separates mobile planning, decision-making, and reflection, with a memory unit for task progress and relevant content. - https://awesomecua.com/resources/arxiv-2406-01014/ · source: https://arxiv.org/abs/2406.01014v1 - Paper · reviewed 2026-10-16 · environments: android · topics: planning, memory, mobile-navigation - Published: 2024-06-03 · Authors: Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang ### Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents Presents GUI-Owl-1.5 variants and a hybrid data pipeline for multimodal GUI interaction, tool use, memory, and multi-platform tasks. - https://awesomecua.com/resources/arxiv-2602-16855/ · source: https://arxiv.org/abs/2602.16855v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: tool-integration, memory, reinforcement-learning - Published: 2026-02-15 · Authors: Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, Zhiyuan Chen, Jitong Liao, Qi Zheng, Jiahui Zeng, Ze Xu, Shuai Bai, Junyang Lin, Jingren Zhou, Ming Yan ### Mobile-Agent-v3: Fundamental Agents for GUI Automation Introduces GUI-Owl models and Mobile-Agent-v3 orchestration, using environment-generated trajectories to support GUI capabilities. - https://awesomecua.com/resources/arxiv-2508-15144/ · source: https://arxiv.org/abs/2508.15144v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation, planning, mobile-navigation - Published: 2025-08-21 · Authors: Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, Ming Yan ### Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automation Extracts operational guidance from video for mobile task execution and introduces Mobile-Knowledge to evaluate the contribution of external knowledge. - https://awesomecua.com/resources/arxiv-2502-17110/ · source: https://arxiv.org/abs/2502.17110v3 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, video, memory - Published: 2025-02-24 · Authors: Junyang Wang, Haiyang Xu, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, Jitao Sang ### Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception Uses visual perception tools and task decomposition to operate mobile applications without relying on app interface metadata. - https://awesomecua.com/resources/arxiv-2401-16158/ · source: https://arxiv.org/abs/2401.16158v2 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, planning - Published: 2024-01-29 · Authors: Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang ### Mobile-Aptus: Confidence-Driven Proactive and Robust Interaction in MLLM-based Mobile-Using Agents Mobile-Aptus trains mobile agents to output actions and confidence scores, then corrects confidence bias to guide when they request human help. - https://awesomecua.com/resources/arxiv-2605-28629/ · source: https://arxiv.org/abs/2605.28629v1 - Paper · reviewed 2026-11-06 · environments: android · topics: mobile-navigation, human-in-the-loop, action-prediction - Published: 2026-05-27 · Authors: Zheng Wu, Pengzhou Cheng, Zongru Wu, Yuan Guo, Tianjie Ju, Aston Zhang, Gongshen Liu, Zhuosheng Zhang ### Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agents Mobile-Bench-v2 evaluates mobile GUI agents using multiple acceptable offline action paths, noisy interfaces with ads or pop-ups, and ambiguous instructions with scripted clarification exchanges. These splits separate step-level execution, robustness, and proactive interaction. - https://awesomecua.com/resources/arxiv-2505-11891/ · source: https://arxiv.org/abs/2505.11891v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, robustness, human-in-the-loop - Published: 2025-05-17 · Authors: Weikai Xu, Zhizheng Jiang, Yuxuan Liu, Pengzhi Gao, Wei Liu, Jian Luan, Yuanchun Li, Yunxin Liu, Bin Wang, Bo An ### Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents Evaluates mobile agents using combined UI and API actions, tasks with varying app scope, and checkpoints for intermediate progress. - https://awesomecua.com/resources/arxiv-2407-00993/ · source: https://arxiv.org/abs/2407.00993v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, cross-app, tool-integration - Published: 2024-07-01 · Authors: Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang ### Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction Provides an Android benchmark toolkit with controlled environments, intermediate feedback, and task sets for reproducible GUI-agent evaluation. - https://awesomecua.com/resources/arxiv-2305-08144/ · source: https://arxiv.org/abs/2305.08144v4 - Paper · reviewed 2026-10-16 · environments: android · topics: evaluation, sandbox - Published: 2023-05-14 · Authors: Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, Kai Yu ### Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Training Uses a staged curriculum from response-format alignment through action feedback to interactive task-level training for mobile agents. - https://awesomecua.com/resources/arxiv-2506-20332/ · source: https://arxiv.org/abs/2506.20332v4 - Paper · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, mobile-navigation - Published: 2025-06-25 · Authors: Jihao Gu, Qihang Ai, Yingyao Wang, Pi Bu, Jingxuan Xing, Zekun Zhu, Wei Jiang, Ziming Wang, Yingxiu Zhao, Ming-Liang Zhang, Jun Song, Yuning Jiang, Bo Zheng ### MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users Studies automatic annotation and federated training of mobile GUI agents from decentralized user interaction data with heterogeneous distributions. - https://awesomecua.com/resources/arxiv-2502-02982/ · source: https://arxiv.org/abs/2502.02982v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, data-generation - Published: 2025-02-05 · Authors: Wenhao Wang, Mengying Yuan, Zijie Yu, Guangyi Liu, Rui Ye, Tian Jin, Siheng Chen, Yanfeng Wang ### MobileAgent: enhancing mobile control via human-machine interaction and SOP integration Combines operating-procedure context with user interaction to guide mobile task automation and handle personalized or sensitive requests. - https://awesomecua.com/resources/arxiv-2401-04124/ · source: https://arxiv.org/abs/2401.04124v3 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, workflow, human-in-the-loop - Published: 2024-01-04 · Authors: Tinghe Ding ### MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents Introduces a mobile-agent benchmark with tasks across open-source applications, organized by difficulty to reduce manual evaluation effort. - https://awesomecua.com/resources/arxiv-2406-08184/ · source: https://arxiv.org/abs/2406.08184v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, mobile-navigation - Published: 2024-06-12 · Authors: Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, Shoufa Chen ### MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment Introduces MobileBench-OL, an online benchmark of Chinese-app tasks spanning execution, reasoning, exploration, and noisy conditions. Its automatic evaluation and reset framework supports repeated testing of mobile GUI agents in real environments. - https://awesomecua.com/resources/arxiv-2601-20335/ · source: https://arxiv.org/abs/2601.20335v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, robustness - Published: 2026-01-28 · Authors: Qinzhuo Wu, Zhizhuo Yang, Hanhao Li, Pengzhi Gao, Wei Liu, Jian Luan ### MobileDreamer: Generative Sketch World Model for GUI Agent MobileDreamer predicts post-action GUI states as task-relevant textual sketches with an order-invariant learning objective. Agents use these predictions for lookahead action selection, evaluated on AndroidWorld. - https://awesomecua.com/resources/arxiv-2601-04035/ · source: https://arxiv.org/abs/2601.04035v2 - Paper · reviewed 2026-10-17 · environments: android · topics: world-models, planning, mobile-navigation - Published: 2026-01-07 · Authors: Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Siran Dai, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, Wenji Mao ### MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices Introduces MobileExperts, which assembles mobile-agent teams, lets agents explore and formulate tools, and coordinates them through two levels of planning. A benchmark organizes tasks by hierarchical intelligence levels. - https://awesomecua.com/resources/arxiv-2407-03913/ · source: https://arxiv.org/abs/2407.03913v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, planning, tool-integration - Published: 2024-07-04 · Authors: Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, Jianping Fan ### MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Exploration MobileExplorer probes relevant UI elements while an on-device model is reasoning, then supplies the exploration trace as compact context. A two-level rollback mechanism restores the working state when simple backtracking fails. - https://awesomecua.com/resources/arxiv-2605-26546/ · source: https://arxiv.org/abs/2605.26546v1 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, efficiency, memory, robustness - Published: 2026-05-26 · Authors: Runxi Huang, Liyu Zhang, Shengzhong Liu, Xiaomin Ouyang ### MobileFlow: A Multimodal LLM For Mobile GUI Agent Introduces a mobile GUI model using hybrid visual encoders, mixture-of-experts expansion, and alignment training for multilingual interfaces. - https://awesomecua.com/resources/arxiv-2407-04346/ · source: https://arxiv.org/abs/2407.04346v3 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, screen-understanding - Published: 2024-07-05 · Authors: Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, Wenhao Xu ### MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization MobileForge couples real-app exploration and generated tasks with hierarchical feedback for mobile-agent adaptation. Its optimizer turns trajectory outcomes, step feedback and corrective hints into contextualized GRPO updates, with evaluation on AndroidWorld and a GUI-only MobileWorld split. - https://awesomecua.com/resources/arxiv-2606-19930/ · source: https://arxiv.org/abs/2606.19930v2 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, reinforcement-learning, data-generation, evaluation - Published: 2026-06-18 · Authors: Guangyi Liu, Pengxiang Zhao, Gao Wu, Yiwen Yin, Mading Li, Liang Liu, Congxiao Liu, Zhang Qi, Mengyan Wang, Liang Guo, Jiangning Zhang, Yong Liu ### MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment MobileGUI-RL constructs an online training curriculum through agent exploration and task filtering. It adapts GRPO with trajectory-aware advantages and rewards for task completion and execution efficiency, evaluated on three online mobile-agent benchmarks. - https://awesomecua.com/resources/arxiv-2507-05720/ · source: https://arxiv.org/abs/2507.05720v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, reinforcement-learning, data-generation, efficiency - Published: 2025-07-08 · Authors: Yucheng Shi, Wenhao Yu, Zaitang Li, Yonglin Wang, Hongming Zhang, Ninghao Liu, Haitao Mi, Dong Yu ### MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research Introduces MobileGym, a browser-hosted simulation of mobile app interactions whose configurable JSON state supports deterministic judging and parallel rollouts. MobileGym-Bench adds parameterized tasks, with experiments examining online training and transfer to real devices. - https://awesomecua.com/resources/arxiv-2605-26114/ · source: https://arxiv.org/abs/2605.26114v3 - Paper · reviewed 2026-10-17 · environments: web · topics: sandbox, evaluation, reinforcement-learning - Published: 2026-05-25 · Authors: Dingbang Wu, Rui Hao, Haiyang Wang, Shuzhe Wu, Han Xiao, Zhenghong Li, Bojiang Zhou, Zheng Ju, Zichen Liu, Lue Fan, Zhaoxiang Zhang ### MobileInject: Evaluating Reasoning-to-Action Security of MLLM-Based Mobile Agents under Cross-Modal Consistency Injection MobileInject evaluates visual/textual observation injection in stateful Android workflows using resettable emulators, benign/adversarial state oracles and trace-based diagnosis of emitted rationales and actions. - https://awesomecua.com/resources/arxiv-2610-23307/ · source: https://arxiv.org/abs/2610.23307v1 - Paper · reviewed 2026-10-20 · environments: android · topics: safety-security, mobile-navigation, evaluation, observability, cross-app - Published: 2026-10-16 · Authors: Songping Wang, Peiyang Li, Yi Huang, Fengting Li, Xuhao Ran, Yueming Lyu, Chao Feng, Caifeng Shan ### MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning MobileIPL samples trees of action-planning thoughts, propagates rule-based leaf rewards backward, and forms preference pairs for reasoning-level optimization. It also expands screenshot-based instruction data through staged evolution before evaluation on mobile GUI-agent benchmarks. - https://awesomecua.com/resources/arxiv-2505-12299/ · source: https://arxiv.org/abs/2505.12299v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, reinforcement-learning, planning, data-generation - Published: 2025-05-18 · Authors: Kun Huang, Weikai Xu, Yuxuan Liu, Quandong Wang, Pengzhi Gao, Wei Liu, Jian Luan, Bin Wang, Bo An ### MobileJailBench: Red Queen Jailbreak Benchmarking for Mobile Agents MobileJailBench evaluates mobile-agent jailbreaks in a browser-based Android-like simulator, combining feedback-driven attack evolution, separately admitted task expansions and evidence-grounded outcome adjudication. - https://awesomecua.com/resources/arxiv-2610-23303/ · source: https://arxiv.org/abs/2610.23303v1 - Paper · reviewed 2026-10-20 · environments: web · topics: safety-security, mobile-navigation, evaluation, observability, cross-app - Published: 2026-10-16 · Authors: Songping Wang, Borui Li, Qinglong Liu, Yueming Lyu, Yi Huang, Fengting Li, Chao Feng, Caifeng Shan ### MobileRAG: Enhancing Mobile Agent with Retrieval-Augmented Generation MobileRAG combines retrieval and memory components to support mobile tasks that need external knowledge or long action sequences. MobileRAG-Eval is introduced to study these requirements on complex real-world app tasks. - https://awesomecua.com/resources/arxiv-2509-03891/ · source: https://arxiv.org/abs/2509.03891v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, mobile-navigation, evaluation, long-horizon - Published: 2025-09-04 · Authors: Gowen Loo, Chang Liu, Qinghong Yin, Xiang Chen, Jiawei Chen, Jingyuan Zhang, Yu Tian ### MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents MobileRL adapts GRPO to mobile GUI training with difficulty-dependent replay of successes and filtering of failure curricula. A shortest-path reward adjustment accounts for task length, with evaluations on AndroidWorld and AndroidLab. - https://awesomecua.com/resources/arxiv-2509-18119/ · source: https://arxiv.org/abs/2509.18119v2 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, mobile-navigation, efficiency - Published: 2025-09-10 · Authors: Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Wenyi Zhao, Yuxiao Dong ### MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control Evaluates mobile agents on misuse, side effects, and indirect prompt-injection risks in an Android emulator environment. - https://awesomecua.com/resources/arxiv-2410-17520/ · source: https://arxiv.org/abs/2410.17520v3 - Paper · reviewed 2026-10-16 · environments: android · topics: safety-security, evaluation - Published: 2024-10-23 · Authors: Juyong Lee, Dongyoon Hahm, June Suk Choi, W. Bradley Knox, Kimin Lee ### MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions Introduces MobileSteward, which coordinates app-specific phone agents, schedules cross-app tasks, and reuses successful experience; CAPBench evaluates cross-app instructions. - https://awesomecua.com/resources/arxiv-2502-16796/ · source: https://arxiv.org/abs/2502.16796v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: cross-app, planning, memory, mobile-navigation - Published: 2025-02-24 · Authors: Yuxuan Liu, Hongda Sun, Wei Liu, Jian Luan, Bo Du, Rui Yan ### MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation MobileUse reflects on errors at action and task timescales, invoking reflection when needed and using proactive exploration to learn unfamiliar apps. It evaluates this mobile execution framework on AndroidWorld and AndroidLab and describes a toolkit for physical-device operation. - https://awesomecua.com/resources/arxiv-2507-16853/ · source: https://arxiv.org/abs/2507.16853v1 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, self-reflection, robustness, long-horizon - Published: 2025-07-21 · Authors: Ning Li, Xiangmou Qu, Jiamu Zhou, Jun Wang, Muning Wen, Kounianhua Du, Xingyu Lou, Qiuying Peng, Jun Wang, Weinan Zhang ### MobileViews: A Million-scale and Diverse Mobile GUI Dataset Builds mobile screenshot and view-hierarchy pairs through automated application traversal and evaluates their usefulness for GUI grounding. - https://awesomecua.com/resources/arxiv-2409-14337/ · source: https://arxiv.org/abs/2409.14337v3 - Paper · reviewed 2026-10-16 · environments: android · topics: screen-understanding, data-generation - Published: 2024-09-22 · Authors: Longxi Gao, Li Zhang, Shihe Wang, Pengzhi Gao, Wei Liu, Jian Luan, Shangguang Wang, Yuanchun Li, Mengwei Xu ### MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding MobileVLM adds two pretraining stages and four UI tasks to learn element detail and relationships between successive screens. Its Mobile3M dataset contains three million Chinese mobile UI pages connected by recorded transition actions. - https://awesomecua.com/resources/arxiv-2409-14818/ · source: https://arxiv.org/abs/2409.14818v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, data-generation, action-prediction - Published: 2024-09-23 · Authors: Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Shuo Shang ### MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments Introduces long mobile workflows that combine agent-user interaction, cross-application tasks, and MCP-based tool access. - https://awesomecua.com/resources/arxiv-2512-19432/ · source: https://arxiv.org/abs/2512.19432v3 - Paper · reviewed 2026-10-16 · environments: android · topics: evaluation, tool-integration, long-horizon - Published: 2025-12-22 · Authors: Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, Yue Wang ### MobileWorldBench: Towards Semantic World Modeling For Mobile Agents Evaluates semantic world models for mobile interfaces and studies natural-language state transitions as a tool for agent planning. - https://awesomecua.com/resources/arxiv-2512-14014/ · source: https://arxiv.org/abs/2512.14014v2 - Paper · reviewed 2026-10-16 · environments: android · topics: world-models, planning, evaluation - Published: 2025-12-16 · Authors: Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Aditya Grover ### MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps Evaluates environmental injection risks in Android tasks using programmatic risk checks and model-based adjudication of ambiguous cases. - https://awesomecua.com/resources/arxiv-2608-17659/ · source: https://arxiv.org/abs/2608.17659v1 - Paper · reviewed 2026-10-16 · environments: android · topics: safety-security, evaluation - Published: 2026-08-18 · Authors: Sujin Chen, Lijun Li, Tianyi Du, Jing Shao ### Modeling Distinct Human Interaction in Web Agents CowCorpus records interleaved human and agent actions during web navigation and identifies distinct patterns of user intervention. Models trained to anticipate these interventions are integrated into live web agents and assessed in a user study. - https://awesomecua.com/resources/arxiv-2602-17588/ · source: https://arxiv.org/abs/2602.17588v4 - Paper · reviewed 2026-10-17 · environments: web · topics: human-in-the-loop, web-navigation, data-generation, action-prediction - Published: 2026-02-19 · Authors: Faria Huq, Zora Zhiruo Wang, Zhanqiu Guo, Venu Arvind Arangarajan, Tianyue Ou, Frank Xu, Shuyan Zhou, Graham Neubig, Jeffrey P. Bigham ### MolmoPoint: Better Pointing for VLMs with Grounding Tokens MolmoPoint replaces textual coordinate generation with pointing tokens that select visual regions and refine them through subpatches. The general image/video method includes GUI-pointing evaluation on ScreenSpotPro alongside other visual grounding tasks. - https://awesomecua.com/resources/arxiv-2603-28069/ · source: https://arxiv.org/abs/2603.28069v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, video, efficiency - Published: 2026-03-30 · Authors: Christopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, Taira Anderson, Winson Han, Ranjay Krishna ### MolmoWeb: Open Visual Web Agent and Open Data for the Open Web Presents MolmoWebMix, combining synthetic browser trajectories, human demonstrations and GUI perception data, and MolmoWeb screenshot-to-action policies trained from it. Browser benchmarks also test parallel rollouts with best-of-N selection. - https://awesomecua.com/resources/arxiv-2604-08516/ · source: https://arxiv.org/abs/2604.08516v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, data-generation, imitation-learning - Published: 2026-04-09 · Authors: Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Taylor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, Ranjay Krishna ### Monitoring Web Agents Without Internal Signals: Observable Trajectories and Key-Step Supervision Predicts web-agent failure risk from observable trajectory prefixes using cross-step behavior and black-box consistency queries. Key-step labels preserve valid early prefixes of failed runs, and evaluation tests early intervention and transfer across website categories. - https://awesomecua.com/resources/arxiv-2609-02057/ · source: https://arxiv.org/abs/2609.02057v1 - Paper · reviewed 2026-10-18 · environments: web · topics: observability, evaluation, robustness, interaction-history - Published: 2026-09-02 · Authors: Sitong Pan, Yipeng Shen, Yilin Lu, Caiwen Ding, Lu Cheng, Qianwen Wang ### Morae: Proactively Pausing UI Agents for User Choices Introduces Morae, a UI agent that detects decision points and pauses for user choices using interface code and screenshots. A study with blind and low-vision participants evaluates this mixed-initiative approach on real web tasks. - https://awesomecua.com/resources/arxiv-2508-21456/ · source: https://arxiv.org/abs/2508.21456v1 - Paper · reviewed 2026-10-17 · environments: web · topics: human-in-the-loop, web-navigation, evaluation - Published: 2025-08-29 · Authors: Yi-Hao Peng, Dingzeyu Li, Jeffrey P. Bigham, Amy Pavel ### MP-GUI: Modality Perception with MLLMs for GUI Understanding Uses specialized graphical, textual, and spatial perception components with a data-collection pipeline to improve GUI understanding. - https://awesomecua.com/resources/arxiv-2503-14021/ · source: https://arxiv.org/abs/2503.14021v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: screen-understanding, grounding, data-generation - Published: 2025-03-18 · Authors: Ziwei Wang, Weizhi Chen, Leyang Yang, Sheng Zhou, Shengchu Zhao, Hanbei Zhan, Jiongchao Jin, Liangcheng Li, Zirui Shao, Jiajun Bu ### MPIDetector: Detecting Multi-modal Prompt Injection via LLM-based Reasoning MPIDetector uses few-shot prompts to detect text and visual prompt injections, with environmental web-agent injections included in its evaluation. - https://awesomecua.com/resources/arxiv-2610-30202/ · source: https://arxiv.org/abs/2610.30202v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, training-free - Published: 2026-10-21 · Authors: Yiheng Li, Jingyue Li ### MPO: Boosting LLM Agents with Meta Plan Optimization Refines reusable high-level meta plans with execution feedback to guide language-agent planning across interactive environments. - https://awesomecua.com/resources/arxiv-2503-02682/ · source: https://arxiv.org/abs/2503.02682v2 - Paper · reviewed 2026-10-16 · environments: web · topics: planning, memory - Published: 2025-03-04 · Authors: Weimin Xiong, Yifan Song, Qingxiu Dong, Bingchan Zhao, Feifan Song, Xun Wang, Sujian Li ### MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents MPR-GUI evaluates GUI perception and reasoning through strictly aligned tasks in six languages. GUI-XLI adjusts non-English hidden representations toward English representations at language-sensitive layers during inference to address the measured cross-language gaps. - https://awesomecua.com/resources/arxiv-2512-00756/ · source: https://arxiv.org/abs/2512.00756v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, screen-understanding, robustness - Published: 2025-11-30 · Authors: Ruihan Chen, Qiming Li, Xiaocheng Feng, Weihong Zhong, Xiaoliang Yang, Yuxuan Gu, Zekun Zhou, Yunfei Lu, Haoyu Ren, Kun Chen, Dandan Tu, Bing Qin ### MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling MUD combines LLM-guided Android app exploration, noise filtering and human annotation to collect modern UI screenshots. The study evaluates the resulting data for element detection and UI retrieval. - https://awesomecua.com/resources/arxiv-2405-07090/ · source: https://arxiv.org/abs/2405.07090v1 - Paper · reviewed 2026-10-18 · environments: android · topics: data-generation, screen-understanding - Published: 2024-05-11 · Authors: Sidong Feng, Suyu Ma, Han Wang, David Kong, Chunyang Chen ### MUG: Interactive Multimodal Grounding on User Interfaces Studies iterative UI grounding in which users refine or correct an agent's interpretation over multiple conversational turns. - https://awesomecua.com/resources/arxiv-2209-15099/ · source: https://arxiv.org/abs/2209.15099v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: grounding, human-in-the-loop - Published: 2022-09-29 · Authors: Tao Li, Gang Li, Jingjie Zheng, Purple Wang, Yang Li ### MUIAnno: An Expert-Annotated Dataset and Evaluation Benchmark for Mobile UI Understanding MUIAnno provides expert labels for elements in manually explored mobile-app screens, with a web annotation tool that emits structured JSON. It also supplies UI-element detection baselines for the dataset. - https://awesomecua.com/resources/arxiv-2605-17656/ · source: https://arxiv.org/abs/2605.17656v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding, evaluation, data-generation - Published: 2026-05-17 · Authors: Athar Parvez, Muhammad Jawad Mufti, Muqaddas Gull, Omar Hammad ### Multi-Agent Computer Use MACU represents computer-use objectives as a dependency graph whose manager dispatches parallel subagents and revises tasks as new information arrives. Desktop and web experiments examine coordination, information handoff and task-completion time. - https://awesomecua.com/resources/arxiv-2606-01533/ · source: https://arxiv.org/abs/2606.01533v1 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: planning, long-horizon, workflow, state-tracking, efficiency - Published: 2026-06-01 · Authors: Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried ### Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making Introduces Multi-Head Latent Control, a layer that reads frozen LLM or VLM hidden-state trajectories to signal when an agent should defer, seek clarification, use tools, abstain, or answer. - https://awesomecua.com/resources/arxiv-2607-14277/ · source: https://arxiv.org/abs/2607.14277v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, tool-integration - Published: 2026-07-15 · Authors: Amirhosein Ghasemabadi, Ruichen Chen, Bahador Rashidi, Di Niu ### Multimodal Auto Validation For Self-Refinement in Web Agents Studies text and visual validation for web agents and uses the resulting validator to detect failures and guide workflow self-refinement. - https://awesomecua.com/resources/arxiv-2410-00689/ · source: https://arxiv.org/abs/2410.00689v2 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, robustness, web-navigation - Published: 2024-10-01 · Authors: Ruhana Azam, Tamer Abuelsaad, Aditya Vempaty, Ashish Jagmohan ### Multimodal Web Navigation with Instruction-Finetuned Foundation Models Trains WebGUM offline from demonstrations by jointly tuning an instruction-following language model and a vision encoder over webpage screenshots and HTML. The agent predicts browser actions and is evaluated on MiniWoB, WebShop, and Mind2Web. - https://awesomecua.com/resources/arxiv-2305-11854/ · source: https://arxiv.org/abs/2305.11854v4 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, imitation-learning, action-prediction - Published: 2023-05-19 · Authors: Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, Izzeddin Gur ### MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks MUZZLE evaluates web-agent exposure to indirect prompt injection by finding attack surfaces in agent trajectories and refining injected instructions from execution feedback. - https://awesomecua.com/resources/arxiv-2602-09222/ · source: https://arxiv.org/abs/2602.09222v2 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, interaction-history, web-navigation - Published: 2026-02-09 · Authors: Georgios Syros, Evan Rose, Brian Grinstead, Christoph Kerschbaumer, William Robertson, Cristina Nita-Rotaru, Alina Oprea ### MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions MVISU-Bench evaluates bilingual mobile tasks involving cross-app work, vague requests, user interaction and unethical instructions alongside ordinary single-app tasks. Its Aider prompting module is designed to clarify intent and mitigate risky execution within agent workflows. - https://awesomecua.com/resources/arxiv-2508-09057/ · source: https://arxiv.org/abs/2508.09057v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, human-in-the-loop, safety-security - Published: 2025-08-12 · Authors: Zeyu Huang, Juyuan Wang, Longfeng Chen, Boyi Xiao, Leng Cai, Yawen Zeng, Jin Xu ### MVP: Multiple View Prediction Improves GUI Grounding MVP addresses unstable GUI coordinates by predicting across several attention-guided cropped views. It clusters the resulting coordinates and selects the densest cluster centroid, evaluating this training-free inference method across grounding models. - https://awesomecua.com/resources/arxiv-2512-08529/ · source: https://arxiv.org/abs/2512.08529v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, robustness - Published: 2025-12-09 · Authors: Yunzhu Zhang, Zeyu Pan, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Linchao Zhu ### MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents MyPCBench tests personal-assistant tasks on a Linux desktop with simulated web applications and data for a single fictional persona. A shared computer-and-shell interface enables evaluation of cross-application workflows that depend on personal context and account-like state. - https://awesomecua.com/resources/arxiv-2606-16748/ · source: https://arxiv.org/abs/2606.16748v1 - Paper · reviewed 2026-10-17 · environments: desktop, linux, web · topics: evaluation, cross-app, workflow - Published: 2026-06-15 · Authors: Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov ### Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents This study finds that full-image visual memory can reduce some state errors while worsening action-level failures. Its AGMem alternative stores local crops associated with successful actions or recovery and evaluates their use in desktop GUI tasks. - https://awesomecua.com/resources/arxiv-2606-14106/ · source: https://arxiv.org/abs/2606.14106v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: memory, interaction-history, grounding, robustness, evaluation - Published: 2026-06-12 · Authors: Seoyoung Choi, Minseok Ko, Hyunseok Lee, Kunwoong Kim, Woomin Song, Chanseok Jeon, Jinwoo Shin ### nanoMuse: An Open-Source Personal Agent for Every Device You Own Presents nanoMuse, an open-source personal agent designed to act on phone and computer screens, with a shared conversation, action-checking Sentinel, and readable file memory. - https://awesomecua.com/resources/arxiv-2610-08699/ · source: https://arxiv.org/abs/2610.08699v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: cross-app, memory, safety-security - Published: 2026-10-06 · Authors: Guangyi Liu, Yong Liu, Jiangning Zhang ### NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks NaturalGAIA builds verifiable long-horizon GUI tasks from human interaction intents while separating causal task structure from its verbal description. Its LightManus-Jarvis framework divides dynamic planning and context management from execution based on visual and structural interface information. - https://awesomecua.com/resources/arxiv-2508-01330/ · source: https://arxiv.org/abs/2508.01330v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, long-horizon, planning - Published: 2025-08-02 · Authors: Zihan Zheng, Tianle Cui, Taoran Wang, Fengtao Wang, Jiahui Pan, Lewei He, Qianglong Chen ### Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questions Navi-plus evaluates GUI agents that ask follow-up questions to complete ambiguous user instructions. It provides GUI clarification question-answer pairs and Dual-Stream Trajectory Evaluation for this interactive setting. - https://awesomecua.com/resources/arxiv-2503-24180/ · source: https://arxiv.org/abs/2503.24180v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, evaluation, planning - Published: 2025-03-31 · Authors: Ziming Cheng, Zhiyuan Huang, Junting Pan, Zhaohui Hou, Mingjie Zhan ### Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents Introduces UGround, a visual target-localization model trained with large-scale synthetic references to elements in GUI screenshots. - https://awesomecua.com/resources/arxiv-2410-05243/ · source: https://arxiv.org/abs/2410.05243v3 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, data-generation - Published: 2024-10-07 · Authors: Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, Yu Su ### Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning Combines supervised and reinforcement learning for MiniWoB web tasks and investigates whether agents understand HTML structure or memorize target elements. - https://awesomecua.com/resources/arxiv-2405-00516/ · source: https://arxiv.org/abs/2405.00516v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, reinforcement-learning, evaluation - Published: 2024-05-01 · Authors: Lucas-Andreï Thil, Mirela Popa, Gerasimos Spanakis ### Navigation Alone Is Not Enough: Evaluating Explanatory Assistive UI Agents Introduces NeXUI, an assistive UI-agent benchmark that evaluates navigation together with interface-grounded explanations and user oversight in recorded interface states. - https://awesomecua.com/resources/arxiv-2608-09944/ · source: https://arxiv.org/abs/2608.09944v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, human-in-the-loop, web-navigation - Published: 2026-07-01 · Authors: Santosh Patapati ### NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks NaviMaster trains a shared policy for GUI and embodied navigation using a common visual-trajectory formulation. It combines trajectory collection, reinforcement learning and distance-aware rewards, with evaluation across both task families. - https://awesomecua.com/resources/arxiv-2508-02046/ · source: https://arxiv.org/abs/2508.02046v4 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: reinforcement-learning, data-generation, robustness - Published: 2025-08-04 · Authors: Zhihao Luo, Wentao Yan, Jingyu Gong, Min Wang, Zhizhong Zhang, Xuhong Wang, Yuan Xie, Xin Tan ### NaviQAte: Functionality-Guided Web Application Navigation Frames web-application exploration as functionality-oriented question answering, generating navigation actions from multimodal context. - https://awesomecua.com/resources/arxiv-2409-10741/ · source: https://arxiv.org/abs/2409.10741v1 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, automation - Published: 2024-09-16 · Authors: Mobina Shahbandeh, Parsa Alian, Noor Nashid, Ali Mesbah ### Nested Browser-Use Learning for Agentic Information Seeking NestBrowse organizes browser actions in a nested framework that separates interaction control from page exploration. It evaluates this design for deep information seeking beyond snippet retrieval and direct URL fetching. - https://awesomecua.com/resources/arxiv-2512-23647/ · source: https://arxiv.org/abs/2512.23647v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, planning, efficiency, tool-integration - Published: 2025-12-29 · Authors: Baixuan Li, Jialong Wu, Wenbiao Yin, Kuan Li, Zhongwang Zhang, Huifeng Yin, Zhengwei Tao, Liwen Zhang, Pengjun Xie, Jingren Zhou, Yong Jiang ### Neuro-Symbolic Computer Use: Learning Reusable Policies for Reliable and Efficient Execution Learns reusable computer-use policies that encode stable workflow logic in programs while delegating grounding and state checks to neural models. Policy iteration executes variants, diagnoses failures, revises the program, and adds verification before state-changing actions. - https://awesomecua.com/resources/arxiv-2609-36927/ · source: https://arxiv.org/abs/2609.36927v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: automation, workflow, efficiency - Published: 2026-09-29 · Authors: Hyewon Suh, Thanh Minh Nguyen, Chih-Lun Lee, Darrow Hartman, Lizhao Liu, Xin Eric Wang, Ang Li, Jiachen Yang ### Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense Next-Gen CAPTCHAs introduces generated interactive challenges intended to distinguish human users from GUI agents. The benchmark targets perception, memory, decisions and actions, with backend-supported challenge types designed for scalable instance generation. - https://awesomecua.com/resources/arxiv-2602-09012/ · source: https://arxiv.org/abs/2602.09012v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, safety-security, data-generation - Published: 2026-02-09 · Authors: Jiacheng Liu, Yaxin Luo, Jiacheng Cui, Xinyi Shang, Xiaohan Zhao, Zhiqiang Shen ### NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild Generates browser-agent demonstrations through autonomous exploration and retrospective task labeling, pruning trajectories that lack meaningful subtasks. - https://awesomecua.com/resources/arxiv-2410-02907/ · source: https://arxiv.org/abs/2410.02907v2 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation - Published: 2024-10-03 · Authors: Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, Christopher D. Manning ### Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents Studies how to route different parts of GUI experience into model weights or runtime context rather than treating whole trajectories as a single unit. It separates locators, procedures, state facts, and lessons, then predicts their destination from recurrence and dependence on the current state. - https://awesomecua.com/resources/arxiv-2610-01787/ · source: https://arxiv.org/abs/2610.01787v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, evaluation, efficiency - Published: 2026-10-01 · Authors: Beining Wu, Zihao Ding, Jun Huang ### Not All Graphs Are Maps: A Task-Agnostic Exploration Framework for Mobile GUI Navigation TAG-Nav supplies Android GUI agents with destination and route memory distilled from offline app exploration. DestBench measures arrival at a reference pre-action screen, separately from downstream task execution. - https://awesomecua.com/resources/arxiv-2610-22672/ · source: https://arxiv.org/abs/2610.22672v1 - Paper · reviewed 2026-10-20 · environments: android · topics: mobile-navigation, memory, evaluation - Published: 2026-10-15 · Authors: Adamo Orsini, Seyed Mehdi Ayyoubzadeh, Sadman Rakib Pinon, Hannuo Zhang, Koosha Gholipour Baradari, Yuanhao Yu, Xinxin Zuo, Yang Wang ### Not an A11y: How Android Accessibility Exposes Mobile AI Agents to Indirect Prompt Injection Demonstrates that unsanitized Android accessibility tree metadata exposes mobile GUI agents to stealthy indirect prompt injection attacks. - https://awesomecua.com/resources/arxiv-2608-08939/ · source: https://arxiv.org/abs/2608.08939v1 - Paper · reviewed 2026-10-18 · environments: android · topics: safety-security, mobile-navigation, screen-understanding - Published: 2026-08-09 · Authors: Rahul Deivasigamani, Sayeda Faatin Alvi, Derqui Andrea, Kaushal Punjabi, Stjepan Picek ### Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? Compares seven traditional and learned GUI element detectors on more than 50,000 interface images. The analysis motivates a coarse-to-fine non-text detector combined with a learned text detector to address GUI-specific localization requirements. - https://awesomecua.com/resources/arxiv-2008-05132/ · source: https://arxiv.org/abs/2008.05132v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, evaluation - Published: 2020-08-12 · Authors: Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li ### OctoNest: Adaptive Cross-Device Execution through Stateful Control Introduces OctoNest, which coordinates device agents, revises task assignments from execution feedback, and switches local interaction modalities; CAPEBench evaluates cross-device tasks. - https://awesomecua.com/resources/arxiv-2606-20487/ · source: https://arxiv.org/abs/2606.20487v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: cross-app, planning, state-tracking, evaluation - Published: 2026-06-18 · Authors: Shu Yao, Yuhua Luo, Qian Long, Jingru Fan, Yuheng Wang, Lin Wu, Zhuoyuan Yu, Yufan Dang, Huatao Li, Chen Qian ### OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows Introduces OdysseyBench for evaluating long-horizon workflows across office applications, plus HomerAgents to generate tasks from environment exploration and dialogue synthesis. - https://awesomecua.com/resources/arxiv-2508-09124/ · source: https://arxiv.org/abs/2508.09124v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon, cross-app, interaction-history - Published: 2025-08-12 · Authors: Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, Saravan Rajmohan ### Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Introduces Odysseys, a live-web benchmark of long, multi-site tasks derived from browsing sessions. Graded task rubrics and a rubric-score-per-step measure evaluate both partial progress and trajectory efficiency. - https://awesomecua.com/resources/arxiv-2604-24964/ · source: https://arxiv.org/abs/2604.24964v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, long-horizon, efficiency - Published: 2026-04-27 · Authors: Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried, Ruslan Salakhutdinov ### OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation Evaluates language agents on office workflows requiring information integration, long-term planning, and switching between applications. - https://awesomecua.com/resources/arxiv-2407-19056/ · source: https://arxiv.org/abs/2407.19056v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: evaluation, cross-app, long-horizon - Published: 2024-07-26 · Authors: Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, Jingbo Shang ### Okara: Detection and Attribution of TLS Man-in-the-Middle Vulnerabilities in Android Apps with Foundation Models Okara uses foundation-model-driven GUI agents to explore Android apps for TLS vulnerability detection, then applies instrumentation and classification to attribute vulnerable code. - https://awesomecua.com/resources/arxiv-2601-22770/ · source: https://arxiv.org/abs/2601.22770v2 - Paper · reviewed 2026-11-06 · environments: android · topics: automation, mobile-navigation, safety-security - Published: 2026-01-30 · Authors: Haoyun Yang, Ronghong Huang, Yong Fang, Beizeng Zhang, Junpu Guo, Zhanyu Wu, Xianghang Mi ### OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding Introduces OmegaUse-OfficeVal, a benchmark for long-horizon office-suite agent tasks with labor-time and price signals, plus rubric-based code verifiers. - https://awesomecua.com/resources/arxiv-2607-27155/ · source: https://arxiv.org/abs/2607.27155v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, workflow, industry, long-horizon - Published: 2026-07-29 · Authors: Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu ### OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations OmegaUse-SOP converts expert GUI demonstrations into configurable procedures and refines them with domain rules and human input. It records, abstracts, grounds and verifies live execution, with a case study of photovoltaic simulation workflows in PVsyst 7.2. - https://awesomecua.com/resources/arxiv-2609-02149/ · source: https://arxiv.org/abs/2609.02149v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, recording, workflow, automation - Published: 2026-09-02 · Authors: Yixiong Xiao, Lang An, Hucheng Yang, Pinxue Ma, Yongquan Chen, Jingjia Cao, Yusai Zhao, Ting Wang, Ting Liu, Siqi Bao, Jingbo Zhou, Hua Wu ### OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution OmegaUse combines exploratory and taxonomy-guided data synthesis with supervised fine-tuning followed by GRPO for GUI control. The report introduces OS-Nav for offline Android and Ubuntu evaluation alongside established grounding and action-prediction benchmarks. - https://awesomecua.com/resources/arxiv-2601-20380/ · source: https://arxiv.org/abs/2601.20380v1 - Paper · reviewed 2026-10-17 · environments: android, desktop, linux · topics: data-generation, reinforcement-learning, grounding, evaluation - Published: 2026-01-28 · Authors: Le Zhang, Yixiong Xiao, Xinjiang Lu, Jingjia Cao, Yusai Zhao, Jingbo Zhou, Lang An, Zikan Feng, Wanxiang Sha, Yu Shi, Congxi Xiao, Jian Xiong, Yankai Zhang, Hua Wu, Haifeng Wang ### OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web Introduces a dataset and benchmark for generating executable task scripts from screenshots and visually grounded desktop or web instructions. - https://awesomecua.com/resources/arxiv-2402-17553/ · source: https://arxiv.org/abs/2402.17553v3 - Paper · reviewed 2026-10-16 · environments: desktop, web · topics: evaluation, grounding - Published: 2024-02-27 · Authors: Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, Ruslan Salakhutdinov ### OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds Introduces OmniActor, which shares shallow model layers while separating deeper parameters to train jointly on GUI and embodied interactions. A unified action representation and mixed training data support evaluation in both task families. - https://awesomecua.com/resources/arxiv-2509-02322/ · source: https://arxiv.org/abs/2509.02322v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: action-prediction, data-generation, imitation-learning - Published: 2025-09-02 · Authors: Longrong Yang, Zhixiong Zeng, Yufeng Zhong, Jing Huang, Liming Zheng, Lei Chen, Haibo Qiu, Zequn Qin, Lin Ma, Xi Li ### OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments OmniGUI evaluates step-level action prediction from interleaved smartphone screenshots, synchronous audio and video. Expert demonstrations carry multimodal-dependency annotations, and initial baselines use native multimodal models as agent proxies to study temporal and audio-related failures. - https://awesomecua.com/resources/arxiv-2605-18758/ · source: https://arxiv.org/abs/2605.18758v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: evaluation, action-prediction, video, mobile-navigation - Published: 2026-04-03 · Authors: Felix Henry, Lihan Lin, Jiangyou Zhu, Yangfan, Bingqian Zhang, Min Chen, Shiyu Huang ### OmniParser for Pure Vision Based GUI Agent Parses screenshots into interactable regions and semantic element descriptions to support visually grounded agent actions. - https://awesomecua.com/resources/arxiv-2408-00203/ · source: https://arxiv.org/abs/2408.00203v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, screen-understanding - Published: 2024-08-01 · Authors: Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah ### OmniStepBench: Deployment-Aware Agent Evaluation across Modalities and Task Complexity Introduces OmniStepBench, a shared-environment benchmark for evaluating multimodal agents on multi-step tasks, tool use, and delivered artifacts under one scoring protocol. - https://awesomecua.com/resources/arxiv-2609-06059/ · source: https://arxiv.org/abs/2609.06059v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, workflow, tool-integration - Published: 2026-09-05 · Authors: Yu Liu, Zhilin Liu, Zhiwei Yang, Shaojie Zhang, Zheyuan Deng, Tingwei Huang, Zhenbo Luo, Lei Jiang, Yanbing Liu, Pei Fu ### On the Effects of Data Scale on UI Control Agents Studies how Android task performance changes with demonstration scale and releases AndroidControl with high- and low-level instructions. - https://awesomecua.com/resources/arxiv-2406-03679/ · source: https://arxiv.org/abs/2406.03679v6 - Paper · reviewed 2026-10-16 · environments: android · topics: data-generation, mobile-navigation, evaluation - Published: 2024-06-06 · Authors: Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, Oriana Riva ### On the Internet, Nobody Knows You're an LLM Bot: Unmasking Web Agents with Multi-Layer Fingerprinting Evaluates web agents against anti-bot mechanisms on instrumented honeysites and analyzes network, HTTP and browser fingerprints. The study compares resistance to blocking with the ability to distinguish agents, humans and agent implementations. - https://awesomecua.com/resources/arxiv-2606-30119/ · source: https://arxiv.org/abs/2606.30119v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, observability, evaluation - Published: 2026-06-29 · Authors: Iliana Fayolle, Sihem Bouhenniche, Samuel Pélissier, Pierre Laperdrix, Clémentine Maurice, Walter Rudametkin ### On the Multi-turn Instruction Following for Conversational Web Agents MT-Mind2Web evaluates web navigation through sequences of conversational user instructions. Self-MAP combines memory-augmented planning with reflection to preserve context and dependencies across those interaction turns. - https://awesomecua.com/resources/arxiv-2402-15057/ · source: https://arxiv.org/abs/2402.15057v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, memory, self-reflection, web-navigation - Published: 2024-02-23 · Authors: Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See-Kiong Ng, Tat-Seng Chua ### On the Reliability of Computer Use Agents This study examines repeated OSWorld executions to separate execution stochasticity, task-specification ambiguity and behavioral variation. Paired task-level tests show why one successful run is insufficient evidence of reliable computer use. - https://awesomecua.com/resources/arxiv-2604-17849/ · source: https://arxiv.org/abs/2604.17849v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, robustness - Published: 2026-04-20 · Authors: Gonzalo Gonzalez-Pumariega, Saaket Agashe, Jiachen Yang, Ang Li, Xin Eric Wang ### On the Robustness of GUI Grounding Models Against Image Attacks Evaluates GUI grounding models under natural image noise, untargeted adversarial perturbations, and targeted image attacks. Experiments cover mobile, desktop, and web interfaces and analyze sensitivity to image resolution and adversarial changes. - https://awesomecua.com/resources/arxiv-2504-04716/ · source: https://arxiv.org/abs/2504.04716v1 - Paper · reviewed 2026-10-17 · environments: web, desktop, cross-platform · topics: grounding, robustness, safety-security, evaluation - Published: 2025-04-07 · Authors: Haoren Zhao, Tianyi Chen, Zhen Wang ### One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding InnerZoom preserves target-region cues from intermediate decoder layers and reinjects a compact evidence state during coordinate decoding. Its single-forward GUI grounder is compared with crop-and-rerun zooming and matched supervised/RL baselines for accuracy and inference cost. - https://awesomecua.com/resources/arxiv-2606-30084/ · source: https://arxiv.org/abs/2606.30084v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, efficiency, screen-understanding, evaluation - Published: 2026-06-29 · Authors: Chen Liu, Ling Chen, Hanzhang Zhou, Liangyu Chen, Chenglin Cai, Xin Yu, Steven Hoi, Yue Wang ### One Rollout Is All You Get: Fully Test-Time Adaptation for GUI Agents Defines a test-time adaptation setting with one counted attempt per arriving GUI task and no ground-truth feedback. Its SOLO method filters or relabels completed episodes, then updates a small adapter from a short window of the agent’s own experience. - https://awesomecua.com/resources/arxiv-2609-34321/ · source: https://arxiv.org/abs/2609.34321v1 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, evaluation, robustness - Published: 2026-09-28 · Authors: Ziqiang Wang, Li Gu, Zhixiang Chi, Linlian Jiang, Zihuan Jiang, Linqiang Guo, Siobhan Reid, Zhi Liu, Yang Wang ### Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps Introduces online RL training for agents navigating live iOS apps, separating planning from UI grounding and using step-level and task-completion rewards. - https://awesomecua.com/resources/arxiv-2610-39264/ · source: https://arxiv.org/abs/2610.39264v1 - Paper · reviewed 2026-11-06 · environments: ios · topics: reinforcement-learning, planning, grounding, reward-models - Published: 2026-10-29 · Authors: Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio, Emmanuel Abbe ### Online Skill Learning for Web Agents via State-Grounded Dynamic Retrieval State-Grounded Dynamic Retrieval extracts reusable sub-procedures from completed web trajectories and represents them as paired text and code. Skills are retrieved against both the current webpage state and the task goal at intermediate execution steps. - https://awesomecua.com/resources/arxiv-2606-04391/ · source: https://arxiv.org/abs/2606.04391v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, memory, interaction-history, planning, workflow - Published: 2026-06-03 · Authors: Jiaxi Li, Ke Deng, Yun Wang, Jingyuan Huang, Yucheng Shi, Qiaoyu Tan, Jin Lu, Ninghao Liu ### OpAgent: Operator Agent for Web Navigation OpAgent combines multi-task GUI fine-tuning with online web reinforcement learning using outcome judgments and rule-based progress rewards. A modular planner, grounder, reflector and summarizer support navigation and recovery, evaluated separately from the underlying trained policy. - https://awesomecua.com/resources/arxiv-2602-13559/ · source: https://arxiv.org/abs/2602.13559v2 - Paper · reviewed 2026-10-18 · environments: web · topics: web-navigation, reinforcement-learning, planning, self-reflection - Published: 2026-02-14 · Authors: Yuyu Guo, Wenjie Yang, Siyuan Yang, Ziyang Liu, Cheng Chen, Yuan Wei, Yun Hu, Yang Huang, Guoliang Hao, Dongsheng Yuan, Jianming Wang, Xin Chen, Hang Yu, Lei Lei, Peng Di ### Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents Open CaptchaWorld evaluates multimodal agents on interactive browser-based CAPTCHA puzzles. Its benchmark distinguishes task success from the depth of visual reasoning and interaction needed to complete a puzzle. - https://awesomecua.com/resources/arxiv-2505-24878/ · source: https://arxiv.org/abs/2505.24878v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, web-navigation, screen-understanding - Published: 2025-05-30 · Authors: Yaxin Luo, Zhaoyi Li, Jiacheng Liu, Jiacheng Cui, Xiaohan Zhao, Zhiqiang Shen ### OpenAgents: An Open Platform for Language Agents in the Wild OpenAgents hosts three language agents, for data analysis, API plugins and autonomous web browsing, behind a web interface for non-expert users, with local deployment for developers. - https://awesomecua.com/resources/arxiv-2310-10634/ · source: https://arxiv.org/abs/2310.10634v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration - Published: 2023-10-16 · Authors: Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, Tao Yu ### OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety Evaluates agent safety across real browser, code, filesystem, shell, and messaging tools using rule-based and model-based assessments. - https://awesomecua.com/resources/arxiv-2507-06134/ · source: https://arxiv.org/abs/2507.06134v2 - Paper · reviewed 2026-10-16 · environments: web, desktop · topics: safety-security, evaluation - Published: 2025-07-08 · Authors: Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, Maarten Sap ### OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability Introduces OpenApps, a configurable app ecosystem for measuring UI-agent reliability under changes in appearance and content. Its evaluation compares task success and failure behavior across many versions of the same apps. - https://awesomecua.com/resources/arxiv-2511-20766/ · source: https://arxiv.org/abs/2511.20766v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, robustness, data-generation - Published: 2025-11-25 · Authors: Karen Ullrich, Jingtong Su, Claudia Shi, Arjun Subramonian, Amir Bar, Ivan Evtimov, Nikolaos Tsilivis, Randall Balestriero, Julia Kempe, Mark Ibrahim ### OpenComputer: Verifiable Software Worlds for Computer-Use Agents Introduces OpenComputer, which builds desktop tasks around application-state verifiers and refines those verifiers using execution feedback. An evaluation harness records trajectories and assigns auditable partial credit across professional software workflows. - https://awesomecua.com/resources/arxiv-2605-19769/ · source: https://arxiv.org/abs/2605.19769v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, data-generation, observability - Published: 2026-05-19 · Authors: Jinbiao Wei, Qianran Ma, Yilun Zhao, Xiao Zhou, Kangqi Ni, Guo Gan, Arman Cohan ### OpenCUA: Open Foundations for Computer-Use Agents Presents demonstration collection infrastructure, the AgentNet dataset, and a training pipeline for open computer-use models. - https://awesomecua.com/resources/arxiv-2508-09123/ · source: https://arxiv.org/abs/2508.09123v3 - Paper · reviewed 2026-10-16 · environments: desktop · topics: data-generation, planning - Published: 2025-08-12 · Authors: Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, Tao Yu ### OpenForgeRL: Train Harness-native Agents in Any Environment OpenForgeRL connects stateful agent harnesses to reinforcement-learning infrastructure through a model-call recording proxy and isolated rollout containers. Its validation includes browser and computer-use agents, alongside tool-oriented agents, and examines how harness choice affects learned reliability. - https://awesomecua.com/resources/arxiv-2607-21557/ · source: https://arxiv.org/abs/2607.21557v3 - Paper · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, sandbox, tool-integration, robustness - Published: 2026-07-23 · Authors: Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao ### OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use Introduces an open generalist computer-use stack combining visual and programmatic interaction, with model-controlled observation and task-state decisions. - https://awesomecua.com/resources/arxiv-2610-22200/ · source: https://arxiv.org/abs/2610.22200v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: tool-integration, planning - Published: 2026-10-15 · Authors: Hanzhang Zhou, Long Li, Quyu Kong, Lei Wang, Panrong Tong, Long Chen, Tianyu Xia, Gongjie Zhang, Xu Zhang, Chen Liu, Jianan Zhang, Chenglin Cai, Zhengwen Zeng, Steven Hoi, Yue Wang ### OpenHands: An Open Platform for AI Software Developers as Generalist Agents OpenHands is a platform for agents that write code, use a command line and browse the web, with sandboxed execution and support for agent evaluation. - https://awesomecua.com/resources/arxiv-2407-16741/ · source: https://arxiv.org/abs/2407.16741v3 - Paper · reviewed 2026-11-06 · environments: web · topics: tool-integration, sandbox, evaluation - Published: 2024-07-23 · Authors: Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, Graham Neubig ### OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools Introduces OpenMobile-2, combining simulated mobile apps, emulator tasks, released training data, and policies that can alternate GUI interaction with app-native tools. - https://awesomecua.com/resources/arxiv-2610-14157/ · source: https://arxiv.org/abs/2610.14157v1 - Paper · reviewed 2026-10-17 · environments: android, web · topics: mobile-navigation, tool-integration, data-generation, reinforcement-learning - Published: 2026-10-09 · Authors: Chenyang Yan, Yingying Zhang, Kanzhi Cheng, Qiushi Sun, Zhengyuan Pan, Zheng Ma, Hang Yan, Yian Wang, Nuo Chen, Jialin Cao, Xingdong Gong, Wenpo Song, Han Chen, Zichen Ding, Fangzhi Xu, Shujian Huang, Xinyu Dai, Yichen Liu, Tiankuo Yao, Bo Wang, Ben Kao, Jianbing Zhang, Lewei Lu, Dahua Lin ### OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis Presents OpenMobile, which builds exploration-based environment memory to synthesize task instructions and switches between learner and expert policies during trajectory collection. The resulting data includes recovery behavior and is tested through mobile-agent training and benchmark-overlap analysis. - https://awesomecua.com/resources/arxiv-2604-15093/ · source: https://arxiv.org/abs/2604.15093v1 - Paper · reviewed 2026-10-17 · environments: android · topics: data-generation, imitation-learning, mobile-navigation - Published: 2026-04-16 · Authors: Kanzhi Cheng, Zehao Li, Zheng Ma, Nuo Chen, Jialin Cao, Qiushi Sun, Zichen Ding, Fangzhi Xu, Hang Yan, Jiajun Chen, Anh Tuan Luu, Jianbing Zhang, Lewei Lu, Dahua Lin ### OpenPhone: Mobile Agentic Foundation Models OpenPhone trains a compact mobile policy with supervised learning followed by GRPO on synthetic GUI data, incorporating reasoning and memory management. Its device-cloud design handles routine work locally and delegates complex subtasks, with online evaluation on AndroidLab. - https://awesomecua.com/resources/arxiv-2510-22009/ · source: https://arxiv.org/abs/2510.22009v3 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, memory, efficiency, mobile-navigation - Published: 2025-10-24 · Authors: Yangqin Jiang, Chao Huang ### OpenVAM: Open-World Visual Attention Modeling with VLMs Combines dense saliency prediction with explanations of attended elements, including webpage layouts; evaluates human-attention agreement and textual rationales rather than agent actions. - https://awesomecua.com/resources/arxiv-2609-31364/ · source: https://arxiv.org/abs/2609.31364v1 - Paper · reviewed 2026-10-20 · environments: web · topics: screen-understanding, ui-design, evaluation, data-generation - Published: 2026-09-25 · Authors: Kiana Hooshanfar, Amirhossein Kazerouni, Alireza Hosseini, Michael Brudno, Babak Taati - Licenses: document CC-BY-4.0 ### OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents Introduces OpenWebRL, a framework for training visual web agents with online multi-turn reinforcement learning on live websites, including browser infrastructure and success judging. - https://awesomecua.com/resources/arxiv-2606-02031/ · source: https://arxiv.org/abs/2606.02031v2 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, web-navigation, sandbox, evaluation - Published: 2026-06-01 · Authors: Rui Yang, Qianhui Wu, Yuxi Chen, Hao Bai, Wenlin Yao, Hao Cheng, Baolin Peng, Huan Zhang, Tong Zhang, Jianfeng Gao ### OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization Develops multimodal browser agents through repeated cycles of real-web exploration, trajectory feedback, and policy optimization. - https://awesomecua.com/resources/arxiv-2410-19609/ · source: https://arxiv.org/abs/2410.19609v1 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, data-generation - Published: 2024-10-25 · Authors: Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, Dong Yu ### ORBIT: A Framework for Multi-Agent Safety and Security Evaluations Inspect-based framework that varies multi-agent roles, communication, attacks and defenses, with BrowserGym and virtual-desktop GUI adapters alongside coding and other tool scenarios. - https://awesomecua.com/resources/arxiv-2609-33102/ · source: https://arxiv.org/abs/2609.33102v1 - Paper · reviewed 2026-10-20 · environments: web, desktop · topics: evaluation, safety-security, web-navigation - Published: 2026-09-27 · Authors: Ben Hagag, William L. Anderson, Srija Chakraborty, Christian Schroeder de Witt ### Orchard: An Open-Source Agentic Modeling Framework Orchard provides an environment service and training recipes for several agent domains, including Orchard-GUI, a vision-language computer-use agent. - https://awesomecua.com/resources/arxiv-2605-15040/ · source: https://arxiv.org/abs/2605.15040v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: sandbox, reinforcement-learning, evaluation - Published: 2026-05-14 · Authors: Baolin Peng, Wenlin Yao, Qianhui Wu, Hao Cheng, Xiao Yu, Rui Yang, Tao Ge, Alessandro Sordoni, Xingdi Yuan, Yelong Shen, Pengcheng He, Tong Zhang, Zhou Yu, Jianfeng Gao ### Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent Orcust combines principle-constrained reward modeling with GUI trajectory collection in instrumented virtual machines. Environment-checkable and model-derived principles guide stepwise feedback for grounding, basic operations, and complete task execution. - https://awesomecua.com/resources/arxiv-2509-17917/ · source: https://arxiv.org/abs/2509.17917v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reinforcement-learning, reward-models, data-generation, sandbox - Published: 2025-09-22 · Authors: Junyu Lu, Songxin Zhang, Zejian Xie, Zhuoyang Song, Jiaxing Zhang ### OS-ATLAS: A Foundation Action Model for Generalist GUI Agents Combines cross-platform GUI grounding data synthesis with action-model training to study generalization to unfamiliar interfaces. - https://awesomecua.com/resources/arxiv-2410-23218/ · source: https://arxiv.org/abs/2410.23218v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, data-generation - Published: 2024-10-30 · Authors: Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, Yu Qiao ### OS-Copilot: Towards Generalist Computer Agents with Self-Improvement Provides an operating-system agent framework connecting applications, files, terminals, and web tools, with accumulated skills for subsequent tasks. - https://awesomecua.com/resources/arxiv-2402-07456/ · source: https://arxiv.org/abs/2402.07456v2 - Paper · reviewed 2026-10-16 · environments: desktop, web · topics: tool-integration, planning - Published: 2024-02-12 · Authors: Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, Lingpeng Kong ### OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis Builds GUI trajectories by exploring environments first and deriving tasks retrospectively, with reward-model filtering of generated trajectories. - https://awesomecua.com/resources/arxiv-2412-19723/ · source: https://arxiv.org/abs/2412.19723v3 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation - Published: 2024-12-27 · Authors: Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, Zhiyong Wu ### OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents Extends OSWorld with tasks covering user misuse, prompt injection, and model misbehavior, with assessment of both task success and safety. - https://awesomecua.com/resources/arxiv-2506-14866/ · source: https://arxiv.org/abs/2506.14866v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: safety-security, evaluation - Published: 2025-06-17 · Authors: Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, Maksym Andriushchenko ### OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents Trains native computer-use models to coordinate GUI actions and code execution using hybrid demonstrations and online reinforcement learning. - https://awesomecua.com/resources/arxiv-2610-22234/ · source: https://arxiv.org/abs/2610.22234v1 - Paper · reviewed 2026-10-16 · environments: desktop, linux · topics: tool-integration, reinforcement-learning - Published: 2026-10-15 · Authors: Bowen Yang, Kaiming Jin, Zichen Ding, Zhenyu Wu, Zehao Li, Zhoumianze Liu, Kanzhi Cheng, Fangzhi Xu, Yanan Sun, Qiushi Sun, Yu Qiao ### OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents Uses step-level confidence estimates to decide when a GUI agent should act autonomously or request human intervention in complex situations. - https://awesomecua.com/resources/arxiv-2503-16465/ · source: https://arxiv.org/abs/2503.16465v3 - Paper · reviewed 2026-10-16 · environments: android · topics: human-in-the-loop, safety-security, mobile-navigation - Published: 2025-02-26 · Authors: Pengzhou Cheng, Zheng Wu, Zongru Wu, Aston Zhang, Zhuosheng Zhang, Gongshen Liu ### OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth? OS-MAP organizes 416 computer-use tasks across 15 applications by a five-level automation taxonomy and a hierarchy of user demands. The benchmark separates required autonomy from generalization scope to diagnose where agents struggle as task demands increase. - https://awesomecua.com/resources/arxiv-2507-19132/ · source: https://arxiv.org/abs/2507.19132v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, automation, planning - Published: 2025-07-25 · Authors: Xuetian Chen, Yinghao Chen, Xinfeng Yuan, Zhuo Peng, Lu Chen, Yuekeng Li, Zhoujia Zhang, Yingqian Huang, Leyan Huang, Jiaqing Liang, Tianbao Xie, Zhiyong Wu, Qiushi Sun, Biqing Qi, Bowen Zhou ### OS-Marathon: Benchmarking Computer-Use Agents on Vast-Horizon, Repetitive Tasks Introduces OS-Marathon to evaluate long repetitive computer-use workflows whose execution grows with the amount of input data. It studies failure accumulation under task decomposition and GraphDemo, which adapts agents using a demonstration of the recurring workflow. - https://awesomecua.com/resources/arxiv-2601-20650/ · source: https://arxiv.org/abs/2601.20650v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, long-horizon, human-in-the-loop - Published: 2026-01-28 · Authors: Jing Wu, Wenjie Ai, Daphne Barretto, Yiye Chen, Qingyu Chen, Yuhang He, Pranit Chawla, Nicholas Gydé, Yanan Jian, Vibhav Vineet ### OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models OS-Oracle synthesizes GUI critic examples and trains a pre-action critic with supervised learning followed by consistency-preserving GRPO. It introduces OS-Critic Bench across mobile, web, and desktop settings and tests critic-assisted agents in OSWorld and AndroidWorld. - https://awesomecua.com/resources/arxiv-2512-16295/ · source: https://arxiv.org/abs/2512.16295v1 - Paper · reviewed 2026-10-17 · environments: web, desktop, android · topics: reward-models, reinforcement-learning, data-generation, evaluation - Published: 2025-12-18 · Authors: Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Qiushi Sun, Zhaoyang Liu, Zhoumianze Liu, Yu Qiao, Xiangyu Yue, Zun Wang, Zichen Ding ### OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows Introduces MobileRisk-Live and OS-Sentinel, combining a formal verifier for system-level violations with a multimodal judge for contextual risks in mobile GUI workflows. - https://awesomecua.com/resources/arxiv-2510-24411/ · source: https://arxiv.org/abs/2510.24411v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, evaluation, mobile-navigation - Published: 2025-10-28 · Authors: Qiushi Sun, Mukai Li, Zhoumianze Liu, Zhihui Xie, Fangzhi Xu, Zhangyue Yin, Kanzhi Cheng, Zehao Li, Zichen Ding, Qi Liu, Zhiyong Wu, Zhuosheng Zhang, Ben Kao, Lingpeng Kong ### OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents Provides an evaluation toolkit for joint safety, efficiency, and multimodal robustness analysis of operating-system GUI agents. - https://awesomecua.com/resources/arxiv-2604-24348/ · source: https://arxiv.org/abs/2604.24348v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: evaluation, safety-security, efficiency, robustness - Published: 2026-04-27 · Authors: Zheng Wu, Yi Hua, Zhaoyuan Huang, Chenhao Xue, Yijie Lu, Pengzhou Cheng, Zongru Wu, Lingzhong Dong, Gongshen Liu, Xinghao Jiang, Zhuosheng Zhang ### OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent OS-Symphony coordinates milestone-based reflection and memory with tools that build visually aligned tutorials through browser interaction. The framework evaluates trajectory correction and adaptation on online computer-use benchmarks including OSWorld. - https://awesomecua.com/resources/arxiv-2601-07779/ · source: https://arxiv.org/abs/2601.07779v1 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: memory, self-reflection, planning, tool-integration - Published: 2026-01-12 · Authors: Bowen Yang, Kaiming Jin, Zhenyu Wu, Zhaoyang Liu, Qiushi Sun, Zehao Li, JingJing Xie, Zhoumianze Liu, Fangzhi Xu, Kanzhi Cheng, Qingyun Li, Yian Wang, Yu Qiao, Zun Wang, Zichen Ding ### OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards Presents OS-Themis, a multi-agent critic that divides GUI trajectories into verifiable milestones and reviews their evidence before assigning an outcome. OGRBench evaluates GUI rewards across platforms, with AndroidWorld experiments examining training and trajectory filtering. - https://awesomecua.com/resources/arxiv-2603-19191/ · source: https://arxiv.org/abs/2603.19191v1 - Paper · reviewed 2026-10-17 · environments: android, cross-platform · topics: reward-models, reinforcement-learning, evaluation - Published: 2026-03-19 · Authors: Zehao Li, Zhenyu Wu, Yibo Zhao, Bowen Yang, Jingjing Xie, Zhaoyang Liu, Zhoumianze Liu, Kaiming Jin, Jianze Liang, Zonglin Li, Feng Wu, Bowen Zhou, Zun Wang, Zichen Ding ### OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning OSCAR translates user instructions into Python control code while observing screen images and acting through mouse and keyboard operations. A state-machine controller handles errors and replans tasks during execution across desktop and mobile applications. - https://awesomecua.com/resources/arxiv-2410-18963/ · source: https://arxiv.org/abs/2410.18963v1 - Paper · reviewed 2026-10-17 · environments: desktop, cross-platform · topics: planning, state-tracking, tool-integration, robustness - Published: 2024-10-24 · Authors: Xiaoqiang Wang, Bang Liu ### OSExpert: Computer-Use Agents Learning Professional Skills via Exploration Uses GUI-based depth-first exploration to verify elementary software functions, assemble action primitives, and construct curricula of composed skills. OSExpert-Eval examines how the resulting procedural knowledge supports complex computer-use tasks. - https://awesomecua.com/resources/arxiv-2603-07978/ · source: https://arxiv.org/abs/2603.07978v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: planning, data-generation, evaluation - Published: 2026-03-09 · Authors: Jiateng Liu, Zhenhailong Wang, Rushi Wang, Bingxuan Li, Jeonghwan Kim, Aditi Tiwari, Pengfei Yu, Denghui Zhang, Heng Ji ### OSGuard: A Benchmark for Safety in Computer-Use Agents Evaluates action-level guardrails and full computer tasks with environment-dependent safety constraints, separating safe completion from nominal success. - https://awesomecua.com/resources/arxiv-2606-15034/ · source: https://arxiv.org/abs/2606.15034v3 - Paper · reviewed 2026-10-16 · environments: desktop · topics: safety-security, evaluation - Published: 2026-06-13 · Authors: Mina Mohammadmirzaei, Jeffrey Flanigan ### OSGym: Scalable OS Infra for Computer Use Agents OSGym supplies full operating-system sandboxes with GUIs for parallel computer-use data collection and training. It combines per-replica state management, hardware-aware orchestration, copy-on-write virtual-machine disks, and layered failure recovery. - https://awesomecua.com/resources/arxiv-2511-11672/ · source: https://arxiv.org/abs/2511.11672v5 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: sandbox, efficiency, reinforcement-learning, data-generation - Published: 2025-11-11 · Authors: Zengyi Qin, Jinyuan Chen, Yunze Man, Shengcao Cao, Ziqi Pang, Zhuoyuan Wang, Han Fang, Ling Zhu, Zixin Xie, Zibu Wei, Tianshu Ran, Haoran Geng, Ray Pan, Qizhen Sun, Zachary Bright, Yuyang Cai, Chongye Yang, Jiace Zhao, Tianrui Liu, Han Cao, Yeyang Zhou, Rui Wang, Song Wang, Xiang Ren, Bo Zhang, Yutong Ban, Pieter Abbeel, Brian Anthony ### OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models Introduces OSReward and its harder and multidimensional variants for assessing judgments of computer-use trajectories. It also releases a reasoning-annotated judgment corpus and trains reward models, examining reliability, leniency bias, and evaluation cost. - https://awesomecua.com/resources/arxiv-2607-28609/ · source: https://arxiv.org/abs/2607.28609v2 - Paper · reviewed 2026-10-17 · environments: cross-platform · topics: reward-models, evaluation, data-generation - Published: 2026-07-30 · Authors: Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong ### OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents Introduces OSUniverse, a desktop GUI benchmark spanning precise clicks through multistep and multiapplication tasks. It provides both manual scoring and an automated validation mechanism. - https://awesomecua.com/resources/arxiv-2505-03570/ · source: https://arxiv.org/abs/2505.03570v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, cross-app, long-horizon - Published: 2025-05-06 · Authors: Mariya Davydova, Daniel Jeffries, Patrick Barker, Arturo Márquez Flores, Sinéad Ryan ### OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks Introduces extended computer-use workflows with realistic artifacts, changing information, hidden state, and partial-progress and safety assessments. - https://awesomecua.com/resources/arxiv-2606-29537/ · source: https://arxiv.org/abs/2606.29537v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: evaluation, long-horizon, workflow - Published: 2026-06-28 · Authors: Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu, Jiayang Sun, Jiamin Song, Kaiqian Cui, Bowen Wang, Haoyuan Wu, Yitong Li, Dunjie Lu, Haikong Lu, Qi Zhen, Xinyuan Wang, Jiaqi Deng, Yuhao Yang, Cheng Chen, Boyuan Zheng, Alex Su, Xiao Yu, Hao Zou, Saaket Agashe, Xing Han Lu, Manpreet Kaur, Zhengyang Qi, Vincent Sunn Chen, Frederic Sala, Dayiheng Liu, Junyang Lin, Zhou Yu, Yu Su, Siva Reddy, Xin Eric Wang, Peng Qi, Tianbao Xie, Tao Yu ### OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents OSWorld-Human adds human-determined trajectories to OSWorld to evaluate the efficiency of desktop computer-use agents. The study compares 16 agents and analyzes model-call latency, changes in step duration during task execution, and excess action steps. - https://awesomecua.com/resources/arxiv-2506-16042/ · source: https://arxiv.org/abs/2506.16042v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, efficiency, observability - Published: 2025-06-19 · Authors: Reyna Abhyankar, Qi Qi, Yiying Zhang ### OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents OSWorld-MCP evaluates how agents choose between GUI operations and MCP tool calls in computer tasks. It combines generated and curated tools, with 158 manually validated tools across seven applications, to examine tool use alongside task completion. - https://awesomecua.com/resources/arxiv-2510-24563/ · source: https://arxiv.org/abs/2510.24563v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, tool-integration, planning - Published: 2025-10-28 · Authors: Hongrui Jia, Jitong Liao, Xi Zhang, Haiyang Xu, Tianbao Xie, Chaoya Jiang, Ming Yan, Si Liu, Wei Ye, Fei Huang ### OSWorld-Pro: Process-based Evaluation for Computer Use Agents Adds annotated intermediate subgoals to computer-use tasks and uses process-oriented evaluation to identify action and progress failures. - https://awesomecua.com/resources/arxiv-2609-24890/ · source: https://arxiv.org/abs/2609.24890v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: evaluation, observability - Published: 2026-09-21 · Authors: Zhilin Wang, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou, Karan Sapra, Andrew Tao, Jan Kautz, Yi Dong ### OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software Benchmarks computer agents on scientific-software workflows with task-specific checks of application states and generated research artifacts. - https://awesomecua.com/resources/arxiv-2609-39903/ · source: https://arxiv.org/abs/2609.39903v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: evaluation, science - Published: 2026-09-30 · Authors: Dingyuan Dai, Heli Qi, Lei Liu, Yinxi Li, Baiding Chen, Zijun Dou, Qingcheng Zeng, Qi Kang, Oliver Sun, Eric Wang, Bo Zhou, Haixin Wang, Yufan Du, Shi Bo, Ruihan Lin, Mengqi Yuan, Dunjie Lu, Steven Dillmann, Yiming Shi, Tina Su, Amy Xin, Minghao Liu, Xi Wang, Xu Huang, Ge Zhang, Pengyu Nie, Zhen Yang, Jie Tang, Juanzi Li, Weihao Xuan, Tianyu Liu ### OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments Introduces a real-computer evaluation environment with execution-based checks for tasks spanning desktop applications and operating-system state. - https://awesomecua.com/resources/arxiv-2404-07972/ · source: https://arxiv.org/abs/2404.07972v2 - Paper · reviewed 2026-10-16 · environments: desktop, linux, windows, macos · topics: evaluation, cross-app - Published: 2024-04-11 · Authors: Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu ### OTora: A Unified Red Teaming Framework for Reasoning-Level Denial-of-Service in LLM Agents OTora generates attacks that increase reasoning and tool use in multi-step LLM agents while preserving task outcomes; evaluations include WebShop agents. - https://awesomecua.com/resources/arxiv-2605-08876/ · source: https://arxiv.org/abs/2605.08876v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, efficiency, evaluation - Published: 2026-05-09 · Authors: Xinyu Li, Ronghui Mu, Lin Li, Tianjin Huang, Gaojie Jin ### Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agents Introduces CarryOn, which maintains evidence-linked task objects and remaining-work context across history compression, with evaluations on desktop tasks and web research. - https://awesomecua.com/resources/arxiv-2610-18784/ · source: https://arxiv.org/abs/2610.18784v2 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: memory, long-horizon, efficiency - Published: 2026-10-13 · Authors: Yuchen Lu, Julian McAuley, Lei Zhang ### PageGuide: Browser extension to assist users in navigating a webpage and locating information PageGuide is a browser extension that highlights evidence for answers on a page and guides users through web pages with evidence captured at each step. - https://awesomecua.com/resources/arxiv-2604-23772/ · source: https://arxiv.org/abs/2604.23772v5 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, human-in-the-loop, screen-understanding - Published: 2026-04-26 · Authors: Tin Nguyen, Thang T. Truong, Runtao Zhou, Trung Bui, Chirag Agarwal, Anh Totti Nguyen ### PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control Introduces PAGE Bench for point-precise geometric GUI construction and PAGER, an agent that plans around primitive dependencies and executes pixel-level actions. - https://awesomecua.com/resources/arxiv-2605-15963/ · source: https://arxiv.org/abs/2605.15963v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, grounding, planning, reinforcement-learning - Published: 2026-05-15 · Authors: Jingxuan Wei, Xi Bai, Shan Liu, Caijun Jia, Zheng Sun, Xinglong Xu, Siyuan Li, Linzhuang Sun, Bihui Yu, Conghui He, Cheng Tan ### PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents PAL-UI combines observation/action summaries with selective retrieval of historical screenshots during planning. It trains mobile navigation models from step-level supervision and evaluates transfer to web navigation without additional training. - https://awesomecua.com/resources/arxiv-2510-00413/ · source: https://arxiv.org/abs/2510.00413v2 - Paper · reviewed 2026-10-17 · environments: web, cross-platform · topics: memory, planning, interaction-history, mobile-navigation - Published: 2025-10-01 · Authors: Zikang Liu, Junyi Li, Wayne Xin Zhao, Dawei Gao, Yaliang Li, Ji-rong Wen ### PalmClaw: A Native On-Device Agent Framework for Mobile Phones Introduces PalmClaw, an on-device phone agent framework that manages sessions, memory, and tools, exposing mobile capabilities through structured device-tool calls. - https://awesomecua.com/resources/arxiv-2607-13027/ · source: https://arxiv.org/abs/2607.13027v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: tool-integration, memory, mobile-navigation - Published: 2026-07-14 · Authors: Hongru Cai, Yongqi Li, Ran Wei, Wenjie Li ### PANDO: Efficient Multimodal AI Agents via Online Skill Distillation Analyzes inefficiency in VisualWebArena trajectories and distills reusable online skills to reduce repeated actions and token overhead in multimodal web agents. - https://awesomecua.com/resources/arxiv-2605-24785/ · source: https://arxiv.org/abs/2605.24785v2 - Paper · reviewed 2026-10-18 · environments: web · topics: efficiency, memory, web-navigation - Published: 2026-05-24 · Authors: Yubo Li, Yidi Miao, Yuntian Shen, Yuxin Liu ### PastForward: Faster On-Device GUI Agents via Computational Experience Reuse Introduces PastForward, which reuses earlier output sequences as verified decoding proposals and starts future action computation from predicted GUI transitions. Screen checks determine whether speculative computation and cached model states can be retained. - https://awesomecua.com/resources/arxiv-2609-32166/ · source: https://arxiv.org/abs/2609.32166v1 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, efficiency, mobile-navigation - Published: 2026-09-26 · Authors: Taehwan Park, Changmin Lee, Hayeon Lee, Taesik Gong ### PatchScout: Thematic Web Data Collection via Information Foraging Introduces PatchScout, a multi-agent method for collecting thematic documents from the live web by deciding when to explore a new information region. - https://awesomecua.com/resources/arxiv-2611-00169/ · source: https://arxiv.org/abs/2611.00169v1 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, long-horizon, web-navigation - Published: 2026-10-30 · Authors: Michael West, Eduard Dragut ### PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World Combines PC Tracker interaction capture, completion of action and reasoning annotations, and planning and grounding agents for desktop workflows. - https://awesomecua.com/resources/arxiv-2412-17589/ · source: https://arxiv.org/abs/2412.17589v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: recording, data-generation, planning - Published: 2024-12-23 · Authors: Yanheng He, Jiahe Jin, Shijie Xia, Jiadi Su, Runze Fan, Haoyang Zou, Xiangkun Hu, Pengfei Liu ### PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC Uses active perception and hierarchical agents for instruction decomposition, progress tracking, action selection, and reflection in PC workflows. - https://awesomecua.com/resources/arxiv-2502-14282/ · source: https://arxiv.org/abs/2502.14282v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: planning, cross-app - Published: 2025-02-20 · Authors: Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, Fei Huang ### Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations Introduces PriceBlind, a visual perturbation method for controlled screenshot-based evaluation of price-constrained multimodal agents. The reported transfer experiments use a simplified single-turn coordinate-selection protocol. - https://awesomecua.com/resources/arxiv-2604-16515/ · source: https://arxiv.org/abs/2604.16515v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: safety-security, robustness - Published: 2026-04-15 · Authors: Jiachen Qian, Zhaolu Kang ### Perceive Before Reasoning: A Pre-Reasoning Perception Framework for Efficient and Reliable Proactive Mobile Agents Introduces PRPF, a two-stage framework for proactive mobile agents that gates intervention with MPP and invokes PAR only when assistance is warranted. - https://awesomecua.com/resources/arxiv-2606-03236/ · source: https://arxiv.org/abs/2606.03236v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, planning, mobile-navigation - Published: 2026-06-02 · Authors: Zhijie Ding, Weinan Hong, Zicheng Zhu, Lei Li, Dezhi Kong, Hao Wang, Peng Zhou, Xuchu Jiang, Jiaming Xu ### PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation Introduces PerceptUI, a persona-conditioned model for predicting how particular users answer questions about interfaces, supporting synthetic UI/UX evaluation. - https://awesomecua.com/resources/arxiv-2606-05697/ · source: https://arxiv.org/abs/2606.05697v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: ui-design, evaluation, imitation-learning - Published: 2026-06-04 · Authors: Nicolas Bougie, Xiaotong Ye, Gian Maria Marconi, Narimasa Watanabe ### PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM PeriGuru interprets smartphone GUI screenshots with computer vision and uses a language model to decide the next action. A peripheral robotic arm executes the phone interactions, providing an alternative mechanism for operating mobile apps. - https://awesomecua.com/resources/arxiv-2409-09354/ · source: https://arxiv.org/abs/2409.09354v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, screen-understanding, automation - Published: 2024-09-14 · Authors: Kelin Fu, Yang Tian, Kaigui Bian ### Permission Manifests for Web Agents Proposes agent-permissions.json, a website manifest for declaring which interactions web agents may perform, with optional references to APIs. - https://awesomecua.com/resources/arxiv-2601-02371/ · source: https://arxiv.org/abs/2601.02371v2 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation, tool-integration - Published: 2025-12-07 · Authors: Samuele Marro, Alan Chan, Xinxing Ren, Lewis Hammond, Jesse Wright, Gurjyot Wanga, Tiziano Piccardi, Nuno Campos, Tobin South, Jialin Yu, Sunando Sengupta, Eric Sommerlade, Alex Pentland, Philip Torr, Jiaxin Pei ### PerPilot: Personalizing VLM-based Mobile Agents via Memory and Exploration Introduces PerPilot and the PerInstruct dataset for completing user-specific mobile instructions through memory retrieval, app exploration, and clarification when missing context remains unresolved. - https://awesomecua.com/resources/arxiv-2508-18040/ · source: https://arxiv.org/abs/2508.18040v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: memory, planning, human-in-the-loop, mobile-navigation - Published: 2025-08-25 · Authors: Xin Wang, Zhiyao Cui, Hao Li, Ya Zeng, Chenxu Wang, Ruiqi Song, Yihang Chen, Kun Shao, Qiaosheng Zhang, Jinzhuo Liu, Siyue Ren, Shuyue Hu, Zhen Wang ### Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History Persona2Web tests whether web agents can resolve ambiguous requests using preferences implicit in a user’s history. Its reasoning-aware evaluation compares agent architectures and history-access strategies on personalized tasks on the open web. - https://awesomecua.com/resources/arxiv-2602-17003/ · source: https://arxiv.org/abs/2602.17003v3 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, memory, web-navigation - Published: 2026-02-19 · Authors: Serin Kim, Sangam Lee, Dongha Lee ### PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas PersonaCUABench varies simulated user requests and clarification dialogue over fixed web tasks, comparing task completion with requirement handling and persona-agent satisfaction ratings. - https://awesomecua.com/resources/arxiv-2610-22990/ · source: https://arxiv.org/abs/2610.22990v1 - Paper · reviewed 2026-10-20 · environments: web · topics: evaluation, web-navigation, human-in-the-loop - Published: 2026-10-16 · Authors: Haoming Li, Defu Cao, Xu Hu, Yingtao Luo, Yiyang Wang, Xianghe Xu, Jiashu He, Wanting Jiang, Yifan Zhang, Yunhan Zhao, Siyang Wu, Chen Yang, Haixin Wang, Zheyuan Deng, Taoran Li, Jingwen Zhang, Zhejian Peng, Shushu Liang, Yue Zhao, Bingzhe Li, Feng Chen, Yifan Wang, Jintao Huang, Yijun Wang, Yuexing Hao, Xiaomin Li ### PersonaFingerprint: Measuring Persona Inference on Modern Websites with LLM-Driven Browsing PersonaFingerprint studies persona inference from encrypted web traffic, using a persona-constrained computer-use agent to browse real websites and collect traces. - https://awesomecua.com/resources/arxiv-2605-15962/ · source: https://arxiv.org/abs/2605.15962v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation, data-generation - Published: 2026-05-15 · Authors: Chuxu Song, Hao Wang, Richard Martin ### PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records Introduces AndroidIntent to evaluate how GUI agents infer omitted preferences and recurring routines from long-term user records. Its HIM-Agent maintains hierarchical memory of preferences and routines for personalized execution and proactive suggestions. - https://awesomecua.com/resources/arxiv-2601-09636/ · source: https://arxiv.org/abs/2601.09636v2 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, evaluation, mobile-navigation - Published: 2026-01-14 · Authors: Yibo Lyu, Gongwei Chen, Rui Shao, Weili Guan, Liqiang Nie ### PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails Introduces PersonaTrail, a benchmark for personalized web agents using browsing histories, and PACMem, a structured memory method for guiding navigation. - https://awesomecua.com/resources/arxiv-2607-20482/ · source: https://arxiv.org/abs/2607.20482v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, memory, interaction-history, web-navigation - Published: 2026-05-30 · Authors: Seungbin Yang, Chaewoon Ki, Dohyun Lee, Jaegul Choo, ChaeHun Park ### PG-Agent: An Agent Powered by Page Graph Introduces PG-Agent, which turns GUI operation episodes into page graphs and retrieves guidance from them to support task decomposition in new scenarios. - https://awesomecua.com/resources/arxiv-2509-03536/ · source: https://arxiv.org/abs/2509.03536v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, planning, interaction-history, screen-understanding - Published: 2025-08-27 · Authors: Weizhi Chen, Ziwei Wang, Leyang Yang, Sheng Zhou, Xiaoxuan Tang, Jiajun Bu, Yong Li, Wei Jiang ### Phi-Ground Tech Report: Advancing Perception in GUI Grounding Reports a GUI-grounding model family and empirical design choices covering source data, perception training, and coordinate prediction. - https://awesomecua.com/resources/arxiv-2507-23779/ · source: https://arxiv.org/abs/2507.23779v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, screen-understanding - Published: 2025-07-31 · Authors: Miaosen Zhang, Ziqiang Xu, Jialiang Zhu, Qi Dai, Kai Qiu, Yifan Yang, Chong Luo, Tianyi Chen, Justin Wagle, Tim Franklin, Baining Guo ### PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents Introduces PhoneCLI, which explores mobile app interfaces and compiles navigation paths into reusable commands without internal app APIs or instrumentation. At execution time it verifies compiled paths and falls back to a visual agent for open-ended interaction or failed navigation. - https://awesomecua.com/resources/arxiv-2609-35671/ · source: https://arxiv.org/abs/2609.35671v2 - Paper · reviewed 2026-10-17 · environments: android · topics: automation, efficiency, mobile-navigation - Published: 2026-09-28 · Authors: Yangqin Jiang, Lingrui Xu, Chao Huang ### PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions PhoneHarness routes phone workflows across GUI actions, device-side commands and host tools while retaining auditable execution traces. Its accompanying benchmark checks observable task side effects, separating mixed-action execution from plausible final-answer generation. - https://awesomecua.com/resources/arxiv-2606-14832/ · source: https://arxiv.org/abs/2606.14832v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: mobile-navigation, tool-integration, evaluation, workflow - Published: 2026-06-12 · Authors: Chenxin Li, Zhengyao Fang, Zhengyang Tang, Pengyuan Lyu, Xingran Zhou, Xin Lai, Fei Tang, Liang Wu, Yiduo Guo, Weinong Wang, Junyi Li, Yi Zhang, Yang Ding, Huawen Shen, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Benyou Wang, Chengquan Zhang, Han Hu ### PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents Introduces PhoneWorld, which converts observed app pages, transitions and state changes into executable Android environments through a build-inspect-repair process. Generated tasks and programmatic verifiers support benchmarking, trajectory collection and online training. - https://awesomecua.com/resources/arxiv-2605-29486/ · source: https://arxiv.org/abs/2605.29486v3 - Paper · reviewed 2026-10-17 · environments: android · topics: data-generation, sandbox, reinforcement-learning, evaluation - Published: 2026-05-28 · Authors: Yuxuan Liu, Xin Lai, Junyi Li, Pengyuan Lyu, Jason, Yiduo Guo, Zhengyao Fang, Yang Ding, Yi Zhang, Weinong Wang, Huawen Shen, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Chengquan Zhang, Han Hu, Benyou Wang, Ji-Rong Wen, Rui Yan, Zhengyang Tang ### PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents PIRA-Bench evaluates proactive intent recommendations from noisy GUI screenshot sequences containing interleaved tasks and user context. Its PIRF baseline combines memory and state tracking to maintain multiple task threads and identify opportunities to assist. - https://awesomecua.com/resources/arxiv-2603-08013/ · source: https://arxiv.org/abs/2603.08013v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, screen-understanding, memory, state-tracking, human-in-the-loop - Published: 2026-03-09 · Authors: Yuxiang Chai, Shunye Tang, Han Xiao, Rui Liu, Hongsheng Li ### Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding Pix2Struct pretrains an image-to-text model by reconstructing simplified HTML from masked webpage screenshots. Its variable-resolution visual representation is evaluated across several domains, including user-interface understanding. - https://awesomecua.com/resources/arxiv-2210-03347/ · source: https://arxiv.org/abs/2210.03347v2 - Paper · reviewed 2026-10-18 · environments: web · topics: screen-understanding, data-generation - Published: 2022-10-07 · Authors: Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova ### PixelPrune: Pixel-Level Adaptive Visual Token Reduction via Predictive Coding PixelPrune removes pixel-redundant patches before the vision encoder, reducing both encoding and downstream language-model work. The training-free method evaluates lossless and controlled-lossy compression on document and GUI tasks. - https://awesomecua.com/resources/arxiv-2604-00886/ · source: https://arxiv.org/abs/2604.00886v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: efficiency, screen-understanding, training-free, evaluation - Published: 2026-04-01 · Authors: Nan Wang, Zhiwei Jin, Chen Chen, Haonan Lu ### PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels Introduces PixelWeb, a webpage GUI dataset with element images, contours, and pixel masks. Its annotation pipeline combines bitmap channel information with DOM-based visibility and stacking analysis to address occlusion and overlapping elements. - https://awesomecua.com/resources/arxiv-2504-16419/ · source: https://arxiv.org/abs/2504.16419v2 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, grounding, screen-understanding - Published: 2025-04-23 · Authors: Qi Yang, Weichen Bi, Haiyang Shen, Yaoqi Guo, Yun Ma ### Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks Separates planning from execution and creates synthetic plan annotations for training agents on extended web-navigation tasks. - https://awesomecua.com/resources/arxiv-2503-09572/ · source: https://arxiv.org/abs/2503.09572v3 - Paper · reviewed 2026-10-16 · environments: web · topics: planning, long-horizon, data-generation - Published: 2025-03-12 · Authors: Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, Amir Gholami ### Planner Matters! An Efficient and Unbalanced Multi-agent Collaboration Framework for Long-horizon Planning A planner-centric RL framework divides agent automation among a planner, actor, and memory manager, with evaluations that include web navigation and OS control. - https://awesomecua.com/resources/arxiv-2605-02168/ · source: https://arxiv.org/abs/2605.02168v1 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, reinforcement-learning, memory, long-horizon - Published: 2026-05-04 · Authors: Wenyi Wu, Sibo Zhu, Kun Zhou, Biwei Huang ### PlayCoder: Making LLM-Generated GUI Code Playable PlayEval measures whether generated GUI applications remain logically playable, using PlayTester to execute task-oriented GUI playthroughs. PlayCoder closes the loop between repository-aware generation, interactive testing and targeted code repair. - https://awesomecua.com/resources/arxiv-2604-19742/ · source: https://arxiv.org/abs/2604.19742v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, automation, ui-design, workflow - Published: 2026-04-21 · Authors: Zhiyuan Peng, Wei Tao, Xin Yin, Chenhao Ying, Yuan Luo, Yiwen Guo ### Player-1: Learning to Play from Synthetic Experience Extends computer-use policies to game control through synthetic training environments and richer keyboard and pointer actions. Its evaluation concerns games rather than ordinary productivity workflows. - https://awesomecua.com/resources/arxiv-2610-20584/ · source: https://arxiv.org/abs/2610.20584v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, reinforcement-learning, evaluation - Published: 2026-10-14 · Authors: Mingyu Ouyang, Yiran Wang, Siyuan Hu, Xingyilang Yin, Huiqiang Sun, Guangzhi Wang, Junfu Pu, Cheng Cheng, Honghao Chen, Ying Shan, Mike Zheng Shou, Hwee Tou Ng ### PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence Introduces PlaySuite, a closed-loop video-game benchmark that evaluates interactive visual agents, including computer-use agents, over extended play. - https://awesomecua.com/resources/arxiv-2610-07127/ · source: https://arxiv.org/abs/2610.07127v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon, screen-understanding - Published: 2026-10-05 · Authors: Dheeraj Varghese, Anna Vettoruzzo, Walter Simoncini, Michelle Lorena Acevedo Callejas, Mohammad Mahdi Derakhshani, Kristof Meding, Joaquin Vanschoren, Cees G. M. Snoek ### Plover: Steering GUI Agents through Plan-Centric Interaction Plover exposes GUI-agent plans and replanning as persistent artifacts that users can inspect and edit during execution. Its planner–executor design supports localized correction through plan edits, language guidance and screenshot-grounded interventions, studied through failure repair and workflow scenarios. - https://awesomecua.com/resources/arxiv-2607-15193/ · source: https://arxiv.org/abs/2607.15193v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: planning, human-in-the-loop, observability, workflow - Published: 2026-07-16 · Authors: Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo, Jorge Piazentin Ono, Liu Ren, Dongyu Liu ### PLOW: A Collaborative Task Learning Agent Describes PLOW, which learns executable web-task models from a session combining demonstrations, natural-language explanations, and dialogue. Browser instrumentation connects user actions with DOM structure to support task generalization. - https://awesomecua.com/resources/url-628b5251757bb3888e89/ · source: https://aaai.org/Papers/AAAI/2007/AAAI07-240.pdf - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, human-in-the-loop, planning - Published: 2007 · Authors: James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, William Taysom ### PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents Introduces PlugMem, a knowledge-graph memory module for general LLM agents, evaluated on web-agent tasks alongside conversational question answering and knowledge retrieval. - https://awesomecua.com/resources/arxiv-2603-03296/ · source: https://arxiv.org/abs/2603.03296v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, evaluation - Published: 2026-02-06 · Authors: Ke Yang, Zixi Chen, Xuan He, Jize Jiang, Michel Galley, Chenglong Wang, Jianfeng Gao, Jiawei Han, ChengXiang Zhai ### PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding POINTERBENCH is a diagnostic GUI grounding suite covering spreadsheet sub-elements, document characters and targets across professional applications. - https://awesomecua.com/resources/arxiv-2610-31274/ · source: https://arxiv.org/abs/2610.31274v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, grounding, screen-understanding - Published: 2026-10-22 · Authors: Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher, Maximilian Schilling ### POINTS-GUI-G: GUI-Grounding Journey Studies GUI-grounding development from a weakly grounded base model through data preparation, visual training, and verifiable rewards. - https://awesomecua.com/resources/arxiv-2602-06391/ · source: https://arxiv.org/abs/2602.06391v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, reinforcement-learning - Published: 2026-02-06 · Authors: Zhongyin Zhao, Yuan Liu, Yikun Liu, Haicheng Wang, Le Tian, Xiao Zhou, Yangxiu You, Zilin Yu, Yang Yu, Jie Zhou ### Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents Demonstrates clean-text visual backdoor attacks (VIBMA) that compromise fine-tuned VLM mobile agents through visual triggers while keeping text instructions intact. - https://awesomecua.com/resources/arxiv-2506-13205/ · source: https://arxiv.org/abs/2506.13205v6 - Paper · reviewed 2026-10-18 · environments: android · topics: safety-security, mobile-navigation, robustness - Published: 2025-06-16 · Authors: Xuan Wang, Siyuan Liang, Zhe Liu, Yi Yu, Aishan Liu, Yuliang Lu, Xitong Gao, Ee-Chien Chang ### Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agents Studies persistent web-agent memory contamination introduced through environmental observations instead of direct memory access. Experiments on WebArena and VisualWebArena examine cross-session effects and how interaction failures alter susceptibility. - https://awesomecua.com/resources/arxiv-2604-02623/ · source: https://arxiv.org/abs/2604.02623v2 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, memory, web-navigation - Published: 2026-04-03 · Authors: Wei Zou, Mingwen Dong, Miguel Romero Calvo, Shuaichen Chang, Jiang Guo, Dongkyu Lee, Xing Niu, Xiaofei Ma, Yanjun Qi, Jiarong Jiang ### PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction Introduces PolySkill, which separates a web-agent skill's goal from its execution so agents can learn and reuse skills across websites. - https://awesomecua.com/resources/arxiv-2510-15863/ · source: https://arxiv.org/abs/2510.15863v2 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, workflow, web-navigation - Published: 2025-10-17 · Authors: Simon Yu, Gang Li, Weiyan Shi, Peng Qi ### Ponder & Press: Advancing Visual GUI Agent towards General Computer Control Separates interpretation of user instructions from precise GUI localization using general-purpose and specialized visual-language components. - https://awesomecua.com/resources/arxiv-2412-01268/ · source: https://arxiv.org/abs/2412.01268v1 - Paper · reviewed 2026-10-16 · environments: cross-platform, web, desktop · topics: grounding, planning - Published: 2024-12-02 · Authors: Yiqin Wang, Haoji Zhang, Jingqi Tian, Yansong Tang ### PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks Introduces PPT-Eval, a benchmark of PowerPoint creation and editing tasks for computer-use agents, with rubrics that score partial progress and presentation quality. - https://awesomecua.com/resources/arxiv-2606-31154/ · source: https://arxiv.org/abs/2606.31154v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, workflow - Published: 2026-06-30 · Authors: Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar, Firoz Shaik, Shubhang Desai, Thong Q. Nguyen, Muhammad Taqi Raza, Vishal Chowdhary, Graham Neubig ### PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion Evaluates multi-turn PowerPoint creation and editing, scoring the resulting presentation files rather than requiring a fixed API sequence. - https://awesomecua.com/resources/arxiv-2311-01767/ · source: https://arxiv.org/abs/2311.01767v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: evaluation, tool-integration - Published: 2023-11-03 · Authors: Yiduo Guo, Zekai Zhang, Yaobo Liang, Dongyan Zhao, Nan Duan ### PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion Benchmarks robustness of API-based PowerPoint completion under sentence-level, semantic, and multilingual instruction perturbations and changes to the available API set. - https://awesomecua.com/resources/arxiv-2403-03788/ · source: https://arxiv.org/abs/2403.03788v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, robustness, workflow - Published: 2024-03-06 · Authors: Zekai Zhang, Yiduo Guo, Yaobo Liang, Dongyan Zhao, Nan Duan ### Practical, Automated Scenario-based Mobile App Testing ScenTest builds scenario-specific event knowledge graphs from crowdsourced test reports containing GUI and action information. It uses the current app state and testing context to generate scenario tests across applications. - https://awesomecua.com/resources/arxiv-2406-08340/ · source: https://arxiv.org/abs/2406.08340v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, cross-app, screen-understanding, evaluation - Published: 2024-06-12 · Authors: Shengcheng Yu, Chunrong Fang, Mingzhe Du, Zimin Ding, Zhenyu Chen, Zhendong Su ### PreAct: Computer-Using Agents that Get Faster on Repeated Tasks PreAct compiles successful screen-interaction runs into guarded state-machine programs for repeated tasks. It validates programs from clean states before storage and returns control to the agent when replayed screen expectations fail, with tests across mobile, desktop and web. - https://awesomecua.com/resources/arxiv-2606-17929/ · source: https://arxiv.org/abs/2606.17929v1 - Paper · reviewed 2026-10-18 · environments: cross-platform · topics: recording, memory, efficiency, automation - Published: 2026-06-16 · Authors: Bojie Li ### PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors Studies iterative cursor grounding in dense code-editor interfaces, using visual feedback from earlier attempts to correct displacement. PrecisionCUA evaluates this closed-loop process across multiple model families on coding-interface tasks. - https://awesomecua.com/resources/arxiv-2604-13019/ · source: https://arxiv.org/abs/2604.13019v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, robustness, screen-understanding - Published: 2026-04-14 · Authors: Himangi Mittal, Gaurav Mittal, Nelson Daniel Troncoso, Yu Hu ### PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval Presents PrecogUI, which retrieves prior anomaly and success patterns and predicts symbolic UI transitions before choosing actions. It also introduces InterfereBench, generated for evaluating long GUI workflows with disruptive events and recovery requirements. - https://awesomecua.com/resources/arxiv-2609-36923/ · source: https://arxiv.org/abs/2609.36923v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, world-models, robustness, long-horizon - Published: 2026-09-29 · Authors: Bin Kang, Jiarui Ouyang, Li Jiang, Bin Chen, Zhuotao Tian ### Preference Redirection via Attention Concentration: An Attack on Computer Use Agents Introduces PRAC, an adversarial-patch attack that redirects a computer-use agent's visual attention to steer product selection on an online shopping platform. - https://awesomecua.com/resources/arxiv-2604-08005/ · source: https://arxiv.org/abs/2604.08005v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, screen-understanding - Published: 2026-04-09 · Authors: Dominik Seip, Matthias Hein ### PreView: Enhancing GUI Grounding with Preprocessed Views and Cues Introduces PreView, a screenshot preprocessor that adds cropped views and candidate cues to help existing GUI grounding models identify controls for clicks. - https://awesomecua.com/resources/arxiv-2610-33645/ · source: https://arxiv.org/abs/2610.33645v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, screen-understanding - Published: 2026-10-24 · Authors: Wenjun Hou, Kaishuai Xu ### PriMobiBench: Characterizing Visual Privacy Leakage in VLM-Driven Mobile GUI Agents PriMobiBench evaluates direct privacy leakage and user profiling from mobile GUI screenshot trajectories, supported by the MobiLeak trace dataset. It also tests masking sensitive, task-irrelevant interface elements before cloud processing and measures the resulting privacy and utility trade-off. - https://awesomecua.com/resources/arxiv-2609-13873/ · source: https://arxiv.org/abs/2609.13873v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, safety-security, screen-understanding, mobile-navigation - Published: 2026-09-12 · Authors: Qihang Cen, Tianshuo Cong, Da Song, Xinlei He, Jiaxing Song, Ke Xu, Qi Li ### Prismata: Confining Cross-Site Prompt Injection in Web Agents Introduces Prismata, a defense for web agents that assigns trust labels to page content and enforces them by redacting content and restricting agent actions. - https://awesomecua.com/resources/arxiv-2607-08147/ · source: https://arxiv.org/abs/2607.08147v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation - Published: 2026-07-09 · Authors: Corban Villa, Alp Eren Ozdarendeli, Sijun Tan, Raluca Ada Popa ### Privacy Practices of Browser Agents Presents a privacy evaluation framework for browser agents covering component weaknesses, website defenses, tracking, privacy-related prompts and personal-data leakage. The framework is applied to deployed browser-agent tools. - https://awesomecua.com/resources/arxiv-2512-07725/ · source: https://arxiv.org/abs/2512.07725v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2025-12-08 · Authors: Alisha Ukani, Hamed Haddadi, Ali Shahin Shamsabadi, Peter Snyder ### PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI Agents Introduces PrivacyGuard, which sanitizes source-linked personal information and maintains locally resolvable references across mobile-agent turns to reduce exposure to remote planners. - https://awesomecua.com/resources/arxiv-2610-16778/ · source: https://arxiv.org/abs/2610.16778v1 - Paper · reviewed 2026-10-17 · environments: android · topics: safety-security, mobile-navigation, evaluation - Published: 2026-10-12 · Authors: Zian Yuan, Wei Li ### PrivWeb: Unobtrusive and Content-aware Privacy Protection For Web Agents Introduces PrivWeb, which uses a local language model to anonymize interface information according to user preferences. Adaptive notifications and selective pauses give users control over sensitive information access during web-agent tasks. - https://awesomecua.com/resources/arxiv-2509-11939/ · source: https://arxiv.org/abs/2509.11939v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, human-in-the-loop, web-navigation - Published: 2025-09-15 · Authors: Shuning Zhang, Yutong Jiang, Rongjun Ma, Yuting Yang, Mingyao Xu, Zhixin Huang, Xin Yi, Hewu Li ### PRO-CUA: Process-Reward Optimization for Computer Use Agents PRO-CUA trains computer-use agents with step-level RL, scoring candidate actions from live rollouts with a process reward model and evaluating on live web benchmarks. - https://awesomecua.com/resources/arxiv-2605-29119/ · source: https://arxiv.org/abs/2605.29119v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, reward-models, web-navigation - Published: 2026-05-27 · Authors: Yifei He, Rui Yang, Hao Bai, Tong Zhang, Han Zhao ### ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices ProactiveMobile benchmarks mobile agents on inferring latent user intent from device context and generating executable sequences from a pool of 63 APIs. - https://awesomecua.com/resources/arxiv-2602-21858/ · source: https://arxiv.org/abs/2602.21858v4 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, action-prediction - Published: 2026-02-25 · Authors: Dezhi Kong, Zhengzhao Feng, Qiliang Liang, Hao Wang, Haofei Sun, Changpeng Yang, Yang Li, Peng Zhou, Shuai Nie, Hongzhen Wang, Linfeng Zhou, Hao Jia, Jiaming Xu, Runyu Shi, Ying Huang ### ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent Studies proactive assistance in visual-analytics interfaces, using interaction logs to detect support needs and provide context-aware suggestions and guidance. - https://awesomecua.com/resources/arxiv-2507-18165/ · source: https://arxiv.org/abs/2507.18165v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: human-in-the-loop, planning - Published: 2025-07-24 · Authors: Yuheng Zhao, Xueli Shu, Liwen Fan, Lin Gao, Yu Zhang, Siming Chen ### ProAgent: From Robotic Process Automation to Agentic Process Automation Introduces ProAgent, an LLM-based agent that builds automation workflows from human instructions and coordinates specialized agents during workflow execution. - https://awesomecua.com/resources/arxiv-2311-10751/ · source: https://arxiv.org/abs/2311.10751v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: workflow, planning, automation - Published: 2023-11-02 · Authors: Yining Ye, Xin Cong, Shizuo Tian, Jiannan Cao, Hao Wang, Yujia Qin, Yaxi Lu, Heyang Yu, Huadong Wang, Yankai Lin, Zhiyuan Liu, Maosong Sun ### Probe to Act: Elevating Browser-Use Agent via Active Visual Probing Introduces Probe to Act, which lets browser agents request visual evidence linking DOM handles and screen regions before committing an operation. It retains selected interaction evidence as memory and studies both prompted use and distillation into open models. - https://awesomecua.com/resources/arxiv-2609-33646/ · source: https://arxiv.org/abs/2609.33646v1 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, memory, web-navigation - Published: 2026-09-27 · Authors: Keliang Li, Heng Wang, Chen Hu, Daxin Jiang, Hong Chang, Shiguang Shan ### ProBench: Benchmarking GUI Agents with Accurate Process Information ProBench extends mobile GUI evaluation with tasks whose correctness depends on the interaction process as well as the final state. Its ProcessProvider supplies process information for assessing agent behavior and analyzing failure modes. - https://awesomecua.com/resources/arxiv-2511-09157/ · source: https://arxiv.org/abs/2511.09157v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation, observability - Published: 2025-11-12 · Authors: Leyang Yang, Ziwei Wang, Xiaoxuan Tang, Sheng Zhou, Dajun Chen, Wei Jiang, Yong Li ### ProCUA-SFT Technical Report ProCUA-SFT generates tasks and synthetic trajectories on live desktops seeded with document content, checking task preconditions before rollout. The resulting step-prefix examples are formatted to match inference context and used for supervised GUI-agent fine-tuning. - https://awesomecua.com/resources/arxiv-2606-17321/ · source: https://arxiv.org/abs/2606.17321v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: data-generation, imitation-learning, evaluation - Published: 2026-06-15 · Authors: Jaehun Jung, Ximing Lu, Brandon Cui, Muhammad Khalifa, Shaokun Zhang, Hao Zhang, Jin Xu, Amala Sanjay Deshmukh, Karan Sapra, Andrew Tao, Yejin Choi, Jan Kautz, Mingjie Liu, Yi Dong ### Programming with Pixels: Can Computer-Use Agents do Software Engineering? Introduces a visual IDE environment and benchmark for software-engineering tasks, comparing screenshot-based control with access to editing and shell tools. - https://awesomecua.com/resources/arxiv-2502-18525/ · source: https://arxiv.org/abs/2502.18525v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: evaluation, tool-integration, automation - Published: 2025-02-24 · Authors: Pranjal Aggarwal, Sean Welleck ### ProgRM: Build Better GUI Agents with Progress Rewards ProgRM predicts per-step task completion progress to provide intermediate rewards during online GUI-agent training. It derives progress labels by using longest common subsequences to identify key trajectory steps, allowing useful partial progress to contribute even when a trajectory ultimately fails. - https://awesomecua.com/resources/arxiv-2505-18121/ · source: https://arxiv.org/abs/2505.18121v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reinforcement-learning, data-generation - Published: 2025-05-23 · Authors: Danyang Zhang, Situo Zhang, Ziyue Yang, Zichen Zhu, Zihan Zhao, Ruisheng Cao, Lu Chen, Kai Yu ### Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts Prompt2Task converts textual goals or procedures into smartphone UI automation using agents for intent interpretation, external-information handling, and execution. User feedback and accumulated task knowledge support subsequent task generation and operation. - https://awesomecua.com/resources/arxiv-2404-02475/ · source: https://arxiv.org/abs/2404.02475v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, human-in-the-loop, workflow - Published: 2024-04-03 · Authors: Tian Huang, Chun Yu, Weinan Shi, Zijian Peng, David Yang, Weiqi Sun, Yuanchun Shi ### Prompting Is All You Need: Automated Android Bug Replay with Large Language Models Introduces AdbGPT, a prompting-based approach that uses LLMs to replay app bugs from written reports, applying few-shot examples and reasoning without model training. - https://awesomecua.com/resources/arxiv-2306-01987/ · source: https://arxiv.org/abs/2306.01987v3 - Paper · reviewed 2026-11-06 · environments: android · topics: automation, training-free, mobile-navigation - Published: 2023-06-03 · Authors: Sidong Feng, Chunyang Chen ### Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents Combines autonomous task proposal, web interaction, and visual success evaluation to discover skills and improve browser-agent policies. - https://awesomecua.com/resources/arxiv-2412-13194/ · source: https://arxiv.org/abs/2412.13194v1 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, reinforcement-learning, web-navigation - Published: 2024-12-17 · Authors: Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, Erran Li ### ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration ProRe evaluates GUI task outcomes by having a reasoner assign targeted environment-probing tasks to specialist actors. The additional observations support reward judgments, with separate evaluation of judge accuracy and downstream agent task success. - https://awesomecua.com/resources/arxiv-2509-21823/ · source: https://arxiv.org/abs/2509.21823v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reward-models, evaluation, planning - Published: 2025-09-26 · Authors: Gaole Dai, Shiqi Jiang, Ting Cao, Yuqing Yang, Yuanchun Li, Rui Tan, Mo Li, Lili Qiu ### ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environments ProSoftArena evaluates 436 professional-software tasks across six disciplines and 13 applications in a Windows 11 virtual-machine environment. Its capability hierarchy, execution-based checks, and human-in-the-loop evaluation cover precise GUI operations and multi-software workflows. - https://awesomecua.com/resources/arxiv-2601-02399/ · source: https://arxiv.org/abs/2601.02399v1 - Paper · reviewed 2026-10-17 · environments: windows, desktop · topics: evaluation, industry, science, cross-app - Published: 2025-12-30 · Authors: Jiaxin Ai, Yukang Feng, Fanrui Zhang, Jianwen Sun, Zizhen Li, Chuanhao Li, Yifan Chang, Wenxiao Wu, Ruoxi Wang, Mingliang Zhai, Kaipeng Zhang ### Prune4Web: DOM Tree Pruning Programming for Web Agent Introduces Prune4Web, where generated scoring programs filter DOM elements according to decomposed subtasks before grounding. An annotation pipeline and two-turn training procedure coordinate the planner, programmatic filter and grounder. - https://awesomecua.com/resources/arxiv-2511-21398/ · source: https://arxiv.org/abs/2511.21398v1 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, screen-understanding, efficiency - Published: 2025-11-26 · Authors: Jiayuan Zhang, Kaiquan Chen, Zhihao Lu, Enshen Zhou, Qian Yu, Jing Zhang ### PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent PSPA-Bench evaluates personalized smartphone GUI behavior using user-specific instructions and a process-aware assessment of interaction traces. Its comparison of GUI agents diagnoses weaknesses in adapting workflows and preferences rather than measuring generic task completion alone. - https://awesomecua.com/resources/arxiv-2603-29318/ · source: https://arxiv.org/abs/2603.29318v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, mobile-navigation - Published: 2026-03-31 · Authors: Hongyi Nie, Xunyuan Liu, Yudong Bai, Yaqing Wang, Yang Liu, Quanming Yao, Zhen Wang ### Qwen-CUA: Native Computer Use for (almost) Everything Qwen-CUA operates through screenshots and keyboard/mouse events while compressing older visual history into reusable context blocks. Its report combines verifiable task generation, trajectory-based training and separate browser and Bash-augmented studies. - https://awesomecua.com/resources/arxiv-2608-02352/ · source: https://arxiv.org/abs/2608.02352v1 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: reinforcement-learning, imitation-learning, data-generation, interaction-history, long-horizon, evaluation - Published: 2026-08-03 · Authors: Dunjie Lu, Shuai Bai, Tianyi Bai, Sicheng Fan, Chang Gao, Jian Guan, Feng Hu, Mianqiu Huang, Xingyang Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Ning Li, Dayiheng Liu, Shixuan Liu, Zheng Liu, Que Shen, Bowen Wang, Junli Wang, Chencan Wu, Rui Xie, Tianbao Xie, Zhihui Xie, Haiyang Xu, An Yang, Tao Yu, Wenzhen Yuan, Xi Zhang, Zhenru Zhang, Mingkang Zhu, Zhaoqing Zhu, Yizhong Cao, Kai Dang, Binyuan Hui, Kaixin Li, Junyang Lin, Haiquan Wang, Zekun Wang, Yiheng Xu, Fan Yan, Mengqi Yuan, Danyang Zhang, Jiajun Zhang, Zhipeng Zhang, Fan Zhou, Fan Zhou ### Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agents Trains a mobile planner with supervised trajectories and competence-adaptive RL, then couples typed tool execution with memory, skills and offline model-harness refinement. - https://awesomecua.com/resources/arxiv-2609-29892/ · source: https://arxiv.org/abs/2609.29892v1 - Paper · reviewed 2026-10-20 · environments: unspecified · topics: planning, data-generation, reinforcement-learning, memory, mobile-navigation, efficiency - Published: 2026-09-24 · Authors: Tingyu Qu, Weigao Sun, Yuecheng Liu, Yucheng Zhao, Yi Zhu, Yifeng Ding, Qiyi Wang, Sihan Cao, Pengkun Jiao, Hanlei Xie, Xiongwei Wu, Qichao Wang, Haodong Zhang, Jiajun Liu, Yuhao Wang, Yuqing Xie, Junpeng Zhao, Long Chen, Ming Ma, Sihan Yang, Ziwang Zhao, Yanhao Jia, Liangquan Gong, Feida Zhu, Yiran Zhong, Steven Hoi - Licenses: document CC-BY-4.0 ### Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents Describes a foundation GUI agent with combined GUI and command-line actions, large-scale interaction infrastructure, and iterative environment and task generation. - https://awesomecua.com/resources/arxiv-2607-28227/ · source: https://arxiv.org/abs/2607.28227v1 - Paper · reviewed 2026-10-16 · environments: cross-platform, web, desktop · topics: reinforcement-learning, long-horizon, tool-integration - Published: 2026-07-30 · Authors: Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia, Gongjie Zhang, Jianan Zhang, Long Li, Long Chen, Lei Wang, Gaole Dai, Pengxiang Li, Liangyu Chen, Yue Wang, Steven Hoi ### Qwen2.5-VL Technical Report Reports Qwen2.5-VL, a vision-language model that can act as a visual agent for tasks involving computer and mobile-device operation, alongside broader visual tasks. - https://awesomecua.com/resources/arxiv-2502-13923/ · source: https://arxiv.org/abs/2502.13923v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, tool-integration - Published: 2025-02-19 · Authors: Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, Junyang Lin ### R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding R-VLM localizes GUI elements through zoomed-in region proposals and an intersection-over-union-aware training objective. It evaluates the grounding model on ScreenSpot and AgentStudio and studies its effect on navigation in AITW and Mind2Web. - https://awesomecua.com/resources/arxiv-2507-05673/ · source: https://arxiv.org/abs/2507.05673v1 - Paper · reviewed 2026-10-17 · environments: web, cross-platform · topics: grounding, screen-understanding, web-navigation - Published: 2025-07-08 · Authors: Joonhyung Park, Peng Tang, Sagnik Das, Srikar Appalaraju, Kunwar Yashraj Singh, R. Manmatha, Shabnam Ghadar ### R-WoM: Retrieval-augmented World Model For Computer-use Agents R-WoM evaluates how well world models predict nearby GUI states, align complete procedures, and recognize milestone transitions. It retrieves external tutorials to guide simulated future interactions, with experiments on subsets of OSWorld and WebArena. - https://awesomecua.com/resources/arxiv-2510-11892/ · source: https://arxiv.org/abs/2510.11892v2 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: world-models, memory, planning, evaluation - Published: 2025-10-13 · Authors: Kai Mei, Jiang Guo, Shuaichen Chang, Mingwen Dong, Dongkyu Lee, Xing Niu, Jiarong Jiang ### R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory R2D2 gives web agents a replay-buffer map of visited pages and a reflection step for analyzing errors and revising navigation strategies. - https://awesomecua.com/resources/arxiv-2501-12485/ · source: https://arxiv.org/abs/2501.12485v3 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, self-reflection, web-navigation, interaction-history - Published: 2025-01-21 · Authors: Tenghao Huang, Kinjal Basu, Ibrahim Abdelaziz, Pavan Kapanipathi, Jonathan May, Muhao Chen ### RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection RankGround trains a lightweight reranker to select a promising screenshot crop before one VLM localization call. Ranking supervision uses containment and boundary-aware augmentation, with pointwise and listwise training stages. - https://awesomecua.com/resources/arxiv-2609-18690/ · source: https://arxiv.org/abs/2609.18690v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, efficiency, data-generation, screen-understanding - Published: 2026-09-16 · Authors: Liyang Fan, Xinping Bi, Yitai Li, Shuaimin Li, Hui Li, Min Yang ### ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation Splits mobile tasks into reaching the required page and operating on it, using subtask data and preferred GUI trajectories for training. - https://awesomecua.com/resources/arxiv-2502-02955/ · source: https://arxiv.org/abs/2502.02955v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, planning, data-generation - Published: 2025-02-05 · Authors: Qinzhuo Wu, Wei Liu, Jian Luan, Bin Wang ### ReAct: Synergizing Reasoning and Acting in Language Models Introduces ReAct, a method that interleaves reasoning traces and actions for language-model agents; WebShop is one of its interactive decision-making evaluations. - https://awesomecua.com/resources/arxiv-2210-03629/ · source: https://arxiv.org/abs/2210.03629v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: planning, training-free, evaluation - Published: 2022-10-06 · Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao ### Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding Introduces Screen Point-and-Read and a hierarchical layout approach that interprets screen regions indicated by a user-selected point. - https://awesomecua.com/resources/arxiv-2406-19263/ · source: https://arxiv.org/abs/2406.19263v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: screen-understanding, grounding - Published: 2024-06-27 · Authors: Yue Fan, Lei Ding, Ching-Chen Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jie Yang, Yi Zhang, Xin Eric Wang ### Read More, Think More: Revisiting Observation Reduction for Web Agents Studies HTML and accessibility-tree observations for web agents, relating their usefulness to model capability and thinking budget, and proposes diff-based observation history to save tokens. - https://awesomecua.com/resources/arxiv-2604-01535/ · source: https://arxiv.org/abs/2604.01535v1 - Paper · reviewed 2026-11-06 · environments: web · topics: screen-understanding, interaction-history, efficiency, web-navigation - Published: 2026-04-02 · Authors: Masafumi Enomoto, Ryoma Obara, Haochen Zhang, Masafumi Oyamada ### Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops Pairs factual probes with next-action choices on frozen agent traces, including Mind2Web, to diagnose reading-decision gaps without closed-loop execution. - https://awesomecua.com/resources/arxiv-2610-18579/ · source: https://arxiv.org/abs/2610.18579v1 - Paper · reviewed 2026-10-20 · environments: web · topics: evaluation, web-navigation, interaction-history - Published: 2026-10-13 · Authors: Yubo Li, Ramayya Krishnan, Rema Padman - Licenses: document CC-BY-4.0 ### Real-Time Procedural Learning From Experience for AI Agents PRAXIS stores action outcomes and retrieves past state-action-result examples to guide agent decisions, with evaluation on the REAL web browsing benchmark. - https://awesomecua.com/resources/arxiv-2511-22074/ · source: https://arxiv.org/abs/2511.22074v2 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, interaction-history, web-navigation - Published: 2025-11-27 · Authors: Dasheng Bi, Yubin Hu, Mohammed N. Nasir ### REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites Introduces REAL, a benchmark of deterministic website replicas with 112 tasks covering information retrieval and state-changing browser actions. - https://awesomecua.com/resources/arxiv-2504-11543/ · source: https://arxiv.org/abs/2504.11543v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, sandbox, web-navigation - Published: 2025-04-15 · Authors: Divyansh Garg, Shaun VanWeelden, Diego Caples, Andis Draguns, Nikil Ravi, Pranav Putta, Naman Garg, Tomas Abraham, Michael Lara, Federico Lopez, James Liu, Atharva Gundawar, Prannay Hebbar, Youngchul Joo, Jindong Gu, Charles London, Christian Schroeder de Witt, Sumeet Motwani ### RealGUINoise: An Interactive Cross-Platform Benchmark for GUI Agent Robustness under Real-World Interface Noise Evaluates interactive GUI agents under varied realistic interface noise, comparing task completion, safety, and behavior with clean-environment runs. - https://awesomecua.com/resources/arxiv-2609-38184/ · source: https://arxiv.org/abs/2609.38184v1 - Paper · reviewed 2026-10-16 · environments: web, desktop, cross-platform · topics: robustness, evaluation, safety-security - Published: 2026-08-07 · Authors: Yongjiang Wu, Junyuan Zhang, Ada Chen, Kuiyi Gao, Wenxuan Wang ### ReALM: Reference Resolution As Language Modeling Frames conversational and on-screen reference resolution as language modeling, allowing textual representations of screen entities to support user-reference interpretation. Evaluation separately examines on-screen references against an existing resolver and language-model baselines. - https://awesomecua.com/resources/arxiv-2403-20329/ · source: https://arxiv.org/abs/2403.20329v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding - Published: 2024-03-29 · Authors: Joel Ruben Antony Moniz, Soundarya Krishnan, Melis Ozyildirim, Prathamesh Saraf, Halim Cagri Ates, Yuan Zhang, Hong Yu ### RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users RealWebAssist evaluates web assistance through sequences of instructions collected from real users across multiple websites. It tests intent interpretation, user-specific routines, long-term context, and visual grounding when instructions are ambiguous or evolve during interaction. - https://awesomecua.com/resources/arxiv-2504-10445/ · source: https://arxiv.org/abs/2504.10445v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, long-horizon, human-in-the-loop, grounding - Published: 2025-04-14 · Authors: Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, Tianmin Shu ### Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach Introduces UXBench for diagnosing usability issues from UI screenshots and UI-UX, a reinforcement-trained model for reasoning about those interface problems. - https://awesomecua.com/resources/arxiv-2606-13192/ · source: https://arxiv.org/abs/2606.13192v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: ui-design, screen-understanding, evaluation, reinforcement-learning - Published: 2026-06-11 · Authors: Ruichao Mao, Zhou Fang, Teng Guo, Hao Yang, Yaping Li, Shaohua Peng, Maji Huang, Xiaoyu Lin, Shuoyang Liu, Xuepeng Li, Yuyu Zhang, Hai Rao ### ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory Distills generalizable reasoning strategies from both successful and failed agent trajectories into a persistent memory bank for test-time retrieval. - https://awesomecua.com/resources/arxiv-2509-25140/ · source: https://arxiv.org/abs/2509.25140v2 - Paper · reviewed 2026-10-18 · environments: web · topics: memory, self-reflection, web-navigation - Published: 2025-09-29 · Authors: Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, Tomas Pfister ### REBASE: Device-Cloud Experience Coherence for GUI Agents Across App Updates Introduces REBASE, a device–cloud protocol for repairing cached GUI experience after app updates. It combines cloud replay, device-side step verification, and progressively richer failure reports to produce repairs tied to application versions. - https://awesomecua.com/resources/arxiv-2609-32130/ · source: https://arxiv.org/abs/2609.32130v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, robustness, efficiency, mobile-navigation - Published: 2026-09-26 · Authors: Beining Wu, Jun Huang, Yanxiao Zhao ### Recon-Act: A Self-Evolving Multi-Agent Browser-Use System via Web Reconnaissance, Tool Generation, and Task Execution Introduces Recon-Act, a browser-use framework that compares failed and successful web trajectories to create reusable tools for planning and executing long-horizon website tasks. - https://awesomecua.com/resources/arxiv-2509-21072/ · source: https://arxiv.org/abs/2509.21072v1 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, long-horizon, planning, tool-integration - Published: 2025-09-25 · Authors: Kaiwen He, Zhiwei Wang, Chenyi Zhuang, Jinjie Gu ### Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents GUI-RobustEval measures how GUI agents recover from their own errors, while RoTS synthesizes recovery trajectories for training on varied failure modes. - https://awesomecua.com/resources/arxiv-2605-29447/ · source: https://arxiv.org/abs/2605.29447v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: robustness, evaluation, data-generation - Published: 2026-05-28 · Authors: Tianpeng Bu, Xin Liu, Qihua Chen, Hao Jiang, Shurui Li, Hongtao Duan, Lu Jiang, Lulu Hu, Bin Yang, Minying Zhang ### RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents RecreationWorld gives hybrid agents running reference applications to explore and recreate using native GUI controls and coding tools. Its multi-platform benchmark checks interaction-dependent behavior with programmatic and visual assertions, while trajectory-training transfer is evaluated separately. - https://awesomecua.com/resources/arxiv-2609-22000/ · source: https://arxiv.org/abs/2609.22000v2 - Paper · reviewed 2026-10-17 · environments: desktop, linux, macos, windows, android, web · topics: evaluation, data-generation, tool-integration, workflow - Published: 2026-09-18 · Authors: Shuai Bai, Jiayong Deng, Sicheng Fan, Yikun Fu, Chang Gao, Xuhao Hu, Mianqiu Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Keliang Li, Ning Li, Wanli Li, Dayiheng Liu, Dunjie Lu, Changwei Luo, Que Shen, Zheyuan Wang, Zijian Wang, Jie Wu, Gao Wu, Zhihui Xie, Rui Xie, Haiyang Xu, An Yang, Jiakang Yuan, Yanming Zhang, Jiajun Zhang, Xi Zhang, Zhenru Zhang, Zhuo Zhen, Mingkang Zhu, Bowen Zhou ### Recurrent Latent Visual Search for GUI Grounding Introduces ReLaViS, which performs recurrent visual search inside a grounding model by repeatedly querying screenshot tokens and feeding the resulting spatial evidence back into the model. Coarse-to-fine supervision is constructed from existing interface-element annotations. - https://awesomecua.com/resources/arxiv-2610-05185/ · source: https://arxiv.org/abs/2610.05185v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, efficiency - Published: 2026-10-04 · Authors: Kaiyu Wu, Beichen Zheng, Weiyao Huang, Keze Wang ### RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments RedTeamCUA combines a virtual-machine OS environment with containerized websites for controlled prompt-injection tests spanning both interfaces. Its RTC-Bench contains 864 examples and can initialize agents at injection points to separate attack susceptibility from navigation failures. - https://awesomecua.com/resources/arxiv-2505-21936/ · source: https://arxiv.org/abs/2505.21936v5 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: safety-security, evaluation, sandbox - Published: 2025-05-28 · Authors: Zeyi Liao, Jaylen Jones, Linxi Jiang, Yuting Ning, Eric Fosler-Lussier, Yu Su, Zhiqiang Lin, Huan Sun ### Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents EvoSkill-GUI revises reusable skill files from execution feedback through an executor–critic loop. It studies how training-free updates to localization, recovery and procedure information help subsequent mobile and desktop tasks. - https://awesomecua.com/resources/arxiv-2609-17653/ · source: https://arxiv.org/abs/2609.17653v1 - Paper · reviewed 2026-10-17 · environments: android, desktop · topics: memory, training-free, self-reflection, robustness, long-horizon, workflow - Published: 2026-09-15 · Authors: Bofan Chen, Boxuan Zhang, Fei Tang, Zhengxi Lu, Yong Du, Tongbo Chen, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen ### Reflection with Action-Induced Visual Differences for Desktop GUI Agents Introduces Evidence-First Reflection, which separates extraction of action-induced screen changes from judgment of whether an operation achieved its objective. It filters visual differences into relevant evidence before a desktop agent’s reflector decides how execution should continue. - https://awesomecua.com/resources/arxiv-2608-24015/ · source: https://arxiv.org/abs/2608.24015v1 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: self-reflection, screen-understanding, robustness - Published: 2026-08-25 · Authors: Yijie Ma, Chaoyue Niu, Fan Wu, Guihai Chen ### Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents Introduces BrowserART and studies whether chat-model refusal behavior transfers to browser agents carrying out potentially harmful instructions. - https://awesomecua.com/resources/arxiv-2410-13886/ · source: https://arxiv.org/abs/2410.13886v2 - Paper · reviewed 2026-10-16 · environments: web · topics: safety-security, evaluation - Published: 2024-10-11 · Authors: Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx, Shuyan Zhou, Matt Fredrikson, Summer Yue, Zifan Wang ### Region4Web: Rethinking Observation Space Granularity for Web Agents Reorganizes webpage accessibility trees into functional regions through hierarchical decomposition and semantic abstraction. PageDigest supplies a persistent region-level observation to web agents, evaluated on WebArena. - https://awesomecua.com/resources/arxiv-2605-07134/ · source: https://arxiv.org/abs/2605.07134v1 - Paper · reviewed 2026-10-17 · environments: web · topics: screen-understanding, web-navigation, efficiency - Published: 2026-05-08 · Authors: Donguk Kwon, Dongha Lee ### ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search Combines self-generated localization reasoning, spatial consistency checks, and inference-time search to improve data-efficient web grounding. - https://awesomecua.com/resources/arxiv-2505-15259/ · source: https://arxiv.org/abs/2505.15259v2 - Paper · reviewed 2026-10-16 · environments: web · topics: grounding, reinforcement-learning - Published: 2025-05-21 · Authors: Hyunseok Lee, Jeonghoon Kim, Beomjun Kim, Jihoon Tack, Chansong Jo, Jaehong Lee, Cheonbok Park, Sookyo In, Jinwoo Shin, Kang Min Yoo ### ReInAgent: A Context-Aware GUI Agent Enabling Human-in-the-Loop Mobile Task Navigation Introduces ReInAgent, coordinating information management, planning, and reflection around a shared memory for mobile tasks. It proactively asks users for information and checks consistency when instructions are ambiguous or change during execution. - https://awesomecua.com/resources/arxiv-2510-07988/ · source: https://arxiv.org/abs/2510.07988v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, memory, mobile-navigation - Published: 2025-10-09 · Authors: Haitao Jia, Ming He, Zimo Yin, Likang Wu, Jianping Fan, Jitao Sang ### Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API Trains a screenshot-based instruction-grounding model to predict target coordinates, using reinforcement learning to improve spatial decoding. - https://awesomecua.com/resources/arxiv-2310-04716/ · source: https://arxiv.org/abs/2310.04716v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, reinforcement-learning - Published: 2023-10-07 · Authors: Zhizheng Zhang, Wenxuan Xie, Xiaoyi Zhang, Yan Lu ### Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation The framework uses a vision-language judge’s final-screen assessment as a terminal reward for GUI reinforcement learning. It models evaluator noise and corrects the reward estimator used by PPO across desktop benchmarks. - https://awesomecua.com/resources/arxiv-2606-24515/ · source: https://arxiv.org/abs/2606.24515v1 - Paper · reviewed 2026-10-17 · environments: desktop, macos, windows · topics: reinforcement-learning, reward-models, evaluation, robustness - Published: 2026-06-23 · Authors: Marta Sumyk, Oleksandr Kosovan ### Reinforcement Learning for Long-Horizon Interactive LLM Agents LOOP trains interactive digital agents with reinforcement learning in stateful, multi-app environments where tasks are performed through API calls. - https://awesomecua.com/resources/arxiv-2502-01600/ · source: https://arxiv.org/abs/2502.01600v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, cross-app, workflow, efficiency - Published: 2025-02-03 · Authors: Kevin Chen, Marco Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, Philipp Krähenbühl ### Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration Uses workflows extracted from demonstrations to constrain reinforcement-learning exploration for interactive web tasks. - https://awesomecua.com/resources/arxiv-1802-08802/ · source: https://arxiv.org/abs/1802.08802v1 - Paper · reviewed 2026-10-16 · environments: web · topics: reinforcement-learning, workflow - Published: 2018-02-24 · Authors: Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, Percy Liang ### Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents Introduces ExpActivator, which matches the current GUI state to relevant historical states before supplying past actions as guidance. A separate activation rule decides whether a recurring user intent is warranted by the current time and situation or should be withheld. - https://awesomecua.com/resources/arxiv-2609-33304/ · source: https://arxiv.org/abs/2609.33304v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, planning, training-free - Published: 2026-09-27 · Authors: Fuyao Zhang, Xuan Wang, Zherui Li, Jiaming Zhang, Longtao Huang, Wei Yang Bryan Lim ### Reliable Weak-to-Strong Monitoring of LLM Agents Introduces CUA-SHADE-Arena and a red-teaming workflow for testing monitors that detect covert misbehavior in computer-use agent trajectories. - https://awesomecua.com/resources/arxiv-2508-19461/ · source: https://arxiv.org/abs/2508.19461v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, observability - Published: 2025-08-26 · Authors: Neil Kale, Chen Bo Calvin Zhang, Kevin Zhu, Ankit Aich, Paula Rodriguez, Scale Red Team, Christina Q. Knight, Zifan Wang ### ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation Introduces ReSAIL, an augmentation for iterative agent self-distillation that selects informative interaction steps; it also tests GUI-agent action prediction. - https://awesomecua.com/resources/arxiv-2609-39306/ · source: https://arxiv.org/abs/2609.39306v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: imitation-learning, action-prediction, interaction-history - Published: 2026-09-30 · Authors: Shengjie Jin, Hengbo Xu, Zelong Sun, YuJie Guo, Zhiwu Lu ### Responsible Task Automation: Empowering Large Language Models as Responsible Task Automators Presents ResponsibleTA, a coordinator-executor framework that checks command feasibility and task completion while using local memory for privacy-related controls. The paper compares prompted and learned implementations and evaluates them on UI task automation. - https://awesomecua.com/resources/arxiv-2306-01242/ · source: https://arxiv.org/abs/2306.01242v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, evaluation, automation - Published: 2023-06-02 · Authors: Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, Yan Lu ### ResTest: Targeted Coverage of Residual Not-Covered Code Using Large Language Models for Web GUI Testing Introduces ResTest, which uses coverage reports and a state transition graph to infer untested web functions, then directs a GUI agent to execute them. - https://awesomecua.com/resources/arxiv-2506-00520/ · source: https://arxiv.org/abs/2506.00520v2 - Paper · reviewed 2026-11-06 · environments: web · topics: automation, web-navigation, state-tracking, planning - Published: 2025-05-31 · Authors: Chenxu Liu, Zhiyu Gu, Guoquan Wu, Ying Zhang, Wei Yang, Jun Wei, Tao Xie ### ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization Introduces ReSum, which compresses interaction histories during long web searches, and ReSum-GRPO, which trains agents to reason across the resulting segments. - https://awesomecua.com/resources/arxiv-2509-13313/ · source: https://arxiv.org/abs/2509.13313v3 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, interaction-history, efficiency, reinforcement-learning - Published: 2025-09-16 · Authors: Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Xinmiao Yu, Dingchu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Minhao Cheng, Shuai Wang, Hong Cheng, Jingren Zhou ### Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs Examines inference-time scaling for local computer-use models by varying context, step budgets, decomposition, and parallel execution. OSWorld experiments track both task performance and changing failure modes, exposing cases where extra computation extends errors or adds coordination overhead. - https://awesomecua.com/resources/arxiv-2607-28573/ · source: https://arxiv.org/abs/2607.28573v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: efficiency, evaluation, robustness - Published: 2026-07-30 · Authors: Woongkyu Lee, Jungwook Choi ### Retrieval-augmented GUI Agents with Generative Guidelines RAG-GUI trains a small guidance adapter to filter and summarize retrieved web tutorials while keeping the acting agent fixed. Supervised and rejection-sampling stages prepare the adapter for inference-time assistance, tested on offline web/mobile tasks and interactive Android tasks. - https://awesomecua.com/resources/arxiv-2509-24183/ · source: https://arxiv.org/abs/2509.24183v1 - Paper · reviewed 2026-10-17 · environments: web, android · topics: memory, planning, imitation-learning, web-navigation, mobile-navigation - Published: 2025-09-29 · Authors: Ran Xu, Kaixin Ma, Wenhao Yu, Hongming Zhang, Joyce C. Ho, Carl Yang, Dong Yu ### Reuse, Rewrite, or Reject: Evidence-Constrained Trace Reuse for Repeated Mobile GUI Control Proposes graph-text templates that let mobile GUI agents reuse, adapt, or discard steps from earlier executions based on the current instruction and UI state. - https://awesomecua.com/resources/arxiv-2610-31712/ · source: https://arxiv.org/abs/2610.31712v1 - Paper · reviewed 2026-11-06 · environments: android, harmonyos · topics: mobile-navigation, interaction-history, efficiency - Published: 2026-10-22 · Authors: Bingyu Bai, Cheng Zhang, Erhu Feng ### ReuseDroid: A VLM-empowered Android UI Test Migrator Boosted by Active Feedback ReuseDroid migrates Android GUI tests between functionally similar apps using VLM agents that separate shared test intent from app-specific interaction flows. It introduces LinPro, a dataset of 578 migration tasks across 39 apps, to evaluate test transfer. - https://awesomecua.com/resources/arxiv-2504-02357/ · source: https://arxiv.org/abs/2504.02357v1 - Paper · reviewed 2026-10-17 · environments: android · topics: automation, evaluation, cross-app - Published: 2025-04-03 · Authors: Xiaolei Li, Jialun Cao, Yepang Liu, Shing-Chi Cheung, Hailong Wang ### ReUseIt: Synthesizing Reusable AI Agent Workflows for Web Automation Introduces ReUseIt, which synthesizes reusable web-agent workflows from successful and failed attempts and adds guards to detect and fix execution errors. - https://awesomecua.com/resources/arxiv-2510-14308/ · source: https://arxiv.org/abs/2510.14308v2 - Paper · reviewed 2026-11-06 · environments: web · topics: workflow, automation, state-tracking - Published: 2025-10-16 · Authors: Yimeng Liu, Misha Sra, Jeevana Priya Inala, Chenglong Wang ### ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction Trains GUI agents with a learned selector that removes repeated visual patches across consecutive screenshots while preserving spatial structure. ReVision evaluates this history representation on OSWorld, WebTailBench and AgentNetBench. - https://awesomecua.com/resources/arxiv-2605-11212/ · source: https://arxiv.org/abs/2605.11212v3 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: efficiency, interaction-history, screen-understanding - Published: 2026-05-11 · Authors: Amirhossein Abaskohi, Yuhang He, Peter West, Giuseppe Carenini, Pranit Chawla, Vibhav Vineet ### Revisiting Observation Reduction for Web Agents: Comprehensive Evaluation with a Lightweight Framework Introduces a lightweight framework for evaluating HTML reduction in web agents, using Minimal Failure Set coverage to assess pruning and optimize latency. - https://awesomecua.com/resources/arxiv-2605-29397/ · source: https://arxiv.org/abs/2605.29397v1 - Paper · reviewed 2026-11-06 · environments: web · topics: efficiency, evaluation, web-navigation - Published: 2026-05-28 · Authors: Masafumi Enomoto, Ryoma Obara, Haochen Zhang, Masafumi Oyamada ### Rico: A Mobile App Dataset for Building Data-Driven Design Applications Introduces Rico, collecting Android interface designs and interaction traces through crowdsourced use followed by automated exploration. The paper demonstrates UI-layout retrieval using learned representations and describes visual, textual, structural, and interaction data. - https://awesomecua.com/resources/url-b560b5d6abaab98483f6/ · source: https://doi.org/10.1145/3126594.3126651 - Paper · reviewed 2026-10-17 · environments: android · topics: data-generation, screen-understanding, ui-design - Published: 2017-10-20 · Authors: Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, Ranjitha Kumar ### Right Click, Wrong Box: Grading GUI Grounding by What a Click Does Proposes grading GUI grounding clicks by their executed outcomes and audits box-based scores on live web applications and ScreenSpot-v2 screenshots. - https://awesomecua.com/resources/arxiv-2610-36831/ · source: https://arxiv.org/abs/2610.36831v1 - Paper · reviewed 2026-11-06 · environments: web · topics: grounding, evaluation, web-navigation - Published: 2026-10-27 · Authors: Xiaokang Yuan, Binhui Yao, Luwei Yang, Ruimin Chen, Zeng Wang ### Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents Presents LGWM, a decoder-free GUI world model that predicts the next screen representation from an action and compares it with the observed state to verify transitions. Its paired diagnostic separates legitimate and hijacked transitions that lead to the same credential screen. - https://awesomecua.com/resources/arxiv-2610-11942/ · source: https://arxiv.org/abs/2610.11942v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: world-models, safety-security, evaluation - Published: 2026-10-08 · Authors: Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim ### Ringer: Web Automation by Demonstration Introduces Ringer, which converts demonstrated webpage interactions into scripts combining actions, target selectors, and waiting conditions. It evaluates replay robustness on tasks recorded from real websites. - https://awesomecua.com/resources/url-347925d0f440d6396080/ · source: https://doi.org/10.1145/2983990.2984020 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, recording, robustness - Published: 2016-11 · Authors: Shaon Barman, Sarah Chasins, Ras Bodik, Sumit Gulwani ### RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents Introduces RiOSWorld, a computer-use risk benchmark separating user-originated and environmental threats. It evaluates both risky-goal intent and goal completion across web and other computer applications. - https://awesomecua.com/resources/arxiv-2506-00618/ · source: https://arxiv.org/abs/2506.00618v3 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation - Published: 2025-05-31 · Authors: Jingyi Yang, Shuai Shao, Dongrui Liu, Jing Shao ### RISK: A Framework for GUI Agents in E-commerce Risk Management RISK combines browser-interaction training data, a benchmark, and reinforcement fine-tuning for e-commerce risk-management workflows. Its trajectories and rewards distinguish individual actions from multi-step stateful tasks involving deeply embedded web content. - https://awesomecua.com/resources/arxiv-2509-21982/ · source: https://arxiv.org/abs/2509.21982v2 - Paper · reviewed 2026-10-17 · environments: web · topics: industry, data-generation, evaluation, reinforcement-learning - Published: 2025-09-26 · Authors: Renqi Chen, Zeyin Tao, Jianming Guo, Jingzhe Zhu, Yiheng Peng, Qingqing Sun, Tianyi Zhang, Shuai Chen ### RiskChainBench: A Benchmark for Obfuscated Platform Message Restoration and Evidence-Grounded Web Investigation Introduces RiskChainBench, pairing obfuscated-message restoration with web-agent investigation of linked sites in labeled local web environments. - https://awesomecua.com/resources/arxiv-2609-16900/ · source: https://arxiv.org/abs/2609.16900v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, safety-security, sandbox - Published: 2026-09-15 · Authors: ZhuoXin Liu, Zhiming Ma, Ying Zhang, Mengzheng Yang, Yifan Wang, Zhengqi Huang, Yanhan Zhou, Zekun Lin, Jun Zhang, Shun Zhang, Yue Chen, Qiao Zhao, Peng Chen ### RiskWebWorld: A Realistic Interactive Benchmark for GUI Agents in E-commerce Risk Management Introduces RiskWebWorld, an interactive benchmark of e-commerce risk-management tasks for GUI agents on websites, with Gymnasium infrastructure for evaluation and agentic RL. - https://awesomecua.com/resources/arxiv-2604-13531/ · source: https://arxiv.org/abs/2604.13531v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, industry, reinforcement-learning, web-navigation - Published: 2026-04-15 · Authors: Renqi Chen, Zeyin Tao, Jianming Guo, Jing Wang, Zezhou Xu, Jingzhe Zhu, Qingqing Sun, Tianyi Zhang, Shuai Chen ### RMSWeb: Reflection, Failure-Mode Mining, and Salvage-DS for Web Agent Reinforcement Learning Introduces RMSWeb, a training recipe for web agents that uses reflection retries, failure-mode mining, and Salvage-DS reward and sampling methods for web actions. - https://awesomecua.com/resources/arxiv-2608-00335/ · source: https://arxiv.org/abs/2608.00335v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, data-generation, efficiency - Published: 2026-07-31 · Authors: Chengbo Liu, Lifang Zhou, Ruijie Yan, Pei Tan, Ao Sun, Haojun Huang, Guichun Hua, Sining Wei, Yining Chen, Yingying He, Yutao Xie ### Robust, Observable, and Evolvable Agentic Systems Engineering: A Principled Framework Validated via the Fairy GUI Agent Presents an agent-engineering framework combining runtime goal clarification, an observable component architecture and evolving execution memory. The Fairy mobile GUI agent evaluates these mechanisms on ambiguous tasks and through controlled and user studies. - https://awesomecua.com/resources/arxiv-2509-20729/ · source: https://arxiv.org/abs/2509.20729v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: observability, human-in-the-loop, memory, robustness - Published: 2025-09-25 · Authors: Jiazheng Sun, Ruimeng Yang, Xu Han, Jiayang Niu, Mingxuan Li, Te Yang, Yongyong Lu, Xin Peng ### Rousillon: Scraping Distributed Hierarchical Web Data Introduces Rousillon, which generalizes a demonstration of collecting one hierarchical data row into a browser-automation program that traverses related pages. The paper develops relation-selection and generalization methods and evaluates users building such scripts. - https://awesomecua.com/resources/url-97c0b1896525632434e2/ · source: https://doi.org/10.1145/3242587.3242661 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, recording, automation - Published: 2018 · Authors: Sarah E. Chasins, Maria Mueller, Rastislav Bodik ### Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents Measures the complementarity, cost and rerun variability of six browser observation modes on WebArena and VisualWebArena. Its routing experiments distinguish an oracle cost opportunity from policies that fail to robustly outperform a fixed observation mode. - https://awesomecua.com/resources/arxiv-2608-06171/ · source: https://arxiv.org/abs/2608.06171v1 - Paper · reviewed 2026-10-17 · environments: web · topics: screen-understanding, efficiency, evaluation, robustness - Published: 2026-08-06 · Authors: Jiaming Wei, Zekun Wu, Adriano Koshiyama, Maria Perez-Ortiz ### RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments Introduces RSIAgent, a training-free framework that explores unfamiliar digital interfaces, stores environment-specific action knowledge, and reuses that memory for later tasks. - https://awesomecua.com/resources/arxiv-2609-15364/ · source: https://arxiv.org/abs/2609.15364v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, training-free, planning - Published: 2026-09-14 · Authors: Sibo Zhu, Shicheng Fan, Xinyue Wang, Wenyi Wu, Kun Zhou, Biwei Huang ### RunAgent SuperBrowser: A Theory of Autonomous Web Navigation Grounded in Human Browsing Behaviour SUPERBROWSER combines screenshot-based target proposals with separate orchestration, planning and action roles. A bounded ledger and context-eviction process retain recent actions, task facts and checkpoints during browser execution. - https://awesomecua.com/resources/arxiv-2606-09399/ · source: https://arxiv.org/abs/2606.09399v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, grounding, planning, memory, efficiency - Published: 2026-06-08 · Authors: Radeen Mostafa, Sawradip Saha ### Running the Gauntlet: Hard Agentic Tasks Introduces GauntletBench, a web-based benchmark of vision-intensive tasks in five professional applications for assessing agent perception, 3D reasoning, and generalisation. - https://awesomecua.com/resources/arxiv-2606-14397/ · source: https://arxiv.org/abs/2606.14397v5 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, screen-understanding, science, web-navigation - Published: 2026-06-12 · Authors: Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel, Michal Zakrzewski, Sebastian Montagna, Damian Rynczak, Shreyansh Padarha, Kumail Alhamoud, Zihao Fu, William Lugoloobi, Kai Rawal, Hanna Yershova, Taras Rumezhak, Guohao Li, Fazl Barez, Baoyuan Wu, Arkadiusz Drohomirecki, Chris Russell, Christopher Summerfield, Adam Mahdi, Volodymyr Karpiv, Philip Torr, Adel Bibi ### RWKV-UI: UI Understanding with Enhanced Perception and Reasoning RWKV-UI is a visual language model for high-resolution web interfaces that uses layout detection and reasoning-chain visual prompts to understand pages. - https://awesomecua.com/resources/arxiv-2502-03971/ · source: https://arxiv.org/abs/2502.03971v1 - Paper · reviewed 2026-11-06 · environments: web · topics: screen-understanding, planning - Published: 2025-02-06 · Authors: Jiaxi Yang, Haowen Hou ### SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? SaaS-Bench evaluates browser agents on professional workflows across self-hosted applications. Weighted checkpoints measure task completion and partial progress, with application-state checks and configured content judges documented in the official harness. - https://awesomecua.com/resources/arxiv-2605-15777/ · source: https://arxiv.org/abs/2605.15777v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, evaluation, long-horizon, workflow, cross-app, state-tracking - Published: 2026-05-15 · Authors: Kean Shi, Zihang Li, Tianyi Ma, Zengji Tu, Jialong Wu, Xinbo Xu, Qingyao Yang, Ruoyu Wu, Weichu Xie, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, Baobao Chang ### Safe and Scalable Web Agent Learning via Recreated Websites Introduces VeriEnv, which recreates websites as executable training environments and exposes internal state through a Python SDK for task generation and deterministic reward checks. Experiments examine web-agent learning across new and expanded site environments. - https://awesomecua.com/resources/arxiv-2603-10505/ · source: https://arxiv.org/abs/2603.10505v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, sandbox, reinforcement-learning - Published: 2026-03-11 · Authors: Hyungjoo Chae, Jungsoo Park, Alan Ritter ### SafeArena: Evaluating the Safety of Autonomous Web Agents Pairs benign and harmful web tasks to assess deliberate misuse risks and categorize agent behavior by risk level. - https://awesomecua.com/resources/arxiv-2503-04957/ · source: https://arxiv.org/abs/2503.04957v1 - Paper · reviewed 2026-10-16 · environments: web · topics: safety-security, evaluation - Published: 2025-03-06 · Authors: Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Stańczak, Siva Reddy ### SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems Introduces SafeFlow, a semantic information-flow defense for multi-agent delegation, evaluated on safety tasks that include harmful web-agent behavior. - https://awesomecua.com/resources/arxiv-2607-25255/ · source: https://arxiv.org/abs/2607.25255v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, workflow - Published: 2026-07-28 · Authors: Haowen Dai, Zonghao Ying, Wenfeng Li, Xiangfan Wu, Yisong Xiao, Tianyuan Zhang, Jiaye Lin, Lei Wei, Guangyuan Dong, Xitong Ling, Xixun Lin, Quanchen Zou, Xiangzheng Zhang ### SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration Introduces SafeGround, which estimates GUI-grounding uncertainty from the spatial dispersion of sampled predictions and calibrates a decision threshold before testing. It studies risk-aware selection on ScreenSpot-Pro, with statistical control scoped to the threshold’s false-discovery rate. - https://awesomecua.com/resources/arxiv-2602-02419/ · source: https://arxiv.org/abs/2602.02419v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, robustness, evaluation - Published: 2026-02-02 · Authors: Qingni Wang, Yue Fan, Xin Eric Wang ### SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents Studies chain-level risk detection for mobile multimodal agents and a language-model-assisted evaluator, assessing safety interventions together with functional preservation. - https://awesomecua.com/resources/arxiv-2507-00841/ · source: https://arxiv.org/abs/2507.00841v1 - Paper · reviewed 2026-10-18 · environments: android · topics: safety-security, evaluation, mobile-navigation - Published: 2025-07-01 · Authors: Siyuan Liang, Tianmeng Fang, Zhe Liu, Aishan Liu, Yan Xiao, Jinyuan He, Ee-Chien Chang, Xiaochun Cao ### SafePred: A Predictive Guardrail for Computer-Using Agents via World Models SafePred uses world-model risk forecasts to guide action filtering and task replanning. Its desktop and web safety evaluations distinguish policy compliance, task success and success under the specified policy. - https://awesomecua.com/resources/arxiv-2602-01725/ · source: https://arxiv.org/abs/2602.01725v1 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: safety-security, world-models, planning, evaluation - Published: 2026-02-02 · Authors: Yurun Chen, Zeyi Liao, Ping Yin, Taotao Xie, Keting Yin, Shengyu Zhang ### Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems Presents a trace-level framework for comparing GUI agents with people on task outcomes, queries, and interface navigation in a production audio-search application. - https://awesomecua.com/resources/arxiv-2604-07929/ · source: https://arxiv.org/abs/2604.07929v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, observability, interaction-history - Published: 2026-04-09 · Authors: Maria Movin, Claudia Hauff, Aron Henriksson, Panagiotis Papapetrou ### Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize? Introduces AnyAppBench, a live Android benchmark that holds user goals fixed while changing the application used to achieve them. It combines category-controlled task pairs with failure annotations to distinguish task understanding from application-specific execution skill. - https://awesomecua.com/resources/arxiv-2609-34139/ · source: https://arxiv.org/abs/2609.34139v1 - Paper · reviewed 2026-10-17 · environments: android · topics: evaluation, robustness, mobile-navigation - Published: 2026-09-28 · Authors: Tien Tran, Namho Koh, Daiki E. Matsunaga, Ayush Jain, Kee Eung Kim ### Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents Introduces an evaluation framework for mobile GUI agents that compares actions implied by their reasoning with intended and executed actions to diagnose mismatches. - https://awesomecua.com/resources/arxiv-2510-02204/ · source: https://arxiv.org/abs/2510.02204v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, mobile-navigation, observability, safety-security - Published: 2025-10-02 · Authors: Lingzhong Dong, Ziqi Zhou, Shuaibo Yang, Haiyue Sheng, Pengzhou Cheng, Zongru Wu, Zheng Wu, Gongshen Liu, Zhuosheng Zhang ### SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents Introduces Spatial Credit Assignment, which uses click coordinates to refine group-relative reinforcement-learning updates for GUI grounding. It distinguishes failed clicks by spatial evidence while leaving the deployed action policy unchanged. - https://awesomecua.com/resources/arxiv-2609-36939/ · source: https://arxiv.org/abs/2609.36939v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, evaluation - Published: 2026-09-29 · Authors: Shengtian Yang, Ziyu Xiong, Kaibing Yang, Guangfeng Cai, Yewen Li, Peng Jiang, Gai Kun, Qingpeng Cai, Lei Feng ### SCAFFOLD: Self-Improving Web Agents via Recursive Parametric Skill Abstraction Introduces SCAFFOLD, which extracts executable skills from web trajectories, composes them into a hierarchy, and compresses redundant skills through behavioral checks. It periodically distills skill-guided trajectories into the policy and studies repeated improvement across web benchmarks. - https://awesomecua.com/resources/arxiv-2609-05511/ · source: https://arxiv.org/abs/2609.05511v1 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, automation, web-navigation - Published: 2026-08-31 · Authors: Bowei He, Xiaokun Zhang, Meng Ding, Xue Liu ### Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents Introduces MONDAY and an automated pipeline that extracts mobile-navigation training data from instructional videos across operating-system platforms. - https://awesomecua.com/resources/arxiv-2505-12632/ · source: https://arxiv.org/abs/2505.12632v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: video, data-generation, mobile-navigation - Published: 2025-05-19 · Authors: Yunseok Jang, Yeda Song, Sungryull Sohn, Lajanugen Logeswaran, Tiange Luo, Dong-Ki Kim, Kyunghoon Bae, Honglak Lee ### SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL Scales computer-agent reinforcement learning through verifiable task generation, capability-aware rollout allocation, and segmented visual context. - https://awesomecua.com/resources/arxiv-2607-11185/ · source: https://arxiv.org/abs/2607.11185v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: data-generation, reinforcement-learning, efficiency - Published: 2026-07-13 · Authors: Bowen Lv, Xiao Liu, Yanyu Ren, Hanyu Lai, Bohao Jing, Hanchen Zhang, Yanxiao Zhao, Shuntian Yao, Jie Tang, Yuxiao Dong ### ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data Introduces a cross-platform computer-use dataset and agents trained with a collection pipeline combining automated interaction and human input. - https://awesomecua.com/resources/arxiv-2509-15221/ · source: https://arxiv.org/abs/2509.15221v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: data-generation, automation - Published: 2025-09-18 · Authors: Zhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li, Bowen Yang, Zhenyu Wu, Xuehui Wang, Qiushi Sun, Shi Liu, Weiyun Wang, Shenglong Ye, Qingyun Li, Xuan Dong, Yue Yu, Chenyu Lu, YunXiang Mo, Yao Yan, Zeyue Tian, Xiao Zhang, Yuan Huang, Yiqian Liu, Weijie Su, Gen Luo, Xiangyu Yue, Biqing Qi, Kai Chen, Bowen Zhou, Yu Qiao, Qifeng Chen, Wenhai Wang ### ScaleTrack: Scaling and back-tracking Automated GUI Agents ScaleTrack combines a unified collection of GUI grounding samples with a planning objective that predicts the next action while reconstructing prior actions. This backward prediction links the current screenshot to interaction history during agent training. - https://awesomecua.com/resources/arxiv-2505-00416/ · source: https://arxiv.org/abs/2505.00416v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, planning, data-generation, interaction-history - Published: 2025-05-01 · Authors: Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, Lin Ma ### ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis Introduces ScaleWoB, which synthesizes interactive GUI environments as backend-free webpages with verifiable rewards. Its mobile-oriented benchmark evaluates agent execution and compares selected synthetic tasks with real-app behavior. - https://awesomecua.com/resources/arxiv-2605-25160/ · source: https://arxiv.org/abs/2605.25160v2 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, evaluation, sandbox - Published: 2026-05-24 · Authors: Guohong Liu, Jialei Ye, Pengzhi Gao, Wei Liu, Jian Luan, Yunxin Liu, Yuanchun Li ### Scaling Agent Learning via Experience Synthesis DreamGym synthesizes agent experiences with a reasoning-based environment model for online RL; its evaluations include the web task setting WebArena. - https://awesomecua.com/resources/arxiv-2511-03773/ · source: https://arxiv.org/abs/2511.03773v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, reinforcement-learning, world-models - Published: 2025-11-05 · Authors: Zhaorun Chen, Zhuokai Zhao, Kai Zhang, Bo Liu, Qi Qi, Yifan Wu, Tarun Kalluri, Sara Cao, Yuanhao Xiong, Haibo Tong, Huaxiu Yao, Hengduo Li, Jiacheng Zhu, Xian Li, Dawn Song, Bo Li, Jason Weston, Dat Huynh ### Scaling Agents for Computer Use Introduces Behavior Judge, which converts execution trajectories into behavior narratives and compares multiple candidate rollouts. Experiments examine this selection approach on OSWorld, WindowsAgentArena and AndroidWorld. - https://awesomecua.com/resources/arxiv-2510-02250/ · source: https://arxiv.org/abs/2510.02250v2 - Paper · reviewed 2026-10-17 · environments: desktop, windows, android · topics: evaluation, reward-models, long-horizon - Published: 2025-10-02 · Authors: Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, Xin Eric Wang ### Scaling Autonomous Agents via Automatic Reward Modeling And Planning Introduces a framework that learns a reward model from agent trajectories to guide multi-step planning, with online shopping among its decision-making settings. - https://awesomecua.com/resources/arxiv-2502-12130/ · source: https://arxiv.org/abs/2502.12130v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reward-models, planning, data-generation - Published: 2025-02-17 · Authors: Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, Chuang Gan ### Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis Introduces OSWorld-G grounding evaluation and the Jedi synthesis pipeline for examples spanning layout reasoning and fine-grained interaction. - https://awesomecua.com/resources/arxiv-2505-13227/ · source: https://arxiv.org/abs/2505.13227v3 - Paper · reviewed 2026-10-16 · environments: desktop · topics: grounding, data-generation, evaluation - Published: 2025-05-19 · Authors: Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, Caiming Xiong ### Scaling GUI Agents with Visual State Transitions Introduces State Transition Pretraining, which jointly learns to infer actions from GUI changes and predict future states from current screens and actions. Subsequent instruction-trajectory fine-tuning tests whether these representations transfer to desktop and mobile agent tasks. - https://awesomecua.com/resources/arxiv-2607-24112/ · source: https://arxiv.org/abs/2607.24112v1 - Paper · reviewed 2026-10-17 · environments: desktop, android · topics: world-models, action-prediction, data-generation - Published: 2026-07-27 · Authors: Xiangyan Liu, Kaixin Li, Haonan Wang, Biao Wu, Meng Fang, Longxu Dou, Chao Du, Michael Qizhe Shieh, Tianyu Pang ### Scaling Instructable Agents Across Many Simulated Worlds The SIMA project trains agents to follow language instructions in simulated 3D worlds, using images as input and keyboard-and-mouse actions as output. - https://awesomecua.com/resources/arxiv-2404-10179/ · source: https://arxiv.org/abs/2404.10179v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: action-prediction, imitation-learning - Published: 2024-03-13 · Authors: SIMA Team, Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, Stephanie C. Y. Chan, Jeff Clune, Adrian Collister, Vikki Copeman, Alex Cullum, Ishita Dasgupta, Dario de Cesare, Julia Di Trapani, Yani Donchev, Emma Dunleavy, Martin Engelcke, Ryan Faulkner, Frankie Garcia, Charles Gbadamosi, Zhitao Gong, Lucy Gonzales, Kshitij Gupta, Karol Gregor, Arne Olav Hallingstad, Tim Harley, Sam Haves, Felix Hill, Ed Hirst, Drew A. Hudson, Jony Hudson, Steph Hughes-Fitt, Danilo J. Rezende, Mimi Jasarevic, Laura Kampis, Rosemary Ke, Thomas Keck, Junkyung Kim, Oscar Knagg, Kavya Kopparapu, Rory Lawton, Andrew Lampinen, Shane Legg, Alexander Lerchner, Marjorie Limont, Yulan Liu, Maria Loks-Thompson, Joseph Marino, Kathryn Martin Cussons, Loic Matthey, Siobhan Mcloughlin, Piermaria Mendolicchio, Hamza Merzic, Anna Mitenkova, Alexandre Moufarek, Valeria Oliveira, Yanko Oliveira, Hannah Openshaw, Renke Pan, Aneesh Pappu, Alex Platonov, Ollie Purkiss, David Reichert, John Reid, Pierre Harvey Richemond, Tyson Roberts, Giles Ruscoe, Jaume Sanchez Elias, Tasha Sandars, Daniel P. Sawyer, Tim Scholtes, Guy Simmons, Daniel Slater, Hubert Soyer, Heiko Strathmann, Peter Stys, Allison C. Tam, Denis Teplyashin, Tayfun Terzi, Davide Vercelli, Bojan Vujatovic, Marcus Wainwright, Jane X. Wang, Zhengdong Wang, Daan Wierstra, Duncan Williams, Nathaniel Wong, Sarah York, Nick Young ### Scaling Synthetic Task Generation for Agents via Exploration AutoPlay explores application states and functions to generate tasks, then uses a model executor and verifier to construct demonstrations. It produces 20,000 tasks across 20 Android apps and 10,000 tasks across 13 Ubuntu apps for supervised and reinforcement learning of UI agents. - https://awesomecua.com/resources/arxiv-2509-25047/ · source: https://arxiv.org/abs/2509.25047v1 - Paper · reviewed 2026-10-17 · environments: android, linux, desktop, cross-platform · topics: data-generation, reinforcement-learning, imitation-learning - Published: 2025-09-29 · Authors: Ram Ramrakhya, Andrew Szot, Omar Attia, Yuhao Yang, Anh Nguyen, Bogdan Mazoure, Zhe Gan, Harsh Agrawal, Alexander Toshev ### Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation This work uses constraint-based progress evaluation to retain partially successful web trajectories for training. It introduces BookingArena for structured booking tasks and studies a distilled agent trained with the resulting data. - https://awesomecua.com/resources/arxiv-2602-12544/ · source: https://arxiv.org/abs/2602.12544v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, evaluation, web-navigation - Published: 2026-02-13 · Authors: Lajanugen Logeswaran, Jaekyeom Kim, Sungryull Sohn, Creighton Glasscock, Honglak Lee ### Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation HyperTrack provides mobile GUI navigation tasks across Chinese apps, while GUIEvalKit benchmarks VLM agents offline and supports analysis of training scale and history. - https://awesomecua.com/resources/arxiv-2605-27134/ · source: https://arxiv.org/abs/2605.27134v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: mobile-navigation, evaluation, data-generation, interaction-history - Published: 2026-05-26 · Authors: Heng Qu, Yike Liu, Renren Jin, Wenzong Zhang, Pengzhi Gao, Wei Liu, Jian Luan ### Scenario-Guided LLM-based Mobile App GUI Testing ScenGen organizes mobile GUI testing around business scenarios rather than unrestricted exploration. Five collaborating agents observe the interface, choose and execute actions, check scenario completion, and record operations and runtime bugs. - https://awesomecua.com/resources/arxiv-2506-05079/ · source: https://arxiv.org/abs/2506.05079v4 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: automation, mobile-navigation, planning - Published: 2025-06-05 · Authors: Shengcheng Yu, Yuchen Ling, Chunrong Fang, Quan Zhou, Yi Zhao, Chunyang Chen, Shaomin Zhu, Zhenyu Chen ### ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows ScienceBoard combines an interactive environment with professional scientific software and 169 human-curated tasks in areas including biochemistry, astronomy, and geoinformatics. It evaluates computer-use agents on visually rich research workflows and analyzes their execution limitations. - https://awesomecua.com/resources/arxiv-2505-19897/ · source: https://arxiv.org/abs/2505.19897v3 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, science, workflow - Published: 2025-05-26 · Authors: Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, Jianing Wang, Qintong Li, Xiangru Tang, Tianbao Xie, Xiachong Feng, Xiang Li, Ben Kao, Wenhai Wang, Biqing Qi, Lingpeng Kong, Zhiyong Wu ### ScienceCUA: Resource-to-Skill Construction and Execution-Guided Skill Evolution for Scientific Computer Use Agents Introduces ScienceCUA, which turns scientific software documentation into reusable agent skills and revises them using execution feedback, with two audited evaluation variants. - https://awesomecua.com/resources/arxiv-2611-03238/ · source: https://arxiv.org/abs/2611.03238v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: science, memory, self-reflection, evaluation - Published: 2026-11-02 · Authors: Daiqiang Li, Jingyi Peng, Rui Hu, Jia Li, Xuyang Liu, Guanren Qiao, Yujia Xiang, Qiuzhuang Sun, Zhiping Xu, Mingrui Luo ### ScopeMem: Latent Memory That Learns Where to Revise Frozen GUI Agents ScopeMem conditions latent-memory prefixes on a frozen GUI agent's initial action and trains them to correct errors while preserving correct decisions. Its evaluation separates action repairs from harmful revisions. - https://awesomecua.com/resources/arxiv-2610-26503/ · source: https://arxiv.org/abs/2610.26503v1 - Paper · reviewed 2026-10-20 · environments: android · topics: memory, interaction-history, imitation-learning, robustness, evaluation - Published: 2026-10-19 · Authors: Zhicheng Zheng, Cheng Zhang, Shengping Yu, Tianxiang Zhao ### SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety Introduces SCOUT, a safety verifier that first derives task-specific completion and safety rubrics from a computer-use trajectory, then actively probes the resulting environment for evidence. It evaluates whether these two stages detect harmful consequences that screenshots alone can miss. - https://awesomecua.com/resources/arxiv-2609-36201/ · source: https://arxiv.org/abs/2609.36201v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, evaluation, tool-integration - Published: 2026-09-28 · Authors: Jianxing Chen, Xiao Yu, Shipra Agrawal, Zhou Yu ### Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Infers UI elements and accessibility metadata from mobile screenshots to support screen-reader interaction with iOS applications. - https://awesomecua.com/resources/arxiv-2101-04893/ · source: https://arxiv.org/abs/2101.04893v1 - Paper · reviewed 2026-10-16 · environments: ios · topics: screen-understanding - Published: 2021-01-13 · Authors: Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, Jeffrey P. Bigham ### Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation Introduces Screen2AX, which creates hierarchical accessibility metadata from macOS screenshots, plus datasets and Screen2AX-Task for desktop-agent evaluation. - https://awesomecua.com/resources/arxiv-2507-16704/ · source: https://arxiv.org/abs/2507.16704v1 - Paper · reviewed 2026-11-06 · environments: desktop, macos · topics: screen-understanding, data-generation, evaluation - Published: 2025-07-22 · Authors: Viktor Muryn, Marta Sumyk, Mariya Hirna, Sofiya Garkot, Maksym Shamrai ### Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning Studies mobile-screen summarization using visual, textual, and structural UI information, with a dataset of human-written descriptions. - https://awesomecua.com/resources/arxiv-2108-03353/ · source: https://arxiv.org/abs/2108.03353v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: screen-understanding - Published: 2021-08-07 · Authors: Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, Yang Li ### ScreenAgent: A Vision Language Model-driven Computer Control Agent Combines a screenshot-and-action dataset with a planning, acting, and reflecting loop for computer control through mouse and keyboard actions. - https://awesomecua.com/resources/arxiv-2402-07945/ · source: https://arxiv.org/abs/2402.07945v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: automation, planning - Published: 2024-02-09 · Authors: Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, Qi Wang ### ScreenAI: A Vision-Language Model for UI and Infographics Understanding Introduces ScreenAI and screen-annotation training that supports UI question answering, navigation, and summarization data generation. - https://awesomecua.com/resources/arxiv-2402-04615/ · source: https://arxiv.org/abs/2402.04615v3 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: screen-understanding, data-generation - Published: 2024-02-07 · Authors: Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin, Jindong Chen, Abhanshu Sharma ### ScreenExplorer: Training a Vision-Language Model for Diverse Exploration in Open GUI World Trains ScreenExplorer with reinforcement learning in dynamic GUI environments, using a world-model curiosity reward and experience-stream distillation to support sustained exploration. - https://awesomecua.com/resources/arxiv-2505-19095/ · source: https://arxiv.org/abs/2505.19095v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reinforcement-learning, world-models, data-generation - Published: 2025-05-25 · Authors: Runliang Niu, Jinglong Ji, Yi Chang, Qi Wang ### ScreenHaystack: Finding Blind Zones in GUI Grounding Moves controlled target icons across high-resolution interfaces to diagnose spatial blind zones in grounding models and study targeted augmentation. - https://awesomecua.com/resources/arxiv-2609-32036/ · source: https://arxiv.org/abs/2609.32036v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, evaluation, robustness - Published: 2026-09-25 · Authors: Chenyue Li, Xiaoxiao Sun, Yubo Deng, Qinlin Zhao, Serena Yeung-Levy, Yuhui Zhang ### ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction Represents interface interactions with a stateful screen schema to model user behavior and support action understanding and prediction. - https://awesomecua.com/resources/arxiv-2503-20978/ · source: https://arxiv.org/abs/2503.20978v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: screen-understanding, memory - Published: 2025-03-26 · Authors: Yiqiao Jin, Stefano Petrangeli, Yu Shen, Gang Wu ### ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision ScreenParse supplies dense annotations of UI elements, types and text in web screenshots through an automated rendering and filtering pipeline. ScreenVLM learns to output structured screen markup, and the study evaluates dense parsing and transfer to GUI grounding. - https://awesomecua.com/resources/arxiv-2602-14276/ · source: https://arxiv.org/abs/2602.14276v2 - Paper · reviewed 2026-10-17 · environments: web · topics: screen-understanding, data-generation, grounding, evaluation - Published: 2026-02-15 · Authors: A. Said Gurbuz, Sunghwan Hong, Ahmed Nassar, Marc Pollefeys, Peter Staar ### ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots Introduces screenshot question answering over mobile interfaces, linking answers to screen content and bounding boxes. - https://awesomecua.com/resources/arxiv-2209-08199/ · source: https://arxiv.org/abs/2209.08199v4 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: screen-understanding, evaluation - Published: 2022-09-16 · Authors: Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Srinivas Sunkara, Victor Carbune, Jason Lin, Maria Wang, Yun Zhu, Jindong Chen ### ScreenSearch: Uncertainty-Aware OS Exploration ScreenSearch explores desktop GUI states by indexing and deduplicating UIA-tree screens, then using outcome ambiguity and frontier rewards to guide graph search. - https://awesomecua.com/resources/arxiv-2605-16024/ · source: https://arxiv.org/abs/2605.16024v2 - Paper · reviewed 2026-11-06 · environments: desktop · topics: screen-understanding, state-tracking, planning, data-generation - Published: 2026-05-15 · Authors: Michael Solodko, Justin Wagle ### Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents This study separates tool-adoption behavior from task competence in hybrid GUI–MCP agents. It tests tool-use incentives and screenshot-history compression, showing why changed action preferences alone need not improve held-out accuracy. - https://awesomecua.com/resources/arxiv-2608-03327/ · source: https://arxiv.org/abs/2608.03327v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: tool-integration, reinforcement-learning, efficiency, interaction-history, evaluation - Published: 2026-08-04 · Authors: Siqi Fan, Minghao Li, Xiaoqian Ma, Wenhui Tan, Xiusheng Huang, Juntong Wu, Liujie Zhang, Shuo Shang, Weihang Chen ### ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use Evaluates GUI target localization in high-resolution professional software and studies cascaded visual search to narrow the target region. - https://awesomecua.com/resources/arxiv-2504-07981/ · source: https://arxiv.org/abs/2504.07981v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: grounding, evaluation - Published: 2025-04-04 · Authors: Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, Tat-Seng Chua ### ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data Studies fine-tuning open language models on production web-workflow traces, including training choices for specialized web agents. - https://awesomecua.com/resources/arxiv-2411-15004/ · source: https://arxiv.org/abs/2411.15004v2 - Paper · reviewed 2026-10-16 · environments: web · topics: workflow, web-navigation - Published: 2024-11-22 · Authors: Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, Ameet Talwalkar ### SCUBA: Salesforce Computer Use Benchmark SCUBA evaluates enterprise UI navigation and CRM workflows with 300 task instances derived from Salesforce user interviews. Salesforce sandbox execution and milestone-based assessment support comparisons between zero-shot agents and agents given demonstrations. - https://awesomecua.com/resources/arxiv-2509-26506/ · source: https://arxiv.org/abs/2509.26506v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, industry, workflow, sandbox - Published: 2025-09-30 · Authors: Yutong Dai, Krithika Ramakrishnan, Jing Gu, Matthew Fernandez, Yanqi Luo, Viraj Prabhu, Zhenyu Hu, Silvio Savarese, Caiming Xiong, Zeyuan Chen, Ran Xu ### SE-GA: Memory-Augmented Self-Evolution for GUI Agents SE-GA retrieves hierarchical memories during inference and uses the collected experience in a separate policy-improvement pipeline. Its evaluation includes offline grounding/action tasks and online Android interaction, which measure different capabilities. - https://awesomecua.com/resources/arxiv-2605-16883/ · source: https://arxiv.org/abs/2605.16883v1 - Paper · reviewed 2026-10-17 · environments: android · topics: memory, interaction-history, mobile-navigation, grounding, evaluation - Published: 2026-05-16 · Authors: Shilong Jin, Lanjun Wang, Zhuosheng Zhang ### SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience SEAgent explores unfamiliar software using a world-state assessor and an automatically generated task curriculum. It learns from failed and successful interactions through adversarial imitation and GRPO, then combines specialist experience into a generalist agent evaluated on five OSWorld software environments. - https://awesomecua.com/resources/arxiv-2508-04700/ · source: https://arxiv.org/abs/2508.04700v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: reinforcement-learning, data-generation, world-models, imitation-learning - Published: 2025-08-06 · Authors: Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, Jiaqi Wang ### SecAgent: Efficient Mobile GUI Agent with Semantic Context Presents a Chinese mobile GUI dataset and navigation benchmark together with SecAgent, which compresses previous screenshots and actions into semantic summaries. The agent is trained with supervised and reinforcement fine-tuning to use this compact interaction context. - https://awesomecua.com/resources/arxiv-2603-08533/ · source: https://arxiv.org/abs/2603.08533v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, memory, reinforcement-learning - Published: 2026-03-09 · Authors: Yiping Xie, Song Chen, Jingxuan Xing, Wei Jiang, Zekun Zhu, Yingyao Wang, Pi Bu, Jun Song, Yuning Jiang, Bo Zheng ### Secure and Efficient Access Control for Computer-Use Agents via Context Space Introduces CSAgent, which combines static access policies with user intent and context and enforces them through an operating-system service. A supporting toolchain constructs and refines policies for agents using API, CLI and GUI interfaces. - https://awesomecua.com/resources/arxiv-2509-22256/ · source: https://arxiv.org/abs/2509.22256v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, tool-integration - Published: 2025-09-26 · Authors: Haochen Gong, Chenxiao Li, Rui Chang, Wenbo Shen ### Secure-CUA: Controlling Untrusted Influence in Computer-Use Agents Proposes Secure-CUA, which commits to per-action transactions before reading untrusted page content and separates content queries from GUI target selection. The paper formalizes its protection assumptions and evaluates benign task utility on WebArena. - https://awesomecua.com/resources/arxiv-2610-09469/ · source: https://arxiv.org/abs/2610.09469v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, web-navigation - Published: 2026-10-07 · Authors: Sarthak Choudhary, Mihai Christodorescu, Ashish Hooda, Somesh Jha, Tongxin Li, Damien Octeau ### SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents Introduces SecureWebArena with simulated interactive websites, an attack-vector taxonomy and labeled agent trajectories. Its evaluation separates failures in reasoning, behavioral traces and task outcomes across vision-language agent families. - https://awesomecua.com/resources/arxiv-2510-10073/ · source: https://arxiv.org/abs/2510.10073v2 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, robustness - Published: 2025-10-11 · Authors: Zonghao Ying, Yangguang Shao, Jianle Gan, Gan Xu, Wenxin Zhang, Quanchen Zou, Junzheng Shi, Zhenfei Yin, Mingchuan Zhang, Aishan Liu, Xianglong Liu ### Securing Computer-Use Agents Against Branch Steering Attacks Introduces STEER-Bench to study branch-steering attacks that manipulate untrusted web data toward hazardous paths already permitted by an agent’s plan. Its COBRA architecture combines trusted branching plans with advance constraints on the parameters and destinations each branch can use. - https://awesomecua.com/resources/arxiv-2610-03089/ · source: https://arxiv.org/abs/2610.03089v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, planning, evaluation - Published: 2026-10-02 · Authors: Giulio Zingrillo, Hanna Foerster, Ilia Shumailov, Yiren Zhao, Robert Mullins ### Securing Web Agents with the Compile-Then-Execute Paradigm Baya compiles a web task into a program using a Trusted Website API before viewing live content, limiting how webpage prompt injections can alter agent control flow. - https://awesomecua.com/resources/arxiv-2611-06039/ · source: https://arxiv.org/abs/2611.06039v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation, planning, tool-integration - Published: 2026-11-04 · Authors: Annabella Chow, Jinhao Zhu, Sylvie Venuto, Abdulaziz Alharbi, Basel Alomair, Raluca Popa, David Wagner, Julien Piet ### See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch ScratchWorld evaluates program construction through the Scratch interface using fine-grained drag-and-drop actions and a complementary semantic-API mode. Runtime tests assess the resulting programs, allowing analysis of the gap between program reasoning and GUI manipulation. - https://awesomecua.com/resources/arxiv-2602-10814/ · source: https://arxiv.org/abs/2602.10814v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, grounding, automation - Published: 2026-02-11 · Authors: Xingyi Zhang, Yulei Ye, Kaifeng Huang, Wenhao Li, Xiangfeng Wang ### See, Think, Act: Online Shopper Behavior Simulation with VLM Agents Studies online shopper action prediction with VLM agents using webpage screenshots, HTML, and action history from OPeRA, trained with SFT and hierarchical RL rewards. - https://awesomecua.com/resources/arxiv-2510-19245/ · source: https://arxiv.org/abs/2510.19245v1 - Paper · reviewed 2026-11-06 · environments: web · topics: action-prediction, interaction-history, reinforcement-learning - Published: 2025-10-22 · Authors: Yimeng Zhang, Jiri Gesi, Ran Xue, Tian Wang, Ziyi Wang, Yuxuan Lu, Sinong Zhan, Huimin Zeng, Qingjun Cui, Yufan Guo, Jing Huang, Mubarak Shah, Dakuo Wang ### See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles Introduces a benchmark for binary GUI-toggle instructions and State-aware Reasoning, which compares the observed control state with the requested state before choosing an action. It evaluates agents on toggle tasks and broader interactive benchmarks. - https://awesomecua.com/resources/arxiv-2509-13615/ · source: https://arxiv.org/abs/2509.13615v5 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: state-tracking, grounding, evaluation - Published: 2025-09-17 · Authors: Zongru Wu, Rui Mao, Zhiyuan Tian, Pengzhou Cheng, Tianjie Ju, Zheng Wu, Lingzhong Dong, Haiyue Sheng, Zhuosheng Zhang, Gongshen Liu ### SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis Introduces SEE, which first explores mobile interfaces to build a screen-and-element transition graph, then composes task trajectories through graph planning and controlled sampling. It studies coverage, loop avoidance, and transfer after training agents on the generated sequences. - https://awesomecua.com/resources/arxiv-2607-18046/ · source: https://arxiv.org/abs/2607.18046v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, long-horizon, mobile-navigation - Published: 2026-07-20 · Authors: Zhuohang Fan, Beichen Zhang, Yuanfa Li, Changqiao Wu, Wei Liu, Jian Luan, Weigang Zhang ### SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents Trains a screenshot-driven GUI agent with grounding pretraining and introduces ScreenSpot for evaluating interface target localization. - https://awesomecua.com/resources/arxiv-2401-10935/ · source: https://arxiv.org/abs/2401.10935v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, data-generation - Published: 2024-01-17 · Authors: Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu ### Seed1.5-VL Technical Report Reports Seed1.5-VL, a general vision-language model evaluated on GUI control and gameplay alongside visual understanding and reasoning tasks. - https://awesomecua.com/resources/arxiv-2505-07062/ · source: https://arxiv.org/abs/2505.07062v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, screen-understanding - Published: 2025-05-11 · Authors: Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang, Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, Shixiong Zhao, Shuai Peng, Shuangye Li, Sihang Yuan, Sijin Wu, Tianheng Cheng, Weiwei Liu, Wenqian Wang, Xianhan Zeng, Xiao Liu, Xiaobo Qin, Xiaohan Ding, Xiaojun Xiao, Xiaoying Zhang, Xuanwei Zhang, Xuehan Xiong, Yanghua Peng, Yangrui Chen, Yanwei Li, Yanxu Hu, Yi Lin, Yiyuan Hu, Yiyuan Zhang, Youbin Wu, Yu Li, Yudong Liu, Yue Ling, Yujia Qin, Zanbo Wang, Zhiwu He, Aoxue Zhang, Bairen Yi, Bencheng Liao, Can Huang, Can Zhang, Chaorui Deng, Chaoyi Deng, Cheng Lin, Cheng Yuan, Chenggang Li, Chenhui Gou, Chenwei Lou, Chengzhi Wei, Chundian Liu, Chunyuan Li, Deyao Zhu, Donghong Zhong, Feng Li, Feng Zhang, Gang Wu, Guodong Li, Guohong Xiao, Haibin Lin, Haihua Yang, Haoming Wang, Heng Ji, Hongxiang Hao, Hui Shen, Huixia Li, Jiahao Li, Jialong Wu, Jianhua Zhu, Jianpeng Jiao, Jiashi Feng, Jiaze Chen, Jianhui Duan, Jihao Liu, Jin Zeng, Jingqun Tang, Jingyu Sun, Joya Chen, Jun Long, Junda Feng, Junfeng Zhan, Junjie Fang, Junting Lu, Kai Hua, Kai Liu, Kai Shen, Kaiyuan Zhang, Ke Shen, Ke Wang, Keyu Pan, Kun Zhang, Kunchang Li, Lanxin Li, Lei Li, Lei Shi, Li Han, Liang Xiang, Liangqiang Chen, Lin Chen, Lin Li, Lin Yan, Liying Chi, Longxiang Liu, Mengfei Du, Mingxuan Wang, Ningxin Pan, Peibin Chen, Pengfei Chen, Pengfei Wu, Qingqing Yuan, Qingyao Shuai, Qiuyan Tao, Renjie Zheng, Renrui Zhang, Ru Zhang, Rui Wang, Rui Yang, Rui Zhao, Shaoqiang Xu, Shihao Liang, Shipeng Yan, Shu Zhong, Shuaishuai Cao, Shuangzhi Wu, Shufan Liu, Shuhan Chang, Songhua Cai, Tenglong Ao, Tianhao Yang, Tingting Zhang, Wanjun Zhong, Wei Jia, Wei Weng, Weihao Yu, Wenhao Huang, Wenjia Zhu, Wenli Yang, Wenzhi Wang, Xiang Long, XiangRui Yin, Xiao Li, Xiaolei Zhu, Xiaoying Jia, Xijin Zhang, Xin Liu, Xinchen Zhang, Xinyu Yang, Xiongcai Luo, Xiuli Chen, Xuantong Zhong, Xuefeng Xiao, Xujing Li, Yan Wu, Yawei Wen, Yifan Du, Yihao Zhang, Yining Ye, Yonghui Wu, Yu Liu, Yu Yue, Yufeng Zhou, Yufeng Yuan, Yuhang Xu, Yuhong Yang, Yun Zhang, Yunhao Fang, Yuntao Li, Yurui Ren, Yuwen Xiong, Zehua Hong, Zehua Wang, Zewei Sun, Zeyu Wang, Zhao Cai, Zhaoyue Zha, Zhecheng An, Zhehui Zhao, Zhengzhuo Xu, Zhipeng Chen, Zhiyong Wu, Zhuofan Zheng, Zihao Wang, Zilong Huang, Ziyu Zhu, Zuquan Song ### Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps Trident combines exploration, monitoring, and detection agents to identify functional bugs that do not crash a mobile app. It reasons over screenshot sequences and interface transitions to construct test oracles, evaluated on 590 non-crash bugs and additional live apps. - https://awesomecua.com/resources/arxiv-2407-03037/ · source: https://arxiv.org/abs/2407.03037v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, evaluation, screen-understanding, state-tracking - Published: 2024-07-03 · Authors: Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yawen Wang, Jun Hu, Qing Wang ### Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models Studies reinforcement learning for vision-language models facing conflicting instructions across text and images, including evaluation on web-agent safety tasks. - https://awesomecua.com/resources/arxiv-2609-22234/ · source: https://arxiv.org/abs/2609.22234v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, safety-security, robustness - Published: 2026-09-03 · Authors: Nicholas Sansoterra, Zishuo Zheng, Sachin Kumar ### SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents Introduces SeekJudge, which iteratively searches and analyzes GUI trajectories to assess instruction fulfillment. The work combines dense judgment-data synthesis, a step-label benchmark, and reward-server scheduling to study accurate and practical reward feedback for agent training. - https://awesomecua.com/resources/arxiv-2607-23263/ · source: https://arxiv.org/abs/2607.23263v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reward-models, evaluation, data-generation - Published: 2026-07-25 · Authors: Yang Wan, Zhenhao Zhang, Jierui Wang, Xiangxin Zhou, Linchao Zhu ### SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction Uses instruction screening and a safety-augmented world model to assess likely consequences of mobile GUI actions before execution. - https://awesomecua.com/resources/arxiv-2607-15550/ · source: https://arxiv.org/abs/2607.15550v3 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: safety-security, world-models - Published: 2026-07-17 · Authors: Xue Yu, Bo Yuan, Kailin Zhao, Pengshuai Yang, Hong Hu, Junlan Feng ### Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding Recasts GUI grounding as predicting a mask of valid click locations from bounding-box supervision, then derives a click point with a deterministic selection rule. - https://awesomecua.com/resources/arxiv-2610-16234/ · source: https://arxiv.org/abs/2610.16234v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding - Published: 2026-10-12 · Authors: Jingchao Wang, Dingjiang Huang ### Selective Knowledge Control for Continual GUI Agent Learning over Application Streams Introduces activation-conditioned gradient control for GUI agents learning successive applications. It protects inactive application-specific neurons, constrains updates to shared neurons, and accumulates a compact historical state to balance adaptation with retention. - https://awesomecua.com/resources/arxiv-2609-06530/ · source: https://arxiv.org/abs/2609.06530v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: memory, robustness - Published: 2026-09-06 · Authors: Zirui Shang, Xin Shu, Yang Liu, Zhi Gao, Xinxiao Wu, Lifeng Fan ### Self-evolving LLM agents with in-distribution Optimization Introduces Q-Evolve, which derives step-wise rewards from an in-distribution critic for LLM agent learning; WebShop is one of its interactive evaluation settings. - https://awesomecua.com/resources/arxiv-2606-07367/ · source: https://arxiv.org/abs/2606.07367v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, reward-models, long-horizon - Published: 2026-06-05 · Authors: Yudi Zhang, Meng Fang, Zhenfang Chen, Mykola Pechenizkiy ### Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V Overlays labeled image regions to help multimodal models refer to and reason about visual targets; a general visual-prompting method relevant to GUI grounding. - https://awesomecua.com/resources/arxiv-2310-11441/ · source: https://arxiv.org/abs/2310.11441v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: grounding - Published: 2023-10-17 · Authors: Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao ### SheetAgent: Towards A Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Models Introduces SheetRM, a benchmark of long-horizon spreadsheet tasks, and SheetAgent, an LLM agent that plans, retrieves information and reflects while manipulating spreadsheets. - https://awesomecua.com/resources/arxiv-2403-03636/ · source: https://arxiv.org/abs/2403.03636v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, planning, long-horizon, self-reflection - Published: 2024-03-06 · Authors: Yibin Chen, Yifu Yuan, Zeyu Zhang, Yan Zheng, Jinyi Liu, Fei Ni, Jianye Hao, Hangyu Mao, Fuzheng Zhang ### SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheets Introduces SheetBrain, a spreadsheet question-answering and manipulation agent that uses a Python sandbox and Excel tools, plus the SheetBench benchmark. - https://awesomecua.com/resources/arxiv-2510-19247/ · source: https://arxiv.org/abs/2510.19247v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: tool-integration, evaluation, science - Published: 2025-10-22 · Authors: Ziwei Wang, Jiayuan Su, Mengyu Zhou, Huaxing Zeng, Mengni Jia, Xiao Lv, Haoyu Dong, Xiaojun Ma, Shi Han, Dongmei Zhang ### SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models Introduces SheetCopilot, an agent that steers spreadsheet software via atomic virtual APIs, along with a 221-task spreadsheet manipulation benchmark. - https://awesomecua.com/resources/arxiv-2305-19308/ · source: https://arxiv.org/abs/2305.19308v2 - Paper · reviewed 2026-10-18 · environments: desktop, windows · topics: workflow, tool-integration, evaluation - Published: 2023-05-30 · Authors: Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, Zhaoxiang Zhang ### ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning Proposes a guardrail agent that reasons over explicit safety policies and checks action trajectories, with a web-agent safety benchmark. - https://awesomecua.com/resources/arxiv-2503-22738/ · source: https://arxiv.org/abs/2503.22738v2 - Paper · reviewed 2026-10-16 · environments: web · topics: safety-security - Published: 2025-03-26 · Authors: Zhaorun Chen, Mintong Kang, Bo Li ### ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents ShopGym builds sandbox e-commerce storefronts from live shops and generates catalog-grounded tasks for evaluating web shopping agents in resettable environments. - https://awesomecua.com/resources/arxiv-2605-16116/ · source: https://arxiv.org/abs/2605.16116v1 - Paper · reviewed 2026-11-06 · environments: web · topics: sandbox, data-generation, evaluation, web-navigation - Published: 2026-05-15 · Authors: Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang ### ShowUI-$π$: Flow-based Generative Models as GUI Dexterous Hands Models discrete clicks and continuous drags with flow-based action generation, and introduces drag trajectories and the ScreenDrag benchmark. - https://awesomecua.com/resources/arxiv-2512-24965/ · source: https://arxiv.org/abs/2512.24965v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: grounding, evaluation, data-generation - Published: 2025-12-31 · Authors: Siyuan Hu, Kevin Qinghong Lin, Mike Zheng Shou ### ShowUI-Aloha: Human-Taught GUI Agent Transforms human screen recordings and input events into annotated demonstrations used by planning and execution components for GUI tasks. - https://awesomecua.com/resources/arxiv-2601-07181/ · source: https://arxiv.org/abs/2601.07181v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: recording, workflow, data-generation - Published: 2026-01-12 · Authors: Yichun Zhang, Xiangwu Guo, Yauhong Goh, Jessica Hu, Zhiheng Chen, Xin Wang, Difei Gao, Mike Zheng Shou ### ShowUI: One Vision-Language-Action Model for GUI Visual Agent Combines visual token selection, interleaved visual-action history, and curated instruction data in a compact GUI action model. - https://awesomecua.com/resources/arxiv-2411-17465/ · source: https://arxiv.org/abs/2411.17465v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, efficiency - Published: 2024-11-26 · Authors: Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, Mike Zheng Shou ### Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents Introduces Sidekick, a communication interface that uses ambient status cues, multimodal action summaries, and visual and verbal explanations of reasoning to help people monitor computer-use agents. A user study examines progress awareness, error traceability, and multitasking across background and foreground interaction. - https://awesomecua.com/resources/arxiv-2607-17527/ · source: https://arxiv.org/abs/2607.17527v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, observability, long-horizon - Published: 2026-07-20 · Authors: Ruei-Che Chang, Wenqian Xu, Dingzeyu Li, Bryan Wang, Anhong Guo ### Signal-Driven Observation for Long-Horizon Web Agents Proposes Signal-Driven Observation: a separate call selects relevant DOM elements and selectors, and a detector requests new observations only after meaningful browser events or failures. The paper frames observation timing and compression as an architectural research problem for long-horizon web agents. - https://awesomecua.com/resources/arxiv-2606-06708/ · source: https://arxiv.org/abs/2606.06708v2 - Paper · reviewed 2026-10-18 · environments: web · topics: efficiency, screen-understanding, long-horizon - Published: 2026-06-04 · Authors: Shubham Gaur, Ian Lane ### Sikuli: Using GUI screenshots for search and automation Introduces screenshot-based GUI search and visual scripting, using image patterns to locate interface targets and direct mouse and keyboard events. It evaluates screenshot queries and demonstrates automation and interactive-help tasks. - https://awesomecua.com/resources/url-a4cdb2ee0d07d3980651/ · source: https://doi.org/10.1145/1622176.1622213 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, automation, screen-understanding - Published: 2009-10 · Authors: Tom Yeh, Tsung-Hsiang Chang, Robert C. Miller ### SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce SimGym runs traffic-grounded synthetic buyers as browser agents to compare e-commerce storefront variants offline, using buyer profiles derived from production interactions. - https://awesomecua.com/resources/arxiv-2602-01443/ · source: https://arxiv.org/abs/2602.01443v1 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, web-navigation, evaluation, data-generation - Published: 2026-02-01 · Authors: Alberto Castelo, Zahra Zanjani Foumani, Ailin Fan, Keat Yang Koay, Vibhor Malik, Yuanzheng Zhu, Han Li, Meysam Feghhi, Ronie Uliana, Shuang Xie, Zhaoyu Zhang, Angelo Ocana Martins, Mingyu Zhao, Francis Pelland, Jonathan Faerman, Nikolas LeBlanc, Aaron Glazer, Andrew McNamara, Lingyun Wang, Zhong Wu ### SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agents SimPersona learns buyer types from clickstreams and uses persona tokens to condition web agents trained on browsing traces for live storefront tasks. - https://awesomecua.com/resources/arxiv-2605-14205/ · source: https://arxiv.org/abs/2605.14205v2 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, interaction-history, imitation-learning, data-generation - Published: 2026-05-14 · Authors: Zahra Zanjani Foumani, Alberto Castelo, Shuang Xie, Ted Chaiwachirasak, Han Li, Lingyun Wang ### SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling Introduces SimTrace, a framework that generates synthetic multimodal web clickstreams by pairing a computer-use client agent's actions with web observations and user context. - https://awesomecua.com/resources/arxiv-2609-38397/ · source: https://arxiv.org/abs/2609.38397v1 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, interaction-history, action-prediction - Published: 2026-09-29 · Authors: Yunan Lu, Shuang Xie, Meghna Allamudi, Mingyu Zhao, Han Li, Lingyun Wang, Zhou Yu ### SIR: Self-improving Red-teaming for Compute Use Agents Introduces SIR, which iteratively refines a reusable library of prompt-injection strategies from failed computer-use attack trajectories. Its evaluation uses deterministic system-state checks and counts attacks only when the adversarial objective and legitimate task both complete. - https://awesomecua.com/resources/arxiv-2608-30207/ · source: https://arxiv.org/abs/2608.30207v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: safety-security, evaluation, data-generation - Published: 2026-08-31 · Authors: Chen Xiong, Zhiyuan He, Pin-Yu Chen, Stjepan Picek, Tsung-Yi Ho ### Skill-Guided Continuation Distillation for GUI Agents Skill-Guided Continuation Distillation first lets a policy reach its own off-trajectory states, then uses skill guidance to collect successful continuations. Mixing those continuations with expert traces supplies supervision for recovery states absent from demonstrations. - https://awesomecua.com/resources/arxiv-2606-18890/ · source: https://arxiv.org/abs/2606.18890v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: imitation-learning, data-generation, robustness, planning - Published: 2026-06-17 · Authors: Zhimin Fan, Hongwei Yu, Yeqing Shen, Haolong Yan, Guozhen Peng, Tianhao Peng, Yudong Zhang, Xiaowen Zhang, Kaijun Tan, Zheng Ge, Xiangyu Zhang, Daxin Jiang ### SkillDroid: Compile Once, Reuse Forever Introduces SkillDroid, which compiles successful mobile GUI trajectories into reusable action templates and replays them without model calls, recompiling when they degrade. - https://awesomecua.com/resources/arxiv-2604-14872/ · source: https://arxiv.org/abs/2604.14872v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, efficiency, mobile-navigation, automation - Published: 2026-04-16 · Authors: Qijia Chen, Andrea Bellucci, Zhida Sun, Giulio Jacucci ### SkillHarness: Harnessing Safe Skills for Computer-Use Agents Introduces SkillHarness, a framework for computer-use agents to learn and reuse skills under safety constraints when interactions include prompt injections or pop-ups. - https://awesomecua.com/resources/arxiv-2606-20636/ · source: https://arxiv.org/abs/2606.20636v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, memory, workflow, robustness - Published: 2026-06-02 · Authors: Yurun Chen, Biao Yi, Keting Yin, Shengyu Zhang ### SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation Introduces SkillLens visual skill cards for retrieved GUI action guidance and CardDistill for training an action-prediction student that runs without retrieval on website tasks. - https://awesomecua.com/resources/arxiv-2608-10775/ · source: https://arxiv.org/abs/2608.10775v2 - Paper · reviewed 2026-11-06 · environments: web · topics: action-prediction, memory, imitation-learning, web-navigation - Published: 2026-08-11 · Authors: Zhou Liu, Chenxu Zhang, Ligang Huang, Zeli Su, Zewei Pan, Changhang Tian, Zhaoyang Han, Xing Chen, Yuanfeng Song, Wentao Zhang ### SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents Introduces SkillShift-Bench to evaluate how web agents reuse learned skills when task context, visual perception, or execution conditions change across web systems. - https://awesomecua.com/resources/arxiv-2610-33528/ · source: https://arxiv.org/abs/2610.33528v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, robustness, memory, web-navigation - Published: 2026-10-24 · Authors: Bo-Yu Chen, Zhi Zhou, Yu-Feng Li ### SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills SkillWeaver explores websites, practices interactions, and converts the resulting experience into a reusable library of callable skill APIs. Its WebArena and live-website experiments examine skill refinement, composition, and transfer from stronger agents to weaker ones. - https://awesomecua.com/resources/arxiv-2504-07079/ · source: https://arxiv.org/abs/2504.07079v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration, planning - Published: 2025-04-09 · Authors: Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, Yu Su ### Skim: Speculative Execution for Fast and Efficient Web Agents Introduces Skim, a speculative execution framework for web agents that learns site URL and answer patterns offline, then uses them to answer matching queries with a small model. - https://awesomecua.com/resources/arxiv-2605-16565/ · source: https://arxiv.org/abs/2605.16565v2 - Paper · reviewed 2026-11-06 · environments: web · topics: efficiency, web-navigation, automation - Published: 2026-05-15 · Authors: Mike Wong, Kevin Hsieh, Suman Nath, Ravi Netravali ### SlowBA: An efficiency backdoor attack towards VLM-based GUI agents Introduces SlowBA, a backdoor attack on vision-language GUI agents that uses pop-up triggers to induce longer reasoning chains and increase response latency. - https://awesomecua.com/resources/arxiv-2603-08316/ · source: https://arxiv.org/abs/2603.08316v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, efficiency, reinforcement-learning - Published: 2026-03-09 · Authors: Junxian Li, Tu Lan, Haozhen Tan, Yan Meng, Haojin Zhu ### SmartFlow: Robotic Process Automation using LLMs Presents SmartFlow, which converts visible GUI elements into text for language-model action planning and executes the plan through a scripting engine. A collection of enterprise applications evaluates adaptation to different layouts and tasks. - https://awesomecua.com/resources/arxiv-2405-12842/ · source: https://arxiv.org/abs/2405.12842v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, workflow, screen-understanding - Published: 2024-05-21 · Authors: Arushi Jain, Shubham Paliwal, Monika Sharma, Lovekesh Vig, Gautam Shroff ### SmartSnap: Proactive Evidence Seeking for Self-Verifying Agents SmartSnap trains GUI agents to collect concise screenshot evidence of task completion for an LLM judge, with experiments on mobile tasks. - https://awesomecua.com/resources/arxiv-2512-22322/ · source: https://arxiv.org/abs/2512.22322v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: self-reflection, evaluation, reinforcement-learning, mobile-navigation - Published: 2025-12-26 · Authors: Shaofei Cai, Yulei Qin, Haojia Lin, Zihan Xu, Gang Li, Yuchen Shi, Zongyi Li, Yong Mao, Siqi Cai, Xiaoyu Tan, Yitao Liang, Ke Li, Xing Sun ### Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks Uses inference of possible user queries from screenshots and target coordinates as a training bridge between GUI grounding and action reasoning. - https://awesomecua.com/resources/arxiv-2503-00401/ · source: https://arxiv.org/abs/2503.00401v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning - Published: 2025-03-01 · Authors: Zongru Wu, Pengzhou Cheng, Zheng Wu, Tianjie Ju, Zhuosheng Zhang, Gongshen Liu ### SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agents SnapGuard detects screenshot-based prompt injection through complementary visual-stability and recovered textual cues. Its evaluation compares detection quality and inference cost across attack and benign-page settings, rather than establishing universal runtime prevention. - https://awesomecua.com/resources/arxiv-2604-25562/ · source: https://arxiv.org/abs/2604.25562v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, screen-understanding, evaluation, efficiency - Published: 2026-04-28 · Authors: Mengyao Du, Han Fang, Haokai Ma, Jiahao Chen, Kai Xu, Quanjun Yin, Ee-Chien Chang ### SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments Introduces SoftwareGym, a pipeline that synthesizes runnable GUI applications and iteratively repairs them using execution checks. It also provides a task benchmark and studies whether training on synthesized application trajectories transfers to other computer-use environments. - https://awesomecua.com/resources/arxiv-2610-12611/ · source: https://arxiv.org/abs/2610.12611v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, evaluation, sandbox - Published: 2026-10-08 · Authors: Tian Jin, Wenhao Wang, Jing Kang, Haoting Shi, Rui Ye, Yijun Lu, Siheng Chen ### SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning SOLAR-RL trains GUI navigation agents by reconstructing rollout candidates from static data and assigning step rewards that reflect trajectory quality. - https://awesomecua.com/resources/arxiv-2604-22558/ · source: https://arxiv.org/abs/2604.22558v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, reward-models, long-horizon - Published: 2026-04-24 · Authors: Jichao Wang, Liuyang Bian, Yufeng Zhou, Han Xiao, Yue Pan, Guozhi Wang, Hao Wang, Zhaoxiong Wang, Yafei Wen, Xiaoxin Chen, Shuai Ren, Lingfang Zeng ### SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation Introduces a smartphone-agent benchmark with English and Chinese tasks, real-time Android interaction, and completion and resource-use metrics. - https://awesomecua.com/resources/arxiv-2410-15164/ · source: https://arxiv.org/abs/2410.15164v3 - Paper · reviewed 2026-10-16 · environments: android · topics: evaluation, efficiency, mobile-navigation - Published: 2024-10-19 · Authors: Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, Kun Shao ### SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution Introduces SPA, a reward redistribution method for multi-step LLM agents that estimates each action’s contribution to task completion; Webshop is among its evaluations. - https://awesomecua.com/resources/arxiv-2505-20732/ · source: https://arxiv.org/abs/2505.20732v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, reward-models, long-horizon - Published: 2025-05-27 · Authors: Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, Wenjie Li ### SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing Introduces SparkUI-Parser, which adds continuous coordinate decoding and a rejection mechanism for nonexistent targets to a multimodal model. It also presents ScreenParse for evaluating structural interface perception. - https://awesomecua.com/resources/arxiv-2509-04908/ · source: https://arxiv.org/abs/2509.04908v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, evaluation - Published: 2025-09-05 · Authors: Hongyi Jing, Jiafu Chen, Chen Rao, Ziqiang Dang, Jiajie Teng, Tianyi Chu, Juncheng Mo, Shuo Fang, Huaizhong Lin, Rui Lv, Chenguang Ma, Lei Zhao ### Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents GUIPruner compresses high-resolution screenshots and interaction history for GUI navigation through training-free resizing and layout-aware token pruning. - https://awesomecua.com/resources/arxiv-2602-23235/ · source: https://arxiv.org/abs/2602.23235v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, training-free, interaction-history, grounding - Published: 2026-02-26 · Authors: Zhou Xu, Bowen Zhou, Qi Wang, Shuwen Feng, Jingyu Xiao ### SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environments SpecOps divides testing of GUI-based agents into test generation, environment setup, execution and validation handled by specialist agents. Its evaluation spans five real-world agents, including web applications and browser extensions, and measures execution reliability and bug discovery. - https://awesomecua.com/resources/arxiv-2603-10268/ · source: https://arxiv.org/abs/2603.10268v1 - Paper · reviewed 2026-10-17 · environments: web · topics: automation, evaluation, planning - Published: 2026-03-10 · Authors: Syed Yusuf Ahmed, Shiwei Feng, Chanwoo Bae, Calix Barrus Xiangyu Zhang ### SPECTRA: On-Device Cognitive Perturbation and Trajectory Analysis for Autonomous Edge-Cloud GUI Grounding Introduces SPECTRA, an edge-cloud GUI grounding framework that uses visual perturbations and trajectory analysis to decide when to request a cloud model. - https://awesomecua.com/resources/arxiv-2609-35775/ · source: https://arxiv.org/abs/2609.35775v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, efficiency, robustness - Published: 2026-07-31 · Authors: Zhan Qu, Hui Zang, Ran Chen, Tao Wang, Shengyu Zhang ### Speculative Rollback Correction for Quality-Diverse Web Agent Imitation Speculative Rollback Correction lets a student web agent execute short branches before teacher review localizes harmful deviations. Verified continuations and localized corrections feed supervised training while an archive retains diverse successful solutions. - https://awesomecua.com/resources/arxiv-2606-12485/ · source: https://arxiv.org/abs/2606.12485v2 - Paper · reviewed 2026-10-18 · environments: web · topics: imitation-learning, data-generation, robustness - Published: 2026-06-10 · Authors: Longkun Hao, Hongyu Lin, Hao Li, Zhuowen Liu, Zhichao Yang, Haojie Hao, Dongshuo Huang, Haitao Yang, Hongyu Ge, Ming jie Xie, Yanjun Wu, Zi Hao Yin, Yan Bai, Yihang Lou ### Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? Benchmarks multimodal agents on data-science and engineering workflows involving enterprise applications, code, and GUI operations in real computer environments. - https://awesomecua.com/resources/arxiv-2407-10956/ · source: https://arxiv.org/abs/2407.10956v1 - Paper · reviewed 2026-10-16 · environments: desktop, web · topics: evaluation, science, workflow - Published: 2024-07-15 · Authors: Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, Tao Yu ### SPIDES: Information-Flow Control for Browser Agents SPIDES propagates HTML integrity and confidentiality labels into agent context and mediates browser actions with information-flow policies and approval gates, evaluated in the labeled MicroNet ecosystem. - https://awesomecua.com/resources/arxiv-2610-24277/ · source: https://arxiv.org/abs/2610.24277v1 - Paper · reviewed 2026-10-20 · environments: web · topics: safety-security, web-navigation, human-in-the-loop, evaluation - Published: 2026-10-16 · Authors: Egor Zverev, Juhee Kim, Leon Trampert, Boris Köpf, Aashish Kolluri, Shruti Tople, Santiago Zanella-Béguelin ### SPILLage: Agentic Oversharing on the Web Formalizes and measures natural agentic oversharing when browser agents handle private user emails and calendars across live websites. - https://awesomecua.com/resources/arxiv-2602-13516/ · source: https://arxiv.org/abs/2602.13516v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2026-02-13 · Authors: Jaechul Roh, Eugene Bagdasarian, Hamed Haddadi, Ali Shahin Shamsabadi ### Spine-Branch Coordination for Multi-agent Computer Use Introduces Spine-Branch Coordination, which assigns a main task flow to one virtual-machine state and uses parallel branch machines to gather information for computer-use agents. - https://awesomecua.com/resources/arxiv-2608-22077/ · source: https://arxiv.org/abs/2608.22077v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: planning, long-horizon, sandbox, efficiency - Published: 2026-08-22 · Authors: Mian Zhang, Manasi Sharma, Sheng Zhang, Minglai Yang, Kejian Shi, Ying Liu, Zhiyu Zoey Chen, Daniel Yue Zhang ### SpiritSight Agent: Advanced GUI Agent with One Look Introduces SpiritSight and GUI-Lasagne data, combining multilevel GUI annotations with visual block parsing for high-resolution interface navigation. - https://awesomecua.com/resources/arxiv-2503-03196/ · source: https://arxiv.org/abs/2503.03196v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, screen-understanding, data-generation - Published: 2025-03-05 · Authors: Zhiyuan Huang, Ziming Cheng, Junting Pan, Zhaohui Hou, Mingjie Zhan ### Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus Introduces Spotlight, a vision-language model conditioned on a UI screenshot and a selected region of interest. It evaluates screenshot-based mobile UI understanding and explores multitask learning and few-shot prompting. - https://awesomecua.com/resources/arxiv-2209-14927/ · source: https://arxiv.org/abs/2209.14927v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding - Published: 2022-09-29 · Authors: Gang Li, Yang Li ### Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning Spreadsheet-RL trains spreadsheet agents with RL in a Microsoft Excel environment exposed through a Python sandbox and supplies the Domain-Spreadsheet benchmark. - https://awesomecua.com/resources/arxiv-2605-22642/ · source: https://arxiv.org/abs/2605.22642v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, automation, evaluation, tool-integration - Published: 2026-05-21 · Authors: Banghao Chi, Yining Xie, Mingyuan Wu, Jingcheng Yang, Jize Jiang, Zhaoheng Li, Shengyi Qian, Minjia Zhang, Klara Nahrstedt, Rui Hou, Xiangjun Fan, Hanchao Yu ### SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows Introduces SpreadsheetBench 2, a benchmark of multi-sheet business spreadsheet workflows for agents, covering generation, debugging, and visualization tasks. - https://awesomecua.com/resources/arxiv-2606-29955/ · source: https://arxiv.org/abs/2606.29955v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, workflow, industry - Published: 2026-06-29 · Authors: Jian Zhu, Yuzheng Zhang, Zeyao Ma, Bohan Zhang, Armin Schoepf, Daniel Woloch, Peter Yiliu Wang, Guangyu Robert Yang, Samuel Jacob, Siddharth Nagisetty, Abhiram Chundru, Jean Lin, Spencer Mateega, Jing Zhang ### SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation SpreadsheetBench evaluates LLM spreadsheet manipulation with 912 questions from Excel forums and multiple spreadsheet files as test cases for each instruction. - https://awesomecua.com/resources/arxiv-2406-14991/ · source: https://arxiv.org/abs/2406.14991v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, automation - Published: 2024-06-21 · Authors: Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, Jie Tang ### SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization SSL uses progressively tiered rewards for agent training, including distance-based guidance on GUI perception tasks alongside planning and reasoning tasks. - https://awesomecua.com/resources/arxiv-2601-22491/ · source: https://arxiv.org/abs/2601.22491v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, reward-models, efficiency - Published: 2026-01-30 · Authors: Jinyang Wu, Changpeng Yang, Yuhao Shen, Fangzhi Xu, Bolin Ni, Chonghua Liao, Yuchen Liu, Hongzhen Wang, Shuai Nie, Shuai Zhang, Haoran Luo, Jiaming Xu ### ST-Lite: Training-Free KV Cache Compression with Spatio-Trajectory Guidance for Long-Horizon GUI Agents Introduces ST-Lite, a training-free KV cache compression scheme for vision-language GUI agents that filters redundant frames and preserves small UI elements in interaction traces. - https://awesomecua.com/resources/arxiv-2603-00188/ · source: https://arxiv.org/abs/2603.00188v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, interaction-history, screen-understanding, training-free - Published: 2026-02-27 · Authors: Bowen Zhou, Zhou Xu, Wanli Li, Jingyu Xiao, Pingan Gan, Haoqian Wang ### ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents Evaluates web-task completion under explicit enterprise policies, measuring policy-compliant completion and safety violations separately. - https://awesomecua.com/resources/arxiv-2410-06703/ · source: https://arxiv.org/abs/2410.06703v7 - Paper · reviewed 2026-10-16 · environments: web · topics: safety-security, evaluation - Published: 2024-10-09 · Authors: Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, Nir Mashkif, Segev Shlomov ### StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents StainFlow tracks visually verifiable task entities through GUI trajectories to identify phases and retrieve focused evidence windows. Its process-reward model is tested for both online policy learning and trajectory-completion judgments. - https://awesomecua.com/resources/arxiv-2606-07027/ · source: https://arxiv.org/abs/2606.07027v2 - Paper · reviewed 2026-10-18 · environments: android · topics: reward-models, state-tracking, reinforcement-learning, evaluation - Published: 2026-06-05 · Authors: Haojie Hao, Longkun Hao, Yihang Lou, Yan Bai, Zhenyang Li, Zhichao Yang, Dongshuo Huang, Hongyu Lin, Lanqing Hong, Jiakai Wang, Xianglong Liu ### STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments STAMP trains explicit memory for mobile GUI agents in virtual environments by injecting facts into generated tasks and using supervised data and online RL. - https://awesomecua.com/resources/arxiv-2605-29324/ · source: https://arxiv.org/abs/2605.29324v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, mobile-navigation, data-generation, reinforcement-learning - Published: 2026-05-28 · Authors: Junyang Wang, Haiyang Xu, Xi Zhang, Zhaoqing Zhu, Ming Yan, Jieping Ye, Jitao Sang ### STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models STaR-KV compresses GUI-agent KV caches by calibrating visual-token importance across attention subspaces and interaction time. Its training-free scoring combines spatial specialization, temporal redundancy and an adaptive temperature, evaluated at matched cache budgets. - https://awesomecua.com/resources/arxiv-2606-01790/ · source: https://arxiv.org/abs/2606.01790v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: efficiency, training-free, screen-understanding, interaction-history - Published: 2026-06-01 · Authors: Yuhang Han, Wenzheng Yang, Yujie Chen, Xiangqi Jin, Yaojie Zhang, Siteng Huang, Linfeng Zhang ### State Your Intention to Steer Your Attention: An AI Assistant for Intentional Digital Living Introduces an assistant that reads screenshots, application titles, and URLs to compare computer activity with a user's stated intention and send notifications when it diverges. - https://awesomecua.com/resources/arxiv-2510-14513/ · source: https://arxiv.org/abs/2510.14513v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, human-in-the-loop, state-tracking - Published: 2025-10-16 · Authors: Juheon Choi, Juyong Lee, Jian Kim, Chanyoung Kim, Taywon Min, W. Bradley Knox, Min Kyung Lee, Kimin Lee ### StateAct: Enhancing LLM Base Agents via Self-prompting and State-tracking Maintains explicit task goals and evolving state descriptions to improve base-agent decisions across interactive tasks, including web shopping. - https://awesomecua.com/resources/arxiv-2410-02810/ · source: https://arxiv.org/abs/2410.02810v3 - Paper · reviewed 2026-10-16 · environments: web · topics: planning, memory - Published: 2024-09-21 · Authors: Nikolai Rozanov, Marek Rei ### StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents Introduces StateAct, a code-first computer-use harness that manipulates program state directly and delegates screenshot-and-click subgoals to a GUI agent. An independent completion gate checks saved outputs, while fresh subagents handle subgoals to limit accumulated context. - https://awesomecua.com/resources/arxiv-2607-22798/ · source: https://arxiv.org/abs/2607.22798v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: tool-integration, state-tracking, long-horizon - Published: 2026-07-24 · Authors: Yan Yang, Xiangru Jian, Ziyang Luo, Zirui Zhao, Yutong Dai, Ziji Shi, Hanshu Yan, Jun Hao Liew, Silvio Savarese, Junnan Li ### StateFork: Branchable Infrastructure for Agent Exploration Introduces StateFork and Waypoint for branching and restoring terminal-agent sessions so agents can explore multiple paths while preserving execution state. - https://awesomecua.com/resources/arxiv-2609-38648/ · source: https://arxiv.org/abs/2609.38648v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: sandbox, planning, tool-integration - Published: 2026-09-29 · Authors: Jiakai Xu, Tianle Zhou, Georgios Liargkovas, Danielle Gillai, Ruizhe Fu, Patrick Shen, Eugene Wu, Kostis Kaffes ### StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling Introduces StateM, a stateful runtime with checked transitions and reusable runbooks for long-horizon agents; its reported evaluations use terminal and business tasks. - https://awesomecua.com/resources/arxiv-2608-15089/ · source: https://arxiv.org/abs/2608.15089v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: state-tracking, memory, workflow, long-horizon - Published: 2026-08-15 · Authors: Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, Kai Wang ### Steering Black-Box Large Language Model Agents with Pitfall Anticipation World Model PAWM predicts likely multi-step pitfalls and selectively guides black-box agents; its evaluations include a web-agent benchmark. - https://awesomecua.com/resources/arxiv-2610-29139/ · source: https://arxiv.org/abs/2610.29139v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: world-models, planning, robustness - Published: 2026-10-21 · Authors: Jiongxiao Wang, Han Ding, Youzhi Luo, Yawei Wang, Yun Zhou, Linbo Liu, Panpan Xu ### Step-Aligned Script Generation with Verification and Recovery for Web Automation Testing Presents TESSARA, which turns natural-language web test intent into Playwright scripts, linking each browser interaction to selector evidence, outcomes, and repair history. - https://awesomecua.com/resources/arxiv-2610-25601/ · source: https://arxiv.org/abs/2610.25601v1 - Paper · reviewed 2026-11-06 · environments: web · topics: automation, workflow, self-reflection - Published: 2026-10-19 · Authors: Hridya Dhulipala, Xiaokai Rong, Aashish Yadavally, Tien N. Nguyen ### Step-GUI Technical Report Step-GUI uses calibrated trajectory rewards to turn generated GUI interaction data into training signals for compact agent models. The report also introduces GUI-MCP for hierarchical GUI operations and local task delegation, and AndroidDaily for static-action and end-to-end mobile evaluation. - https://awesomecua.com/resources/arxiv-2512-15431/ · source: https://arxiv.org/abs/2512.15431v2 - Paper · reviewed 2026-10-17 · environments: android, desktop · topics: data-generation, reward-models, tool-integration, evaluation, mobile-navigation - Published: 2025-12-17 · Authors: Haolong Yan, Jia Wang, Xin Huang, Yeqing Shen, Ziyang Meng, Zhimin Fan, Kaijun Tan, Jin Gao, Lieyu Shi, Mi Yang, Shiliang Yang, Zhirui Wang, Brian Li, Kang An, Chenyang Li, Lei Lei, Mengmeng Duan, Danxun Liang, Guodong Liu, Hang Cheng, Hao Wu, Jie Dong, Junhao Huang, Mei Chen, Renjie Yu, Shunshan Li, Xu Zhou, Yiting Dai, Yineng Deng, Yingdan Liang, Zelin Chen, Wen Sun, Chengxu Yan, Chunqin Xu, Dong Li, Fengqiong Xiao, Guanghao Fan, Guopeng Li, Guozhen Peng, Hongbing Li, Hang Li, Hongming Chen, Jingjing Xie, Jianyong Li, Jingyang Zhang, Jiaju Ren, Jiayu Yuan, Jianpeng Yin, Kai Cao, Liang Zhao, Liguo Tan, Liying Shi, Mengqiang Ren, Min Xu, Manjiao Liu, Mao Luo, Mingxin Wan, Na Wang, Nan Wu, Ning Wang, Peiyao Ma, Qingzhou Zhang, Qiao Wang, Qinlin Zeng, Qiong Gao, Qiongyao Li, Shangwu Zhong, Shuli Gao, Shaofan Liu, Shisi Gao, Shuang Luo, Xingbin Liu, Xiaojia Liu, Xiaojie Hou, Xin Liu, Xuanti Feng, Xuedan Cai, Xuan Wen, Xianwei Zhu, Xin Liang, Xin Liu, Xin Zhou, Yifan Sui, Yingxiu Zhao, Yukang Shi, Yunfang Xu, Yuqing Zeng, Yixun Zhang, Zejia Weng, Zhonghao Yan, Zhiguo Huang, Zhuoyu Wang, Zihan Yan, Zheng Ge, Jing Li, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, Daxin Jiang ### Step-level Optimization for Efficient Computer-use Agents Proposes a step-level model cascade that runs a smaller computer-use policy until learned monitors flag stalled progress or semantic drift. Separate history-based stuck detection and milestone checks determine when to escalate for recovery or verification without retraining the larger model. - https://awesomecua.com/resources/arxiv-2604-27151/ · source: https://arxiv.org/abs/2604.27151v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: efficiency, robustness, state-tracking, interaction-history, long-horizon - Published: 2026-04-29 · Authors: Jinbiao Wei, Kangqi Ni, Yilun Zhao, Guo Gan, Arman Cohan ### SteP: Stacked LLM Policies for Web Actions SteP dynamically composes specialized web-action policies through a stack that records the active chain of policy calls. It studies this control structure across WebArena, MiniWoB++, and CRM workflows. - https://awesomecua.com/resources/arxiv-2310-03720/ · source: https://arxiv.org/abs/2310.03720v4 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, web-navigation, workflow - Published: 2023-10-05 · Authors: Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, Ryan McDonald ### STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimization STEP allocates RL sampling by task success rate and trains on individual steps from interaction trajectories, with evaluations on OSWorld and AndroidWorld. - https://awesomecua.com/resources/arxiv-2511-13091/ · source: https://arxiv.org/abs/2511.13091v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, efficiency, long-horizon - Published: 2025-11-17 · Authors: Yuhan Chen, Yuxuan Liu, Long Zhang, Pengzhi Gao, Jian Luan, Wei Liu ### StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection Introduces StepJack, a benchmark of indirect prompt injections that split an adversarial goal across pages on a computer-use agent's web navigation path. - https://awesomecua.com/resources/arxiv-2608-06477/ · source: https://arxiv.org/abs/2608.06477v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, evaluation, web-navigation, long-horizon - Published: 2026-08-06 · Authors: Zhuoxin Zhan, Akbar Rafiey, Avery Ma, Leila Pishdad, Layla El Asri ### StepReflect: Structured UI Transition Reflection for Mobile GUI Agents StepReflect predicts structured per-action reflections from transition specifications and paired visual evidence. The paper distinguishes offline transition-label accuracy from online task success when the reflection model is integrated into mobile agents. - https://awesomecua.com/resources/arxiv-2608-05587/ · source: https://arxiv.org/abs/2608.05587v1 - Paper · reviewed 2026-10-17 · environments: android · topics: self-reflection, state-tracking, imitation-learning, reward-models, evaluation, efficiency - Published: 2026-08-06 · Authors: Linqiang Guo, Wei Liu, Li Gu, Yang Wang, Tse-Hsun, Chen ### StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design Introduces StepX-Edge, an on-device UI vision-language model for screen understanding, OCR, question answering, and element grounding on mobile chips. - https://awesomecua.com/resources/arxiv-2607-22708/ · source: https://arxiv.org/abs/2607.22708v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, grounding, efficiency - Published: 2026-07-20 · Authors: Yin Wang, Haotian Hu, Jineng Han, Wentao Qiu, Zhenhua Ge, Liujian Tang, Fanyi Wang ### STEVE: A Step Verification Pipeline for Computer-use Agent Training Labels individual computer-use actions by comparing surrounding screenshots, then trains agents from both successful and unsuccessful steps. - https://awesomecua.com/resources/arxiv-2503-12532/ · source: https://arxiv.org/abs/2503.12532v2 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: data-generation, evaluation, reinforcement-learning - Published: 2025-03-16 · Authors: Fanbin Lu, Zhisheng Zhong, Ziqin Wei, Shu Liu, Chi-Wing Fu, Jiaya Jia ### Steward: Natural Language Web Automation Introduces Steward, a natural-language browser automation system that repeatedly plans and executes website actions. The paper examines completion detection, state representation, responsiveness, and caching in real-site tasks. - https://awesomecua.com/resources/arxiv-2409-15441/ · source: https://arxiv.org/abs/2409.15441v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, automation, efficiency - Published: 2024-09-23 · Authors: Brian Tang, Kang G. Shin ### STP-GUI: Spatially-Aware Token Pruning for Efficient Web-GUI Agents Introduces STP-GUI, which selects spatially coherent screenshot windows conditioned on the instruction to reduce visual tokens while retaining useful interface structure. - https://awesomecua.com/resources/arxiv-2610-20390/ · source: https://arxiv.org/abs/2610.20390v1 - Paper · reviewed 2026-10-17 · environments: web, android · topics: efficiency, grounding, screen-understanding - Published: 2026-10-14 · Authors: Shuo Wang, Sifan Zhou, Xiaobo Wang, Tianyu Fu, Chenxu Zhao, Anyang Su, Hanning Wang, Ping Jiang, Minghui Wu ### StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability Introduces a web-agent stress-testing benchmark with controlled changes to layouts, interaction semantics, and execution for diagnosing behavior under varied conditions. - https://awesomecua.com/resources/arxiv-2604-16385/ · source: https://arxiv.org/abs/2604.16385v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, robustness, web-navigation - Published: 2026-03-27 · Authors: Haoyue Bai, Dong Wang, Long Chen, Bingguang Hao, Pengyang Shao, Yonghui Yang, Yicheng He, Chenyi Zhuang ### StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure StructAgent maintains compact task state and regulates long-horizon execution through verified transitions and progress checkpoints. Its workflow supports evidence-based completion and targeted recovery, with desktop computer-use evaluation and a separate Minecraft extension. - https://awesomecua.com/resources/arxiv-2607-11388/ · source: https://arxiv.org/abs/2607.11388v1 - Paper · reviewed 2026-10-18 · environments: desktop · topics: state-tracking, planning, long-horizon, robustness - Published: 2026-07-13 · Authors: Wenyi Wu, Sibo Zhu, Kun Zhou, Aayush Salvi, Zixuan Song, Biwei Huang ### Structured Distillation of Web Agent Capabilities Enables Generalization Introduces Agent-as-Annotators, which uses modular LLM roles to synthesize and filter web-agent trajectories for supervised training of a smaller web agent. - https://awesomecua.com/resources/arxiv-2604-07776/ · source: https://arxiv.org/abs/2604.07776v1 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, imitation-learning, web-navigation - Published: 2026-04-09 · Authors: Xing Han Lù, Siva Reddy ### Structured Interaction, Visual Localization, and Robust Execution for Complex Web Tasks: A Technical Report on the WebRetriever Challenge Describes a WebRetriever agent that combines semantic webpage operations with visual localization when structured representations are insufficient. Hierarchical context management and fault-aware execution support processing tasks across concurrent browsers. - https://awesomecua.com/resources/arxiv-2609-35904/ · source: https://arxiv.org/abs/2609.35904v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, grounding, robustness, memory - Published: 2026-09-28 · Authors: Ziqi Zhang, Shaohui Li, Bing Li ### STRUCTUREDAGENT: Planning with AND/OR Trees for Long-Horizon Web Tasks Introduces STRUCTUREDAGENT, a web-agent planning framework that interleaves execution with dynamic AND/OR trees, supporting node repair and exploration of alternative plans. - https://awesomecua.com/resources/arxiv-2603-05294/ · source: https://arxiv.org/abs/2603.05294v3 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, long-horizon, web-navigation - Published: 2026-03-05 · Authors: ELita Lobo, Xu Chen, Jingjing Meng, Nan Xi, Yang Jiao, Chirag Agarwal, Yair Zick, Yan Gao ### Structuring GUI Elements through Vision Language Models: Towards Action Space Generation Introduces IAML training, which augments coordinate samples to help multimodal models locate GUI elements from screen content and user instructions. - https://awesomecua.com/resources/arxiv-2508-16271/ · source: https://arxiv.org/abs/2508.16271v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, screen-understanding, data-generation - Published: 2025-08-22 · Authors: Yi Xu, Yesheng Zhang, Jiajia Liu, Jingdong Chen ### Subgoal-aware Memory Graph for GUI Agent Proposes a subgoal-aware memory graph that turns past mobile GUI agent trajectories into reusable evidence for decisions across tasks. - https://awesomecua.com/resources/arxiv-2610-31307/ · source: https://arxiv.org/abs/2610.31307v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, mobile-navigation, long-horizon, interaction-history - Published: 2026-10-22 · Authors: Dezhao Luo, Daniel Beechey, Derek Yuen, Weilin Luo ### Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control Succeed or Learn Slowly applies direct off-policy updates to successful interactions and more conservative regularized updates to negative examples. Successful Transition Replay prioritizes successful experience during policy learning, with evaluation on AndroidWorld. - https://awesomecua.com/resources/arxiv-2509-01720/ · source: https://arxiv.org/abs/2509.01720v2 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, mobile-navigation, efficiency - Published: 2025-09-01 · Authors: Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, Kun Shao ### sudo rm -rf agentic_security Evaluates security failures of computer-use agents under adversarial requests and screen-context manipulation. The study examines refusal behavior while agents perform tasks in desktop and web environments. - https://awesomecua.com/resources/arxiv-2503-20279/ · source: https://arxiv.org/abs/2503.20279v3 - Paper · reviewed 2026-10-18 · environments: desktop, web · topics: safety-security, evaluation - Published: 2025-03-26 · Authors: Sejin Lee, Jian Kim, Haon Park, Ashkan Yousefpour, Sangyoon Yu, Min Song ### SUGILITE: Creating Multimodal Smartphone Automation by Demonstration Introduces SUGILITE, which combines spoken instructions, demonstrated Android actions, and UI hierarchies to generate reusable automation scripts. Its interface includes context checking and repairs for unfamiliar situations and changing app interfaces. - https://awesomecua.com/resources/url-392a602c660ffa91be46/ · source: https://doi.org/10.1145/3025453.3025483 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, recording, human-in-the-loop, cross-app - Published: 2017 · Authors: Toby Jia-Jun Li, Amos Azaria, Brad A. Myers ### Surfer 2: The Next Generation of Cross-Platform Computer Use Agents Surfer2 combines hierarchical context management with separate planning and execution, self-verification, and recovery for visual computer use. It evaluates the framework across web, desktop, and Android tasks without task-specific fine-tuning. - https://awesomecua.com/resources/arxiv-2510-19949/ · source: https://arxiv.org/abs/2510.19949v2 - Paper · reviewed 2026-10-17 · environments: web, desktop, android · topics: planning, memory, long-horizon, robustness - Published: 2025-10-22 · Authors: Mathieu Andreux, Märt Bakler, Yanael Barbier, Hamza Benchekroun, Emilien Biré, Antoine Bonnet, Riaz Bordie, Nathan Bout, Matthias Brunel, Aleix Cambray, Pierre-Louis Cedoz, Antoine Chassang, Gautier Cloix, Ethan Connelly, Alexandra Constantinou, Ramzi De Coster, Hubert de la Jonquiere, Aurélien Delfosse, Maxime Delpit, Alexis Deprez, Augustin Derupti, Mathieu Diaz, Shannon D'Souza, Julie Dujardin, Abai Edmund, Michael Eickenberg, Armand Fatalot, Wissem Felissi, Isaac Herring, Xavier Koegler, Erwan Le Jumeau de Kergaradec, Aurélien Lac, Maxime Langevin, Corentin Lauverjat, Antonio Loison, Avshalom Manevich, Axel Moyal, Axel Nguyen Kerbel, Marinela Parovic, Julien Revelle, Guillaume Richard, Mats Richter, Ronan Riochet, María Santos, Romain Savidan, Laurent Sifre, Maxime Theillard, Marc Thibault, Ivan Valentini, Tony Wu, Laura Yie, Kai Yuan, Jevgenij Zubovskij ### Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights Describes Surfer-H, a web agent paired with the Holo1 vision-language model family trained on web content, synthetic examples, and agent-generated data. The work introduces WebClick for web-interface localization and evaluates browser task execution on WebVoyager. - https://awesomecua.com/resources/arxiv-2506-02865/ · source: https://arxiv.org/abs/2506.02865v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, grounding, data-generation, evaluation - Published: 2025-06-03 · Authors: Mathieu Andreux, Breno Baldas Skuk, Hamza Benchekroun, Emilien Biré, Antoine Bonnet, Riaz Bordie, Nathan Bout, Matthias Brunel, Pierre-Louis Cedoz, Antoine Chassang, Mickaël Chen, Alexandra D. Constantinou, Antoine d'Andigné, Hubert de La Jonquière, Aurélien Delfosse, Ludovic Denoyer, Alexis Deprez, Augustin Derupti, Michael Eickenberg, Mathïs Federico, Charles Kantor, Xavier Koegler, Yann Labbé, Matthew C. H. Lee, Erwan Le Jumeau de Kergaradec, Amir Mahla, Avshalom Manevich, Adrien Maret, Charles Masson, Rafaël Maurin, Arturo Mena, Philippe Modard, Axel Moyal, Axel Nguyen Kerbel, Julien Revelle, Mats L. Richter, María Santos, Laurent Sifre, Maxime Theillard, Marc Thibault, Louis Thiry, Léo Tronchon, Nicolas Usunier, Tony Wu ### SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agents Introduces SusBench, an online benchmark that inserts dark-pattern designs into consumer websites for agent evaluation. A human study checks the realism of the interventions, and agent trials compare susceptibility across pattern types. - https://awesomecua.com/resources/arxiv-2510-11035/ · source: https://arxiv.org/abs/2510.11035v2 - Paper · reviewed 2026-10-17 · environments: web · topics: ui-design, safety-security, evaluation - Published: 2025-10-13 · Authors: Longjie Guo, Chenjie Yuan, Mingyuan Zhong, Robert Wolfe, Ruican Zhong, Yue Xu, Bingbing Wen, Hua Shen, Lucy Lu Wang, Alexis Hiniker ### SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis Introduces SwipeGen for synthesizing varied swipe trajectories and SwipeBench for assessing their execution quality. The study examines how richer control of trajectory, speed, and timing changes GUI agents’ interaction performance. - https://awesomecua.com/resources/arxiv-2601-18305/ · source: https://arxiv.org/abs/2601.18305v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: action-prediction, data-generation, evaluation - Published: 2026-01-26 · Authors: Xuan Wang, Siyuan Su, Quantong Fu, Yongxiang Hu, Yangfan Zhou ### SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control Introduces SWIRL, staged interleaved RL for mobile GUI control, pairing a Navigator that plans from screens and language with an Interactor that produces actions. - https://awesomecua.com/resources/arxiv-2508-20018/ · source: https://arxiv.org/abs/2508.20018v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, planning, grounding, mobile-navigation - Published: 2025-08-27 · Authors: Quanfeng Lu, Zhantao Ma, Shuai Zhong, Jin Wang, Dahai Yu, Michael K. Ng, Ping Luo ### Syll: Open-Source Personal Automation with Cross-Surface Execution Syll combines tool calls, shell execution and visual GUI control in a personal-automation runtime. It turns user demonstrations into reusable skills and exposes execution logs, keyframes and approval checkpoints for inspection. - https://awesomecua.com/resources/arxiv-2606-07594/ · source: https://arxiv.org/abs/2606.07594v1 - Paper · reviewed 2026-10-17 · environments: desktop, macos · topics: automation, human-in-the-loop, workflow, memory, recording, observability - Published: 2026-05-28 · Authors: Bo Zhang, Borui Zhang, Chenghao Jiang, Minglei Shi, Xiaofeng Wang, Zheng Zhu, Jie Zhou, Jiwen Lu ### Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs Couples large- and small-model web-agent exploration with trajectory synthesis and distillation to improve both model types iteratively. - https://awesomecua.com/resources/arxiv-2502-07942/ · source: https://arxiv.org/abs/2502.07942v2 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation, efficiency - Published: 2025-02-11 · Authors: Ruichen Zhang, Mufan Qiu, Zhen Tan, Mohan Zhang, Vincent Lu, Jie Peng, Kaidi Xu, Leandro Z. Agudelo, Peter Qian, Tianlong Chen ### Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control Synapse filters task-irrelevant page state and places complete abstracted state-action trajectories into the prompt. An embedding-based exemplar memory retrieves relevant demonstrations for new tasks, evaluated on MiniWoB++ and Mind2Web. - https://awesomecua.com/resources/arxiv-2306-07863/ · source: https://arxiv.org/abs/2306.07863v3 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, web-navigation, planning, efficiency - Published: 2023-06-13 · Authors: Longtao Zheng, Rundong Wang, Xinrun Wang, Bo An ### Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale Synatra converts indirect procedural knowledge, including human-oriented tutorials, into direct demonstrations for digital agents. It trains a 7B CodeLlama on 100,000 synthesized demonstrations and evaluates the resulting policy on Mind2Web, MiniWoB++, and WebArena. - https://awesomecua.com/resources/arxiv-2409-15637/ · source: https://arxiv.org/abs/2409.15637v2 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, imitation-learning, web-navigation - Published: 2024-09-24 · Authors: Tianyue Ou, Frank F. Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, Shuyan Zhou ### SynChain: Inducing Computer-Use Agent Systems to Construct Their Own Attack Chains Introduces SynChain, an attack that induces persistent agents to create poisoned skills or memory entries, and CUAChain for evaluating later attack-chain activation. - https://awesomecua.com/resources/arxiv-2608-06862/ · source: https://arxiv.org/abs/2608.06862v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, memory, evaluation, workflow - Published: 2026-08-07 · Authors: Fuyao Zhang, Jiaming Zhang, Che Wang, Boyang Chen, Yurong Hao, Xiongtao Sun, Guowei Guan, Blaise Delattre, Yang Cao, Wei Yang Bryan Lim ### SynthAgent: Adapting Web Agents with Synthetic Supervision Introduces SynthAgent, which generates tasks by exploring website elements and refines them when observations reveal conflicts. A later trajectory-refinement stage removes misaligned behavior before fine-tuning agents for the target websites. - https://awesomecua.com/resources/arxiv-2511-06101/ · source: https://arxiv.org/abs/2511.06101v3 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, imitation-learning, web-navigation - Published: 2025-11-08 · Authors: Zhaoyang Wang, Yiming Liang, Xuchao Zhang, Qianhui Wu, Siwei Han, Anson Bastos, Rujia Wang, Chetan Bansal, Baolin Peng, Jianfeng Gao, Saravan Rajmohan, Huaxiu Yao ### SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents SynWeaver explores a website to build a map of page states and executable interactions, then learns website priors for task proposals. Joint task/trajectory synthesis revises inconsistent pairs and verifies or repairs the resulting supervision. - https://awesomecua.com/resources/arxiv-2608-12429/ · source: https://arxiv.org/abs/2608.12429v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, web-navigation, state-tracking, robustness - Published: 2026-08-12 · Authors: Ruitao Wang, Yuwen Hao, Menglin Yang ### Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agents Develops a threat taxonomy and evaluation framework for attacks against multimodal mobile GUI agents, with case studies and dataset experiments. - https://awesomecua.com/resources/arxiv-2407-09295/ · source: https://arxiv.org/abs/2407.09295v3 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: safety-security, evaluation - Published: 2024-07-12 · Authors: Yulong Yang, Xinshan Yang, Shuaidong Li, Chenhao Lin, Zhengyu Zhao, Chao Shen, Tianwei Zhang ### Tactile: Giving Computer-Using Agents Hands and Feet Tactile turns accessibility, OCR and visual evidence into explicit desktop targets with executable actions and verification cues. Its execution loop chooses semantic actions or coordinate fallbacks while retaining provenance for replay and diagnosis. - https://awesomecua.com/resources/arxiv-2607-14443/ · source: https://arxiv.org/abs/2607.14443v1 - Paper · reviewed 2026-10-17 · environments: desktop, macos · topics: grounding, automation, observability, tool-integration, robustness, evaluation - Published: 2026-07-16 · Authors: Yong Liu, Zhenyi Zhong, Zhanpeng Shi ### Tail-Likelihood Reinforcement Learning Introduces Tail-Likelihood Reinforcement Learning (TailRL), an RL objective that favors rare high-reward rollouts; GUI grounding is one of several evaluation tasks. - https://awesomecua.com/resources/arxiv-2609-02987/ · source: https://arxiv.org/abs/2609.02987v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, reward-models, grounding - Published: 2026-09-02 · Authors: Shrinivas Ramasubramanian, Daman Arora, Fahim Tajwar, Guanning Zeng, Qingyang Wu, Zhongzhu Zhou, Chenfeng Xu, Haiwen Feng, Yuda Song, Aarti Singh, Ruslan Salakhutdinov, J. Andrew Bagnell, Jeff Schneider, Andrea Zanette ### Talk2Agent: Benchmarking Voice Interfaces for Text Agents Introduces Talk2Agent, a benchmark of spoken instructions for computer-use tasks that measures whether voice interfaces preserve task-critical information. It pairs human speech with an execution-free intention-based metric and checks how that metric relates to downstream task completion. - https://awesomecua.com/resources/arxiv-2609-38867/ · source: https://arxiv.org/abs/2609.38867v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, robustness - Published: 2026-09-30 · Authors: Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang ### Tandem: Serving Hybrid Linear-Attention GUI Agents on Mobile NPUs Tandem serves hybrid GUI-agent models on Android NPUs using divergence-point state checkpoints, rollback-aware speculative decoding and idle power release between request bursts. - https://awesomecua.com/resources/arxiv-2610-26660/ · source: https://arxiv.org/abs/2610.26660v1 - Paper · reviewed 2026-10-20 · environments: android · topics: efficiency, mobile-navigation - Published: 2026-10-19 · Authors: Zhuomin Chen, Jingchao Ni, Xu Zheng, Yong Chen, Mo Sha, Wei Cheng, Dongsheng Luo ### Task-Adaptive Rubrics for GUI Reward Modeling Introduces AdaptRubric, which retrieves reusable criteria for a GUI task family and then generates instruction-specific checks for concrete values and constraints. It evaluates these adaptive rubrics in offline reward assessment and online agent optimization. - https://awesomecua.com/resources/arxiv-2608-24174/ · source: https://arxiv.org/abs/2608.24174v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: reward-models, evaluation, reinforcement-learning - Published: 2026-08-25 · Authors: Tao Xiong, Xavier Hu, Wenkai Wang, Qinzhuo Wu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, Shengyu Zhang ### Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution Presents TaskVM, an interface for specifying desired and preserved application state, checking agent actions, and exposing whether changes can be reversed or compensated. - https://awesomecua.com/resources/arxiv-2610-20100/ · source: https://arxiv.org/abs/2610.20100v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, safety-security, workflow - Published: 2026-10-14 · Authors: Wenkui Yang, Yiming Hao, Kai Zhou, Chengyu Pu, Yuzhe Zhang, Jie Cao, Jin Huang, Shuai Ma, Feng Tian ### TaskSense: Cognitive Chain Modeling and Difficulty Estimation for GUI Tasks Cognitive Chain estimates GUI task difficulty by extracting sequences of cognitive steps from execution traces, then uses those estimates to assess GUI agents. - https://awesomecua.com/resources/arxiv-2511-09309/ · source: https://arxiv.org/abs/2511.09309v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, observability, data-generation - Published: 2025-11-12 · Authors: Yiwen Yin, Zhian Hu, Xiaoxi Xu, Chun Yu, Xintong Wu, Wenyu Fan, Yuanchun Shi ### TClone: Low-Latency Forking of Live GUI Environments for Computer-Use Agents TClone versions live GUI workspaces through snapshots, isolated forks, rollback and selective commit or merge. It separates fast branch creation from durable checkpoints using copy-on-write sharing, filesystem versioning and asynchronous checkpointing, with end-to-end agent-loop latency measurements. - https://awesomecua.com/resources/arxiv-2605-17320/ · source: https://arxiv.org/abs/2605.17320v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: sandbox, efficiency, safety-security, automation - Published: 2026-05-17 · Authors: Yutong Huang, Vikranth Srivatsa, Alex Asch, Hansin Tushar Patwa, Yiying Zhang ### Teach it to stop, not just to click Studies verifier-guided repair of a 35B computer-use agent across five environments, examining variation across runs and data draws in done detection, GUI clicks, and field entry. - https://awesomecua.com/resources/arxiv-2607-17136/ · source: https://arxiv.org/abs/2607.17136v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, reinforcement-learning, grounding, robustness - Published: 2026-07-19 · Authors: Barada Sahu, Shivesh Pandey ### Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents Introduces WebKD, which repairs failed student web-agent trajectories with minimal teacher edits and trains on corrections that preserve the student's reasoning style. - https://awesomecua.com/resources/arxiv-2610-19834/ · source: https://arxiv.org/abs/2610.19834v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, web-navigation, efficiency - Published: 2026-10-14 · Authors: Md Farhan Ishmam, Kenneth Marino ### TeleTune: Evolving Agent Skills From Offline Telemetry Introduces TeleTune, which edits a textual skill library using held-out action prediction on offline software-use logs with missing goals or interleaved tasks. The learned workflows also guide retrieval of demonstrations for subsequent web-agent execution. - https://awesomecua.com/resources/arxiv-2610-05437/ · source: https://arxiv.org/abs/2610.05437v2 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, workflow, efficiency - Published: 2026-10-04 · Authors: Justin Chih-Yao Chen, Elias Stengel-Eskin, Yan Chen, Pol Llado, Scott Counts, Mohit Bansal, Benjamin Van Durme, Harsh Jhamtani, Gaurav Verma ### Tell Me What's Next: Textual Foresight for Generic UI Representations Textual Foresight learns UI representations by describing the future screen expected after an action on the current interface. The OpenApp dataset supports pretraining and downstream evaluation of representations that connect individual elements with whole-screen changes. - https://awesomecua.com/resources/arxiv-2406-07822/ · source: https://arxiv.org/abs/2406.07822v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, world-models, data-generation - Published: 2024-06-12 · Authors: Andrea Burns, Kate Saenko, Bryan A. Plummer ### Temporal UI State Inconsistency in Desktop GUI Agents: Formalizing and Defending Against TOCTOU Attacks on Computer-Use Agents This study examines UI changes between a desktop agent’s screenshot observation and action dispatch. Its pre-execution state checks address overlay and focus manipulation, while the reported DOM-injection blind spot limits the defense’s coverage. - https://awesomecua.com/resources/arxiv-2604-18860/ · source: https://arxiv.org/abs/2604.18860v1 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: safety-security, robustness, evaluation, state-tracking - Published: 2026-04-20 · Authors: Wenpeng Xu ### Terminal Agents Suffice for Enterprise Automation Studies enterprise automation by terminal agents that call platform APIs, comparing their task results and cost with other agent approaches, including web agents using graphical interfaces. - https://awesomecua.com/resources/arxiv-2604-00073/ · source: https://arxiv.org/abs/2604.00073v4 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, industry, tool-integration, evaluation - Published: 2026-03-31 · Authors: Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, Sagar Davasam, Srinivas Sunkara, Vikas Yadav, Sai Rajeswar ### Terminal Is All You Need: Design Properties for Human-AI Agent Collaboration This position paper derives interface design principles from terminal-based human-agent collaboration: representational compatibility, transparent actions and low participation barriers. It argues that graphical and other agent-facing modalities should provide these properties explicitly. - https://awesomecua.com/resources/arxiv-2603-10664/ · source: https://arxiv.org/abs/2603.10664v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, ui-design, observability, tool-integration - Published: 2026-03-11 · Authors: Alexandre De Masi ### Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces Terminal-Bench 2.0 benchmarks agents on 89 workflow-inspired tasks in computer terminal environments, with a distinct environment, human solution and verification tests for each task. - https://awesomecua.com/resources/arxiv-2601-11868/ · source: https://arxiv.org/abs/2601.11868v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon, workflow - Published: 2026-01-17 · Authors: Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, Ludwig Schmidt ### Test-Time Reinforcement Learning for GUI Grounding via Region Consistency GUI-RC aggregates multiple grounding predictions into spatial voting grids to choose a consensus region. GUI-RCPO then turns agreement with that region into a reward for test-time policy updates on unlabeled GUI screenshots. - https://awesomecua.com/resources/arxiv-2508-05615/ · source: https://arxiv.org/abs/2508.05615v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, training-free - Published: 2025-08-07 · Authors: Yong Du, Yuchen Yan, Fei Tang, Zhengxi Lu, Chang Zong, Weiming Lu, Shengpei Jiang, Yongliang Shen ### Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation Introduces a Test-Time Self-Evolving framework for GUI grounding that explores unseen interfaces, reflects on predicted coordinates, and distills those reflections into the model. - https://awesomecua.com/resources/arxiv-2608-11191/ · source: https://arxiv.org/abs/2608.11191v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, self-reflection, action-prediction - Published: 2026-08-11 · Authors: Shiyu Xuan, Zechao Li ### TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning TGPO merges semantically equivalent web states into tree-structured trajectories for offline preference optimization. A process reward model scores subgoal progress, redundant behavior, and action validity, while dynamic weighting emphasizes consequential decisions. - https://awesomecua.com/resources/arxiv-2509-14172/ · source: https://arxiv.org/abs/2509.14172v2 - Paper · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, reward-models, web-navigation - Published: 2025-09-17 · Authors: Ziyuan Chen, Zhenghui Zhao, Zhangye Han, Miancan Liu, Xianhang Ye, Yiqing Li, Hongbo Min, Jinkui Ren, Xiantao Zhang, Guitao Cao ### The Agent Clicks Approve: Injected Page Text Makes Browser Agents Authorize Access the Task Never Named Studies prompt-injected account-access approvals in browser agents and proposes a service-side authority envelope that checks authorization requests against the user task before allowing them to complete. - https://awesomecua.com/resources/arxiv-2610-15138/ · source: https://arxiv.org/abs/2610.15138v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation - Published: 2026-10-10 · Authors: Md Imran Hossen ### The Art of Building Verifiers for Computer Use Agents Describes a web-trajectory verifier built around structured rubrics, separate process and outcome judgments, and explicit treatment of uncontrollable failures. CUAVerifierBench supplies human labels for evaluating these choices and screenshot-based context management. - https://awesomecua.com/resources/arxiv-2604-06240/ · source: https://arxiv.org/abs/2604.06240v1 - Paper · reviewed 2026-10-17 · environments: web · topics: reward-models, evaluation, long-horizon - Published: 2026-04-05 · Authors: Corby Rosset, Pratyusha Sharma, Andrew Zhao, Miguel Gonzalez-Fernandez, Ahmed Awadallah ### The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes Introduces a controlled web-task testbed for studying how stated preferences and human values influence GUI-agent decisions. It analyzes how personas alter navigation and how prominent interface cues can override the intended preference guidance. - https://awesomecua.com/resources/arxiv-2601-16356/ · source: https://arxiv.org/abs/2601.16356v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, ui-design, human-in-the-loop - Published: 2026-01-22 · Authors: Simret Araya Gebreegziabher, Yukun Yang, Charles Chiang, Hojun Yoo, Chaoran Chen, Hyo Jin Do, Zahra Ashktorab, Werner Geyer, Diego Gómez-Zará, Toby Jia-Jun Li ### The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents Introduces OS-BLIND, a computer-use agent safety benchmark for harmful outcomes arising from benign instructions and threats in task environments. - https://awesomecua.com/resources/arxiv-2604-10577/ · source: https://arxiv.org/abs/2604.10577v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation - Published: 2026-04-12 · Authors: Xuwei Ding, Skylar Zhai, Linxin Song, Jiate Li, Taiwei Shi, Nicholas Meade, Siva Reddy, Jian Kang, Jieyu Zhao ### The BrowserGym Ecosystem for Web Agent Research Describes BrowserGym's shared web interaction environment and AgentLab tooling for consistent experiments across multiple agent benchmarks. - https://awesomecua.com/resources/arxiv-2412-05467/ · source: https://arxiv.org/abs/2412.05467v4 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, sandbox - Published: 2024-12-06 · Authors: Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, Alexandre Lacoste ### The Cognitive Firewall:Securing Browser Based AI Agents Against Indirect Prompt Injection Via Hybrid Edge Cloud Defense Proposes a split-compute browser defense pairing a local visual sentinel, a cloud planner, and deterministic policy enforcement against indirect prompt injection. - https://awesomecua.com/resources/arxiv-2603-23791/ · source: https://arxiv.org/abs/2603.23791v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, web-navigation - Published: 2026-03-24 · Authors: Qianlong Lan, Anuj Kaul ### The Compliance Trap: Diagnosing How AI Agents Consume Conflicting Memory The Entry–Propagation–Recovery framework traces where conflicting memory first changes an agent action, how the effect propagates and whether recovery occurs. WebArena and the controlled MemTrapBench separate memory-consumption failures across these stages. - https://awesomecua.com/resources/arxiv-2607-10608/ · source: https://arxiv.org/abs/2607.10608v1 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, safety-security, robustness, evaluation, long-horizon - Published: 2026-07-12 · Authors: Yixiong Chen, Xinyi Bai, Alan Yuille ### The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use Presents a preliminary study of Claude 3.5 Computer Use across desktop tasks and software, analyzing planning, action, and critique behavior and providing a framework for deploying GUI-automation models. - https://awesomecua.com/resources/arxiv-2411-10323/ · source: https://arxiv.org/abs/2411.10323v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, automation, planning - Published: 2024-11-15 · Authors: Siyuan Hu, Mingyu Ouyang, Difei Gao, Mike Zheng Shou ### The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models Introduces StateAliasBench to isolate GUI world-model ambiguities in which matching visible screens and actions can lead to different futures. It augments frozen world models with structured state recovered from interaction history and evaluates the effect on prediction and agent execution. - https://awesomecua.com/resources/arxiv-2609-32679/ · source: https://arxiv.org/abs/2609.32679v1 - Paper · reviewed 2026-10-17 · environments: android · topics: world-models, state-tracking, evaluation - Published: 2026-09-26 · Authors: Dongsheng Liu, Chao Jin, Wenkui Yang, Hejin Wang, Junwei Yang, Zeren Zhang, Ziwei Chen, Huaibo Huang, Jie Cao, Ran He ### The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge KNOWS benchmarks browser workflows that retrieve and synthesize information into documents, presentations and spreadsheets. Task-specific evaluators combine deterministic checks with model judgments, distinguishing partial progress from usable end-to-end artifact completion. - https://awesomecua.com/resources/arxiv-2609-30604/ · source: https://arxiv.org/abs/2609.30604v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, web-navigation, workflow, long-horizon - Published: 2026-09-24 · Authors: Alexander Gill, Md Farhan Ishmam, Xuyen Nguyen, Neha Bhat, Parker Henry DeYoung, Fateme Hashemi Chaleshtori, Nathan Stringham, Kenneth Marino, Ana Marasović ### The Hidden Dangers of Browsing AI Agents Outlines an end-to-end threat model and empirical security evaluation across architectural layers of autonomous browser agents. - https://awesomecua.com/resources/arxiv-2505-13076/ · source: https://arxiv.org/abs/2505.13076v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2025-05-19 · Authors: Mykyta Mudryi, Markiyan Chaklosh, Grzegorz Wójcik ### The Impact of Element Ordering on LM Agent Performance Studies how the ordering of UI elements in an agent observation affects web and desktop performance. A pixel-based element detector and dimensionality-reduction ordering are evaluated on OmniACT when document hierarchy is unavailable. - https://awesomecua.com/resources/arxiv-2409-12089/ · source: https://arxiv.org/abs/2409.12089v3 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: screen-understanding, grounding, evaluation - Published: 2024-09-18 · Authors: Wayne Chi, Ameet Talwalkar, Chris Donahue ### The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents Gated Hindsight Distillation gives a teacher access to the next screenshot during training while the student sees only the available trajectory prefix. Distillation is applied when hindsight repairs a student failure, and the resulting policies are evaluated on Android tasks. - https://awesomecua.com/resources/arxiv-2608-06065/ · source: https://arxiv.org/abs/2608.06065v1 - Paper · reviewed 2026-10-17 · environments: android · topics: imitation-learning, interaction-history, mobile-navigation - Published: 2026-08-06 · Authors: Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Wen Li ### The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections Characterizes six forms of GUI-content injection and studies their effects on GUI agents and human users using adversarial webpages. The analysis examines privacy failures and the limits of human oversight. - https://awesomecua.com/resources/arxiv-2504-11281/ · source: https://arxiv.org/abs/2504.11281v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, human-in-the-loop, evaluation - Published: 2025-04-15 · Authors: Chaoran Chen, Zhiping Zhang, Bingcan Guo, Shang Ma, Ibrahim Khalilov, Simret A Gebreegziabher, Yanfang Ye, Ziang Xiao, Yaxing Yao, Tianshi Li, Toby Jia-Jun Li ### The Tool Illusion: Rethinking Tool Use in Web Agents Presents a controlled study of higher-level tool use in web agents across tool sources, backbone models, frameworks, and evaluation benchmarks. - https://awesomecua.com/resources/arxiv-2604-03465/ · source: https://arxiv.org/abs/2604.03465v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, tool-integration, web-navigation - Published: 2026-04-03 · Authors: Renze Lou, Baolin Peng, Wenlin Yao, Qianhui Wu, Hao Cheng, Suman Nath, Wenpeng Yin, Jianfeng Gao ### The Verifiable Action Card: Trustworthy Human-in-the-Loop Control for Secure Autonomous Agents Presents the Verifiable Action Card (VAC), a browser approval mechanism that derives confirmations from pending actions and binds user approval to execution. - https://awesomecua.com/resources/arxiv-2609-18411/ · source: https://arxiv.org/abs/2609.18411v2 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, human-in-the-loop, web-navigation - Published: 2026-09-16 · Authors: Hasnain Irshad, Anam Mughees, Neelam Mughees, Abdullah Mughees, Imtiaz Ali Soomro ### TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks Simulates a software-company workplace to evaluate agents on professional tasks involving websites, programs, files, and coworker communication. - https://awesomecua.com/resources/arxiv-2412-14161/ · source: https://arxiv.org/abs/2412.14161v3 - Paper · reviewed 2026-10-16 · environments: web, desktop · topics: evaluation, long-horizon, workflow - Published: 2024-12-18 · Authors: Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, Graham Neubig ### Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems Combines rapid target prediction with deliberate interface analysis, adaptively selecting the grounding process according to task complexity. - https://awesomecua.com/resources/arxiv-2503-06470/ · source: https://arxiv.org/abs/2503.06470v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, efficiency, planning - Published: 2025-03-09 · Authors: Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, Yueting Zhuang ### Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction Test-Time Interaction scales the interaction horizon so web agents can explore, backtrack and revise plans within a rollout. Its online reinforcement-learning curriculum adjusts rollout lengths and is evaluated on WebVoyager and WebArena. - https://awesomecua.com/resources/arxiv-2506-07976/ · source: https://arxiv.org/abs/2506.07976v2 - Paper · reviewed 2026-10-18 · environments: web · topics: reinforcement-learning, web-navigation, planning, long-horizon - Published: 2025-06-09 · Authors: Junhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, Aviral Kumar ### Throttling Web Agents Using Reasoning Gates Introduces reasoning gates that require agents to solve generated, verifiable puzzles before receiving access to resources. The throttling approach is deployed on a custom website and MCP servers and evaluated with web agents. - https://awesomecua.com/resources/arxiv-2509-01619/ · source: https://arxiv.org/abs/2509.01619v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, tool-integration - Published: 2025-09-01 · Authors: Abhinav Kumar, Jaechul Roh, Ali Naseh, Amir Houmansadr, Eugene Bagdasarian ### TimeWarp: Evaluating Web Agents by Revisiting the Past TimeWarp tests web agents across multiple versions of the same sites, varying visual design, frontend code and workflows. TimeTraj transfers plans across versions to collect training trajectories and study robustness to interface change. - https://awesomecua.com/resources/arxiv-2603-04949/ · source: https://arxiv.org/abs/2603.04949v4 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, robustness, web-navigation, data-generation - Published: 2026-03-05 · Authors: Md Farhan Ishmam, Kenneth Marino ### TinyClick: Single-Turn Agent for Empowering GUI Automation Adapts a compact Florence-2 model to map user instructions to screen coordinates using multitask training and augmented data. - https://awesomecua.com/resources/arxiv-2410-11871/ · source: https://arxiv.org/abs/2410.11871v3 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, efficiency - Published: 2024-10-09 · Authors: Pawel Pawlowski, Krystian Zawistowski, Wojciech Lapacz, Adam Wiacek, Marcin Skorupa, Sebastien Postansque, Jakub Hoscilowicz ### Token Efficient Task Execution via Application Behavior Modeling for Web Agents OdoBot builds a model of web-application behavior from successful task demonstrations and uses it to guide task execution. Its Canvas LMS evaluation compares token use and completion with existing web agents. - https://awesomecua.com/resources/arxiv-2609-13491/ · source: https://arxiv.org/abs/2609.13491v1 - Paper · reviewed 2026-10-18 · environments: web · topics: memory, efficiency, web-navigation - Published: 2026-09-11 · Authors: Alexandru Ianta, Eleni Stroulia ### TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents TongUI converts multimodal online tutorials, including videos and articles, into GUI-agent trajectories. Its GUI-Net dataset contains 143,000 trajectories spanning five operating systems and more than 200 applications, and supports fine-tuning for grounding and navigation. - https://awesomecua.com/resources/arxiv-2504-12679/ · source: https://arxiv.org/abs/2504.12679v4 - Paper · reviewed 2026-10-17 · environments: cross-platform · topics: data-generation, grounding, video - Published: 2025-04-17 · Authors: Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, Qing Li ### ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents ToolCUA synthesizes interleaved GUI/tool trajectories and trains agents to choose between graphical actions and higher-level tools. Staged supervised and reinforcement learning culminate in online hybrid-environment training with a reward for efficient execution paths. - https://awesomecua.com/resources/arxiv-2605-12481/ · source: https://arxiv.org/abs/2605.12481v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: tool-integration, data-generation, imitation-learning, reinforcement-learning, planning, efficiency - Published: 2026-05-12 · Authors: Xuhao Hu, Xi Zhang, Haiyang Xu, Kyle Qiao, Jingyi Yang, Xuanjing Huang, Jing Shao, Ming Yan, Jieping Ye ### ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents ToolTok represents GUI operations with learned tool tokens and frames interaction as progressive tool selection. Semantic anchoring and an easy-to-hard curriculum teach tool meanings before visual pathfinding, with evaluation across grounding scenarios. - https://awesomecua.com/resources/arxiv-2602-02548/ · source: https://arxiv.org/abs/2602.02548v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: tool-integration, grounding, data-generation - Published: 2026-01-30 · Authors: Xiaoce Wang, Guibin Zhang, Junzhe Li, Jinzhe Tu, Chun Li, Ming Li ### Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents Presents a position paper on privacy and security evaluation for GUI agents, identifying risk categories and challenges in involving human evaluators. It proposes risk-aware assessment and in-context consent as design directions. - https://awesomecua.com/resources/arxiv-2504-17934/ · source: https://arxiv.org/abs/2504.17934v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, safety-security, human-in-the-loop - Published: 2025-04-24 · Authors: Chaoran Chen, Zhiping Zhang, Ibrahim Khalilov, Bingcan Guo, Simret A Gebreegziabher, Yanfang Ye, Ziang Xiao, Yaxing Yao, Tianshi Li, Toby Jia-Jun Li ### Towards Automated Crowdsourced Testing via Personified-LLM Introduces PersonaTester, a framework that gives LLM agents tester personas to vary their exploration and interaction habits during automated crowdsourced GUI testing. - https://awesomecua.com/resources/arxiv-2603-24160/ · source: https://arxiv.org/abs/2603.24160v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: automation, evaluation, workflow - Published: 2026-03-25 · Authors: Shengcheng Yu, Yuchen Ling, Chunrong Fang, Zhenyu Chen, Chunyang Chen ### Towards Better Semantic Understanding of Mobile Interfaces Augments RICO screens and view hierarchies with human labels for icon shape and meaning and for links between UI elements and text labels. It compares image-only and multimodal models on semantic understanding of previously unseen apps. - https://awesomecua.com/resources/arxiv-2210-02663/ · source: https://arxiv.org/abs/2210.02663v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding, data-generation - Published: 2022-10-06 · Authors: Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong, Chen, Abhanshu Sharma, James Stout ### Towards Computer-Using Personal Agents Proposes computer-using personal agents connected to user-controlled data repositories, discussing privacy, interoperability, and collaborative tasks. - https://awesomecua.com/resources/arxiv-2503-15515/ · source: https://arxiv.org/abs/2503.15515v1 - Paper · reviewed 2026-10-16 · environments: web, cross-platform · topics: safety-security, memory, human-in-the-loop - Published: 2025-01-31 · Authors: Piero A. Bonatti, John Domingue, Anna Lisa Gentile, Andreas Harth, Olaf Hartig, Aidan Hogan, Katja Hose, Ernesto Jimenez-Ruiz, Deborah L. McGuinness, Chang Sun, Ruben Verborgh, Jesse Wright ### Towards Enterprise-Ready Computer Using Generalist Agent Describes the CUGA enterprise-agent development process, emphasizing iterative evaluation, failure analysis, and refinement on software-task benchmarks. - https://awesomecua.com/resources/arxiv-2503-01861/ · source: https://arxiv.org/abs/2503.01861v3 - Paper · reviewed 2026-10-16 · environments: web · topics: automation, evaluation, workflow - Published: 2025-02-24 · Authors: Sami Marreed, Alon Oved, Avi Yaeli, Segev Shlomov, Ido Levy, Offer Akrabi, Aviad Sela, Asaf Adi, Nir Mashkif ### Towards Faithful Simulation of Human Shopping Behavior Presents RecVerse, a screenshot-based shopping simulation agent with hierarchical memory and trajectory-level RL, alongside USB, an interactive e-commerce GUI trajectory dataset. - https://awesomecua.com/resources/arxiv-2608-20707/ · source: https://arxiv.org/abs/2608.20707v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, reinforcement-learning, data-generation, long-horizon - Published: 2026-08-21 · Authors: Jiakai Tang, Yan Mi, Jing Yu, Yang Zhang, See-Kiong Ng, Qi Cao, Fei Sun, Xu Chen, Wen Chen, Jian Wu, Han Zhu, Bo Zheng ### Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding Adapts a discrete vision-language diffusion model to single-turn GUI action and bounding-box prediction, using a hybrid masking schedule for coordinate geometry. Experiments across web, desktop and mobile datasets examine grounding accuracy and inference tradeoffs. - https://awesomecua.com/resources/arxiv-2603-26211/ · source: https://arxiv.org/abs/2603.26211v1 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: grounding, action-prediction, efficiency - Published: 2026-03-27 · Authors: Shrinidhi Kumbhar, Haofu Liao, Srikar Appalaraju, Kunwar Yashraj Singh ### Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction MaP unifies GUI-navigation objectives by masking and predicting components of multi-turn trajectories. Role-aware adapters address heterogeneous training data, and evaluations compare the approach with direct task-mixture training across navigation benchmarks. - https://awesomecua.com/resources/arxiv-2609-25769/ · source: https://arxiv.org/abs/2609.25769v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: imitation-learning, action-prediction, state-tracking, planning - Published: 2026-09-22 · Authors: Yan Zhang, Pei Fu, Daiqing Wu, Huawen Shen, Ruoceng Zhang, Shaojie Zhang, Jiahui Yang, Yu Zhou, Can Ma, Zhenbo Luo, Jian Luan ### Towards Safe Autonomy for Computer Use Presents SA-CUA, separating trusted and untrusted visual content using accessibility-derived masking rules and a controlled channel for verifying information before endorsement. - https://awesomecua.com/resources/arxiv-2610-18508/ · source: https://arxiv.org/abs/2610.18508v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, screen-understanding - Published: 2026-10-13 · Authors: Dominik Seip, Bartol Markovinović, Matthias Hein ### Towards Scalable Lightweight GUI Agents via Multi-role Orchestration Introduces LAMO, a training framework for a lightweight GUI agent that can execute tasks alone or serve as an action executor alongside planners in multi-agent workflows. - https://awesomecua.com/resources/arxiv-2604-13488/ · source: https://arxiv.org/abs/2604.13488v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, reinforcement-learning, planning, workflow - Published: 2026-04-15 · Authors: Ziwei Wang, Junjie Zheng, Leyang Yang, Sheng Zhou, Xiaoxuan Tang, Zhouhua Fang, Zhiwei Liu, Dajun Chen, Yong Li, Jiajun Bu ### Towards Sustainable Web Agents: A Plea for Transparency and Dedicated Metrics for Energy Consumption Examines energy and carbon-accounting limitations in web-agent evaluation and argues for transparent, dedicated sustainability measurements. - https://awesomecua.com/resources/arxiv-2502-17903/ · source: https://arxiv.org/abs/2502.17903v1 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, efficiency - Published: 2025-02-25 · Authors: Lars Krupp, Daniel Geißler, Paul Lukowicz, Jakob Karolus ### TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agents TRACE orders screenshot evidence for monotonic visual-token reduction using layout priors, instruction relevance and feature novelty. Spatial token reserves and incremental KV contraction aim to preserve useful evidence for future GUI targets without repeatedly encoding retired frames. - https://awesomecua.com/resources/arxiv-2609-10297/ · source: https://arxiv.org/abs/2609.10297v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: efficiency, interaction-history, screen-understanding, training-free - Published: 2026-09-09 · Authors: Yuhao Wang, Mu Qiao, Xindong Zhang, Yunzhi Zhuge, Lei Zhang, Huchuan Lu ### Training a Vision Language Model as Smartphone Assistant Trains a smartphone assistant from sequences of screenshots paired with earlier actions. The model predicts UI-only interactions such as taps and swipes, with evaluation on Android in the Wild. - https://awesomecua.com/resources/arxiv-2404-08755/ · source: https://arxiv.org/abs/2404.08755v1 - Paper · reviewed 2026-10-17 · environments: android · topics: mobile-navigation, interaction-history, action-prediction - Published: 2024-04-12 · Authors: Nicolai Dorka, Janusz Marecki, Ammar Anwar ### Training Computer Use Agents to Assess the Usability of Graphical User Interfaces uxCUA learns to assess interface usability by choosing interaction flows, executing them and predicting a numerical usability score. Training uses interactive UIs with usability labels and human preferences, with evaluations on synthetic and real interfaces. - https://awesomecua.com/resources/arxiv-2604-26020/ · source: https://arxiv.org/abs/2604.26020v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: ui-design, evaluation, automation - Published: 2026-04-28 · Authors: Alice Gao, Weixi Tong, Rishab Vempati, Katharina Reinecke, R. Benjamin Shapiro, Tianyi Zhang, Jason Wu ### Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation This work trains high-level schedulers with staged reinforcement learning driven by execution feedback. Its CES framework separates a Coordinator for planning and decomposition, a StateTracker for compressed context, and a low-level Executor. - https://awesomecua.com/resources/arxiv-2511-22235/ · source: https://arxiv.org/abs/2511.22235v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: planning, state-tracking, reinforcement-learning, long-horizon - Published: 2025-11-27 · Authors: Zehao Deng, Tianjie Ju, Zheng Wu, Zhuosheng Zhang, Gongshen Liu ### Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Masking Introduces STeP, which trains agents on synthetic trajectories containing error reflections and corrections while masking poor steps; WebShop is one of three evaluated tasks. - https://awesomecua.com/resources/arxiv-2505-20023/ · source: https://arxiv.org/abs/2505.20023v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: data-generation, imitation-learning, self-reflection - Published: 2025-05-26 · Authors: Yihan Chen, Benfeng Xu, Xiaorui Wang, Yongdong Zhang, Zhendong Mao ### Training Needs Trustworthy Worlds: Verified Synthetic Web Environments for Agent Learning Builds synthetic websites with verified navigation, backend state, and feasible tasks to provide reliable supervision for web-agent training. - https://awesomecua.com/resources/arxiv-2608-21898/ · source: https://arxiv.org/abs/2608.21898v2 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, sandbox, reinforcement-learning - Published: 2026-08-22 · Authors: Chenghao Zhang, Yuxi Cheng, Saisai Hu, Canran Xiao, Yikai Mao, Dan Roth ### TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments Introduces TransBench to evaluate GUI grounding transfer across app versions, iOS, Android, Web, and functionally distinct applications. - https://awesomecua.com/resources/arxiv-2505-17629/ · source: https://arxiv.org/abs/2505.17629v2 - Paper · reviewed 2026-11-06 · environments: ios, android, web, cross-platform · topics: evaluation, grounding, robustness, cross-app - Published: 2025-05-23 · Authors: Yuheng Lu, Qian Yu, Hongru Wang, Zeming Liu, Wei Su, Yanping Liu, Yuhang Guo, Maocheng Liang, Yunhong Wang, Haifeng Wang ### Tree Search for Language Model Agents Applies best-first search to agent actions within interactive web environments, allowing exploration and multistep planning at inference time. WebArena and VisualWebArena experiments examine search behavior and test-time computation. - https://awesomecua.com/resources/arxiv-2407-01476/ · source: https://arxiv.org/abs/2407.01476v4 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, web-navigation, evaluation - Published: 2024-07-01 · Authors: Jing Yu Koh, Stephen McAleer, Daniel Fried, Ruslan Salakhutdinov ### TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution TreeCUA organizes GUI exploration trajectories as reusable trees and balances deeper tasks against broader coverage. Its data pipeline combines action verification, memory-backed recovery and a DPO variant that uses information from adjacent branches. - https://awesomecua.com/resources/arxiv-2602-09662/ · source: https://arxiv.org/abs/2602.09662v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, planning, memory, efficiency - Published: 2026-02-10 · Authors: Deyang Jiang, Jing Huang, Xuanle Zhao, Lei Chen, Liming Zheng, Fanfan Liu, Haibo Qiu, Peng Shi, Zhixiong Zeng ### TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks TriAxialKV assigns KV-cache precision by token recency, modality, and role for agentic inference, with a computer-use agent on OSWorld among its evaluations. - https://awesomecua.com/resources/arxiv-2605-17170/ · source: https://arxiv.org/abs/2605.17170v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency - Published: 2026-05-16 · Authors: Hanzhang Shen, Haoran Wu, Yiren Zhao, Robert Mullins ### Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion Trifuse combines attention-based localization with OCR text cues and icon captions to ground GUI instructions. Its fusion procedure favors cross-modal agreement and sharp location peaks, and is evaluated without task-specific fine-tuning. - https://awesomecua.com/resources/arxiv-2602-06351/ · source: https://arxiv.org/abs/2602.06351v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding, training-free - Published: 2026-02-06 · Authors: Longhui Ma, Di Zhao, Siwei Wang, Zhao Lv, Miao Wang ### TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents Combines hierarchical screen parsing with spatially enriched element descriptions to support both locating GUI targets and describing indicated elements. - https://awesomecua.com/resources/arxiv-2502-08226/ · source: https://arxiv.org/abs/2502.08226v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, screen-understanding - Published: 2025-02-12 · Authors: Kunal Singh, Shreyas Singh, Mukund Khanna ### Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding Proposes quality-aware self-distillation for GUI coordinate generation under student-produced prefixes. A correctness-aware gate suppresses coordinate-token supervision that cannot reach the target box, while teacher-probability scaling adjusts the remaining signal strength. - https://awesomecua.com/resources/arxiv-2606-18101/ · source: https://arxiv.org/abs/2606.18101v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, imitation-learning, evaluation - Published: 2026-06-16 · Authors: Jingyuan Huang, Zuming Huang, Yucheng Shi, Tianze Yang, Xiaoming Zhai, Wei Chu, Ninghao Liu ### TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents TUA-Bench evaluates terminal-use agents on general digital tasks, including document editing, email, web information seeking and specialist software workflows. Tasks run in real terminals with deterministic setup and execution-based scoring. - https://awesomecua.com/resources/arxiv-2606-28480/ · source: https://arxiv.org/abs/2606.28480v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, automation - Published: 2026-06-26 · Authors: Shoufa Chen, Luyuan Wang, Xuan Yang, Zhiheng Liu, Yuren Cong, Yuanfeng Ji, Feiyan Zhou, Xiaohui Zhang, Fanny Yang, Belinda Zeng ### Tur\[k\]ingBench: A Challenge Benchmark for Web Agents Evaluates web agents on multimodal annotation tasks adapted from HTML interfaces originally built for crowdsourcing workers. - https://awesomecua.com/resources/arxiv-2403-11905/ · source: https://arxiv.org/abs/2403.11905v4 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation - Published: 2024-03-18 · Authors: Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi ### Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization Introduces the Agent Humanization Benchmark (AHB) and mobile touch-dynamics data to assess whether GUI agents' interactions resemble human touch behavior. - https://awesomecua.com/resources/arxiv-2604-09574/ · source: https://arxiv.org/abs/2604.09574v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, data-generation, mobile-navigation - Published: 2026-02-24 · Authors: Jiachen Zhu, Lingyu Yang, Rong Shan, Congmin Zheng, Zeyu Zheng, Weiwen Liu, Yong Yu, Weinan Zhang, Jianghao Lin ### TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training Proposes TurnOPD, a rollout-depth and loss-budgeting method for training long-horizon language agents; WebShop is one of its evaluation settings. - https://awesomecua.com/resources/arxiv-2607-05804/ · source: https://arxiv.org/abs/2607.05804v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, long-horizon - Published: 2026-07-07 · Authors: Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, Jingjing Chen ### UEyes: An Eye-Tracking Dataset across User Interface Types Introduces UEyes, an eye-tracking dataset of webpage, desktop, mobile, and poster interfaces, and analyzes how people view these different UI types. - https://awesomecua.com/resources/arxiv-2402-05202/ · source: https://arxiv.org/abs/2402.05202v1 - Paper · reviewed 2026-11-06 · environments: web, desktop · topics: data-generation, screen-understanding - Published: 2024-02-07 · Authors: Yue Jiang, Luis A. Leiva, Paul R. B. Houssel, Hamed R. Tavakoli, Julia Kylmälä, Antti Oulasvirta ### UFO2: The Desktop AgentOS Introduces UFO2, a Windows desktop agent architecture combining a coordinating HostAgent, application-specialized agents, and a shared GUI/API action layer. It integrates UI Automation, visual parsing, speculative planning, and an isolated picture-in-picture desktop. - https://awesomecua.com/resources/arxiv-2504-14603/ · source: https://arxiv.org/abs/2504.14603v2 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: automation, cross-app, tool-integration - Published: 2025-04-20 · Authors: Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, Liqun Li, Yu Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang ### UFO: A UI-Focused Agent for Windows OS Interaction UFO uses two agents to inspect Windows application interfaces and their control metadata, then grounds and executes user requests within and across applications. The study evaluates this UI-focused architecture on nine Windows applications. - https://awesomecua.com/resources/arxiv-2402-07939/ · source: https://arxiv.org/abs/2402.07939v5 - Paper · reviewed 2026-10-17 · environments: windows, desktop · topics: cross-app, planning, grounding, automation - Published: 2024-02-08 · Authors: Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang ### UGIF: UI Grounded Instruction Following Introduces multilingual smartphone instruction-following data and a pipeline that retrieves help instructions and maps them to UI actions. - https://awesomecua.com/resources/arxiv-2211-07615/ · source: https://arxiv.org/abs/2211.07615v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: mobile-navigation, grounding - Published: 2022-11-14 · Authors: Sagar Gubbi Venkatesh, Partha Talukdar, Srini Narayanan ### UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface UISCGD detects semantic groups of adjacent text and non-text UI elements. It learns from Android and iOS screenshots to support interface retrieval, UI-to-code structure and screen-reader accessibility. - https://awesomecua.com/resources/arxiv-2403-04984/ · source: https://arxiv.org/abs/2403.04984v1 - Paper · reviewed 2026-10-18 · environments: android, ios · topics: screen-understanding, grounding - Published: 2024-03-08 · Authors: Shuhong Xiao, Yunnong Chen, Yaxuan Song, Liuqing Chen, Lingyun Sun, Yankun Zhen, Yanfang Chang ### UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding Combines grounding-oriented reward and sampling improvements with inference-time image decomposition and target selection. - https://awesomecua.com/resources/arxiv-2507-22025/ · source: https://arxiv.org/abs/2507.22025v4 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, reinforcement-learning - Published: 2025-07-29 · Authors: Shuquan Lian, Yuhang Wu, Jia Ma, Yifan Ding, Zihan Song, Bingqi Chen, Xiawu Zheng, Hui Li, Rongrong Ji ### UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization UI-Copilot separates GUI execution from on-demand memory retrieval and numerical assistance. Its training procedure treats tool selection and multi-turn task execution as distinct optimization targets while keeping persistent observations outside transient context. - https://awesomecua.com/resources/arxiv-2604-13822/ · source: https://arxiv.org/abs/2604.13822v1 - Paper · reviewed 2026-10-17 · environments: android · topics: tool-integration, memory, planning, long-horizon, mobile-navigation - Published: 2026-04-15 · Authors: Zhengxi Lu, Fei Tang, Guangyi Liu, Kaitao Song, Xu Tan, Jin Ma, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen ### UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliability UI-CUBE evaluates 226 tasks ranging from individual controls to copy-paste and enterprise workflows. Interface variations, multiple resolutions, and application-state validation support diagnosis of operational reliability beyond task accuracy alone. - https://awesomecua.com/resources/arxiv-2511-17131/ · source: https://arxiv.org/abs/2511.17131v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, industry, workflow, robustness - Published: 2025-11-21 · Authors: Horia Cristescu, Charles Park, Trong Canh Nguyen, Sergiu Talmacel, Alexandru-Gabriel Ilie, Stefan Adam ### UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis UI-E2I-Synth uses GPT-4o to synthesize GUI grounding instructions with varied complexity, addressing small elements, imbalanced element types, and implicit requests. The accompanying UI-I2E-Bench evaluates instruction-to-element localization across these annotation challenges. - https://awesomecua.com/resources/arxiv-2504-11257/ · source: https://arxiv.org/abs/2504.11257v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, grounding, evaluation - Published: 2025-04-15 · Authors: Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, Yan Lu ### UI-Evol: Automatic Knowledge Evolving for Computer Use Agents Introduces UI-Evol, which extracts action sequences from agent interactions and critiques external guidance against those traces. Evaluation on OSWorld examines how revised operational knowledge affects execution and behavioral consistency. - https://awesomecua.com/resources/arxiv-2505-21964/ · source: https://arxiv.org/abs/2505.21964v2 - Paper · reviewed 2026-10-17 · environments: desktop · topics: memory, self-reflection, robustness - Published: 2025-05-28 · Authors: Ziyun Zhang, Xinyi Liu, Xiaoyi Zhang, Jun Wang, Gang Chen, Yan Lu ### UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents UI-Genie jointly improves a mobile GUI agent and a reward model that processes screenshots with interaction history and supplies action- and task-level feedback. Its data pipeline uses rule-based verification, corrupted trajectories, and hard negatives to support reward-guided exploration and iterative training. - https://awesomecua.com/resources/arxiv-2505-21496/ · source: https://arxiv.org/abs/2505.21496v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: data-generation, reinforcement-learning, mobile-navigation, interaction-history - Published: 2025-05-27 · Authors: Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Aojun Zhou, Hongsheng Li ### UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning Treats alternative instruction formulations as reasoning paths and trains GUI-grounding models to select and combine useful perspectives. - https://awesomecua.com/resources/arxiv-2510-20286/ · source: https://arxiv.org/abs/2510.20286v1 - Paper · reviewed 2026-10-16 · environments: cross-platform, android · topics: grounding, reinforcement-learning - Published: 2025-10-23 · Authors: Liangyu Chen, Hanzhang Zhou, Chenglin Cai, Jianan Zhang, Panrong Tong, Quyu Kong, Xu Zhang, Chen Liu, Yuqi Liu, Wenxuan Wang, Yue Wang, Qin Jin, Steven Hoi ### UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents UI-KOBE explores mobile applications to build graphs of UI states and executable transitions. A lightweight agent then matches its current screenshot to the graph and chooses a transition, local action, completion signal or fallback action. - https://awesomecua.com/resources/arxiv-2605-29534/ · source: https://arxiv.org/abs/2605.29534v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, planning, state-tracking, memory, efficiency - Published: 2026-05-28 · Authors: Yuxiang Chai, Han Xiao, Xinyu Fu, Jinpeng Chen, Rui Liu, Hongsheng Li ### UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations Combines environment-grounded training with demonstration-conditioned workflows, and introduces OSWorkerBench for extended office-task evaluation. - https://awesomecua.com/resources/arxiv-2608-15930/ · source: https://arxiv.org/abs/2608.15930v1 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: workflow, data-generation, long-horizon - Published: 2026-08-16 · Authors: Zihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke, Mengcheng Lan, Weixian Lei, Hanxuan Li, Honglin Li, Xiyun Li, Zaitang Li, Leowei Liang, Xin Luo, Haozhe Ma, Jiayi Mao, Zhoujie Pan, Can Qin, Tianyuan Qu, Weiqi Wang, Wenkai Wang, Yonglin Wang, Yuxin Wang, Chenxu Wu, Yingchen Yu, Chenyu Zhang, Yuhao Zheng ### UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents UI-Mem stores workflows, skills and failure patterns as reusable templates to guide online GUI reinforcement learning. It varies guidance within rollout groups and continually updates the memory so an unguided policy can learn from guided experiences. - https://awesomecua.com/resources/arxiv-2602-05832/ · source: https://arxiv.org/abs/2602.05832v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, reinforcement-learning, cross-app - Published: 2026-02-05 · Authors: Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yuxiang Chai, Yue Pan, Yufeng Zhou, Xiaoxin Chen, Yafei Wen, Hongsheng Li ### UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents Introduces Uni-GUI, a collection of executable desktop and mobile trajectories, and UI-MOPD, which trains a shared policy from its own rollouts using platform-specific teachers. The distillation procedure routes each visited state to the appropriate teacher to preserve distinct interaction conventions. - https://awesomecua.com/resources/arxiv-2607-04425/ · source: https://arxiv.org/abs/2607.04425v2 - Paper · reviewed 2026-10-17 · environments: desktop, cross-platform · topics: data-generation, imitation-learning, action-prediction - Published: 2026-07-05 · Authors: Niu Lian, Tongbo Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Heng Qu, Shu-Tao Xia, Jinpeng Wang ### UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics UI-Oceanus converts autonomous GUI exploration and execution feedback into supervision for predicting future interface states. It compares forward and inverse objectives during continual pretraining and examines downstream offline and online navigation as synthetic data scales. - https://awesomecua.com/resources/arxiv-2604-02345/ · source: https://arxiv.org/abs/2604.02345v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: world-models, data-generation, state-tracking, evaluation - Published: 2026-02-11 · Authors: Mengzhou Wu, Yuzhe Guo, Yuan Cao, Haochuan Lu, Songhe Zhu, Pingzhe Qu, Xin Chen, Kang Qin, Zhongpu Wang, Xiaode Zhang, Xinyi Wang, Wei Dai, Gang Cao, Yuetang Deng, Zhi Gong, Dezhi Ran, Linyi Li, Wei Yang, Tao Xie ### UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning Uses rule-based action rewards and policy optimization to train GUI action prediction from a compact set of mobile tasks. - https://awesomecua.com/resources/arxiv-2503-21620/ · source: https://arxiv.org/abs/2503.21620v5 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: reinforcement-learning, grounding - Published: 2025-03-27 · Authors: Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, Hongsheng Li ### UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning UI-S1 simulates multi-turn online learning over offline trajectories, using a patch module to recover divergences while retaining the model’s own outputs in context. It combines discounted future returns with step- and episode-level advantages and introduces a semi-online performance proxy. - https://awesomecua.com/resources/arxiv-2509-11543/ · source: https://arxiv.org/abs/2509.11543v2 - Paper · reviewed 2026-10-17 · environments: android · topics: reinforcement-learning, evaluation, mobile-navigation - Published: 2025-09-15 · Authors: Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, Yueting Zhuang ### UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning UI-TARS-2 combines scalable trajectory generation, stabilized multi-turn reinforcement learning, and a sandbox that supports GUI, filesystem, and terminal actions. The report evaluates the resulting agent across web, desktop, Android, and game tasks and analyzes training stability. - https://awesomecua.com/resources/arxiv-2509-02544/ · source: https://arxiv.org/abs/2509.02544v2 - Paper · reviewed 2026-10-17 · environments: web, desktop, windows, android, cross-platform · topics: reinforcement-learning, data-generation, sandbox, tool-integration - Published: 2025-09-02 · Authors: Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, Wanjun Zhong, Yining Ye, Yujia Qin, Yuwen Xiong, Yuxin Song, Zhiyong Wu, Aoyan Li, Bo Li, Chen Dun, Chong Liu, Daoguang Zan, Fuxing Leng, Hanbin Wang, Hao Yu, Haobin Chen, Hongyi Guo, Jing Su, Jingjia Huang, Kai Shen, Kaiyu Shi, Lin Yan, Peiyao Zhao, Pengfei Liu, Qinghao Ye, Renjie Zheng, Shulin Xin, Wayne Xin Zhao, Wen Heng, Wenhao Huang, Wenqian Wang, Xiaobo Qin, Yi Lin, Youbin Wu, Zehui Chen, Zihao Wang, Baoquan Zhong, Xinchun Zhang, Xujing Li, Yuanfan Li, Zhongkai Zhao, Chengquan Jiang, Faming Wu, Haotian Zhou, Jinlin Pang, Li Han, Qi Liu, Qianli Ma, Siyao Liu, Songhua Cai, Wenqi Fu, Xin Liu, Yaohui Wang, Zhi Zhang, Bo Zhou, Guoliang Li, Jiajun Shi, Jiale Yang, Jie Tang, Li Li, Qihua Han, Taoran Lu, Woyu Lin, Xiaokang Tong, Xinyao Li, Yichi Zhang, Yu Miao, Zhengxuan Jiang, Zili Li, Ziyuan Zhao, Chenxin Li, Dehua Ma, Feng Lin, Ge Zhang, Haihua Yang, Hangyu Guo, Hongda Zhu, Jiaheng Liu, Junda Du, Kai Cai, Kuanye Li, Lichen Yuan, Meilan Han, Minchao Wang, Shuyue Guo, Tianhao Cheng, Xiaobo Ma, Xiaojun Xiao, Xiaolong Huang, Xinjie Chen, Yidi Du, Yilin Chen, Yiwen Wang, Zhaojian Li, Zhenzhu Yang, Zhiyuan Zeng, Chaolin Jin, Chen Li, Hao Chen, Haoli Chen, Jian Chen, Qinghao Zhao, Guang Shi ### UI-TARS: Pioneering Automated GUI Interaction with Native Agents Studies a screenshot-driven native GUI model with unified actions, deliberate reasoning, and iterative training on interaction traces. - https://awesomecua.com/resources/arxiv-2501-12326/ · source: https://arxiv.org/abs/2501.12326v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, planning, data-generation - Published: 2025-01-21 · Authors: Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, Guang Shi ### UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning Introduces UI-UG 2, a mobile UI model trained for component understanding, GUI grounding, and structured interface generation with render-aware RL. - https://awesomecua.com/resources/arxiv-2611-00926/ · source: https://arxiv.org/abs/2611.00926v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: screen-understanding, grounding, ui-design, reinforcement-learning - Published: 2026-10-31 · Authors: Hao Yang, Shuoyang Liu, Xiaoyu Lin, Zhou Fang, Maji Huang, Zhaosong Huang, Ruichao Mao, Teng Guo, Hai Rao ### UI-Venus Technical Report: Building High-performance UI Agents with RFT Studies screenshot-based GUI grounding and navigation using curated interaction data, reward design, and reinforcement fine-tuning. - https://awesomecua.com/resources/arxiv-2508-10833/ · source: https://arxiv.org/abs/2508.10833v2 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: grounding, reinforcement-learning - Published: 2025-08-14 · Authors: Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Liang Chen, Weiqiang Wang ### UI-Venus-1.5 Technical Report Describes a GUI-agent family combining broad intermediate training, full-trajectory online reinforcement learning, and merging of specialized models. - https://awesomecua.com/resources/arxiv-2602-09082/ · source: https://arxiv.org/abs/2602.09082v2 - Paper · reviewed 2026-10-16 · environments: cross-platform, web, android · topics: reinforcement-learning, grounding, mobile-navigation - Published: 2026-02-09 · Authors: Venus Team, Changlong Gao, Zhangxuan Gu, Yulin Liu, Xinyu Qiu, Shuheng Shen, Yue Wen, Tianyu Xia, Zhenyu Xu, Zhengwen Zeng, Beitong Zhou, Xingran Zhou, Weizhi Chen, Sunhao Dai, Jingya Dou, Yichen Gong, Yuan Guo, Zhenlin Guo, Feng Li, Qian Li, Jinzhen Lin, Yuqi Zhou, Linchao Zhu, Liang Chen, Zhenyu Guo, Changhua Meng, Weiqiang Wang ### UI-Venus-2 Technical Report Describes a GUI foundation agent trained through expanded environments, function-grounded tasks, and trajectory verification with safety-aware execution mechanisms. - https://awesomecua.com/resources/arxiv-2609-00028/ · source: https://arxiv.org/abs/2609.00028v1 - Paper · reviewed 2026-10-16 · environments: web, desktop, cross-platform · topics: automation, reinforcement-learning, safety-security - Published: 2026-08-27 · Authors: Venus Team, Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Yusong Hu, Jianrong Jiang, Jianguo Li, Runze Li, Jinzhen Lin, Zhenyu Ma, Changhua Meng, Han Peng, Xinyu Qiu, Shuheng Shen, Zhongyi Shui, Weiqiang Wang, Ming Wen, Zhuoer Xu, Hang Yan, Kaiwen Yang, Ruilin Yao, Nanjun Yu, Zhengwen Zeng, Lianrui Zhang, Yunzhu Zhang, Zhe Zhao, Beitong Zhou ### UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction Provides annotated desktop demonstrations for evaluating element grounding, layout grounding, and action prediction. - https://awesomecua.com/resources/arxiv-2503-15661/ · source: https://arxiv.org/abs/2503.15661v2 - Paper · reviewed 2026-10-16 · environments: desktop · topics: grounding, evaluation, screen-understanding - Published: 2025-03-19 · Authors: Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M. Tamer Özsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, Perouz Taslakian, Spandana Gella, Sai Rajeswar ### UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience Introduces a two-stage mobile GUI learner that first co-evolves trajectories and policies through rejection fine-tuning. Group Relative Self-Distillation then uses successful rollout branches to provide step-level supervision for failed trajectories, evaluated on AndroidWorld. - https://awesomecua.com/resources/arxiv-2603-24533/ · source: https://arxiv.org/abs/2603.24533v1 - Paper · reviewed 2026-10-17 · environments: android · topics: imitation-learning, data-generation, mobile-navigation - Published: 2026-03-25 · Authors: Zichuan Lin, Feiyu Liu, Yijun Yang, Jiafei Lyu, Yiming Gao, Yicheng Liu, Zhicong Lu, Yangbin Yu, Mingyu Yang, Junyou Li, Deheng Ye, Jie Jiang ### UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding Introduces UI-Zoomer, a training-free grounding method that uses candidate agreement and token confidence to decide when to crop and retry. Prediction variance determines a separate crop radius for each uncertain interface target. - https://awesomecua.com/resources/arxiv-2604-14113/ · source: https://arxiv.org/abs/2604.14113v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, efficiency - Published: 2026-04-15 · Authors: Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen ### UIBert: Learning Generic Multimodal Representations for UI Understanding Learns shared image, text, and structural UI representations through self-supervised tasks based on relationships among interface components. - https://awesomecua.com/resources/arxiv-2107-13731/ · source: https://arxiv.org/abs/2107.13731v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: screen-understanding - Published: 2021-07-29 · Authors: Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, Blaise Aguera y Arcas ### UIClip: A Data-driven Model for Assessing User Interface Design UIClip assesses UI design quality and visual relevance from screenshots and descriptions, using crawled, augmented and human-rated interfaces for training. - https://awesomecua.com/resources/arxiv-2404-12500/ · source: https://arxiv.org/abs/2404.12500v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: ui-design, data-generation, evaluation - Published: 2024-04-18 · Authors: Jason Wu, Yi-Hao Peng, Amanda Li, Amanda Swearngin, Jeffrey P. Bigham, Jeffrey Nichols ### UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset Introduces designer-authored mobile UI critiques and quality ratings, studying how targeted examples improve automated design feedback. - https://awesomecua.com/resources/arxiv-2407-08850/ · source: https://arxiv.org/abs/2407.08850v3 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: screen-understanding, evaluation - Published: 2024-07-11 · Authors: Peitong Duan, Chin-yi Chen, Gang Li, Bjoern Hartmann, Yang Li ### UINav: A Practical Approach to Train On-Device Automation Agents Introduces UINav, a demonstration-based method for training mobile UI automation agents, with a referee model for failure feedback and automatic augmentation of examples. - https://awesomecua.com/resources/arxiv-2312-10170/ · source: https://arxiv.org/abs/2312.10170v4 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: imitation-learning, automation, human-in-the-loop, data-generation - Published: 2023-12-15 · Authors: Wei Li, Fu-Lin Hsu, Will Bishop, Folawiyo Campbell-Ajala, Max Lin, Oriana Riva ### UIPro: Unleashing Superior Interaction Capability For GUI Agents UIPro pretrains GUI understanding on 20.6 million tasks and then harmonizes interaction datasets through a unified action space. Continued fine-tuning uses the merged multi-platform data to train action prediction for generalist GUI operation. - https://awesomecua.com/resources/arxiv-2509-17328/ · source: https://arxiv.org/abs/2509.17328v1 - Paper · reviewed 2026-10-17 · environments: cross-platform · topics: data-generation, action-prediction, grounding, imitation-learning - Published: 2025-09-22 · Authors: Hongxin Li, Jingran Su, Jingfan Chen, Zheng Ju, Yuntao Chen, Qing Li, Zhaoxiang Zhang ### UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments Introduces UISim, an image-based mobile UI simulator that predicts the next screen layout after an action and renders the resulting screen for testing and data generation. - https://awesomecua.com/resources/arxiv-2509-21733/ · source: https://arxiv.org/abs/2509.21733v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: world-models, sandbox, data-generation, mobile-navigation - Published: 2025-09-26 · Authors: Jiannan Xiang, Yun Zhu, Lei Shu, Maria Wang, Lijun Yu, Gabriel Barcik, James Lyon, Srinivas Sunkara, Jindong Chen ### UITron-Speech: Towards Automated GUI Agents Based on Speech Instructions Introduces UITron-Speech, which predicts GUI actions from spoken instructions and screenshots. It combines synthetic speech data and mixed-modality training with a training-free grounding refinement step. - https://awesomecua.com/resources/arxiv-2506-11127/ · source: https://arxiv.org/abs/2506.11127v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, action-prediction, data-generation - Published: 2025-06-10 · Authors: Wenkang Han, Zhixiong Zeng, Jing Huang, Shu Jiang, Liming Zheng, Longrong Yang, Haibo Qiu, Chang Yao, Jingyuan Chen, Lin Ma ### UItron: Foundational GUI Agent with Advanced Perception and Planning UItron combines GUI perception and planning data engineering with supervised training and curriculum reinforcement learning in connected mobile and PC environments. It also collects more than one million operation steps across 100 Chinese apps and builds offline and online evaluation environments. - https://awesomecua.com/resources/arxiv-2508-21767/ · source: https://arxiv.org/abs/2508.21767v1 - Paper · reviewed 2026-10-17 · environments: desktop, cross-platform · topics: data-generation, reinforcement-learning, planning, grounding - Published: 2025-08-29 · Authors: Zhixiong Zeng, Jing Huang, Liming Zheng, Wenkang Han, Yufeng Zhong, Lei Chen, Longrong Yang, Yingjie Chu, Yuzhi He, Lin Ma ### UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action UltraCUA trains a shared policy for primitive GUI actions and high-level tool calls using synthesized tasks and hybrid trajectories. Supervised fine-tuning followed by online reinforcement learning is evaluated on OSWorld and WindowsAgentArena. - https://awesomecua.com/resources/arxiv-2510-17790/ · source: https://arxiv.org/abs/2510.17790v3 - Paper · reviewed 2026-10-17 · environments: desktop, windows · topics: tool-integration, reinforcement-learning, data-generation - Published: 2025-10-20 · Authors: Yuhao Yang, Zhen Yang, Zi-Yi Dou, Anh Nguyen, Keen You, Omar Attia, Andrew Szot, Michael Feng, Ram Ramrakhya, Alexander Toshev, Chao Huang, Yinfei Yang, Zhe Gan ### Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets Argus benchmarks post-hoc uncertainty methods for single-step GUI grounding across open-weight and closed-source observation regimes. It tests calibration, rejection, error severity and conformal click regions, finding that uncertainty rankings transfer less reliably across model/interface regimes than across datasets for one model. - https://awesomecua.com/resources/arxiv-2606-25760/ · source: https://arxiv.org/abs/2606.25760v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, evaluation, robustness, safety-security - Published: 2026-06-24 · Authors: Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi ### Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement RecAgent narrows attention to recommended UI components and requests user feedback when task intent or decisions remain uncertain. It introduces ComplexAction to evaluate specified single-step GUI actions in difficult interface scenarios. - https://awesomecua.com/resources/arxiv-2508-04025/ · source: https://arxiv.org/abs/2508.04025v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: mobile-navigation, human-in-the-loop, screen-understanding, evaluation - Published: 2025-08-06 · Authors: Chao Hao, Shuai Wang, Kaiwen Zhou ### Understanding GUI Agent Localization Biases through Logit Sharpness Introduces a finer-grained taxonomy of GUI localization errors and a logit-based Peak Sharpness Score for analyzing coordinate-prediction uncertainty. A context-aware cropping method refines the visual input without retraining the grounding model. - https://awesomecua.com/resources/arxiv-2506-15425/ · source: https://arxiv.org/abs/2506.15425v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, evaluation, robustness, training-free - Published: 2025-06-18 · Authors: Xingjian Tao, Yiwei Wang, Yujun Cai, Zhicheng Yang, Jing Tang ### Understanding HTML with Large Language Models Studies language models for HTML element classification, page description and autonomous web navigation. Fine-tuned models are evaluated on MiniWoB, and a distilled, automatically labeled CommonCrawl HTML corpus supports the training approach. - https://awesomecua.com/resources/arxiv-2210-03945/ · source: https://arxiv.org/abs/2210.03945v2 - Paper · reviewed 2026-10-17 · environments: web · topics: screen-understanding, web-navigation, data-generation - Published: 2022-10-08 · Authors: Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, Aleksandra Faust ### Understanding Mobile GUI: from Pixel-Words to Screen-Sentences Models mobile screens as atomic text and graphic components whose relationships are learned by a Screen Transformer. The work introduces RICO-PW annotations and evaluates metadata-free screen understanding on relation, clickability, retrieval, and app-classification tasks. - https://awesomecua.com/resources/arxiv-2105-11941/ · source: https://arxiv.org/abs/2105.11941v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding, data-generation - Published: 2021-05-25 · Authors: Jingwen Fu, Xiaoyi Zhang, Yuwang Wang, Wenjun Zeng, Sam Yang, Grayson Hilliard ### Understanding the Weakness of Large Language Model Agents within a Complex Android Environment Introduces AndroidArena and analyzes agent limitations in cross-application tasks, user constraints, exploration, and reflection. - https://awesomecua.com/resources/arxiv-2402-06596/ · source: https://arxiv.org/abs/2402.06596v1 - Paper · reviewed 2026-10-16 · environments: android · topics: evaluation, cross-app - Published: 2024-02-09 · Authors: Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, Zhen Xiao ### Understanding Tool-Mediated Interaction Failures in Agentic GUI Software: An Empirical Study Malva catalogs failures that make GUI agents' tool-mediated view diverge from the live interface; MalvaGuard checks related tool contracts before deployment. - https://awesomecua.com/resources/arxiv-2610-31698/ · source: https://arxiv.org/abs/2610.31698v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: observability, state-tracking, evaluation, tool-integration - Published: 2026-10-21 · Authors: Yuchen Shao, Wenjing Liu, Yuheng Huang, Mengqian Xu, Ziqun Bao, Long Yang, Lei Ma, Ting Su, Chengcheng Wan ### Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes Maps design considerations for computer-use agent experiences and introduces AgentUXlab, a browser-based probe for prototyping and evaluating agent UX on a website. - https://awesomecua.com/resources/arxiv-2510-04452/ · source: https://arxiv.org/abs/2510.04452v3 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, human-in-the-loop, evaluation - Published: 2025-10-06 · Authors: Jenny T. Liang, Titus Barik, Jeffrey Nichols, Eldon Schoop, Ruijia Cheng ### UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks Introduces UniClawBench, a benchmark of proactive agents on bilingual real-world tasks, evaluated in live Docker containers with stepwise checkpoints and simulated user feedback. - https://awesomecua.com/resources/arxiv-2607-08768/ · source: https://arxiv.org/abs/2607.08768v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, sandbox, human-in-the-loop - Published: 2026-07-09 · Authors: Zhekai Chen, Chengqi Duan, Kaiyue Sun, Bohao Li, Yuqing Wang, Manyuan Zhang, Xihui Liu ### Unlocking Smarter Device Control: Foresighted Planning with a World Model-Driven Code Execution Approach FPWC builds and refines a task-oriented world model before planning mobile-device actions. It executes plans as code and evaluates the approach in simulated environments and on real mobile devices. - https://awesomecua.com/resources/arxiv-2505-16422/ · source: https://arxiv.org/abs/2505.16422v3 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: world-models, planning, mobile-navigation - Published: 2025-05-22 · Authors: Xiaoran Yin, Xu Luo, Hao Wu, Lianli Gao, Jingkuan Song ### Untrusted Content Masking for Web Agents with Security Guarantees Untrusted Content Masking uses DOM structure to redact untrusted webpage regions before agent observation. A sandboxed interaction interface separates privileges so browser actions can proceed under the proposed trust-boundary model without exposing protected content to the planner. - https://awesomecua.com/resources/arxiv-2607-05277/ · source: https://arxiv.org/abs/2607.05277v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, sandbox, web-navigation - Published: 2026-07-06 · Authors: Kristina Nikolić, Egor Zverev, Javier Rando, Matthew Jagielski, Edoardo Debenedetti, Florian Tramèr ### Unveiling Disparities in Web Task Handling Between Human and Web Agent Compares human think-aloud web-task behavior with web-agent planning, action, and reflection. The analysis highlights differences in updating plans, handling ambiguity, and investigating failures. - https://awesomecua.com/resources/arxiv-2405-04497/ · source: https://arxiv.org/abs/2405.04497v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, human-in-the-loop, planning - Published: 2024-05-07 · Authors: Kihoon Son, Jinhyeon Kwon, DaEun Choi, Tae Soo Kim, Young-Ho Kim, Sangdoo Yun, Juho Kim ### Unveiling the Tricks: Automated Detection of Dark Patterns in Mobile Applications Introduces UIGuard, a system that examines mobile UI visuals and text to detect dark patterns, together with a labeled dataset of benign and malicious interfaces. - https://awesomecua.com/resources/arxiv-2308-05898/ · source: https://arxiv.org/abs/2308.05898v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: ui-design, screen-understanding, data-generation - Published: 2023-08-11 · Authors: Jieshan Chen, Jiamou Sun, Sidong Feng, Zhenchang Xing, Qinghua Lu, Xiwei Xu, Chunyang Chen ### UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model Introduces UReader, which combines unified instruction tuning, auxiliary text-reading tasks and shape-adaptive image crops for OCR-free visual language understanding. Its evaluation includes webpage screenshots alongside documents, tables, charts and natural images. - https://awesomecua.com/resources/arxiv-2310-05126/ · source: https://arxiv.org/abs/2310.05126v1 - Paper · reviewed 2026-10-17 · environments: web · topics: screen-understanding, efficiency - Published: 2023-10-08 · Authors: Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, Fei Huang ### Using GUI Agent for Electronic Design Automation GUI-EDA collects more than 2,000 screenshot-answer-action pairs from engineers working across five CAD tools and five physical domains. The accompanying benchmark evaluates more than 30 GUI agents on specialized electronic-design workflows. - https://awesomecua.com/resources/arxiv-2512-11611/ · source: https://arxiv.org/abs/2512.11611v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: industry, data-generation, evaluation, workflow - Published: 2025-12-12 · Authors: Chunyi Li, Longfei Li, Zicheng Zhang, Xiaohong Liu, Min Tang, Weisi Lin, Guangtao Zhai ### UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design Introduces UXAgent, which uses simulated users and a browser connector to test website usability and present their actions, responses, and recordings to UX researchers. - https://awesomecua.com/resources/arxiv-2502-12561/ · source: https://arxiv.org/abs/2502.12561v3 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, web-navigation, video, evaluation - Published: 2025-02-18 · Authors: Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Yang Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, Dakuo Wang ### V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM Describes V-Zen, a multimodal GUI model with dual-resolution image encoders for grounding and next-action prediction, fine-tuned using GUIDE interface elements and task sequences. - https://awesomecua.com/resources/arxiv-2405-15341/ · source: https://arxiv.org/abs/2405.15341v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, action-prediction, screen-understanding - Published: 2024-05-24 · Authors: Abdur Rahman, Rajat Chawla, Muskaan Kumar, Arkajit Datta, Adarsh Jha, Mukunda NS, Ishaan Bhola ### V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peaking V2P trains GUI grounding by suppressing attention to background regions and emphasizing the center of the target element. It represents target importance with size-dependent Gaussian heatmaps and evaluates the contribution of each component on ScreenSpot-v2 and ScreenSpot-Pro. - https://awesomecua.com/resources/arxiv-2508-13634/ · source: https://arxiv.org/abs/2508.13634v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, screen-understanding - Published: 2025-08-19 · Authors: Jikai Chen, Long Chen, Dong Wang, Qinglin Su, Zhixuan Chu, Bingguang Hao, Leilei Gan, Chenyi Zhuang, Jinjie Gu ### VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model Trains GUI agents with a frozen value model learned from offline interactions, guiding action exploration without predicting complete next observations. - https://awesomecua.com/resources/arxiv-2502-18906/ · source: https://arxiv.org/abs/2502.18906v1 - Paper · reviewed 2026-10-16 · environments: android · topics: reinforcement-learning, world-models - Published: 2025-02-26 · Authors: Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang ### VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks VenusBench-GD is a bilingual, cross-platform GUI grounding benchmark organized into basic and advanced tasks. Its annotation pipeline and six-subtask taxonomy support comparisons of general multimodal models and GUI-specialized models across different grounding demands. - https://awesomecua.com/resources/arxiv-2512-16501/ · source: https://arxiv.org/abs/2512.16501v1 - Paper · reviewed 2026-10-17 · environments: cross-platform · topics: grounding, evaluation, data-generation, robustness - Published: 2025-12-18 · Authors: Beitong Zhou, Zhexiao Huang, Yuan Guo, Zhangxuan Gu, Tianyu Xia, Zichen Luo, Fei Tang, Dehan Kong, Yanyi Shang, Suling Ou, Zhenlin Guo, Changhua Meng, Shuheng Shen ### VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics Introduces VenusBench-Mobile, an online benchmark for mobile GUI agents with user-intent-driven tasks and capability annotations for analyzing agent behavior and failures. - https://awesomecua.com/resources/arxiv-2604-06182/ · source: https://arxiv.org/abs/2604.06182v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, mobile-navigation, observability - Published: 2026-02-06 · Authors: Yichen Gong, Zhuohan Cai, Sunhao Dai, Yuqi Zhou, Zhangxuan Gu, Changhua Meng, Shuheng Shen ### VeriCUA: Learning Proactive Verification for Reliable GUI Agent Rewards VeriCUA trains a Judge to select verification checks and stopping decisions while a frozen Probe inspects the post-task GUI. Single-attempt reinforcement learning balances verdict correctness and Probe use. - https://awesomecua.com/resources/arxiv-2610-26469/ · source: https://arxiv.org/abs/2610.26469v1 - Paper · reviewed 2026-10-20 · environments: desktop, linux, web, android · topics: reward-models, reinforcement-learning, evaluation, efficiency - Published: 2026-10-19 · Authors: Zhizhou Wang, Yaxi Lu, Zhong Zhang, Zikun Hu, Xin Cong, Yankai Lin, Fangming Liu ### VeriCUA: Verification as Agentic Exploration for Computer-Using Agents Introduces VeriCUA, a cross-platform task verifier that gathers evidence through GUI or code checks to assess computer-use agent completion. - https://awesomecua.com/resources/arxiv-2610-40578/ · source: https://arxiv.org/abs/2610.40578v1 - Paper · reviewed 2026-11-06 · environments: desktop, web, cross-platform · topics: evaluation, tool-integration, observability - Published: 2026-10-29 · Authors: Qiushi Sun, Da Yu, Alexander Knop, Amer Sinha, Lingpeng Kong, Ben Kao, Ravi Kumar, Pritish Kamath ### VerificAgent: Domain-Specific Memory Verification for Scalable Oversight of Aligned Computer-Use Agents Introduces VerificAgent, which grows domain-specific memories from task trajectories and subjects them to human fact-checking before deployment. Its OSWorld evaluation treats persistent memory as an auditable constraint on later agent behavior. - https://awesomecua.com/resources/arxiv-2506-02539/ · source: https://arxiv.org/abs/2506.02539v3 - Paper · reviewed 2026-10-17 · environments: desktop · topics: memory, human-in-the-loop, safety-security - Published: 2025-06-03 · Authors: Thong Q. Nguyen, Shubhang Desai, Raja Hasnain Anwar, Firoz Shaik, Vishwas Suryanarayanan, Vishal Chowdhary ### VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents Introduces VeriOS-Agent, which learns when to execute GUI actions autonomously and when to query a user under untrustworthy conditions. Its staged training combines supervised learning and group-relative policy optimization. - https://awesomecua.com/resources/arxiv-2509-07553/ · source: https://arxiv.org/abs/2509.07553v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: human-in-the-loop, safety-security, reinforcement-learning - Published: 2025-09-09 · Authors: Zheng Wu, Heyuan Huang, Xingyu Lou, Xiangmou Qu, Pengzhou Cheng, Zongru Wu, Weiwen Liu, Weinan Zhang, Jun Wang, Zhaoxiang Wang, Zhuosheng Zhang ### VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification Translates user instructions into formal specifications and applies runtime logic-based checks to proposed mobile GUI actions. - https://awesomecua.com/resources/arxiv-2503-18492/ · source: https://arxiv.org/abs/2503.18492v2 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: safety-security, mobile-navigation - Published: 2025-03-24 · Authors: Jungjae Lee, Dongjae Lee, Chihun Choi, Youngmin Im, Jaeyoung Wi, Kihong Heo, Sangeun Oh, Sunjae Lee, Insik Shin ### VeriWeb: Verifiable Long-Chain Web Benchmark for Agentic Information-Seeking Introduces VeriWeb, a benchmark of long-chain web information-seeking tasks with verifiable subtasks and a complete human-annotated trajectory for each task. - https://awesomecua.com/resources/arxiv-2508-04026/ · source: https://arxiv.org/abs/2508.04026v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, long-horizon, data-generation - Published: 2025-08-06 · Authors: Shunyu Liu, Minghao Liu, Huichi Zhou, Zhenyu Cui, Yang Zhou, Yuhao Zhou, Jialiang Gao, Heng Zhou, Yunhao Yang, Wendong Fan, puzhen zhang, Ge Zhang, Jiajun Shi, Weihao Xuan, Jiaxing Huang, Shuang Luo, Fang Wu, Heli Qi, Qingcheng Zeng, Junjie Wang, Aosong Feng, Jindi Lv, Sicong Jiang, Ziqi Ren, Wangchunshu Zhou, Zhenfei Yin, Wenlong Zhang, Guohao Li, Wenhao Yu, Lei Ma, Lei Bai, Qunshu Lin, Mingli Song, Dacheng Tao ### VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning Uses image-dependent question-answer data and staged fine-tuning to improve GUI comprehension and reduce unsupported visual answers. - https://awesomecua.com/resources/arxiv-2406-14056/ · source: https://arxiv.org/abs/2406.14056v3 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: screen-understanding, grounding - Published: 2024-06-20 · Authors: Ziyang Meng, Yu Dai, Zezheng Gong, Shaoxiong Guo, Minglong Tang, Tongquan Wei ### Video-Based Reward Modeling for Computer-Use Agents Builds ExeVR-53k from execution-video, instruction and reward triplets, including negative instructions with step annotations. A reward model uses spatiotemporal token pruning to judge task completion from the instruction and execution keyframes. - https://awesomecua.com/resources/arxiv-2603-10178/ · source: https://arxiv.org/abs/2603.10178v1 - Paper · reviewed 2026-10-17 · environments: desktop, linux, macos, windows, android · topics: reward-models, video, evaluation - Published: 2026-03-10 · Authors: Linxin Song, Jieyu Zhang, Huanxin Sheng, Taiwei Shi, Gupta Rahul, Yang Liu, Ranjay Krishna, Jian Kang, Jieyu Zhao ### Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining Video2GUI extracts GUI interaction trajectories from tutorial videos to build WildGUI, a pretraining dataset spanning applications and websites. - https://awesomecua.com/resources/arxiv-2605-14747/ · source: https://arxiv.org/abs/2605.14747v1 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, video, imitation-learning - Published: 2026-05-14 · Authors: Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, Hao Tian ### VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos Extracts structured GUI actions from screen-recorded tutorials using inverse dynamics, then uses the inferred trajectories for computer-use training. - https://awesomecua.com/resources/arxiv-2510-19488/ · source: https://arxiv.org/abs/2510.19488v1 - Paper · reviewed 2026-10-16 · environments: desktop, web · topics: data-generation, video - Published: 2025-10-22 · Authors: Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang, Zekun Wang, Junlin Yang, Hongjin Su, Jixuan Chen, Junda Chen, Yuchen Mao, Jingren Zhou, Junyang Lin, Binyuan Hui, Tao Yu ### VideoCAD: A Dataset and Model for Learning Long-Horizon 3D CAD UI Interactions from Video Introduces synthetic CAD interaction videos, a model for learning interface actions from video, and evaluation tasks for spatial and long-horizon reasoning. - https://awesomecua.com/resources/arxiv-2505-24838/ · source: https://arxiv.org/abs/2505.24838v2 - Paper · reviewed 2026-10-18 · environments: web · topics: video, data-generation, science, long-horizon - Published: 2025-05-30 · Authors: Brandon Man, Ghadi Nehme, Md Ferdous Alam, Faez Ahmed ### VideoGUI: A Benchmark for GUI Automation from Instructional Videos Evaluates GUI automation from instructional videos at separate levels of high-level planning, action narration, and primitive execution. - https://awesomecua.com/resources/arxiv-2406-10227/ · source: https://arxiv.org/abs/2406.10227v1 - Paper · reviewed 2026-10-16 · environments: desktop, web · topics: evaluation, video, planning - Published: 2024-06-14 · Authors: Kevin Qinghong Lin, Linjie Li, Difei Gao, Qinchen WU, Mingyi Yan, Zhengyuan Yang, Lijuan Wang, Mike Zheng Shou ### VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks Evaluates whether web agents can retain procedural skills and factual information from long video tutorials and use them during tasks. - https://awesomecua.com/resources/arxiv-2410-19100/ · source: https://arxiv.org/abs/2410.19100v3 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, video, memory - Published: 2024-10-24 · Authors: Lawrence Jang, Yinheng Li, Dan Zhao, Charles Ding, Justin Lin, Paul Pu Liang, Rogerio Bonatti, Kazuhito Koishida ### ViMo: A Generative Visual GUI World Model for App Agents ViMo predicts the next mobile-app screenshot after a candidate action by separately modeling graphics and text. A symbolic placeholder representation preserves layout while a second predictor supplies readable text, enabling visual lookahead for GUI-agent planning. - https://awesomecua.com/resources/arxiv-2504-13936/ · source: https://arxiv.org/abs/2504.13936v2 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: world-models, planning, mobile-navigation - Published: 2025-04-15 · Authors: Dezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis, Jifei Song, Shaogang Gong, Jianye Hao, Jun Wang, Kun Shao ### VisCritic: Visual State Comparison as Process Reward for GUI Agents VisCritic compares before/after screenshots through change-aware visual features and an action-conditioned critic. Weakly supervised samples from existing traces train judgments of action success, progress and error type for GUI-agent verification. - https://awesomecua.com/resources/arxiv-2606-24525/ · source: https://arxiv.org/abs/2606.24525v1 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: reward-models, state-tracking, data-generation, evaluation - Published: 2026-06-23 · Authors: Jiachen Qian ### Visible but Not Avoided: State-Dependent Re-Proposal of Rejected Candidates in Catalog-Based LLM Optimization Loops Studies repeated rejected edits while optimizing a screenshot-only GUI agent, contrasting static probes with reached loop states and testing how rejection wording changes subsequent edit proposals. - https://awesomecua.com/resources/arxiv-2610-15738/ · source: https://arxiv.org/abs/2610.15738v1 - Paper · reviewed 2026-10-20 · environments: unspecified · topics: evaluation, grounding, training-free - Published: 2026-10-11 · Authors: Jianlin Chen, Wenhui Chen, Chi Man Vong - Licenses: document CC-BY-4.0 ### Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification Introduces Vision2Web, a benchmark for website development from images, spanning UI-to-code, multi-page frontend and full-stack tasks, with a GUI agent verifier and VLM-based judge. - https://awesomecua.com/resources/arxiv-2603-26648/ · source: https://arxiv.org/abs/2603.26648v3 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, ui-design, long-horizon - Published: 2026-03-27 · Authors: Zehai He, Wenyi Hong, Zhen Yang, Ziyang Pan, Mingdao Liu, Xiaotao Gu, Jie Tang ### VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning VisionTasker converts screenshots into textual UI interpretations and presents one interface at a time to an LLM planner for action selection. It evaluates visual representations and execution of 147 tasks on an Android smartphone, including integration with programming by demonstration. - https://awesomecua.com/resources/arxiv-2312-11190/ · source: https://arxiv.org/abs/2312.11190v2 - Paper · reviewed 2026-10-17 · environments: android · topics: screen-understanding, planning, mobile-navigation - Published: 2023-12-18 · Authors: Yunpeng Song, Yiheng Bian, Yongtao Tang, Guiyu Ma, Zhongmin Cai ### VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents VISTA evaluates coding agents that turn design handoffs into working web and Android applications. Its evaluator locates annotated interactive components and probes their behavior, keeping visual localization distinct from functional correctness. - https://awesomecua.com/resources/arxiv-2605-26144/ · source: https://arxiv.org/abs/2605.26144v4 - Paper · reviewed 2026-10-17 · environments: web, android · topics: evaluation, ui-design, automation - Published: 2026-05-22 · Authors: JunJia Guo, Yuhang Yao, Chang Li, Zimeng Pan, Jiawei, Zhou, Jingdi Chen ### VISTA: View-Consistent Self-Verified Training for GUI Grounding VISTA forms GRPO comparison groups from target-preserving crops of the same GUI instance with exactly remapped target boxes. A conditional, advantage-weighted cross-view anchor supplies additional supervision without entering the group baseline, and evaluation includes view-consistency robustness. - https://awesomecua.com/resources/arxiv-2606-14579/ · source: https://arxiv.org/abs/2606.14579v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, reinforcement-learning, robustness, data-generation - Published: 2026-06-12 · Authors: Xinyu Qiu, Yunzhu Zhang, Heng Jia, Shuheng Shen, Changhua Meng, Linchao Zhu ### Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agents Formalizes a visual confused-deputy failure in which a computer agent acts on a misperceived UI object. A separate guardrail checks both the visual target and the agent's action reasoning against deployment-specific knowledge before permitting execution. - https://awesomecua.com/resources/arxiv-2603-14707/ · source: https://arxiv.org/abs/2603.14707v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: safety-security, grounding, robustness - Published: 2026-03-16 · Authors: Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen ### Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces Compares modular OCR-based and end-to-end methods for locating desktop GUI targets from natural-language instructions. - https://awesomecua.com/resources/arxiv-2407-01558/ · source: https://arxiv.org/abs/2407.01558v3 - Paper · reviewed 2026-10-16 · environments: desktop · topics: grounding, evaluation - Published: 2024-05-05 · Authors: El Hassane Ettifouri, Jessica López Espejel, Laura Minkova, Tassnim Dardouri, Walid Dahhane ### Visual Test-time Scaling for GUI Agent Grounding RegionFocus adds visual test-time scaling by zooming into selected screenshot regions and maintaining an image-based map of landmarks and actions. It evaluates this grounding procedure with UI-TARS and Qwen2.5-VL on ScreenSpot-Pro and WebVoyager. - https://awesomecua.com/resources/arxiv-2505-00684/ · source: https://arxiv.org/abs/2505.00684v2 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, screen-understanding, web-navigation - Published: 2025-05-01 · Authors: Tiange Luo, Lajanugen Logeswaran, Justin Johnson, Honglak Lee ### Visual Token Routing: Learning to keep tokens that matter Introduces Visual Token Routing (VTR), which selects visual tokens from current and earlier screenshots to reduce computer-use agent inference cost while retaining action-relevant information. - https://awesomecua.com/resources/arxiv-2611-06854/ · source: https://arxiv.org/abs/2611.06854v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, action-prediction, interaction-history, screen-understanding - Published: 2026-11-04 · Authors: Pranjal Aggarwal, Sean Welleck ### VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents Benchmarks visual agents across GUI, embodied, and design tasks, with training trajectories collected through solvers, agents, and human demonstrations. - https://awesomecua.com/resources/arxiv-2408-06327/ · source: https://arxiv.org/abs/2408.06327v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: evaluation, data-generation - Published: 2024-08-12 · Authors: Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, Jie Tang ### VISUALSKILL: Multimodal Skills for Computer-Use Agents VISUALSKILL organizes application-specific text and figures into hierarchical skills retrieved on demand through an MCP tool. Skills combine authored documentation with live UI exploration, and matched multimodal/text-only comparisons test their utility for interface identification and workflow-state verification. - https://awesomecua.com/resources/arxiv-2606-18448/ · source: https://arxiv.org/abs/2606.18448v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: memory, tool-integration, screen-understanding, long-horizon - Published: 2026-06-16 · Authors: Ziyan Jiang, Li An, Yujian Liu, Jiabao Ji, Qiucheng Wu, Jacob Andreas, Yang Zhang, Shiyu Chang ### VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation Studies backdoors that redirect GUI visual grounding even when the task plan is correct. VisualTrap evaluates whether poisoned grounding training transfers to downstream tasks and across web, mobile, and desktop interface settings. - https://awesomecua.com/resources/arxiv-2507-06899/ · source: https://arxiv.org/abs/2507.06899v2 - Paper · reviewed 2026-10-17 · environments: web, desktop · topics: safety-security, grounding, evaluation - Published: 2025-07-09 · Authors: Ziang Ye, Yang Zhang, Wentao Shi, Xiaoyu You, Fuli Feng, Tat-Seng Chua ### VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks Introduces visually grounded website tasks for evaluating multimodal agents that must interpret both images and text. - https://awesomecua.com/resources/arxiv-2401-13649/ · source: https://arxiv.org/abs/2401.13649v2 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, grounding, web-navigation - Published: 2024-01-24 · Authors: Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, Daniel Fried ### VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding? Separately evaluates web-page reading, understanding, and element grounding across tasks drawn from real websites. - https://awesomecua.com/resources/arxiv-2404-05955/ · source: https://arxiv.org/abs/2404.05955v1 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, screen-understanding, grounding - Published: 2024-04-09 · Authors: Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, Xiang Yue ### VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation VLAA-GUI combines mandatory checks for visually supported completion, multi-tier loop recovery and on-demand workflow search. Optional coding and grounding agents support execution, with component ablations on Linux and Windows GUI tasks. - https://awesomecua.com/resources/arxiv-2604-21375/ · source: https://arxiv.org/abs/2604.21375v3 - Paper · reviewed 2026-10-18 · environments: desktop, linux, windows · topics: robustness, self-reflection, tool-integration, evaluation - Published: 2026-04-23 · Authors: Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie ### VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought ICAL turns imperfect agent trajectories into reusable strategies and action annotations through reflection and human feedback, with VisualWebArena among its evaluations. - https://awesomecua.com/resources/arxiv-2406-14596/ · source: https://arxiv.org/abs/2406.14596v6 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, self-reflection, human-in-the-loop, data-generation - Published: 2024-06-20 · Authors: Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, Katerina Fragkiadaki ### VLM-Fuzz: Vision Language Model Assisted Recursive Depth-first Search Exploration for Effective UI Testing of Android Apps VLM-Fuzz combines depth-first UI exploration with Android manifest analysis, runtime widget hierarchies, and on-demand visual reasoning to generate app test inputs. It evaluates coverage on 59 benchmark apps and uses additional Google Play apps to study crash discovery. - https://awesomecua.com/resources/arxiv-2504-11675/ · source: https://arxiv.org/abs/2504.11675v1 - Paper · reviewed 2026-10-17 · environments: android · topics: automation, evaluation, mobile-navigation - Published: 2025-04-16 · Authors: Biniam Fisseha Demissie, Yan Naing Tun, Lwin Khin Shar, Mariano Ceccato ### Voluntary Servitude: Weaponized Safety Warning Mechanisms for Controlling AI Agents Studies fabricated hazard warnings in reviews that steer agents’ choices without explicit commands, including transfer to browser agents, and tests a corroboration requirement. - https://awesomecua.com/resources/arxiv-2611-04393/ · source: https://arxiv.org/abs/2611.04393v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, robustness, web-navigation - Published: 2026-11-03 · Authors: Ilan Zendel, Matan Levy, Stav Cohen, Avi Mendelson, Amit LeVi ### VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents Introduces VPI-Bench, interactive web-platform variants containing visually embedded malicious instructions. It evaluates computer-use and browser-use agents under these threats and examines system-prompt defenses. - https://awesomecua.com/resources/arxiv-2506-02456/ · source: https://arxiv.org/abs/2506.02456v2 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation - Published: 2025-06-03 · Authors: Tri Cao, Bennett Lim, Yue Liu, Yuan Sui, Yuexin Li, Shumin Deng, Lin Lu, Nay Oo, Shuicheng Yan, Bryan Hooi ### VRL-Bench: Benchmarking agents on computer control tasks under finite trial budgets Introduces VRL-Bench to compare trial-and-error learning for computer-control agents under limited attempts, and VEX² to schedule exploration and exploitation. - https://awesomecua.com/resources/arxiv-2609-12404/ · source: https://arxiv.org/abs/2609.12404v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, memory, planning - Published: 2026-09-11 · Authors: Yu Bai, Yukai Miao, Dawei Wang, Li Chen, Yanyu Ren, Yuqian Shi, Dan Li, Ying Xiong, Chengqiu Tan, Run Zhou, Li Li ### VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling The Versatile UI Transformer jointly models interface screenshots, structure, and language for five UI tasks. Its encoder supports element detection when structural metadata is absent, while an autoregressive component handles UI question answering and command grounding. - https://awesomecua.com/resources/arxiv-2112-05692/ · source: https://arxiv.org/abs/2112.05692v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding, efficiency - Published: 2021-12-10 · Authors: Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani, Alexey Gritsenko ### WAAA! Web Adversaries Against Agentic Browsers Evaluates agentic browsers against classical web attacks and visual social-engineering lures beyond indirect prompt injection. - https://awesomecua.com/resources/arxiv-2605-05509/ · source: https://arxiv.org/abs/2605.05509v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2026-05-06 · Authors: Sohom Datta, Alex Nahapetyan, William Enck, Alexandros Kapravelos ### WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents Introduces WAInjectBench with a threat-model taxonomy and malicious and benign text-image datasets for web-agent prompt-injection detection. It compares detector families across explicit and less visible forms of injected content. - https://awesomecua.com/resources/arxiv-2510-01354/ · source: https://arxiv.org/abs/2510.01354v2 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, robustness - Published: 2025-10-01 · Authors: Yinuo Liu, Xilong Wang, Ruohan Xu, Yuqi Jia, Neil Zhenqiang Gong ### WaitLens: Temporal VLM-Based Adaptive Waiting for Industrial GUI Testing Agents Introduces WaitLens, a vision-language model trained on GUI loading trajectories to decide when testing agents can stop waiting for an interface to load. - https://awesomecua.com/resources/arxiv-2610-32482/ · source: https://arxiv.org/abs/2610.32482v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: efficiency, screen-understanding, industry - Published: 2026-10-23 · Authors: Shika Ikeura, Xuan Wang, Ziyi Kuang, Jianping Xu, Chao Kang, Yongxiang Hu, Xiaoyu Wang, Juxing Yuan, Yangfan Zhou ### WALT: Web Agents that Learn Tools WALT explores website functionality and converts demonstrated interactions into validated, reusable tools combining DOM actions, URL navigation, and agentic fallbacks. It evaluates this expanded browser action space on WebArena and VisualWebArena. - https://awesomecua.com/resources/arxiv-2510-01524/ · source: https://arxiv.org/abs/2510.01524v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration, efficiency, automation - Published: 2025-10-01 · Authors: Viraj Prabhu, Yutong Dai, Matthew Fernandez, Jing Gu, Krithika Ramakrishnan, Yanqi Luo, Silvio Savarese, Caiming Xiong, Junnan Li, Zeyuan Chen, Ran Xu ### WARC-Bench: Web Archive Based Benchmark for GUI Subtask Executions WARC-Bench packages dynamic web pages as replayable archives and evaluates short navigation subtasks involving components such as date pickers and scrollable containers. It also studies supervised training followed by reinforcement learning with verifiable rewards on these browser interactions. - https://awesomecua.com/resources/arxiv-2510-09872/ · source: https://arxiv.org/abs/2510.09872v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, sandbox, reinforcement-learning - Published: 2025-10-10 · Authors: Sanjari Srivastava, Gang Li, Cheng Chang, Rishu Garg, Manpreet Kaur, Charlene Y. Lee, Yuezhang Li, Yining Mao, Ignacio Cases, Yanan Xie, Peng Qi ### WARD: Adversarially Robust Defense of Web Agents Against Prompt Injections WARD guards web agents against prompt injections in HTML and visual interfaces, using web data and adaptive attacker–guard training. - https://awesomecua.com/resources/arxiv-2605-15030/ · source: https://arxiv.org/abs/2605.15030v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, robustness, data-generation - Published: 2026-05-14 · Authors: Tri Cao, Yulin Chen, Hieu Cao, Yibo Li, Khoi Le, Thong Nguyen, Yuexin Li, Yufei He, Yue Liu, Shuicheng Yan, Bryan Hooi ### WAREX: Web Agent Reliability Evaluation on Existing Benchmarks WAREX augments existing web-agent benchmarks with reliability challenges arising from unstable systems and changing or disruptive website behavior. Evaluation on WebArena, WebVoyager and REAL examines how these disturbances affect task completion. - https://awesomecua.com/resources/arxiv-2510-03285/ · source: https://arxiv.org/abs/2510.03285v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, robustness, web-navigation - Published: 2025-09-28 · Authors: Su Kara, Fazle Faisal, Suman Nath ### WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks Introduces WASP, an end-to-end benchmark for prompt-injection security in web agents. Its evaluation separates partial attack progress from completion of the attacker’s full objective. - https://awesomecua.com/resources/arxiv-2504-18575/ · source: https://arxiv.org/abs/2504.18575v3 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2025-04-22 · Authors: Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, Kamalika Chaudhuri ### Watch and Learn: Learning to Use Computers from Online Videos Introduces Watch & Learn (W&L), which converts human computer-use videos into UI action trajectories by predicting actions between consecutive screen states. - https://awesomecua.com/resources/arxiv-2510-04673/ · source: https://arxiv.org/abs/2510.04673v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: video, data-generation, action-prediction - Published: 2025-10-06 · Authors: Chan Hee Song, Yiwen Song, Palash Goyal, Yu Su, Oriana Riva, Hamid Palangi, Tomas Pfister ### Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents Studies backdoor attacks on LLM agents, including triggers in user queries or intermediate observations, with web shopping among the evaluated agent tasks. - https://awesomecua.com/resources/arxiv-2402-11208/ · source: https://arxiv.org/abs/2402.11208v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation - Published: 2024-02-17 · Authors: Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, Xu Sun ### Weasel: Out-of-Domain Generalization for Web Agents via Importance-Diversity Data Selection Weasel selects web-agent training steps by balancing importance and diversity across states, sites and interaction patterns. It also prunes training accessibility trees around known action targets and regenerates rationales to match the model’s reasoning style. - https://awesomecua.com/resources/arxiv-2605-20291/ · source: https://arxiv.org/abs/2605.20291v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, imitation-learning, efficiency, robustness, data-generation - Published: 2026-05-19 · Authors: Fatemeh Pesaran Zadeh, Seyeon Choi, Xing Han Lù, Siva Reddy, Gunhee Kim ### WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces WeaveBench requires agents to combine GUI, command-line and code operations within long tasks on an Ubuntu desktop. A trajectory-aware judge checks deliverables and interaction evidence, including shortcut behavior that outcome-only grading can miss. - https://awesomecua.com/resources/arxiv-2606-09426/ · source: https://arxiv.org/abs/2606.09426v3 - Paper · reviewed 2026-10-17 · environments: desktop, linux · topics: evaluation, long-horizon, workflow, tool-integration - Published: 2026-06-08 · Authors: Wanli Li, Bowen Zhou, Yunyao Yu, Zhou Xu, Yifan Yang, Dongsheng Li, Caihua Shan ### Web Agent Agentic Reinforcement Learning Decision Model Under Multi-Cost and Failure Risk Proposes constrained reinforcement learning for web agents that weighs request costs, latency and failure risk across multi-step website tasks. - https://awesomecua.com/resources/arxiv-2610-23857/ · source: https://arxiv.org/abs/2610.23857v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, web-navigation, efficiency, planning - Published: 2026-01-12 · Authors: Qianli Ma, Limengxi Yue, Shuyang Xu, Yanpei Shi, Hongrui Liu ### Web Agents Should Adopt the Plan-Then-Execute Paradigm Argues for compiling web tasks into predefined execution graphs before exposing an agent to untrusted runtime content. Its WebArena task analysis identifies typed, semantically complete website tools as a prerequisite and distinguishes this interface proposal from reactive click-level control. - https://awesomecua.com/resources/arxiv-2605-14290/ · source: https://arxiv.org/abs/2605.14290v1 - Paper · reviewed 2026-10-18 · environments: web · topics: planning, safety-security, tool-integration - Published: 2026-05-14 · Authors: Julien Piet, Annabella Chow, Yiwei Hou, Muxi Lyu, Sylvie Venuto, Jinhao Zhu, Raluca Ada Popa, David Wagner ### Web Agents Should Use Typed Actions Instead of Click-Based Browsing Proposes web verbs as typed actions for web agents, exposing operations with structured inputs, outputs and conditions so agents can compose auditable workflows. - https://awesomecua.com/resources/arxiv-2602-17245/ · source: https://arxiv.org/abs/2602.17245v2 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, tool-integration, workflow, observability - Published: 2026-02-19 · Authors: Linxi Jiang, Rui Xi, Zhijie Liu, Shuo Chen, Zhiqiang Lin, Suman Nath ### Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation Uses a learned web world model to predict important state changes and compare candidate actions before executing them. - https://awesomecua.com/resources/arxiv-2410-13232/ · source: https://arxiv.org/abs/2410.13232v2 - Paper · reviewed 2026-10-16 · environments: web · topics: world-models, planning - Published: 2024-10-17 · Authors: Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, Jinyoung Yeo ### Web$hell: A Unified Interface for Agentic Web Access Introduces Web$hell, an interface that compiles live pages into persistent information and action objects so agents can gather evidence, navigate websites, and reuse acquired content through shared operations. - https://awesomecua.com/resources/arxiv-2610-13416/ · source: https://arxiv.org/abs/2610.13416v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration, memory, efficiency - Published: 2026-10-09 · Authors: Sunghwan Kim, Deogyong Kim, Sangam Lee, Donguk Kwon, Dongha Lee ### Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agents Web-CogReasoner separates factual and conceptual web knowledge from procedural reasoning, using a structured curriculum to train a multimodal agent. Web-CogDataset and Web-CogBench support study of these capabilities alongside web-navigation evaluation. - https://awesomecua.com/resources/arxiv-2508-01858/ · source: https://arxiv.org/abs/2508.01858v3 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, planning, evaluation, web-navigation - Published: 2025-08-03 · Authors: Yuhan Guo, Cong Guo, Aiwen Sun, Hongliang He, Xinyu Yang, Yue Lu, Yingji Zhang, Xuntao Guo, Dong Zhang, Jianzhuang Liu, Jiang Duan, Yijia Xiao, Liangjian Wen, Hai-Ming Xu, Yong Dai ### Web-Shepherd: Advancing PRMs for Reinforcing Web Agents Web-Shepherd is a process reward model for assessing individual steps of web-navigation trajectories. It introduces a collection of 40,000 step-level preference pairs with checklists and a separate benchmark for comparing reward models, then evaluates verification during WebArena-lite navigation. - https://awesomecua.com/resources/arxiv-2505-15277/ · source: https://arxiv.org/abs/2505.15277v2 - Paper · reviewed 2026-10-17 · environments: web · topics: reinforcement-learning, evaluation, data-generation, web-navigation - Published: 2025-05-21 · Authors: Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Seungjun Moon, Gyeom Hwangbo, Dongha Lim, Minjin Kim, Yeonjun Hwang, Minju Gwak, Dongwook Choi, Minseok Kang, Gwanhoon Im, ByeongUng Cho, Hyojun Kim, Jun Hee Han, Taeyoon Kwon, Minju Kim, Beong-woo Kwak, Dongjin Kang, Jinyoung Yeo ### WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning Trains web agents through asynchronous multi-turn environment interaction, using task-success rewards to optimize complete trajectories. - https://awesomecua.com/resources/arxiv-2505-16421/ · source: https://arxiv.org/abs/2505.16421v2 - Paper · reviewed 2026-10-16 · environments: web · topics: reinforcement-learning, web-navigation - Published: 2025-05-22 · Authors: Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, Lihong Li ### WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web Agents WebAgentGuard separates prompt-injection detection from the operating web agent through a parallel multimodal guard. The guard is trained on synthetic visual/textual examples using reasoning-focused supervised fine-tuning followed by reinforcement learning. - https://awesomecua.com/resources/arxiv-2604-12284/ · source: https://arxiv.org/abs/2604.12284v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, data-generation, imitation-learning, reinforcement-learning, evaluation - Published: 2026-04-14 · Authors: Yulin Chen, Tri Cao, Haoran Li, Yue Liu, Yibo Li, Yufei He, Le Minh Khoi, Yangqiu Song, Shuicheng Yan, Bryan Hooi ### WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Models Introduces WebAggregator, a pipeline for synthesizing compositional questions from websites to train deep-research agents, plus the WebAggregatorQA testbed. - https://awesomecua.com/resources/arxiv-2510-14438/ · source: https://arxiv.org/abs/2510.14438v2 - Paper · reviewed 2026-11-06 · environments: web · topics: data-generation, evaluation, web-navigation - Published: 2025-10-16 · Authors: Rui Wang, Ce Zhang, Jun-Yu Ma, Jianshu Zhang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu, Kam-Fai Wong ### WebAnything: From the Wild Web to Scalable Agent Environments Introduces WebAnything, a framework that reconstructs reusable, resettable website environments from real URLs for web-agent training and evaluation. - https://awesomecua.com/resources/arxiv-2610-37786/ · source: https://arxiv.org/abs/2610.37786v1 - Paper · reviewed 2026-11-06 · environments: web · topics: sandbox, data-generation, evaluation, web-navigation - Published: 2026-10-27 · Authors: Tianci Xie, Zikun Cai, Chaoyun Zhang, Pinxue Guo, Bingxuan Li, Rui Zheng, Hang Yan ### WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents WebArbiter generates reasoned action-preference verdicts for web navigation, using reasoning distillation followed by reinforcement learning. It introduces WebPRMBench and evaluates both preference judgments and reward-guided trajectory search. - https://awesomecua.com/resources/arxiv-2601-21872/ · source: https://arxiv.org/abs/2601.21872v2 - Paper · reviewed 2026-10-17 · environments: web · topics: reward-models, reinforcement-learning, evaluation, web-navigation - Published: 2026-01-29 · Authors: Yao Zhang, Shijie Tang, Zeyu Li, Zhen Han, Volker Tresp ### WebArena: A Realistic Web Environment for Building Autonomous Agents Introduces reproducible web environments with task-completion checks for evaluating autonomous web agents. - https://awesomecua.com/resources/arxiv-2307-13854/ · source: https://arxiv.org/abs/2307.13854v4 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation, long-horizon - Published: 2023-07-25 · Authors: Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, Graham Neubig ### WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks WebArxiv provides 510 static-snapshot tasks with deterministic answers for scholarly browsing, including constrained paper retrieval, content extraction, and cross-paper comparison. The study also tests dynamic retrieval of interaction history to address repetitive or incomplete agent reasoning. - https://awesomecua.com/resources/arxiv-2507-00938/ · source: https://arxiv.org/abs/2507.00938v4 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, memory - Published: 2025-07-01 · Authors: Zihao Sun, Zijing Shi, Ling Chen ### WebATLAS: An LLM Agent with Experience-Driven Memory and Action Simulation WebATLAS is a web agent that stores interaction outcomes and simulates candidate actions before acting, using past experience to plan navigation on unseen websites. - https://awesomecua.com/resources/arxiv-2510-22732/ · source: https://arxiv.org/abs/2510.22732v2 - Paper · reviewed 2026-11-06 · environments: web · topics: memory, world-models, planning, web-navigation - Published: 2025-10-26 · Authors: Jiali Cheng, Anjishnu Kumar, Roshan Lal, Rishi Rajasekaran, Hani Ramezani, Omar Zia Khan, Oleg Rokhlenko, Sunny Chiu-Webster, Gang Hua, Hadi Amiri ### WebCanvas: Benchmarking Web Agents in Online Environments Introduces WebCanvas and Mind2Web-Live, evaluating online task progress through meaningful intermediate states on changing websites. - https://awesomecua.com/resources/arxiv-2406-12373/ · source: https://arxiv.org/abs/2406.12373v3 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation - Published: 2024-06-18 · Authors: Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, Zhengyang Wu ### WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces WebChain aligns human web-interaction trajectories with visual, structural and action data for agent training. It also introduces WebChainBench and a mid-training approach that separates spatial grounding from planning. - https://awesomecua.com/resources/arxiv-2603-05295/ · source: https://arxiv.org/abs/2603.05295v3 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, evaluation, grounding, planning - Published: 2026-03-05 · Authors: Sicheng Fan, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, Dehan Kong ### WebChallenger: A Reliable and Efficient Generalist Web Agent WebChallenger builds a hierarchical DOM-derived page representation for selective observation, reusable site maps and compound actions. The framework combines these mechanisms without model fine-tuning and evaluates generalization across multiple web-agent benchmarks. - https://awesomecua.com/resources/arxiv-2606-10423/ · source: https://arxiv.org/abs/2606.10423v1 - Paper · reviewed 2026-10-18 · environments: web · topics: web-navigation, memory, efficiency, training-free, workflow - Published: 2026-06-09 · Authors: Jayoo Hwang, Xiaowen Zhang, Vedant Padwal ### WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks Introduces WebChoreArena, 532 web chores built on WebArena environments to evaluate agents’ retention of page information, calculation, and tracking across pages. - https://awesomecua.com/resources/arxiv-2506-01952/ · source: https://arxiv.org/abs/2506.01952v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, memory, web-navigation, long-horizon - Published: 2025-06-02 · Authors: Atsuyuki Miyai, Zaiying Zhao, Kazuki Egashira, Atsuki Sato, Tatsumi Sunada, Shota Onohara, Hiromasa Yamanishi, Mashiro Toyooka, Kunato Nishina, Ryoma Maeda, Kiyoharu Aizawa, Toshihiko Yamasaki ### WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance Introduces WebCoach, which condenses navigation logs into persistent episodic memory and retrieves experiences by similarity and recency. Runtime hooks selectively supply advice to browser agents, evaluated on WebVoyager without retraining the underlying models. - https://awesomecua.com/resources/arxiv-2511-12997/ · source: https://arxiv.org/abs/2511.12997v2 - Paper · reviewed 2026-10-17 · environments: web · topics: memory, interaction-history, web-navigation - Published: 2025-11-17 · Authors: Genglin Liu, Shijie Geng, Sha Li, Hejie Cui, Sarah Zhang, Xin Liu, Tianyi Liu ### WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback WebCoT turns reflection, look-ahead, branching and rollback procedures into chain-of-thought training rationales. Fine-tuning on these reconstructed trajectories is evaluated for web-agent self-improvement and transfer across navigation tasks. - https://awesomecua.com/resources/arxiv-2505-20013/ · source: https://arxiv.org/abs/2505.20013v2 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, planning, self-reflection, web-navigation - Published: 2025-05-26 · Authors: Minda Hu, Tianqing Fang, Jianshu Zhang, Junyu Ma, Zhisong Zhang, Jingyan Zhou, Hongming Zhang, Haitao Mi, Dong Yu, Irwin King ### WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model WebEvolver trains a co-evolving world model to predict web observations and generate additional agent-training trajectories. The same model supports look-ahead action selection at inference, with evaluation on live web-task benchmarks. - https://awesomecua.com/resources/arxiv-2504-21024/ · source: https://arxiv.org/abs/2504.21024v2 - Paper · reviewed 2026-10-17 · environments: web · topics: world-models, data-generation, web-navigation, planning - Published: 2025-04-23 · Authors: Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, Dong Yu ### WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents Introduces WebFactory, a GUI-agent training pipeline that synthesizes websites, generates tasks and trajectories, and trains with decomposed rewards. Offline and online transfer evaluations examine agents trained on the generated environments. - https://awesomecua.com/resources/arxiv-2603-05044/ · source: https://arxiv.org/abs/2603.05044v2 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, reinforcement-learning, web-navigation - Published: 2026-03-05 · Authors: Sicheng Fan, Qingyun Shi, Shengze Xu, Shengbo Cai, Tieyong Zeng, Li Ling, Yanyi Shang, Dehan Kong ### WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark Presents WebForge, a pipeline that plans, generates, refines and validates self-contained interactive websites. WebForge-Bench uses these environments to evaluate browser agents across domains and controlled difficulty levels. - https://awesomecua.com/resources/arxiv-2604-10988/ · source: https://arxiv.org/abs/2604.10988v1 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, evaluation, web-navigation - Published: 2026-04-13 · Authors: Peng Yuan, Yuyang Yin, Yuxuan Cai, Zheng Wei ### WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites Describes WebFovea, a vision-based live-web agent whose execution loop checks action parsing, browser effects, feedback accuracy, and observation delivery. The report analyzes failures such as coordinate mismatches and silent control interactions and documents corresponding harness changes. - https://awesomecua.com/resources/arxiv-2610-03036/ · source: https://arxiv.org/abs/2610.03036v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, robustness, observability - Published: 2026-10-02 · Authors: Jiangang Han ### WebGames: Challenging General-Purpose Web-Browsing AI Agents Introduces WebGames, a browser-agent benchmark with 50+ interactive challenges in a self-contained test environment and verifiable solutions. - https://awesomecua.com/resources/arxiv-2502-18356/ · source: https://arxiv.org/abs/2502.18356v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, sandbox - Published: 2025-02-25 · Authors: George Thomas, Alex J. Chan, Jikun Kang, Wenqi Wu, Filippos Christianos, Fraser Greenlee, Andy Toulis, Marvin Purtorab ### WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning Introduces WebGen-Agent for website code generation, using screenshot critiques and GUI-agent tests to refine code and provide step-level training rewards. - https://awesomecua.com/resources/arxiv-2509-22644/ · source: https://arxiv.org/abs/2509.22644v1 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, reward-models, reinforcement-learning, self-reflection - Published: 2025-09-26 · Authors: Zimu Lu, Houxing Ren, Yunqiao Yang, Ke Wang, Zhuofan Zong, Junting Pan, Mingjie Zhan, Hongsheng Li ### WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch Introduces WebGen-Bench for evaluating agents that generate multi-file websites, using a web-navigation agent to check whether the resulting pages pass interaction tests. - https://awesomecua.com/resources/arxiv-2505-03733/ · source: https://arxiv.org/abs/2505.03733v2 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, evaluation, web-navigation - Published: 2025-05-06 · Authors: Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, Hongsheng Li ### WebGPT: Browser-assisted question-answering with human feedback Trains a question-answering model to search and navigate a text-based web browser, collect references and improve long-form answers using demonstrations and human feedback. - https://awesomecua.com/resources/arxiv-2112-09332/ · source: https://arxiv.org/abs/2112.09332v3 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, imitation-learning, human-in-the-loop, reward-models - Published: 2021-12-17 · Authors: Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman ### WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader Introduces WebGrader, a programmatic grader for website generation that plans interaction flows, runs them in a live browser, and uses observed outcomes as RL rewards. - https://awesomecua.com/resources/arxiv-2608-06474/ · source: https://arxiv.org/abs/2608.06474v1 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, evaluation, reward-models, reinforcement-learning - Published: 2026-08-06 · Authors: Boshui Chen, Huiping Liu, Shaolei Zhang ### WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation Introduces WebGraphEval, which merges multiple agents' trajectories into a weighted graph of canonical actions. Structural analyses examine recurring behavior, inefficient paths and important decision points beyond a final success score. - https://awesomecua.com/resources/arxiv-2510-19205/ · source: https://arxiv.org/abs/2510.19205v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, observability, interaction-history - Published: 2025-10-22 · Authors: Yaoyao Qian, Yuanli Wang, Jinda Zhang, Yun Zong, Meixu Chen, Hanhan Zhou, Jindan Huang, Yifan Zeng, Xinyu Hu, Chan Hee Song, Danqing Zhang ### WebGuard: Building a Generalizable Guardrail for Web Agents Introduces WebGuard, a human-annotated dataset for predicting outcomes and risk levels of state-changing website actions. It provides generalization splits and evaluates specialized guardrail models. - https://awesomecua.com/resources/arxiv-2507-14293/ · source: https://arxiv.org/abs/2507.14293v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, reward-models, evaluation - Published: 2025-07-18 · Authors: Boyuan Zheng, Zeyi Liao, Scott Salisbury, Zeyuan Liu, Michael Lin, Qinyuan Zheng, Zifan Wang, Xiang Deng, Dawn Song, Huan Sun, Yu Su ### WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks WebGym supplies realistic web tasks with rubric-based evaluation and an asynchronous rollout system for reinforcement learning. It studies how training on an agent’s own interaction traces scales with task coverage and transfers to unseen websites. - https://awesomecua.com/resources/arxiv-2601-02439/ · source: https://arxiv.org/abs/2601.02439v6 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, reinforcement-learning, data-generation, evaluation, efficiency - Published: 2026-01-05 · Authors: Hao Bai, Alexey Taymanov, Tong Zhang, Aviral Kumar, Spencer Whitehead ### WebHarbor: Docking Real Websites for Evolving GUI Agent Environments Introduces WebHarbor, a suite of 153 executable website environments and 3,518 tasks for evaluating and training GUI web agents on interactive workflows. - https://awesomecua.com/resources/arxiv-2611-03634/ · source: https://arxiv.org/abs/2611.03634v1 - Paper · reviewed 2026-11-06 · environments: web · topics: sandbox, evaluation, reinforcement-learning, web-navigation - Published: 2026-11-02 · Authors: see the paper ### WebInject: Prompt Injection Attack to Web Agents Introduces WebInject, a prompt-injection attack that optimizes rendered webpage pixels to make screenshot-based web agents take attacker-chosen actions. - https://awesomecua.com/resources/arxiv-2505-11717/ · source: https://arxiv.org/abs/2505.11717v4 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation - Published: 2025-05-16 · Authors: Xilong Wang, John Bloch, Zedian Shao, Yuepeng Hu, Shuyan Zhou, Neil Zhenqiang Gong ### Weblica: Scalable and Reproducible Training Environments for Visual Web Agents Introduces Weblica, which combines HTTP-level capture and replay with website-grounded environment synthesis. The resulting interactive web environments support reinforcement learning and tests of navigation transfer and inference-time scaling. - https://awesomecua.com/resources/arxiv-2605-06761/ · source: https://arxiv.org/abs/2605.06761v1 - Paper · reviewed 2026-10-17 · environments: web · topics: sandbox, data-generation, reinforcement-learning - Published: 2026-05-07 · Authors: Oğuzhan Fatih Kar, Roman Bachmann, Yuanzheng Gong, Anders Boesen Lindbo Larsen, Afshin Dehghan ### WebLINX: Real-World Website Navigation with Multi-Turn Dialogue Collects expert demonstrations of conversational web navigation and studies retrieval-based element selection for processing large pages. - https://awesomecua.com/resources/arxiv-2402-05930/ · source: https://arxiv.org/abs/2402.05930v2 - Paper · reviewed 2026-10-16 · environments: web · topics: data-generation, web-navigation, evaluation - Published: 2024-02-08 · Authors: Xing Han Lù, Zdeněk Kasner, Siva Reddy ### WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents Introduces WebLists, a benchmark for structured extraction from interactive websites, and BardeenAgent, which turns web-agent runs into replayable CSS-selector programs. - https://awesomecua.com/resources/arxiv-2504-12682/ · source: https://arxiv.org/abs/2504.12682v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, automation - Published: 2025-04-17 · Authors: Arth Bohra, Manvel Saroyan, Danil Melkozerov, Vahe Karufanyan, Gabriel Maher, Pascal Weinberger, Artem Harutyunyan, Giovanni Campagna ### WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents WebMall provides four simulated online shops with heterogeneous product data for comparison shopping, related-product searches, and checkout tasks. It evaluates web agents with different observation formats and memory configurations in a fixed offline environment. - https://awesomecua.com/resources/arxiv-2508-13024/ · source: https://arxiv.org/abs/2508.13024v3 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, sandbox - Published: 2025-08-18 · Authors: Ralph Peeters, Aaron Steiner, Luca Schwarz, Julian Yuya Caspary, Christian Bizer ### WebMCP-Phalanx: Enforcing and Characterizing Trust Boundaries for Browser-Integrated LLM Agents Proposes WebMCP-Phalanx, a browser-integrated runtime that binds page-exposed tools to their registering principals and separates content inspection from privileged invocation. It studies lifecycle and prompt-injection attacks, including a timing bypass that motivates validating all visible metadata before tool use. - https://awesomecua.com/resources/arxiv-2608-24017/ · source: https://arxiv.org/abs/2608.24017v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, tool-integration - Published: 2026-08-25 · Authors: Lin-Fa Lee, YI-YU Chang, Kuo-Hui Yeh ### WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation Introduces WebMMU, a multilingual benchmark for website visual question answering, element grounding, code editing, and mockup-to-code generation. - https://awesomecua.com/resources/arxiv-2508-16763/ · source: https://arxiv.org/abs/2508.16763v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, grounding, screen-understanding, ui-design - Published: 2025-08-22 · Authors: Rabiul Awal, Mahsa Massoud, Aarash Feizi, Zichao Li, Suyuchen Wang, Christopher Pal, Aishwarya Agrawal, David Vazquez, Siva Reddy, Juan A. Rodriguez, Perouz Taslakian, Spandana Gella, Sai Rajeswar ### WebNav: An Intelligent Agent for Voice-Controlled Web Navigation Introduces WebNav, a web navigation agent that turns natural-language commands into GUI actions using screenshot context, DOM labels, and separate LLMs for planning and action parameters. - https://awesomecua.com/resources/arxiv-2503-13843/ · source: https://arxiv.org/abs/2503.13843v2 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, screen-understanding, planning - Published: 2025-03-18 · Authors: Trisanth Srinivasan, Santosh Patapati ### WebOperator: Action-Aware Tree Search for Autonomous Agents in Web Environment WebOperator uses tree search, action filtering, and path checks before replay to explore web tasks and backtrack while accounting for irreversible actions. - https://awesomecua.com/resources/arxiv-2512-12692/ · source: https://arxiv.org/abs/2512.12692v1 - Paper · reviewed 2026-11-06 · environments: web · topics: planning, safety-security, web-navigation - Published: 2025-12-14 · Authors: Mahir Labib Dihan, Tanzima Hashem, Mohammed Eunus Ali, Md Rizwan Parvez ### WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents Verifies web tasks through typed interface events and generates controlled UI variations while preserving instructions and success conditions. - https://awesomecua.com/resources/arxiv-2609-35026/ · source: https://arxiv.org/abs/2609.35026v1 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, robustness - Published: 2026-09-28 · Authors: Anton Emelyanov, Maria Tikhonova, Zaven Martirosian, Sergei Averkiev, Alena Fenogenova ### WebPII: Benchmarking Visual PII Detection for Computer-Use Agents WebPII provides synthetic e-commerce screenshots with fine-grained annotations for sensitive information, including partially filled forms and transaction identifiers. It also trains WebRedact to detect visual PII as a privacy component for computer-use agents. - https://awesomecua.com/resources/arxiv-2603-17357/ · source: https://arxiv.org/abs/2603.17357v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, data-generation, safety-security, screen-understanding - Published: 2026-03-18 · Authors: Nathan Zhao ### WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration WebPilot separates global subtask planning from local Monte Carlo tree search during web execution. It continually refines the high-level plan and adapts local search to uncertain transitions and incomplete observations, evaluated on WebArena and MiniWoB++. - https://awesomecua.com/resources/arxiv-2408-15978/ · source: https://arxiv.org/abs/2408.15978v1 - Paper · reviewed 2026-10-17 · environments: web · topics: planning, web-navigation, long-horizon - Published: 2024-08-28 · Authors: Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, Volker Tresp ### WebQuest: A Benchmark for Multimodal QA on Web Page Sequences Introduces question answering over individual screens, multiple web pages, and navigation traces to test information synthesis across views. - https://awesomecua.com/resources/arxiv-2409-13711/ · source: https://arxiv.org/abs/2409.13711v2 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, screen-understanding - Published: 2024-09-06 · Authors: Maria Wang, Srinivas Sunkara, Gilles Baechler, Jason Lin, Yun Zhu, Fedir Zubach, Lei Shu, Jindong Chen ### WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation WebRetriever evaluates navigation, knowledge-assisted interaction and end-to-end information-extraction workflows under separate protocols. Its NavEval judge uses richer interaction context to diagnose cases where navigation success does not translate into useful task completion. - https://awesomecua.com/resources/arxiv-2607-06118/ · source: https://arxiv.org/abs/2607.06118v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, web-navigation, observability - Published: 2026-07-07 · Authors: Wei Dong, Tianyu Fu, Zhe Yu, Hanning Wang, Anyang Su, Zhizhou Fang, Yuyang Chen, Shuo Wang, Minghui Wu, Ping Jiang, Zhen Lei, Chenxu Zhao ### WebRider: Persona-Conditioned Intent Controllers for Live-Web Assistance Introduces WebRider, a live-web agent that maintains intent contracts while carrying out guarded browser actions, and RiderBench for auditing those contracts across websites. - https://awesomecua.com/resources/arxiv-2608-06704/ · source: https://arxiv.org/abs/2608.06704v2 - Paper · reviewed 2026-11-06 · environments: web · topics: web-navigation, planning, evaluation, tool-integration - Published: 2026-08-07 · Authors: Zhi Li, Tao Zhou, Yeqing Li, Eugene Ie, Demetri Terzopoulos ### WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning Trains web agents with an evolving task curriculum, outcome supervision, and reinforcement-learning updates intended to manage policy drift. - https://awesomecua.com/resources/arxiv-2411-02337/ · source: https://arxiv.org/abs/2411.02337v3 - Paper · reviewed 2026-10-16 · environments: web · topics: reinforcement-learning, web-navigation - Published: 2024-11-04 · Authors: Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, Yuxiao Dong ### WebRollback: Enhancing Web Agents with Explicit Rollback Mechanisms WebRollback gives a web agent an explicit action for returning to an earlier state in its navigation trajectory. It studies this recovery mechanism on two live web-navigation benchmarks under both zero-shot and fine-tuned settings. - https://awesomecua.com/resources/arxiv-2504-11788/ · source: https://arxiv.org/abs/2504.11788v3 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, planning, robustness - Published: 2025-04-16 · Authors: Zhisong Zhang, Tianqing Fang, Kaixin Ma, Wenhao Yu, Hongming Zhang, Haitao Mi, Dong Yu ### WebRouter: Query-specific Router via Variational Information Bottleneck for Cost-sensitive Web Agent Introduces WebRouter, a cost-aware router that compresses web-agent prompts to choose among LLMs, with cost and accuracy measured on WebVoyager websites. - https://awesomecua.com/resources/arxiv-2510-11221/ · source: https://arxiv.org/abs/2510.11221v1 - Paper · reviewed 2026-11-06 · environments: web · topics: efficiency, web-navigation - Published: 2025-10-13 · Authors: Tao Li, Jinlong Hu, Yang Wang, Junfeng Liu, Xuejun Liu ### WebSentinel: Detecting and Localizing Prompt Injection Attacks for Web Agents WebSentinel identifies suspicious webpage segments and checks each against its surrounding page context to detect and locate prompt injections. Its evaluation uses both contaminated and clean webpage datasets. - https://awesomecua.com/resources/arxiv-2602-03792/ · source: https://arxiv.org/abs/2602.03792v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2026-02-03 · Authors: Xilong Wang, Yinuo Liu, Zhun Wang, Dawn Song, Neil Gong ### WEBSERV: A Full-Stack and RL-Ready Web Environment for Training Web Agents at Scale WebServ combines copy-on-write container environments with a compact DOM-derived browser interface for training web agents. Network-aware waiting supports action execution in single-page applications, with evaluations and reinforcement-learning experiments on WebArena-Lite. - https://awesomecua.com/resources/arxiv-2510-16252/ · source: https://arxiv.org/abs/2510.16252v2 - Paper · reviewed 2026-10-17 · environments: web · topics: sandbox, reinforcement-learning, efficiency, web-navigation - Published: 2025-10-17 · Authors: Yuxuan Lu, Ziyi Wang, Jing Huang, Hui Liu, Jiri Gesi, Yan Han, Shihan Fu, Tianqi Zheng, Xianfeng Tang, Chen Luo, Yisi Sang, Jin Lai, Dakuo Wang ### WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents Provides a simulated shopping environment where language agents search, configure, and select products from natural-language requirements. - https://awesomecua.com/resources/arxiv-2207-01206/ · source: https://arxiv.org/abs/2207.01206v4 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation - Published: 2022-07-04 · Authors: Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan ### WebSight: A Vision-First Architecture for Robust Web Agents WebSight uses a vision-language model trained on web-interface examples within a modular agent with planning, action, verification, and episodic memory components. It evaluates element clicking on Showdown Clicks and full browsing tasks on WebVoyager using visual observations. - https://awesomecua.com/resources/arxiv-2508-16987/ · source: https://arxiv.org/abs/2508.16987v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, grounding, planning, memory - Published: 2025-08-23 · Authors: Tanvir Bhathal, Asanshay Gupta ### WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks WebSP-Eval evaluates agents carrying out website security and privacy tasks such as preference changes and session management. A browser extension controls account and initial state across runs, enabling analysis of failures on stateful UI controls. - https://awesomecua.com/resources/arxiv-2604-06367/ · source: https://arxiv.org/abs/2604.06367v2 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, web-navigation, state-tracking - Published: 2026-04-07 · Authors: Guruprasad Viswanathan Ramesh, Asmit Nayak, Basieem Siddique, Kassem Fawaz ### WebSRC: A Dataset for Web-Based Structural Reading Comprehension Introduces question answering over webpage text and structure, with aligned HTML, screenshots, and metadata for structural reading comprehension. - https://awesomecua.com/resources/arxiv-2101-09465/ · source: https://arxiv.org/abs/2101.09465v2 - Paper · reviewed 2026-10-16 · environments: web · topics: screen-understanding, evaluation - Published: 2021-01-23 · Authors: Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, Kai Yu ### WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering Introduces a pipeline that filters individual steps from noisy computer-agent rollouts and augments them with reasoning for training. WebSTAR and WebSCORE support GUI-policy supervision and a distilled process reward model, StepRM. - https://awesomecua.com/resources/arxiv-2512-10962/ · source: https://arxiv.org/abs/2512.10962v3 - Paper · reviewed 2026-10-17 · environments: web · topics: data-generation, reward-models, imitation-learning - Published: 2025-11-22 · Authors: Yifei He, Pranit Chawla, Yaser Souri, Subhojit Som, Xia Song ### WebSuite: Systematically Evaluating Why Web Agents Fail WebSuite organizes browser operations into an action taxonomy and tests both isolated controls and complete workflows. Its diagnostic tasks attribute failures to specific actions, enabling comparison of text-based and multimodal web agents. - https://awesomecua.com/resources/arxiv-2406-01623/ · source: https://arxiv.org/abs/2406.01623v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, observability, web-navigation - Published: 2024-06-01 · Authors: Eric Li, Jim Waldo ### WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis Introduces WebSynthesis, which uses a learned web world model and tree-based planning to synthesize interaction trajectories for training web agents. - https://awesomecua.com/resources/arxiv-2507-04370/ · source: https://arxiv.org/abs/2507.04370v1 - Paper · reviewed 2026-11-06 · environments: web · topics: world-models, data-generation, planning, web-navigation - Published: 2025-07-06 · Authors: Yifei Gao, Junhong Ye, Jiaqi Wang, Jitao Sang ### WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing Introduces WebTestBench for end-to-end website testing, separating checklist generation from defect detection. Its WebTester baseline evaluates test coverage, detection and sustained interaction across web application categories. - https://awesomecua.com/resources/arxiv-2603-25226/ · source: https://arxiv.org/abs/2603.25226v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, web-navigation, automation - Published: 2026-03-26 · Authors: Fanheng Kong, Jingyuan Zhang, Yang Yue, Chenxi Sun, Yang Tian, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Jun Du, Wenchong Zeng, Han Li, Kun Gai ### WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements WebTestPilot turns natural-language requirements into web test steps with inferred preconditions and postconditions over symbolized GUI elements. A benchmark of bug-injected web applications tests whether these context-aware oracles distinguish application defects from agent mistakes. - https://awesomecua.com/resources/arxiv-2602-11724/ · source: https://arxiv.org/abs/2602.11724v3 - Paper · reviewed 2026-10-17 · environments: web · topics: automation, evaluation, planning - Published: 2026-02-12 · Authors: Xiwen Teoh, Yun Lin, Duc-Minh Nguyen, Ruofei Ren, Wenjie Zhang, Jin Song Dong ### WebThinker: Empowering Large Reasoning Models with Deep Research Capability Combines iterative web search and link-following with interleaved reasoning and report drafting for deep research. Its reported web explorer retrieves page content through a crawler rather than controlling a visual browser interface. - https://awesomecua.com/resources/arxiv-2504-21776/ · source: https://arxiv.org/abs/2504.21776v2 - Paper · reviewed 2026-10-18 · environments: web · topics: web-navigation, planning, tool-integration - Published: 2025-04-30 · Authors: Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, Zhicheng Dou ### WebTrap Park: An Automated Platform for Systematic Security Evaluation of Web Agents Introduces WebTrap Park, a platform that turns web-agent security risks into executable tasks and assesses concrete interactions with live pages. It compares agent frameworks without requiring modifications to their implementations. - https://awesomecua.com/resources/arxiv-2601-08406/ · source: https://arxiv.org/abs/2601.08406v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2026-01-13 · Authors: Xinyi Wu, Jiagui Chen, Geng Hong, Jiayi Dong, Xudong Pan, Jiarun Dai, Min Yang ### WebTrap: Stealthy Mid-Task Hijacking of Browser Agents During Navigation WebTrap injects mid-task instructions into browser-agent navigation to steer an agent toward an attacker goal before it resumes the user's task. - https://awesomecua.com/resources/arxiv-2605-08310/ · source: https://arxiv.org/abs/2605.08310v1 - Paper · reviewed 2026-11-06 · environments: web · topics: safety-security, web-navigation, long-horizon - Published: 2026-05-08 · Authors: Zhichao Liu, Wenbo Pan, Haining Yu, Ge Gao, Tianqing Zhu, Xiaohua Jia ### WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics Collects rendered webpages and structural metadata for visual UI modeling, studying how web semantics transfer to interface-understanding tasks. - https://awesomecua.com/resources/arxiv-2301-13280/ · source: https://arxiv.org/abs/2301.13280v1 - Paper · reviewed 2026-10-16 · environments: web · topics: screen-understanding, data-generation - Published: 2023-01-30 · Authors: Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, Jeffrey P. Bigham ### WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness Introduces an execution-based benchmark for generated web interfaces, with a headless-browser UI-agent harness that checks DOM interactions and functional assertions. - https://awesomecua.com/resources/arxiv-2610-02617/ · source: https://arxiv.org/abs/2610.02617v1 - Paper · reviewed 2026-10-18 · environments: web · topics: evaluation, web-navigation, ui-design - Published: 2026-10-02 · Authors: Yun-Yun Tsai, Yuning Mao, Shiqi Wang, Junfeng Yang, Sinong Wang ### WebUncertainty: Dual-Level Uncertainty Driven Planning and Reasoning For Autonomous Web Agent WebUncertainty adapts planning mode to task uncertainty and guides action-level tree search with confidence-derived uncertainty estimates. Its evaluations examine autonomous execution of dynamic, long-horizon web tasks on WebArena and WebVoyager. - https://awesomecua.com/resources/arxiv-2604-17821/ · source: https://arxiv.org/abs/2604.17821v2 - Paper · reviewed 2026-10-18 · environments: web · topics: planning, web-navigation, long-horizon, robustness - Published: 2026-04-20 · Authors: Lingfeng Zhang, Yongan Sun, Jinpeng Hu, Hui Ma, Yang Ying, Kuien Liu, Zenglin Shi, Meng Wang ### WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation WebVIA combines a web UI exploration agent, interactive UI-to-Code generation, and validation of generated code's interactivity. - https://awesomecua.com/resources/arxiv-2511-06251/ · source: https://arxiv.org/abs/2511.06251v1 - Paper · reviewed 2026-11-06 · environments: web · topics: ui-design, evaluation, web-navigation - Published: 2025-11-09 · Authors: Mingde Xu, Zhen Yang, Wenyi Hong, Lihang Pan, Xinyue Fan, Yan Wang, Xiaotao Gu, Bin Xu, Jie Tang ### WebVLN: Vision-and-Language Navigation on Websites WebVLN trains website navigation from question-based instructions using both rendered visual information and underlying HTML. It introduces WebVLN-v1 and a website-aware navigation network to study this digital variant of vision-and-language navigation. - https://awesomecua.com/resources/arxiv-2312-15820/ · source: https://arxiv.org/abs/2312.15820v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, data-generation, evaluation - Published: 2023-12-25 · Authors: Qi Chen, Dileepa Pitawela, Chongyang Zhao, Gengze Zhou, Hsiang-Ting Chen, Qi Wu ### WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models Combines multimodal web interaction with a real-website task collection and an automated evaluation protocol. - https://awesomecua.com/resources/arxiv-2401-13919/ · source: https://arxiv.org/abs/2401.13919v4 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, evaluation - Published: 2024-01-25 · Authors: Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, Dong Yu ### WebWalker: Benchmarking LLMs in Web Traversal Introduces WebWalkerQA and an exploration-and-critique framework for gathering information by traversing multiple levels of a website. - https://awesomecua.com/resources/arxiv-2501-07572/ · source: https://arxiv.org/abs/2501.07572v3 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, web-navigation - Published: 2025-01-13 · Authors: Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, Fei Huang ### WebWISE: Web Interface Control and Sequential Exploration with Large Language Models WebWISE filters DOM elements into observations and generates short programs for click, scroll, and text-entry steps. It evaluates in-context learning from either one supplied example or an example collected during a successful zero-shot MiniWoB++ trial. - https://awesomecua.com/resources/arxiv-2310-16042/ · source: https://arxiv.org/abs/2310.16042v2 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, tool-integration, planning - Published: 2023-10-24 · Authors: Heyi Tao, Sethuraman T, Michal Shlapentokh-Rothman, Derek Hoiem ### WebWorld: A Large-Scale World Model for Web Agent Training Introduces WebWorld, an open-web simulator trained on interaction trajectories for generating agent training data and testing candidate actions. WebWorld-Bench evaluates simulated transitions, alongside downstream web-agent training experiments. - https://awesomecua.com/resources/arxiv-2602-14721/ · source: https://arxiv.org/abs/2602.14721v1 - Paper · reviewed 2026-10-17 · environments: web · topics: world-models, data-generation, evaluation - Published: 2026-02-16 · Authors: Zikai Xiao, Jianhong Tu, Chuhang Zou, Yuxin Zuo, Zhi Li, Peng Wang, Bowen Yu, Fei Huang, Junyang Lin, Zuozhu Liu ### WebWorld: The Browser as a World Model for Self-Improving Web Code Uses typed browser interaction contracts to accept web-code improvements only when target progress and previously verified capabilities are preserved. Accepted transitions supply self-improvement data. - https://awesomecua.com/resources/arxiv-2608-30530/ · source: https://arxiv.org/abs/2608.30530v1 - Paper · reviewed 2026-10-20 · environments: web · topics: evaluation, workflow, robustness, data-generation - Published: 2026-08-31 · Authors: see the paper ### WebXSkill: Skill Learning for Autonomous Web Agents WebXSkill pairs parameterized action programs with step-level language guidance, mining reusable skills from synthetic browser-agent traces. A URL-based graph supports retrieval, and deployment alternates between direct program execution and guided native planning. - https://awesomecua.com/resources/arxiv-2604-13318/ · source: https://arxiv.org/abs/2604.13318v2 - Paper · reviewed 2026-10-18 · environments: web · topics: memory, planning, web-navigation, automation - Published: 2026-04-14 · Authors: Zhaoyang Wang, Qianhui Wu, Xuchao Zhang, Chaoyun Zhang, Wenlin Yao, Fazle Elahi Faisal, Baolin Peng, Si Qin, Suman Nath, Qingwei Lin, Chetan Bansal, Dongmei Zhang, Saravan Rajmohan, Jianfeng Gao, Huaxiu Yao ### WEPO: Web Element Preference Optimization for LLM-based Web Navigation Uses web-element preference optimization with sampled negative HTML elements to align high-level user intent with navigation actions. - https://awesomecua.com/resources/arxiv-2412-10742/ · source: https://arxiv.org/abs/2412.10742v1 - Paper · reviewed 2026-10-16 · environments: web · topics: grounding, web-navigation - Published: 2024-12-14 · Authors: Jiarun Liu, Jia Hao, Chunhong Zhang, Zheng Hu ### What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents SERL trains multi-turn agents by using task rewards and selective step feedback to weight updates, with WebShop among its evaluation settings. - https://awesomecua.com/resources/arxiv-2605-19447/ · source: https://arxiv.org/abs/2605.19447v1 - Paper · reviewed 2026-11-06 · environments: web · topics: reinforcement-learning, reward-models, long-horizon - Published: 2026-05-19 · Authors: Xiaozhe Li, Tianyi Lyu, Yang Li, Yichuan Ma, Peiji Li, Linyang Li, Qipeng Guo, Dahua Lin, Kai Chen ### What Does "Now" Mean to Your Computer-Use Agent? Stale Screenshots, Page Loads, and Which Check to Run Before a Click Studies clicks made from stale screenshots on web pages and provides a benchmark generator and pre-click checks for pages that change during loading. - https://awesomecua.com/resources/arxiv-2610-39282/ · source: https://arxiv.org/abs/2610.39282v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, state-tracking, robustness, efficiency - Published: 2026-10-29 · Authors: Vasundra Srinivasan ### What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs Re-Prefill adds an attention-guided second prefill pass that revisits candidate screen regions before coordinate generation. The study evaluates this training-free intervention across GUI-grounding models and benchmarks. - https://awesomecua.com/resources/arxiv-2605-12549/ · source: https://arxiv.org/abs/2605.12549v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, screen-understanding, evaluation - Published: 2026-05-10 · Authors: Jiaping Lin, Fei Shen, Junzhe Li, Ping Nie, Fei Yu, Ming Li, Haizhou Li ### What Is This Memory Useful For? Measuring Task-Conditioned Utility in Long-Term Agent Memory Introduces task-conditioned measurements and a dataset for assessing whether stored agent memories help later tasks, using factual-recall tasks built from web-agent memories. - https://awesomecua.com/resources/arxiv-2611-03930/ · source: https://arxiv.org/abs/2611.03930v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, evaluation, efficiency - Published: 2026-11-02 · Authors: Chih-Hsuan Yang, Anjir Ahmed Chowdhury, Feng Yan, Rajeev Thakur ### What It Is, Not What It Says: Securing Agentic Browsing Interactions with Web Semantics Intact records webpage element provenance and capabilities, isolates planning per element, and uses a browser harness to constrain actions and request consent for third-party interactions. - https://awesomecua.com/resources/arxiv-2610-22697/ · source: https://arxiv.org/abs/2610.22697v1 - Paper · reviewed 2026-10-20 · environments: web · topics: safety-security, web-navigation, planning, human-in-the-loop - Published: 2026-10-15 · Authors: Umar Iqbal ### What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States Proposes ATMem, an updated execution-state memory for long mobile GUI tasks, and STR-GRPO to train selective memory use; also introduces a mobile task benchmark. - https://awesomecua.com/resources/arxiv-2606-31612/ · source: https://arxiv.org/abs/2606.31612v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: memory, state-tracking, reinforcement-learning, mobile-navigation - Published: 2026-06-30 · Authors: Chen Liu, Ling Chen, Hanzhang Zhou, Xu Zhang, Quyu Kong, Panrong Tong, Wenhao Wang, Xin Yu, Steven Hoi, Yue Wang ### What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning Introduces UILoop, which inserts explicit UI-element localization, function and usage reasoning between screen observation and action. UI Comprehension-Bench evaluates these element-level capabilities with three metrics. - https://awesomecua.com/resources/arxiv-2604-06995/ · source: https://arxiv.org/abs/2604.06995v3 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: screen-understanding, grounding, evaluation - Published: 2026-04-08 · Authors: Songze Li, Xiaoke Guo, Tianqi Liu, Biao Yi, Zhaoyan Gong, Zhiqiang Liu, Huajun Chen, Wen Zhang ### When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition Studies silent failures in web content fetched by static and rendered-browser clients, and evaluates checks for whether HTTP 200 pages contain usable content. - https://awesomecua.com/resources/arxiv-2611-00884/ · source: https://arxiv.org/abs/2611.00884v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, web-navigation, robustness - Published: 2026-10-31 · Authors: Shuyang Zhang ### When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents MisActBench provides action-level labels for task alignment, while DeAction checks proposed computer-use actions and requests revisions through structured feedback. The paper separately evaluates offline detection and interactive operation in OSWorld and RedTeamCUA. - https://awesomecua.com/resources/arxiv-2602-08995/ · source: https://arxiv.org/abs/2602.08995v2 - Paper · reviewed 2026-10-17 · environments: desktop, web · topics: safety-security, evaluation, robustness - Published: 2026-02-09 · Authors: Yuting Ning, Jaylen Jones, Zhehao Zhang, Chentao Ye, Weitong Ruan, Junyi Li, Rahul Gupta, Huan Sun ### When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents Investigates mismatches between what humans see on a mobile display and what agents receive through screenshots or accessibility metadata. An automated evaluation framework embeds instruction-agnostic perturbations into functional APK clones and compares static and dynamic agent deviations with human noticeability. - https://awesomecua.com/resources/arxiv-2609-16732/ · source: https://arxiv.org/abs/2609.16732v1 - Paper · reviewed 2026-10-18 · environments: android · topics: safety-security, screen-understanding, evaluation, robustness - Published: 2026-09-15 · Authors: Heng Li, Fulin Zhao, Zhe Geng, Zhiyuan Yao, Wei Yuan, Xiapu Luo ### When and What to Teach: Budget-Aware Online Adaptation for Web Agents Proposes an online teaching framework that decides which web-agent episodes warrant a stronger teacher and which trajectory turns merit training. A solvability gate and score-guided trimming aim to preserve first-attempt performance while reducing teaching and training costs. - https://awesomecua.com/resources/arxiv-2609-05513/ · source: https://arxiv.org/abs/2609.05513v1 - Paper · reviewed 2026-10-17 · environments: web · topics: efficiency, data-generation, web-navigation - Published: 2026-08-31 · Authors: Jianwei Zhang, Sihan Cao, Pengcheng Zheng, Ya Wen, Pei Ke, Kuien Liu, Shen Gao, Wei Dong, Yang Yang, Chaoning Zhang ### When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents Proposes diagnostic controls for hidden-state probes of indirect prompt injection in multimodal computer-use agents, using a teacher-forced Mind2Web case study. - https://awesomecua.com/resources/arxiv-2606-22864/ · source: https://arxiv.org/abs/2606.22864v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation - Published: 2026-06-22 · Authors: Yanhang Li, Zhichao Fan, Zexin Zhuang ### When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents AutoElicit iteratively revises benign instructions using execution feedback to surface harmful unintended actions by computer-use agents in OS workflows. - https://awesomecua.com/resources/arxiv-2602-08235/ · source: https://arxiv.org/abs/2602.08235v2 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, data-generation, workflow - Published: 2026-02-09 · Authors: Jaylen Jones, Zhehao Zhang, Yuting Ning, Eric Fosler-Lussier, Pierre-Luc St-Charles, Yoshua Bengio, Dawn Song, Yu Su, Huan Sun ### When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agent Studies deceptive social-engineering web patterns targeting browser automation agents and proposes defenses against bait-based manipulation. - https://awesomecua.com/resources/arxiv-2601-07263/ · source: https://arxiv.org/abs/2601.07263v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, web-navigation, robustness - Published: 2026-01-12 · Authors: Xinyi Wu, Geng Hong, Yueyue Chen, MingXuan Liu, Feier Jin, Xudong Pan, Jiarun Dai, Baojun Liu ### When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks Introduces MisVisAgentBench, a paired executable benchmark that changes chart evidence while holding web tasks fixed, to trace how misleading visualizations affect agent decisions and downstream actions. - https://awesomecua.com/resources/arxiv-2610-12823/ · source: https://arxiv.org/abs/2610.12823v1 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, robustness, screen-understanding - Published: 2026-10-09 · Authors: Yisheng Li, Dongming Han, Hongkun Pan, Bingcheng Mao, Zijian Wang, Yunlong Mao, Wei Tong, Sheng Zhong ### When Faster Decisions Do Not Make Better Agents: An Empirical Study of Jev in Agent Workflows Evaluates Jev as a bounded action selector inside three AndroidWorld agents and two text-based ALFWorld agents, separating decision savings from task success, recovery and resources per delivered outcome. - https://awesomecua.com/resources/arxiv-2610-13092/ · source: https://arxiv.org/abs/2610.13092v1 - Paper · reviewed 2026-10-20 · environments: android · topics: efficiency, action-prediction, evaluation, mobile-navigation, self-reflection - Published: 2026-10-09 · Authors: Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang ### When to Compile a Computer-Use Agent? Measuring Payback and Making Compilation Decisions for Token Efficiency Introduces PACE, a protocol and online decision rule for deciding when repeated GUI procedures are worth compiling into programs. It accounts for failed compilation attempts, measured per-use savings, uncertain reuse, and execution costs under an explicit cumulative budget. - https://awesomecua.com/resources/arxiv-2610-02932/ · source: https://arxiv.org/abs/2610.02932v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: automation, efficiency, workflow - Published: 2026-10-02 · Authors: Yulong Ming, Jie Xu, Zihan Wu, Xiaohua Jia ### When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents Introduces ReInBench, annotated computer-use and web-agent error traces, and ReCoF, a method for predicting whether an agent will recover without intervention. - https://awesomecua.com/resources/arxiv-2610-36471/ · source: https://arxiv.org/abs/2610.36471v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, observability, robustness - Published: 2026-10-27 · Authors: Chiming Duan, Tong Jia, Yingxi Chen, Minghua He, Lingzhe Zhang, Ying Li ### When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigation Introduces InterruptBench, a WebArena-Lite-derived benchmark that tests how long-horizon web-navigation agents handle mid-task user additions, revisions, and retractions. - https://awesomecua.com/resources/arxiv-2604-00892/ · source: https://arxiv.org/abs/2604.00892v1 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, human-in-the-loop, long-horizon, web-navigation - Published: 2026-04-01 · Authors: Henry Peng Zou, Chunyu Miao, Wei-Chieh Huang, Yankai Chen, Yue Zhou, Hanrong Zhang, Yaozu Wu, Liancheng Fang, Zhengyao Gu, Zhen Zhang, Kening Zheng, Fangxin Wang, Yi Nian, Shanghao Li, Wenzhe Fan, Langzhou He, Weizhi Zhang, Xue Liu, Philip S. Yu ### When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration Introduces Parallel WebBench for parallel web exploration and uses verified URL trajectories to diagnose search loops, incomplete answers, and failures after evidence retrieval. - https://awesomecua.com/resources/arxiv-2606-20724/ · source: https://arxiv.org/abs/2606.20724v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, observability, web-navigation, long-horizon - Published: 2026-06-16 · Authors: Aagam Sogani, Botao Rui, Swetha Vaidyanathan, Rishi Agarwal, Minghao Yan, Shivaram Venkataraman ### Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation Studies visual token pruning for GUI navigation, separating reusable screenshot encoder embeddings from later language-model computation. The analysis examines foreground-background token allocation and spatial coverage when pruning historical screenshots. - https://awesomecua.com/resources/arxiv-2603-26041/ · source: https://arxiv.org/abs/2603.26041v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: efficiency, interaction-history, screen-understanding - Published: 2026-03-27 · Authors: Daiqiang Li, Zihao Pan, Zeyu Zhang, Xuyang Liu, Shijia Xu, Ronghao Chen, Huacan Wang, Honggang Chen, Linfeng Zhang, Zhangquan Chen, Haiyun Jiang ### Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking Introduces WebStep, a web-agent benchmark that records semantic states behind GUI interactions to analyze exploration, execution accuracy, and decisive errors across task steps. - https://awesomecua.com/resources/arxiv-2606-15673/ · source: https://arxiv.org/abs/2606.15673v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, state-tracking, observability, web-navigation - Published: 2026-04-08 · Authors: Jiwan Chung, JiHyuk Byun, Vibhav Vineet, Seon Joo Kim ### Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints Proposes training with subset-based attribution constraints so models attend to intended evidence, evaluated on image classification and GUI-agent click decisions. - https://awesomecua.com/resources/arxiv-2602-07008/ · source: https://arxiv.org/abs/2602.07008v6 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: action-prediction, screen-understanding - Published: 2026-01-30 · Authors: Ruoyu Chen, Shangquan Sun, Xiaoqing Guo, Kangwei Liu, Sanyi Zhang, Zhangcheng Wang, Shiming Liu, Qunli Zhang, Hua Zhang, Wei Wang, Xiaochun Cao ### Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents Introduces a two-stage GUI-agent RL framework that selects uncertain action checkpoints and expands rollout length as training progresses. - https://awesomecua.com/resources/arxiv-2610-39251/ · source: https://arxiv.org/abs/2610.39251v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: reinforcement-learning, efficiency, action-prediction - Published: 2026-10-29 · Authors: Yuxiang Chai, Quyu Kong, Panrong Tong, Steven Hoi, Yue Wang ### Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents StakeBench evaluates prompt-injection harm to different stakeholders in shopping web-agent workflows. Its executable cases separate attack objectives and assess both outcomes and execution processes to reveal how the same attack pattern can affect different parties. - https://awesomecua.com/resources/arxiv-2606-13385/ · source: https://arxiv.org/abs/2606.13385v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, safety-security, web-navigation - Published: 2026-06-11 · Authors: Zihao Wang, Yiming Li, Yutong Wu, Kangjie Chen, Zheyu Liu, Fok Kar Wai, Pin-Yu Chen, Vrizlynn L. L. Thing, Bo Li, Dacheng Tao, Tianwei Zhang ### Who Verifies the Agent's Success Claim? Self-Report Leakage in GUI-Agent Completion Judging Studies how GUI agents' success claims affect completion judgments and builds a claim-blind auditor that checks reports against recorded traces. - https://awesomecua.com/resources/arxiv-2610-29609/ · source: https://arxiv.org/abs/2610.29609v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, observability, safety-security - Published: 2026-10-21 · Authors: Yunfeng Zhu, Shengcheng Yu, Zihan Li, Zhenyu Chen ### Whose Agent Are You? Multi-Layer Fingerprinting and Attribution of Autonomous Web Agents Builds a logging framework that combines network/protocol fingerprints with browser interaction behavior to attribute web-agent traffic. A decision-tree classifier is evaluated across six agent implementations, human browsing and conventional crawlers. - https://awesomecua.com/resources/arxiv-2606-20910/ · source: https://arxiv.org/abs/2606.20910v1 - Paper · reviewed 2026-10-18 · environments: web · topics: safety-security, observability, evaluation - Published: 2026-06-18 · Authors: Dayeon Kang, Hyejun Jeong, Jade Sheffey, Pubali Datta, Amir Houmansadr ### Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees Introduces Adaptive Anticipatory Policy Trees, which precompute guarded actions and deadlines during idle periods so transient GUI events can be handled without new text generation. Paired experiments examine when this prepared execution helps and when reactive control remains preferable. - https://awesomecua.com/resources/arxiv-2607-28399/ · source: https://arxiv.org/abs/2607.28399v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: efficiency, planning, robustness - Published: 2026-07-30 · Authors: Zihan Dong, Rui Qian, Qishi Zhan, Dongshen Peng, Kaixin Li, Yu Li ### Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis Analyzes web-agent vulnerabilities at the component level, examining goal placement in prompts, multistep action generation, and observations as sources of increased susceptibility. - https://awesomecua.com/resources/arxiv-2502-20383/ · source: https://arxiv.org/abs/2502.20383v3 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, evaluation, web-navigation - Published: 2025-02-27 · Authors: Jeffrey Yang Fan Chiang, Seungjae Lee, Jia-Bin Huang, Furong Huang, Yizheng Chen ### Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective Presents a hierarchical planning framework for evaluating web agents at planning, execution and replanning stages, and reports low-level execution as the main failure source. - https://awesomecua.com/resources/arxiv-2603-14248/ · source: https://arxiv.org/abs/2603.14248v2 - Paper · reviewed 2026-11-06 · environments: web · topics: evaluation, planning, observability, web-navigation - Published: 2026-03-15 · Authors: Mohamed Aghzal, Gregory J. Stein, Ziyu Yao ### Why Generate a Click? Plug-in Decision Heads for Web Agents Adds decision heads to a frozen vision-language generator for selecting web actions and targets, with a confidence gate that falls back to text generation. - https://awesomecua.com/resources/arxiv-2610-20107/ · source: https://arxiv.org/abs/2610.20107v1 - Paper · reviewed 2026-10-17 · environments: web · topics: efficiency, grounding, web-navigation - Published: 2026-10-14 · Authors: Chengguang Gan, Yunhao Liang, Hanjun Wei, Qinghao Zhang, Zhixi Cai, Shiwen Ni ### Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements Introduces crowdsourced descriptions of mobile UI elements and studies multimodal models that generate useful widget captions. - https://awesomecua.com/resources/arxiv-2010-04295/ · source: https://arxiv.org/abs/2010.04295v1 - Paper · reviewed 2026-10-16 · environments: unspecified · topics: screen-understanding, grounding - Published: 2020-10-08 · Authors: Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan ### WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents WILBUR learns to rank prior task demonstrations for a web agent’s prompt and synthesizes instructions from past runs. A generated task curriculum and a backtracking mechanism support adaptation and recovery, evaluated on WebVoyager. - https://awesomecua.com/resources/arxiv-2404-05902/ · source: https://arxiv.org/abs/2404.05902v1 - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, memory, robustness, data-generation - Published: 2024-04-08 · Authors: Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, Giovanni Campagna ### WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation WildClawBench evaluates CLI agents on long, multimodal tasks in Docker containers with real tools, using task checks, side-effect audits, and a semantic judge. - https://awesomecua.com/resources/arxiv-2605-10912/ · source: https://arxiv.org/abs/2605.10912v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, long-horizon, sandbox, tool-integration - Published: 2026-05-11 · Authors: Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, Yubo Ma, Haodong Duan, Jing Shao, Jiaqi Wang, Dahua Lin, Kai Chen, Yuhang Zang ### WinClick: GUI Grounding with Multimodal Large Language Models Introduces a screenshot-based Windows GUI agent and the WinSpot benchmark for evaluating localization of actionable interface regions. - https://awesomecua.com/resources/arxiv-2503-04730/ · source: https://arxiv.org/abs/2503.04730v1 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: grounding, evaluation - Published: 2025-01-27 · Authors: Zheng Hui, Yinheng Li, Dan zhao, Tianyi Chen, Colby Banbury, Kazuhito Koishida ### WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments WinDeskGround synthesizes multi-window desktop layouts with controlled occlusion, density and semantic similarity to test GUI grounding robustness. Its instruction-target benchmark measures how localization changes when interfaces depart from simple single-layer scenes. - https://awesomecua.com/resources/arxiv-2605-16402/ · source: https://arxiv.org/abs/2605.16402v1 - Paper · reviewed 2026-10-18 · environments: desktop · topics: grounding, evaluation, robustness, data-generation - Published: 2026-05-13 · Authors: Haoren Zhao, Tianyi Chen, Zhen Wang ### WinDOM: Self-Family Distillation for Small-Model GUI Grounding WinDOM obtains bounding-box supervision from a Windows 11 web reimplementation driven through Playwright, and studies rejection-sampling distillation for small GUI grounders. It treats the depth of the supervised cold start as a GRPO choice, comparing same-size EMA and larger same-family teachers. - https://awesomecua.com/resources/arxiv-2606-25964/ · source: https://arxiv.org/abs/2606.25964v1 - Paper · reviewed 2026-10-17 · environments: web · topics: grounding, data-generation, imitation-learning, reinforcement-learning - Published: 2026-06-24 · Authors: Chengheng Li-Chen, Zhiqian Zhou, Hao Chen, Nicolas Chauvin ### Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale Adapts real-desktop evaluation to Windows applications and supports parallel execution of benchmark tasks. - https://awesomecua.com/resources/arxiv-2409-08264/ · source: https://arxiv.org/abs/2409.08264v2 - Paper · reviewed 2026-10-16 · environments: desktop, windows · topics: evaluation, cross-app - Published: 2024-09-12 · Authors: Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, Zack Hui ### WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments Introduces WindowsWorld for professional workflows spanning multiple desktop applications, with generated tasks refined through human review. Intermediate subgoals support process-oriented evaluation of cross-application reasoning and execution efficiency. - https://awesomecua.com/resources/arxiv-2604-27776/ · source: https://arxiv.org/abs/2604.27776v1 - Paper · reviewed 2026-10-17 · environments: desktop · topics: evaluation, cross-app, workflow - Published: 2026-04-30 · Authors: Jinchao Li, Yunxin Li, Chenrui Zhao, Zhenran Xu, Baotian Hu, Min Zhang ### WiP: Characterizing and Defending Against Mobile-Agent-Driven MFA Automation Studies how mobile automation can combine authentication factors within one agent environment and evaluates motion sensing as a user-presence signal. - https://awesomecua.com/resources/arxiv-2609-02154/ · source: https://arxiv.org/abs/2609.02154v1 - Paper · reviewed 2026-10-16 · environments: android · topics: safety-security, evaluation - Published: 2026-09-02 · Authors: Yimeng Liu, Hua Huang ### WIPI: A New Web Threat for LLM-Driven Web Agents Studies WIPI, a black-box indirect-instruction threat in which webpage content redirects LLM-driven web agents. Evaluation spans plugin-based agents, Web GPTs, and open-source web-agent implementations. - https://awesomecua.com/resources/arxiv-2402-16965/ · source: https://arxiv.org/abs/2402.16965v1 - Paper · reviewed 2026-10-17 · environments: web · topics: safety-security, web-navigation, evaluation - Published: 2024-02-26 · Authors: Fangzhou Wu, Shutong Wu, Yulong Cao, Chaowei Xiao ### WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning Uses learned world models for mobile-agent training rollouts and for reasoning about the consequences of candidate GUI actions. - https://awesomecua.com/resources/arxiv-2608-27508/ · source: https://arxiv.org/abs/2608.27508v1 - Paper · reviewed 2026-10-16 · environments: android · topics: world-models, reinforcement-learning - Published: 2026-08-27 · Authors: Yu Han, Tianwen Qian ### WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks Introduces WONDERBREAD, a benchmark pairing human WebArena workflow demonstrations with screen recordings, action logs, and written procedures. Six evaluation tasks cover documentation, knowledge transfer, and process improvement beyond end-to-end execution. - https://awesomecua.com/resources/arxiv-2406-13264/ · source: https://arxiv.org/abs/2406.13264v2 - Paper · reviewed 2026-10-17 · environments: web · topics: evaluation, workflow, video - Published: 2024-06-19 · Authors: Michael Wornow, Avanika Narayan, Ben Viggiano, Ishan S. Khare, Tathagat Verma, Tibor Thompson, Miguel Angel Fuentes Hernandez, Sudharsan Sundar, Chloe Trujillo, Krrish Chawla, Rongfei Lu, Justin Shen, Divya Nagaraj, Joshua Martinez, Vardhan Agrawal, Althea Hudson, Nigam H. Shah, Christopher Re ### Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction Aligns human screen recordings, input events, and accessibility information to derive reusable workflow skills from desktop demonstrations. - https://awesomecua.com/resources/arxiv-2610-12984/ · source: https://arxiv.org/abs/2610.12984v1 - Paper · reviewed 2026-10-16 · environments: desktop · topics: recording, workflow, data-generation - Published: 2026-10-09 · Authors: Yuqi Huang, Hui Li, Yifei Cao, Zikun Cai, Tianci Xie, Yangfan Gao, Pinxue Guo, Chaoyun Zhang, Bingxuan Li, Rui Zheng, Hang Yan, Ning Liao ### WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks Defines compositional enterprise workflows to evaluate planning and reasoning, with generation of ground-truth interaction traces. - https://awesomecua.com/resources/arxiv-2407-05291/ · source: https://arxiv.org/abs/2407.05291v2 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, planning, workflow - Published: 2024-07-07 · Authors: Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, Alexandre Drouin ### WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? Introduces enterprise web tasks on ServiceNow and BrowserGym interfaces for evaluating browser-based agents. - https://awesomecua.com/resources/arxiv-2403-07718/ · source: https://arxiv.org/abs/2403.07718v5 - Paper · reviewed 2026-10-16 · environments: web · topics: evaluation, workflow - Published: 2024-03-12 · Authors: Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, Alexandre Lacoste ### Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields Workflow-GYM evaluates long GUI workflows in professional software rather than isolated interface actions. Its analysis examines failures such as omitted stages, accumulated errors and goal drift during specialized end-to-end work. - https://awesomecua.com/resources/arxiv-2606-11042/ · source: https://arxiv.org/abs/2606.11042v4 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, long-horizon, workflow, robustness - Published: 2026-06-09 · Authors: see the paper ### WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models WorkflowLLM builds WorkflowBench from application API workflows and synthetic queries, then trains WorkflowLlama to generate multi-application workflows. - https://awesomecua.com/resources/arxiv-2411-05451/ · source: https://arxiv.org/abs/2411.05451v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: workflow, data-generation, automation - Published: 2024-11-08 · Authors: Shengda Fan, Xin Cong, Yuepeng Fu, Zhong Zhang, Shuyan Zhang, Yuanwei Liu, Yesai Wu, Yankai Lin, Zhiyuan Liu, Maosong Sun ### WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning Uses rule-based rewards for output structure and action correctness to train reasoning for enterprise web navigation. - https://awesomecua.com/resources/arxiv-2505-22942/ · source: https://arxiv.org/abs/2505.22942v2 - Paper · reviewed 2026-10-16 · environments: web · topics: reinforcement-learning, workflow - Published: 2025-05-28 · Authors: Yuchen Zhuang, Di Jin, Jiaao Chen, Wenqi Shi, Hanrui Wang, Chao Zhang ### WorkRover: Scaling Long-Horizon Cowork Agent Training Via Recursive Environment Evolution Synthesizes resettable workspaces with files, application observations and skills; repairs environment incompatibilities and trains long-task agents using verifier-linked turn credit. - https://awesomecua.com/resources/arxiv-2610-18067/ · source: https://arxiv.org/abs/2610.18067v1 - Paper · reviewed 2026-10-20 · environments: unspecified · topics: data-generation, long-horizon, evaluation, reward-models, reinforcement-learning, workflow - Published: 2026-10-13 · Authors: Hang He, Linting Wang, Guanming Liu, Bo Mao, Jinxiu Liu, Yiran Suo, Maosen Zhou, Zhiheng Xi, Guanting Dong, Zhou Shao, Rui Zheng, Jie Zhou, Chengcheng Wan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang - Licenses: document CC-BY-4.0 ### World of Bits: An Open-Domain Platform for Web-Based Agents Introduces World of Bits, a web-task platform combining human demonstrations, keyboard and mouse actions, and cached HTTP traffic for repeatable task environments. It evaluates agents trained through behavioral cloning and reinforcement learning. - https://awesomecua.com/resources/url-52a3261eb6a19b402e88/ · source: https://proceedings.mlr.press/v70/shi17a.html - Paper · reviewed 2026-10-17 · environments: web · topics: web-navigation, reinforcement-learning, evaluation, data-generation - Published: 2017-07-17 · Authors: Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, Percy Liang ### World-Model-Augmented Web Agents with Action Correction WAC has a web-agent action model consult a world model, simulate action outcomes and use judge feedback to correct actions before execution. - https://awesomecua.com/resources/arxiv-2602-15384/ · source: https://arxiv.org/abs/2602.15384v1 - Paper · reviewed 2026-11-06 · environments: web · topics: world-models, planning, self-reflection, web-navigation - Published: 2026-02-17 · Authors: Zhouzhou Shen, Xueyu Hu, Xiyun Li, Tianqing Fang, Juncheng Li, Shengyu Zhang ### WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point Introduces desktop and web tasks with varied initial states to test planning recovery and adaptability beyond canonical starting configurations. - https://awesomecua.com/resources/arxiv-2502-08047/ · source: https://arxiv.org/abs/2502.08047v5 - Paper · reviewed 2026-10-16 · environments: desktop, web · topics: evaluation, robustness, planning - Published: 2025-02-12 · Authors: Henry Hengyuan Zhao, Kaiming Yang, Wendi Yu, Difei Gao, Mike Zheng Shou ### Wuying-Browser-Agent: Real-World Centric Fundamental Long-Horizon Browser Agents Combines a browser execution framework, recovery-oriented supervision, and online reinforcement learning, with a bilingual live-web task benchmark. - https://awesomecua.com/resources/arxiv-2608-17319/ · source: https://arxiv.org/abs/2608.17319v1 - Paper · reviewed 2026-10-16 · environments: web · topics: web-navigation, long-horizon, reinforcement-learning - Published: 2026-08-18 · Authors: AIMAE Team, Tianxiang Chen, Yan Cheng, Zhangye Han, Xiaowei Li, Chang Liu, Cheng Liu, Zhongqiang Ma, Long Peng, Xiaobing Tu, Yinggui Wang, Hongliang Wei, Chen Wu, Daiping Xin, Kunyu Zhou, Pengyang Zhou, Peiyuan Chen, Ziyuan Chen, Yutao Deng, Chunyu Dong, Xiangyu Fu, Yicheng Feng, Ruian He, Haochen Li, Miancan Liu, Zhengqin Liu, Wei Peng, Jinkui Ren, Haoyu Tan, Dong Xiao, Rongkun Xue, Shujian Yang, Xianhang Ye, Ziqi Yuan, Ziyang Yu, Linghan Zhang, Xiantao Zhang, Xuanpu Zhao, Yinan Zhao, Zhenghui Zhao, Bin Zhu, Likai Zou ### X-OmniClaw Technical Report: A Unified Mobile Agent for Multimodal Understanding and Interaction Describes an Android agent architecture combining multimodal perception, working and personal memory, hybrid XML/visual grounding, and reusable skills captured from user interactions. - https://awesomecua.com/resources/arxiv-2605-05765/ · source: https://arxiv.org/abs/2605.05765v2 - Paper · reviewed 2026-10-18 · environments: android · topics: mobile-navigation, memory, grounding, workflow - Published: 2026-05-07 · Authors: Xiaoming Ren, Ru Zhen, Chao Li, Yang Song, Qiuxia Hou, Yanhao Zhang, Peng Liu, Qi Qi, Quanlong Zheng, Qi Wu, Zhenyi Liao, Binqiang Pan, Haobo Ji, Haonan Lu ### XBOUND: Exploring Capability Boundaries of Device-Control Agents at the State Level XBOUND evaluates GUI agents at the level of a screen state, considering multiple instructions and possible target actions within it. The study separates action matching from instruction unification and compares effects of grounding and trajectory data. - https://awesomecua.com/resources/arxiv-2505-21279/ · source: https://arxiv.org/abs/2505.21279v2 - Paper · reviewed 2026-10-18 · environments: unspecified · topics: evaluation, grounding - Published: 2025-05-27 · Authors: Shaoqing Zhang, Kehai Chen, Zhuosheng Zhang, Rumei Li, Rongxiang Weng, Yang Xiang, Min Zhang ### Xiaomi-GUI-0 Technical Report Describes a mobile GUI agent trained through real-device interaction, corrected failure trajectories, supervised learning, and reinforcement learning. - https://awesomecua.com/resources/arxiv-2606-31410/ · source: https://arxiv.org/abs/2606.31410v2 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, reinforcement-learning, robustness - Published: 2026-06-30 · Authors: Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao, Niu Lian, Fazhan Liu, Hui Liu, Heng Qu, Qinzhuo Wu, Zhehao Yu, Tongbo Chen, Shiqi Cui, Anan Du, Shukai Jia, Yuanfa Li, Wei Liu, Yike Liu, Wenchao Lu, Zhenbo Luo, Haoyuan Sun, Jiatong Sun, Cheng Tan, Yajie Wang, Changqiao Wu, Tao Xiong, Jiahui Yang, Yuxuan Yuan, Ruoceng Zhang, Shaojie Zhang, Jian Zhu, Jian Luan, Cong Zou ### You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation RealDevWorld evaluates generated software repositories with 194 development tasks and AppEvalPilot, an agent judge that operates the resulting GUIs. The evaluation examines functional behavior, visual fidelity, and runtime responses to produce task-specific diagnostic feedback. - https://awesomecua.com/resources/arxiv-2508-14104/ · source: https://arxiv.org/abs/2508.14104v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: evaluation, automation, observability - Published: 2025-08-17 · Authors: Yutong Bian, Xianhao Lin, Yupeng Xie, Tianyang Liu, Mingchen Zhuge, Siyuan Lu, Haoming Tang, Jinlin Wang, Jiayi Zhang, Jiaqi Chen, Xiangru Tang, Yongxin Ni, Sirui Hong, Chenglin Wu ### You Only Look at Screens: Multimodal Chain-of-Action Agents Introduces Auto-GUI, a screenshot-based agent using previous actions and future action plans to guide device interaction. - https://awesomecua.com/resources/arxiv-2309-11436/ · source: https://arxiv.org/abs/2309.11436v4 - Paper · reviewed 2026-10-16 · environments: android · topics: mobile-navigation, planning - Published: 2023-09-20 · Authors: Zhuosheng Zhang, Aston Zhang ### You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agents Introduces ReadSecBench, a README-based benchmark for measuring whether high-privilege agents follow injected instructions that leak private data; a computer-use agent is evaluated. - https://awesomecua.com/resources/arxiv-2603-11862/ · source: https://arxiv.org/abs/2603.11862v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation - Published: 2026-03-12 · Authors: Ching-Yu Kao, Xinfeng Li, Shenyu Dai, Tianze Qiu, Pengcheng Zhou, Eric Hanchen Jiang, Philip Sperl ### Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw Introduces a safety evaluation of OpenClaw and the CIK taxonomy for analyzing how changes to a personal agent's persistent state affect attacks on its local and service access. - https://awesomecua.com/resources/arxiv-2604-04759/ · source: https://arxiv.org/abs/2604.04759v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: safety-security, evaluation, memory - Published: 2026-04-06 · Authors: Zijun Wang, Haoqin Tu, Letian Zhang, Hardy Chen, Juncheng Wu, Xiangyan Liu, Zhenlong Yuan, Tianyu Pang, Michael Qizhe Shieh, Fengze Liu, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie ### Your Harness is Not Secure: Benchmarking Real-world Threat of Command Line Interface Agent Introduces AdvCLI, a sandbox benchmark for evaluating whether command-line agents carry out security-relevant operating-system tasks across multiple hosts. - https://awesomecua.com/resources/arxiv-2510-06607/ · source: https://arxiv.org/abs/2510.06607v3 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: evaluation, safety-security, sandbox - Published: 2025-10-08 · Authors: Weidi Luo, Qiming Zhang, Tianyu Lu, Xiaogeng Liu, Bin Hu, Hung-Chun Chiu, Siyuan Ma, Yizhe Zhang, Xusheng Xiao, Yinzhi Cao, Zhen Xiang, Chaowei Xiao ### ZeroGUI: Automating Online GUI Learning at Zero Human Cost Combines generated tasks, estimated success rewards, and online reinforcement learning to automate GUI-agent interaction training. - https://awesomecua.com/resources/arxiv-2505-23762/ · source: https://arxiv.org/abs/2505.23762v1 - Paper · reviewed 2026-10-16 · environments: cross-platform · topics: reinforcement-learning, data-generation - Published: 2025-05-29 · Authors: Chenyu Yang, Shiqian Su, Shi Liu, Xuan Dong, Yue Yu, Weijie Su, Xuehui Wang, Zhaoyang Liu, Jinguo Zhu, Hao Li, Wenhai Wang, Yu Qiao, Xizhou Zhu, Jifeng Dai ### ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding ZonUI-3B trains GUI grounding with a curated 24,000-example collection of mobile, desktop, and web screenshots. Its two-stage procedure first learns across platforms and then specializes on high-resolution images, with ablations on data diversity and redundant examples. - https://awesomecua.com/resources/arxiv-2506-23491/ · source: https://arxiv.org/abs/2506.23491v3 - Paper · reviewed 2026-10-17 · environments: web, desktop, cross-platform · topics: grounding, data-generation, efficiency - Published: 2025-06-30 · Authors: ZongHan Hsieh, Tzer-Jen Wei, ShengJing Yang ### Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines Proposes zoom consistency as a confidence signal for multistep GUI grounding and uses it to route between specialist and generalist visual grounding models. - https://awesomecua.com/resources/arxiv-2604-15376/ · source: https://arxiv.org/abs/2604.15376v1 - Paper · reviewed 2026-11-06 · environments: unspecified · topics: grounding, evaluation, efficiency - Published: 2026-04-15 · Authors: Keon Kim, Krish Chelikavada ### Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding ZoomClick uses repeated zooming and context adjustment to improve GUI grounding without training. The paper studies zoom depth, crop sizes, and related choices and introduces GUIZoom-Bench to evaluate how grounding models respond to zoomed views. - https://awesomecua.com/resources/arxiv-2512-05941/ · source: https://arxiv.org/abs/2512.05941v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, evaluation, screen-understanding - Published: 2025-12-05 · Authors: Zhiyuan Jiang, Shenghao Xie, Wenyi Li, Wenqiang Zu, Peihang Li, Jiahao Qiu, Siqi Pei, Lei Ma, Tiejun Huang, Mengdi Wang, Shilong Liu ### Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements Presents ZoomUI, a training-free grounding method that expresses instructions as visual element descriptions and uses model attention to zoom progressively into relevant interface regions. Evaluation examines instruction-to-element localization on GUI benchmarks. - https://awesomecua.com/resources/arxiv-2603-14448/ · source: https://arxiv.org/abs/2603.14448v1 - Paper · reviewed 2026-10-17 · environments: unspecified · topics: grounding, training-free, screen-understanding - Published: 2026-03-15 · Authors: Ziwei Liu, Tao Feng, Borui Kang, Yanbing Yang, Jun Luo