arXiv v1
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
Adapts multimodal web agents to unfamiliar websites using a small number of human demonstrations through prompting or meta-learning.
RESEARCH LIBRARY
Computer-use research library
Source-backed papers on GUI grounding, agents, training, evaluation, and safety.
Read the research survey ↗ · Methods, evaluation, and human agency — an English working manuscript with a source-by-source reading register.
PAPER TOPICSEvaluationData generationGUI groundingWeb navigationPlanningMobile navigationReinforcement learningSafety & securityAll topics ↗
AI-assisted editorial review of primary sources; experiments are not independently reproduced. Dates keep their recorded precision. Resource links below follow reviewed introduction or documentation relationships. Method ↗
arXiv v7
Improved GUI Grounding via Iterative Narrowing
Uses iterative visual narrowing to improve localization of requested GUI elements with general or specially trained vision-language models.
Cross-platformGUI grounding
arXiv v1Survey
Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms
Surveys virtual agents for browsers, applications and operating systems, organizing prior work by task type, observation and action spaces, model strategies and evaluation methods. It examines screen-based interaction and keyboard, mouse and touch control, alongside transferability and long-sequence decision-making challenges.
arXiv v1
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use
Presents a preliminary study of Claude 3.5 Computer Use across desktop tasks and software, analyzing planning, action, and critique behavior and providing a framework for deploying GUI-automation models.
arXiv v2
Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
Introduces WebDreamer, which simulates candidate web actions with language-model world models and value estimates before acting.
arXiv v2Survey
GUI Agents with Foundation Models: A Comprehensive Survey
Survey of (M)LLM-based GUI agents organized around their datasets and benchmarks, a unified agent framework with a taxonomy, and commercial applications.
Cross-platformSurvey
arXiv v1
Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
Introduces an orchestrated team of browsing, file, and coding agents with progress tracking and error recovery for general digital tasks.
arXiv v3
From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
Introduces StepAgent, which compares agent and expert actions to supply intermediate rewards for policy learning, with implicit-reward and inverse-RL variants. Evaluation includes WebShop and Mind2Web alongside non-web tasks.
arXiv v2
Attacking Vision-Language Computer Agents via Pop-ups
Tests whether adversarial visual pop-ups divert computer agents from intended tasks in desktop and web benchmark environments.
arXiv v3
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
Trains web agents with an evolving task curriculum, outcome supervision, and reinforcement-learning updates intended to manage policy drift.
arXiv v3Survey
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey
Surveys multimodal mobile agents, organizing prompt-based and training-based approaches alongside evaluation benchmarks, supporting technologies, and open challenges in mobile task execution.
arXiv v2
AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents
Provides a shared Android action environment, reproducible tasks, and instruction data for training and comparing language and multimodal agents.
arXiv v1
From Context to Action: Analysis of the Impact of State Representation and Context on the Generalization of Multi-Turn Web Navigation Agents
Analyzes how interaction history and webpage representation affect multiturn web-agent generalization across unfamiliar websites, task categories, and geographic settings.
arXiv v3
Evaluating Cultural and Social Awareness of LLM Web Agents
Introduces CASA to assess web agents' responses to cultural and social norms in shopping and forum tasks, including misleading observations.
Findings of the Association for Computational Linguistics: NAACL 2025 ↗
arXiv v1
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Combines cross-platform GUI grounding data synthesis with action-model training to study generalization to unfamiliar interfaces.
arXiv v1
Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents
Discovers compact intents from demonstrations and uses intent predictions to guide a language-model web agent's decisions and exploration.
arXiv v1
AutoGLM: Autonomous Foundation Agents for GUIs
Presents AutoGLM agents for browser and phone control, separating planning from grounding and training through an evolving interaction curriculum.
arXiv v1
OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
Develops multimodal browser agents through repeated cycles of real-web exploration, trajectory feedback, and policy optimization.
arXiv v2
EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
Synthesizes GUI training examples at multiple levels of detail from webpages and studies transfer of learned understanding to other interface environments.
arXiv v3
VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks
Evaluates whether web agents can retain procedural skills and factual information from long video tutorials and use them during tasks.
arXiv v2
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Extends Ferret-UI across device platforms with adaptive visual scaling and synthetic data for referring, grounding, and user-centered UI tasks.
arXiv v1
OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning
OSCAR translates user instructions into Python control code while observing screen images and acting through mouse and keyboard operations. A state-machine controller handles errors and replans tasks during execution across desktop and mobile applications.
arXiv v1
AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
Introduces a platform that integrates specialized agents under a coordinating MetaAgent for varied computer tasks.
arXiv v2
Lightweight Neural App Control
LiMAC combines a compact Action Transformer with a fine-tuned vision-language model for Android app control. It uses textual goals and histories of screenshots and UI trees to predict actions, with evaluation on two mobile-control datasets.
arXiv v3
MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control
Evaluates mobile agents on misuse, side effects, and indirect prompt-injection risks in an Android emulator environment.
arXiv v4
AdvAgent: Controllable Blackbox Red-teaming on Web Agents
Introduces a black-box web-agent red-teaming framework that learns adversarial prompts from agent feedback to study security weaknesses.
arXiv v3
Beyond Browsing: API-Based Web Agents
Compares browsing-only, API-only, and hybrid web agents on WebArena, examining how access to application APIs changes task execution alongside browser interaction.
Findings of the Association for Computational Linguistics: ACL 2025 ↗
arXiv v3
SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation
Introduces a smartphone-agent benchmark with English and Chinese tasks, real-time Android interaction, and completion and resource-use metrics.
arXiv v5
DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents
Introduces asynchronous distributed reinforcement learning with centralized updates and decentralized interaction collection for device-control agents.
arXiv v2
AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents
Studies how simplifying and aligning a web agent's observations and actions with language-model capabilities affects task completion.
arXiv v3
Harnessing Webpage UIs for Text-Rich Visual Understanding
Builds MultiUI by generating instructions from webpage accessibility trees and pairing them with screenshots for multimodal training. Evaluation examines web-interface understanding, including VisualWebBench and Mind2Web, and transfer to other text-rich visual tasks.
arXiv v3
MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation
Combines adaptive mobile-task planning, reflection-based recovery, and multiple memory components, evaluated with MobBench and AndroidArena.
arXiv v2
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
Uses a learned web world model to predict important state changes and compare candidate actions before executing them.
arXiv v2
Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents
Introduces BrowserART and studies whether chat-model refusal behavior transfers to browser agents carrying out potentially harmful instructions.
arXiv v2
From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts
Develops a taxonomy of mobile-action consequences and annotated interaction data to evaluate whether models recognize impactful or risky UI operations.
arXiv v1
Agent S: An Open Agentic Framework that Uses Computers Like a Human
Combines hierarchical planning, retrieved experience, and a computer interaction interface to execute multistep GUI tasks.
arXiv v2
ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
Separates multimodal reasoning and action planning from a dedicated UI localization model, with evaluation on Android devices.
Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue ↗
arXiv v3
TinyClick: Single-Turn Agent for Empowering GUI Automation
Adapts a compact Florence-2 model to map user instructions to screen coordinates using multitask training and augmented data.
arXiv v7
ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
Evaluates web-task completion under explicit enterprise policies, measuring policy-compliant completion and safety violations separately.
arXiv v3
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Introduces UGround, a visual target-localization model trained with large-scale synthetic references to elements in GUI screenshots.
arXiv v2
NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild
Generates browser-agent demonstrations through autonomous exploration and retrospective task labeling, pruning trajectories that lack meaningful subtasks.
arXiv v2
Multimodal Auto Validation For Self-Refinement in Web Agents
Studies text and visual validation for web agents and uses the resulting validator to detect failures and guide workflow self-refinement.
arXiv v3
Dynamic Planning for LLM-based Graphical User Interface Automation
Dynamic Planning of Thoughts revises a mobile GUI plan using current environmental feedback and execution history. The study compares this planning strategy with long-dialogue prompting and examines its behavior on unfamiliar tasks and different language-model backbones.
Findings of the Association for Computational Linguistics: EMNLP 2024 ↗
arXiv v1
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation
Generates labeled slides and interface visuals through code to support visual element detection, description, and classification with less manual annotation.
arXiv v1
Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interface
Studies agent-system and interface co-design that incorporates user interruptions and intermediate feedback to reduce planning latency.
arXiv v2
Inferring Alt-text For UI Icons With Large Language Models During App Development
Introduces IconDesc, which generates mobile icon alt-text from partial UI context such as element identifiers, bounds, nearby text and hierarchy. Model evaluation and a user study examine the resulting accessibility descriptions.
arXiv v2
AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents
Introduces AXIS, an API-first application agent that can use UI actions and expand its available APIs through automated application exploration. The work evaluates the approach on Microsoft Word tasks.
arXiv v2
Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
Synatra converts indirect procedural knowledge, including human-oriented tutorials, into direct demonstrations for digital agents. It trains a 7B CodeLlama on 100,000 synthesized demonstrations and evaluates the resulting policy on Mind2Web, MiniWoB++, and WebArena.
arXiv v1
Steward: Natural Language Web Automation
Introduces Steward, a natural-language browser automation system that repeatedly plans and executes website actions. The paper examines completion detection, state representation, responsiveness, and caching in real-site tasks.
arXiv v2
MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding
MobileVLM adds two pretraining stages and four UI tasks to learn element detail and relationships between successive screens. Its Mobile3M dataset contains three million Chinese mobile UI pages connected by recorded transition actions.
Findings of the Association for Computational Linguistics: EMNLP 2024 ↗
arXiv v3
MobileViews: A Million-scale and Diverse Mobile GUI Dataset
Builds mobile screenshot and view-hierarchy pairs through automated application traversal and evaluates their usefulness for GUI grounding.
arXiv v3
StateAct: Enhancing LLM Base Agents via Self-prompting and State-tracking
Maintains explicit task goals and evolving state descriptions to improve base-agent decisions across interactive tasks, including web shopping.
arXiv v3
The Impact of Element Ordering on LM Agent Performance
Studies how the ordering of UI elements in an agent observation affects web and desktop performance. A pixel-based element detector and dimensionality-reduction ordering are evaluated on OmniACT when document hierarchy is unavailable.
arXiv v5
EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
Evaluates privacy-directed webpage instruction injection against generalist web agents using PII-related interaction steps from Mind2Web, including attempts to expose specific user information or entire requests.
arXiv v4
Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps
Orienter detects interactable GUI elements in XR applications by interpreting scene context and revisiting candidate elements through a validation and reflection loop. It addresses open-vocabulary element types and context-dependent interactability.
arXiv v1
NaviQAte: Functionality-Guided Web Application Navigation
Frames web-application exploration as functionality-oriented question answering, generating navigation actions from multimodal context.
arXiv v1
PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM
PeriGuru interprets smartphone GUI screenshots with computer vision and uses a language model to decide the next action. A peripheral robotic arm executes the phone interactions, providing an alternative mechanism for operating mobile apps.
arXiv v2
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Adapts real-desktop evaluation to Windows applications and supports parallel execution of benchmark tasks.
arXiv v1
Agent Workflow Memory
Agent Workflow Memory extracts reusable task routines and selectively supplies them to guide later web actions. It studies both offline workflow induction from training examples and online induction from test experience on Mind2Web and WebArena.
arXiv v2
WebQuest: A Benchmark for Multimodal QA on Web Page Sequences
Introduces question answering over individual screens, multiple web pages, and navigation traces to test information synthesis across views.
Showing 961–1,020 of 1,202, newest first.