arXiv v1
UEyes: An Eye-Tracking Dataset across User Interface Types
Introduces UEyes, an eye-tracking dataset of webpage, desktop, mobile, and poster interfaces, and analyzes how people view these different UI types.
RESEARCH LIBRARY
Computer-use research library
Source-backed papers on GUI grounding, agents, training, evaluation, and safety.
Read the research survey ↗ · Methods, evaluation, and human agency — an English working manuscript with a source-by-source reading register.
PAPER TOPICSEvaluationData generationWeb navigationPlanningGUI groundingSafety & securityReinforcement learningMobile navigationAll topics ↗
AI-assisted editorial review of primary sources. Dates keep their recorded precision. Resource links below follow reviewed introduction or documentation relationships. Method ↗
arXiv v3
ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Introduces ScreenAI and screen-annotation training that supports UI question answering, navigation, and summarization data generation.
arXiv v2
Dual-View Visual Contextualization for Web Navigation
Uses screenshot geometry and visual content to enrich HTML element representations for web navigation. Nearby elements contribute textual and visual context, with evaluation on cross-task, cross-website, and cross-domain Mind2Web splits.
arXiv v2
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Uses visual perception tools and task decomposition to operate mobile applications without relying on app interface metadata.
arXiv v3
GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning
Introduces GPTVoiceTasker, a mobile assistant that interprets user commands, executes device interactions, and learns from prior commands to streamline later task execution. The work includes a user study of task efficiency.
arXiv v4
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
Combines multimodal web interaction with a real-website task collection and an automated evaluation protocol.
arXiv v2
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Introduces visually grounded website tasks for evaluating multimodal agents that must interpret both images and text.
arXiv v2
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Introduces AgentBoard, a multiturn agent evaluation framework with intermediate progress metrics and analysis tools. Its broader task suite includes WebShop and WebArena interactions with stepwise progress measurements.
arXiv v2
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Trains a screenshot-driven GUI agent with grounding pretraining and introduces ScreenSpot for evaluating interface target localization.
arXiv v3
MobileAgent: enhancing mobile control via human-machine interaction and SOP integration
Combines operating-procedure context with user interaction to guide mobile task automation and handle personalized or sensitive requests.
arXiv v2
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Introduces SeeAct and live-website evaluation, examining how visual planning and element grounding affect web-agent task completion.
Proceedings of the 41st International Conference on Machine Learning ↗
arXiv v1
AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI
Executes voice-requested mobile tasks through GUI exploration, backtracking, and accumulated interaction experience.
arXiv v1
WebVLN: Vision-and-Language Navigation on Websites
WebVLN trains website navigation from question-based instructions using both rendered visual information and underlying HTML. It introduces WebVLN-v1 and a website-aware navigation network to study this digital variant of vision-and-language navigation.
arXiv v3
AppAgent: Multimodal Agents as Smartphone Users
Builds app knowledge through exploration or human demonstrations and uses it to guide multimodal smartphone interactions.
arXiv v2
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Introduces Windows application tasks and an actor-critic agent framework for mouse-and-keyboard automation of desktop workflows.
arXiv v2
VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
VisionTasker converts screenshots into textual UI interpretations and presents one interface at a time to an LLM planner for action selection. It evaluates visual representations and execution of 147 tasks on an Android smartphone, including integration with programming by demonstration.
arXiv v4
UINav: A Practical Approach to Train On-Device Automation Agents
Introduces UINav, a demonstration-based method for training mobile UI automation agents, with a referee model for failure feedback and automatic augmentation of examples.
arXiv v3
CogAgent: A Visual Language Model for GUI Agents
Introduces a visual-language model with low- and high-resolution image encoders for screenshot understanding and GUI navigation.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ↗
arXiv v1
Intelligent Virtual Assistants with LLM-based Process Automation
Presents LLMPA, a mobile process-automation architecture that decomposes requests, identifies interface elements, predicts actions and checks errors. Demonstrations evaluate multistep operations in Alipay.
arXiv v3
Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation
Introduces MobileGPT, which organizes mobile-app experience into reusable subtasks using an explore, select, derive, and recall cycle. Its memory supports learning task procedures and adapting them to new contexts.
arXiv v3
Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web
CompWoB adds 50 composed web-automation tasks to study generalization from simpler interactions. The paper compares prompted and fine-tuned agents, balances task training data, and probes sensitivity to changes in instruction order.
arXiv v1
GAIA: a benchmark for General AI Assistants
Introduces questions that test assistant reasoning, multimodal understanding, browsing, and tool use beyond screen-action execution alone.
arXiv v1
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
Studies MM-Navigator, a multimodal agent that interprets smartphone screenshots and predicts localized actions without task-specific training.
arXiv v2
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion
Evaluates multi-turn PowerPoint creation and editing, scoring the resulting presentation files rather than requiring a fixed API sequence.
arXiv v2
ProAgent: From Robotic Process Automation to Agentic Process Automation
Introduces ProAgent, an LLM-based agent that builds automation workflows from human instructions and coordinates specialized agents during workflow execution.
arXiv v2
WebWISE: Web Interface Control and Sequential Exploration with Large Language Models
WebWISE filters DOM elements into observations and generates short programs for click, scroll, and text-entry steps. It evaluates in-context learning from either one supplied example or an example collected during a successful zero-shot MiniWoB++ trial.
arXiv v1
Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions
Introduces GPTDroid for mobile GUI testing: it feeds app page information to an LLM, runs generated test scripts and uses feedback and memory to guide further exploration.
arXiv v1
InstructExcel: A Benchmark for Natural Language Instruction in Excel
Introduces InstructExcel, a benchmark for generating Excel OfficeScripts from natural-language requests, built using scripts recorded from user actions and a collection of existing spreadsheets.
arXiv v2
AllTogether: Investigating the Efficacy of Spliced Prompt for Web Navigation using Large Language Models
AllTogether standardizes prompts for HTML-based web navigation and compares prompting with instruction fine-tuning. Its analysis examines how HTML context, interaction history, and current environmental feedback affect task execution.
arXiv v3Survey
Vision-Based Mobile App GUI Testing: A Survey
Surveys vision-based mobile app GUI testing across test generation, record-and-replay and testing frameworks, drawing on 271 papers including 92 vision-based studies.
arXiv v2
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Overlays labeled image regions to help multimodal models refer to and reason about visual targets; a general visual-prompting method relevant to GUI grounding.
GUI groundingAdjacent research
arXiv v1
OpenAgents: An Open Platform for Language Agents in the Wild
OpenAgents hosts three language agents, for data analysis, API plugins and autonomous web browsing, behind a web interface for non-expert users, with local deployment for developers.
arXiv v3
A Zero-Shot Language Agent for Computer Control with Structured Reflection
Studies a zero-shot computer-control agent that plans executable actions and organizes its own reflections after mistakes. MiniWoB++ experiments examine learning to complete tasks without supplied expert trajectories.
arXiv v1
CrashTranslator: Automatically Reproducing Mobile Application Crashes Directly from Stack Trace
Introduces CrashTranslator, which predicts GUI exploration steps from stack traces with an LLM and uses RL-guided search to reproduce crashes in Android apps.
arXiv v1
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
Introduces UReader, which combines unified instruction tuning, auxiliary text-reading tasks and shape-adaptive image crops for OCR-free visual language understanding. Its evaluation includes webpage screenshots alongside documents, tables, charts and natural images.
arXiv v1
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
ILuvUI generates 335,000 conversational training examples paired with UI screenshots, covering questions, descriptions, and plans. The resulting vision-language model is assessed on element detection, response quality, and examples of multi-step UI navigation.
arXiv v1
Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API
Trains a screenshot-based instruction-grounding model to predict target coordinates, using reinforcement learning to improve spatial decoding.
arXiv v4
SteP: Stacked LLM Policies for Web Actions
SteP dynamically composes specialized web-action policies through a stack that records the active chain of policy calls. It studies this control structure across WebArena, MiniWoB++, and CRM workflows.
arXiv v3
AXNav: Replaying Accessibility Tests from Natural Language
Introduces an LLM and pixel-based UI system that executes natural-language accessibility tests, records navigable videos and flags potential interface issues.
arXiv v4
You Only Look at Screens: Multimodal Chain-of-Action Agents
Introduces Auto-GUI, a screenshot-based agent using previous actions and future action plans to guide device interaction.
Findings of the Association for Computational Linguistics: ACL 2024 ↗
arXiv v2
LASER: LLM Agent with State-Space Exploration for Web Navigation
LASER represents web navigation as exploration over a defined set of states, allowing an agent to revisit earlier states after mistakes. The study evaluates this backtracking approach on WebShop and Amazon tasks.
arXiv v4
AutoDroid: LLM-powered Task Automation in Android
Combines language-model reasoning with app analysis, UI representations, and exploration-derived memory for Android task automation.
arXiv v2
Unveiling the Tricks: Automated Detection of Dark Patterns in Mobile Applications
Introduces UIGuard, a system that examines mobile UI visuals and text to detect dark patterns, together with a labeled dataset of benign and malicious interfaces.
arXiv v3
AgentBench: Evaluating LLMs as Agents
Evaluates language agents across eight interactive environments and analyzes failures in reasoning, decisions, and instruction following.
arXiv v4
WebArena: A Realistic Web Environment for Building Autonomous Agents
Introduces reproducible web environments with task-completion checks for evaluating autonomous web agents.
arXiv v4
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Combines task decomposition, HTML summarization, and generated Python programs to carry out instructions on websites.
arXiv v2
Android in the Wild: A Large-Scale Dataset for Android Device Control
Collects Android interaction demonstrations with screens, actions, and instructions across device and platform variations for robustness evaluation.
arXiv v3
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
Synapse filters task-irrelevant page state and places complete abstracted state-action trajectories into the prompt. An embedding-based exemplar memory retrieves relevant demonstrations for new tasks, evaluated on MiniWoB++ and Mind2Web.
arXiv v3
Mind2Web: Towards a Generalist Agent for the Web
Introduces a web-task dataset spanning diverse websites and evaluates generalization to unseen sites and domains.
arXiv v3
Prompting Is All You Need: Automated Android Bug Replay with Large Language Models
Introduces AdbGPT, a prompting-based approach that uses LLMs to replay app bugs from written reports, applying few-shot examples and reasoning without model training.
arXiv v2
Responsible Task Automation: Empowering Large Language Models as Responsible Task Automators
Presents ResponsibleTA, a coordinator-executor framework that checks command feasibility and task completion while using local memory for privacy-related controls. The paper compares prompted and learned implementations and evaluates them on UI task automation.
arXiv v2
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Studies instruction-following agents that observe screenshots and act through a generic mouse-and-keyboard action space, evaluating pixel-based pretraining for GUI tasks on MiniWoB++.
arXiv v2
SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models
Introduces SheetCopilot, an agent that steers spreadsheet software via atomic virtual APIs, along with a 221-task spreadsheet manipulation benchmark.
arXiv v1
AdaPlanner: Adaptive Planning from Feedback with Language Models
Introduces AdaPlanner, which revises code-structured plans in response to environmental feedback and reuses successful plans as exemplars. Experiments include MiniWoB++ web tasks alongside ALFWorld.
arXiv v3
Hierarchical Prompting Assists Large Language Model on Web Navigation
Separates web navigation prompting into a SUMMARIZER that extracts action-relevant observations and an ACTOR that chooses the next action from the condensed context.
arXiv v4
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Trains WebGUM offline from demonstrations by jointly tuning an instruction-following language model and a vision encoder over webpage screenshots and HTML. The agent predicts browser actions and is evaluated on MiniWoB, WebShop, and Mind2Web.
arXiv v1
Mobile User Interface Element Detection Via Adaptively Prompt Tuning
Introduces MUI-zh and APT for detecting mobile UI elements, combining visual features with OCR descriptions to adjust category prompts in CLIP-based detectors.
arXiv v4
Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
Provides an Android benchmark toolkit with controlled environments, intermediate feedback, and task sets for reproducible GUI-agent evaluation.
arXiv v5
DroidBot-GPT: GPT-powered UI Automation for Android
Transforms Android UI states and available actions into language prompts for an LLM to choose and execute task steps.
arXiv v3
Language Models can Solve Computer Tasks
Introduces recursive criticism and improvement prompting for language-model agents performing computer tasks, evaluated on MiniWoB++ with a small number of demonstrations per task.
Showing 1,561–1,620 of 1,673, newest first.