Evaluation

MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents

Luyuan Wang, Yongyu Deng et al.

Introduces a mobile-agent benchmark with tasks across open-source applications, organized by difficulty to reduce manual evaluation effort.

On the Effects of Data Scale on UI Control Agents

Wei Li, William Bishop et al.

Studies how Android task performance changes with demonstration scale and releases AndroidControl with high- and low-level instructions.

WebSuite: Systematically Evaluating Why Web Agents Fail

Eric Li, Jim Waldo

WebSuite organizes browser operations into an action taxonomy and tests both isolated controls and complete workflows. Its diagnostic tasks attribute failures to specific actions, enabling comparison of text-based and multimodal web agents.

Large Language Models Can Self-Improve At Web Agent Tasks

Ajay Patel, Markus Hofmarcher et al.

Examines self-improvement by fine-tuning web agents on three mixtures of model-generated training data. WebArena experiments and trajectory-level evaluation measures study task performance, robustness, and behavior beyond aggregate success rates.

AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

Christopher Rawles, Sarah Clinckemaillie et al.

Introduces parameterized Android tasks with initialization, success checks, and cleanup for reproducible interactive evaluation.

Latent State Estimation Helps UI Agents to Reason

William E Bishop, Alice Li et al.

Studies explicit textual estimates of latent UI state, including whether commanded actions occurred and how far a task has progressed. The experiments compare agents that reason with these estimates against agents that omit them across public and internal task benchmarks.

Unveiling Disparities in Web Task Handling Between Human and Web Agent

Kihoon Son, Jinhyeon Kwon et al.

Compares human think-aloud web-task behavior with web-agent planning, action, and reflection. The analysis highlights differences in updating plans, handling ambiguity, and investigating failures.

Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces

El Hassane Ettifouri, Jessica López Espejel et al.

Compares modular OCR-based and end-to-end methods for locating desktop GUI targets from natural-language instructions.

Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning

Lucas-Andreï Thil, Mirela Popa et al.

Combines supervised and reinforcement learning for MiniWoB web tasks and investigates whether agents understand HTML structure or memorize target elements.

MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot

Zirui Song, Yaohang Li et al.

Introduces MMAC-Copilot, which coordinates agents with different expertise to operate PC applications. VIBench evaluates visual interaction with applications that lack usable APIs, spanning office, recreation and game tasks.

Benchmarking Mobile Device Control Agents across Diverse Configurations

Juyong Lee, Taywon Min et al.

Introduces B-MoCA, an interactive mobile-control benchmark that varies device settings, layouts, and languages to test generalization.

MMInA: Benchmarking Multihop Multimodal Internet Agents

Shulin Tian, Ziniu Zhang et al.

Evaluates multimodal agents on compositional tasks across changing websites and studies replaying prior trajectories as memory support.

Large Language Models for Mobile GUI Text Input Generation: An Empirical Study

Chenhui Cui, Tao Li et al.

Studies language-model generation of mobile text inputs, comparing UI representations, execution feedback, and human assistance for app testing.

Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking

Lei Ding, Jeshwanth Bheemanpally et al.

Executes instructions from mobile how-to search results inside a controlled Android environment, then uses the observed success of those instructions to verify and rerank technical-support results.

LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation

Li Zhang, Shihe Wang et al.

LlamaTouch executes mobile tasks on devices and evaluates whether an agent visits manually annotated essential application or system states. Screenshot and hierarchy annotations, combined with exact and fuzzy state matching, support evaluation of 496 tasks across multiple agents.

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Tianbao Xie, Danyang Zhang et al.

Introduces a real-computer evaluation environment with execution-based checks for tasks spanning desktop applications and operating-system state.

Autonomous Evaluation and Refinement of Digital Agents

Jiayi Pan, Yichi Zhang et al.

Compares automatic trajectory evaluators for web navigation and device control against oracle judgments. The resulting feedback is used for agent fine-tuning and inference-time guidance, including experiments on WebArena.

VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

Junpeng Liu, Yifan Song et al.

Separately evaluates web-page reading, understanding, and element grounding across tasks drawn from real websites.

AgentStudio: A Toolkit for Building General Virtual Agents

Longtao Zheng, Zhiyuan Huang et al.

AgentStudio combines an interactive GUI/API environment with tools for creating tasks, annotating interface elements, and labeling actions in videos. Its online suite and GroundUI, IDMBench, and CriticBench datasets examine interaction, grounding, learning from video, and success detection.

Tur[k]ingBench: A Challenge Benchmark for Web Agents

Kevin Xu, Yeganeh Kordi et al.

Evaluates web agents on multimodal annotation tasks adapted from HTML interfaces originally built for crowdsourcing workers.

WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?

Alexandre Drouin, Maxime Gasse et al.

Introduces enterprise web tasks on ServiceNow and BrowserGym interfaces for evaluating browser-based agents.

PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion

Zekai Zhang, Yiduo Guo et al.

Benchmarks robustness of API-based PowerPoint completion under sentence-level, semantic, and multilingual instruction perturbations and changes to the available API set.

OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web

Raghav Kapoor, Yash Parag Butala et al.

Introduces a dataset and benchmark for generating executable task scripts from screenshots and visually grounded desktop or web instructions.

WIPI: A New Web Threat for LLM-Driven Web Agents

Fangzhou Wu, Shutong Wu et al.

Studies WIPI, a black-box indirect-instruction threat in which webpage content redirects LLM-driven web agents. Evaluation spans plugin-based agents, Web GPTs, and open-source web-agent implementations.

On the Multi-turn Instruction Following for Conversational Web Agents

Yang Deng, Xuan Zhang et al.

MT-Mind2Web evaluates web navigation through sequences of conversational user instructions. Self-MAP combines memory-augmented planning with reflection to preserve context and dependencies across those interaction turns.

Understanding the Weakness of Large Language Model Agents within a Complex Android Environment

Mingzhe Xing, Rongkai Zhang et al.

Introduces AndroidArena and analyzes agent limitations in cross-application tasks, user constraints, exploration, and reflection.

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lù, Zdeněk Kasner et al.

Collects expert demonstrations of conversational web navigation and studies retrieval-based element selection for processing large pages.

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

Hongliang He, Wenlin Yao et al.

Combines multimodal web interaction with a real-website task collection and an automated evaluation protocol.

VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

Jing Yu Koh, Robert Lo et al.

Introduces visually grounded website tasks for evaluating multimodal agents that must interpret both images and text.

AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

Chang Ma, Junlei Zhang et al.

Introduces AgentBoard, a multiturn agent evaluation framework with intermediate progress metrics and analysis tools. Its broader task suite includes WebShop and WebArena interactions with stepwise progress measurements.

WebVLN: Vision-and-Language Navigation on Websites

Qi Chen, Dileepa Pitawela et al.

WebVLN trains website navigation from question-based instructions using both rendered visual information and underlying HTML. It introduces WebVLN-v1 and a website-aware navigation network to study this digital variant of vision-and-language navigation.

ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation

Difei Gao, Lei Ji et al.

Introduces Windows application tasks and an actor-critic agent framework for mouse-and-keyboard automation of desktop workflows.

Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web

Hiroki Furuta, Yutaka Matsuo et al.

CompWoB adds 50 composed web-automation tasks to study generalization from simpler interactions. The paper compares prompted and fine-tuned agents, balances task training data, and probes sensitivity to changes in instruction order.

GAIA: a benchmark for General AI Assistants

Grégoire Mialon, Clémentine Fourrier et al.

Introduces questions that test assistant reasoning, multimodal understanding, browsing, and tool use beyond screen-action execution alone.

PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion

Yiduo Guo, Zekai Zhang et al.

Evaluates multi-turn PowerPoint creation and editing, scoring the resulting presentation files rather than requiring a fixed API sequence.

InstructExcel: A Benchmark for Natural Language Instruction in Excel

Justin Payan, Swaroop Mishra et al.

Introduces InstructExcel, a benchmark for generating Excel OfficeScripts from natural-language requests, built using scripts recorded from user actions and a collection of existing spreadsheets.

AllTogether: Investigating the Efficacy of Spliced Prompt for Web Navigation using Large Language Models

Jiarun Liu, Wentao Hu et al.

AllTogether standardizes prompts for HTML-based web navigation and compares prompting with instruction fine-tuning. Its analysis examines how HTML context, interaction history, and current environmental feedback affect task execution.

AgentBench: Evaluating LLMs as Agents

Xiao Liu, Hao Yu et al.

Evaluates language agents across eight interactive environments and analyzes failures in reasoning, decisions, and instruction following.

WebArena: A Realistic Web Environment for Building Autonomous Agents

Shuyan Zhou, Frank F. Xu et al.

Introduces reproducible web environments with task-completion checks for evaluating autonomous web agents.

Android in the Wild: A Large-Scale Dataset for Android Device Control

Christopher Rawles, Alice Li et al.

Collects Android interaction demonstrations with screens, actions, and instructions across device and platform variations for robustness evaluation.

Mind2Web: Towards a Generalist Agent for the Web

Xiang Deng, Yu Gu et al.

Introduces a web-task dataset spanning diverse websites and evaluates generalization to unseen sites and domains.

Responsible Task Automation: Empowering Large Language Models as Responsible Task Automators

Zhizheng Zhang, Xiaoyi Zhang et al.

Presents ResponsibleTA, a coordinator-executor framework that checks command feasibility and task completion while using local memory for privacy-related controls. The paper compares prompted and learned implementations and evaluates them on UI task automation.

SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models

Hongxin Li, Jingran Su et al.

Introduces SheetCopilot, an agent that steers spreadsheet software via atomic virtual APIs, along with a 221-task spreadsheet manipulation benchmark.

AdaPlanner: Adaptive Planning from Feedback with Language Models

Haotian Sun, Yuchen Zhuang et al.

Introduces AdaPlanner, which revises code-structured plans in response to environmental feedback and reuses successful plans as exemplars. Experiments include MiniWoB++ web tasks alongside ALFWorld.

Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction

Danyang Zhang, Zhennan Shen et al.

Provides an Android benchmark toolkit with controlled environments, intermediate feedback, and task sets for reproducible GUI-agent evaluation.

ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots

Yu-Chung Hsiao, Fedir Zubach et al.

Introduces screenshot question answering over mobile interfaces, linking answers to screen content and bounding boxes.

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Shunyu Yao, Howard Chen et al.

Provides a simulated shopping environment where language agents search, configure, and select products from natural-language requirements.

A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility

Andrea Burns, Deniz Arsan et al.

Introduces MoTIF mobile-app interactions with feasible and infeasible commands, feasibility explanations, and follow-up questions for ambiguous requests.

WebSRC: A Dataset for Web-Based Structural Reading Comprehension

Xingyu Chen, Zihan Zhao et al.

Introduces question answering over webpage text and structure, with aligned HTML, screenshots, and metadata for structural reading comprehension.

Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?

Jieshan Chen, Mulong Xie et al.

Compares seven traditional and learned GUI element detectors on more than 50,000 interface images. The analysis motivates a coarse-to-fine non-text detector combined with a learned text detector to address GUI-specific localization requirements.

Mapping Natural Language Commands to Web Elements

Panupong Pasupat, Tian-Shun Jiang et al.

Defines language grounding on webpages as choosing the UI element targeted by a natural-language command. A collection of more than 50,000 commands and three baseline models study functional references, relationships between elements, and visual position cues.

World of Bits: An Open-Domain Platform for Web-Based Agents

Tianlin Shi, Andrej Karpathy et al.

Introduces World of Bits, a web-task platform combining human demonstrations, keyboard and mouse actions, and cached HTTP traffic for repeatable task environments. It evaluates agents trained through behavioral cloning and reinforcement learning.

End-to-End Goal-Driven Web Navigation

Rodrigo Nogueira, Kyunghyun Cho

Introduces WebNav for converting websites into goal-directed hyperlink navigation tasks and WikiNav for evaluating language-conditioned navigation and question answering on Wikipedia.

GUI Testing Using Computer Vision

Tsung-Hsiang Chang, Tom Yeh et al.

Presents visual GUI test scripts that use images to specify actions and expected screen feedback. Recording both input events and screenshots can generate tests from demonstrations, supporting unit, regression, and test-driven workflows.

AgentBench

Agent evaluation suite with containerized tasks and a function-calling interface, including operating-system and shopping environments.

AgentDAM

Web-agent benchmark for measuring unnecessary use of sensitive personal information during end-to-end task execution.

AgentRewardBench

A benchmark and evaluation toolkit for comparing automatic web-agent judges against expert judgments of task success, side effects and repetition.

AndroidArena

Android environment for evaluating language-model-driven agents.

AndroidLab

Reproducible Android benchmark of 138 tasks across nine apps on predefined virtual devices, run via AVD on Mac or Docker on Linux, released with the Android Instruct training set.

AndroidWorld

Android environment with programmatic task evaluation for mobile agents.