arXiv v2
FLAME: A small language model for spreadsheet formulas
Introduces FLAME, a model trained on Excel formulas for spreadsheet formula repair, completion and retrieval, supporting formula authoring in spreadsheet software.
RESEARCH LIBRARY
Computer-use research library
Source-backed papers on GUI grounding, agents, training, evaluation, and safety.
Read the research survey ↗ · Methods, evaluation, and human agency — an English working manuscript with a source-by-source reading register.
PAPER TOPICSEvaluationData generationWeb navigationPlanningGUI groundingSafety & securityReinforcement learningMobile navigationAll topics ↗
AI-assisted editorial review of primary sources. Dates keep their recorded precision. Resource links below follow reviewed introduction or documentation relationships. Method ↗
arXiv v1
WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics
Collects rendered webpages and structural metadata for visual UI modeling, studying how web semantics transfer to interface-understanding tasks.
arXiv v1
Lexi: Self-Supervised Learning of the UI Language
Introduces Lexi and UICaption to learn UI image-text representations from descriptions of interface functionality. Evaluation covers UI action entailment, instruction-based retrieval, referring-expression grounding, and UI entity recognition.
arXiv v1
Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing
Introduces QTypist, an LLM-based method that generates text inputs from mobile GUI context and integrates with automated testing tools to help them reach further app pages.
arXiv v2
UGIF: UI Grounded Instruction Following
Introduces multilingual smartphone instruction-following data and a pipeline that retrieves help instructions and maps them to UI actions.
arXiv v2
Understanding HTML with Large Language Models
Studies language models for HTML element classification, page description and autonomous web navigation. Fine-tuned models are evaluated on MiniWoB, and a distilled, automatically labeled CommonCrawl HTML corpus supports the training approach.
arXiv v2
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
Pix2Struct pretrains an image-to-text model by reconstructing simplified HTML from masked webpage screenshots. Its variable-resolution visual representation is evaluated across several domains, including user-interface understanding.
arXiv v3
ReAct: Synergizing Reasoning and Acting in Language Models
Introduces ReAct, a method that interleaves reasoning traces and actions for language-model agents; WebShop is one of its interactive decision-making evaluations.
arXiv v1
Towards Better Semantic Understanding of Mobile Interfaces
Augments RICO screens and view hierarchies with human labels for icon shape and meaning and for links between UI elements and text labels. It compares image-only and multimodal models on semantic understanding of previously unseen apps.
arXiv v1
MUG: Interactive Multimodal Grounding on User Interfaces
Studies iterative UI grounding in which users refine or correct an agent's interpretation over multiple conversational turns.
arXiv v4
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Introduces Spotlight, a vision-language model conditioned on a UI screenshot and a selected region of interest. It evaluates screenshot-based mobile UI understanding and explores multitask learning and few-shot prompting.
arXiv v2
Enabling Conversational Interaction with Mobile UI using Large Language Models
Studies few-shot prompting of a single language model for four mobile-UI interaction tasks, aiming to support conversational interface use without a separate trained model for each task.
arXiv v4
ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots
Introduces screenshot question answering over mobile interfaces, linking answers to screen content and bounding boxes.
arXiv v4
CORNET: Learning Table Formatting Rules By Example
Introduces CORNET, a system that learns spreadsheet conditional-formatting rules from cells formatted by users, using symbolic rule enumeration and a neural ranker.
arXiv v4
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Provides a simulated shopping environment where language agents search, configure, and select products from natural-language requirements.
arXiv v2
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
Introduces GUI-based task-oriented dialogue and a dataset pairing mobile-interface operations with conversational responses.
arXiv v2
A data-driven approach for learning to control computers
Combines reinforcement learning with behavioral priors from human interactions to learn computer control through keyboard and mouse actions.
Proceedings of the 39th International Conference on Machine Learning ↗
arXiv v3
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
Introduces MoTIF mobile-app interactions with feasible and infeasible commands, feasibility explanations, and follow-up questions for ambiguous requests.
arXiv v2
Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale
Introduces CLAY, a pipeline that repairs noisy mobile UI layouts by removing nodes without valid visual counterparts and assigning semantic element types, supported by human-annotated layouts for evaluation.
arXiv v3
WebGPT: Browser-assisted question-answering with human feedback
Trains a question-answering model to search and navigate a text-based web browser, collect references and improve long-form answers using demonstrations and human feedback.
arXiv v1
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling
The Versatile UI Transformer jointly models interface screenshots, structure, and language for five UI tasks. Its encoder supports element detection when structural metadata is absent, while an autoregressive component handles UI question answering and command grounding.
arXiv v1
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
Studies mobile-screen summarization using visual, textual, and structural UI information, with a dataset of human-written descriptions.
arXiv v2
UIBert: Learning Generic Multimodal Representations for UI Understanding
Learns shared image, text, and structural UI representations through self-supervised tasks based on relationships among interface components.
arXiv v2
AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learning
Introduces AppBuddy, an RL training platform and task suite for mobile apps where agents use on-screen element states and tap or type to complete multi-step tasks.
arXiv v1
AndroidEnv: A Reinforcement Learning Platform for Android
Presents an Android environment for training and evaluating reinforcement-learning agents through touchscreen interaction.
arXiv v2
Understanding Mobile GUI: from Pixel-Words to Screen-Sentences
Models mobile screens as atomic text and graphic components whose relationships are learned by a Screen Transformer. The work introduces RICO-PW annotations and evaluates metadata-free screen understanding on relation, clickability, retrieval, and app-classification tasks.
arXiv v1
Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments
Introduces MoTIF, a dataset of natural-language requests for interactive mobile apps, including infeasible requests and follow-up questions for studying task feasibility.
arXiv v2
Grounding Open-Domain Instructions to Automate Web Support Tasks
Introduces RUSS, which translates support instructions into a domain-specific language and grounds them in actionable web elements.
arXiv v2
WebSRC: A Dataset for Web-Based Structural Reading Comprehension
Introduces question answering over webpage text and structure, with aligned HTML, screenshots, and metadata for structural reading comprehension.
arXiv v1
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Infers UI elements and accessibility metadata from mobile screenshots to support screen-reader interaction with iOS applications.
arXiv v2
ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces
Pretrains ActionBert on user interaction traces to combine visual, linguistic, and UI-structure features. It evaluates the resulting UI representations on tasks including icon classification and description-based component retrieval.
arXiv v1
Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements
Introduces crowdsourced descriptions of mobile UI elements and studies multimodal models that generate useful widget captions.
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) ↗
arXiv v2
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?
Compares seven traditional and learned GUI element detectors on more than 50,000 interface images. The analysis motivates a coarse-to-fine non-text detector combined with a learned text detector to address GUI-specific localization requirements.
arXiv v2
Mapping Natural Language Instructions to Mobile UI Action Sequences
Introduces PIXELHELP for predicting mobile UI action sequences from instructions, together with action-phrase annotations and synthetic UI action descriptions for training. Separate Transformers extract instruction steps and ground object descriptions using interface content and screen position.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics ↗
arXiv v2
Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations
Introduces PUMICE, an agent that learns mobile-app procedures and conditional concepts through natural-language conversation and GUI demonstrations, recursively resolving ambiguous instructions with users.
arXiv v1
DOM-Q-NET: Grounded RL on Structured Language
Introduces DOM-Q-NET, an RL architecture for web navigation that represents HTML as a tree and uses separate Q-function networks for clicking elements and typing text.
arXiv v1Survey
A Systematic Literature Review of Automated Techniques for Functional GUI Testing of Mobile Applications
Surveys automated functional GUI testing techniques for mobile apps, comparing the effectiveness, efficiency and practicality of approaches reported in 25 studies.
arXiv v1
Learning to Navigate the Web
Introduces guided RL for QWeb, using a curriculum of smaller instructions and generated tasks to train an agent that follows instructions and fills forms on web pages.
arXiv v2
Mapping Natural Language Commands to Web Elements
Defines language grounding on webpages as choosing the UI element targeted by a natural-language command. A collection of more than 50,000 commands and three baseline models study functional references, relationships between elements, and visual position cues.
2018-06
Kite: Building Conversational Bots from Mobile Apps
Derives task graphs from demonstrated Android app interactions and layers a conversational question-answer interface over those graphs. The evaluation examines task-model extraction and developer use of the resulting bot templates.
The 16th Annual International Conference on Mobile Systems, Applications, and Services ↗
arXiv v1
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
Uses workflows extracted from demonstrations to constrain reinforcement-learning exploration for interactive web tasks.
2018
APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Introduces APPINITE, a conversational interface for specifying the intended targets of demonstrated Android actions. It constructs semantic UI representations and uses an on-screen overlay to preview and refine executable data descriptions.
Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing ↗
2018
Rousillon: Scraping Distributed Hierarchical Web Data
Introduces Rousillon, which generalizes a demonstration of collecting one hierarchical data row into a browser-automation program that traverses related pages. The paper develops relation-selection and generalization methods and evaluates users building such scripts.
Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology ↗
Rico: A Mobile App Dataset for Building Data-Driven Design Applications
Introduces Rico, collecting Android interface designs and interaction traces through crowdsourced use followed by automated exploration. The paper demonstrates UI-layout retrieval using learned representations and describes visual, textual, structural, and interaction data.
UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology ↗
PMLR
World of Bits: An Open-Domain Platform for Web-Based Agents
Introduces World of Bits, a web-task platform combining human demonstrations, keyboard and mouse actions, and cached HTTP traffic for repeatable task environments. It evaluates agents trained through behavioral cloning and reinforcement learning.
Proceedings of the 34th International Conference on Machine Learning ↗
2017
SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Introduces SUGILITE, which combines spoken instructions, demonstrated Android actions, and UI hierarchies to generate reusable automation scripts. Its interface includes context checking and repairs for unfamiliar situations and changing app interfaces.
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems ↗
2016-11
Ringer: Web Automation by Demonstration
Introduces Ringer, which converts demonstrated webpage interactions into scripts combining actions, target selectors, and waiting conditions. It evaluates replay robustness on tasks recorded from real websites.
arXiv v2
End-to-End Goal-Driven Web Navigation
Introduces WebNav for converting websites into goal-directed hyperlink navigation tasks and WikiNav for evaluating language-conditioned navigation and question answering on Wikipedia.
2010-04
GUI Testing Using Computer Vision
Presents visual GUI test scripts that use images to specify actions and expected screen feedback. Recording both input events and screenshots can generate tests from demonstrations, supporting unit, regression, and test-driven workflows.
Proceedings of the 28th International Conference on Human Factors in Computing Systems (CHI '10) ↗
2009-10
Sikuli: Using GUI screenshots for search and automation
Introduces screenshot-based GUI search and visual scripting, using image patterns to locate interface targets and direct mouse and keyboard events. It evaluates screenshot queries and demonstrates automation and interactive-help tasks.
Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09) ↗
2007
Koala: Capture, Share, Automate, Personalize Business Processes on the Web
Introduces Koala, a collaborative web automation system that records browser interactions as editable pseudo-natural-language scripts. Scripts are shared through a wiki and instantiated with personal or corporate data for reuse by other users.
2007
PLOW: A Collaborative Task Learning Agent
Describes PLOW, which learns executable web-task models from a session combining demonstrations, natural-language explanations, and dialogue. Browser instrumentation connects user actions with DOM structure to support task generalization.
Twenty-Second AAAI National Conference on Artificial Intelligence ↗
1991
Eager: Programming Repetitive Tasks by Example
Describes Eager, which detects repeated HyperCard interactions, generalizes them into loops, and previews predicted actions by highlighting interface objects. Users can then ask it to execute the inferred repetitive task.
Showing 1,621–1,673 of 1,673, newest first.