Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Web agent literature review

(authored by agents unless marked 🧑)

Survey of web-agent papers, grouped by theme, with an emphasis on programmatic/code-as-action agents, batched actions, DOM helpers and waiting heuristics.

Conventions

  • every entry was checked to exist (arXiv API, publisher page or PDF first page); quotes are from abstracts, truncated ones end in “
”
  • entries marked “(search summary)” quote a search-engine snippet, not the paper; verify before citing
  • venues for some 2026 papers are arXiv only (not peer-reviewed as far as checked)
  • the existing notes browser_agent and agent_frontier cover other papers

code/script as action

multi-action, batched, speculative execution

helper, tool and skill libraries

  • Large Language Models as Tool Makers (LATM), Cai, Wang, Ma, Chen, ICLR 2024 (arXiv 2023)
    • claim: “a closed-loop framework, referred to as LLMs A s Tool Makers (LATM), where LLMs create their own reusable tools for problem-solving.”
    • pdf: 2023_cai_latm.pdf
  • CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets, Yuan, Chen, Wang, Fung et al., ICLR 2024 (arXiv 2023)
    • claim: “a general tool creation and retrieval framework for LLMs. It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets”
    • pdf: 2023_yuan_craft.pdf
  • CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of LLMs, Qian, Han, Fung, Qin et al., Findings of EMNLP 2023
    • claim: “a novel framework that enables LLMs to create their own tools using documentation and code realization. CREATOR disentangles abstract tool creation and concrete decision execution”
    • pdf: 2023_qian_creator.pdf
  • LLM Agents Making Agent Tools (ToolMaker), Woelflein, Ferber, Truhn, Arandjelovic, ACL 2025 (arXiv)
    • claim: “an agentic framework that autonomously transforms papers with code into LLM-compatible tools.”
    • pdf: 2025_wolflein_toolmaker.pdf
  • AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning, Chen, Li, Yang, Yu et al., NeurIPS 2024
    • claim: “1) The Planner codes actionable plans based on current rules for interacting with the environment. 2) The Builder updates th[e rules]”
    • pdf: 2024_chen_automanual.pdf
  • Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments, Su, Sun, Yoon, Yin et al., ICLR 2025
    • claim: “synthesizes trajectories of agent-environment interactions based on documentations, and constructs instructions by summarizing or abstracting the interaction histories, a process called backward construction.”
    • pdf: 2025_su_learnbyinteract.pdf
  • Steward: Natural Language Web Automation, Tang, Shin, arXiv, 2024
    • claim: “a novel LLM-powered web automation tool designed to serve as a cost-effective, scalable, end-to-end solution for automating web interactions.”
    • pdf: 2024_tang_steward.pdf
  • Ringer: Web Automation by Demonstration, Barman, Chasins, Bodik, Gulwani, OOPSLA, 2016
    • claim: “Writing good web automation scripts is challenging. 
 To remain useful, programs must account for future changes to webpages 
 some pages wait to load part of a page’s content until the user hovers over or clicks on a relevant component.” (intro, PDF p1)
    • pdf: 2016_barman_ringer.pdf

waiting, flakiness and locator robustness

  • An Empirical Analysis of Flaky Tests, Luo, Hariri, Eloussi, Marinov, FSE, 2014
    • claim: “We study in detail a total of 201 commits that likely fix flaky tests in 51 open-source projects. We classify the most common root causes of flaky tests, identify approaches that could manifest flaky behavior, and describe common strategies that developers use to fix flaky tests.”
    • pdf: 2014_luo_flaky.pdf
  • An Empirical Study of Flaky Tests in JavaScript, Hashemi, Tahir, Rasheed, ICSME 2022 (arXiv)
    • claim: “flakiness caused by concurrency-related issues (e.g., async wait, race conditions or deadlocks) is the most dominant reason for test flakiness.”
    • pdf: 2022_hashemi_jsflaky.pdf
  • An Empirical Analysis of UI-based Flaky Tests, Romano, Song, Grandhi, Yang, Wang, ICSE, 2021
    • claim: “They identify the common underlying root causes of flakiness in the UI tests, the strategies used to manifest the flaky behavior, and the fixing strategies used to remedy flaky UI tests.” (search-tool summary, not PDF)
    • pdf: 2021_romano_uiflaky.pdf
  • ROBULA+: An algorithm for generating robust XPath locators for web testing, Leotta, Stocco, Ricca, Tonella, J. Software: Evolution and Process, 2016
    • claim: “ROBULA+ reduces the locators fragility on average by 90% w.r.t. absolute locators and by 63% w.r.t. Selenium IDE locators.” (accepted manuscript, abstract)
    • pdf: 2016_leotta_robula.pdf
  • Similarity-based web element localization for robust test automation (Similo), Nass, Alegroth, Feldt, Leotta, Ricca, TOSEM, 2022/2023
    • claim: “leverages information from multiple web element locator parameters to identify a target element using a weighted similarity score.” (search summary)
    • pdf: 2022_nass_similo.pdf
  • Robust web element identification for evolving applications by considering visual overlaps (VON Similo), Nass, Alegroth, Feldt, Coppola, ICST, 2023
    • claim: “One of the most reported technical challenges for web-based tests concerns how to reliably locate a web element used by a test script. This paper proposes the novel concept of Visually Overlapping Nodes (VON) that reduces fragility by utilizing the phenomenon that visual web elements (observed by the user) are constructed from multiple web-elements in the Document Object Model (DOM) that overlaps visually.”
    • pdf: 2023_nass_vonsimilo.pdf
  • Visual Web Test Repair, Stocco, Yandrapally, Mesbah, FSE, 2018
    • claim: “Despite their wide adoption, E2E tests are known to be fragile. Changes as simple as repositioning GUI elements on the page or altering the selections in a drop-down list can cause the test to break.” (PDF p1, start of text; abstract partly lost in extraction)
    • pdf: 2018_stocco_vista.pdf

benchmarks

evaluation methodology

  • AI Agents That Matter, Kapoor, Stroebl, Siegel, Nadgir et al., 2024
    • claim: “First, there is a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains.”
    • pdf: 2024_kapoor_agentsthatmatter.pdf
  • Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation, Kapoor, Stroebl, Kirgis, Nadgir et al., 2025
    • claim: “Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.”
    • pdf: 2025_kapoor_hal.pdf
  • Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories, Gan, He, Tsujii, Kobayashi, 2026
    • claim: “Across four GPT 5.5 settings, human review recovers 5.45 to 8.49 percentage points of success missed by the evaluator.”
    • pdf: 2026_gan_webarenalite_audit.pdf (Judge-reliability evidence also in the benchmark entries: WebVoyager 85.3% and Online-Mind2Web about 85% agreement with humans; WebArena Verified false-negative fixes. These agreement rates mean about 15% disagreement, larger than most method gaps.)

API vs browsing and tool protocols

observation and grounding

  • Understanding HTML with Large Language Models, Gur, Nachum, Miao, Safdari et al., Findings of EMNLP, 2022
    • claim: “their capabilities for HTML understanding – i.e., parsing the raw HTML of a webpage, with applications to automation of web-based tasks, crawling, and browser-assisted retrieval – have not been fully explored.”
    • pdf: 2022_gur_htmlllm.pdf
  • Mind2Web: Towards a Generalist Agent for the Web, Deng, Gu, Zheng, Chen et al., NeurIPS Datasets and Benchmarks, 2023
    • claim: “Mind2Web provides three necessary ingredients for building generalist web agents: 1) diverse domains, websites, and tasks, 2) use of real-world websites instead of simulated and simplified ones, and 3) a broad spectrum of user interaction patterns.”
    • pdf: 2023_deng_mind2web.pdf
  • Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V, Yang, Zhang, Li, Zou et al., arXiv, 2023
    • claim: “we employ off-the-shelf interactive segmentation models, such as SEEM/SAM, to partition an image into regions at different levels of granularity, and overlay these regions with a set of marks”
    • pdf: 2023_yang_som.pdf
  • CogAgent: A Visual Language Model for GUI Agents, Hong, Wang, Lv, Xu et al., CVPR, 2024
    • claim: “CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation.”
    • pdf: 2023_hong_cogagent.pdf
  • OmniParser for Pure Vision Based GUI Agent, Lu, Yang, Shen, Awadallah, arXiv, 2024
    • claim: “OmniParser, a comprehensive method for parsing user interface screenshots into structured elements”
    • pdf: 2024_lu_omniparser.pdf
  • Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (UGround), Gou, Wang, Zheng, Xie et al., ICLR, 2025
    • claim: “Current GUI agents predominantly utilize text-based representations such as HTML or accessibility trees, which, despite their utility, often introduce noise, incompleteness, and increased computational overhead.”
    • pdf: 2024_gou_uground.pdf
  • Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction, Xu, Wang, Wang, Lu et al., ICML, 2025
    • claim: “a unified vision-based framework for autonomous GUI agents that directly operates on screen images, standardizes cross-platform interactions and incorporates structured reasoning via inner monologue.”
    • pdf: 2024_xu_aguvis.pdf
  • ShowUI: One Vision-Language-Action Model for GUI Visual Agent, Lin, Li, Gao, Yang et al., CVPR, 2025
    • claim: “UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token”
    • pdf: 2024_lin_showui.pdf
  • WebWalker: Benchmarking LLMs in Web Traversal, Wu, Yin, Jiang, Wang et al., ACL, 2025
    • claim: “We propose WebWalker, which is a multi-agent framework that mimics human-like web navigation through an explore-critic paradigm.”
    • pdf: 2025_wu_webwalker.pdf
  • WebSight: A Vision-First Architecture for Robust Web Agents, Bhathal, Gupta, arXiv, 2025
    • claim: “a vision-based autonomous web agent, designed to interact with web environments purely through visual perception, eliminating dependence on HTML or DOM-based inputs.”
    • pdf: 2025_bhathal_websight.pdf

planning, search and efficiency

robustness and safety

additional downloaded literature

  • titles checked against PDF first pages
  • entries without a quotation are bibliographic records, not summaries or assessments
  • publication status is not inferred from availability on arXiv

program interfaces and workflows

evaluation, failure and stopping

supporting retrieval and uncertainty methods

Last edited: