Web and computer-use agents

A web or computer-use agent looks at a screen, decides what to do, and moves a mouse or presses keys, just like you. Evaluating one means building a whole computer that can be reset on demand, and then deciding, from the wreckage the agent leaves behind, whether it did the job.

What makes these agents different

In Module 06 the agent called neat, typed functions such as get_order(order_id="A17"). A web agent instead works through a browser, and a computer-use agent (also called a GUI agent, where GUI means graphical user interface) works through a whole desktop or phone. Nobody wrote an API for it. It has to read the screen the way a person does, figure out which of hundreds of pixels or page elements matters, and then act through the same crude channels a person uses: clicking, typing, scrolling, pressing shortcuts.

That buys enormous generality (any app a human can use, the agent can in principle use) and it creates three evaluation headaches that this module is about:

The perceive, ground, act loop

Every GUI agent runs the same loop you met in Module 05, but with a special middle step. Perceive: get an observation of the screen. Reason: decide what should happen next ("I need to type the search query"). Ground: turn that intention into a concrete target, either a pixel coordinate or an element ID. Act: send the click or keystroke to the environment. Then look again.

Why separate grounding from reasoning? Because they fail differently and are fixed differently. A model that knows it should click "Add to cart" but clicks 20 pixels too low has a grounding problem (fix: better vision, zoom, higher-resolution screenshots, or element IDs). A model that clicks exactly on "Add to wishlist" because it thinks that is right has a reasoning problem. End-to-end success rates blur the two, which is why dedicated grounding benchmarks such as ScreenSpot exist (covered below).

Observation spaces: what the agent sees

The observation space is the format in which the environment describes itself to the agent at each step. For GUI agents there are four common choices, often combined.

Screenshots (pixels)

The agent receives an image of the screen. This is the most general option: it works on any app, including canvas-based editors, games, remote desktops and legacy software with no accessible markup. It is also the hardest, because the model must read small text and locate targets precisely. Pure-screenshot agents are what the Anthropic and OpenAI computer-use tools are built around.

Accessibility tree

Operating systems and browsers maintain an accessibility tree for screen readers: a hierarchy of elements, each with a role (button, link, textbox), an accessible name ("Add to cart") and some state (checked, disabled). It is compact and names things explicitly, but it can be incomplete or wrong (unlabelled icon buttons, custom widgets, images and charts that have no text). OSWorld offers it as an input option on Linux; WebArena offers it for web pages.

DOM / HTML

The DOM (Document Object Model) is the browser's live tree of the page's HTML. It contains everything, including hidden elements, scripts and styling, so it is precise but very long. Real pages often run to hundreds of thousands of tokens, so harnesses prune or filter it first. Mind2Web's original baseline, for example, used a small ranking model to shortlist candidate elements before asking an LLM to pick one.

Set-of-marks (SoM)

Set-of-Mark prompting (Yang et al., Microsoft, 2023) draws numbered labels on top of detected regions of an image so that a vision model can refer to regions by number. Web agents borrowed the idea: the harness finds interactive elements (from the DOM or a detector), draws a box and number on each, and gives the model both the annotated screenshot and a text list. VisualWebArena's strongest baseline used this style of input. The model then says click [7] and the harness maps 7 back to an element, which removes pixel-level grounding error at the cost of depending on the element detector.

Action spaces: how the agent acts

The action space is the set of commands the agent may issue. It is chosen together with the observation space, and it sits on a spectrum from low-level and universal to high-level and specific.

• Pixel coordinates. Click, double-click, drag, move, scroll at (x, y), type text, press keys. Universal but the agent must ground every target. The resolution matters: if the harness downscales a 2560x1440 screen to 1280x720 for the model, it must scale coordinates back up, and a bug here silently tanks scores.
• Element IDs. Refer to an element from the tree or set-of-marks list. Much easier, but only as good as the element extraction (hidden elements, iframes, and canvas widgets are frequent gaps).
• Keyboard shortcuts. Often the most reliable way to do something (ctrl+L to focus the address bar, ctrl+S to save). Strong agents use them heavily.
• API tools. Some harnesses give the agent a shell, a Python interpreter or site-specific functions alongside the GUI. OSWorld tasks can often be solved from a terminal; this is allowed in some setups and not in others, which is one reason leaderboards list the agent framework next to the score.

How commercial computer-use tools work

Both Anthropic and OpenAI ship computer-use tools that follow the same screenshot-and-action loop. The important thing to understand for evals is the division of labor: the model only proposes actions. Your harness owns the computer, executes each action, captures the new screenshot and sends it back. So your harness is also where you reset environments, log trajectories, enforce step budgets and run graders.

At a high level, according to the vendors' documentation (checked September 2026):

• Anthropic's computer use tool. You declare the computer tool in the Messages API. Claude replies with tool_use blocks naming actions such as screenshot, left_click (with a coordinate), type, key, scroll, left_click_drag, mouse_move, wait and zoom (capture a region at full resolution). Your application runs each action in order and returns tool_result blocks (images for screenshots and zooms). The tool has been versioned over time (for example computer_20250124, and the current computer_toolset_20260801). The docs recommend a dedicated VM or container with minimal privileges, a domain allowlist, no sensitive credentials, and human confirmation for consequential actions, and note that Anthropic runs classifiers on screenshots to flag possible prompt injections.
• OpenAI's computer use tool. In the Responses API the model emits a computer_call containing actions such as click, double_click, drag, move, scroll, keypress, type, wait and screenshot. You execute them and reply with a computer_call_output carrying the new screenshot. OpenAI's guidance likewise says to isolate the environment, use allowlists, confirm consequential actions, and "verify actual outcomes rather than relying solely on model assertions", which is exactly the evaluator lesson of this module.

Offline vs online evaluation

This is the single most important design split for web and GUI agent evals.

• Offline (static) evaluation records a human doing the task once, saving a snapshot of the page at each step and the action they took. At test time the agent is shown snapshot k plus the correct history of earlier actions and asked to predict action k. You compare its prediction with the human's. Mind2Web is the canonical example.
• Online (interactive) evaluation puts the agent in a live, running environment and lets it act freely until it stops. You then inspect the end state (was the order placed? does the file exist?) or the answer. WebArena, OSWorld and AndroidWorld are online.

Worked example: Mind2Web step metrics

Mind2Web (Deng et al., 2023) defines four metrics. Element accuracy: did the agent pick an acceptable element? Operation F1: a token-level F1 on the operation (for Click it equals accuracy; for Type and Select it also checks the typed value). Step success rate: a step succeeds only if both element and operation are correct. Task success rate: a task succeeds only if every step succeeded. Step-wise metrics are macro-averaged across tasks (average within each task first, then across tasks).

Notice the gap between step and task success. With 10 steps at 90% each, independent errors would give roughly 0.9¹⁰ ≈ 35% task success. That is why step-level numbers flatter agents, and why online end-to-end evaluation is the headline metric for release decisions. A 2025 follow-up, Online-Mind2Web, runs agents on live websites and uses an LLM judge ("WebJudge"), precisely to measure the end-to-end picture.

The benchmark landscape

Sources: the papers linked at the end, via the course research notes (September 2026). Current top scores move monthly and aggregator sites disagree, so check each official leaderboard rather than trusting a number in a course.

WebArena up close

WebArena (Zhou et al., 2023) solved the reproducibility problem of live websites by self-hosting realistic open-source web apps in Docker and filling them with data: an e-commerce store, its admin panel, a Reddit-like forum, GitLab, and an OpenStreetMap-based map. Each of its 812 tasks is a JSON config with a natural-language intent, a start URL, login state, and an eval block. In the public config file, the evaluator types break down as 325 string_match, 282 program_html, 129 url_match + program_html, 66 url_match, and 10 string_match + url_match (counted from config_files/test.raw.json in the WebArena repo). A real example of a state check:

{
  "sites": ["gitlab"],
  "intent": "Post \"lgtm\" for the merge request related to semantic HTML post in a11yproject/a11yproject.com project",
  "eval": {
    "eval_types": ["program_html"],
    "program_html": [{
      "url": "__GITLAB__/a11yproject/a11yproject.com/-/merge_requests/1531",
      "locator": "document.querySelector('[id=\"notes-list\"').lastElementChild.querySelector('.timeline-discussion-body').outerText",
      "required_contents": {"exact_match": "lgtm"}
    }]
  }
}

After the agent stops, the evaluator navigates to the merge request, runs that JavaScript locator to read the last comment, and checks it equals "lgtm". It does not care how the agent got there. It does, however, care about exact text: "LGTM!" would fail. This is the typical trade-off of programmatic checks: precise and cheap, but occasionally too strict (false negatives) or too loose (false positives). Audits found enough of both that cleaned-up versions such as WebArena-Verified exist.

OSWorld up close

OSWorld (Xie et al., 2024) does the same thing for whole computers. Each task runs in a real Ubuntu virtual machine restored from a snapshot, the task's config commands prepare the scene (download a file, open an app, change a setting), the agent works, and then the task's evaluator fetches some piece of final state from the VM and compares it with an expectation. The 369 tasks split across domains as Chrome 46, GIMP 26, LibreOffice Calc 47, Impress 47, Writer 23, multi-app 101, OS 24, Thunderbird 15, VLC 17 and VS Code 23 (counted from evaluation_examples/test_all.json). Some tasks are deliberately infeasible (the requested feature does not exist) and the correct behavior is to say so, which tests whether the agent will bluff.

Here is a real OSWorld task, lightly trimmed. Notice the setup deliberately sets the value away from the goal, so an agent that does nothing fails:

{
  "id": "bedcedc4-4d72-425e-ad62-21960b11fe0d",
  "snapshot": "os",
  "instruction": "Could you set the 'Dim screen when inactive' to off in setting?",
  "config": [
    {"type": "execute",
     "parameters": {"command": ["gsettings", "set", "org.gnome.desktop.session", "idle-delay", "300"]}}
  ],
  "related_apps": ["os"],
  "evaluator": {
    "func": ["exact_match", "exact_match"],
    "conj": "or",
    "result": [
      {"type": "vm_command_line",
       "command": ["gsettings", "get", "org.gnome.desktop.session", "idle-delay"]},
      {"type": "vm_command_line",
       "command": ["gsettings", "get", "org.gnome.settings-daemon.plugins.power", "idle-dim"]}
    ],
    "expected": [
      {"type": "rule", "rules": {"expected": "uint32 0\n"}},
      {"type": "rule", "rules": {"expected": "false\n"}}
    ]
  },
  "possibility_of_env_change": "low"
}

The evaluator accepts either of two settings ("conj": "or") because two different GNOME settings reasonably satisfy "dim screen off". That is an evaluator author anticipating alternative solutions. Below is a compact re-implementation of the idea (a sketch, not OSWorld's actual code) so you can build your own tasks in the same style:

import json, subprocess

# --- metric functions: (actual_result, rules) -> score in [0, 1] -----------
def exact_match(result: str, rules: dict) -> float:
    return float(result == rules["expected"])

def check_include_exclude(result: str, rules: dict) -> float:
    inc = all(s in result for s in rules.get("include", []))
    exc = not any(s in result for s in rules.get("exclude", []))
    return float(inc and exc)

def file_contains_csv_row(result: str, rules: dict) -> float:
    # result is the text of a file pulled from the VM
    return float(rules["row"] in result.splitlines())

METRICS = {f.__name__: f for f in (exact_match, check_include_exclude, file_contains_csv_row)}

# --- getters: fetch a piece of FINAL state from the VM -----------------------
def get_result(vm, spec: dict) -> str:
    if spec["type"] == "vm_command_line":
        return vm.run(spec["command"], shell=spec.get("shell", False))   # stdout
    if spec["type"] == "vm_file":
        return vm.read_file(spec["path"])
    raise ValueError(f"unknown getter {spec['type']}")

def evaluate(task: dict, vm) -> float:
    ev = task["evaluator"]
    if ev["func"] == "infeasible":                 # correct answer is to refuse
        return float(vm.agent_said_infeasible)
    as_list = lambda x: x if isinstance(x, list) else [x]
    funcs, results, expected = map(as_list, (ev["func"], ev["result"], ev["expected"]))
    scores = []
    for fn, r_spec, e_spec in zip(funcs, results, expected):
        try:
            actual = get_result(vm, r_spec)
        except Exception:
            actual = None                          # missing file etc. counts as fail
        scores.append(0.0 if actual is None else METRICS[fn](actual, e_spec["rules"]))
    return max(scores) if ev.get("conj") == "or" else min(scores)   # "and" by default

# usage, after the agent loop finished:
# task = json.load(open("tasks/bedcedc4.json")); print(evaluate(task, vm))

AndroidWorld: dynamic, parameterized tasks

AndroidWorld (Rawles et al., 2024) runs an Android emulator with 20 real apps and 116 task templates. Each template has parameters (a contact name, a date, an expense amount) that are randomly generated at run time, which yields millions of distinct instances. That fights memorization, because the exact task you are tested on did not exist before the run, and it lets you run the same template with several seeds to measure reliability. Success is computed from device state such as app databases and files, not from screenshots.

Live-web benchmarks: WebVoyager and BrowseComp

WebVoyager runs agents on 15 real, live websites. Since you cannot query Amazon's database, success is judged by humans, or automatically by GPT-4V looking at the agent's screenshots and answer, which agreed with humans about 85% of the time in the paper. Live sites change daily: prices move, flights sell out, some tasks become impossible, and bot protections appear. Two runs a month apart are not the same benchmark.

BrowseComp flips the problem: 1,266 questions whose answers are short and easy to verify but extremely hard to find, so the agent must browse persistently and creatively. Because only the answer is graded, it avoids end-state checking entirely. Its answers are encrypted in the public release to reduce training-data contamination.

Grounding evaluation

A GUI grounding benchmark isolates step 3 of the loop. Each sample is a screenshot, an instruction ("click the bold button"), and the target element's bounding box. The model outputs a point; it is correct if the point falls inside the box. ScreenSpot (from the SeeClick paper, 2024) has 1,272 such instructions across mobile, desktop and web and is now largely saturated above 90%. ScreenSpot-Pro (2025) moves to 1,581 instructions on high-resolution professional software (CAD, IDEs, creative and scientific tools) where targets are tiny, and at release the best model scored only about 18.9%.

from collections import defaultdict

def in_box(point, box):
    """point = (x, y); box = (x0, y0, x1, y1), all in ORIGINAL screenshot pixels."""
    x, y = point
    x0, y0, x1, y1 = box
    return x0 <= x <= x1 and y0 <= y <= y1

def to_pixels(pred, img_w, img_h, model_w=None, model_h=None):
    """Models may answer in normalised [0,1] coords, or in the pixels of a DOWNSCALED
    image. Convert back to original pixels, or you will score garbage."""
    x, y = pred
    if 0 <= x <= 1 and 0 <= y <= 1:                  # normalised output
        return x * img_w, y * img_h
    if model_w:                                        # model saw a resized image
        return x * img_w / model_w, y * img_h / model_h
    return x, y

def grounding_accuracy(samples, predict):
    """samples: dicts with image, instruction, bbox, size=(w, h), group (e.g. 'text'/'icon', app).
    predict: fn(image, instruction) -> (x, y) or None (unparseable)."""
    hits, totals = defaultdict(int), defaultdict(int)
    for s in samples:
        pred = predict(s["image"], s["instruction"])
        ok = pred is not None and in_box(to_pixels(pred, *s["size"]), s["bbox"])
        for key in ("all", s["group"]):                # report overall AND per slice
            totals[key] += 1
            hits[key] += ok
    return {k: hits[k] / totals[k] for k in totals}

# e.g. {'all': 0.61, 'icon': 0.42, 'text': 0.80}: icons are usually the hard slice

Always report slices (text vs icon targets, app, platform). A single average hides the fact that most failures cluster on small unlabelled icons. And test your coordinate conversion on a few hand-checked samples before you trust any number.

Evaluator designs and how they lie

For online tasks, someone has to decide success from the final situation. There are four families of evaluator, and each fails in characteristic ways. Recall from Module 03: a false positive is when the grader says "pass" but the task was not really done; a false negative is when it says "fail" but the task was done.

Validating your evaluator

Treat an evaluator like any other classifier and measure it (see Module 13). For a new task suite: (1) write a known-good solution for each task and confirm the checker passes it; (2) run the do-nothing agent and confirm it fails (the OSWorld example above deliberately sets the starting state away from the goal for this reason); (3) hand-label 50 to 100 real agent trajectories and compute the checker's false-positive and false-negative rates; (4) re-examine every task where strong agents fail in the same way, since that is often a broken task, not a hard one.

A mini web-agent eval harness

Here is the whole idea in about 100 lines using Playwright for Python: reset a local site to a golden database, run a (stubbed) agent in a step loop with a budget, record the trajectory, then check the end state directly in the database. Swap the stub for a real model call and you have the skeleton of WebArena.

"""mini_webarena.py: reset -> run agent loop -> check end state.
Assumes a small local shop app on :8000 that stores data in shop.db (SQLite)."""
import shutil, sqlite3, subprocess, time, json
from dataclasses import dataclass, field
from typing import Callable
from playwright.sync_api import sync_playwright

BASE = "http://localhost:8000"
DB, GOLDEN_DB = "site/shop.db", "site/shop.golden.db"

@dataclass
class Task:
    id: str
    intent: str
    start_path: str
    check: Callable[[str | None], bool]      # end-state checker, gets final answer
    max_steps: int = 15

# ---------- environment control ------------------------------------------------
def reset_site() -> subprocess.Popen:
    """Stop server, restore golden DB, restart. Full restart avoids cached state."""
    subprocess.run(["pkill", "-f", "uvicorn shop:app"], check=False)
    shutil.copyfile(GOLDEN_DB, DB)
    srv = subprocess.Popen(["uvicorn", "shop:app", "--port", "8000"], cwd="site")
    time.sleep(1.5)                               # real harness: poll a /health URL
    return srv

def observe(page) -> dict:
    return {
        "url": page.url,
        "screenshot": page.screenshot(),                   # PNG bytes
        "aria": page.locator("body").aria_snapshot(),      # accessibility tree as YAML
    }

def execute(page, a: dict) -> None:
    t = a["type"]
    if t == "click_xy":     page.mouse.click(a["x"], a["y"])
    elif t == "click_role": page.get_by_role(a["role"], name=a["name"]).first.click()
    elif t == "type":       page.keyboard.type(a["text"])
    elif t == "key":        page.keyboard.press(a["key"])          # e.g. "Enter"
    elif t == "scroll":     page.mouse.wheel(0, a.get("dy", 600))
    elif t == "goto":       page.goto(BASE + a["path"])
    else: raise ValueError(f"unknown action {t}")

# ---------- the agent (stub: replace with a model call) ------------------------
def scripted_agent(intent: str, obs: dict, history: list) -> dict:
    plan = [
        {"type": "click_role", "role": "searchbox", "name": "Search"},
        {"type": "type", "text": "red mug"},
        {"type": "key", "key": "Enter"},
        {"type": "click_role", "role": "button", "name": "Add Red mug to cart"},
        {"type": "stop", "answer": None},
    ]
    return plan[min(len(history), len(plan) - 1)]

# ---------- one episode ---------------------------------------------------------
def run_task(task: Task, policy, browser) -> dict:
    srv = reset_site()
    ctx = browser.new_context(viewport={"width": 1280, "height": 800})   # fresh cookies
    page = ctx.new_page()
    page.goto(BASE + task.start_path)
    traj, answer, t0 = [], None, time.time()
    try:
        for step in range(task.max_steps):
            obs = observe(page)
            action = policy(task.intent, obs, traj)
            traj.append({"step": step, "url": obs["url"], "action": action})
            if action["type"] == "stop":
                answer = action.get("answer")
                break
            try:
                execute(page, action)
                page.wait_for_load_state("networkidle", timeout=5000)
            except Exception as e:                  # a failed action is data, not a crash
                traj[-1]["error"] = repr(e)
        passed = task.check(answer)                 # grade the BACKEND, not the page
    finally:
        ctx.close(); srv.terminate()
    return {"task": task.id, "passed": passed, "steps": len(traj),
            "seconds": round(time.time() - t0, 1), "trajectory": traj}

# ---------- end-state checkers ---------------------------------------------------
def cart_has_exactly_one_red_mug(answer) -> bool:
    con = sqlite3.connect(DB)
    rows = con.execute("SELECT sku, qty FROM cart_items WHERE user_id = 1").fetchall()
    orders = con.execute("SELECT COUNT(*) FROM orders").fetchone()[0]
    con.close()
    # goal check AND collateral check: no surprise purchases
    return rows == [("MUG-RED", 1)] and orders == 0

TASKS = [Task("cart-001", "Add the cheapest red mug to my cart", "/", cart_has_exactly_one_red_mug)]

if __name__ == "__main__":
    with sync_playwright() as p:
        browser = p.chromium.launch()
        results = [run_task(t, scripted_agent, browser) for t in TASKS for _ in range(3)]  # 3 trials
        browser.close()
    print(json.dumps([{k: r[k] for k in ("task", "passed", "steps", "seconds")} for r in results], indent=1))
    print("success rate:", sum(r["passed"] for r in results) / len(results))

Design points worth copying: the environment is reset before every trial, not after (a crash can't poison the next run); each trial gets a fresh browser context (no leaked cookies); failed actions are logged, not fatal; the checker reads the database, not the agent's claim; and each task runs several trials so you can report pass^k (Module 04).

Environment engineering

For online benchmarks, most of the engineering effort is not the agent or the grader: it is making thousands of identical, isolated computers appear and disappear on demand.

Safety: injection, destruction, confirmation

Web and computer-use agents read untrusted content all day. Any web page, email, PDF or file name can contain text that tries to take over the agent. This is indirect prompt injection: the attacker is not the user, but someone who controls content the agent reads.

AgentDojo (Debenedetti et al., ETH Zurich, 2024) is a widely used framework for exactly this. It provides 97 realistic tool-using tasks (email client, e-banking, travel booking and so on) and 629 security test cases, and evaluates both task utility and whether injected instructions succeed. It found that capable models already fail many tasks without attacks and that existing attacks break some, but not all, security properties. AgentDojo is tool-based rather than GUI-based, but the method carries over: take your normal web tasks, plant an injection in a page the agent must visit (a product review, a sponsored banner like the one in the set-of-marks figure, an email body), and check the end state for both the user's goal and the attacker's goal. WebArena has a safety-focused descendant, ST-WebAgentBench, for policy compliance in web tasks.

Destructive actions and confirmation policies

Beyond injection, agents make honest mistakes with real consequences: emptying the trash, force-pushing to GitLab, submitting a payment. Both vendors' docs recommend asking a human before consequential actions. That creates something new to evaluate: a confirmation policy. Build tasks where the right behavior is to pause and ask ("Delete all files in Downloads": do you mean all 2,000?), and tasks where asking would be annoying ("Close this tab"). Score two error types: under-asking (took an irreversible action without confirmation) and over-asking (interrupted the user for trivia). In your harness, intercept actions such as submitting a checkout form or running rm, record whether the agent requested confirmation, and have the simulated user decline sometimes, to check the agent respects "no".

Putting it together: a checklist for a GUI agent eval

1. Define the interface. Observation space (screenshot / tree / DOM / SoM), action space (pixels / IDs / keys / tools), resolution, step and time budget. Write them in the report.
2. Build a resettable world. Self-host where possible. Snapshot, freeze the clock, pre-load logins.
3. Write tasks with checkers first. For each task: a setup that puts the world away from the goal, a state-based goal check, a collateral check, and a known-good solution that passes.
4. Include hard cases. Infeasible tasks, ambiguous tasks that need a question, injection pages, destructive-action traps.
5. Validate the evaluator. Do-nothing agent fails, reference solution passes, human-label a sample to estimate false-positive and false-negative rates.
6. Run multiple trials. Report success rate with confidence intervals and pass^k; slice by app and task type.
7. Read trajectories. Tag failures as perception, grounding, reasoning, environment flake, or evaluator bug (Module 12).

Check yourself

Sources and further reading

• Zhou et al., WebArena: A Realistic Web Environment for Building Autonomous Agents (2023); task configs in the WebArena repo.
• Koh et al., VisualWebArena (2024).
• Deng et al., Mind2Web: Towards a Generalist Agent for the Web (2023); Online-Mind2Web (2025).
• He et al., WebVoyager (2024).
• Drouin et al., WorkArena (2024); WorkArena++ (2024).
• Wei et al., BrowseComp (OpenAI, 2025).
• Xie et al., OSWorld (2024); OSWorld site and OSWorld-Verified leaderboard; task files in the OSWorld repo.
• Bonatti et al., Windows Agent Arena (Microsoft, 2024).
• Rawles et al., AndroidWorld (Google DeepMind, 2024).
• Cheng et al., SeeClick / ScreenSpot (2024); Li et al., ScreenSpot-Pro (2025).
• Yang et al., Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V (2023).
• Debenedetti et al., AgentDojo (2024).
• Anthropic, Computer use tool documentation.
• OpenAI, Computer use guide.
• Playwright for Python, ARIA snapshots and Mouse API.