What is a computer-use agent, and what can you do with one?

The loop, and how it differs from API agents

A computer-use agent runs a loop that looks surprisingly human. The model receives a screenshot of a screen, decides what to do, and returns an action: click at these coordinates, type this text, scroll, press a key. Your application executes the action in a real browser or desktop, captures a new screenshot, and sends it back. The loop repeats until the task is done or something stops it. Anthropic's documentation calls this the agent loop; OpenAI notes that you provide the environment and execute the model's requests; Google requires the same client-side execution environment and returns an explanation field called intent with each suggested action.

The phrase agentic computer control covers two integration styles, and the difference is worth knowing before you read a tutorial. In one, the model returns structured actions that your code translates into mouse and keyboard input. In the other, the model writes code, for example Playwright in a browser or PyAutoGUI on a desktop, and your sandbox runs that code and returns the output, including screenshots. OpenAI's documentation recommends the code-execution style for GPT-6 Astra and keeps the structured computer tool as an alternative; both end up in the same screenshot-decide-act loop.

What separates this from the API agents most developers already use is the surface it operates on. An API agent calls functions with defined contracts. A computer-use agent works through the same interface a person uses, which is why it can operate software that has no API at all, and also why its mistakes are visible as wrong clicks in your real applications rather than as a failed JSON call.

What you can actually do with one

The documented use cases are refreshingly mundane, which is a good sign for usefulness. Microsoft's documentation lists automated data entry, invoice processing, and data extraction, and sums the pitch up as: if a person can use an app or website, computer use can too. Google lists filling forms and repetitive data entry, automated testing of web applications and user flows, and research that gathers information across sites such as prices and reviews. OpenAI lists filling out forms, testing user flows, and completing tasks through an application's interface.

The pattern underneath all of those: workflows trapped behind a GUI. Legacy desktop software, vendor portals with no API, internal admin tools, and websites that forbid or do not offer automation. That is exactly where a computer-use agent earns its keep, because the alternative is brittle scripts tied to pixel positions or a human doing the clicks.

It is equally important to say what this is not. If a workflow has a good API, calling the API remains faster, cheaper, and more reliable, and nothing in the vendor documentation argues otherwise. These agents are also not fire-and-forget in 2026: the documentation we read is full of environment hygiene, confirmation gates, and verification steps, for reasons the next sections cover. A reasonable mental model is a capable contractor who works through your screen and needs supervision around anything consequential.

• Good fits: no-API software, form and data-entry workflows, cross-site research, UI-level test automation.
• Bad fits: anything with a solid API, high-volume deterministic jobs, and workflows involving sensitive accounts until you have isolation in place.
• Reality check: every documented deployment pattern includes supervision and verification, not autonomy at any cost.

The options in 2026

Anthropic ships computer use as a toolset on the Claude API. One entry in the tools array adds 17 member tools such as screenshot, left_click, type, and zoom, and your application runs each call in an environment you control. Supported models per the documentation include Fable 5, Mythos 5, Opus 4.8 and 5, and Sonnet 5, with Amazon Bedrock and Microsoft Foundry available in beta, and a Docker-based reference implementation for getting started. Anthropic's engineering blog adds an unusually candid set of integration notes, including that screenshot resolution and coordinate scaling are where most click accuracy is won or lost.

Google's computer use tool runs on Gemini models, with Gemini 3.8 Flash documented as the recommended model. It supports browser, mobile, and desktop environments, and Gemini 3.x adds per-action intent explanations plus configurable safety policies and opt-in screenshot scanning for hidden prompt-injection instructions. Google's docs point at a Docker-based reference implementation for the execution environment and a Playwright-based handler example.

OpenAI supports computer use on its models with two integration paths: code execution, which it recommends for GPT-6 Astra, and the structured computer tool. A public sample app implements both browser (Playwright) and desktop (PyAutoGUI) workflows. Microsoft is the no-code on-ramp: Copilot Studio exposes computer use as a configurable tool where you describe the task in plain language and pick a model, currently including OpenAI's Computer-Using Agent and Anthropic's Claude Sonnet 4.5 with newer Anthropic models marked experimental in that platform.

The open-source branch is real and moving quickly. Cua describes itself as infrastructure for scaling computer-use with drivers, cross-OS fleets, sandboxes, and benchmarks, and it is MIT licensed with about 22,500 stars and commits the hour of our check. browser-use is the browser-focused library, MIT licensed, around 114,000 stars, with a macOS harness and a benchmark project in the same family. We have not run any of these; they are listed as options to evaluate, and the sandbox vendors build is the part that matters most for safety.

The safety rules every vendor writes down

Read the four major documentation sets and a consensus emerges, which is rare enough to be worth repeating. First, the screen is untrusted input. OpenAI's guidance states it plainly: text in a page, document, or tool result cannot grant permission or override the user's instructions. Anthropic warns that Claude can in some circumstances follow commands found in content even when they conflict with your instructions, and runs classifiers that flag suspected prompt injection in screenshots and steer the model to ask for confirmation. Google offers opt-in prompt-injection scanning of screenshots. Our own answer on prompt injection covers the general defenses; here the point is that a computer-use agent gives injected instructions a path to your actual desktop.

Second, isolate the environment. Google tells developers to run the agent in a sandboxed VM or container. OpenAI says to restrict it to an isolated browser or VM with an allow list of sites and actions. Anthropic says to isolate the model from sensitive data and actions. Microsoft's version runs on a machine you configure for the agent, with credentials either maker-provided or end-user provided, and a documented warning that sharing an agent with maker-provided credentials lets anyone who uses it act with the original author's access. That is the same blast-radius logic as sandboxing a coding agent, applied to a whole desktop.

Third, gate and bound the run. OpenAI lists confirming consequential actions, keeping humans in control of purchases, data transmission, and destructive changes, and treating typed sensitive information as transmission. Microsoft sends a human-supervision email when it detects potentially harmful instructions, with a response time limit. Fourth, verify outcomes instead of trusting the final message, and log what the agent actually did, which Google's guidance makes explicit. None of this is optional advice in the vendor documents; it is the expected deployment shape.

• Treat everything on the screen as untrusted input that cannot grant permission.
• Run the agent in an isolated VM or container with an allow list, not on your logged-in desktop.
• Confirm consequential actions: purchases, sending data, typing credentials, irreversible changes.
• Bound each run with step, time, and cost limits, and verify the real outcome afterward.

What it costs and what it takes to run

Budget for tokens, infrastructure, and integration time, in that order. Anthropic's documentation says the computer use toolset adds about 4,500 input tokens of overhead to every request once member tools are attached, and screenshots returned in tool results bill as image input on top. Those screenshots are not incidental; they are the loop's whole sensory channel, which means token bills scale with how often the model looks at the screen.

The no-code route has the most legible pricing: Copilot Studio bills computer use at 5 Copilot credits per step, 15 with a premium model, and the docs walk through a four-step form example costing 20 or 60 credits. If you are inside the Microsoft ecosystem, that is close to a per-task price you can forecast.

The hidden cost is the environment. OpenAI states that you provide the environment and execute the model's requests; Google requires you to implement the client-side execution environment; Anthropic's best-practices post is largely about the unglamorous engineering of resolution, coordinate scaling, and screenshot handling. Expect real work on action handlers, sandbox images, session state, logging, and a test harness before the first useful task completes, and measure success rate on your actual tasks rather than trusting a demo.

Where it breaks today

The honest failure modes are known and documented. Long-horizon reliability is the common thread in community discussion; a September 2026 Hacker News thread framed controllability, not raw capability, as the bottleneck for computer-use agents, and the recent Show HN cluster we found was mostly builders selling sandboxes and managed environments for exactly this reason. Treat those as community signals rather than results; they are directionally useful but not benchmarks.

Concrete brittleness is also documented. Google's guidance notes that unexpected pop-ups, notifications, or layout changes can confuse the model and recommends starting each task from a clean state. Microsoft's docs flag whole app categories, including Electron, Java, Unity, games, command-line interfaces, and virtualized environments such as Citrix, as possibly unsupported. Click accuracy needs engineering; screenshots in the wrong coordinate space produce consistently wrong clicks. Some vendor channels are still beta, and model availability differs by platform, so the same task can behave differently in Anthropic's API, Google's, OpenAI's, and Microsoft's versions.

The largest risk is social rather than technical: an agent that operates your real browser profile, email, or admin console has the keys to everything those sessions can touch. That is why every vendor's first instruction is isolation rather than a model setting. RepoRadar has not run any of these agents end to end, so treat this article as a map of documented capability and documented caution, not as hands-on test results.

A useful next action

Try the smallest safe version this week. Pick one reference implementation, Anthropic's or Google's Docker-based demos are the most self-contained starting points, and run it inside a container or throwaway VM rather than on your working machine. Use a fresh browser profile or a test account, never a profile that is logged into your real services, and write down the sites and actions the agent is allowed to touch before the first run.

Then measure one thing honestly: give the agent five to ten real tasks from your own workflow, count how many complete correctly without intervention, and note where the failures cluster. That success rate, plus the token and credit math from the previous section, is the number that decides whether computer use is a tool you deploy or a demo you watch. If it holds up, escalate to a supervised production pattern: isolated environment, human confirmation on consequential steps, and logging on. If it does not, you have spent an afternoon, not a quarter.