Vercel's agent-desktop: the AI agent that reads the screen unseen

Vercel Labs, the lab of the company known for website hosting and Next.js, has released agent-desktop: an open source tool that lets an AI agent read and use a computer’s software on Mac, Windows and Linux. The idea sounds modest. Yet it addresses one of the most concrete obstacles to AI in business: most everyday software was never designed to be driven by a machine.

What the tool does

Until now, most agents that “use a computer” work like an intern behind glass: they take a screenshot, guess where the button is, move the mouse, click, then take another screenshot to check. It is slow, expensive in compute, and fragile at the slightest change of theme or resolution. Meanwhile, the machine is blocked for its user.

agent-desktop takes another route. Every operating system already describes its windows for assistive technologies: the AX API on Mac, UI Automation on Windows, AT-SPI on Linux. That description, the accessibility tree, says there is a “Downloads” window here, a list there, a “Send” button over there. The tool reads it, returns it as structured data and gives every usable element a reference. The agent no longer guesses a pixel: it asks to click the button’s reference, reads the screen again to check, and repeats. It is the loop of any agent, which we describe in our article on the agent loop, applied to desktop software.

The tool has about sixty commands: read a window, click, type, tick, scroll, open and close applications, read the clipboard, handle notifications. Actions run in the background by default, with no pointer movement; some still need the real keyboard and mouse, and the documentation lists, system by system, what works in the background and what does not. It also plugs in as an MCP server, the protocol that lets an assistant call tools, which makes it usable by most agents on the market, whatever model sits behind them. It is released under the Apache 2.0 licence, so it can be used in a commercial project.

Who it is for

Teams juggling software that does not talk to each other. An old management tool on one side, a CRM on the other, and someone copying orders from one to the other every morning. When that software has no API, classic automation stops there. An agent that reads the screen can do the back-and-forth, and the person who did the copying keeps what needs judgement.

Roles that live in desktop applications. Accounting, logistics, administration, engineering: many business tools are installed software, not websites. Browser agents, now multiplying, cannot reach them. agent-desktop fills that gap.

Teams testing their own software. Checking that a desktop application behaves after an update is repetitive work. An agent that walks the interface by its structure does it without depending on a button’s colour or the screen size.

Vendors who took care of accessibility. This is the most unexpected effect: the tool only sees what the software describes. Software that works well for a visually impaired person becomes easy to automate at the same time. Accessibility stops being a separate obligation and becomes a selling point.

What you can do with it

A few realistic uses, within reach of a team of a few dozen people:

• export a report every morning from an application with no API, and drop it where the team reads it;
• carry orders from an old management tool into the CRM, with a human review of doubtful cases;
• fill in an administrative form from a spreadsheet, line by line, while the team does something else;
• sort the notifications on a shared workstation and only surface the ones that matter.

In every case, the first question stays the same: does the software have an API? If so, going through the screen is a detour, and as we wrote about automations that do not need AI, the right tool is often the simplest one. agent-desktop is mostly useful where there is no other way in, the case covered in our guide to automating software that has no API.

The limits to know

The tool’s strength is also its limit: it cannot see an interface that publishes no accessibility tree. Old software, some fully custom-drawn tools, graphic areas such as a plan or a drawn chart remain out of reach. For the web content of applications built on Chromium or Electron, like many modern messaging tools, it hands over to its cousin agent-browser.

Installation takes one command, but you then have to grant the program accessibility permission on Mac, or run it in the user’s signed-in session on Windows and Linux. That is reassuring: a program able to click anywhere should not install without anyone knowing. The documentation also warns that screen and clipboard reads can contain private data. An agent that can click anything must run on an account with limited rights, with a record of what it does.

Finally, the tool is young (first version published in February 2026) and moving fast. It deserves a trial on a specific case before going into production.

A door to the future of agentic AI?

Yes, for a simple reason. AI agents first settled where machines already had a grip: APIs, then browsers. The desktop remained the last territory, where a large share of business software still lives. By relying on a layer every system already provides, agent-desktop gives agents clean access to that territory, without asking vendors to change anything.

What is taking shape is an agent working on the same computer as you, in the background, without taking your mouse. Not a replacement, more a colleague you hand data entry and checks to while you keep what needs judgement. Several agents could one day share the same machine; the tool already handles sessions, but its documentation does not yet say how two agents touching the same application coordinate. That is one of the developments to watch.

For a business, the question is therefore not “should every workstation have an agent?” but “which back-and-forth between programs costs the team the most time, and which have no way in other than the screen?”. That is exactly what we look at during an audit: which tools have an API, which can only be automated through the screen, and whether the agent is worth its cost.