Computer Use: Driving the Desktop GUI

Hermes Agent can drive your desktop — clicking, typing, scrolling, and dragging — in the background on macOS, Windows, and Linux. Your cursor doesn't move, keyboard focus doesn't change, and your virtual desktops or Spaces don't switch on you. You and the agent co-work on the same machine, and unlike most computer-use integrations this works with any tool-capable model — Claude, GPT, Gemini, or an open model on a local OpenAI-compatible endpoint.

How It Works

The built-in computer_use toolset is the recommended Hermes integration. It speaks MCP over stdio to cua-driver, an open-source background computer-use driver. Each platform uses its appropriate accessibility-plus-input stack:

• macOS — AX (private SkyLight SPIs) and pid-scoped event posting, no cursor warp.
• Windows — UIAutomation with SendInput + PostMessage, no focus steal.
• Linux — AT-SPI on X11 and Wayland, with XTest (X11) or virtual-keyboard (Wayland) input.

The result is the same everywhere: the agent can read the accessibility tree of any visible window and post synthesized events without bringing it to front, switching virtual desktops, or moving the real OS cursor.

Enabling Computer Use

Fresh installs likely already have the driver — the Hermes installer (install.sh / install.ps1) pre-installs cua-driver best-effort. Enabling is a config flip:

• hermes tools → pick 🖱️ Computer Use, which installs the driver automatically if missing;
• the Dashboard or desktop app → toggle the Computer Use toolset (install kicks off in the background);
• manual fallback: hermes computer-use install, then verify with hermes computer-use status.

Then start a session with the toolset enabled: hermes -t computer_use chat, or add computer_use to your enabled toolsets in ~/.hermes/config.yaml.

Permission Modes

Hermes maps its approval UX onto cua-driver's immutable runtime modes:

• standard (default) — normal Hermes approvals; Cua stops at its protected boundary.
• bounded — a private daemon where you review and approve a capability manifest once at launch.
• unrestricted — via --yolo or approvals.mode: off; one explicit Hermes risk acceptance, no runtime prompts.

Bounded mode is ideal for repeatable automation — cron jobs or scheduled research against an authenticated app. A capability manifest names the apps, browser profile kinds, allowed origins, and typed tools the session may use; anything outside it fails closed inside cua-driver.

The Agent Cursor and Safety

When the agent acts, you'll see a tinted overlay cursor glide across the screen to where each click, type, or scroll lands — the real OS cursor never moves. Each Hermes run declares a public session name, so concurrent runs and subagents get distinct cursors.

Hermes applies multi-layer guardrails: destructive actions require approval via CLI dialog or messaging-platform buttons; hard-blocked key combos (empty trash, force delete, lock screen, log out) and type patterns (curl | bash, sudo rm -rf /, fork bombs) are blocked; and the system prompt forbids clicking permission dialogs, typing passwords, or following instructions embedded in screenshots. If anything goes wrong, hermes computer-use doctor runs cua-driver's structured health report and prints a per-check matrix — the single fastest way to find why an action isn't working.

Example Workflow

Given "find my latest email from Stripe and summarise what they want me to do," the agent's plan looks the same on every platform:

1. computer_use(action="capture", mode="som", app="Mail") — numbered screenshot of the app.
2. computer_use(action="click", element=14) — click the search field.
3. computer_use(action="type", text="from:stripe").
4. computer_use(action="key", keys="return", capture_after=True) — submit and get the new screenshot.
5. Click the top result, read the body, summarise.

All the while your cursor stays where you left it and the email app never comes to front. Screenshots taken during control are normally internal, but every capture saves a shareable copy — on Telegram, Discord, Desktop, and other attachment-capable surfaces you can simply ask for the image to be delivered as a native attachment.

Key Takeaways

• computer_use drives your desktop in the background on macOS, Windows, and Linux with no cursor movement or focus stealing.
• It works with any tool-capable model and speaks MCP over stdio to the open-source cua-driver.
• Standard, bounded, and unrestricted permission modes map Hermes' approval UX onto cua-driver's immutable runtime modes.
• hermes computer-use doctor is the first triage stop for any action that isn't working.