Browser Agent Commands

AI agents running in Codemux terminals can control the embedded browser programmatically using CLI commands or the socket API.

Detect Codemux

Check if you're inside Codemux before using browser commands:

if [ -n "$CODEMUX_WORKSPACE_ID" ]; then
  # Inside Codemux — browser commands available
fi

Environment variables set by Codemux:

• CODEMUX_WORKSPACE_ID — Current workspace ID
• CODEMUX_SURFACE_ID — Current terminal surface ID

Setup

Create a browser pane first (only needed once per workspace):

codemux browser create

CLI Commands

Navigate

codemux browser open <url>

Opens a URL in the browser pane. Always use this instead of xdg-open or open.

Get Accessibility Snapshot

codemux browser snapshot [browser_id]
codemux browser snapshot --dom          # DOM-based element list with selectors

Returns the page's accessibility tree. Use this to discover elements before interacting.

Click

codemux browser click <selector> [browser_id]

Clicks an element matching the CSS selector.

Fill Input

codemux browser fill <selector> <text> [browser_id]

Fills an input field with text.

Screenshot

codemux browser screenshot [browser_id]

Takes a screenshot and returns it as base64-encoded PNG.

Console Logs

codemux browser console-logs [browser_id]

Returns captured console output from the page.

Coordinate-Based Commands (Tier 2)

Each of these has both a codemux browser ... CLI form and an MCP tool. They use pixel coordinates instead of CSS selectors. Useful for canvas elements, iframes, shadow DOM, or when selectors aren't available. Agents typically get coordinates from a screenshot.

Click at Coordinates

codemux browser click-at 150 300
codemux browser click-at 150 300 --click-type right   # left | right | middle

// MCP tool: browser_click_at
{"x": 150, "y": 300, "click_type": "left"}

Moves the mouse along a human-like Bezier curve to (x, y), then clicks.

Type at Coordinates

codemux browser type-at "hello world" --x 150 --y 300

// MCP tool: browser_type_at
{"x": 150, "y": 300, "text": "hello world"}

Clicks at the coordinates, then types the text with per-character delays.

Scroll at Coordinates

codemux browser scroll-at 400 300 --direction down --amount 3

// MCP tool: browser_scroll_at
{"x": 400, "y": 300, "direction": "down", "amount": 3}

Scrolls at the specified position. direction is up / down / left / right (default down); amount is 1–10 wheel ticks (default 3). There are no deltaX / deltaY fields — passing them is silently ignored and you get the default 3-tick down-scroll.

Key Press

codemux browser key-press Enter

// MCP tool: browser_key_press
{"key": "Enter"}

Sends a keyboard event. Supports keys like Enter, Tab, Escape, ArrowDown, etc.

Drag

codemux browser drag 100 200 300 200

// MCP tool: browser_drag
{"start_x": 100, "start_y": 200, "end_x": 300, "end_y": 200}

All four coordinates are snake_case and required. The camelCase spelling (startX…) fails schema validation.

Drags from start to end coordinates with human-like mouse movement.

OS-Level Commands (Tier 3 — Stealth)

These use ydotool to generate kernel-level input events that are indistinguishable from human interaction. Requires ydotool + ydotoold, headed browser mode, and Hyprland.

OS Click

codemux browser click-os 150 300

// MCP tool: browser_click_os
{"x": 150, "y": 300}

OS Type

codemux browser type-os "hello world" --x 150 --y 300

// MCP tool: browser_type_os
{"text": "hello world", "x": 150, "y": 300}

Socket API

For programmatic control, send JSON commands over the Unix socket at $XDG_RUNTIME_DIR/codemux.sock:

echo '{"command":"browser_automation","params":{"browser_id":"default","action":{"kind":"open_url","url":"https://example.com"}}}' | nc -U $XDG_RUNTIME_DIR/codemux.sock

Available Actions

Snapshot naming is inverted between surfaces. The CLI's bare codemux browser snapshot returns the accessibility tree (use --dom for the DOM list), while the MCP tool browser_snapshot is DOM-based and browser_accessibility_snapshot is the ARIA tree. Check which surface you are on before parsing the output.

Common Workflows

Testing a Web App

npm run dev &
codemux browser open http://localhost:3000
codemux browser snapshot
codemux browser fill "#search" "test query"
codemux browser click "#submit"
codemux browser snapshot

Debugging JavaScript Errors

codemux browser console-logs
codemux browser snapshot

Tips

1. Always get a snapshot before interacting — know what elements exist
2. Prefer explicit CSS selectors over guessing
3. Check console logs when behavior is unexpected
4. The browser_id parameter is optional — defaults to the active browser