cmux/docs/remote-daemon-spec.md

# Remote SSH Living Spec

Last updated: March 12, 2026
Tracking issue: https://github.com/manaflow-ai/cmux/issues/151
Primary PR: https://github.com/manaflow-ai/cmux/pull/239
CLI relay PR: https://github.com/manaflow-ai/cmux/pull/374

This document is the working source of truth for:
1. what is implemented now
2. what is intentionally temporary
3. what must be built next

## 1. Document Type

This is a **living implementation spec** (also called an **execution spec**): a spec-level document with status tracking (`DONE`, `IN PROGRESS`, `TODO`) and acceptance tests.

## 2. Objective

`cmux ssh` should provide:
1. durable remote terminals with reconnect/reuse
2. browser traffic that egresses from the remote host via proxying
3. tmux-style PTY resize semantics (`smallest screen wins`)

## 3. Current State (Implemented)

### 3.1 Remote Workspace + Reconnect UX
- `DONE` `cmux ssh` creates remote-tagged workspaces and does not require `--name`.
- `DONE` scoped shell niceties are applied only for `cmux ssh` launches.
- `DONE` context menu actions exist for remote workspaces (`Reconnect Workspace(s)`, `Disconnect Workspace(s)`).
- `DONE` socket API includes `workspace.remote.reconnect`.

### 3.2 Bootstrap + Daemon
- `DONE` local app probes remote platform, verifies a release-pinned `cmuxd-remote` artifact by embedded manifest SHA-256, uploads it when missing, and runs `serve --stdio`.
- `DONE` daemon `hello` handshake is enforced.
- `DONE` daemon now exposes proxy stream RPC (`proxy.open`, `proxy.close`, `proxy.write`, `proxy.read`).
- `DONE` local proxy broker now tunnels SOCKS5/CONNECT traffic over daemon stream RPC instead of `ssh -D`.
- `DONE` daemon now exposes session resize-coordinator RPC (`session.open`, `session.attach`, `session.resize`, `session.detach`, `session.status`, `session.close`).
- `DONE` transport-level proxy failures now escalate from broker retry to full daemon re-bootstrap/reconnect in the session controller.
- `DONE` SOCKS handshake parsing now preserves pipelined post-connect payload bytes instead of dropping request-prefix bytes.
- `DONE` `workspace.remote.configure.local_proxy_port` exists as an internal deterministic test hook for bind-conflict regression coverage.
- `DONE` bootstrap/probe failures surface actionable details.
- `DONE` bootstrap installs `~/.cmux/bin/cmux` wrapper (also tries `/usr/local/bin/cmux`) so `cmux` is available in PATH on the remote.

### 3.5 CLI Relay (Running cmux Commands From Remote)
- `DONE` `cmuxd-remote` includes a table-driven CLI relay (`cli` subcommand) that maps CLI args to v1 text or v2 JSON-RPC messages.
- `DONE` busybox-style argv[0] detection: when invoked as `cmux` via wrapper/symlink, auto-dispatches to CLI relay.
- `DONE` background `ssh -N -R 127.0.0.1:PORT:127.0.0.1:LOCAL_RELAY_PORT` process reverse-forwards a TCP port to a dedicated authenticated local relay server. Uses TCP instead of Unix socket forwarding because many servers have `AllowStreamLocalForwarding` disabled.
- `DONE` relay process uses `ControlPath=none` (avoids ControlMaster multiplexing and inherited `RemoteForward` directives) and `ExitOnForwardFailure=no` (inherited forwards from user ssh config failing should not kill the relay).
- `DONE` relay address written to `~/.cmux/socket_addr` on the remote with a 3s delay after the relay process starts, giving SSH time to establish the `-R` forward.
- `DONE` Go CLI re-reads `~/.cmux/socket_addr` on each TCP retry to pick up updated relay ports when multiple workspaces overwrite the file.
- `DONE` `cmux ssh` startup exports session-local `CMUX_SOCKET_PATH=127.0.0.1:<relay_port>` so parallel sessions pin to their own relay instead of racing on shared socket_addr.
- `DONE` relay startup writes `~/.cmux/relay/<relay_port>.daemon_path`; remote `cmux` wrapper uses this to select the right daemon binary per session, including mixed local cmux versions.
- `DONE` relay startup writes `~/.cmux/relay/<relay_port>.auth` with a relay ID and token; the local relay requires HMAC-SHA256 challenge-response before forwarding any command to the real local socket.
- `DONE` ephemeral port range (49152-65535) filtered from probe results to exclude relay ports from other workspaces.
- `DONE` multi-workspace port conflict detection uses TCP connect check (`isLoopbackPortReachable`) so ports already forwarded by another workspace are silently skipped instead of flagged as conflicts.
- `DONE` orphaned relay SSH processes from previous app sessions are cleaned up before starting a new relay.

### 3.6 Artifact Trust
- `DONE` release and nightly workflows publish `cmuxd-remote` assets for `darwin/linux × arm64/amd64`.
- `DONE` release and nightly apps embed a compact `CMUXRemoteDaemonManifestJSON` in `Info.plist` with exact asset URLs and SHA-256 digests.
- `DONE` `cmux remote-daemon-status` exposes the current manifest entry, local cache verification state, release download command, and GitHub attestation verification command.

### 3.3 Error Surfacing
- `DONE` remote errors are surfaced in sidebar status + logs + notifications.
- `DONE` reconnect retry count/time is included in surfaced error text (for example, `retry 1 in 4s`).

### 3.4 Removed Temporary Behavior
- `DONE` removed remote listening-port probe loop and per-port SSH `-L` mirroring.
- `DONE` remote browser routing now uses a single shared local proxy endpoint instead of detected-port mirroring.
- `DONE` remote status now includes structured proxy metadata (`remote.proxy`) and `proxy_unavailable` error code when proxy setup fails.

## 4. Target Architecture (No Port Mirroring)

### 4.1 Browser Networking Path
1. `DONE` one local proxy endpoint is created per SSH transport/session key (not per detected port).
2. `DONE` endpoint is provided by a local broker that supports SOCKS5 + HTTP CONNECT and tunnels via daemon stream RPC.
3. `DONE` browser panels in remote workspaces are auto-wired to the workspace proxy endpoint.
4. `DONE` browser panels in local workspaces are not force-proxied.
5. `DONE` identical SSH transports share one endpoint via a transport-scoped broker.

### 4.2 WKWebView Wiring
1. `DONE` use workspace-scoped `WKWebsiteDataStore(forIdentifier:)`.
2. `DONE` apply workspace/browser scoped `proxyConfigurations`.
3. `DONE` prefer SOCKS5 proxy config.
4. `DONE` keep HTTP CONNECT proxy config as fallback.
5. `DONE` re-apply proxy config on reconnect/state updates.

### 4.3 Remote Daemon + Transport
1. `DONE` `cmuxd-remote` now supports proxy stream RPC (`proxy.open`, `proxy.close`, `proxy.write`, `proxy.read`).
2. `DONE` local side now runs a shared local broker that serves SOCKS5/CONNECT and tunnels each stream over persistent daemon stdio RPC.
3. `DONE` removed remote service-port discovery/probing from browser routing path.

### 4.4 Explicit Non-Goal
1. Automatic mirroring of every remote listening port to local loopback is not a goal for browser support.

## 5. PTY Resize Semantics (tmux-style)

### 5.1 Core Rule
For each session with multiple attachments, the effective PTY size is:
1. `cols = min(cols_i over attached clients)`
2. `rows = min(rows_i over attached clients)`

This is the `smallest screen wins` rule.

### 5.2 State Model
Per session track:
1. set of active attachments `{attachment_id -> cols, rows, updated_at}`
2. effective size currently applied to PTY
3. last-known size when temporarily unattached

### 5.3 Recompute Triggers
Recompute effective size on:
1. attachment create
2. attachment detach
3. resize event from any attachment
4. reconnect reattach

### 5.4 Correctness Requirements
1. Never shrink history because of UI relayout noise; only PTY viewport changes.
2. On reconnect, reuse persisted session and recompute from active attachments.
3. If no attachments remain, keep last-known PTY size (do not force 80x24 reset).

## 6. Milestones (Living Status)

| ID | Milestone | Status | Notes |
|---|---|---|---|
| M-001 | `cmux ssh` workspace creation + metadata + optional `--name` | DONE | Covered by `tests_v2/test_ssh_remote_cli_metadata.py` |
| M-002 | Remote bootstrap/upload/start + hello handshake | DONE | Includes daemon capability handshake + status surfacing |
| M-003 | Reconnect/disconnect UX + API + improved error surfacing | DONE | Includes retry count in surfaced errors |
| M-004 | Docker e2e for bootstrap/reconnect shell niceties | DONE | Docker suites validate proxy-path bootstrap and reconnect behavior |
| M-004b | CLI relay: run cmux commands from within SSH sessions | DONE | Reverse TCP forward + Go CLI relay + bootstrap wrapper |
| M-005 | Remove automatic remote port mirroring path | DONE | `WorkspaceRemoteSessionController` now uses one shared daemon-backed proxy endpoint |
| M-006 | Transport-scoped local proxy broker (SOCKS5 + CONNECT) | DONE | Identical SSH transports now reuse one local proxy endpoint |
| M-007 | Remote proxy stream RPC in `cmuxd-remote` | DONE | `proxy.open/close/write/read` implemented |
| M-008 | WebView proxy auto-wiring for remote workspaces | DONE | Workspace-scoped `WKWebsiteDataStore.proxyConfigurations` wiring is active |
| M-009 | PTY resize coordinator (`smallest screen wins`) | DONE | Daemon session RPC now tracks attachments and applies min cols/rows semantics with unit tests |
| M-010 | Resize + proxy reconnect e2e test suites | DONE | `tests_v2/test_ssh_remote_docker_forwarding.py` validates HTTP/websocket egress plus SOCKS pipelined-payload handling; `tests_v2/test_ssh_remote_docker_reconnect.py` verifies reconnect recovery and repeats SOCKS pipelined-payload checks after host restart; `tests_v2/test_ssh_remote_proxy_bind_conflict.py` validates structured `proxy_unavailable` bind-conflict surfacing and `local_proxy_port` status retention under bind conflict; `tests_v2/test_ssh_remote_daemon_resize_stdio.py` validates session resize semantics over real stdio RPC process boundaries; `tests_v2/test_ssh_remote_cli_metadata.py` validates `workspace.remote.configure` numeric-string compatibility, explicit `null` clear semantics (including `workspace.remote.status` reflection), strict `port`/`local_proxy_port` validation (bounds/type), case-insensitive SSH option override precedence for StrictHostKeyChecking/control-socket keys, and `local_proxy_port` payload echo for deterministic bind-conflict test hook behavior |

## 7. Acceptance Test Matrix (With Status)

### 7.1 Terminal + Reconnect

| ID | Scenario | Status |
|---|---|---|
| T-001 | baseline remote connect | DONE |
| T-002 | identical host reuse semantics | DONE |
| T-003 | no `--name` | DONE |
| T-004 | reconnect API success/error paths | DONE |
| T-005 | retry count visible in daemon error detail | DONE |

### 7.2 CLI Relay

| ID | Scenario | Status |
|---|---|---|
| C-001 | `cmux ping` from remote session | DONE |
| C-002 | `cmux list-workspaces --json` from remote | DONE |
| C-003 | `cmux new-workspace` from remote | DONE |
| C-004 | `cmux rpc system.capabilities` passthrough | DONE |
| C-005 | TCP retry handles relay not yet established | DONE |
| C-006 | multi-workspace port conflict silent skip | DONE |
| C-007 | ephemeral port filtering excludes relay ports | DONE |

### 7.3 Browser Proxy (Target)

| ID | Scenario | Status |
|---|---|---|
| W-001 | remote workspace browser auto-proxied | DONE |
| W-002 | browser egress equals remote network path | DONE |
| W-003 | websocket via SOCKS5/CONNECT through remote daemon | DONE |
| W-004 | reconnect restores browser proxy path automatically | DONE |
| W-005 | local proxy bind conflict yields structured `proxy_unavailable` | DONE |
| W-006 | proxy transport failure triggers daemon re-bootstrap and recovers after host recreation | DONE |
| W-007 | SOCKS greeting/connect + immediate pipelined payload in same write remains intact | DONE |

### 7.4 Resize

| ID | Scenario | Status |
|---|---|---|
| RZ-001 | two attachments, smallest wins | DONE |
| RZ-002 | grow one attachment, PTY stays bounded by smallest | DONE |
| RZ-003 | detach smallest, PTY expands to next smallest | DONE |
| RZ-004 | reconnect preserves session + applies recomputed size | DONE |
| RZ-005 | daemon stdio RPC round-trip enforces resize semantics end-to-end | DONE |

## 8. Removal Checklist (Port Mirroring)

Before declaring browser proxying complete:
1. `DONE` remove remote port probe loop and `-L` auto-forward orchestration
2. `DONE` remove mirror-specific routing behavior as default remote behavior
3. `DONE` replace mirroring docker assertions with proxy egress assertions
4. `DONE` keep optional explicit user-driven forwarding out of this path; no automatic mirroring remains in browser routing

## 9. Open Decisions

1. Proxy auth policy for local broker (`none` vs optional credentials).
2. Reconnect backoff profile and max retry budget.

## 10. Socket API Contract Notes

### 10.1 `workspace.remote.configure` Port Fields
1. `port` and `local_proxy_port` accept integer values and numeric strings.
2. Explicit `null` clears each field.
3. Out-of-range values and invalid types (for example booleans/non-numeric strings/fractional numbers) return `invalid_params`.
4. `local_proxy_port` is an internal deterministic test hook to force local bind conflicts in regression coverage.

### 10.2 SSH Option Precedence
1. `StrictHostKeyChecking` default (`accept-new`) is only injected when no user override is present.
2. Control-socket defaults (`ControlMaster`, `ControlPersist`, `ControlPath`) are only injected when missing.
3. SSH option key matching is case-insensitive for precedence checks in both CLI-built commands and remote configure payloads.

### 10.3 SSH Docker E2E Harness Knobs
1. `CMUX_SSH_TEST_DOCKER_HOST` sets the SSH destination host/IP used by docker-backed SSH fixtures (default `127.0.0.1`).
2. `CMUX_SSH_TEST_DOCKER_BIND_ADDR` sets the bind address used in fixture container publish mappings (default `127.0.0.1`).
3. Defaults preserve loopback behavior on a single host; override both when docker runs on a different host (for example VM -> host OrbStack).