# RUNBOOK — deploy the rendezvous-relay to VPS `8.138.1.192` (Alibaba Cloud, mainland) > **Staging, durable single-tenant** (PLAN_RELAY_PHASE1 §0). Real Postgres + Redis (Docker Compose on > the VPS), real Let's Encrypt TLS on `:443` against an **already ICP-filed** domain, agent dials OUT > from the operator's laptop. Dev in-process CA signer is accepted for staging (KMS → Phase 2). > > Everything below is a **staging template** — replace every `` and confirm each path on > the box. Config is **env-only**; no host/port/secret is hardcoded in code or units. Two independent trust chains — never cross-wire them: | Chain | Issued by | Protects | Script | |---|---|---|---| | **Public web PKI** | Let's Encrypt | browser `:443` WSS (`TLS_CERT_PATH`) | `issue-tls-cert.sh` | | **Private enrollment CA** | your offline root | agent mTLS + relay agent-server cert (`AGENT_CA_*`) | `gen-agent-ca.sh` | And one shared key: the **P5 capability keypair** — control-plane **signs**, relay + admin API **verify** (`gen-capability-key.sh`). `CAPABILITY_SIGN_PUBKEY_B64` (CP) and `RELAY_AUTH_VERIFY_PUBKEY` (relay) are the **same public key**, two encodings. --- ## 0. Prerequisites (on `8.138.1.192`) ```bash # Docker Engine + Compose plugin curl -fsSL https://get.docker.com | sh sudo systemctl enable --now docker # Node 20 (satisfies control-plane>=18, relay-run>=20, agent>=18) via NodeSource curl -fsSL https://deb.nodesource.com/setup_20.x | sudo bash - sudo apt-get install -y nodejs git openssl # Clone (this runbook assumes /opt/web-terminal; adjust the systemd --prefix if you change it) sudo git clone /opt/web-terminal cd /opt/web-terminal npm ci # root deps npm --prefix control-plane ci npm --prefix relay-run ci npm --prefix relay-web ci # Dedicated non-root service user + secret dir sudo useradd --system --home /opt/web-terminal relay || true sudo mkdir -p /etc/relay ``` **DNS (do this first — LE HTTP-01 and the browser both need it):** create an A-record `.` → `8.138.1.192`. The domain must be **ICP-filed** (mainland Aliyun blocks `:80/:443` on unfiled domains; if unfiled, you must use DNS-01 in step 2). --- ## 1. Bring up Postgres + Redis (Docker Compose, loopback-only) ```bash cd /opt/web-terminal cp deploy/.env.example deploy/.env # gitignored; fill POSTGRES_PASSWORD (strong random) docker compose --env-file deploy/.env -f deploy/docker-compose.yml up -d docker compose -f deploy/docker-compose.yml ps # both healthy; bound to 127.0.0.1 only ``` Both services publish to `127.0.0.1` only — never the public interface (they are the relay's private state). --- ## 2. Generate the CA, the capability key, and the TLS cert ```bash # (a) PRIVATE enrollment CA (agent mTLS) — NOT Let's Encrypt. sudo AGENT_FACING_HOST="." \ AGENT_FACING_IP="8.138.1.192" \ RELAY_TRUST_DOMAIN="" \ bash deploy/scripts/gen-agent-ca.sh # writes /etc/relay/ca/{root,intermediate,agent-ca.*,relay-agent-server.*} # (b) P5 capability keypair — prints BOTH public-key encodings; private key -> 0600 file. sudo bash deploy/scripts/gen-capability-key.sh # -> note CAPABILITY_SIGN_PUBKEY_B64 (base64) and RELAY_AUTH_VERIFY_PUBKEY (base64url); same key. # (c) PUBLIC TLS cert for the browser :443 (Let's Encrypt). # http-01 needs inbound :80 open DURING issuance (step 4); dns-01 needs no inbound port. sudo BASE_DOMAIN="" SUBDOMAIN="" ACME_EMAIL="" \ ACME_METHOD="http-01" \ TLS_CERT_PATH="/etc/relay/tls/fullchain.pem" TLS_KEY_PATH="/etc/relay/tls/privkey.pem" \ bash deploy/scripts/issue-tls-cert.sh sudo chown -R relay:relay /etc/relay ``` > **Staging CA seam (A-wave, not this task):** the control-plane's dev in-process signer must sign > agent leaves with `/etc/relay/ca/intermediate.key.pem` so they chain to `agent-ca.bundle.pem`. > `CA_INTERMEDIATE_KMS_KEY_REF=dev-local` selects the dev signer; point it at that key. Phase 2 = KMS. > After first boot, move `/etc/relay/ca/root.key.pem` **off** the host (offline anchor). --- ## 3. Fill the two env files Copy the relevant blocks of `deploy/.env.example` into two 0600 files. **The linchpin:** `CAPABILITY_SIGN_PUBKEY_B64` (control-plane) and `RELAY_AUTH_VERIFY_PUBKEY` (relay) are the SAME key from step 2b — paste both. ```bash sudo install -m 600 -o relay -g relay /dev/null /etc/relay/control-plane.env sudo install -m 600 -o relay -g relay /dev/null /etc/relay/relay.env ``` `/etc/relay/control-plane.env` (P3): ```ini PG_URL=postgres://relay:@127.0.0.1:5432/relay REDIS_URL=redis://127.0.0.1:6379 CAPABILITY_SIGN_PUBKEY_B64= CA_INTERMEDIATE_KMS_KEY_REF=dev-local CA_INTERMEDIATE_CERT_PATH=/etc/relay/ca/intermediate.cert.pem NODE_MTLS_TRUST_BUNDLE_PATH=/etc/relay/ca/agent-ca.bundle.pem BASE_DOMAIN= CP_BIND_HOST=127.0.0.1 CP_BIND_PORT=8080 # HEARTBEAT_TTL_SEC / PAIRING_TTL_SEC / PAIRING_MAX_REDEEM_ATTEMPTS use defaults if unset ``` `/etc/relay/relay.env` (P1 data-plane): ```ini BASE_DOMAIN= BIND_HOST=0.0.0.0 BIND_PORT=443 TLS_CERT_PATH=/etc/relay/tls/fullchain.pem TLS_KEY_PATH=/etc/relay/tls/privkey.pem AGENT_BIND_PORT=8444 AGENT_CA_CERT_PATH=/etc/relay/ca/agent-ca.cert.pem AGENT_CA_CHAIN_PATH=/etc/relay/ca/agent-ca.bundle.pem RELAY_NODE_ID=relay-1 RELAY_AUTH_VERIFY_PUBKEY= RELAY_TRUST_DOMAIN= PG_URL=postgres://relay:@127.0.0.1:5432/relay REDIS_URL=redis://127.0.0.1:6379 ``` --- ## 4. Open the Aliyun security group (inbound) Open **only** these, in the ECS instance's security group: | Port | Who | Note | |---|---|---| | `443/tcp` | browsers (WSS) | permanent | | `8444/tcp` (`AGENT_BIND_PORT`) | agents (mTLS) | permanent | | `80/tcp` | Let's Encrypt HTTP-01 | **temporary** — only during cert issue/renew; skip entirely if using DNS-01 | **Keep loopback-only (never open):** Postgres `5432`, Redis `6379`, control-plane admin `8080`. Reach the admin API from your laptop via SSH tunnel: `ssh -L 8080:127.0.0.1:8080 root@8.138.1.192`. --- ## 5. Build the browser bundle ```bash cd /opt/web-terminal npm --prefix relay-web run build # emits relay-web/public/build/, served same-origin by D1 ``` --- ## 6. Enable + start both units ```bash sudo cp deploy/systemd/relay-control-plane.service deploy/systemd/relay-data-plane.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now relay-control-plane.service # migrates PG, listens on 127.0.0.1:8080 sudo systemctl enable --now relay-data-plane.service # :443 browser WSS + :8444 agent mTLS sudo systemctl status relay-control-plane.service relay-data-plane.service journalctl -u relay-data-plane.service -f # watch for bind + verify-key load ``` --- ## 7. Create an account + pairing code (control-plane admin) The admin API is **deny-by-default**: `POST /accounts` and `POST /accounts/:id/pairing-codes` require a **capability token** with the `manage` right whose **`aud` equals `BASE_DOMAIN`** (control-plane `authz.ts` / `main.ts` `expectedAud: env.baseDomain`). `accountId` is taken ONLY from the verified token (INV3) — never from the body. Mint the token with the **capability PRIVATE key** from step 2b (the B6 token-mint owns the signing helper; it imports `/etc/relay/capability/capability-sign.key.pem`). Token claims: `rights=['manage']`, `aud=`. Send it as `Authorization: Bearer `. Bootstrap has a two-token nuance (grounded in `provision.ts`): 1. `POST /accounts` checks `manage` only (no account-ownership check) → mint a `manage` token, create the account, read back its `id`. 2. `POST /accounts/:id/pairing-codes` additionally enforces `token.sub == :id` (own-account). Mint a second `manage` token with `sub=`, then request the pairing code. Over the SSH tunnel from step 4: ```bash # 1) create account curl -sS -X POST http://127.0.0.1:8080/accounts \ -H "Authorization: Bearer " \ -H 'content-type: application/json' -d '{"plan":"personal"}' # -> { "id": "", ... } # 2) pairing code (token.sub must equal ) curl -sS -X POST http://127.0.0.1:8080/accounts//pairing-codes \ -H "Authorization: Bearer " # -> { "code": "", ... } (single-use, TTL 600s default) ``` --- ## 8. On the LAPTOP — build the agent, pair, run The agent dials OUT; nothing inbound is opened on the laptop. ```bash cd /agent npm ci && npm run build # emits dist/cli.js (C1) export RELAY_URL="wss://.:8444" # AGENT_BIND_PORT export ENROLL_URL="https://./enroll" export HOST_ID="" export SUBDOMAIN="" export LOCAL_TARGET_URL="ws://127.0.0.1:3000" # the base web-terminal app export STATE_DIR="$HOME/.web-terminal-agent" node dist/cli.js pair # redeems the code -> SPIFFE mTLS cert + hostContentSecret node dist/cli.js run # dials the relay, holds the mux tunnel ``` `pair` redeems the single-use code and stores the enrolled SPIFFE cert under `STATE_DIR`; `run` opens the outbound mTLS tunnel and keeps it alive (heartbeat/backoff). --- ## 9. Browser — log in and click through to the shell 1. Open `https://.` (served same-origin as the WSS endpoint). 2. Operator login (B6) mints a short-lived connect capability token. 3. Pick the host → the relay splices browser ↔ agent. The relay only sees **ciphertext** (INV2); the E2E is browser ↔ agent. --- ## 10. Verify restart-safety + revocation teardown ```bash # Restart-safety (INV7): bounce the relay while a shell is open — the PTY and host registration survive. sudo systemctl restart relay-data-plane.service # the browser auto-reconnects; the session is still there (state is in PG/Redis, not the process). # Revocation teardown (INV12): revoke the host and watch the tunnel drop within budget. curl -sS -X DELETE http://127.0.0.1:8080/hosts/ \ -H "Authorization: Bearer " # -> control-plane publishes a KillSignal on Redis relay:revocations; the data-plane subscriber # closes the spliced stream (browser WS closes 4403). Measure revoke -> close latency. ``` --- ## Troubleshooting | Symptom | Meaning | Fix | |---|---|---| | Browser WS closes **1013** | `WS_TRY_LATER` — **no agent tunnel** for that host (agent offline / not dialed) | Start the agent (`node dist/cli.js run`); check `RELAY_URL` host+port; confirm `:8444` open; watch `journalctl -u relay-data-plane`. | | Browser WS closes **401** | **bad Origin** (CSWSH exact-match failed) — the `Origin` header is not in the relay's `allowedOrigins` | Open via `https://.` (not the raw IP, not `http:`, no stray port). `allowedOrigins` is the port-less `https://.` on `:443`; must match EXACTLY. Verify `BASE_DOMAIN` in `relay.env`. | | Agent mTLS closes **4401** | cert chains to the CA but the **pubkey is not enrolled** in the host registry (INV14 registry-gated) | The host was never enrolled or was revoked. Re-run `pair ` with a fresh code; confirm the CP signed the leaf with `/etc/relay/ca/intermediate.key.pem` (chains to `agent-ca.bundle.pem`). | | Browser WS closes **4403** | `WS_REVOKED` — the host/session was revoked (expected after step 10) | Re-enroll the host to reconnect. | | CP fails to boot: "CAPABILITY_SIGN_PUBKEY_B64 must decode to 32 bytes" | wrong key encoding in `control-plane.env` | Use the **base64** value (not base64url) from step 2b for the CP; base64url is the relay's `RELAY_AUTH_VERIFY_PUBKEY`. | | Admin API returns **401** on `POST /accounts` | missing/expired/foreign-`aud` capability token | Mint a `manage` token with `aud=`, signed by the step-2b private key; send `Authorization: Bearer`. | | Admin API returns **403** on `/pairing-codes` | token lacks `manage` right, or `token.sub != :id` | Mint with `rights:['manage']` and `sub=` (own-account rule). | | LE issuance fails (timeout/connection) | HTTP-01 `:80` not reachable, or domain not ICP-filed | Open `:80` in the security group for issuance, or switch `ACME_METHOD=dns-01`; confirm the A-record + ICP filing. | | `docker compose ps` unhealthy | PG/Redis not up | `docker compose -f deploy/docker-compose.yml logs`; confirm `POSTGRES_PASSWORD` set in `deploy/.env`. | | Mixed-content / `ws:` blocked in browser | page is HTTPS but WS scheme resolved to `ws:` | Serve over HTTPS; the client follows page scheme (`wss:` on HTTPS, M6). Confirm the LE cert is valid for the FQDN. |