web-terminal/deploy/RUNBOOK.md

RUNBOOK — deploy the rendezvous-relay to VPS 8.138.1.192 (Alibaba Cloud, mainland)

Staging, durable single-tenant (PLAN_RELAY_PHASE1 §0). Real Postgres + Redis (Docker Compose on the VPS), real Let's Encrypt TLS on :443 against an already ICP-filed domain, agent dials OUT from the operator's laptop. Dev in-process CA signer is accepted for staging (KMS → Phase 2).

Everything below is a staging template — replace every <placeholder> and confirm each path on the box. Config is env-only; no host/port/secret is hardcoded in code or units.

Two independent trust chains — never cross-wire them:

Chain	Issued by	Protects	Script
Public web PKI	Let's Encrypt	browser :443 WSS (TLS_CERT_PATH)	issue-tls-cert.sh
Private enrollment CA	your offline root	agent mTLS + relay agent-server cert (AGENT_CA_*)	gen-agent-ca.sh

And one shared key: the P5 capability keypair — control-plane signs, relay + admin API verify (gen-capability-key.sh). CAPABILITY_SIGN_PUBKEY_B64 (CP) and RELAY_AUTH_VERIFY_PUBKEY (relay) are the same public key, two encodings.

---

0. Prerequisites (on 8.138.1.192)

# Docker Engine + Compose plugin
curl -fsSL https://get.docker.com | sh
sudo systemctl enable --now docker

# Node 20 (satisfies control-plane>=18, relay-run>=20, agent>=18) via NodeSource
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo bash -
sudo apt-get install -y nodejs git openssl

# Clone (this runbook assumes /opt/web-terminal; adjust the systemd --prefix if you change it)
sudo git clone <REPO_URL> /opt/web-terminal
cd /opt/web-terminal
npm ci                          # root deps
npm --prefix control-plane ci
npm --prefix relay-run ci
npm --prefix relay-web ci

# Dedicated non-root service user + secret dir
sudo useradd --system --home /opt/web-terminal relay || true
sudo mkdir -p /etc/relay

DNS (do this first — LE HTTP-01 and the browser both need it): create an A-record <SUBDOMAIN>.<BASE_DOMAIN> → 8.138.1.192. The domain must be ICP-filed (mainland Aliyun blocks :80/:443 on unfiled domains; if unfiled, you must use DNS-01 in step 2).

---

1. Bring up Postgres + Redis (Docker Compose, loopback-only)

cd /opt/web-terminal
cp deploy/.env.example deploy/.env        # gitignored; fill POSTGRES_PASSWORD (strong random)
docker compose --env-file deploy/.env -f deploy/docker-compose.yml up -d
docker compose -f deploy/docker-compose.yml ps      # both healthy; bound to 127.0.0.1 only

Both services publish to 127.0.0.1 only — never the public interface (they are the relay's private state).

---

2. Generate the CA, the capability key, and the TLS cert

# (a) PRIVATE enrollment CA (agent mTLS) — NOT Let's Encrypt.
sudo AGENT_FACING_HOST="<SUBDOMAIN>.<BASE_DOMAIN>" \
     AGENT_FACING_IP="8.138.1.192" \
     RELAY_TRUST_DOMAIN="<RELAY_TRUST_DOMAIN>" \
     bash deploy/scripts/gen-agent-ca.sh
# writes /etc/relay/ca/{root,intermediate,agent-ca.*,relay-agent-server.*}

# (b) P5 capability keypair — prints BOTH public-key encodings; private key -> 0600 file.
sudo bash deploy/scripts/gen-capability-key.sh
#   -> note CAPABILITY_SIGN_PUBKEY_B64 (base64) and RELAY_AUTH_VERIFY_PUBKEY (base64url); same key.

# (c) PUBLIC TLS cert for the browser :443 (Let's Encrypt).
#     http-01 needs inbound :80 open DURING issuance (step 4); dns-01 needs no inbound port.
sudo BASE_DOMAIN="<BASE_DOMAIN>" SUBDOMAIN="<SUBDOMAIN>" ACME_EMAIL="<you@example.com>" \
     ACME_METHOD="http-01" \
     TLS_CERT_PATH="/etc/relay/tls/fullchain.pem" TLS_KEY_PATH="/etc/relay/tls/privkey.pem" \
     bash deploy/scripts/issue-tls-cert.sh

sudo chown -R relay:relay /etc/relay

Staging CA seam (A-wave, not this task): the control-plane's dev in-process signer must sign agent leaves with /etc/relay/ca/intermediate.key.pem so they chain to agent-ca.bundle.pem. CA_INTERMEDIATE_KMS_KEY_REF=dev-local selects the dev signer; point it at that key. Phase 2 = KMS. After first boot, move /etc/relay/ca/root.key.pem off the host (offline anchor).

---

3. Fill the two env files

Copy the relevant blocks of deploy/.env.example into two 0600 files. The linchpin: CAPABILITY_SIGN_PUBKEY_B64 (control-plane) and RELAY_AUTH_VERIFY_PUBKEY (relay) are the SAME key from step 2b — paste both.

sudo install -m 600 -o relay -g relay /dev/null /etc/relay/control-plane.env
sudo install -m 600 -o relay -g relay /dev/null /etc/relay/relay.env

/etc/relay/control-plane.env (P3):

PG_URL=postgres://relay:<POSTGRES_PASSWORD>@127.0.0.1:5432/relay
REDIS_URL=redis://127.0.0.1:6379
CAPABILITY_SIGN_PUBKEY_B64=<base64 from step 2b>
CA_INTERMEDIATE_KMS_KEY_REF=dev-local
CA_INTERMEDIATE_CERT_PATH=/etc/relay/ca/intermediate.cert.pem
NODE_MTLS_TRUST_BUNDLE_PATH=/etc/relay/ca/agent-ca.bundle.pem
BASE_DOMAIN=<BASE_DOMAIN>
CP_BIND_HOST=127.0.0.1
CP_BIND_PORT=8080
# HEARTBEAT_TTL_SEC / PAIRING_TTL_SEC / PAIRING_MAX_REDEEM_ATTEMPTS use defaults if unset

/etc/relay/relay.env (P1 data-plane):

BASE_DOMAIN=<BASE_DOMAIN>
BIND_HOST=0.0.0.0
BIND_PORT=443
TLS_CERT_PATH=/etc/relay/tls/fullchain.pem
TLS_KEY_PATH=/etc/relay/tls/privkey.pem
AGENT_BIND_PORT=8444
AGENT_CA_CERT_PATH=/etc/relay/ca/agent-ca.cert.pem
AGENT_CA_CHAIN_PATH=/etc/relay/ca/agent-ca.bundle.pem
RELAY_NODE_ID=relay-1
RELAY_AUTH_VERIFY_PUBKEY=<base64url from step 2b — SAME key as CAPABILITY_SIGN_PUBKEY_B64>
RELAY_TRUST_DOMAIN=<RELAY_TRUST_DOMAIN>
PG_URL=postgres://relay:<POSTGRES_PASSWORD>@127.0.0.1:5432/relay
REDIS_URL=redis://127.0.0.1:6379

---

4. Open the Aliyun security group (inbound)

Open only these, in the ECS instance's security group:

Port	Who	Note
443/tcp	browsers (WSS)	permanent
8444/tcp (AGENT_BIND_PORT)	agents (mTLS)	permanent
80/tcp	Let's Encrypt HTTP-01	temporary — only during cert issue/renew; skip entirely if using DNS-01

Keep loopback-only (never open): Postgres 5432, Redis 6379, control-plane admin 8080. Reach the admin API from your laptop via SSH tunnel: ssh -L 8080:127.0.0.1:8080 root@8.138.1.192.

---

5. Build the browser bundle

cd /opt/web-terminal
npm --prefix relay-web run build        # emits relay-web/public/build/, served same-origin by D1

---

6. Enable + start both units

sudo cp deploy/systemd/relay-control-plane.service deploy/systemd/relay-data-plane.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now relay-control-plane.service   # migrates PG, listens on 127.0.0.1:8080
sudo systemctl enable --now relay-data-plane.service      # :443 browser WSS + :8444 agent mTLS
sudo systemctl status relay-control-plane.service relay-data-plane.service
journalctl -u relay-data-plane.service -f                 # watch for bind + verify-key load

---

7. Create an account + pairing code (control-plane admin)

The admin API is deny-by-default: POST /accounts and POST /accounts/:id/pairing-codes require a capability token with the manage right whose aud equals BASE_DOMAIN (control-plane authz.ts / main.ts expectedAud: env.baseDomain). accountId is taken ONLY from the verified token (INV3) — never from the body.

Mint the token with the capability PRIVATE key from step 2b (the B6 token-mint owns the signing helper; it imports /etc/relay/capability/capability-sign.key.pem). Token claims: rights=['manage'], aud=<BASE_DOMAIN>. Send it as Authorization: Bearer <token>.

Bootstrap has a two-token nuance (grounded in provision.ts):

1. POST /accounts checks manage only (no account-ownership check) → mint a manage token, create the account, read back its id.
2. POST /accounts/:id/pairing-codes additionally enforces token.sub == :id (own-account). Mint a second manage token with sub=<the new accountId>, then request the pairing code.

Over the SSH tunnel from step 4:

# 1) create account
curl -sS -X POST http://127.0.0.1:8080/accounts \
  -H "Authorization: Bearer <MANAGE_TOKEN aud=BASE_DOMAIN>" \
  -H 'content-type: application/json' -d '{"plan":"personal"}'
#   -> { "id": "<ACCOUNT_ID>", ... }

# 2) pairing code (token.sub must equal <ACCOUNT_ID>)
curl -sS -X POST http://127.0.0.1:8080/accounts/<ACCOUNT_ID>/pairing-codes \
  -H "Authorization: Bearer <MANAGE_TOKEN sub=ACCOUNT_ID aud=BASE_DOMAIN>"
#   -> { "code": "<PAIRING_CODE>", ... }   (single-use, TTL 600s default)

---

8. On the LAPTOP — build the agent, pair, run

The agent dials OUT; nothing inbound is opened on the laptop.

cd <repo>/agent
npm ci && npm run build                 # emits dist/cli.js (C1)
export RELAY_URL="wss://<SUBDOMAIN>.<BASE_DOMAIN>:8444"     # AGENT_BIND_PORT
export ENROLL_URL="https://<SUBDOMAIN>.<BASE_DOMAIN>/enroll"
export HOST_ID="<pick-a-host-id>"
export SUBDOMAIN="<SUBDOMAIN>"
export LOCAL_TARGET_URL="ws://127.0.0.1:3000"              # the base web-terminal app
export STATE_DIR="$HOME/.web-terminal-agent"

node dist/cli.js pair <PAIRING_CODE>    # redeems the code -> SPIFFE mTLS cert + hostContentSecret
node dist/cli.js run                    # dials the relay, holds the mux tunnel

pair redeems the single-use code and stores the enrolled SPIFFE cert under STATE_DIR; run opens the outbound mTLS tunnel and keeps it alive (heartbeat/backoff).

---

9. Browser — log in and click through to the shell

1. Open https://<SUBDOMAIN>.<BASE_DOMAIN> (served same-origin as the WSS endpoint).
2. Operator login (B6) mints a short-lived connect capability token.
3. Pick the host → the relay splices browser ↔ agent. The relay only sees ciphertext (INV2); the E2E is browser ↔ agent.

---

10. Verify restart-safety + revocation teardown

# Restart-safety (INV7): bounce the relay while a shell is open — the PTY and host registration survive.
sudo systemctl restart relay-data-plane.service
#   the browser auto-reconnects; the session is still there (state is in PG/Redis, not the process).

# Revocation teardown (INV12): revoke the host and watch the tunnel drop within budget.
curl -sS -X DELETE http://127.0.0.1:8080/hosts/<HOST_ID> \
  -H "Authorization: Bearer <MANAGE_TOKEN sub=ACCOUNT_ID aud=BASE_DOMAIN>"
#   -> control-plane publishes a KillSignal on Redis relay:revocations; the data-plane subscriber
#      closes the spliced stream (browser WS closes 4403). Measure revoke -> close latency.

---

Troubleshooting

Symptom	Meaning	Fix
Browser WS closes 1013	WS_TRY_LATER — no agent tunnel for that host (agent offline / not dialed)	Start the agent (node dist/cli.js run); check RELAY_URL host+port; confirm :8444 open; watch journalctl -u relay-data-plane.
Browser WS closes 401	bad Origin (CSWSH exact-match failed) — the Origin header is not in the relay's allowedOrigins	Open via https://<SUBDOMAIN>.<BASE_DOMAIN> (not the raw IP, not http:, no stray port). allowedOrigins is the port-less https://<sub>.<BASE_DOMAIN> on :443; must match EXACTLY. Verify BASE_DOMAIN in relay.env.
Agent mTLS closes 4401	cert chains to the CA but the pubkey is not enrolled in the host registry (INV14 registry-gated)	The host was never enrolled or was revoked. Re-run pair <CODE> with a fresh code; confirm the CP signed the leaf with /etc/relay/ca/intermediate.key.pem (chains to agent-ca.bundle.pem).
Browser WS closes 4403	WS_REVOKED — the host/session was revoked (expected after step 10)	Re-enroll the host to reconnect.
CP fails to boot: "CAPABILITY_SIGN_PUBKEY_B64 must decode to 32 bytes"	wrong key encoding in control-plane.env	Use the base64 value (not base64url) from step 2b for the CP; base64url is the relay's RELAY_AUTH_VERIFY_PUBKEY.
Admin API returns 401 on POST /accounts	missing/expired/foreign-aud capability token	Mint a manage token with aud=<BASE_DOMAIN>, signed by the step-2b private key; send Authorization: Bearer.
Admin API returns 403 on /pairing-codes	token lacks manage right, or token.sub != :id	Mint with rights:['manage'] and sub=<ACCOUNT_ID> (own-account rule).
LE issuance fails (timeout/connection)	HTTP-01 :80 not reachable, or domain not ICP-filed	Open :80 in the security group for issuance, or switch ACME_METHOD=dns-01; confirm the A-record + ICP filing.
docker compose ps unhealthy	PG/Redis not up	docker compose -f deploy/docker-compose.yml logs; confirm POSTGRES_PASSWORD set in deploy/.env.
Mixed-content / ws: blocked in browser	page is HTTPS but WS scheme resolved to ws:	Serve over HTTPS; the client follows page scheme (wss: on HTTPS, M6). Confirm the LE cert is valid for the FQDN.