aa1912b962
RELAY-PHASE1 Wave A2/B/C/D/E (12-agent workflow, all tsc-clean, 314/314 tests pass): - A2: control-plane server.ts entry + boot/redis.ts revocation-bus wiring + start script. - B1: relay-run shared-store EnforceDeps (relay-auth ports over the SAME Postgres+Redis as P3). - B2: registry-backed MtlsVerifier (verifyAgentCert, fail-closed, INV14). - B3: store-backed RouteResolver (subdomain->hostId). - B4: Redis relay:revocations subscriber -> tunnel teardown (INV12). - B5: main-phase1.ts production entry (public bind, real TLS, async-mTLS prefetch bridge) + staging /auth/mint. - B6 (PARTIAL): relay-web operator login + browser DPoP; proof offered via term.dpop.<b64u> subprotocol. - C: agent dist/cli.js build (esbuild) + runTunnel run-loop + CliDeps. - D1: same-origin static serve of relay-web/public from the browser WSS. - E: systemd units + gen-ca/gen-capability-key/issue-tls-cert scripts + deploy/RUNBOOK.md. Adversarial review: all hard invariants PASS. Follow-ups (B7): close DPoP-subprotocol read on browser-server (blocks browser connect); rate-limit /auth/mint (F1); wire activeSessionCount (F2); scrub error logs (F5). Excludes unrelated public/style.css (concurrent iOS job).
278 lines
13 KiB
Markdown
278 lines
13 KiB
Markdown
# RUNBOOK — deploy the rendezvous-relay to VPS `8.138.1.192` (Alibaba Cloud, mainland)
|
|
|
|
> **Staging, durable single-tenant** (PLAN_RELAY_PHASE1 §0). Real Postgres + Redis (Docker Compose on
|
|
> the VPS), real Let's Encrypt TLS on `:443` against an **already ICP-filed** domain, agent dials OUT
|
|
> from the operator's laptop. Dev in-process CA signer is accepted for staging (KMS → Phase 2).
|
|
>
|
|
> Everything below is a **staging template** — replace every `<placeholder>` and confirm each path on
|
|
> the box. Config is **env-only**; no host/port/secret is hardcoded in code or units.
|
|
|
|
Two independent trust chains — never cross-wire them:
|
|
|
|
| Chain | Issued by | Protects | Script |
|
|
|---|---|---|---|
|
|
| **Public web PKI** | Let's Encrypt | browser `:443` WSS (`TLS_CERT_PATH`) | `issue-tls-cert.sh` |
|
|
| **Private enrollment CA** | your offline root | agent mTLS + relay agent-server cert (`AGENT_CA_*`) | `gen-agent-ca.sh` |
|
|
|
|
And one shared key: the **P5 capability keypair** — control-plane **signs**, relay + admin API
|
|
**verify** (`gen-capability-key.sh`). `CAPABILITY_SIGN_PUBKEY_B64` (CP) and `RELAY_AUTH_VERIFY_PUBKEY`
|
|
(relay) are the **same public key**, two encodings.
|
|
|
|
---
|
|
|
|
## 0. Prerequisites (on `8.138.1.192`)
|
|
|
|
```bash
|
|
# Docker Engine + Compose plugin
|
|
curl -fsSL https://get.docker.com | sh
|
|
sudo systemctl enable --now docker
|
|
|
|
# Node 20 (satisfies control-plane>=18, relay-run>=20, agent>=18) via NodeSource
|
|
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo bash -
|
|
sudo apt-get install -y nodejs git openssl
|
|
|
|
# Clone (this runbook assumes /opt/web-terminal; adjust the systemd --prefix if you change it)
|
|
sudo git clone <REPO_URL> /opt/web-terminal
|
|
cd /opt/web-terminal
|
|
npm ci # root deps
|
|
npm --prefix control-plane ci
|
|
npm --prefix relay-run ci
|
|
npm --prefix relay-web ci
|
|
|
|
# Dedicated non-root service user + secret dir
|
|
sudo useradd --system --home /opt/web-terminal relay || true
|
|
sudo mkdir -p /etc/relay
|
|
```
|
|
|
|
**DNS (do this first — LE HTTP-01 and the browser both need it):** create an A-record
|
|
`<SUBDOMAIN>.<BASE_DOMAIN>` → `8.138.1.192`. The domain must be **ICP-filed** (mainland Aliyun blocks
|
|
`:80/:443` on unfiled domains; if unfiled, you must use DNS-01 in step 2).
|
|
|
|
---
|
|
|
|
## 1. Bring up Postgres + Redis (Docker Compose, loopback-only)
|
|
|
|
```bash
|
|
cd /opt/web-terminal
|
|
cp deploy/.env.example deploy/.env # gitignored; fill POSTGRES_PASSWORD (strong random)
|
|
docker compose --env-file deploy/.env -f deploy/docker-compose.yml up -d
|
|
docker compose -f deploy/docker-compose.yml ps # both healthy; bound to 127.0.0.1 only
|
|
```
|
|
|
|
Both services publish to `127.0.0.1` only — never the public interface (they are the relay's private
|
|
state).
|
|
|
|
---
|
|
|
|
## 2. Generate the CA, the capability key, and the TLS cert
|
|
|
|
```bash
|
|
# (a) PRIVATE enrollment CA (agent mTLS) — NOT Let's Encrypt.
|
|
sudo AGENT_FACING_HOST="<SUBDOMAIN>.<BASE_DOMAIN>" \
|
|
AGENT_FACING_IP="8.138.1.192" \
|
|
RELAY_TRUST_DOMAIN="<RELAY_TRUST_DOMAIN>" \
|
|
bash deploy/scripts/gen-agent-ca.sh
|
|
# writes /etc/relay/ca/{root,intermediate,agent-ca.*,relay-agent-server.*}
|
|
|
|
# (b) P5 capability keypair — prints BOTH public-key encodings; private key -> 0600 file.
|
|
sudo bash deploy/scripts/gen-capability-key.sh
|
|
# -> note CAPABILITY_SIGN_PUBKEY_B64 (base64) and RELAY_AUTH_VERIFY_PUBKEY (base64url); same key.
|
|
|
|
# (c) PUBLIC TLS cert for the browser :443 (Let's Encrypt).
|
|
# http-01 needs inbound :80 open DURING issuance (step 4); dns-01 needs no inbound port.
|
|
sudo BASE_DOMAIN="<BASE_DOMAIN>" SUBDOMAIN="<SUBDOMAIN>" ACME_EMAIL="<you@example.com>" \
|
|
ACME_METHOD="http-01" \
|
|
TLS_CERT_PATH="/etc/relay/tls/fullchain.pem" TLS_KEY_PATH="/etc/relay/tls/privkey.pem" \
|
|
bash deploy/scripts/issue-tls-cert.sh
|
|
|
|
sudo chown -R relay:relay /etc/relay
|
|
```
|
|
|
|
> **Staging CA seam (A-wave, not this task):** the control-plane's dev in-process signer must sign
|
|
> agent leaves with `/etc/relay/ca/intermediate.key.pem` so they chain to `agent-ca.bundle.pem`.
|
|
> `CA_INTERMEDIATE_KMS_KEY_REF=dev-local` selects the dev signer; point it at that key. Phase 2 = KMS.
|
|
> After first boot, move `/etc/relay/ca/root.key.pem` **off** the host (offline anchor).
|
|
|
|
---
|
|
|
|
## 3. Fill the two env files
|
|
|
|
Copy the relevant blocks of `deploy/.env.example` into two 0600 files. **The linchpin:**
|
|
`CAPABILITY_SIGN_PUBKEY_B64` (control-plane) and `RELAY_AUTH_VERIFY_PUBKEY` (relay) are the SAME key
|
|
from step 2b — paste both.
|
|
|
|
```bash
|
|
sudo install -m 600 -o relay -g relay /dev/null /etc/relay/control-plane.env
|
|
sudo install -m 600 -o relay -g relay /dev/null /etc/relay/relay.env
|
|
```
|
|
|
|
`/etc/relay/control-plane.env` (P3):
|
|
|
|
```ini
|
|
PG_URL=postgres://relay:<POSTGRES_PASSWORD>@127.0.0.1:5432/relay
|
|
REDIS_URL=redis://127.0.0.1:6379
|
|
CAPABILITY_SIGN_PUBKEY_B64=<base64 from step 2b>
|
|
CA_INTERMEDIATE_KMS_KEY_REF=dev-local
|
|
CA_INTERMEDIATE_CERT_PATH=/etc/relay/ca/intermediate.cert.pem
|
|
NODE_MTLS_TRUST_BUNDLE_PATH=/etc/relay/ca/agent-ca.bundle.pem
|
|
BASE_DOMAIN=<BASE_DOMAIN>
|
|
CP_BIND_HOST=127.0.0.1
|
|
CP_BIND_PORT=8080
|
|
# HEARTBEAT_TTL_SEC / PAIRING_TTL_SEC / PAIRING_MAX_REDEEM_ATTEMPTS use defaults if unset
|
|
```
|
|
|
|
`/etc/relay/relay.env` (P1 data-plane):
|
|
|
|
```ini
|
|
BASE_DOMAIN=<BASE_DOMAIN>
|
|
BIND_HOST=0.0.0.0
|
|
BIND_PORT=443
|
|
TLS_CERT_PATH=/etc/relay/tls/fullchain.pem
|
|
TLS_KEY_PATH=/etc/relay/tls/privkey.pem
|
|
AGENT_BIND_PORT=8444
|
|
AGENT_CA_CERT_PATH=/etc/relay/ca/agent-ca.cert.pem
|
|
AGENT_CA_CHAIN_PATH=/etc/relay/ca/agent-ca.bundle.pem
|
|
RELAY_NODE_ID=relay-1
|
|
RELAY_AUTH_VERIFY_PUBKEY=<base64url from step 2b — SAME key as CAPABILITY_SIGN_PUBKEY_B64>
|
|
RELAY_TRUST_DOMAIN=<RELAY_TRUST_DOMAIN>
|
|
PG_URL=postgres://relay:<POSTGRES_PASSWORD>@127.0.0.1:5432/relay
|
|
REDIS_URL=redis://127.0.0.1:6379
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Open the Aliyun security group (inbound)
|
|
|
|
Open **only** these, in the ECS instance's security group:
|
|
|
|
| Port | Who | Note |
|
|
|---|---|---|
|
|
| `443/tcp` | browsers (WSS) | permanent |
|
|
| `8444/tcp` (`AGENT_BIND_PORT`) | agents (mTLS) | permanent |
|
|
| `80/tcp` | Let's Encrypt HTTP-01 | **temporary** — only during cert issue/renew; skip entirely if using DNS-01 |
|
|
|
|
**Keep loopback-only (never open):** Postgres `5432`, Redis `6379`, control-plane admin
|
|
`8080`. Reach the admin API from your laptop via SSH tunnel: `ssh -L 8080:127.0.0.1:8080 root@8.138.1.192`.
|
|
|
|
---
|
|
|
|
## 5. Build the browser bundle
|
|
|
|
```bash
|
|
cd /opt/web-terminal
|
|
npm --prefix relay-web run build # emits relay-web/public/build/, served same-origin by D1
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Enable + start both units
|
|
|
|
```bash
|
|
sudo cp deploy/systemd/relay-control-plane.service deploy/systemd/relay-data-plane.service /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now relay-control-plane.service # migrates PG, listens on 127.0.0.1:8080
|
|
sudo systemctl enable --now relay-data-plane.service # :443 browser WSS + :8444 agent mTLS
|
|
sudo systemctl status relay-control-plane.service relay-data-plane.service
|
|
journalctl -u relay-data-plane.service -f # watch for bind + verify-key load
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Create an account + pairing code (control-plane admin)
|
|
|
|
The admin API is **deny-by-default**: `POST /accounts` and `POST /accounts/:id/pairing-codes` require a
|
|
**capability token** with the `manage` right whose **`aud` equals `BASE_DOMAIN`** (control-plane
|
|
`authz.ts` / `main.ts` `expectedAud: env.baseDomain`). `accountId` is taken ONLY from the verified
|
|
token (INV3) — never from the body.
|
|
|
|
Mint the token with the **capability PRIVATE key** from step 2b (the B6 token-mint owns the signing
|
|
helper; it imports `/etc/relay/capability/capability-sign.key.pem`). Token claims:
|
|
`rights=['manage']`, `aud=<BASE_DOMAIN>`. Send it as `Authorization: Bearer <token>`.
|
|
|
|
Bootstrap has a two-token nuance (grounded in `provision.ts`):
|
|
|
|
1. `POST /accounts` checks `manage` only (no account-ownership check) → mint a `manage` token, create
|
|
the account, read back its `id`.
|
|
2. `POST /accounts/:id/pairing-codes` additionally enforces `token.sub == :id` (own-account). Mint a
|
|
second `manage` token with `sub=<the new accountId>`, then request the pairing code.
|
|
|
|
Over the SSH tunnel from step 4:
|
|
|
|
```bash
|
|
# 1) create account
|
|
curl -sS -X POST http://127.0.0.1:8080/accounts \
|
|
-H "Authorization: Bearer <MANAGE_TOKEN aud=BASE_DOMAIN>" \
|
|
-H 'content-type: application/json' -d '{"plan":"personal"}'
|
|
# -> { "id": "<ACCOUNT_ID>", ... }
|
|
|
|
# 2) pairing code (token.sub must equal <ACCOUNT_ID>)
|
|
curl -sS -X POST http://127.0.0.1:8080/accounts/<ACCOUNT_ID>/pairing-codes \
|
|
-H "Authorization: Bearer <MANAGE_TOKEN sub=ACCOUNT_ID aud=BASE_DOMAIN>"
|
|
# -> { "code": "<PAIRING_CODE>", ... } (single-use, TTL 600s default)
|
|
```
|
|
|
|
---
|
|
|
|
## 8. On the LAPTOP — build the agent, pair, run
|
|
|
|
The agent dials OUT; nothing inbound is opened on the laptop.
|
|
|
|
```bash
|
|
cd <repo>/agent
|
|
npm ci && npm run build # emits dist/cli.js (C1)
|
|
export RELAY_URL="wss://<SUBDOMAIN>.<BASE_DOMAIN>:8444" # AGENT_BIND_PORT
|
|
export ENROLL_URL="https://<SUBDOMAIN>.<BASE_DOMAIN>/enroll"
|
|
export HOST_ID="<pick-a-host-id>"
|
|
export SUBDOMAIN="<SUBDOMAIN>"
|
|
export LOCAL_TARGET_URL="ws://127.0.0.1:3000" # the base web-terminal app
|
|
export STATE_DIR="$HOME/.web-terminal-agent"
|
|
|
|
node dist/cli.js pair <PAIRING_CODE> # redeems the code -> SPIFFE mTLS cert + hostContentSecret
|
|
node dist/cli.js run # dials the relay, holds the mux tunnel
|
|
```
|
|
|
|
`pair` redeems the single-use code and stores the enrolled SPIFFE cert under `STATE_DIR`; `run` opens
|
|
the outbound mTLS tunnel and keeps it alive (heartbeat/backoff).
|
|
|
|
---
|
|
|
|
## 9. Browser — log in and click through to the shell
|
|
|
|
1. Open `https://<SUBDOMAIN>.<BASE_DOMAIN>` (served same-origin as the WSS endpoint).
|
|
2. Operator login (B6) mints a short-lived connect capability token.
|
|
3. Pick the host → the relay splices browser ↔ agent. The relay only sees **ciphertext** (INV2); the
|
|
E2E is browser ↔ agent.
|
|
|
|
---
|
|
|
|
## 10. Verify restart-safety + revocation teardown
|
|
|
|
```bash
|
|
# Restart-safety (INV7): bounce the relay while a shell is open — the PTY and host registration survive.
|
|
sudo systemctl restart relay-data-plane.service
|
|
# the browser auto-reconnects; the session is still there (state is in PG/Redis, not the process).
|
|
|
|
# Revocation teardown (INV12): revoke the host and watch the tunnel drop within budget.
|
|
curl -sS -X DELETE http://127.0.0.1:8080/hosts/<HOST_ID> \
|
|
-H "Authorization: Bearer <MANAGE_TOKEN sub=ACCOUNT_ID aud=BASE_DOMAIN>"
|
|
# -> control-plane publishes a KillSignal on Redis relay:revocations; the data-plane subscriber
|
|
# closes the spliced stream (browser WS closes 4403). Measure revoke -> close latency.
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Meaning | Fix |
|
|
|---|---|---|
|
|
| Browser WS closes **1013** | `WS_TRY_LATER` — **no agent tunnel** for that host (agent offline / not dialed) | Start the agent (`node dist/cli.js run`); check `RELAY_URL` host+port; confirm `:8444` open; watch `journalctl -u relay-data-plane`. |
|
|
| Browser WS closes **401** | **bad Origin** (CSWSH exact-match failed) — the `Origin` header is not in the relay's `allowedOrigins` | Open via `https://<SUBDOMAIN>.<BASE_DOMAIN>` (not the raw IP, not `http:`, no stray port). `allowedOrigins` is the port-less `https://<sub>.<BASE_DOMAIN>` on `:443`; must match EXACTLY. Verify `BASE_DOMAIN` in `relay.env`. |
|
|
| Agent mTLS closes **4401** | cert chains to the CA but the **pubkey is not enrolled** in the host registry (INV14 registry-gated) | The host was never enrolled or was revoked. Re-run `pair <CODE>` with a fresh code; confirm the CP signed the leaf with `/etc/relay/ca/intermediate.key.pem` (chains to `agent-ca.bundle.pem`). |
|
|
| Browser WS closes **4403** | `WS_REVOKED` — the host/session was revoked (expected after step 10) | Re-enroll the host to reconnect. |
|
|
| CP fails to boot: "CAPABILITY_SIGN_PUBKEY_B64 must decode to 32 bytes" | wrong key encoding in `control-plane.env` | Use the **base64** value (not base64url) from step 2b for the CP; base64url is the relay's `RELAY_AUTH_VERIFY_PUBKEY`. |
|
|
| Admin API returns **401** on `POST /accounts` | missing/expired/foreign-`aud` capability token | Mint a `manage` token with `aud=<BASE_DOMAIN>`, signed by the step-2b private key; send `Authorization: Bearer`. |
|
|
| Admin API returns **403** on `/pairing-codes` | token lacks `manage` right, or `token.sub != :id` | Mint with `rights:['manage']` and `sub=<ACCOUNT_ID>` (own-account rule). |
|
|
| LE issuance fails (timeout/connection) | HTTP-01 `:80` not reachable, or domain not ICP-filed | Open `:80` in the security group for issuance, or switch `ACME_METHOD=dns-01`; confirm the A-record + ICP filing. |
|
|
| `docker compose ps` unhealthy | PG/Redis not up | `docker compose -f deploy/docker-compose.yml logs`; confirm `POSTGRES_PASSWORD` set in `deploy/.env`. |
|
|
| Mixed-content / `ws:` blocked in browser | page is HTTPS but WS scheme resolved to `ws:` | Serve over HTTPS; the client follows page scheme (`wss:` on HTTPS, M6). Confirm the LE cert is valid for the FQDN. |
|