kai/web-terminal
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(tunnel): recover an expired leaf over plain HTTPS, not mTLS

Browse Source 
Correction to the previous commit. Its recovery vhost could not work: nginx
will not forward an expired client certificate in ANY `ssl_verify_client` mode.
`optional` answers a bare `400 The SSL certificate error` before any location
runs, and `optional_no_ca` only tolerates CHAIN failures — see nginx's
`ngx_ssl_verify_error_optional()`, which covers self-signed / unknown-issuer /
unverifiable-leaf and NOT `X509_V_ERR_CERT_HAS_EXPIRED`. Verified live against
the deployed :8472 server, which rejected the real expired leaf.

So recovery drops mTLS instead of trying to bend it:

- agent: `buildTlsOptions` and the mTLS renew transport go back to being
  strictly fail-closed on expiry — the relaxation is gone from the TLS layer
  entirely. The rotator now decides per attempt: valid → mTLS `/renew`,
  expired-inside-grace → plain `/recover`, expired-beyond-grace → terminal
  `onExhausted` with no request issued at all.
- new `recoverCert` POSTs `{cert, csr}` with no client certificate. Possession
  of the private key is still proven: the CSR is self-signed by it and the host
  signer already enforces CSR PoP plus `CSR key == registered key`, so a
  replayed (public) cert without the key yields at most a certificate the
  attacker cannot authenticate with.
- control-plane: `/renew` is strict again (grace 0, matching the terminator).
  The grace lives on the new `POST /recover`, which reads the cert from the BODY
  and ignores the `x-client-cert` header, then runs the identical trust
  pipeline: X.509 path validation to the frp-client-CA anchors, SPIFFE parse,
  `notBefore` (never graced), registry `active` + account match. Revocation
  still bites.
- deploy: no new vhost, no new SNI, no DNS. One `location = /recover` merged
  into the existing enroll vhost, documented in
  `deploy/nginx/enroll-recover-location.md` with the nginx source citation.

The load-bearing new test is "a self-signed cert with a FORGED SPIFFE SAN is
refused → 401": nginx no longer validates the chain on this path, so that
assertion is what keeps `/recover` from being a cert vending machine.

Verified: agent 289/289, control-plane 290/290, tsc clean on both.
This commit is contained in:
Yaojia Wang
2026-07-29 09:52:17 +02:00
parent f3f4d8baa6
commit 5509c81eee
13 changed files with 559 additions and 472 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
deploy/nginx/enroll-recover-location.md Normal file
View File

@@ -0,0 +1,93 @@
# Enroll vhost addition — `/recover` (expired-leaf recovery)
One `location` block to **merge into the existing** enroll vhost on the VPS
(`/etc/nginx/conf.d/enroll.conf`, the `127.0.0.1:8471` server). No new server, no new SNI route, no
DNS work.
> **This is a MERGE, not a file to ship.** The enroll vhost also carries `/enroll`, `/device/enroll`,
> `/auth/login`, `/crl/`, `/renew` and `/device/:id/renew` — do not replace it.
## Why the route exists
`POST /renew` is mTLS-authenticated by the very leaf it renews, so once that leaf lapses the host can
never renew it and the tunnel stays down until an operator re-pairs. That is not hypothetical: a
laptop slept through its 8h renewal window, its 24h leaf expired, and the agent then logged
`client certificate has expired; renew before dialling` 6380 times over 8 days without recovering.
## Why it cannot be fixed on `/renew` itself
**nginx will not forward an expired client certificate, under any `ssl_verify_client` mode.**
- `optional` → nginx answers a bare `400 The SSL certificate error` as soon as verification fails.
The request never reaches the `location`, so no `if ($ssl_client_verify …)` can rescue it.
- `optional_no_ca` → does **not** help either. It only tolerates *chain* failures; see nginx's
`ngx_ssl_verify_error_optional()`, which covers `DEPTH_ZERO_SELF_SIGNED_CERT`,
`SELF_SIGNED_CERT_IN_CHAIN`, `UNABLE_TO_GET_ISSUER_CERT_LOCALLY` and
`UNABLE_TO_VERIFY_LEAF_SIGNATURE` — and **not** `X509_V_ERR_CERT_HAS_EXPIRED`.
- `ssl_verify_client` is a `server`-level directive, so it cannot be relaxed per-location anyway.
So recovery drops mTLS: `/recover` takes **no client certificate**, and the lapsed cert travels in
the request body instead.
## Why that is still authenticated
A certificate is public, so the body alone proves nothing — possession of the **private key** does,
and it is still proven end to end:
- the accompanying CSR is **self-signed by that key**, and the host signer's delegated gate enforces
CSR proof-of-possession plus `CSR key == registered key` (`control-plane/src/ca/csr.ts`
`verifyCsrPoP`);
- `control-plane/src/api/renew.ts` runs the *same* trust pipeline as `/renew` — real X.509 path
validation to the frp-client-CA anchors, SPIFFE SAN parse, `notBefore`, and a registry lookup
requiring an `active`, account-consistent host — differing **only** in a bounded overrun allowance
on `notAfter` (`DEFAULT_EXPIRED_RENEW_GRACE_MS`, 30 days).
Worst case for a replayed cert without the key: the attacker receives a certificate they cannot
authenticate with. Revocation still bites, via registry status.
## The block to add
```nginx
# Expired-leaf recovery: NO client cert (see enroll-recover-location.md). The control-plane is
# the sole verifier; strip any client-supplied cert header so only the body can speak.
location = /recover {
limit_req zone=renew_recover burst=5 nodelay;
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header x-client-cert "";
proxy_read_timeout 60s;
}
```
And once, in the `http` context (top of `enroll.conf` is fine) — the route is reachable
pre-authentication and costs the control-plane an X.509 path validation, so bound it. The
control-plane additionally rate-limits per identity (`createRenewRateLimiter`, 30/hour):
```nginx
limit_req_zone $binary_remote_addr zone=renew_recover:1m rate=10r/m;
```
## Deploy gate
```bash
cp /etc/nginx/conf.d/enroll.conf{,.bak.$(date +%s)} # snapshot first
# ...merge the block above...
nginx -t && systemctl restart nginx # NEVER reload on a failed -t
```
`restart`, not `reload`: a graceful reload has been observed keeping old workers alive for seconds,
which makes post-deploy verification race the change.
## Verify
```bash
# no cert needed — an in-grace expired leaf is re-issued
curl -sS --resolve enroll.terminal.yaojia.wang:443:<vps> \
-X POST -H 'content-type: application/json' \
-d "{\"cert\":\"$(base64 -w0 expired.cert.pem)\",\"csr\":\"$(base64 -w0 new.csr.der)\"}" \
https://enroll.terminal.yaojia.wang/recover # → 201 {cert,caChain,notAfter}
# a forged self-signed cert with a correct-looking SPIFFE SAN must still be refused
# → 401 (this is the load-bearing check; nginx is no longer validating the chain here)
```

65
deploy/nginx/recover-mtls.conf
View File

@@ -1,65 +0,0 @@
# recover-mtls.conf — nginx :8472 RECOVERY vhost: re-issue a leaf that has ALREADY EXPIRED.
#
# Install target: /etc/nginx/conf.d/recover-mtls.conf
# SNI route to merge into the stream map (see stream-sni-additions.md):
# recover.terminal.yaojia.wang 127.0.0.1:8472;
#
# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────────
# `POST /renew` is authenticated by mTLS with the very leaf it renews. Once that leaf lapses the host
# can no longer renew it, so the tunnel stays down until an operator re-pairs by hand. That is not
# hypothetical: a laptop slept through its renewal window, the leaf expired, and the agent then
# logged `client certificate has expired; renew before dialling` 6380 times over 8 days without ever
# recovering.
#
# The enroll vhost cannot host the fix. It runs `ssl_verify_client optional`, and nginx answers a
# bare `400 Bad Request` the moment a PRESENTED client cert fails verification — an expired leaf
# never reaches the `location`, so it can never be forwarded to the control-plane. `ssl_verify_client`
# is a server-level directive and cannot be relaxed per-location, hence a separate server block.
#
# ── THE TRADE (read before changing anything here) ────────────────────────────────────────────────
# This server runs `optional_no_ca`: nginx forwards the presented cert WITHOUT validating its chain.
# The control-plane is therefore the SOLE and FULL verifier on this path. `api/renew.ts` does, in
# order: real X.509 path validation to the frp-client-CA anchors (`assertPresentedCertTrusted` →
# relay-auth `verifyChain`), SPIFFE SAN parse, `notBefore` check (NEVER graced), `notAfter` + bounded
# grace (`DEFAULT_EXPIRED_RENEW_GRACE_MS`, 30d), then a registry lookup requiring an `active`,
# account-consistent host — so REVOCATION IS STILL ENFORCED, via registry status rather than the CRL.
# Do not weaken those checks on the assumption that nginx is still gating this port. It is not.
#
# Blast radius is deliberately tiny: ONLY `POST /renew` is reachable; every other path 404s, and the
# strict enroll vhost (:8471) is left untouched so the normal renewal path keeps its nginx-level
# chain + CRL enforcement.
# Recovery is a once-in-a-blue-moon call per host, but it is reachable pre-authentication and costs
# the control-plane an X.509 path validation, so bound it. CP additionally rate-limits per identity.
limit_req_zone $binary_remote_addr zone=renew_recover:1m rate=10r/m;
server {
listen 127.0.0.1:8472 ssl;
server_name recover.terminal.yaojia.wang;
ssl_certificate /etc/relay/frp-tls/fullchain.pem; # LE wildcard *.terminal.yaojia.wang
ssl_certificate_key /etc/relay/frp-tls/privkey.pem;
# Ask for a client cert and forward whatever is presented — INCLUDING an expired one, which is
# the entire point. `ssl_client_certificate` is kept only so the CertificateRequest carries a CA
# hint and clients auto-select the right identity; it does NOT gate anything under optional_no_ca.
ssl_verify_client optional_no_ca;
ssl_client_certificate /etc/relay/frp-client-ca/frp-client-ca.cert.pem;
location = /renew {
# Under optional_no_ca `$ssl_client_verify` is `NONE` even for a good cert, so presence of the
# cert itself is the only thing nginx can meaningfully assert here.
if ($ssl_client_escaped_cert = "") { return 403; }
limit_req zone=renew_recover burst=5 nodelay;
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header x-client-cert $ssl_client_escaped_cert;
proxy_read_timeout 60s;
}
# Everything else — including /enroll, /device/*, /auth/login — stays on the strict vhost.
location / { return 404; }
}

12
deploy/nginx/stream-sni-additions.md
View File

@@ -17,18 +17,6 @@ frp.terminal.yaojia.wang 127.0.0.1:7000; # frps control channel (TLS passthro
*.terminal.yaojia.wang 127.0.0.1:8470; # nginx :8470 TLS-term + device-CA mTLS (frp-mtls.conf)
```
### Later addition — expired-leaf recovery (recover-mtls.conf)
```nginx
recover.terminal.yaojia.wang 127.0.0.1:8472; # expired-leaf re-issue (recover-mtls.conf)
```
**Order IS load-bearing for this one**: it must sit ABOVE the `*.terminal.yaojia.wang` wildcard, the
same way `enroll.terminal.yaojia.wang` does. `hostnames` gives exact matches priority over wildcards
in nginx's `map`, so a correctly-placed line wins regardless — but keep the exact-match lines grouped
above the wildcard so a future editor cannot mis-read the intent. No DNS work is needed: the zone is
already served by the wildcard record `*.terminal.yaojia.wang → <vps>`.
## Coexistence warning (do NOT retype the existing lines)
The existing `map` body already contains — and MUST be preserved **verbatim**: