fix(tunnel): break the expired-leaf renewal deadlock

`POST /renew` is authenticated by mTLS with the very leaf it renews, so once
that leaf lapsed the host could never renew it and the tunnel stayed down until
an operator re-paired by hand. Production hit exactly this: the Mac slept
through its 8h renewal window, the 24h leaf expired, and the agent then logged
`client certificate has expired; renew before dialling` 6380 times over 8 days
without recovering.

Three layers independently refused an expired leaf, so all three had to move:

- agent `buildTlsOptions` gains an opt-in `expiredGraceMs`. Absent/0 keeps the
  historical fail-closed behaviour, and the TUNNEL dial never passes it — only
  the renew transport does. Past the window it throws the new
  `CertExpiredBeyondGraceError`.
- agent rotator routes an already-expired leaf to a separate recovery endpoint
  and treats "beyond grace" as TERMINAL: report once via `onExhausted`, stop
  retrying, and name the fix (re-pair) instead of spamming warnings forever.
- control-plane `assertPresentedCertTrusted` grants a bounded grace on
  `notAfter` only. Chain validation, SPIFFE identity, `notBefore`, and the
  registry `active`/account checks are all unchanged, so revocation still bites.
- new `deploy/nginx/recover-mtls.conf` (:8472). The strict enroll vhost cannot
  host this: under `ssl_verify_client optional` nginx answers a bare 400 as soon
  as a presented cert fails verification, so an expired leaf never reaches the
  location — and the directive is server-level, not per-location.

Grace defaults to 30 days on both ends and is configurable (`RECOVER_URL`,
`expiredRenewGraceMs`). The honest trade is recorded in the code: a stale stolen
leaf stays reusable for the window, which widens an existing exposure (an
unexpired stolen leaf already renews indefinitely) rather than opening a new one.

Verified: agent 296/296, control-plane 286/286, tsc clean on both.
This commit is contained in:
Yaojia Wang
2026-07-29 09:41:03 +02:00
parent b1bc50ccd1
commit f3f4d8baa6
11 changed files with 643 additions and 16 deletions

View File

@@ -0,0 +1,65 @@
# recover-mtls.conf — nginx :8472 RECOVERY vhost: re-issue a leaf that has ALREADY EXPIRED.
#
# Install target: /etc/nginx/conf.d/recover-mtls.conf
# SNI route to merge into the stream map (see stream-sni-additions.md):
# recover.terminal.yaojia.wang 127.0.0.1:8472;
#
# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────────
# `POST /renew` is authenticated by mTLS with the very leaf it renews. Once that leaf lapses the host
# can no longer renew it, so the tunnel stays down until an operator re-pairs by hand. That is not
# hypothetical: a laptop slept through its renewal window, the leaf expired, and the agent then
# logged `client certificate has expired; renew before dialling` 6380 times over 8 days without ever
# recovering.
#
# The enroll vhost cannot host the fix. It runs `ssl_verify_client optional`, and nginx answers a
# bare `400 Bad Request` the moment a PRESENTED client cert fails verification — an expired leaf
# never reaches the `location`, so it can never be forwarded to the control-plane. `ssl_verify_client`
# is a server-level directive and cannot be relaxed per-location, hence a separate server block.
#
# ── THE TRADE (read before changing anything here) ────────────────────────────────────────────────
# This server runs `optional_no_ca`: nginx forwards the presented cert WITHOUT validating its chain.
# The control-plane is therefore the SOLE and FULL verifier on this path. `api/renew.ts` does, in
# order: real X.509 path validation to the frp-client-CA anchors (`assertPresentedCertTrusted` →
# relay-auth `verifyChain`), SPIFFE SAN parse, `notBefore` check (NEVER graced), `notAfter` + bounded
# grace (`DEFAULT_EXPIRED_RENEW_GRACE_MS`, 30d), then a registry lookup requiring an `active`,
# account-consistent host — so REVOCATION IS STILL ENFORCED, via registry status rather than the CRL.
# Do not weaken those checks on the assumption that nginx is still gating this port. It is not.
#
# Blast radius is deliberately tiny: ONLY `POST /renew` is reachable; every other path 404s, and the
# strict enroll vhost (:8471) is left untouched so the normal renewal path keeps its nginx-level
# chain + CRL enforcement.
# Recovery is a once-in-a-blue-moon call per host, but it is reachable pre-authentication and costs
# the control-plane an X.509 path validation, so bound it. CP additionally rate-limits per identity.
limit_req_zone $binary_remote_addr zone=renew_recover:1m rate=10r/m;
server {
listen 127.0.0.1:8472 ssl;
server_name recover.terminal.yaojia.wang;
ssl_certificate /etc/relay/frp-tls/fullchain.pem; # LE wildcard *.terminal.yaojia.wang
ssl_certificate_key /etc/relay/frp-tls/privkey.pem;
# Ask for a client cert and forward whatever is presented — INCLUDING an expired one, which is
# the entire point. `ssl_client_certificate` is kept only so the CertificateRequest carries a CA
# hint and clients auto-select the right identity; it does NOT gate anything under optional_no_ca.
ssl_verify_client optional_no_ca;
ssl_client_certificate /etc/relay/frp-client-ca/frp-client-ca.cert.pem;
location = /renew {
# Under optional_no_ca `$ssl_client_verify` is `NONE` even for a good cert, so presence of the
# cert itself is the only thing nginx can meaningfully assert here.
if ($ssl_client_escaped_cert = "") { return 403; }
limit_req zone=renew_recover burst=5 nodelay;
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header x-client-cert $ssl_client_escaped_cert;
proxy_read_timeout 60s;
}
# Everything else — including /enroll, /device/*, /auth/login — stays on the strict vhost.
location / { return 404; }
}

View File

@@ -17,6 +17,18 @@ frp.terminal.yaojia.wang 127.0.0.1:7000; # frps control channel (TLS passthro
*.terminal.yaojia.wang 127.0.0.1:8470; # nginx :8470 TLS-term + device-CA mTLS (frp-mtls.conf)
```
### Later addition — expired-leaf recovery (recover-mtls.conf)
```nginx
recover.terminal.yaojia.wang 127.0.0.1:8472; # expired-leaf re-issue (recover-mtls.conf)
```
**Order IS load-bearing for this one**: it must sit ABOVE the `*.terminal.yaojia.wang` wildcard, the
same way `enroll.terminal.yaojia.wang` does. `hostnames` gives exact matches priority over wildcards
in nginx's `map`, so a correctly-placed line wins regardless — but keep the exact-match lines grouped
above the wildcard so a future editor cannot mis-read the intent. No DNS work is needed: the zone is
already served by the wildcard record `*.terminal.yaojia.wang → <vps>`.
## Coexistence warning (do NOT retype the existing lines)
The existing `map` body already contains — and MUST be preserved **verbatim**: