fix(tunnel): break the expired-leaf renewal deadlock

`POST /renew` is authenticated by mTLS with the very leaf it renews, so once
that leaf lapsed the host could never renew it and the tunnel stayed down until
an operator re-paired by hand. Production hit exactly this: the Mac slept
through its 8h renewal window, the 24h leaf expired, and the agent then logged
`client certificate has expired; renew before dialling` 6380 times over 8 days
without recovering.

Three layers independently refused an expired leaf, so all three had to move:

- agent `buildTlsOptions` gains an opt-in `expiredGraceMs`. Absent/0 keeps the
  historical fail-closed behaviour, and the TUNNEL dial never passes it — only
  the renew transport does. Past the window it throws the new
  `CertExpiredBeyondGraceError`.
- agent rotator routes an already-expired leaf to a separate recovery endpoint
  and treats "beyond grace" as TERMINAL: report once via `onExhausted`, stop
  retrying, and name the fix (re-pair) instead of spamming warnings forever.
- control-plane `assertPresentedCertTrusted` grants a bounded grace on
  `notAfter` only. Chain validation, SPIFFE identity, `notBefore`, and the
  registry `active`/account checks are all unchanged, so revocation still bites.
- new `deploy/nginx/recover-mtls.conf` (:8472). The strict enroll vhost cannot
  host this: under `ssl_verify_client optional` nginx answers a bare 400 as soon
  as a presented cert fails verification, so an expired leaf never reaches the
  location — and the directive is server-level, not per-location.

Grace defaults to 30 days on both ends and is configurable (`RECOVER_URL`,
`expiredRenewGraceMs`). The honest trade is recorded in the code: a stale stolen
leaf stays reusable for the window, which widens an existing exposure (an
unexpired stolen leaf already renews indefinitely) rather than opening a new one.

Verified: agent 296/296, control-plane 286/286, tsc clean on both.
This commit is contained in:
Yaojia Wang
2026-07-29 09:41:03 +02:00
parent b1bc50ccd1
commit f3f4d8baa6
11 changed files with 643 additions and 16 deletions

View File

@@ -18,6 +18,13 @@ export interface AgentConfig {
readonly localTargetUrl: string
readonly subdomain: string | null
readonly hostId: string | null
/**
* Renewal endpoint used ONLY when the current leaf has already expired (the strict `/renew` vhost
* rejects an expired client cert before it reaches the control-plane). Optional: when unset it is
* derived from `enrollUrl` by swapping the `enroll.` label for `recover.` — see
* `certs/rotation.ts` `recoveryRenewalUrlFor`.
*/
readonly recoverUrl?: string | null | undefined
}
/**
@@ -59,6 +66,11 @@ export const AgentConfigSchema = z
.refine(isLoopbackWsUrl, 'localTargetUrl must be a ws:// loopback URL (anti-SSRF)'),
subdomain: z.string().min(1).nullable(),
hostId: z.string().min(1).nullable(),
recoverUrl: z
.string()
.refine((u) => hasScheme(u, 'https:'), 'recoverUrl must be an https:// URL')
.nullable()
.optional(),
})
.strict()
.readonly()
@@ -85,6 +97,7 @@ export function loadAgentConfig(
localTargetUrl: argv.localTargetUrl ?? env.LOCAL_TARGET_URL ?? DEFAULT_LOCAL_TARGET,
subdomain: argv.subdomain ?? env.SUBDOMAIN ?? null,
hostId: argv.hostId ?? env.HOST_ID ?? null,
recoverUrl: argv.recoverUrl ?? env.RECOVER_URL ?? null,
}
return AgentConfigSchema.parse(merged)
}