kai/web-terminal
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(tunnel): break the expired-leaf renewal deadlock

Browse Source 
`POST /renew` is authenticated by mTLS with the very leaf it renews, so once
that leaf lapsed the host could never renew it and the tunnel stayed down until
an operator re-paired by hand. Production hit exactly this: the Mac slept
through its 8h renewal window, the 24h leaf expired, and the agent then logged
`client certificate has expired; renew before dialling` 6380 times over 8 days
without recovering.

Three layers independently refused an expired leaf, so all three had to move:

- agent `buildTlsOptions` gains an opt-in `expiredGraceMs`. Absent/0 keeps the
  historical fail-closed behaviour, and the TUNNEL dial never passes it — only
  the renew transport does. Past the window it throws the new
  `CertExpiredBeyondGraceError`.
- agent rotator routes an already-expired leaf to a separate recovery endpoint
  and treats "beyond grace" as TERMINAL: report once via `onExhausted`, stop
  retrying, and name the fix (re-pair) instead of spamming warnings forever.
- control-plane `assertPresentedCertTrusted` grants a bounded grace on
  `notAfter` only. Chain validation, SPIFFE identity, `notBefore`, and the
  registry `active`/account checks are all unchanged, so revocation still bites.
- new `deploy/nginx/recover-mtls.conf` (:8472). The strict enroll vhost cannot
  host this: under `ssl_verify_client optional` nginx answers a bare 400 as soon
  as a presented cert fails verification, so an expired leaf never reaches the
  location — and the directive is server-level, not per-location.

Grace defaults to 30 days on both ends and is configurable (`RECOVER_URL`,
`expiredRenewGraceMs`). The honest trade is recorded in the code: a stale stolen
leaf stays reusable for the window, which widens an existing exposure (an
unexpired stolen leaf already renews indefinitely) rather than opening a new one.

Verified: agent 296/296, control-plane 286/286, tsc clean on both.
This commit is contained in:
Yaojia Wang
2026-07-29 09:41:03 +02:00
parent b1bc50ccd1
commit f3f4d8baa6
11 changed files with 643 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
control-plane/src/api/renew.ts
View File

@@ -57,6 +57,21 @@ export const MAX_CSR_B64_LEN = 8192
* its own current cert. Production wiring can swap in a socket-peer-cert resolver instead.
*/
export const DEFAULT_CLIENT_CERT_HEADER = 'x-client-cert'
/**
* How long after `notAfter` a presented leaf may still authenticate its OWN re-issuance (30 days).
*
* `/renew` is authenticated by the very leaf it renews, so a strict expiry check means a host whose
* leaf lapsed can never renew it — it is bricked until an operator re-pairs (this happened: a laptop
* slept through its renewal window and the tunnel stayed down for 8 days). Accepting a recently
* expired leaf FOR RE-ISSUANCE ONLY breaks that deadlock.
*
* Nothing else is relaxed: the leaf must still chain to the CA anchor set, its SPIFFE identity must
* still resolve to a registry record that is `active` and account-consistent, and `notBefore` is NOT
* graced. The residual risk is that a stale stolen leaf stays usable for this window — an attacker
* holding an unexpired stolen leaf can already renew indefinitely, so this widens an existing
* exposure rather than opening a new one. Set to 0 to restore the strict behaviour.
*/
export const DEFAULT_EXPIRED_RENEW_GRACE_MS = 30 * 24 * 60 * 60 * 1000
/** Uniform reject: 401 = no/invalid current cert (unauthenticated); 403 = cert valid but not allowed. */
export class RenewRejectError extends Error {
@@ -207,6 +222,7 @@ function assertPresentedCertTrusted(
der: Uint8Array,
caAnchorsDer: readonly Uint8Array[],
nowMs: number,
expiredGraceMs: number,
): void {
let leaf: NodeX509Certificate
let anchors: NodeX509Certificate[]
@@ -220,7 +236,10 @@ function assertPresentedCertTrusted(
const notBefore = new Date(leaf.validFrom).getTime()
const notAfter = new Date(leaf.validTo).getTime()
if (Number.isNaN(notBefore) || Number.isNaN(notAfter)) throw new RenewRejectError(401)
if (nowMs < notBefore || nowMs > notAfter) throw new RenewRejectError(401)
// `notBefore` is never graced — a not-yet-valid cert is nonsense, not a recoverable lapse. Only
// `notAfter` gets the bounded renewal grace (see DEFAULT_EXPIRED_RENEW_GRACE_MS).
if (nowMs < notBefore) throw new RenewRejectError(401)
if (nowMs > notAfter + Math.max(0, expiredGraceMs)) throw new RenewRejectError(401)
}
/**
@@ -234,7 +253,11 @@ function assertPresentedCertTrusted(
function parsePresentedCertIdentity(
der: Uint8Array,
expectedKind: SpiffeKind,
verify?: { readonly caAnchorsDer: readonly Uint8Array[]; readonly nowMs: number },
verify?: {
readonly caAnchorsDer: readonly Uint8Array[]
readonly nowMs: number
readonly expiredGraceMs: number
},
): CertIdentity {
let leaf: x509.X509Certificate
try {
@@ -242,7 +265,8 @@ function parsePresentedCertIdentity(
} catch {
throw new RenewRejectError(401)
}
if (verify !== undefined) assertPresentedCertTrusted(der, verify.caAnchorsDer, verify.nowMs)
if (verify !== undefined)
assertPresentedCertTrusted(der, verify.caAnchorsDer, verify.nowMs, verify.expiredGraceMs)
const san = leaf.getExtension(x509.SubjectAlternativeNameExtension)
const uri = san?.names.toJSON().find((n) => n.type === 'url')?.value
if (uri === undefined) throw new RenewRejectError(401)
@@ -279,6 +303,11 @@ export interface RenewDeps {
readonly hostCaAnchorsDer?: readonly Uint8Array[]
/** device-CA anchor DER(s) — same role for the DEVICE renew path. */
readonly deviceCaAnchorsDer?: readonly Uint8Array[]
/**
* Window after `notAfter` in which a presented leaf may still authenticate its own re-issuance.
* Defaults to `DEFAULT_EXPIRED_RENEW_GRACE_MS`; 0 restores the strict fail-closed behaviour.
*/
readonly expiredRenewGraceMs?: number
}
/** Map any thrown error to a uniform HTTP reject — never leak which internal check failed. */
@@ -298,6 +327,7 @@ function sendError(reply: FastifyReply, err: unknown): void {
export function buildRenewRouter(deps: RenewDeps): FastifyPluginAsync {
const presentedCert = deps.presentedCert ?? headerPresentedCert()
const rateLimiter = deps.rateLimiter ?? createRenewRateLimiter()
const expiredGraceMs = deps.expiredRenewGraceMs ?? DEFAULT_EXPIRED_RENEW_GRACE_MS
return async (app) => {
app.post('/renew', async (req, reply) => {
@@ -307,7 +337,9 @@ export function buildRenewRouter(deps: RenewDeps): FastifyPluginAsync {
const identity = parsePresentedCertIdentity(
der,
'host',
deps.hostCaAnchorsDer ? { caAnchorsDer: deps.hostCaAnchorsDer, nowMs: Date.now() } : undefined,
deps.hostCaAnchorsDer
? { caAnchorsDer: deps.hostCaAnchorsDer, nowMs: Date.now(), expiredGraceMs }
: undefined,
)
rateLimiter.check(`host:${identity.accountId}:${identity.id}`)
@@ -340,7 +372,9 @@ export function buildRenewRouter(deps: RenewDeps): FastifyPluginAsync {
const identity = parsePresentedCertIdentity(
der,
'device',
deps.deviceCaAnchorsDer ? { caAnchorsDer: deps.deviceCaAnchorsDer, nowMs: Date.now() } : undefined,
deps.deviceCaAnchorsDer
? { caAnchorsDer: deps.deviceCaAnchorsDer, nowMs: Date.now(), expiredGraceMs }
: undefined,
)
const id = (req.params as { id: string }).id
rateLimiter.check(`device:${identity.id}`)

95
control-plane/test/renew.test.ts
View File

@@ -490,10 +490,16 @@ describe('CP6c POST /renew — presented current-cert chain + expiry verificatio
expect(res.statusCode).toBe(201)
})
test('an EXPIRED current cert (valid chain, notAfter in the past) is rejected → 401', async () => {
/**
* POLICY CHANGE (expired-leaf deadlock fix): `/renew` is mTLS-authenticated by the very leaf it
* renews, so refusing every expired leaf meant a host whose leaf lapsed could NEVER renew and was
* bricked until a manual re-pair. A recently-expired leaf is now accepted for RE-ISSUANCE ONLY,
* inside a bounded window. Everything else stays fail-closed — the tests below pin that down.
*/
test('a leaf expired INSIDE the grace window renews → 201 (breaks the deadlock)', async () => {
const ctx = await hostCtx('alice')
const now = Date.now()
const expired = await mintHostLeaf(ctx, new Date(now - 2 * DAY_MS), new Date(now - DAY_MS))
const expired = await mintHostLeaf(ctx, new Date(now - 9 * DAY_MS), new Date(now - 8 * DAY_MS))
const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
await app.ready()
const res = await app.inject({
@@ -502,6 +508,91 @@ describe('CP6c POST /renew — presented current-cert chain + expiry verificatio
headers: { 'x-client-cert': certHeader(expired) },
payload: { csr: b64Csr(ctx.csr) },
})
expect(res.statusCode).toBe(201)
})
test('a leaf expired BEYOND the grace window is rejected → 401', async () => {
const ctx = await hostCtx('alice')
const now = Date.now()
const stale = await mintHostLeaf(ctx, new Date(now - 60 * DAY_MS), new Date(now - 31 * DAY_MS))
const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
await app.ready()
const res = await app.inject({
method: 'POST',
url: '/renew',
headers: { 'x-client-cert': certHeader(stale) },
payload: { csr: b64Csr(ctx.csr) },
})
expect(res.statusCode).toBe(401)
})
test('grace 0 restores the strict fail-closed behaviour → 401', async () => {
const ctx = await hostCtx('alice')
const now = Date.now()
const expired = await mintHostLeaf(ctx, new Date(now - 2 * DAY_MS), new Date(now - DAY_MS))
const app = appWith(
hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer], expiredRenewGraceMs: 0 }),
)
await app.ready()
const res = await app.inject({
method: 'POST',
url: '/renew',
headers: { 'x-client-cert': certHeader(expired) },
payload: { csr: b64Csr(ctx.csr) },
})
expect(res.statusCode).toBe(401)
})
test('grace NEVER bypasses revocation — revoked host with an in-grace leaf → 403', async () => {
const ctx = await hostCtx('alice')
const now = Date.now()
const expired = await mintHostLeaf(ctx, new Date(now - 9 * DAY_MS), new Date(now - 8 * DAY_MS))
const host = await ctx.hosts.getHostBySubdomain('alice')
await ctx.hosts.setHostStatus(host!.hostId, 'revoked')
const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
await app.ready()
const res = await app.inject({
method: 'POST',
url: '/renew',
headers: { 'x-client-cert': certHeader(expired) },
payload: { csr: b64Csr(ctx.csr) },
})
expect(res.statusCode).toBe(403)
})
test('grace NEVER bypasses chain validation — untrusted CA + in-grace leaf → 401', async () => {
const ctx = await hostCtx('alice')
const rogueCa = await makeP256Ca('rogue-CA')
const now = Date.now()
const rogue = await mintHostLeaf(
ctx,
new Date(now - 9 * DAY_MS),
new Date(now - 8 * DAY_MS),
rogueCa.caSigner,
)
const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
await app.ready()
const res = await app.inject({
method: 'POST',
url: '/renew',
headers: { 'x-client-cert': certHeader(rogue) },
payload: { csr: b64Csr(ctx.csr) },
})
expect(res.statusCode).toBe(401)
})
test('grace applies ONLY to notAfter — a not-yet-valid leaf is still rejected → 401', async () => {
const ctx = await hostCtx('alice')
const now = Date.now()
const future = await mintHostLeaf(ctx, new Date(now + DAY_MS), new Date(now + 2 * DAY_MS))
const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
await app.ready()
const res = await app.inject({
method: 'POST',
url: '/renew',
headers: { 'x-client-cert': certHeader(future) },
payload: { csr: b64Csr(ctx.csr) },
})
expect(res.statusCode).toBe(401)
})