fix(tunnel): break the expired-leaf renewal deadlock

`POST /renew` is authenticated by mTLS with the very leaf it renews, so once
that leaf lapsed the host could never renew it and the tunnel stayed down until
an operator re-paired by hand. Production hit exactly this: the Mac slept
through its 8h renewal window, the 24h leaf expired, and the agent then logged
`client certificate has expired; renew before dialling` 6380 times over 8 days
without recovering.

Three layers independently refused an expired leaf, so all three had to move:

- agent `buildTlsOptions` gains an opt-in `expiredGraceMs`. Absent/0 keeps the
  historical fail-closed behaviour, and the TUNNEL dial never passes it — only
  the renew transport does. Past the window it throws the new
  `CertExpiredBeyondGraceError`.
- agent rotator routes an already-expired leaf to a separate recovery endpoint
  and treats "beyond grace" as TERMINAL: report once via `onExhausted`, stop
  retrying, and name the fix (re-pair) instead of spamming warnings forever.
- control-plane `assertPresentedCertTrusted` grants a bounded grace on
  `notAfter` only. Chain validation, SPIFFE identity, `notBefore`, and the
  registry `active`/account checks are all unchanged, so revocation still bites.
- new `deploy/nginx/recover-mtls.conf` (:8472). The strict enroll vhost cannot
  host this: under `ssl_verify_client optional` nginx answers a bare 400 as soon
  as a presented cert fails verification, so an expired leaf never reaches the
  location — and the directive is server-level, not per-location.

Grace defaults to 30 days on both ends and is configurable (`RECOVER_URL`,
`expiredRenewGraceMs`). The honest trade is recorded in the code: a stale stolen
leaf stays reusable for the window, which widens an existing exposure (an
unexpired stolen leaf already renews indefinitely) rather than opening a new one.

Verified: agent 296/296, control-plane 286/286, tsc clean on both.

control-plane/test/renew.test.ts

@@ -490,10 +490,16 @@ describe('CP6c POST /renew — presented current-cert chain + expiry verificatio
     expect(res.statusCode).toBe(201)
   })
 
-  test('an EXPIRED current cert (valid chain, notAfter in the past) is rejected → 401', async () => {
+  /**
+   * POLICY CHANGE (expired-leaf deadlock fix): `/renew` is mTLS-authenticated by the very leaf it
+   * renews, so refusing every expired leaf meant a host whose leaf lapsed could NEVER renew and was
+   * bricked until a manual re-pair. A recently-expired leaf is now accepted for RE-ISSUANCE ONLY,
+   * inside a bounded window. Everything else stays fail-closed — the tests below pin that down.
+   */
+  test('a leaf expired INSIDE the grace window renews → 201 (breaks the deadlock)', async () => {
     const ctx = await hostCtx('alice')
     const now = Date.now()
-    const expired = await mintHostLeaf(ctx, new Date(now - 2 * DAY_MS), new Date(now - DAY_MS))
+    const expired = await mintHostLeaf(ctx, new Date(now - 9 * DAY_MS), new Date(now - 8 * DAY_MS))
     const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
     await app.ready()
     const res = await app.inject({
@@ -502,6 +508,91 @@ describe('CP6c POST /renew — presented current-cert chain + expiry verificatio
       headers: { 'x-client-cert': certHeader(expired) },
       payload: { csr: b64Csr(ctx.csr) },
     })
+    expect(res.statusCode).toBe(201)
+  })
+
+  test('a leaf expired BEYOND the grace window is rejected → 401', async () => {
+    const ctx = await hostCtx('alice')
+    const now = Date.now()
+    const stale = await mintHostLeaf(ctx, new Date(now - 60 * DAY_MS), new Date(now - 31 * DAY_MS))
+    const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
+    await app.ready()
+    const res = await app.inject({
+      method: 'POST',
+      url: '/renew',
+      headers: { 'x-client-cert': certHeader(stale) },
+      payload: { csr: b64Csr(ctx.csr) },
+    })
+    expect(res.statusCode).toBe(401)
+  })
+
+  test('grace 0 restores the strict fail-closed behaviour → 401', async () => {
+    const ctx = await hostCtx('alice')
+    const now = Date.now()
+    const expired = await mintHostLeaf(ctx, new Date(now - 2 * DAY_MS), new Date(now - DAY_MS))
+    const app = appWith(
+      hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer], expiredRenewGraceMs: 0 }),
+    )
+    await app.ready()
+    const res = await app.inject({
+      method: 'POST',
+      url: '/renew',
+      headers: { 'x-client-cert': certHeader(expired) },
+      payload: { csr: b64Csr(ctx.csr) },
+    })
+    expect(res.statusCode).toBe(401)
+  })
+
+  test('grace NEVER bypasses revocation — revoked host with an in-grace leaf → 403', async () => {
+    const ctx = await hostCtx('alice')
+    const now = Date.now()
+    const expired = await mintHostLeaf(ctx, new Date(now - 9 * DAY_MS), new Date(now - 8 * DAY_MS))
+    const host = await ctx.hosts.getHostBySubdomain('alice')
+    await ctx.hosts.setHostStatus(host!.hostId, 'revoked')
+    const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
+    await app.ready()
+    const res = await app.inject({
+      method: 'POST',
+      url: '/renew',
+      headers: { 'x-client-cert': certHeader(expired) },
+      payload: { csr: b64Csr(ctx.csr) },
+    })
+    expect(res.statusCode).toBe(403)
+  })
+
+  test('grace NEVER bypasses chain validation — untrusted CA + in-grace leaf → 401', async () => {
+    const ctx = await hostCtx('alice')
+    const rogueCa = await makeP256Ca('rogue-CA')
+    const now = Date.now()
+    const rogue = await mintHostLeaf(
+      ctx,
+      new Date(now - 9 * DAY_MS),
+      new Date(now - 8 * DAY_MS),
+      rogueCa.caSigner,
+    )
+    const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
+    await app.ready()
+    const res = await app.inject({
+      method: 'POST',
+      url: '/renew',
+      headers: { 'x-client-cert': certHeader(rogue) },
+      payload: { csr: b64Csr(ctx.csr) },
+    })
+    expect(res.statusCode).toBe(401)
+  })
+
+  test('grace applies ONLY to notAfter — a not-yet-valid leaf is still rejected → 401', async () => {
+    const ctx = await hostCtx('alice')
+    const now = Date.now()
+    const future = await mintHostLeaf(ctx, new Date(now + DAY_MS), new Date(now + 2 * DAY_MS))
+    const app = appWith(hostDeps(ctx, { hostCaAnchorsDer: [ctx.ca.caDer] }))
+    await app.ready()
+    const res = await app.inject({
+      method: 'POST',
+      url: '/renew',
+      headers: { 'x-client-cert': certHeader(future) },
+      payload: { csr: b64Csr(ctx.csr) },
+    })
     expect(res.statusCode).toBe(401)
   })