web-terminal/docs/PLAN_VOICE_COMMANDS.md

Voice Command Mapping — Implementation Plan (v1, LOCKED)

Context-gated confirm-class voice commands: while a permission is HELD on the active tab, spoken confirm/deny phrases resolve the gate via the existing approve/reject WS channel. Produced by a multi-agent plan+review pass (understand → FE/BE plan → 4 adversarial review lenses → synthesis) and hardened against 2 CRITICAL + 1 HIGH findings. This doc is the source of intent for build.

0. Locked decisions

#	Decision	Choice
1	Disambiguation	Context-aware — command only when active tab pendingApproval===true; else ordinary dictation (byte-for-byte unchanged)
2	Scope v1	Confirm-class only — affirmative→{type:'approve'}, negative→{type:'reject'}, reusing the existing H3/A1 channel (never raw bytes)
3	Danger keys	Not wired — Ctrl+C/D/kill out of scope; two-step affordance built but only used for the approve confirm window (B)
A	Plan-gate	tool-gate only — gate!=='tool' falls through to dictation
B	Approve confirm window	Adopted (B2-lite) — approve arms a ~1.5s cancellable window, then commits; reject fires immediately (fail-safe)
C	WS-not-open feedback	Flash "未连接 / not connected" (the confirm overlay already exists)
D	Confidence gate	Adopted — voice.ts forwards ASR confidence; approve requires ≥ MIN_APPROVE_CONFIDENCE (0.6); reject unaffected
E	Recognizer language	navigator.language (unchanged); document that zh/en sets are each only reachable under the matching engine language
F	tabs.ts overage	tabs.ts is already 869 lines (>800 max) — keep the delta minimal, log a follow-up extraction ticket, do not silently normalize

1. Backend verdict — NONE

Zero changes to src/server.ts, src/protocol.ts, src/types.ts. The byte-shuttle stays intact.

• Context signal already delivered: a held POST /hook/permission broadcasts {type:'status',status:'waiting',pending:true,gate:'tool'|'plan'} (server.ts:446) and replays it on attach (server.ts:739-744); the client tracks it as TerminalSession.pendingApproval / pendingGate (terminal-session.ts:297-299).
• Resolve channel already exists: {type:'approve'} / {type:'reject'} are first-class ClientMessages (types.ts:91-92), whitelisted (protocol.ts:27), parsed non-throwing (protocol.ts:77-82), resolved server-side keyed by boundSessionId (server.ts:761-771). No per-request token on the WS path.
• Trust identical to the on-screen button: a voice approve is one user action on the already-Origin-validated, attach-bound socket, same authority as the existing Approve button. No new WS rate-limit/capability.
• Regression obligation (no new BE code): test/protocol.test.ts:111-117 and test/integration/server.test.ts:608-646 must stay green.

2. Files & ownership

File	Action	Lane
public/voice-commands.ts	create — pure matcher (types, phrase sets, normalizeTranscript, matchCommand)	V1
test/voice-commands.test.ts	create — pure unit tests (node env)	V1
public/voice-confirm.ts	create — small cancellable-window controller for the approve confirm (B); timer injectable for tests	V1
test/voice-confirm.test.ts	create — arm/commit/cancel/re-arm with fake timers	V1
public/voice.ts	modify — forward confidence as 2nd onTranscript arg (still transcription-only)	V2
public/terminal-session.ts	modify — add pendingEpoch nonce (stale-gate guard)	V2
public/tabs.ts	modify — wire matcher into onTranscript; PTT-start epoch/session capture; dispatch; arm confirm window; WS-not-open flash	V2
public/main.ts	modify — pass onVoiceTrigger to un-dormant the 🎤	V2
test/tabs.test.ts	modify — jsdom integration via vi.mock('../public/voice.js')	V2
vitest.config.ts	modify (orchestrator only) — add public/voice-commands.ts + public/voice-confirm.ts to coverage.include	orch
public/keybar.ts, src/**	untouched	—

V1 exports must be frozen before V2 consumes them. PROGRESS_LOG.md is orchestrator-only (subagents return a ready-to-paste entry, G1).

3. public/voice-commands.ts — pure matcher (DOM-free, WS-free)

export type VoiceAction = 'approve' | 'reject' | 'text'

export interface VoiceMatchContext {
  readonly pendingApproval: boolean        // active session held-permission flag
  readonly gate: 'tool' | 'plan' | null    // v1 acts ONLY on 'tool'
  readonly confidence?: number             // optional ASR confidence (D)
}

export const MIN_APPROVE_CONFIDENCE = 0.6

export const APPROVE_PHRASES: readonly string[]   // normalized whole utterances
export const REJECT_PHRASES: readonly string[]
export function normalizeTranscript(raw: string): string
export function matchCommand(transcript: string, ctx: VoiceMatchContext): VoiceAction

normalizeTranscript (pure): trim → toLowerCase → strip PUNCTUATION_RE (ASCII + CJK punctuation + apostrophes so don't→dont) → collapse ASCII \s+ → trim. Keeps CJK + ASCII letters/digits.

matchCommand — whole-utterance-exact, reject-first, negation-guarded:

norm = normalizeTranscript(transcript)
if (!ctx.pendingApproval || norm === '') return 'text'      // decision 1 + empty guard
if (ctx.gate !== 'tool') return 'text'                       // decision A: plan → dictation
if (startsWithNegation(norm))                                // leading 不/别/没/未/勿, no/not/dont/cannot/wont/never
  return REJECT_PHRASES.includes(norm) ? 'reject' : 'text'   //   negation can NEVER approve
if (REJECT_PHRASES.includes(norm)) return 'reject'           // reject precedence (fail-safe)
if (APPROVE_PHRASES.includes(norm)) {
  if (ctx.confidence !== undefined && ctx.confidence < MIN_APPROVE_CONFIDENCE) return 'text'  // D
  return 'approve'
}
return 'text'                                                // dictation fallthrough

Whole-utterance-exact is the load-bearing fix: it kills both CRITICALs (不可以≠可以; ok let's not do that≠ok) at the root while keeping the module trivially testable. Cost: yes please→text (must say the command word alone or an enumerated multi-word phrase). Accepted precision-first trade-off for a shell-granting gate.

• APPROVE_PHRASES: 确认 批准 同意 通过 允许 可以 好 好的 好吧 好啊 是 是的 对 行 继续 回车 yes yeah yep ok okay confirm approve accept proceed "go ahead"
• REJECT_PHRASES: 拒绝 取消 不行 不要 不用 不同意 不批准 不可以 不允许 不通过 中断 停止 算了 别 不 no nope reject deny cancel abort decline stop

Invariants (tested): every phrase satisfies normalizeTranscript(p)===p; the two sets are disjoint; no APPROVE_PHRASES entry starts with a negation prefix.

4. public/voice-confirm.ts — approve confirm window (B)

Small controller with an injectable timer (so tests use fake timers, no DOM coupling in the logic):

export interface ApproveConfirm {
  arm(onHeard: () => void, onCommit: () => void): void  // show "听到:批准", start window
  cancel(): void                                        // user aborted before commit
  isArmed(): boolean
}
export function createApproveConfirm(windowMs?: number): ApproveConfirm  // default 1500

arm starts a windowMs timer that calls onCommit unless cancel() runs first; re-arming clears any prior timer. tabs.ts owns the tiny DOM overlay (a #voice-confirm sibling that hideInterim does not clobber) and calls session.approve() in onCommit.

Cancel triggers (corrected during impl): (a) tapping the overlay, (b) a new utterance superseding the armed window (the top of handleVoiceTranscript cancels first), and (c) a reject utterance (cancels the armed approve, then rejects). Note: "releasing the mic" is NOT a cancel trigger — under push-to-talk the mic release is exactly what delivers the final transcript that arms the window, so cancelling on release would make approve impossible.

Commit-time re-validation (TOCTOU fix): onCommit re-checks armedSession.pendingApproval && pendingGate==='tool' && pendingEpoch===armedEpoch && connected immediately before approve(); if the held gate resolved and a new one was raised (epoch bump), or the WS dropped, during the ~1.5s window, the commit is dropped rather than approving a different/unseen gate.

5. public/terminal-session.ts — stale-gate epoch

private pendingEpochValue = 0
get pendingEpoch(): number { return this.pendingEpochValue }
// inside status handling: count false→true flips
if (nextPending && !this.pendingApprovalValue) this.pendingEpochValue++

6. public/tabs.ts / public/voice.ts / public/main.ts — wiring (minimal delta)

• voice.ts: onresult forwards result[0][0].confidence as a 2nd arg to onTranscript. Still transcription-only.
• tabs.ts:
  • On PTT start, capture voicePttSession = active.session and voicePttEpoch = session.pendingEpoch.
  • createVoiceInput((text, confidence) => this.handleVoiceTranscript(text, confidence), { onInterim }).
  • handleVoiceTranscript:

    session = active.session; if (!session) return sendToActive(text)
    action = matchCommand(text, { pendingApproval: session.pendingApproval===true, gate: session.pendingGate, confidence })
    if (action!=='text' && (session!==voicePttSession || session.pendingEpoch!==voicePttEpoch)) return sendToActive(text)  // stale → dictation
    if (action==='reject') { session.reject(); updateApprovalBar(); return }
    if (action==='approve') { approveConfirm.arm(showHeard, () => { session.approve(); updateApprovalBar() }); return }  // B
    sendToActive(text)   // UNCHANGED dictation path
    

  • WS-not-open (session.approve/reject dropped by sendMsg): flash "未连接" (C).
  • Drop the earlier flashVoiceCommand/VOICE_HINT_MS idea — it duplicated the approval-bar vanish and raced with hideInterim.
• main.ts:46: mountKeybar((d)=>app.sendToActive(d), { onVoiceTrigger:(a)=>app.handleVoiceTrigger(a) }) — activates the whole voice path (dictation + commands); note in log.

7. Security safeguards adopted

1. Whole-utterance-exact matching (no substring/token) — no auto-approve from incidental dictation.
2. Leading-negation override — negated phrases can never approve.
3. Reject precedence — fail-safe direction wins ties.
4. tool-gate-only (A) — avoids plan-gate semantic mismatch.
5. Stale-gate epoch (session+epoch bound at PTT-start) — a slow transcript can't approve a newer gate / another tab.
6. Approve confirm window (B) — cancellable ~1.5s before the irreversible approve commits.
7. Confidence gate on approve only (D) — asymmetric-risk aware.
8. Strict pendingApproval===true gate — never triggers the server's spurious status:'working' broadcast.
9. XSS-safe UI — textContent only.
10. SEC-L2 disclosure extended: voice control of permission gates (not just dictation) streams audio to the browser's cloud speech provider.

8. TDD steps (RED-first)

1. RED test/voice-commands.test.ts — normalize + matchCommand incl. the negation/filler regression set → fails (module absent).
2. GREEN implement public/voice-commands.ts → npx vitest run voice-commands green; npm run typecheck clean.
3. RED test/voice-confirm.test.ts (fake timers: arm→commit after window, cancel before window, re-arm clears prior) → fails.
4. GREEN implement public/voice-confirm.ts → green.
5. RED test/tabs.test.ts — vi.mock('../public/voice.js') capturing onTranscript; integration cases → fails.
6. GREEN edit voice.ts, terminal-session.ts, tabs.ts, main.ts → npx vitest run tabs green.
7. REFACTOR — matchCommand<50 lines, no console.log, no mutation, readonly types; log the pre-existing tabs.ts overage + extraction follow-up.
8. COVERAGE (orchestrator) — add modules to vitest.config.ts coverage.include; npx vitest run --coverage meets 80%×4.
9. SHIP-CHECK — npm run build:web bundles the new modules via the tabs.ts import.

Key test cases

• Context gating: matchCommand('yes',{pendingApproval:false,...})==='text'; same for '确认'.
• Held tool approve: 确认/批准/同意/好/好的/好吧/行/可以/yes/ok/confirm/approve/"go ahead" → 'approve'.
• Held tool reject: 拒绝/取消/不行/no/nope/cancel/deny/abort/stop → 'reject'.
• Negation/filler regression (MUST NOT approve): 不可以→'reject'; 不允许/不通过/不同意/不批准→'reject'; 不继续→'text'; "don't approve"/"do not confirm"/"cannot proceed"/"won't continue"→'text'; "ok let's not do that"→'text'; "ok can you also check the logs"→'text'; 我觉得可以先看看→'text'; 请继续→'text' (≠继续); now→'text' (≠no); 我不知道→'text'.
• Plan gate (A): {pendingApproval:true,gate:'plan'} → 'text' for both 'confirm' and 'stop'.
• Confidence (D): confirm@0.4→'text'; confirm@0.9→'approve'; no@0.2→'reject' (reject unaffected).
• Normalize: 'YES'→'yes'; 'confirm.'→'confirm'; '「确认」'→'确认'; "don't"→'dont'; ' '→''.
• tabs integration: pending+tool+'confirm'→arms confirm window, commits session.approve() once after window, send NOT called; +'拒绝'→session.reject() immediately; +'deploy to prod'→send('deploy to prod'); pending=false+'yes'→send('yes'); plan gate+'confirm'→send('confirm'); stale-gate (epoch flip mid-utterance)→dictation; tab-switch mid-utterance→dictation; WS-not-open→no throw + "未连接" flash.

9. Verification

npx vitest run voice-commands      # pure matcher
npx vitest run voice-confirm       # confirm-window controller
npx vitest run tabs                # wiring/integration (jsdom + voice.js mock)
npx vitest run protocol            # BE regression: approve/reject parse (stay green)
npx vitest run integration/server  # BE regression: held-permission → pending → approve resolves
npm test                           # full suite
npm run typecheck                  # frozen src/types.ts unchanged
npx vitest run --coverage          # 80%×4 after coverage.include update
npm run build:web                  # esbuild bundles new modules via tabs.ts import

10. Manual verification (real device)

The command-dispatch logic (matchCommand → approve/reject/dictation, confirm window, stale-gate, WS-drop) is fully covered by jsdom integration tests via the voice.js mock. What CANNOT be automated is the browser's Web Speech engine itself (it needs real microphone audio; there is no reliable headless E2E for it) — so the end-to-end path is verified by hand on a real device.

Setup: Chrome on a phone (Web Speech / webkitSpeechRecognition supported), on the same LAN/Tailscale as the host. Open a session and give Claude Code a task that raises a tool permission gate (e.g. a shell command it must ask to run).

Checklist:

1. Approve (happy path): when the approve/reject bar appears, hold 🎤, say "确认" (or "confirm"), release → overlay shows "听到：批准（点此取消）" → after ~1.5s the gate is approved and the bar disappears.
2. Tap-to-cancel: say "确认", release, then tap the overlay within 1.5s → overlay clears, gate is not approved.
3. Reject: say "拒绝" (or "no") → rejected immediately (no window).
4. Reject cancels a pending approve: say "确认", then within 1.5s say "拒绝" → rejected, and the pending approve does not fire afterwards.
5. Context gate (dictation preserved): with no gate pending, say "确认" → the literal text "确认" is dictated into the terminal, not treated as a command.
6. Plan gate falls through: on a plan gate, say "确认" → dictated as text, not auto-approved (use the on-screen buttons for plan gates).
7. Negation safety: with a tool gate pending, say "不可以" / "不用" → rejected or dictated, never approved.
8. Disconnect feedback: kill Wi-Fi briefly (WS reconnecting), say "确认" → overlay flashes "未连接", nothing is approved.

Report the device + Chrome version with results, since real recognition of the zh vs en phrase sets depends on the engine's active language.