feat: voice command mapping — context-gated approve/reject over voice

While a tool-permission gate is held on the active tab, spoken confirm
phrases resolve it via the existing approve/reject WS channel — no server
change (byte-shuttle intact). Everything else stays ordinary dictation.

Safety: whole-utterance-exact matching + leading-negation guard + reject
precedence + confidence gate (approve only) + a cancellable 1.5s confirm
window whose commit re-validates gate identity/epoch/connectivity (TOCTOU
guard) + stale-gate epoch bound at PTT-start.

New pure modules public/voice-commands.ts and public/voice-confirm.ts at
100% coverage; wiring in voice/terminal-session/tabs/main. 1307 tests pass,
typecheck clean, build:web OK. Independent code + security reviews flagged a
confirm-window TOCTOU and an unwired cancel path — both fixed and covered.

Plan: docs/PLAN_VOICE_COMMANDS.md.
This commit is contained in:
Yaojia Wang
2026-07-01 10:34:51 +02:00
parent 03612323c0
commit e4c327e25e
13 changed files with 1193 additions and 19 deletions

222
docs/PLAN_VOICE_COMMANDS.md Normal file
View File

@@ -0,0 +1,222 @@
# Voice Command Mapping — Implementation Plan (v1, LOCKED)
Context-gated **confirm-class** voice commands: while a permission is HELD on the
active tab, spoken confirm/deny phrases resolve the gate via the existing
`approve`/`reject` WS channel. Produced by a multi-agent plan+review pass
(understand → FE/BE plan → 4 adversarial review lenses → synthesis) and hardened
against 2 CRITICAL + 1 HIGH findings. This doc is the source of intent for build.
## 0. Locked decisions
| # | Decision | Choice |
|---|----------|--------|
| 1 | Disambiguation | **Context-aware** — command only when active tab `pendingApproval===true`; else ordinary dictation (byte-for-byte unchanged) |
| 2 | Scope v1 | **Confirm-class only** — affirmative→`{type:'approve'}`, negative→`{type:'reject'}`, reusing the existing H3/A1 channel (never raw bytes) |
| 3 | Danger keys | **Not wired** — Ctrl+C/D/kill out of scope; two-step affordance built but only used for the approve confirm window (B) |
| A | Plan-gate | **tool-gate only**`gate!=='tool'` falls through to dictation |
| B | Approve confirm window | **Adopted (B2-lite)** — approve arms a ~1.5s cancellable window, then commits; reject fires immediately (fail-safe) |
| C | WS-not-open feedback | Flash "未连接 / not connected" (the confirm overlay already exists) |
| D | Confidence gate | **Adopted**`voice.ts` forwards ASR confidence; approve requires `≥ MIN_APPROVE_CONFIDENCE` (0.6); reject unaffected |
| E | Recognizer language | `navigator.language` (unchanged); document that zh/en sets are each only reachable under the matching engine language |
| F | tabs.ts overage | tabs.ts is already 869 lines (>800 max) — keep the delta minimal, **log a follow-up extraction ticket**, do not silently normalize |
## 1. Backend verdict — NONE
Zero changes to `src/server.ts`, `src/protocol.ts`, `src/types.ts`. The byte-shuttle stays intact.
- **Context signal already delivered:** a held `POST /hook/permission` broadcasts `{type:'status',status:'waiting',pending:true,gate:'tool'|'plan'}` (server.ts:446) and replays it on attach (server.ts:739-744); the client tracks it as `TerminalSession.pendingApproval` / `pendingGate` (terminal-session.ts:297-299).
- **Resolve channel already exists:** `{type:'approve'}` / `{type:'reject'}` are first-class `ClientMessage`s (types.ts:91-92), whitelisted (protocol.ts:27), parsed non-throwing (protocol.ts:77-82), resolved server-side keyed by `boundSessionId` (server.ts:761-771). No per-request token on the WS path.
- **Trust identical to the on-screen button:** a voice approve is one user action on the already-Origin-validated, attach-bound socket, same authority as the existing Approve button. No new WS rate-limit/capability.
- **Regression obligation (no new BE code):** `test/protocol.test.ts:111-117` and `test/integration/server.test.ts:608-646` must stay green.
## 2. Files & ownership
| File | Action | Lane |
|------|--------|------|
| `public/voice-commands.ts` | **create** — pure matcher (types, phrase sets, `normalizeTranscript`, `matchCommand`) | V1 |
| `test/voice-commands.test.ts` | **create** — pure unit tests (node env) | V1 |
| `public/voice-confirm.ts` | **create** — small cancellable-window controller for the approve confirm (B); timer injectable for tests | V1 |
| `test/voice-confirm.test.ts` | **create** — arm/commit/cancel/re-arm with fake timers | V1 |
| `public/voice.ts` | **modify** — forward `confidence` as 2nd `onTranscript` arg (still transcription-only) | V2 |
| `public/terminal-session.ts` | **modify** — add `pendingEpoch` nonce (stale-gate guard) | V2 |
| `public/tabs.ts` | **modify** — wire matcher into `onTranscript`; PTT-start epoch/session capture; dispatch; arm confirm window; WS-not-open flash | V2 |
| `public/main.ts` | **modify** — pass `onVoiceTrigger` to un-dormant the 🎤 | V2 |
| `test/tabs.test.ts` | **modify** — jsdom integration via `vi.mock('../public/voice.js')` | V2 |
| `vitest.config.ts` | **modify (orchestrator only)** — add `public/voice-commands.ts` + `public/voice-confirm.ts` to `coverage.include` | orch |
| `public/keybar.ts`, `src/**` | **untouched** | — |
V1 exports must be frozen before V2 consumes them. `PROGRESS_LOG.md` is orchestrator-only (subagents return a ready-to-paste entry, G1).
## 3. `public/voice-commands.ts` — pure matcher (DOM-free, WS-free)
```ts
export type VoiceAction = 'approve' | 'reject' | 'text'
export interface VoiceMatchContext {
readonly pendingApproval: boolean // active session held-permission flag
readonly gate: 'tool' | 'plan' | null // v1 acts ONLY on 'tool'
readonly confidence?: number // optional ASR confidence (D)
}
export const MIN_APPROVE_CONFIDENCE = 0.6
export const APPROVE_PHRASES: readonly string[] // normalized whole utterances
export const REJECT_PHRASES: readonly string[]
export function normalizeTranscript(raw: string): string
export function matchCommand(transcript: string, ctx: VoiceMatchContext): VoiceAction
```
**`normalizeTranscript`** (pure): `trim → toLowerCase → strip PUNCTUATION_RE (ASCII + CJK punctuation + apostrophes so don't→dont) → collapse ASCII \s+ → trim`. Keeps CJK + ASCII letters/digits.
**`matchCommand`** — whole-utterance-exact, reject-first, negation-guarded:
```
norm = normalizeTranscript(transcript)
if (!ctx.pendingApproval || norm === '') return 'text' // decision 1 + empty guard
if (ctx.gate !== 'tool') return 'text' // decision A: plan → dictation
if (startsWithNegation(norm)) // leading 不/别/没/未/勿, no/not/dont/cannot/wont/never
return REJECT_PHRASES.includes(norm) ? 'reject' : 'text' // negation can NEVER approve
if (REJECT_PHRASES.includes(norm)) return 'reject' // reject precedence (fail-safe)
if (APPROVE_PHRASES.includes(norm)) {
if (ctx.confidence !== undefined && ctx.confidence < MIN_APPROVE_CONFIDENCE) return 'text' // D
return 'approve'
}
return 'text' // dictation fallthrough
```
Whole-utterance-exact is the load-bearing fix: it kills both CRITICALs (`不可以``可以`; `ok let's not do that``ok`) at the root while keeping the module trivially testable. Cost: `yes please``text` (must say the command word alone or an enumerated multi-word phrase). Accepted precision-first trade-off for a shell-granting gate.
- `APPROVE_PHRASES`: `确认 批准 同意 通过 允许 可以 好 好的 好吧 好啊 是 是的 对 行 继续 回车 yes yeah yep ok okay confirm approve accept proceed "go ahead"`
- `REJECT_PHRASES`: `拒绝 取消 不行 不要 不用 不同意 不批准 不可以 不允许 不通过 中断 停止 算了 别 不 no nope reject deny cancel abort decline stop`
Invariants (tested): every phrase satisfies `normalizeTranscript(p)===p`; the two sets are disjoint; no `APPROVE_PHRASES` entry starts with a negation prefix.
## 4. `public/voice-confirm.ts` — approve confirm window (B)
Small controller with an injectable timer (so tests use fake timers, no DOM coupling in the logic):
```ts
export interface ApproveConfirm {
arm(onHeard: () => void, onCommit: () => void): void // show "听到:批准", start window
cancel(): void // user aborted before commit
isArmed(): boolean
}
export function createApproveConfirm(windowMs?: number): ApproveConfirm // default 1500
```
`arm` starts a `windowMs` timer that calls `onCommit` unless `cancel()` runs first; re-arming clears any prior timer. tabs.ts owns the tiny DOM overlay (a `#voice-confirm` sibling that `hideInterim` does not clobber) and calls `session.approve()` in `onCommit`.
**Cancel triggers (corrected during impl):** (a) **tapping the overlay**, (b) **a new utterance superseding** the armed window (the top of `handleVoiceTranscript` cancels first), and (c) **a reject utterance** (cancels the armed approve, then rejects). Note: "releasing the mic" is NOT a cancel trigger — under push-to-talk the mic release is exactly what delivers the final transcript that *arms* the window, so cancelling on release would make approve impossible.
**Commit-time re-validation (TOCTOU fix):** `onCommit` re-checks `armedSession.pendingApproval && pendingGate==='tool' && pendingEpoch===armedEpoch && connected` immediately before `approve()`; if the held gate resolved and a new one was raised (epoch bump), or the WS dropped, during the ~1.5s window, the commit is dropped rather than approving a different/unseen gate.
## 5. `public/terminal-session.ts` — stale-gate epoch
```ts
private pendingEpochValue = 0
get pendingEpoch(): number { return this.pendingEpochValue }
// inside status handling: count false→true flips
if (nextPending && !this.pendingApprovalValue) this.pendingEpochValue++
```
## 6. `public/tabs.ts` / `public/voice.ts` / `public/main.ts` — wiring (minimal delta)
- `voice.ts`: `onresult` forwards `result[0][0].confidence` as a 2nd arg to `onTranscript`. Still transcription-only.
- `tabs.ts`:
- On PTT start, capture `voicePttSession = active.session` and `voicePttEpoch = session.pendingEpoch`.
- `createVoiceInput((text, confidence) => this.handleVoiceTranscript(text, confidence), { onInterim })`.
- `handleVoiceTranscript`:
```
session = active.session; if (!session) return sendToActive(text)
action = matchCommand(text, { pendingApproval: session.pendingApproval===true, gate: session.pendingGate, confidence })
if (action!=='text' && (session!==voicePttSession || session.pendingEpoch!==voicePttEpoch)) return sendToActive(text) // stale → dictation
if (action==='reject') { session.reject(); updateApprovalBar(); return }
if (action==='approve') { approveConfirm.arm(showHeard, () => { session.approve(); updateApprovalBar() }); return } // B
sendToActive(text) // UNCHANGED dictation path
```
- WS-not-open (`session.approve/reject` dropped by `sendMsg`): flash "未连接" (C).
- Drop the earlier `flashVoiceCommand`/`VOICE_HINT_MS` idea — it duplicated the approval-bar vanish and raced with `hideInterim`.
- `main.ts:46`: `mountKeybar((d)=>app.sendToActive(d), { onVoiceTrigger:(a)=>app.handleVoiceTrigger(a) })` — activates the whole voice path (dictation + commands); note in log.
## 7. Security safeguards adopted
1. Whole-utterance-exact matching (no substring/token) — no auto-approve from incidental dictation.
2. Leading-negation override — negated phrases can never approve.
3. Reject precedence — fail-safe direction wins ties.
4. tool-gate-only (A) — avoids plan-gate semantic mismatch.
5. Stale-gate epoch (session+epoch bound at PTT-start) — a slow transcript can't approve a newer gate / another tab.
6. Approve confirm window (B) — cancellable ~1.5s before the irreversible approve commits.
7. Confidence gate on approve only (D) — asymmetric-risk aware.
8. Strict `pendingApproval===true` gate — never triggers the server's spurious `status:'working'` broadcast.
9. XSS-safe UI — `textContent` only.
10. SEC-L2 disclosure extended: voice *control* of permission gates (not just dictation) streams audio to the browser's cloud speech provider.
## 8. TDD steps (RED-first)
1. **RED** `test/voice-commands.test.ts` — normalize + matchCommand incl. the negation/filler regression set → fails (module absent).
2. **GREEN** implement `public/voice-commands.ts` → `npx vitest run voice-commands` green; `npm run typecheck` clean.
3. **RED** `test/voice-confirm.test.ts` (fake timers: arm→commit after window, cancel before window, re-arm clears prior) → fails.
4. **GREEN** implement `public/voice-confirm.ts` → green.
5. **RED** `test/tabs.test.ts` — `vi.mock('../public/voice.js')` capturing `onTranscript`; integration cases → fails.
6. **GREEN** edit `voice.ts`, `terminal-session.ts`, `tabs.ts`, `main.ts` → `npx vitest run tabs` green.
7. **REFACTOR** — `matchCommand`<50 lines, no console.log, no mutation, `readonly` types; log the pre-existing tabs.ts overage + extraction follow-up.
8. **COVERAGE** (orchestrator) — add modules to `vitest.config.ts` `coverage.include`; `npx vitest run --coverage` meets 80%×4.
9. **SHIP-CHECK** — `npm run build:web` bundles the new modules via the tabs.ts import.
### Key test cases
- **Context gating:** `matchCommand('yes',{pendingApproval:false,...})`==='text'; same for '确认'.
- **Held tool approve:** 确认/批准/同意/好/好的/好吧/行/可以/yes/ok/confirm/approve/"go ahead" → 'approve'.
- **Held tool reject:** 拒绝/取消/不行/no/nope/cancel/deny/abort/stop → 'reject'.
- **Negation/filler regression (MUST NOT approve):** 不可以→'reject'; 不允许/不通过/不同意/不批准→'reject'; 不继续→'text'; "don't approve"/"do not confirm"/"cannot proceed"/"won't continue"→'text'; "ok let's not do that"→'text'; "ok can you also check the logs"→'text'; 我觉得可以先看看→'text'; 请继续→'text' (≠继续); now→'text' (≠no); 我不知道→'text'.
- **Plan gate (A):** `{pendingApproval:true,gate:'plan'}` → 'text' for both 'confirm' and 'stop'.
- **Confidence (D):** confirm@0.4→'text'; confirm@0.9→'approve'; no@0.2→'reject' (reject unaffected).
- **Normalize:** 'YES'→'yes'; 'confirm.'→'confirm'; '「确认」'→'确认'; "don't"→'dont'; ' '→''.
- **tabs integration:** pending+tool+'confirm'→arms confirm window, commits `session.approve()` once after window, `send` NOT called; +'拒绝'→`session.reject()` immediately; +'deploy to prod'→`send('deploy to prod')`; pending=false+'yes'→`send('yes')`; plan gate+'confirm'→`send('confirm')`; stale-gate (epoch flip mid-utterance)→dictation; tab-switch mid-utterance→dictation; WS-not-open→no throw + "未连接" flash.
## 9. Verification
```bash
npx vitest run voice-commands # pure matcher
npx vitest run voice-confirm # confirm-window controller
npx vitest run tabs # wiring/integration (jsdom + voice.js mock)
npx vitest run protocol # BE regression: approve/reject parse (stay green)
npx vitest run integration/server # BE regression: held-permission → pending → approve resolves
npm test # full suite
npm run typecheck # frozen src/types.ts unchanged
npx vitest run --coverage # 80%×4 after coverage.include update
npm run build:web # esbuild bundles new modules via tabs.ts import
```
## 10. Manual verification (real device)
The command-dispatch logic (matchCommand → approve/reject/dictation, confirm window,
stale-gate, WS-drop) is fully covered by jsdom integration tests via the `voice.js`
mock. What CANNOT be automated is the browser's Web Speech engine itself (it needs
real microphone audio; there is no reliable headless E2E for it) — so the end-to-end
path is verified by hand on a real device.
**Setup:** Chrome on a phone (Web Speech / `webkitSpeechRecognition` supported), on the
same LAN/Tailscale as the host. Open a session and give Claude Code a task that raises a
**tool** permission gate (e.g. a shell command it must ask to run).
**Checklist:**
1. **Approve (happy path):** when the approve/reject bar appears, hold 🎤, say "确认"
(or "confirm"), release → overlay shows **"听到:批准(点此取消)"** → after ~1.5s the
gate is approved and the bar disappears.
2. **Tap-to-cancel:** say "确认", release, then **tap the overlay** within 1.5s → overlay
clears, gate is **not** approved.
3. **Reject:** say "拒绝" (or "no") → rejected immediately (no window).
4. **Reject cancels a pending approve:** say "确认", then within 1.5s say "拒绝" → rejected,
and the pending approve does **not** fire afterwards.
5. **Context gate (dictation preserved):** with **no** gate pending, say "确认" → the literal
text "确认" is dictated into the terminal, not treated as a command.
6. **Plan gate falls through:** on a *plan* gate, say "确认" → dictated as text, not
auto-approved (use the on-screen buttons for plan gates).
7. **Negation safety:** with a tool gate pending, say "不可以" / "不用" → rejected or
dictated, **never** approved.
8. **Disconnect feedback:** kill Wi-Fi briefly (WS reconnecting), say "确认" → overlay
flashes **"未连接"**, nothing is approved.
Report the device + Chrome version with results, since real recognition of the zh vs en
phrase sets depends on the engine's active language.

View File

@@ -24,6 +24,9 @@
> 新会话读到的第一块。保持准确,只描述"此刻"。
### ✅ VC 语音命令映射 — 语音直接 approve/reject 权限门(完成 — 2026-07-01,分支 `v0.6-projects`,未提交)
上下文门控:仅当活动 tab 有 held 权限门(`pendingApproval===true``gate==='tool'`)时,语音「确认/批准/yes…」→ 复用现有 `{type:'approve'}` WS 通道、「拒绝/取消/no…」→ `{type:'reject'}`;其余一律口述不变。**后端零改动**。安全设计:整句精确匹配(否定/口水词攻不破)+ 前导否定守卫 + reject 优先 + 置信度门(仅 approve)+ **可撤销确认窗口(1.5s,提交时复检门身份/连接,防 TOCTOU)** + stale-gate epoch。新增 `public/voice-commands.ts`(纯匹配器 100% 覆盖)+ `public/voice-confirm.ts`(确认窗口 100%),接线 `tabs/voice/terminal-session/main.ts`。**1307 tests 绿**,双 review(code+security)判 CRITICAL 已全修复并补测。规划 `docs/PLAN_VOICE_COMMANDS.md`,详见 Detailed Log 首条。**待办**:confidence fail-open 取舍留给用户;tabs.ts 968 行待拆分。
### ✅ v0.6 Projects 命名空间自动分组 + 跨设备 prefs(完成 — 2026-07-01,分支 `v0.6-projects`,未提交)
Projects 页去杂乱:按点分隔命名空间自动折叠成 section(`Billo.Platform ·8` / `Billo.Infrastructure ·2` / `Other`)+ 置顶 "Active now" + 无前缀时退化为扁平网格。收藏/折叠态改**服务端持久化**(`GET/PUT /prefs``~/.web-terminal-prefs.json`,离线回退 localStorage 镜像 + 迁移旧 `proj-favs`)跨设备同步。1227 tests 绿,真浏览器验证折叠跨 reload 从服务端载回。详见 Detailed Log 首条。
@@ -149,7 +152,25 @@ Projects 页去杂乱:按点分隔命名空间自动折叠成 section(`Billo.Pla
- **commit**: <hash 或 N/A>
======================================================= -->
### 2026-07-01 · v0.6 Projects 页去杂乱 — 命名空间自动分组 + "Active now" + 跨设备 prefs
### 2026-07-01 · VC 语音命令映射 — 上下文门控 confirm/reject(多 agent 编排 + 双 review)
- **状态**: `[x]` DONE(未提交,分支 `v0.6-projects`)
- **背景**: 已有 v0.7 A2 push-to-talk 只做「语音→文字」口述;本次加**语音命令层**,让语音直接 approve/reject 权限门。规划见 `docs/PLAN_VOICE_COMMANDS.md`(多 agent:理解→前后端计划→4 路对抗 review→综合)。锁定决策:①上下文感知(仅当活动 tab `pendingApproval===true` 才当命令,否则口述不变)②只做 confirm 类,复用现有 `approve`/`reject` WS 通道(**后端零改动**)③危险键不接线,two-step 组件仅用于 approve 确认窗口。
- **改动**:
- 新增 `public/voice-commands.ts`(纯匹配器,**100% 覆盖**):`matchCommand` = 整句精确 + 前导否定守卫 + reject 优先 + 置信度门(仅 approve);中英确认/否定词表。
- 新增 `public/voice-confirm.ts`(确认窗口控制器,**100% 覆盖**):`createApproveConfirm(1500)` arm/cancel/isArmed,纯计时(注入 setTimeout,无 DOM)。
- `public/voice.ts`:`onTranscript` 多传 `confidence`(仍只转写)。`public/terminal-session.ts`:加 `pendingEpoch`(false→true 翻转自增,stale-gate nonce)。`public/tabs.ts`:`handleVoiceTranscript` 分派 + 确认窗口 arm + WS 断开闪「未连接」+ 单独 `#voice-confirm` overlay(避 hideInterim 清屏)。`public/main.ts`:接 `onVoiceTrigger` 点亮 🎤。`vitest.config.ts`:两新模块入 coverage.include(orchestrator 协调编辑)。
- 测试:`test/voice-commands.test.ts`/`test/voice-confirm.test.ts` 新建;`test/tabs.test.ts` 加 VC 集成(含 6 条 review-fix 用例);`test/voice.test.ts` 4 处旧断言随 `(text,confidence)` 新签名更新 + 补 confidence 转发断言。
- **验证**: `npm run typecheck` 干净;`npm test`**1307 passed / 43 files**;`build:web` OK;覆盖率 voice-commands/voice-confirm **100×4**、tabs.ts 94/84/89/97、TOTAL 91.3/84.6/91.7/93.2(80×4 通过)。`git diff src/` 空(后端 NONE 成立)。
- **决策 / 偏离 PLAN**:
- **实现 workflow 首跑失败**(review 阶段某 agent 6 次 stall)——但 Build V1/V2 已落文件;改为 orchestrator 亲自跑验证 + 用普通 agent 并行做 code/security review(避开 workflow 卡死)。
- **两份 review 独立收敛,判 CRITICAL/BLOCK**,均已修复并补测:①**确认窗口 TOCTOU**:`onCommit` 现复检 `pendingApproval && gate==='tool' && pendingEpoch===armedEpoch && connected`,否则丢弃;②**`cancel()` 从未接线**:接入新话语顶替(handler 顶部)、reject 分支、`#voice-confirm` 点击;③**WS 断开延迟提交错位**:提交前复检连接。
- **偏离 PLAN §4**:「松开麦克风取消」在 PTT 时序下不成立(松麦克风正是触发最终转录/arm 的动作),改为 **tap + 新话语顶替 + reject 取消** 三条真实取消路径。
- **`startsWithNegation` 重写**(review MEDIUM):原 `prefix+rest[0]` 实际只测 prefix[0](巧对但脆弱),改为显式测 `rest`
- **遗留 / 待办**:
- **confidence fail-open**(review MEDIUM,**保留**):`confidence===undefined` 时置信度门短路(符合 §D「有置信度时才门控」),靠确认窗口兜底误听;若要 fail-safe(undefined→不批准)是一行改动,但会让不报告置信度的引擎(部分非 Chromium)语音批准失效——留给用户定。
- **`public/tabs.ts` 已 968 行 > 800 上限**(既有债 + 本次 ~99 行):本次仅加最小 diff,**待后续抽 `voice-controller` 拆分**(decision F)。
- **commit**: N/A(未提交)
- **状态**: `[x]` DONE。双 typecheck 干净 · `build:web` OK · 全量 `vitest run` **1227 passed (41 files)** · 真浏览器端到端验证通过。
- **需求**(用户): Projects 页扁平网格太乱(50100 repo)。多 agent 探索(4 路:自动分组/手动文件夹/密度分段/竞品调研)结论一致 → **主方向=按点分隔命名空间自动分组**(repo 名如 `Billo.Platform.Payment` 天然带层级),配 **Active-now 置顶**;手动文件夹只是加分项非主线。用户选:**范围=自动分组+Active band**,**持久化=服务端**(收藏/折叠态跨设备同步,修掉 localStorage 每设备各一份的缺陷)。