smart-support/eng-review-test-plan.md

Test Plan

Generated by /plan-eng-review on 2026-03-29 Branch: unknown Repo: smart-support

Affected Pages/Routes

• WebSocket /ws — Main chat endpoint. Test connection, message flow, streaming, interrupt responses, reconnection
• GET / — Chat UI serving. Test static file serving
• GET /api/replay/{thread_id} — Conversation replay. Test pagination, 404, structured timeline JSON
• GET /api/analytics — Analytics dashboard. Test resolution rate, agent usage, escalation %, zero state
• POST (webhook URL) — Escalation webhook. Test payload shape, retry on failure

Key Interactions to Verify

• Send message via WebSocket → receive streaming tokens back
• Send message triggering write action → receive interrupt prompt → send approval → receive confirmation
• Send message triggering write action → receive interrupt prompt → send rejection → receive cancellation
• Multi-turn: send order lookup, then "cancel that one" → agent resolves reference from context
• Paste OpenAPI spec URL → tools generated → agents registered → chat works with new tools
• View conversation replay → see step-by-step agent decisions and tool calls
• View analytics dashboard → see resolution rate, agent usage metrics

Edge Cases

• Empty message → graceful error response
• Very long message (10K+ chars) → handled without crash
• Rapid-fire messages → no race condition in graph execution
• WebSocket disconnect mid-stream → server cleans up gracefully, client reconnects + resumes interrupt
• LLM API timeout → error message returned to client
• Invalid YAML agent config → clear startup error with file/line reference
• MCP tool timeout → timeout error returned to user with agent name
• Cancel already-cancelled order → appropriate error message
• Ambiguous intent with no context → clarifying question asked
• OpenAPI spec with 100+ endpoints → generation completes without timeout
• Invalid/malformed OpenAPI spec → clear error with what's wrong
• SSRF attempt (private IP, localhost, 169.254.x) → blocked with clear error
• DNS rebinding attack → blocked
• Replay of thread with 200+ turns → pagination works, no slow query
• Analytics with no conversations → zero state displayed correctly
• Webhook URL unreachable → retry with backoff, log failure
• Multi-intent request ("cancel my order and give me a discount") → sequenced correctly

Critical Paths (E2E)

• Happy path: "What's the status of order 1042?" → lookup → answer
• Cancel with approval: "Cancel order 1042" → interrupt → approve → cancel confirmed
• Cancel with rejection: "Cancel order 1042" → interrupt → reject → no action taken
• Multi-turn context: "Check order 1042" then "cancel that one" → correct entity resolution
• OpenAPI import: paste spec URL → tools generated → use new tool in chat
• Conversation replay: select completed conversation → step-by-step replay renders correctly