OPER-177
AI triage writer + AiTriageClassPolicy + AiTriageLog + auto-demote/promote
Band
LARGE
Suggested
Justin Cooke
Body
feature_id: FEAT-studio-qa
## Context
PDT memo v5 ([REDACTED-DOB]) approved the auto-dispatch policy: Sandbox / Scaffold / Safeguard tickets auto-dispatch on CI-passing; Stronghold requires human approval. This ticket implements the writer that classifies failed TestRun evidence into a fix ticket + dispatch decision, plus the config + log tables that hold the policy state.
Tester latency is the goal metric — target end-to-end 5-13 minutes on the 95% case with zero human clicks. Safety comes from CI as ship-gate, PHI floor override (routed via OPER-176), BLOCKER severity override, 3-consecutive-rerun-failure auto-demote, 3-consecutive-successful-rerun auto-promote, and the manual Cancel button on /qa/live-fixes.
## Acceptance Criteria
- New table AiTriageClassPolicy (four columns, no thresholds): `(fix_class TEXT, tier RiskTier, mode AiDispatchMode, updated_at, updated_by)`. Composite PK on (fix_class, tier). Enum AiDispatchMode { AUTO, HUMAN }.
- New table AiTriageLog: `(id, kind AiTriageLogKind, fix_class TEXT?, tier RiskTier?, task_id TEXT?, test_run_id TEXT?, actor TEXT, payload_json JSONB, created_at)`. Enum AiTriageLogKind { DISPATCH_AUTO, DISPATCH_HUMAN, RERUN_PASS, RERUN_FAIL, DEMOTE, PROMOTE, CANCEL }.
- New module `packages/qa/src/ai-triage.ts` with function `triageFailedRun(testRun, evidence): TriageDecision`. Steps:
1. Classify fix_class from evidence patterns (COPY_CHANGE, NULL_CHECK, MISSING_LABEL, VALIDATION_MISSING, STYLE_REGRESSION, etc. — closed vocabulary defined in code).
2. Call deriveRiskTier() from packages/qa/src/risk-tier.ts (from OPER-176) to compute tier.
3. Look up AiTriageClassPolicy for (fix_class, tier). If mode is AUTO AND severity != BLOCKER: dispatch_mode = AUTO. Otherwise HUMAN.
4. Write ticket to Ledger via existing OPER-129 writer with tier + fix_class stamped on the Task. Status starts as PLANNING_VALIDATED if AUTO, AWAITING_HUMAN if HUMAN.
5. Log DISPATCH_AUTO or DISPATCH_HUMAN row in AiTriageLog.
- Rerun subscriber module `packages/qa/src/rerun-watcher.ts` — subscribes to rerun.ts (OPER-175) events. For each RERUN_PASS / RERUN_FAIL on an auto-dispatched fix: write matching AiTriageLog row. Then check the last 3 rerun outcomes for that (fix_class, tier):
- If 3 consecutive FAILED: update AiTriageClassPolicy row to mode=HUMAN, write AiTriageLog kind=DEMOTE with payload_json containing the 3 failed run ids + task ids.
- If (fix_class, tier) is currently HUMAN AND the last 3 rerun outcomes on human-dispatched fixes are PASSED: update mode=AUTO, write AiTriageLog kind=PROMOTE.
- Cancel API endpoint `POST /api/studio/qa/live-fixes/:taskId/cancel` — flips Task status to AWAITING_HUMAN, writes AiTriageLog kind=CANCEL with actor stamp `user:<id>`.
- Failure-path tests BEFORE code (Safeguard tier). Minimum 20 test cases covering: classify never returns unknown fix_class without logging, PHI floor override always wins, BLOCKER override always wins, missing AiTriageClassPolicy row defaults to HUMAN (fail-safe), demote fires on exactly-3 consecutive failures not 2 not 4, promote fires on exactly-3 consecutive successes on human-dispatched fixes only, race between simultaneous demote and promote resolved deterministically, Cancel on already-merged ticket is a no-op with logged reason.
- AuditLog row on every AiTriageClassPolicy row mutation. Actor stamps: `system:qa-ai-triage`, `system:qa-rerun-watcher`, `user:<id>` for Cancel.
## Risk classification
Safeguard (Risk 8 / Reach 6). Autonomously creates tickets AND autonomously modifies dispatch policy. Silently wrong means bad fixes shipping automatically or good classes stuck in HUMAN forever. Failure-path tests before code, AuditLog on every state change, plus the auto-demote circuit breaker as an in-loop safety net.
## Depends on
- OPER-175 (rerun watcher) — emits the RERUN_PASS / RERUN_FAIL events this module subscribes to.
- OPER-176 (Task.risk_tier schema) — provides RiskTier enum + deriveRiskTier() helper this module calls.
## Blocks
- Playwright trace recorder (next ticket — its output is the primary input to this triager)
- /qa/triage-log page (reads from AiTriageLog table created here)
- /qa/live-fixes canvas (reads from Task rows with dispatch_mode=AUTO created here)Attachments
Loading attachments…
Comments
Loading comments…