Operant Studio
OPER-67

Runbook: plan_rejected_missing_fields burst — triage and rollback path

Suggested
Justin Cooke

Body

feature_id: FEAT-studio-dispatcher

## Context

OPER-60 (PR #177, merged [REDACTED-DOB]) shipped the admission gate v0.1. When a ticket fails admission, the runner writes a `PlanningRejection` row and emits an `AuditLog(action="plan_rejected_missing_fields")` event. On the current dashboard, this shows up as a burst of `plan_rejected_missing_fields` audits — a signal that the auto-planner is refusing to promote work.

The signal has three legitimate causes:
1. **A real backlog of undersized tickets** — the healthy case OPER-60 was built for.
2. **A bad batch import** — someone bulk-imported N tickets without the required fields (see OPER-57 for the last time this happened; the recovery took an afternoon).
3. **A bug in `admission` itself** — a regex tightening, a schema drift, or a stale worker snapshot rejecting tickets that should pass.

Priya (Incident Commander) flagged during the OPER-60 memo review: "there is no runbook. When the on-caller sees 40 `plan_rejected_missing_fields` audits in an hour at 2am, they will not know whether to page an engineer, page the person who did the last import, or ignore it."

This ticket ships the runbook.

## Scope

Add `docs/runbooks/plan-rejected-missing-fields.md` covering, in this order:

1. **Signal definition** — what an "unusual burst" is (concrete threshold: >10 `plan_rejected_missing_fields` audits in 15 minutes for a single team, or >30 in 1 hour across all teams). Cite the AuditLog action name and the `PlanningRejection.reasons[]` shape.
2. **Read-only triage steps** — the exact SQL / API calls the on-caller runs first:
   - `SELECT reason, COUNT(*) FROM "PlanningRejection" WHERE "createdAt" > now() - interval '1 hour' GROUP BY reason ORDER BY 2 DESC;` — which reason is dominating.
   - `SELECT identifier, reasons, "createdAt" FROM "PlanningRejection" pr JOIN "Task" t ON t.id = pr."taskId" ORDER BY pr."createdAt" DESC LIMIT 20;` — which tickets are affected.
   - `git log --oneline apps/web/src/lib/planning-validator.ts apps/web/src/lib/writesets-extractor.ts apps/web/src/lib/phi-reviewer-gate.ts --since=1.day` — did anything change in the gates recently.
3. **Decision tree** — three branches, one for each root cause:
   - "one reason dominates + tickets are all from the last hour" -> likely a bad import; find the last `POST /api/studio/[team]/tasks/bulk` and correlate.
   - "reasons are diverse + steady rate" -> legitimate backlog; no action, close alert.
   - "reasons are diverse + sharp cliff" -> likely a gate regression; roll back the last deploy that touched the admission path.
4. **Rollback path** — the exact `gh` command to identify the last admission-touching PR and the exact revert-and-redeploy sequence.
5. **Escalation** — who to page (Justin) and after what threshold (>60 minutes without root-cause identified).

## Acceptance Criteria

- Runbook file exists at `docs/runbooks/plan-rejected-missing-fields.md`.
- Every SQL query in the runbook has been executed against the current production schema at least once (paste the actual-column-names version, not a guess).
- The rollback path names a real `gh` command (`gh pr revert <n>` or the workflow_dispatch equivalent), not a placeholder.
- The runbook is linked from `docs/runbooks/README.md` (or created there if the file does not yet exist) under an "Admission" section.
- No new alert wiring in this ticket — the alert may or may not exist yet; if it does not, that is OPER-68's problem, not this one.

## Non-goals

- Creating the alert / detector that fires on the threshold. That is a separate ticket (see OPER-68).
- Automating any part of the triage. Runbooks are for humans until we have signal that the diagnosis is deterministic.
- Any change to the admission gate's runtime behavior.

## Verification

`grep -l plan-rejected-missing-fields docs/runbooks/README.md` returns the file. All SQL queries in the runbook execute successfully against a fresh `pnpm db:generate` schema (paste output).

Attachments

Loading attachments…

Comments

Loading comments…