OPER-58
Add auto_planner_silent + pipeline_throughput_below_target invariants
Body
# Context The 2026-07-17 -> 2026-07-27 silent stall proved that when the auto-planner falls into an error path BEFORE writing any PlanningValidation row (e.g. rotated LEDGER_BEARER_<team>), no existing invariant catches it. The pipeline can be dark for 10 days and still show green. # Scope Add two new pipeline-health invariants: 1. auto_planner_silent (SEV-1). Fires when a team has >= 3 upstream tickets AND zero PlanningValidation rows have been written in the last 30 minutes. 2. pipeline_throughput_below_target (SEV-2). Fires when upstream depth >= 10 AND merged-Task throughput < 2/hr for a sustained 120 min window. Both must read pure DB (Task counts, PlanningValidation) so they stay measurable when the GitHub API is degraded. # Deliverables - apps/web/src/lib/planner-throughput-invariants.ts with two evaluators + env-tunable thresholds - Wire both into apps/web/src/app/api/cron/pipeline-health/route.ts (write metric row every tick, emit alert when violated) - Runbook sections in docs/operations/pipeline-runbook.md with diagnosis-in-order, remedy, and env tunables - Unit tests covering empty-queue no-op, heartbeat no-op, incident-shape fires, threshold edges - Daily 8am uptime report cron updated to include OPER-58 baseline coverage section # Acceptance - SQL logic applied to prod snapshot from 2026-07-17 shows auto_planner_silent would have fired at ~16:49 UTC and stayed violated the full 10 days. - Zero new typecheck errors vs main. - Tomorrow morning's uptime report shows probe counts for both invariants in the new baseline coverage section. blocked_by: [] module: pipeline-health phi_in_scope: false verification_cmd: pnpm --filter @operant/web test src/lib/__tests__/planner-throughput-invariants.test.ts has_migration: false
Attachments
Loading attachments…
Comments
Loading comments…