Operant Studio
OPER-58

Add auto_planner_silent + pipeline_throughput_below_target invariants

Body

# Context

The 2026-07-17 -> 2026-07-27 silent stall proved that when the auto-planner falls into an error path BEFORE writing any PlanningValidation row (e.g. rotated LEDGER_BEARER_<team>), no existing invariant catches it. The pipeline can be dark for 10 days and still show green.

# Scope

Add two new pipeline-health invariants:

1. auto_planner_silent (SEV-1). Fires when a team has >= 3 upstream tickets AND zero PlanningValidation rows have been written in the last 30 minutes.
2. pipeline_throughput_below_target (SEV-2). Fires when upstream depth >= 10 AND merged-Task throughput < 2/hr for a sustained 120 min window.

Both must read pure DB (Task counts, PlanningValidation) so they stay measurable when the GitHub API is degraded.

# Deliverables

- apps/web/src/lib/planner-throughput-invariants.ts with two evaluators + env-tunable thresholds
- Wire both into apps/web/src/app/api/cron/pipeline-health/route.ts (write metric row every tick, emit alert when violated)
- Runbook sections in docs/operations/pipeline-runbook.md with diagnosis-in-order, remedy, and env tunables
- Unit tests covering empty-queue no-op, heartbeat no-op, incident-shape fires, threshold edges
- Daily 8am uptime report cron updated to include OPER-58 baseline coverage section

# Acceptance

- SQL logic applied to prod snapshot from 2026-07-17 shows auto_planner_silent would have fired at ~16:49 UTC and stayed violated the full 10 days.
- Zero new typecheck errors vs main.
- Tomorrow morning's uptime report shows probe counts for both invariants in the new baseline coverage section.

blocked_by: []
module: pipeline-health
phi_in_scope: false
verification_cmd: pnpm --filter @operant/web test src/lib/__tests__/planner-throughput-invariants.test.ts
has_migration: false

Attachments

Loading attachments…

Comments

Loading comments…