OPER-50
Post-incident: deadman_pipeline_health_stale triggered 2026-07-25T20:26:05.954Z
Body
## Context
Post-incident tombstone for `deadman_pipeline_health_stale` (SEV-2) fired 2026-07-25T20:26:05Z. `lastMetricAt=2026-07-24T17:55:29Z` — pipeline-health cron had not emitted a metric row for ~2h35m when the deadman switch tripped. Slack ping to `#dev-alerts-testing` failed to send (`ts=failed-[REDACTED-MRN]`).
## Acceptance Criteria
- [x] Confirm root cause is the same LEDGER_BEARER rotation window that produced OPE-1..OPE-4.
- [x] Confirm invariant has not fired again since [REDACTED-DOB] (6+ days silent).
- [x] Confirm systemic fix is in-flight or shipped.
- [x] Cancel with resolution note pointing at systemic fix.
## Original alert payload
```
{
"invariant": "deadman_pipeline_health_stale",
"severity": "SEV-2",
"evidence": {
"lastMetricAt": "2026-07-24T17:55:29.797Z",
"staleThresholdMinutes": 15,
"detectedBy": "worker-reaper deadman switch"
},
"slack": {
"channelId": "#dev-alerts-testing",
"ts": "failed-[REDACTED-MRN]"
}
}
```
## Resolution — cancel ([REDACTED-DOB])
**Root cause:** LEDGER_BEARER token rotation window ([REDACTED-DOB] → [REDACTED-DOB]). The pipeline-health cron path called auto-planner over HTTP with the rotated bearer; auto-planner returned 401, cron exited without emitting a metric row, deadman switch fired ~2h35m after the last emit.
**Not recurred:** invariant has been silent for 6 days ([REDACTED-DOB] → [REDACTED-DOB]), consistent with the LEDGER_BEARER outage ending naturally. Sibling post-incident tickets from the same window (OPE-1..OPE-4) were all canceled by the same rationale.
**Systemic fix in-flight:** PR #167 (`refactor(auto-planner): eliminate LEDGER_BEARER round-trip via in-process runPlanValidation`, OPER-59) removes the HTTP hop entirely — the cron can no longer be broken by a token rotation. Nothing to do on this ticket.
**Guardrail already shipped:** PR #166 (OPER-58) adds the `auto_planner_silent` SEV-1 invariant so a future silent-stall trips sooner than the 15-min deadman window.
Canceled by Justin (via Sweetness) after review [REDACTED-DOB].Attachments
Loading attachments…
Comments
Loading comments…