OPER-7A
Pipeline self-heal Phase A: detection + metrics + alerts (DOC-approved)
Body
DOC Council-of-9 unanimous approval. Detection-only, no auto-heal. Deliverables: 1. Extend pipeline-health cron (*/5) with 3 invariants: - #1 IN_PROGRESS tasks with no matching workflow_dispatch run within 60s (SEV-3) - #2 IN_PROGRESS tasks with no branch in target repo (SEV-3) - #3 Queue liveness: PLANNING_VALIDATED>0 AND inFlight<cap AND no new dispatches in 15min (SEV-2) 2. PipelineHealthMetric table + PipelineHealthMonthlyRollup (30d TTL + monthly aggregate) 3. DispatchAudit table per workflow_dispatch attempt 4. Emit via unified audit-event path + structured JSON logs 5. Deadman switch on pipeline-health 6. GitHub-API-health circuit-breaker on checkBranchExists (distinguish missing vs check-failed) 7. Slack alerts to team-scoped channels via TeamAlertRouting map: - Portico team -> #alerts-portico-pipeline - Operant team -> #alerts-operant-pipeline - Coalesced, thread-replied on recurrence, dev-mode flag 8. Auto-file post-incident ticket on SEV-2 9. Runbook docs/operations/pipeline-runbook.md 10. SLO doc docs/operations/pipeline-slos.md (99% dispatched within 5min; 4x burn alert) Follow-up tickets: OPER-7B (auto-heal, tomorrow), OPER-8 (dashboard UI), OPER-9 (PAT rotation alerts).
Attachments
Loading attachments…
Comments
Loading comments…