Nightly queue_liveness incidents recurring + Slack alerts silently broken 5 days

---
feature_id: FEAT-studio-pipeline-observability
---

## Context

`queue_liveness` invariant has fired an AWAITING_HUMAN auto-filed incident ticket **every night for five nights straight** ([REDACTED-DOB] through [REDACTED-DOB]), at approximately the same time (02:00-02:30 UTC). None of the five has been triaged or auto-healed.

- OPER-144 — 2026-09-09T02:05:11Z (CANCELED, not healed)
- OPER-172 — 2026-09-10T02:10:08Z (AWAITING_HUMAN)
- OPER-192 — 2026-09-11T02:10:08Z (AWAITING_HUMAN)
- OPER-203 — 2026-09-12T02:15:08Z (AWAITING_HUMAN)
- OPER-204 — 2026-09-13T02:30:08Z (AWAITING_HUMAN)

OPER-45 (queue_liveness auto-heal runbook) is marked DONE but is clearly not covering this signature — the runbook was written to auto-heal, and it is now just auto-filing a rescope ticket every night that never gets picked up.

### Evidence from most recent incident (OPER-204, 2026-09-13T02:30:08Z)

```json
{
  "invariant": "queue_liveness",
  "severity": "SEV-2",
  "evidence": {
    "planningValidated": 4,
    "inFlight": 2,
    "maxInFlight": 15,
    "recentDispatchCount": 0,
    "windowMinutes": 15,
    "lastSuccessfulDispatchAt": "2026-09-13T02:12:51.174Z",
    "dispatchAttemptsSampled": 30,
    "dispatchSuccessesInSample": 30
  },
  "slack": {
    "channelId": "#dev-alerts-testing",
    "ts": "failed:slack_post_failed:_channel_not_found-[REDACTED-MRN]"
  }
}
```

Two independent bugs surfaced by this evidence:

### Bug 1: Nightly dispatcher sleep at 02:00-02:30 UTC

Evidence: 30 dispatch attempts, all succeeded (`dispatchSuccessesInSample: 30`), but zero dispatches in the 15-minute window before the alert fired despite 4 PLANNING_VALIDATED tickets and only 2 in flight (13 slots of capacity available).

Root cause hypothesis: a scheduled job or maintenance task running at 02:00 UTC (Vercel cron overlap? Neon compute cold-start after autoscale-to-zero window? A GitHub Actions runner limit resetting on UTC midnight rollover?) is starving the auto-planner. The consistent time-of-day signature is the fingerprint.

Diagnostic paths:
- Cross-reference against Vercel cron schedule for jobs firing at 02:00 UTC daily
- Cross-reference against Neon autoscale logs (compute suspend/resume at midnight UTC)
- Check auto-planner lock table for entries held across 02:00 UTC boundary
- Check GitHub Actions runner quota reset time (some plans reset at UTC midnight)

### Bug 2: Broken Slack alert delivery

Evidence: `"slack": {"channelId": "#dev-alerts-testing", "ts": "failed:slack_post_failed:_channel_not_found-[REDACTED-MRN]"}`

The Slack alert has been failing on `channel_not_found` for at least 5 days consecutively. The channel `#dev-alerts-testing` either does not exist or the bot is not a member. This means even if a human WERE watching alerts, they would not see these.

Diagnostic:
- Verify `#dev-alerts-testing` exists in the Operant Slack workspace
- Verify the Studio Slack bot is invited to that channel
- Verify `OPERANT_SLACK_ALERTS_CHANNEL` env var is set correctly (probably should point at `#studio-alerts` or `#dispatcher-incidents`, not a test channel)

## What

Three-part fix, one ticket, three PRs in a stack:

### PR 1 (S / observability) — diagnose the nightly sleep

- Add per-tick auto-planner instrumentation at `apps/web/src/lib/auto-planner.ts`: emit `PlannerTickEvent` with `startedAt`, `finishedAt`, `readyCount`, `inFlightCount`, `dispatchedCount`, `skippedReason` (if any) into `AuditLog`.
- Add a Vercel cron at `/api/cron/planner-tick-report` that runs at 02:45 UTC (right after the incident window) and posts a summary of the previous night's planner activity to the Slack alerts channel.
- Verify the fix by observing 3 consecutive nights of planner activity across 02:00 UTC without an incident (or with an incident + full diagnostic evidence).

### PR 2 (XS / observability) — fix the Slack alert channel

- Change `OPERANT_SLACK_ALERTS_CHANNEL` env var from `#dev-alerts-testing` to `#studio-alerts` (or whichever channel is the active operations channel).
- Add a startup check at `apps/web/src/lib/slack-alerts.ts` that calls `conversations.info` on the configured channel at boot and logs a loud error if the channel is unreachable — no more silent 5-day failures.
- Add a health-check probe at `/api/health/slack` that returns 500 if the alerts channel is unreachable.

### PR 3 (S / observability) — retire OPER-45's auto-file behavior when auto-heal is not actually healing

- OPER-45 runbook fires `queue_liveness` alerts and auto-files an incident ticket. If the runbook has ALSO tried to auto-heal (e.g., unstick a claim, retry a dispatch) and the heal failed, that is the signal to escalate.
- Today the runbook files a ticket every time regardless of whether it attempted a heal.
- Change: auto-file the ticket ONLY when auto-heal was attempted and failed. If the invariant fires and auto-heal was not attempted (because no heal path exists for the signature), the ticket body should say so explicitly rather than pointing at OPER-45 as if it had tried.
- Also: dedupe. If an incident with the same signature (`invariant + evidence.windowMinutes + hour_of_day`) fired in the last 7 days, comment on the existing ticket instead of filing a new one.

## Acceptance Criteria

- [ ] PR 1: PlannerTickEvent emitted on every planner tick with the fields above
- [ ] PR 1: /api/cron/planner-tick-report cron posts nightly summary to Slack
- [ ] PR 1: 3 consecutive nights with 02:00 UTC diagnostic evidence captured
- [ ] PR 2: OPERANT_SLACK_ALERTS_CHANNEL env var updated to a live channel
- [ ] PR 2: Startup Slack channel-reachability check logs loud error on failure
- [ ] PR 2: /api/health/slack probe added
- [ ] PR 3: Runbook only auto-files when auto-heal was attempted and failed
- [ ] PR 3: Body distinguishes "no heal attempted" from "heal attempted and failed"
- [ ] PR 3: Dedupe by signature over 7-day window (comment vs. new ticket)
- [ ] Regression test: mock 5 consecutive same-signature invariant fires, assert 1 ticket + 4 comments

## Blocked-by

None. OPER-45 (queue_liveness auto-heal runbook DONE) provides the substrate to extend.

## Size / Band

M / assurance (3 PRs in a stack; each individual PR is S or XS).

## Discovered by

Session 2026.09.13 dispatcher status check. 5 consecutive nightly incidents, none triaged, alerts silently broken for 5 days.

## Verification

Ships when: (1) 3 consecutive nights pass without a queue_liveness incident firing, OR one fires and has full PlannerTickEvent + Slack delivery + dedupe evidence; (2) Slack health probe returns 200; (3) runbook body clearly says "auto-heal attempted / not attempted".