queue_liveness false-positive-on-idle: fires when dispatcher is provably healthy

---
feature_id: FEAT-studio-pipeline-observability
---

## Context

`queue_liveness` fires SEV-2 post-incidents on the Portico team every day even when the dispatcher is completely healthy. Post-OPER-MONITOR-DRIFT-1 evidence ([REDACTED-DOB] through [REDACTED-DOB]):

- `PipelineHealthMetric` rows for Portico show `queue_liveness violated=true` on most 5-minute ticks
- The alert bodies carry `dispatchAttemptsSampled: 30` + `dispatchSuccessesInSample: 30` — the dispatcher tried 30 dispatches in the sample window and all 30 succeeded
- The invariant still fires because the trigger condition (route.ts:624-629) has no check that the dispatcher is actually broken

**The trigger reads:**

```ts
const queueDead =
  !githubDegraded &&
  !workflowMissing &&
  planningValidated > 0 &&
  inFlight < maxInFlight &&
  recentDispatchCount === 0;   // <-- only signal on "did we dispatch in the last 15 min"
```

`recentDispatchCount === 0` is true whenever a natural quiet window falls inside the 15-min sample — e.g., the planner is between waves, or all PLANNING_VALIDATED tickets are still under admission review. The invariant treats "queue is quiet" and "queue is dead" as the same thing.

The self-close mechanism (OPER-MONITOR-DRIFT-1 PR 3-5, merged [REDACTED-DOB]) closes the post-incident tickets within 15 minutes once the invariant goes green — so the pile stays bounded — but the underlying invariant is still firing false alarms every day, filing tickets that immediately auto-close and generating churn in the daily digest.

## Root cause

The invariant conflates two signals:
1. **Traffic** — is the queue currently dispatching?
2. **Health** — is the dispatcher actually able to dispatch?

Only signal 2 should trigger `queue_liveness`. Signal 1 alone is a normal-operation quiet window.

The evidence for signal 2 is already computed and passed into the alert body: `dispatchAttemptsSampled` and `dispatchSuccessesInSample` come from `getDispatchFreshness()` (route.ts:955-957). If `attemptsSampled > 0 && successesInSample === attemptsSampled`, the dispatcher is provably healthy — no false alarm.

## Fix

Add a dispatcher-health gate to the `queueDead` predicate (route.ts:624-629). Do not fire `queue_liveness` when the recent dispatch attempts we sampled all succeeded.

```ts
const dispatcherProvablyHealthy =
  freshness.attemptsSampled > 0 &&
  freshness.successesInSample === freshness.attemptsSampled;

const queueDead =
  !githubDegraded &&
  !workflowMissing &&
  planningValidated > 0 &&
  inFlight < maxInFlight &&
  recentDispatchCount === 0 &&
  !dispatcherProvablyHealthy;   // <-- new guard
```

**Design notes:**

- `attemptsSampled > 0` gate is important: with zero attempts sampled we have no evidence either way, and the existing `recentDispatchCount === 0` signal is the best we have.
- The gate leaves the invariant able to fire on genuinely dead queues where nothing has been dispatched recently (attemptsSampled == 0) — that's the original OPER-7B failure mode and it should still alarm.
- The gate does NOT touch the "no PLANNING_VALIDATED work" case (`planningValidated > 0` is unchanged) — an empty queue is still not a "dead" queue.

## Known-bad test (KNUCK-A032 discipline)

Add to `docs/monitor-failure-modes.md` per the OPER-MONITOR-DRIFT-1 registry:

- **failure mode name:** `queue_liveness_false_positive_on_idle`
- **shape:** healthy dispatcher (attemptsSampled == successesInSample, both > 0) + no dispatches in the sample window (recentDispatchCount == 0)
- **known-bad test:** vitest case in `apps/web/__tests__/pipeline-health/queue-liveness-guard.test.ts` that constructs the "provably healthy, but quiet" input and asserts `queueDead === false`

## Effort

Single Ledger ticket. One-file change in `apps/web/src/app/api/cron/pipeline-health/route.ts` plus a new test file plus a registry row. Not parallelizable.

## Verification

After merge:
1. `PipelineHealthMetric` rows for Portico with `queue_liveness violated=true` should drop to near-zero
2. No new `POR-*` or `OPER-*` tickets with label `pi:queue_liveness` filed during a period where `dispatchSuccessesInSample == dispatchAttemptsSampled > 0`
3. The known-bad self-test in `check-monitor-selftests.yml` (from OPER-MONITOR-DRIFT-1 PR 5/5) covers this failure mode

## Related

- OPER-MONITOR-DRIFT-1 (DONE) — fixed the stale-cache bug and the phantom-task bug, added the self-close mechanism. Did not touch this false-positive-on-idle bug.
- OPER-209 (in progress) — dedup shim so recurring incidents thread onto one ticket instead of filing daily. Complementary but not a substitute; the right fix is the invariant not firing in the first place.

## Acceptance Criteria

- [ ] `apps/web/src/app/api/cron/pipeline-health/route.ts` `queueDead` predicate adds `!dispatcherProvablyHealthy` gate as specified in the "Fix" section
- [ ] `dispatcherProvablyHealthy` is defined as `freshness.attemptsSampled > 0 && freshness.successesInSample === freshness.attemptsSampled`
- [ ] New vitest file `apps/web/__tests__/pipeline-health/queue-liveness-guard.test.ts` constructs the "provably healthy, quiet queue" input and asserts `queueDead === false`
- [ ] New vitest asserts genuinely-dead case (attemptsSampled == 0) still triggers `queueDead === true`
- [ ] Row added to `docs/monitor-failure-modes.md` for `queue_liveness_false_positive_on_idle`
- [ ] Known-bad self-test wired into `check-monitor-selftests.yml` (per OPER-MONITOR-DRIFT-1 PR 5/5 registry pattern)
- [ ] No functional change to `queueDead` behavior when `attemptsSampled == 0` (preserves original OPER-7B alarm)

---
<!-- planner-rescope: missing_size, missing_verify_cmd, missing_module -->
**Planner rescope (3 reasons):**
- **missing_size** — Task.size is required (XS/S/M/L/XL) — the dispatcher budgets on it.
- **missing_verify_cmd** — Task.verificationCmd is required — without it there is no post-build proof to run.
- **missing_module** — Task.module is required for project/package routing.