Operant Studio
OPER-182

fix(ci): migrate-prod silently skips bot-merged migration PRs

Band
MEDIUM
Suggested
Justin Cooke

Body

feature_id: FEAT-studio-ci-cd

## Context

`.github/workflows/migrate-prod.yml` triggers on `push` to main with paths `packages/database/prisma/schema/migrations/**`. This is the only trigger that runs `prisma migrate deploy` against prod Neon.

**Bug:** GitHub's anti-recursion rule suppresses `push` events fired by `GITHUB_TOKEN`. The fleet's `agent-automerge.yml` uses the default `GITHUB_TOKEN`, so every bot-merged migration PR silently skips `migrate-prod`. Vercel still deploys the newer app code with the newer generated Prisma client. Result: app expects columns that don't exist in prod DB, and every read path that includes the new columns returns 500 (`internal_error`) until someone manually invokes `migrate-prod` via `workflow_dispatch`.

**Observed impact [REDACTED-DOB]:** Five migrations accumulated without running: `oper_103_denormalize_teamid`, `oper_128_qa_member_role_reviewer`, `oper_148_dispatch_run_provenance`, `oper_qa_16_agent_members`, `oper_176_task_risk_tier`. Every `/api/studio/portico/tasks/*` GET/PATCH 500'd (`traceId: b77a75c5-1e2d-4a69-b5c5-af0607f1ec29` and others). Recovery: manual `workflow_dispatch` on `migrate-prod` at 18:32 UTC applied all 5 in ~40s.

**Why the 6-hour scheduled `drift-check` job didn't catch it in time:** It runs every 6 hours (`cron: 17 */6 * * *`), and even when it fires, it Slack-alerts but does not auto-apply. It also has a secondary bug where it exits 1 on ANY non-zero from `prisma migrate status`, including "pending migrations exist" (which is normal), so its alert signal has low information value. Companion ticket recommended to fix drift-check's grep logic to match the pre-flight step's discrimination between drift and pending.

**Design rationale for the current trigger:** The workflow's OPER-83 addition added the `drift-check` cron precisely as a safety net for the silent-skip class. That safety net has caught the silent-skip (Slack alerts have been firing since [REDACTED-DOB]) but nothing has auto-remediated. The right fix is to make the migrate job itself un-silent-skip-able.

## Acceptance Criteria

- [ ] Fix chosen from options below and implemented so that every merge to `main` that touches `packages/database/prisma/schema/migrations/**` triggers `migrate-prod`, regardless of whether the merge was performed by a bot token or a human.
- [ ] The fix works with `agent-automerge.yml`'s default `GITHUB_TOKEN` — do NOT require rotating in a PAT for the automerge flow.
- [ ] `migrate-prod` continues to run on human pushes as before (backward-compatible trigger).
- [ ] A regression test in the form of a small script or workflow assertion that catches "migration PR merged without migrate-prod firing" — e.g. a follow-up check that compares `packages/database/prisma/schema/migrations/**` diffs on main HEAD vs the last successful migrate-prod run and alerts on mismatch.
- [ ] Documentation added to `docs/runbooks/prisma-p3009-recovery.md` (or a new sibling runbook) describing the silent-skip failure mode, how to recognize it (500s on task endpoints, migrate status shows pending migrations), and the manual `workflow_dispatch` recovery.
- [ ] `.github/workflows/migrate-prod.yml` comment header updated to reflect the new trigger and the failure mode this fix guards.

## Fix options (pick one in the plan doc)

**A. `workflow_run` trigger on the automerge workflow.** Add `on.workflow_run` targeting `agent-automerge.yml` completed with conclusion=success, filtered by whether the merged PR touched migration files. `workflow_run` is not suppressed by anti-recursion. Cost: needs extra logic to inspect which files the merged PR touched, since `workflow_run` doesn't have `paths` filter.

**B. `pull_request_target: closed` trigger with `merged == true`.** Fires when a PR closes with merge=true, regardless of what token performed the merge. Cost: `pull_request_target` runs with base-branch permissions, needs careful review of what steps run.

**C. PAT-based push at merge time.** Have `agent-automerge.yml` do the merge via a PAT (not `GITHUB_TOKEN`), which fires normal push events. Cost: adds a PAT to rotate + broader blast radius on that PAT's scope.

**D. Direct trigger from automerge.** After `agent-automerge.yml` merges, have it explicitly `gh workflow run migrate-prod.yml` via API for PRs that touched migrations. Cost: adds a step to automerge, needs the token used to have `actions:write`.

PDT input recommended in the plan doc — this is a CI/CD trigger topology decision.

## Risk classification

Safeguard (Risk 8 / Reach 5). CI/CD workflow change that gates prod DB migrations. Failure mode: a bad trigger fires migrations spuriously, applies migrations twice, or double-triggers with `migrate-prod` running concurrently — the concurrency group `migrate-prod` mitigates the last one but doesn't prevent the first two. Or the new trigger silently misses a case (this is the current state, so the regression bar is "no worse than today"). Failure-path tests come first: test that a human push still triggers, test that a bot merge triggers, test that a non-migration bot merge does NOT trigger, test that two rapid merges serialize correctly.

## Depends on

None. Fix is independent.

## Blocks

Every future migration PR that gets bot-merged. Also implicitly unblocks: any future data-schema work OPER-176, OPER-177, OPER-128, etc. would have shipped without human intervention.

## Related tickets

- Companion ticket recommended: fix `drift-check` job's grep logic to match the pre-flight step (distinguish "drift" from "pending"). Currently fires false-positive Slack alerts on every pending migration. Same file, separate concern.

## Out of scope

- Rewriting `agent-automerge.yml` (out — this ticket is about the migrate trigger, not the merge flow)
- Adding auto-apply to the scheduled `drift-check` job (out — separate ticket)
- Migrating away from `GITHUB_TOKEN` in the automerge flow broadly (out — see fix option C, but this ticket accepts the constraint)
- Adding a Vercel deploy pre-migrate check (out — separate concern about deploy ordering)


---
<!-- admission-rejected -->
**Admission rejected (unknown_feature_id):** feature_id=FEAT-studio-ci-cd is not declared in the feature-map.md of any admission-enabled product owned by this team.

Valid feature_ids for product studio (products/studio/feature-map.md):

- FEAT-studio-agent-workers
- FEAT-studio-audit
- FEAT-studio-audit-log
- FEAT-studio-branch-protection-rulesets
- FEAT-studio-coverage-policy
- FEAT-studio-dependency-map-policy
- FEAT-studio-dispatcher
- FEAT-studio-env-protection
- FEAT-studio-fabricated-evidence-tripwire
- FEAT-studio-guard-inventory
- FEAT-studio-incident-runbook
- FEAT-studio-mutation-testing
- FEAT-studio-phi-guard
- FEAT-studio-pipeline-observability
- FEAT-studio-planning-validation
- FEAT-studio-platform
- FEAT-studio-product-manifest
- FEAT-studio-qa
- FEAT-studio-qa-projects
- FEAT-studio-review-rounds

... and 2 more; see products/studio/feature-map.md

Attachments

Loading attachments…

Comments

Loading comments…