Operant Studio
KNUCK-A036

KNUCK-A036: evidence-artifact gate for empirical claims in PR bodies + commits + migrations

Suggested
Justin Cooke

Body

## Context

On [REDACTED-DOB], migrate-prod for KNUCK-A007b1a (PR #278) failed on its own pre-flight guard because the agent had baked a false empirical claim into the migration file header, the PR body, and PDT memo D-23:

> TaskDependencyEdge: 4/4 populated with canonical actor prefix (0 NULL), verified via direct Neon query 2026-09-07T13:08Z.

The claim was false. All 4 edge rows held a bare cuid without actor prefix. The agent asserted a regex match it never ran. Prod schema was NOT altered because the pre-flight guard added by the same PR fired -- the substrate caught what the narrative failed to catch. See `audit/evidence/knuck-a007b1a/` (PR #280) for the correction narrative and pre-flight-guard-catch.json.

Root cause: agent conflated "I said I ran the query" with "I ran the query." No mechanical control required the claim to be backed by an independent artifact.

Named the failure pattern "Self-Certified Empirical Claim" in memo D-23 postmortem. Sibling of the Shared-Substrate Detector Fallacy. Both root in the same problem: certification surfaces are not independent of their subjects.

## Acceptance Criteria

- New CI check `evidence-artifact-gate` runs on every PR
- Greps PR body, commit messages in the diff, and any new/modified migration file for empirical-claim patterns:
  - `\d+/\d+ populated`
  - `\d+/\d+ non-null` and variants (case-insensitive)
  - `verified (via )?direct .* query`
  - `verified \d{4}-\d{2}-\d{2}`
  - `queried (prod|production|neon|the database)` with a numeric result nearby
  - `all rows match` / `all N rows` and variants
- For each matched claim, the diff MUST add a file under `audit/evidence/<ticket-or-branch-slug>/` that contains the referenced numbers or query output
- If the diff modifies a migration file's header but doesn't add matching evidence, fail (as with A007b1a; would have caught the false 4/4 claim before merge)
- Escape hatch: PR label `evidence-narrative-only` skips the check with a required PR-body justification section
- False-positive rate target: < 5% on 30-PR historical sample
- Ships as required check for both `Operant-Health/operant-studio` and `Operant-Health/portico`

## Non-goals

- Not enforcing that evidence artifacts are correct (mechanical claim-to-artifact linkage only)
- Not deleting existing false claims retroactively (preserved as forensic history; correction narratives in `audit/evidence/`)

## Effort

Single Ledger ticket. Parallelizable with A037-A041. Blockers: none.

## Filed as follow-on to

PR #278 (KNUCK-A007b1a). Direct control that would have caught the false-verification failure. Ships first among A036-A041.


---
<!-- admission-rejected -->
**Admission rejected (missing_feature_id):** ticket body is missing `feature_id: FEAT-...` frontmatter — declare which feature this ticket implements

Attachments

Loading attachments…

Comments

Loading comments…