KNUCK-A036b
KNUCK-A036: evidence-artifact gate for empirical claims in PR bodies + commits + migrations
Suggested
Justin Cooke
Body
feature_id: FEAT-studio-audit ## Context On 2026.09.08 04:17Z, migrate-prod for KNUCK-A007b1a (PR #278) failed on its own pre-flight guard because the agent had baked a false empirical claim into the migration file header, the PR body, and PDT memo D-23: > TaskDependencyEdge: 4/4 populated with canonical actor prefix (0 NULL), verified via direct Neon query 2026.09.07 13:08Z. The claim was false. All 4 edge rows held a bare cuid without actor prefix. The agent asserted a regex match it never ran. Prod schema was NOT altered because the pre-flight guard added by the same PR fired -- the substrate caught what the narrative failed to catch. See `audit/evidence/knuck-a007b1a/` (PR #280) for the correction narrative and pre-flight-guard-catch.json. Root cause: agent conflated "I said I ran the query" with "I ran the query." No mechanical control required the claim to be backed by an independent artifact. Named the failure pattern "Self-Certified Empirical Claim" in memo D-23 postmortem. Sibling of the Shared-Substrate Detector Fallacy. Both root in the same problem: certification surfaces are not independent of their subjects. Related to `FEAT-studio-audit` because the evidence-artifact convention (`audit/evidence/<ticket>/`) is the durable, human-inspectable audit substrate that would have blocked the false claim before merge. ## Acceptance Criteria - New CI check `evidence-artifact-gate` runs on every PR - Greps PR body, commit messages in the diff, and any new or modified migration file for empirical-claim patterns: - `\d+/\d+ populated` - `\d+/\d+ non-null` and variants (case-insensitive) - `verified (via )?direct .* query` - `verified <ISO date>` (any date form) - `queried (prod|production|neon|the database)` with a numeric result nearby - `all rows match` / `all N rows` and variants - For each matched claim, the diff MUST add a file under `audit/evidence/<ticket-or-branch-slug>/` that contains the referenced numbers or query output - If the diff modifies a migration file's header but doesn't add matching evidence, fail (as with A007b1a; would have caught the false 4/4 claim before merge) - Escape hatch: PR label `evidence-narrative-only` skips the check with a required PR-body justification section - False-positive rate target under 5% on 30-PR historical sample - Ships as required check for both `Operant-Health/operant-studio` and `Operant-Health/portico` ## Non-goals - Not enforcing that evidence artifacts are correct (mechanical claim-to-artifact linkage only) - Not deleting existing false claims retroactively (preserved as forensic history; correction narratives in `audit/evidence/`) ## Effort Single Ledger ticket. Parallelizable with A037-A041. Blockers: none. ## Filed as follow-on to PR #278 (KNUCK-A007b1a). Direct control that would have caught the false-verification failure. Ships first among A036-A041. ## Supersedes KNUCK-A036 first attempt (Task cuid `cmts6cx9k001jky04310zihmg`, filed 2026.09.08 04:33Z) -- canceled after admission rejected for missing `feature_id` frontmatter. This body adds the linkage and refiles.
Attachments
Loading attachments…
Comments
Loading comments…