OPER-49
Right-size Prisma connection pool for the reliability loop
Body
# OPER-49 · Right-size Prisma connection pool for the reliability loop
## Problem
Reliability-loop ticks are failing with 4/5 sensors returning `outcome: "error"` on every run since 2026-07-23. Error text across sensors:
```
Timed out fetching a new connection from the connection pool.
(Current connection pool timeout: 10, connection limit: 5)
```
and (from `$queryRawUnsafe` code paths where pool exhaustion surfaces as a different message):
```
Can't reach database server at ep-wild-resonance-atf8ez9o-pooler...
```
Neon itself is healthy — verified from the sandbox:
- `pg_stat_activity` shows 14/901 conns in use, 1 active, 6 idle
- TCP connect + `SELECT 1` returns in <30ms
- Neon status page: All Systems Operational
**Root cause:** The `@operant/database` `PrismaClient` singleton uses Prisma's default `connection_limit` (2× CPU count = **5** on Vercel Fluid), and the reliability loop fans out **5 sensors in parallel** via `Promise.allSettled` — every sensor grabs ≥1 connection and the pool is exhausted before the loop finishes.
## Fix
File: `packages/database/src/index.ts`
Change the `PrismaClient` construction to explicitly set a pooled `datasourceUrl` derived from `DATABASE_URL` (or an existing `DATABASE_URL_POOLED`), appending `?connection_limit=25&pool_timeout=20`.
**Before**
```ts
export const prisma =
globalThis.__prisma ??
new PrismaClient({
log:
process.env.NODE_ENV === "development"
? ["query", "error", "warn"]
: ["error"],
});
```
**After**
```ts
function buildPooledUrl(): string | undefined {
const raw = process.env.DATABASE_URL_POOLED ?? process.env.DATABASE_URL;
if (!raw) return undefined;
try {
const u = new URL(raw);
// Prisma reads these from the query string when connecting via PgBouncer / Neon pooler.
if (!u.searchParams.has("connection_limit")) u.searchParams.set("connection_limit", "25");
if (!u.searchParams.has("pool_timeout")) u.searchParams.set("pool_timeout", "20");
if (!u.searchParams.has("pgbouncer")) u.searchParams.set("pgbouncer", "true");
return u.toString();
} catch {
return raw; // if URL parsing fails, fall back to raw — Prisma will error clearly.
}
}
export const prisma =
globalThis.__prisma ??
new PrismaClient({
datasourceUrl: buildPooledUrl(),
log:
process.env.NODE_ENV === "development"
? ["query", "error", "warn"]
: ["error"],
});
```
Rationale for 25 / 20s:
- Neon pooler supports thousands of client conns; 25 per Vercel instance is well below any limit.
- Reliability loop has 5 sensors, some using `$queryRawUnsafe` which takes a dedicated conn — 25 gives 5× headroom for concurrent ticks + normal request traffic.
- `pool_timeout=20` doubles the current 10s ceiling to survive cold-start bursts.
- `pgbouncer=true` disables prepared statement caching that PgBouncer transaction pooling doesn't support (Neon pooler is transaction-mode).
## Tests
File: `packages/database/src/__tests__/index.test.ts` (create if missing)
- Test `buildPooledUrl` with a URL that has no query params → returns URL with all three params set.
- Test with a URL that already has `connection_limit=10` → preserves the existing value.
- Test with an invalid URL → returns the raw string unchanged.
- Test with `DATABASE_URL_POOLED` set → prefers it over `DATABASE_URL`.
- Test with neither set → returns `undefined`.
## Verification
```bash
npx vitest run packages/database/src/__tests__/index.test.ts
```
Post-merge, verify on next reliability-loop tick:
```sql
SELECT "sensorName", "outcome", COUNT(*)
FROM "ReliabilityEvent"
WHERE "createdAt" > NOW() - INTERVAL '30 minutes'
GROUP BY 1,2;
```
Expected: zero rows with `outcome='error'` and error text about "connection pool" or "Can't reach database".
## Out of scope
- Any Prisma version bump.
- Splitting the pool into read/write.
- Changing the reliability-loop concurrency model (keep `Promise.allSettled` fan-out).
- Fixing the `dispatch_retry_storm` body bug — tracked separately in OPER-48.
## Non-negotiables
- No new tables, no migration.
- `DATABASE_URL` must remain a valid fallback (dev + local tests).
- Do not remove the `globalThis.__prisma` HMR-cache pattern.
- PR title: `fix(database): explicit Prisma pool config for reliability loop (OPER-49)`
- PR body must include `Ticket: OPER-49` trailer.
## Post-merge follow-through
- Add a runbook entry `docs/operations/pipeline-runbook.md#prisma-pool-exhaustion` describing:
- How to spot pool exhaustion in ReliabilityEvent (`error` outcome + "connection pool" errorText)
- Which env var to bump and to what value
- How to verify recovery (query above)
If the runbook file exists, append; do not create a duplicate.
Attachments
Loading attachments…
Comments
Loading comments…