Summary
The Paperclip server became completely unresponsive (TCP accept, zero HTTP response on /api/health, no log output) for ~6 hours while the process stayed alive. Root cause at the DB layer: all 10 pg pool connections were stuck idle in transaction for ~5h50m, every one with its last completed statement being insert into agent_wakeup_requests inside the enqueueWakeup issue-execution transaction (server/src/services/heartbeat.ts:13454). With the pool exhausted, every API request queued forever.
Environment
- Self-hosted, systemd service, embedded Postgres (:54329), Node 22
- Process at the time: RSS 2.3G, peak 5.9G, ~400M swap used
- Pool: 10 connections (default)
Evidence
pg_stat_activity (10 rows, all identical pattern):
state = idle in transaction
xact_age ≈ 5h50m
query = insert into "agent_wakeup_requests" ("id", "company_id", "agent_id", "source", "trigger_detail", "reason", "payload", "s...
curl http://127.0.0.1:3100/api/health → connected, 0 bytes in 10s- Zero application log lines after the wedge began (~10:10 UTC), journal silent for 6h
- Wedge began during a heartbeat run that was executing the issue-execution wakeup path
systemctl restart fully recovered; restart reconciled the stranded heartbeat runs as failed ("Process lost") and the server picked up queued work
Analysis
The frozen transactions sit inside the long multi-statement db.transaction(async (tx) => {...}) in enqueueWakeup (heartbeat.ts:13454), which does SELECT ... FOR UPDATE on issues, several tx.insert(agentWakeupRequests) branches, event inserts, etc. The transactions completed an insert and then never committed — consistent with the Node event loop stalling (memory pressure / GC thrash near the 5.9G peak) while transactions were in flight, leaving them open indefinitely.
Two compounding issues worth addressing:
- No defensive timeout on the DB side. An open transaction should not be able to hold a pool connection forever.
idle_in_transaction_session_timeout (e.g. 5min) on the embedded Postgres would auto-heal this. Consider setting it by default in the embedded-pg bootstrap, plus a per-transaction watchdog in the pool config.
- Long, multi-branch explicit transactions in the wakeup path.
enqueueWakeup's transaction spans many statements and awaits; any stall mid-transaction strands a pool slot. Shortening transaction scope (or moving non-mutating work outside the tx) would shrink the blast radius. A hang-detector on the HTTP side (e.g. /api/health failing → log dump of pg_stat_activity) would also make this diagnosable in minutes instead of hours.
Mitigations applied locally
ALTER DATABASE paperclip SET idle_in_transaction_session_timeout = '5min'- systemd watchdog timer restarting the service when /api/health fails 3x
- Full receipt: wedge began ~10:11 UTC 2026-07-17, recovered 16:03 UTC
Happy to pull more forensics (heap snapshot setup, pool config, pg logs) if useful.
Summary
The Paperclip server became completely unresponsive (TCP accept, zero HTTP response on
/api/health, no log output) for ~6 hours while the process stayed alive. Root cause at the DB layer: all 10 pg pool connections were stuckidle in transactionfor ~5h50m, every one with its last completed statement beinginsert into agent_wakeup_requestsinside theenqueueWakeupissue-execution transaction (server/src/services/heartbeat.ts:13454). With the pool exhausted, every API request queued forever.Environment
Evidence
curl http://127.0.0.1:3100/api/health→ connected, 0 bytes in 10ssystemctl restartfully recovered; restart reconciled the stranded heartbeat runs as failed ("Process lost") and the server picked up queued workAnalysis
The frozen transactions sit inside the long multi-statement
db.transaction(async (tx) => {...})inenqueueWakeup(heartbeat.ts:13454), which doesSELECT ... FOR UPDATEonissues, severaltx.insert(agentWakeupRequests)branches, event inserts, etc. The transactions completed an insert and then never committed — consistent with the Node event loop stalling (memory pressure / GC thrash near the 5.9G peak) while transactions were in flight, leaving them open indefinitely.Two compounding issues worth addressing:
idle_in_transaction_session_timeout(e.g. 5min) on the embedded Postgres would auto-heal this. Consider setting it by default in the embedded-pg bootstrap, plus a per-transaction watchdog in the pool config.enqueueWakeup's transaction spans many statements and awaits; any stall mid-transaction strands a pool slot. Shortening transaction scope (or moving non-mutating work outside the tx) would shrink the blast radius. A hang-detector on the HTTP side (e.g./api/healthfailing → log dump of pg_stat_activity) would also make this diagnosable in minutes instead of hours.Mitigations applied locally
ALTER DATABASE paperclip SET idle_in_transaction_session_timeout = '5min'Happy to pull more forensics (heap snapshot setup, pool config, pg logs) if useful.