fix(db): break probe-failed/restore loop that hangs server startup on sql.js OOM (v3.8.46)

[KooshaPari]

Summary

Fixes the probe-failed/restore loop that hangs the OmniRoute server on Starting server... indefinitely when sql.js fails to load storage.sqlite. Three layered mitigations in src/lib/db/core.ts:

1. Cached fatal on OOM — when the probe error matches an OOM / allocation-failure pattern, set globalThis.__omnirouteDbInitFatal and throw a clear, actionable error. Subsequent callers short-circuit on the cached fatal instead of re-entering the probe.
2. Consecutive-failure counter — even on non-OOM failures, count consecutive probe failures in globalThis.__omnirouteDbProbeFailureCount. After MAX_CONSECUTIVE_PROBE_FAILURES (3) attempts, abort startup with a clear message pointing at the recovery steps.
3. Race-safe async init lock — if sql.js isn't pre-initialized when getDbInstance() is called, kick off a single shared ensureDbInitialized() Promise stored on globalThis.__omnirouteDbInitPromise so concurrent callers share one preInitSqlJs run instead of each racing their own probe.

Root cause

getDbInstance() is not concurrency-safe. On any probe failure, the legacy behaviour renames storage.sqlite → storage.sqlite.probe-failed-<ts>. Concurrent subsystems (STARTUP, HealthCheck, ProviderLimitsSync, ModelSync, BATCH, callLogs) all race their first getDbInstance() during boot; on the next call each one finds the preserved probe-failed backup in listProbeFailureBackups(), renames it back to storage.sqlite, and re-runs the probe, which fails again. The rename/restore cycle repeats indefinitely. The server never reaches OmniRoute is running!.

Reproduction (v3.8.46, Windows, Node 26, sql.js)

omniroute
# Banner prints. Hangs on "⏳ Starting server..." indefinitely.

# Inspect the data dir:
dir $env:APPDATA\omniroute\storage.sqlite*
# storage.sqlite  326 MB (renamed -> probe-failed-1783466372976, restored, renamed, restored, ...)

# Inspect the log:
Get-Content $env:APPDATA\omniroute\logs\application\app.log -Tail 20
# {"component":"DB","message":"[DB] Could not probe existing DB: ... out of memory"}
# {"component":"DB","message":"[DB] Auto-restored preserved database from previous probe failure: ..."}
# {"component":"DB","message":"[DB] Renamed corrupt DB to storage.sqlite.probe-failed-..."}
# ... repeats every ~10 s

After this fix, the same probe OOM produces a single, clear fatal error:

[DB] Aborting startup after 3 consecutive DB probe failures (1 preserved probe-failed backup(s) exist).
     Last error: out of memory. This usually indicates a corrupt database or a runtime that cannot load the SQLite driver.
     Preserved backup: storage.sqlite.probe-failed-*
     To recover, remove the probe-failed backups so OmniRoute can recreate a fresh DB on the next start.

Related

• fix(startup): Cannot create property 'message' on string 'Database closed' at registerInstrumentation #6560 — closed via fix(startup): tolerate 'Database closed' in instrumentation bootstrap (#6560) #6622, fixed the secondary Cannot create property 'message' on string 'Database closed' TypeError in registerInstrumentation (a downstream symptom of the same loop). This PR addresses the upstream loop itself.

Validation

• core.ts parses cleanly under bun build (the only bun build errors are unresolved-import warnings for sql.js and uuid in the fresh test clone — not present in the real node_modules install).
• Local install (C:\Users\koosh\.bun\install\global\node_modules\omniroute) patched with this fix:
  • First test (small heap, no override): server now logs a clear cycle-breaker fatal on the 3rd probe attempt instead of looping silently.
  • Second test (clean data dir, no override): OmniRoute boots in ~6 s. Dashboard at http://localhost:20128/ returns 200 (469 KB). /v1/models returns the model catalog JSON.
  • Third test (326 MB probe-failed backup in db_recovery_2026-07-07\): fatal fires immediately, message includes the data dir path and recovery steps.

Risk

• Additive change only — 167 insertions, 0 deletions. The legacy throw e paths are preserved.
• The cycle-breaker threshold (MAX_CONSECUTIVE_PROBE_FAILURES = 3) is conservative: even on a legitimate transient failure (e.g. the file is briefly locked by a parallel writer), the cap is generous enough that the server will retry at least three times before giving up.
• The new runDbInstanceInitSync() helper extracts the existing body of getDbInstance() verbatim. No behavioural change for callers that don't hit the OOM/race paths.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[KooshaPari]

References #6628