fix(db): break probe-failed/restore loop on large storage.sqlite

[KooshaPari]

fix(db): break probe-failed/restore loop on large storage.sqlite

Summary

Fixes an infinite loop in getDbInstance() that prevents omniroute v3.8.46 from completing startup when storage.sqlite exceeds ~100 MB on a sql.js runtime under the default Node heap limit.

Closes #6631

Root cause

The startup probe flow in src/lib/db/core.ts has two interacting bugs that together produce a tight 5ms-interval loop:

1. Rename-on-any-probe-failure (line ~1054): the probe in readonly mode is wrapped in a try/catch that unconditionally renames storage.sqlite → storage.sqlite.probe-failed-<timestamp> on every failure — including transient out-of-memory errors that are environment-driven (heap limit vs. DB size), not corruption-driven.
2. Auto-restore from most-recent probe-failed (line ~954): the next caller checks !existsSync(R) && probeFailureBackups.length > 0 and unconditionally renames the most recent probe-failed-<ts> back to storage.sqlite. This "recovery" path silently re-loads the same file that just failed.

Combined: probe OOMs → rename → next call restores → probe OOMs → repeat. The loop is driven by multiple startup services (ProviderLimitsSync, ModelSync, HealthCheck, BATCH, callLogs, STARTUP) each independently calling getDbInstance() in parallel.

Symptoms in app.log:

[DB] Auto-restored preserved database from previous probe failure
[DB] Could not probe existing DB: out of memory
[DB] Renamed corrupt DB to storage.sqlite.probe-failed-1783466372979
[DB] Auto-restored preserved database from previous probe failure
[DB] Could not probe existing DB: out of memory
... (repeats every ~5ms, server never reaches "OmniRoute is running")

A 326 MB storage.sqlite reproduces this on a fresh install under default --max-old-space-size. The probe in readonly mode allocates the entire file in sql.js WASM heap (often 3–5x file size) and OOMs.

Fix

Two minimal, surgical changes in src/lib/db/core.ts:

1. Cycle-breaker guard inside the auto-restore condition

if (!fs.existsSync(sqliteFile) && probeFailureBackups.length > 0) {
  if (((globalThis as any).__omnirouteDbProbeRestoreCount =
       ((globalThis as any).__omnirouteDbProbeRestoreCount || 0) + 1) > 3) {
    throw new Error(
      `[DB] Aborting startup: probe-failed/restore loop detected after 3 attempts. ` +
      `The preserved database at ${sqliteFile} is unloadable under this runtime. ` +
      `Remove the probe-failed backups from ${path.dirname(sqliteFile)} and restart, ` +
      `or restore the database from a known-good backup.`
    );
  }
  const mostRecentBackup = probeFailureBackups[0];
  // ... existing restore logic
}

After 3 restoration attempts in the same process, throw a clear, actionable fatal error instead of looping. The counter is per-process and resets on restart (by design — a fresh start with the probe-failed files manually removed should succeed).

2. Skip rename on transient OOM

} catch (probeError) {
  const message = probeError instanceof Error ? probeError.message : String(probeError);
  const isOutOfMemory = /out of memory|allocation (?:failure|failed)|array buffer allocation failed/i.test(message);
  if (!isOutOfMemory) {
    // existing rename-to-probe-failed logic
  }
}

OOM is a runtime/environment condition (heap limit vs. DB size) that cannot be fixed by re-loading the same file. Skipping the rename preserves the working database for the user to manually inspect or VACUUM.

Verification

Repro environment: Windows, Node v26.3.0, omniroute v3.8.46, storage.sqlite ≈ 326 MB.

Without fix: startup hangs forever, app.log shows the rename/restore loop every ~5ms.

With fix:

• First 3 probe failures still produce the original "Could not probe existing DB" + restore cycle.
• 4th attempt throws the new fatal error with the recovery message.
• Server fails fast and stops instead of looping forever.
• User follows the error message: removes storage.sqlite.probe-failed-* files, restarts → boots cleanly.

Test plan

• File parses via bun build --target=bun (no TS errors)
• Manual repro of the loop in local install (326 MB DB)
• Confirmed fatal error fires after 3 attempts in local install
• Upstream CI: full pnpm test and the DB test suite

Related notes

This PR addresses the launch hang specifically. A separate, downstream issue is the Next.js Cannot create property 'message' on string 'Database closed' TypeError that fires from instrumentation.ts when getDbInstance() throws a string from sql.js. That's a different bug (error-as-string vs. error-as-object in the Next.js instrumentation hook) and should be addressed in a separate PR.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!