chore(mcp): instrument streamable-HTTP session retention#1696
Open
giladresisi wants to merge 1 commit into
Open
chore(mcp): instrument streamable-HTTP session retention#1696giladresisi wants to merge 1 commit into
giladresisi wants to merge 1 commit into
Conversation
The MCP service climbs ~330 MB/h from a ~1 GB baseline until it is killed at ~5 GB, roughly twice a day, then restarts and repeats. The climb is load-independent, which points at reconnects rather than traffic. Mechanism: `@mastra/mcp` stores one transport (plus, transitively, one per-session Server holding the whole converted tool schema) per `initialize` request in `streamableHTTPTransports`, and removes it only from `transport.onclose` -- which fires only on an explicit client DELETE. No idle timeout exists. Connectors open a session per conversation and never DELETE, so the map grows for the life of the process. Locally each session retains ~76 KB, and 200 `initialize` requests retained 200 sessions across a forced GC. The session design costs us twice. A restart -- an OOM kill or an ordinary deploy -- wipes the map, and every still-connected client keeps presenting a session id the new process has never seen. Those clients do not recover on their own; they must be restarted, and each reconnect leaves another permanently retained session behind. So crashes beget disconnected customer agents, and reconnecting agents beget more of the leak that caused the crash. This is also a dead end upstream: the 2026-07-28 MCP specification removes the `Mcp-Session-Id` header and protocol-level sessions outright, so session-based serving has to go regardless. This commit measures, it does not fix. Nothing about transport behaviour changes; the session map is only read, via `.size` and `.keys()`. What it records, and why in this shape: - Retained heap, sampled from a `PerformanceObserver` immediately after each major GC. `heapUsed` at an arbitrary moment is mostly uncollected garbage; a leak is exactly what survives a full collection. No forced GC, no pauses. - `live` / `new` / `closed` / `abandoned>60m`, from one 15-minute sweep that diffs the session-id key set against the previous one. No per-session timers. `closed=0` beside a growing `abandoned` count is the proof sessions are never released. - `perSession~KB` and `rate=/h`, which say what one `initialize` costs and how many arrive. Together they test whether this leak accounts for the whole 330 MB/h or whether a second one is hiding behind it. - `rss`, `heapTotal` and `external`, to line the numbers up against the container's memory graph. Correlating a log line with the Railway memory graph: retainedHeap <= heapUsed <= heapTotal <= rss <= Railway's number Railway charts the whole container (every process, plus page cache); we report one process's V8 heap. Absolute values will not match, so compare *slopes* over the window between two log lines. If Railway's slope tracks `delta` on retainedHeap, the growth is heap-resident and this leak explains it. If Railway climbs faster, the excess is outside the heap -- `external` catches Buffers, and if neither moves it is reclaimable page cache. `rss - heapTotal - external` is roughly native overhead. Independently, `perSession x rate` predicts the MB/h Railway should show; agreement means the leak is the whole story. Note that V8 releases pages to the OS lazily, so `rss` is sticky and its slope can briefly outrun `retainedHeap` -- trust `retainedHeap` for leak detection and `rss` only to reconcile with the container. `uptimeSeconds` from `/mcp-metrics` (or the boot line reappearing) marks restarts, which are the cliffs in Railway's sawtooth. The boot line also prints the effective V8 heap limit: if the container dies at ~5 GB while the limit is ~2 GB, the heap cannot be what got there, and we are looking at a kernel OOM on RSS, not a V8 FATAL. One summary line per 15 minutes, suppressed entirely while idle. The `/mcp-metrics` endpoint stays 404 unless `MCP_METRICS_TOKEN` is set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Contribution-checker quality warning Heuristics that flagged:
If this is a genuine contribution, please add detail to your PR description and tighten the diff scope before reviewers look at it. |
|
| Status | Scan Engine |  Critical |
 High |
 Medium |
 Low |
Total (0) |
|---|---|---|---|---|---|---|
| Open Source Security | 0 | 0 | 0 | 0 | See details | |
| Licenses | 0 | 0 | 0 | 0 | See details | |
| ✅ | Code Security | 0 | 0 | 0 | 0 | 0 issues |
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The MCP service climbs ~330 MB/h from a ~1 GB baseline until it is OOM-killed at ~5 GB, roughly twice a day, then restarts and repeats. The climb is load-independent, which points at client reconnects rather than traffic volume.
The mechanism is a session leak in
@mastra/mcp: one transport (and, transitively, one per-sessionServerholding the whole converted tool schema) is retained perinitializerequest, and released only on an explicit clientDELETEthat connectors never send. There is no idle timeout, so the map grows for the life of the process.The same session design also means every restart — an OOM kill or an ordinary deploy — strands connected customer agents, which must be restarted, and each reconnect leaves another permanently retained session behind. And it is a dead end upstream: the 2026-07-28 MCP specification removes
Mcp-Session-Idand protocol-level sessions entirely.What this PR does
Measures, does not fix. No transport behaviour changes; the session map is only read, via
.sizeand.keys(). The fix ships separately in #1697.It records retained heap (sampled right after each major GC via a
PerformanceObserver, so uncollected garbage can't masquerade as a leak), pluslive/new/closed/abandoned>60msession counts from a single 15-minute key-set diff — no per-session timers.The two numbers that matter are
perSession~KBandrate=/h. Their product predicts the MB/h the container should show. If it explains the full 330 MB/h, this leak is the whole story; if it explains a third of it, there is a second leak that the stateless fix would otherwise have masked. That is the reason to land instrumentation first and read production before fixing.Output
One line per 15 minutes, suppressed entirely while idle:
closed=0beside a growingabandonedcount is the direct proof that sessions are never released.Deploying
No environment variables required.
/mcp-metricsreturns404unlessMCP_METRICS_TOKENis set. Overhead is onesetInterval, onePerformanceObserverdoing work only on major GCs, and an O(n) key scan every 15 minutes.The commit message documents how to line these log lines up against the Railway memory graph (compare slopes, not absolute values — Railway charts the whole container, we report one process's V8 heap).