Add Axiom-based API usage observability at the Edge gateway

[SebastienMelki]

Context

We run ~270 API endpoints behind a shared Vercel Edge chokepoint in server/gateway.ts and currently have no reliable usage visibility:

• no per-endpoint req/s or p95 latency
• no stable per-customer or per-key usage attribution
• no long-term queryable request log
• no trustworthy upstream-provider usage view for ACLED / Wingbits / EIA / etc.

That makes capacity planning, enterprise support, and historical drill-downs unnecessarily hard.

This issue is to align on the implementation direction with @koala73 before building it.

Decision Proposal

Use Axiom as the primary backend for this phase.

Key points:

• emit structured usage events from the Edge gateway via fetch
• keep usage telemetry fully separate from existing console.log(...)
• do not add Redis, S3, Parquet, or self-hosted ClickHouse in phase 1
• accept small best-effort loss from Edge -> Axiom direct ingest
• only add a tiny Railway relay later if direct Edge -> Axiom proves insufficient in canary

Final Plan

1. Instrument the chokepoint in server/gateway.ts.

   • Create a request-scoped usage context at the start of each request.
   • Finalize telemetry after the response status and duration are known.

2. Add server/_shared/usage-context.ts backed by AsyncLocalStorage.

   • Store request_id, route, domain, customer_id, user_id, api_key_id, tier, and auth_kind.
   • This keeps deep provider helpers from needing the Request object threaded through every call.

3. Add server/_shared/usage.ts.

   • Provide a dedicated emitUsageEvents(events) helper.
   • Direct fetch to Axiom.
   • 1-2s timeout, no retry, no console.log inside, and not awaited on the request path.

4. Start with one Axiom dataset: wm_api_usage.

   • Use flat rows with event_type rather than nested arrays.
   • event_type=request
   • event_type=upstream

5. Emit one request event per inbound API call with fields like:

   • _time, event_type, request_id
   • domain, route, method, status, duration_ms
   • req_bytes, res_bytes (best effort)
   • customer_id, user_id, api_key_id, auth_kind, tier
   • country, ua_hash, cache_status

6. Emit one upstream event per real outbound provider call with fields like:

   • _time, event_type, request_id
   • customer_id, route, tier
   • provider, operation, host
   • status, duration_ms, request_bytes, response_bytes
   • cache_status='miss' | 'fresh'

7. Instrument upstream calls in two passes.

   • First, add coverage in shared helpers like server/_shared/redis.ts on the fresh-fetch path only, plus a shared trackedFetch() helper for raw provider calls.
   • Then migrate the highest-cost bypasses first: ACLED, Wingbits, EIA / economic shared fetches, then the rest incrementally.

8. Keep existing console.log usage untouched.

   • Usage telemetry goes only to Axiom.
   • No log scraping.

9. Keep phase 1 direct.

   • No Redis spool.
   • No object-store archive.
   • No Parquet generation from Edge.
   • Fallback path only if needed: a small Railway relay that batches to Axiom.

10. Roll out behind USAGE_TELEMETRY=1.

• ship dark to a test dataset
• enable request events first
• add upstream instrumentation next
• build dashboards after real data lands
• canary in prod and compare counts vs Vercel request volume

11. Build initial dashboards in Axiom:

• req/s and p95 by route
• top customers by request count
• top customers by upstream provider consumption
• top providers by call volume and error rate
• long-range drill-down by customer_id + provider + month

Why this shape

This is the adversarial consensus version of two proposed approaches:

• keep the strongest part of the Axiom-first plan: one managed backend, direct Edge emission, no new infra by default
• keep the strongest part of the stricter observability plan: stable customer attribution, explicit upstream-provider tracking, and event shapes that can answer questions like:
  • "which customer is burning provider capacity?"
  • "how much did Acme call EIA last March?"

The main rejected idea is introducing Redis as a spool in phase 1. That adds moving parts before we know direct Edge -> Axiom is actually a problem.

Blocking Questions Before Build

• What is the canonical customer_id for enterprise API traffic?

  • org/account
  • contract/customer record
  • something else

• Which request/query fields are explicitly allowed in telemetry vs must be redacted or omitted?

Acceptance Criteria

• We can view live request rate, p95, top routes, top customers, and top providers.
• We can query historical usage by customer_id, route, and provider over prior months.
• Provider usage reflects real outbound calls, not just inbound request volume.
• Telemetry failures do not materially affect API availability or latency.

Suggested Rollout

• Define event schema and redaction rules
• Add usage-context.ts and usage.ts
• Instrument gateway request events behind USAGE_TELEMETRY
• Send to Axiom test dataset and validate event quality
• Instrument shared upstream paths
• Canary in production
• Create first operational dashboards

[koala73]

Direction in the plan is right. Before anyone starts coding, five things to sharpen based on what's already in the repo — these change what PR 1 looks like.

---

1. waitUntil needs a gateway-signature change (PR 0, small but not free)

server/gateway.ts:277 returns a handler typed (req: Request) => Promise<Response> — no ctx. Vercel Edge will pass ctx as the 2nd arg if the function accepts it, but today createDomainGateway declines it.

Fix is in one place: change the inner handler to (req, ctx: { waitUntil: (p: Promise<unknown>) => void }). The ~30 api/*/v1/[rpc].ts callsites don't need to change — they just export default createDomainGateway(...).

Reference pattern already in the repo: api/notification-channels.ts:166 is the single handler that uses (req, ctx: { waitUntil }) today. Follow that shape.

Usage telemetry must go through ctx.waitUntil(emitUsageEvents(...)). Do not use an unawaited fetch() — the Edge isolate can tear down before it flushes, and you'll see "partial data on low-traffic routes" in canary with no obvious cause.

---

2. Instrument at cachedFetchJsonWithMeta, not cachedFetchJson

Plan step 7 says to instrument server/_shared/redis.ts on the fresh-fetch path. There's a better seam one function down.

server/_shared/redis.ts:300 — cachedFetchJsonWithMeta — already returns { data, source: 'cache' | 'fresh' }. Instrumenting there gives correct cache-vs-upstream labelling by construction, with no discipline required from the caller. The non-meta cachedFetchJson forces you to wrap the fetcher callback and re-derive fresh in every handler — error-prone, and silently wrong when two handlers share a key.

For raw provider calls that bypass Redis entirely, the second seam is server/_shared/fetch-json.ts:8 (fetchJson). Adding an optional { provider, operation } param to that function + cachedFetchJsonWithMeta covers the vast majority of outbound calls with two hook points, zero leaf-handler changes.

Handlers that use raw fetch() directly (not via these helpers) should migrate to fetchJson as part of phase 1 — that's the point where provider attribution becomes enforceable, not aspirational.

---

3. Unblock customer_id now — the gateway already resolves everything

Listed as a blocking question, but the state is already in server/gateway.ts:

• Clerk JWT → sessionUserId (line ~317, via resolveSessionUserId)
• wm_-prefixed user API keys → isUserApiKey = true + userId injected as x-user-id (line ~340, via validateUserApiKey)
• Static enterprise keys → validateApiKey result + header hash
• Tier → getRequiredTier(pathname) + checkEntitlement

Ship a ~20-line server/_shared/usage-identity.ts that returns a pure object:

type UsageIdentity = {
  auth_kind: 'clerk_jwt' | 'user_api_key' | 'enterprise_api_key' | 'widget_key' | 'anon';
  principal_id: string | null;      // user_id OR api_key_hash_id
  customer_id: string | null;       // principal → customer via small resolver
  tier: 0 | 1 | 2;
};

customer_id resolution:

• clerk_jwt → Clerk org id (or user id if personal account)
• user_api_key → Convex apiKeys.customerRef (already persisted for billing)
• enterprise_api_key → ENTERPRISE_KEY_TO_CUSTOMER mapping (small static map; add it explicitly rather than letting it live ambient in an env var)
• widget_key → widget registration id
• anon → null

This is strictly existing state plus a tiny lookup. Don't defer it — deferring turns phase 1 into "usage telemetry that can't answer the per-customer question," which is the only question that matters.

---

4. Sentry-edge already covers exceptions — scope Axiom to usage

api/_sentry-edge.js already reports edge exceptions directly to the Sentry store endpoint. That means Axiom does not need to be an error pipeline, and the plan should say so explicitly. If contributors see "usage observability" without scope, they'll start piping console.error into Axiom too, and you'll end up with duplicate-of-Sentry noise.

Declare in the issue: Axiom = usage attribution (who called what, how often, how expensive). Sentry = exceptions and tracebacks. Cross-link by emitting sentry_trace_id as a field on the request event when present — you get one-click pivot from "this customer is slow" to "here's the exception" without duplicating ingest.

---

5. Schema tightening — lock these before the first PR

Small things that cost nothing now and a migration later if you skip them:

Field provenance (use what's already on the request):

• request_id = req.headers.get('x-vercel-id') — free join with Vercel runtime logs. Generating a UUID throws that away.
• country = req.headers.get('x-vercel-ip-country') ?? req.headers.get('cf-ipcountry') ?? null — no geo lookup cost.
• execution_region = req.headers.get('x-vercel-id')?.split('::')[0] — cheap, useful for cross-region p95.

UA hash requires a rotating pepper.
Hashing UA without a monthly-rotating server-side pepper is a stable per-browser fingerprint across the entire retention window — that's identification, not anonymisation. Define the pepper rotation in the issue, not in the PR.

Body size via Content-Length only.
Edge request bodies are one-shot reads. Teeing streams adds latency and memory. req_bytes = Number(req.headers.get('content-length') || 0); res_bytes same on response construction. Document as "best effort; zero on chunked / streaming responses" — do not try to solve SSE/stream byte counts in phase 1.

Lock enum values now:

• cache_tier — reuse the exact keys from TIER_HEADERS at server/gateway.ts:30: fast | medium | slow | slow-browser | static | daily | no-store. Don't let contributors invent new buckets.
• execution_plane — vercel-edge | vercel-node | railway-relay (the third only if/when a relay is added).
• origin_kind — browser-same-origin | browser-cross-origin | api-key | oauth | mcp | internal-cron.
• cache_status (upstream events) — miss | fresh | stale-while-revalidate | neg-sentinel (that last one is real — see NEG_SENTINEL in server/_shared/redis.ts).

Redaction by construction, not by filter.
Event builders take allowlisted primitives only — never a Request / Response / raw object:

// Good — additive leaks are structurally impossible
buildRequestEvent({
  route, status, durationMs, principalId, customerId, tier,
  cacheTier, country, uaHash, requestId, sentryTraceId,
}): RequestEvent

// Bad — one careless field add leaks
buildRequestEvent(req, res, identity): RequestEvent  // redacted "later"

The day someone adds a free-text field to an internal type, the pass-through-with-filter pattern leaks it. The allowlist-primitives pattern can't.

---

6. Canary reconciliation needs a declared exclusion list

server/gateway.ts:287 returns 403 for disallowed origins before any instrumentation would fire. Rate-limit 429s (checkRateLimit / checkEndpointRateLimit) fire before route match. OPTIONS preflights short-circuit at line ~302.

Either:

• (recommended) instrument those paths — origin-rejects and rate-limit-rejects are high-signal (e.g. "customer X is being throttled and doesn't know"), OR
• declare them excluded from the Vercel-volume canary comparison.

Don't skip this — you'll get a 2–5% phantom gap in the canary and spend a day chasing it before realising it's just rejections that never reached the instrumented path.

Suggested canary comparison: per-route, per-status-class, bucketed by hour. Whole-day sums hide drift.

---

7. Telemetry failure observability — don't fully silent-swallow

"Swallow delivery failures" is right on the happy path but destroys the signal that telemetry itself is broken. Minimum:

• 1% sampled console.warn('[usage-telemetry] drop', { reason, route }) — Sentry-edge will pick it up.
• Breaker at ~5% delivery failure rate over 5 min → one structured console.error + circuit trip so the next N seconds don't hammer a failing Axiom endpoint.
• Surface the breaker state as a header on responses in dev only (x-usage-telemetry: ok | degraded | off) so you can diagnose "is my dev event dropped?" in 2 seconds.

---

8. Phase-2 name correction (before someone writes the wrong thing)

If implicit context propagation is ever added, use getContext() from @vercel/functions, not raw Node AsyncLocalStorage. ALS is not supported on all Edge runtime versions and has shown up in Vercel outages historically. Mentioning @vercel/functions explicitly in the issue prevents someone grepping "AsyncLocalStorage vercel edge" and picking a pattern that works locally but fails in a prod region.

---

9. Explicit non-goals for phase 1

Worth listing so scope doesn't drift in review:

• No SSE / streaming byte counts (api/chat-analyst.ts is the notable one — stream ends when the isolate returns, you can't measure res_bytes without changing the stream shape).
• No instrumentation of static rewrites or the SPA catch-all.
• No MCP handler attribution in phase 1 (api/mcp.ts) — MCP has its own session model and needs a separate attribution design.
• No log scraping from console.log — explicit in the plan, but repeat it in the PR description.
• No Redis spool, no S3 archive, no Parquet, no ClickHouse — explicit in the plan, worth re-stating.

---

TL;DR for the contributor

1. PR 0: change server/gateway.ts:277 handler signature to (req, ctx).
2. Build server/_shared/usage-identity.ts first — it's the unblocker, not the blocker.
3. Instrument cachedFetchJsonWithMeta + fetchJson, not cachedFetchJson.
4. Events from allowlisted primitives only; reuse Vercel/CF headers for request_id / country / region.
5. Lock cache_tier enum to TIER_HEADERS keys; lock other enums in the PR description.
6. Instrument (or explicitly exclude) origin-403 and rate-limit-429 paths so canary counts reconcile.
7. Axiom ≠ Sentry. Usage only. Cross-link via sentry_trace_id.
8. Phase-2 implicit context is @vercel/functions getContext(), not raw AsyncLocalStorage.

[Sushanth012]

Hi @SebastienMelki and @koala73, I've submitted a PR implementing the Axiom-based observability outlined in this issue: #3392

All of koala73's review points from the comments are addressed — ctx.waitUntil, cachedFetchJsonWithMeta instrumentation, usage-identity resolver, allowlisted primitive event builders, and canary-reconcilable 403/429 instrumentation. Rolled out behind USAGE_TELEMETRY=1.

[SebastienMelki]

Final Plan (locked) — Axiom usage observability

Adopting @koala73's review feedback in full. This comment is the source of truth; the original "Final Plan" section above is superseded.

Scope

• Axiom = usage attribution. Sentry-edge already covers exceptions via api/_sentry-edge.js — Axiom is not an error pipeline. Cross-link via sentry_trace_id field on request events; no duplicate ingest.
• One Axiom dataset: wm_api_usage. Flat rows discriminated by event_type ∈ { request, upstream }.

PR 0 — gateway signature (small, isolated)

• server/gateway.ts:277 — change createDomainGateway inner handler from (req: Request) => Promise<Response> to (req, ctx: { waitUntil: (p: Promise<unknown>) => void }). Pattern reference already in repo at api/notification-channels.ts:166.
• server/alias-rewrite.ts rewriteToSebuf propagates ctx.
• ~30 api/*/v1/[rpc].ts files do not change — they export default createDomainGateway(...) and Vercel passes ctx automatically.
• All telemetry I/O goes through ctx.waitUntil(emitUsageEvents(...)). Never unawaited fetch() — Edge isolate teardown drops it.

Identity resolver — server/_shared/usage-identity.ts

• ~20 lines, pure, consumes gateway state — does not re-verify JWT, does not re-hash keys.
• Signature: buildUsageIdentity({ sessionUserId, isUserApiKey, apiKeyHashId, clerkOrgId, tier, authKind }) → UsageIdentity.
• The gateway already resolves: sessionUserId (Clerk JWT), isUserApiKey + x-user-id injection (wm_-prefixed user keys), validateApiKey result + header hash (enterprise keys), getRequiredTier + checkEntitlement (tier).
• customer_id resolution:
  • clerk_jwt → Clerk org id from existing JWT payload (already parsed in gateway), fallback user id for personal accounts
  • user_api_key → Convex apiKeys.customerRef (already persisted for billing — read once, cache in request scope)
  • enterprise_api_key → explicit static ENTERPRISE_KEY_TO_CUSTOMER map in code (not env-ambient)
  • widget_key → widget registration id
  • anon → null

Telemetry sink — server/_shared/usage.ts

• emitUsageEvents(ctx, events) → ctx.waitUntil(sendToAxiom(events)). Direct fetch to Axiom, 1.5s timeout, no retry.
• Builders accept allowlisted primitives only — never Request / Response / raw objects.
  • buildRequestEvent({ route, status, durationMs, principalId, customerId, tier, cacheTier, country, uaHash, requestId, sentryTraceId, executionRegion, reqBytes, resBytes, authKind, originKind, executionPlane, reason? })
  • buildUpstreamEvent({ requestId, customerId, route, tier, provider, operation, host, status, durationMs, requestBytes, responseBytes, cacheStatus })
• Failure observability:
  • 1% sampled console.warn('[usage-telemetry] drop', { reason, route }) — Sentry-edge picks up
  • Circuit breaker: ≥5% delivery failure rate over 5min sliding window → trip; one structured console.error on trip; reject calls until window clears
  • Dev-only response header x-usage-telemetry: ok | degraded | off for 2-second debugging

Schema (locked — do not change without re-review)

Common (both event_types):

• _time, event_type, request_id (= req.headers.get('x-vercel-id'), free join with Vercel logs)

event_type=request:

• route, domain, method, status, duration_ms
• req_bytes (= Number(req.headers.get('content-length') || 0)), res_bytes (best-effort, 0 on chunked/streaming — documented)
• customer_id, principal_id, auth_kind, tier
• country (= req.headers.get('x-vercel-ip-country') ?? req.headers.get('cf-ipcountry') ?? null)
• execution_region (= req.headers.get('x-vercel-id')?.split('::')[0])
• execution_plane ∈ { vercel-edge, vercel-node, railway-relay }
• origin_kind ∈ { browser-same-origin, browser-cross-origin, api-key, oauth, mcp, internal-cron }
• cache_tier ∈ { fast, medium, slow, slow-browser, static, daily, no-store } — mirrors TIER_HEADERS keys at server/gateway.ts:30, derived per-request, not hardcoded
• ua_hash — SHA-256 of UA + monthly-rotating pepper. Pepper key: USAGE_UA_PEPPER, rotation cron documented in this issue, not in the PR
• sentry_trace_id (when present)
• reason ∈ { origin_403, rate_limit_429, ok } — origin-403 and rate-limit-429 paths instrumented so canary reconciles with Vercel volume

event_type=upstream:

• customer_id, route, tier
• provider, operation, host
• status, duration_ms, request_bytes, response_bytes
• cache_status ∈ { miss, fresh, stale-while-revalidate, neg-sentinel } (neg-sentinel matches NEG_SENTINEL in server/_shared/redis.ts)

Upstream instrumentation — two seams, zero leaf-handler discipline

• Seam 1: server/_shared/redis.ts:300 — cachedFetchJsonWithMeta. Already returns { data, source: 'cache' | 'fresh' }, so cache-vs-upstream labelling is correct by construction. Add optional { provider, operation } to its options. Emit upstream event on the fresh branch.
• Seam 2: server/_shared/fetch-json.ts:8 — fetchJson. Add optional { provider, operation }. Emit upstream event on every call.
• Phase-1 migration order for raw fetch() callers → fetchJson: ACLED, Wingbits, EIA / economic shared fetches, then the rest incrementally. Raw fetch() outside these helpers is the migration boundary; if a handler doesn't go through them, attribution is aspirational, not enforceable.
• Do not instrument cachedFetchJson (non-meta). Forces caller-side cache discipline that silently breaks under shared keys.

Non-goals (phase 1)

• No SSE / streaming res_bytes (notably api/chat-analyst.ts)
• No instrumentation of static rewrites or SPA catch-all
• No MCP attribution (api/mcp.ts) — separate session-model design
• No console.log scraping
• No Redis spool, S3 archive, Parquet, ClickHouse
• No raw Node AsyncLocalStorage if implicit context is later added — use getContext() from @vercel/functions (ALS is unsupported on some Edge runtime versions)

Acceptance

• Live: req/s + p95 by route, top customers by request count, top customers by upstream provider consumption, top providers by call volume + error rate
• Historical: query usage by customer_id + route + provider over months
• Provider usage = real outbound calls, not inbound proxy
• Telemetry failure does not affect API availability or latency
• Canary: per-route, per-status-class, hourly buckets vs Vercel request volume reconciles within tolerance

Rollout

Behind USAGE_TELEMETRY=1. Test dataset → request events first → upstream instrumentation → dashboards → prod canary.

Implementation order

1. PR-0: gateway signature + rewriteToSebuf ctx propagation (no telemetry yet)
2. usage-identity.ts (pure, consumes gateway state)
3. usage.ts builders + sink + circuit breaker
4. Gateway happy-path emit + 403/429 emit
5. cachedFetchJsonWithMeta upstream emit
6. fetchJson upstream emit + raw-fetch migration (ACLED → Wingbits → EIA → rest)
7. Axiom dashboards
8. Prod canary

cc @koala73 — confirming I've captured your 9 points correctly and we're free to proceed. PR #3392 is being superseded — it scaffolded files but missed the core deliverable (no happy-path request event, no cachedFetchJsonWithMeta instrumentation, builders take Request directly, cache_tier hardcoded, identity resolver re-verifies JWT instead of consuming gateway state, require() in Edge runtime breaking the build).