Skip to content

feat(routing): propensity logging, learning loop, fingerprint sessions, hardening#82

Open
vishalveerareddy123 wants to merge 9 commits into
mainfrom
feat/routing-hardening-learning-loop
Open

feat(routing): propensity logging, learning loop, fingerprint sessions, hardening#82
vishalveerareddy123 wants to merge 9 commits into
mainfrom
feat/routing-hardening-learning-loop

Conversation

@vishalveerareddy123

Copy link
Copy Markdown
Collaborator

Completes the routing-hardening plan (WS4, WS5, WS1.5) and fixes eight bugs found in live Claude Code sessions. Full incident-by-incident ledger in PLAN-routing-hardening.md.

WS4 — Propensity + candidates logging

  • bandit.pick() returns propensity (exploit: 1−ε+ε/K, explore: ε/K); every decision path stamps propensity + candidates into telemetry, enabling off-policy evaluation from logs alone.

WS5 — Close the learning loop

  • src/routing/feedback.js (new): first real caller of bandit.update(); adds conclusive outcomes (quality ≥70 / ≤40) to the kNN index using the embedding already paid for at decision time. Runs on setImmediate, never throws into the response path.
  • Reward-pipeline state persists across restarts; kNN MIN_INDEX_SIZE 1000→100 with size/1000 confidence damping; in-process auto-calibration re-fits tier thresholds from telemetry every 24h and hot-reloads.

WS1.5 — Sessions that actually work for Claude Code

  • Claude Code sends no session id → every request got a fresh UUID (278 distinct "sessions" per 286 requests measured live), so sticky-session pinning was dead. Session ids are now derived from sha256(first user message + system head + UA).
  • Upward score-drift escape: a "Hi"-opened session pinned SIMPLE re-decides when the real task arrives; force-cloud phrases escape pins regardless of score.

Hardening (all from live incidents, each with a regression test)

  • Risk classifier no longer scans harness-injected <system-reminder> text ("23+45" was risk-escalated on injected "credential/security" strings).
  • Risk-forced decisions are never pinned (kills the pin-poisoning ratchet); suggestion-mode/side requests route static-SIMPLE without touching pins — gated on a detected client profile so generic API clients still get full routing.
  • Tool-history check restricted to true mid-exchange frames: one tool call no longer welds a session to its pin (which silently disabled risk/context/vision guards for the rest of the session).
  • Force-cloud and autonomous-agentic paths honor tier config instead of the credential-priority list (dummy databricks creds caused a self-proxy loop).
  • Badge stripping no longer drops merged badge+answer blocks or produces empty content Anthropic rejects (interrupted-response echo loop).
  • lynkr wrap session stats read the real snake_case metric fields over HTTP (previously showed "No requests tracked" on every exit since shipping).
  • Test suite no longer clobbers the production kNN index.

Tooling

  • benchmark-tier-routing.js measurement layer made honest: served model from response body (not routing headers), real pricing for local models, cache hits billed as zero, expected-tier PASS/FAIL checks, unreachable competitors reported as SKIPPED.
  • Semantic cache hits are now marked in the response body (lynkr_semantic_cache).

Tests

1041/1044 passing — 3 pre-existing test/web-tools.test.js undici mock failures, unchanged from before this branch. ~80 new tests across 8 new test files.

vishal veerareddy added 9 commits July 8, 2026 19:55
…s, hardening

WS4: bandit propensity + candidates on every decision for off-policy evaluation.
WS5: closed the learning loop — feedback module (bandit.update + kNN online
growth), reward-pipeline persistence, in-process auto-calibration.
WS1.5: content-fingerprint session ids (clients send none) + upward score
drift so pinned sessions escalate when the real task arrives.
Hardening from live incidents: risk classifier no longer scans harness-injected
text; risk-forced decisions never pin; suggestion-mode/side requests route
static-SIMPLE without touching pins; tool-history pin weld fixed (guards +
drift now run after completed tool exchanges); force-cloud and autonomous
paths honor tier config instead of the credential list (self-proxy fix);
badge echo fixed; wrap session stats read real fields; benchmark measurement
layer made honest; test suite no longer clobbers the production kNN index.
Encodes this week's live incidents as permanent checks: reminder-injection
immunity, suggestion-mode side requests, force-cloud/path-risk triggers,
autonomous→REASONING, fingerprint pin + mid-session escape, cache
false-positive guard. Route expectations judge Lynkr only (other proxies
synthesize tier labels). Adds a pass/fail scoreboard section.
Pins (6h TTL) and semantic-cache entries persist server-side between runs:
run #2 inherited run #1's P2-written COMPLEX pin and served most scenarios
from cache, corrupting measurements. Per-run nonce in stateful scenarios,
x-lynkr-no-cache on everything outside the cache family, and SC3 now fails
only on LOW-similarity hits (an answer to a different question) — prior
runs' identical question matching at ~0.99 is correct cache behavior.
… reproducible benchmark

Replaces stale headline claims (87.6%/53%/171ms) with numbers the bundled
benchmark reproduces (84% TOON, 18-22% tool stripping, <300ms cache hits),
documents the full routing pipeline (fingerprint sessions, escape ladder,
learning loop, incident-derived rules) and the 17-scenario regression
harness. Test badge 699→1041.
…escalate on failure

Ex-ante difficulty prediction is unreliable for agentic coding traffic
(SWE-Bench evidence: fails trivial baselines; verified cascades gain +14%).
Instead of committing to a predicted tier, cheap-tier (SIMPLE/MEDIUM)
responses are verified post-hoc: structural checks targeting observed
live failures (language drift, degeneration loops, truncation in open
code fences, malformed tool calls, empty/echo) plus a coarse content
score. A failing answer is discarded, recorded as a hard negative
(quality 20 → kNN learns), and the request climbs the tier ladder;
if every escalation fails, the original answer serves (never worse
than no cascade). Expensive-tier answers are never re-verified; fails
open on verifier errors. Flag: LYNKR_CASCADE_VERIFY (default off).
Live-verified: empty minimax response → escalated → kimi served.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant