feat(routing): propensity logging, learning loop, fingerprint sessions, hardening#82
Open
vishalveerareddy123 wants to merge 9 commits into
Open
feat(routing): propensity logging, learning loop, fingerprint sessions, hardening#82vishalveerareddy123 wants to merge 9 commits into
vishalveerareddy123 wants to merge 9 commits into
Conversation
added 9 commits
July 8, 2026 19:55
…s, hardening WS4: bandit propensity + candidates on every decision for off-policy evaluation. WS5: closed the learning loop — feedback module (bandit.update + kNN online growth), reward-pipeline persistence, in-process auto-calibration. WS1.5: content-fingerprint session ids (clients send none) + upward score drift so pinned sessions escalate when the real task arrives. Hardening from live incidents: risk classifier no longer scans harness-injected text; risk-forced decisions never pin; suggestion-mode/side requests route static-SIMPLE without touching pins; tool-history pin weld fixed (guards + drift now run after completed tool exchanges); force-cloud and autonomous paths honor tier config instead of the credential list (self-proxy fix); badge echo fixed; wrap session stats read real fields; benchmark measurement layer made honest; test suite no longer clobbers the production kNN index.
Encodes this week's live incidents as permanent checks: reminder-injection immunity, suggestion-mode side requests, force-cloud/path-risk triggers, autonomous→REASONING, fingerprint pin + mid-session escape, cache false-positive guard. Route expectations judge Lynkr only (other proxies synthesize tier labels). Adds a pass/fail scoreboard section.
Pins (6h TTL) and semantic-cache entries persist server-side between runs: run #2 inherited run #1's P2-written COMPLEX pin and served most scenarios from cache, corrupting measurements. Per-run nonce in stateful scenarios, x-lynkr-no-cache on everything outside the cache family, and SC3 now fails only on LOW-similarity hits (an answer to a different question) — prior runs' identical question matching at ~0.99 is correct cache behavior.
… reproducible benchmark Replaces stale headline claims (87.6%/53%/171ms) with numbers the bundled benchmark reproduces (84% TOON, 18-22% tool stripping, <300ms cache hits), documents the full routing pipeline (fingerprint sessions, escape ladder, learning loop, incident-derived rules) and the 17-scenario regression harness. Test badge 699→1041.
…escalate on failure Ex-ante difficulty prediction is unreliable for agentic coding traffic (SWE-Bench evidence: fails trivial baselines; verified cascades gain +14%). Instead of committing to a predicted tier, cheap-tier (SIMPLE/MEDIUM) responses are verified post-hoc: structural checks targeting observed live failures (language drift, degeneration loops, truncation in open code fences, malformed tool calls, empty/echo) plus a coarse content score. A failing answer is discarded, recorded as a hard negative (quality 20 → kNN learns), and the request climbs the tier ladder; if every escalation fails, the original answer serves (never worse than no cascade). Expensive-tier answers are never re-verified; fails open on verifier errors. Flag: LYNKR_CASCADE_VERIFY (default off). Live-verified: empty minimax response → escalated → kimi served.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the routing-hardening plan (WS4, WS5, WS1.5) and fixes eight bugs found in live Claude Code sessions. Full incident-by-incident ledger in
PLAN-routing-hardening.md.WS4 — Propensity + candidates logging
bandit.pick()returns propensity (exploit:1−ε+ε/K, explore:ε/K); every decision path stampspropensity+candidatesinto telemetry, enabling off-policy evaluation from logs alone.WS5 — Close the learning loop
src/routing/feedback.js(new): first real caller ofbandit.update(); adds conclusive outcomes (quality ≥70 / ≤40) to the kNN index using the embedding already paid for at decision time. Runs onsetImmediate, never throws into the response path.MIN_INDEX_SIZE1000→100 withsize/1000confidence damping; in-process auto-calibration re-fits tier thresholds from telemetry every 24h and hot-reloads.WS1.5 — Sessions that actually work for Claude Code
sha256(first user message + system head + UA).Hardening (all from live incidents, each with a regression test)
<system-reminder>text ("23+45" was risk-escalated on injected "credential/security" strings).lynkr wrapsession stats read the real snake_case metric fields over HTTP (previously showed "No requests tracked" on every exit since shipping).Tooling
benchmark-tier-routing.jsmeasurement layer made honest: served model from response body (not routing headers), real pricing for local models, cache hits billed as zero, expected-tier PASS/FAIL checks, unreachable competitors reported as SKIPPED.lynkr_semantic_cache).Tests
1041/1044 passing — 3 pre-existing
test/web-tools.test.jsundici mock failures, unchanged from before this branch. ~80 new tests across 8 new test files.