Description
The authorized_hits metric (token count) is only emitted for non-streaming responses. When the upstream provider returns an SSE (Server-Sent Events) streaming response (e.g., Content-Type: text/event-stream), the Wasm plugin does not parse the SSE stream to extract usage.total_tokens from the final event, so the ratelimit-report to Limitador never includes the token count.
Impact
authorized_hits{user, model, subscription} — only populated for non-streaming requests
authorized_calls{user, subscription} — works correctly for all requests (streaming and non-streaming)
- Billing/cost dashboards based on
authorized_hits undercount token usage for streaming-heavy workloads (e.g., Claude Code, OpenAI Codex which always use streaming)
Reproduction
- Configure an ExternalModel pointing to Anthropic API with
stream: true
- Send a request through the gateway
- Check Limitador metrics —
authorized_calls increments but authorized_hits does not
- Send the same request with
stream: false — both metrics increment
Expected Behavior
The Wasm plugin should parse SSE responses to extract token usage from the final event:
- Anthropic:
event: message_delta → data: {"usage": {"output_tokens": N}}
- OpenAI Responses API:
event: response.completed → data: {"response": {"usage": {"total_tokens": N}}}
- OpenAI Chat Completions: final chunk with
usage field when stream_options.include_usage: true
Workaround
We implemented SSE parsing in the IPP (ext_proc) response handler to extract usage data from streaming responses. See:
Environment
- Kuadrant Wasm shim: v0.12.1
- Limitador: v2.3.0
- OpenShift: 4.19
- MaaS: v0.1.0 (kustomize mode)
Description
The
authorized_hitsmetric (token count) is only emitted for non-streaming responses. When the upstream provider returns an SSE (Server-Sent Events) streaming response (e.g.,Content-Type: text/event-stream), the Wasm plugin does not parse the SSE stream to extractusage.total_tokensfrom the final event, so theratelimit-reportto Limitador never includes the token count.Impact
authorized_hits{user, model, subscription}— only populated for non-streaming requestsauthorized_calls{user, subscription}— works correctly for all requests (streaming and non-streaming)authorized_hitsundercount token usage for streaming-heavy workloads (e.g., Claude Code, OpenAI Codex which always use streaming)Reproduction
stream: trueauthorized_callsincrements butauthorized_hitsdoes notstream: false— both metrics incrementExpected Behavior
The Wasm plugin should parse SSE responses to extract token usage from the final event:
event: message_delta→data: {"usage": {"output_tokens": N}}event: response.completed→data: {"response": {"usage": {"total_tokens": N}}}usagefield whenstream_options.include_usage: trueWorkaround
We implemented SSE parsing in the IPP (ext_proc) response handler to extract usage data from streaming responses. See:
Environment