Skip to content

authorized_hits metric not emitted for SSE streaming responses #373

Description

@noyitz

Description

The authorized_hits metric (token count) is only emitted for non-streaming responses. When the upstream provider returns an SSE (Server-Sent Events) streaming response (e.g., Content-Type: text/event-stream), the Wasm plugin does not parse the SSE stream to extract usage.total_tokens from the final event, so the ratelimit-report to Limitador never includes the token count.

Impact

  • authorized_hits{user, model, subscription} — only populated for non-streaming requests
  • authorized_calls{user, subscription} — works correctly for all requests (streaming and non-streaming)
  • Billing/cost dashboards based on authorized_hits undercount token usage for streaming-heavy workloads (e.g., Claude Code, OpenAI Codex which always use streaming)

Reproduction

  1. Configure an ExternalModel pointing to Anthropic API with stream: true
  2. Send a request through the gateway
  3. Check Limitador metrics — authorized_calls increments but authorized_hits does not
  4. Send the same request with stream: false — both metrics increment

Expected Behavior

The Wasm plugin should parse SSE responses to extract token usage from the final event:

  • Anthropic: event: message_deltadata: {"usage": {"output_tokens": N}}
  • OpenAI Responses API: event: response.completeddata: {"response": {"usage": {"total_tokens": N}}}
  • OpenAI Chat Completions: final chunk with usage field when stream_options.include_usage: true

Workaround

We implemented SSE parsing in the IPP (ext_proc) response handler to extract usage data from streaming responses. See:

Environment

  • Kuadrant Wasm shim: v0.12.1
  • Limitador: v2.3.0
  • OpenShift: 4.19
  • MaaS: v0.1.0 (kustomize mode)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions