Handle nested token usage details in oaieval

[kayametehan]

Fixes #1582.

Overview

add_token_usage_to_result currently assumes every usage value can be summed as a plain integer. That breaks with newer OpenAI SDK usage payloads where fields like completion_tokens_details and prompt_tokens_details are nested objects rather than scalar counts.

What changed

• Flatten nested usage payloads into numeric leaf metrics before aggregation
• Sum the union of all observed usage keys across sampling events
• Preserve top-level usage metrics like prompt_tokens, completion_tokens, and total_tokens
• Add a regression test covering nested CompletionUsage details

Example

A sampling event that includes completion_tokens_details.reasoning_tokens now records:

• usage_completion_tokens
• usage_completion_tokens_details_reasoning_tokens
• usage_prompt_tokens
• usage_total_tokens

instead of raising a TypeError during result aggregation.

[Copilot]

Pull request overview

This PR updates the oaieval CLI’s token-usage aggregation to support newer OpenAI SDK usage payloads that include nested “details” objects (e.g., completion_tokens_details) instead of only scalar token counts, preventing TypeError during result aggregation (Fixes #1582).

Changes:

• Add a usage-metric flattener that converts nested usage objects/mappings into numeric leaf metrics with underscore-delimited keys.
• Aggregate token usage across the union of all observed usage keys across sampling events (instead of assuming a fixed key set).
• Add a regression test to ensure nested usage details are flattened and summed correctly.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
evals/cli/oaieval.py	Adds _flatten_usage_metrics and updates add_token_usage_to_result to flatten and safely sum nested usage metrics across events.
evals/cli/oaieval_test.py	Adds a regression test covering nested CompletionUsage detail fields and verifying aggregated result keys/values.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.