Skip to content

feat: Integrate OpenRouterModel into Metric utilities#2632

Open
Djalal-H wants to merge 1 commit intoconfident-ai:mainfrom
Djalal-H:fix/openrouter-for-metrics
Open

feat: Integrate OpenRouterModel into Metric utilities#2632
Djalal-H wants to merge 1 commit intoconfident-ai:mainfrom
Djalal-H:fix/openrouter-for-metrics

Conversation

@Djalal-H
Copy link
Copy Markdown

PR: Add OpenRouter support to Metrics

This PR integrates the OpenRouterModel into the deepeval metrics layer, allowing it to be used as a native evaluation provider.

Key Changes

  • deepeval/metrics/utils.py:
    • Added OpenRouterModel to recognized native models.
    • Implemented should_use_openrouter_model() for automated provider detection.
    • Updated initialize_model() and is_native_model() to support OpenRouter.
  • Tests: Added tests/test_integrations/test_openrouter/test_g_eval_openrouter.py to verify GEval integration, including sync/async execution and cost tracking.

Testing

Run pytest tests/test_integrations/test_openrouter/test_g_eval_openrouter.py with OPENROUTER_API_KEY configured.

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 26, 2026

@Djalal-H is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant