Skip to content

docs(ai-evals): Document opt-in evaluation report inbox signals #18140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
inkeep wants to merge 2 commits into
base: master
Choose a base branch
from
Open

docs(ai-evals): Document opt-in evaluation report inbox signals #18140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
302c903
Add AI observability signal source to inbox sources docs
inkeep[bot] Jul 2, 2026
1528e63
docs(ai-evals): add inbox signals subsection for opt-in eval report s…
inkeep[bot] Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions contents/docs/ai-evals/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ Evaluations automatically assess the quality of your LLM [generations](/docs/ai-

## Choosing an evaluation type

| | LLM-as-a-judge | Code-based (Hog) | Sentiment analysis |
| --------------- | ------------------------------------------------------------ | ---------------------------------------------------------- | -------------------------------------------------------- |
| **Best for** | Subjective quality checks (tone, helpfulness, hallucination) | Deterministic rule-based checks (format, keywords, length) | Tracking how users feel during conversations |
| **Cost** | LLM API call per evaluation | Free | Free |
| **Speed** | Seconds | Milliseconds | Milliseconds |
| **Output** | Pass/fail with reasoning | Pass/fail with reasoning | Positive, neutral, or negative with a confidence score |
| **Setup** | Write a prompt | Write Hog code | Pick the type — no prompt or code required |
| | LLM-as-a-judge | Code-based (Hog) | Sentiment analysis |
| ------------ | ------------------------------------------------------------ | ---------------------------------------------------------- | ------------------------------------------------------ |
| **Best for** | Subjective quality checks (tone, helpfulness, hallucination) | Deterministic rule-based checks (format, keywords, length) | Tracking how users feel during conversations |
| **Cost** | LLM API call per evaluation | Free | Free |
| **Speed** | Seconds | Milliseconds | Milliseconds |
| **Output** | Pass/fail with reasoning | Pass/fail with reasoning | Positive, neutral, or negative with a confidence score |
| **Setup** | Write a prompt | Write Hog code | Pick the type — no prompt or code required |

## LLM-as-a-judge evaluations

Expand Down Expand Up @@ -351,6 +351,12 @@ You can adjust the report configuration after creating an evaluation:
- Add email or Slack delivery targets to receive notifications when reports are generated
- Disable the report if you don't need automated summaries

### Inbox signals

Evaluation reports can appear as signals in your [inbox](/docs/self-driving/inbox). To receive these signals, enable the **AI observability** source in your inbox signal sources settings.

When enabled, the inbox receives a signal each time a report is generated, letting you review evaluation trends alongside other product signals. This is opt-in and disabled by default.

## Evaluation status and errors

Each evaluation has a status that indicates whether it's running:
Expand All @@ -363,10 +369,10 @@ Each evaluation has a status that indicates whether it's running:

PostHog automatically disables evaluations and sets them to error status when:

| Error reason | Cause | How to resolve |
| ------------------------ | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| Error reason | Cause | How to resolve |
| ------------------------ | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Trial limit reached** | The 100 free trial evaluation runs have been exhausted. | Add your own API key in [**Settings** > **AI Observability**](https://app.posthog.com/settings/environment-ai-observability#ai-observability-byok). |
| **Model not allowed** | The selected model isn't available on the trial plan. | Switch to a trial-allowed model, or add your own provider API key. |
| **Model not allowed** | The selected model isn't available on the trial plan. | Switch to a trial-allowed model, or add your own provider API key. |
| **Provider key deleted** | The provider API key used by this evaluation was removed. | Add a new provider API key in [**Settings** > **AI Observability**](https://app.posthog.com/settings/environment-ai-observability#ai-observability-byok). |

### Identifying errored evaluations
Expand Down
13 changes: 10 additions & 3 deletions contents/docs/self-driving/inbox/sources.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,16 @@ sidebar: Docs
showTitle: true
---

import { CallToAction } from 'components/CallToAction'
import List from 'components/List'
import { CallToAction } from "components/CallToAction";
import List from "components/List";

<CalloutBox icon="IconFlask" title="Open beta" type="action">

Self-driving is currently in open beta.

</CalloutBox>

Signal sources are what fill your inbox. A [signal](/docs/self-driving/signals) can be a production error, a support conversation, a Session Replay pattern, a log alert changing state, a Replay Vision scanner finding, or an issue from an external tracker. Each source watches one of these streams and turns what it finds into work worth investigating.
Signal sources are what fill your inbox. A [signal](/docs/self-driving/signals) can be a production error, a support conversation, a Session Replay pattern, a log alert changing state, a Replay Vision scanner finding, an AI evaluation report, or an issue from an external tracker. Each source watches one of these streams and turns what it finds into work worth investigating.

## Configure sources

Expand Down Expand Up @@ -75,6 +75,13 @@ These sources are available when the matching PostHog product has data in your p
description: "Outdated SDKs, proxy gaps, ingestion warnings, broken data models, and more.",
lineClamp: 2,
},
{
label: "AI observability",
url: "/docs/ai-evals/evaluations",
icon: "IconAI",
description: "Findings from evaluation reports on your LLM traffic.",
lineClamp: 2,
},
]}
/>

Expand Down
Loading