test(llm-validation): A/B experiment on AGENTS.md usefulness + new config-migration benchmark case#4988
Draft
anna-git wants to merge 7 commits into
Draft
test(llm-validation): A/B experiment on AGENTS.md usefulness + new config-migration benchmark case#4988anna-git wants to merge 7 commits into
anna-git wants to merge 7 commits into
Conversation
Introduces the .llm-validation/ scaffold that wires dd-trace-go into the Datadog LLM Validation Platform gate. PRs that modify the monitored instruction files (AGENTS.md, contrib/AGENTS.md, ddtrace/tracer/AGENTS.md, internal/AGENTS.md) will be automatically judged: the gate compares the PR's instruction files against main through the Datadog AI Gateway and posts a PASS/WARN/FAIL verdict, blocking the merge on a confident regression. Cases in suites/dd-trace-go-agent-v0.1.yaml: - dd-trace-go-contrib-structure-001 (blocking): Go workspace layout, per-contrib go.mod, go test must run from inside the contrib dir - dd-trace-go-contrib-tags-002 (blocking): span.kind and component tags - dd-trace-go-tracer-api-break-003 (warn-only): breaking-api-acknowledged label and make apidiff/incompatible - dd-trace-go-control-general-go-004 (warn-only): control case (AGENTS.md must not bloat answers to unrelated Go questions) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add the cross-project include from llm-validation-platform so the gate job runs automatically when a monitored instruction file changes in a PR. The job self-skips when no monitored file changed; a FAIL blocks the merge. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Temporary: reference the generic platform branch until it merges to main. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dd-trace-go uses main, not master. Platform default is master. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on gate Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Deletes the root, contrib/, ddtrace/tracer/, and internal/ AGENTS.md files as an llm-validation candidate variant to measure whether the agent performs better, worse, or the same without them.
Tests whether the agent, when asked to add a new DD_* env var, routes it through internal/config (loadConfig -> provider precedence -> getter/setter with checkProductConflict + configtelemetry.Report) instead of reading it directly via os.Getenv or a legacy helper, per the internal/config migration Mikayla Toffler has been landing (internal/config/README.md, internal/README.md).
anna-git
added
the
AI Generated
github-actions
Bot
added
the
apm:ecosystem
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Contributor
Author
AGENTS.md present vs. deleted — scoring comparison4 runs/case, blind pairwise comparison judged by
Why case 3 flips it: with the real Bottom line: on the two original cases, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Demonstrates an A/B experiment using the LLM validation platform's gateway-backed comparative judge to measure whether
AGENTS.mdinstruction files actually change agent behavior/quality in dd-trace-go, and adds a new, harder benchmark case that specifically targets the recentinternal/configmigration pattern.Commit-by-commit:
607249e98— intentionally-badcontrib/AGENTS.mdvariant, used to confirm the gate can detect a real regression when instructions are actively sabotaged.4b96275df— full deletion of all monitoredAGENTS.mdfiles, used to A/B "real AGENTS.md" vs. "no AGENTS.md at all."973947f01— adds a new suite case,dd-trace-go-config-new-env-var-005, which asks an agent to wire up a new env-var-backed config value and checks whether it follows theinternal/configsingleton/provider/precedence pattern (owning module,loadConfig(), getter/setter,checkProductConflict,configtelemetry.Report, key registration) instead of readingos.Getenvdirectly or copying legacy helpers likeinternal.BoolEnv.Motivation
We wanted empirical evidence (not just intuition) on whether
AGENTS.mdfiles are pulling their weight in this repo. The first two existing cases showed no measurable difference between real and deleted AGENTS.md (50-62% win-rate, within the noise band). The new case was designed to stress-test a scenario where an agent's instinct to mirror nearby (partially legacy) code could lead it astray without explicit guidance — and it did produce a real signal: baseline (real AGENTS.md) failed safety criteria in 3/4 runs by mirroring a legacy env-var read pattern, while candidate scored +23.1 quality and 75% win-rate on that case.This PR is exploratory/demonstrative — it's meant to open the little CI check square so the LLM validation gate result is visible on a real PR, not to merge sabotage/deletion commits into
dd-trace-go.Reviewer's Checklist
make lintlocally.make testlocally.make generatelocally.make fix-moduleslocally.Unsure? Have a question? Request a review!