Skip to content

feat(internal/log,telemetry): add SDK error-reporting API for Error Tracking#4997

Draft
darccio wants to merge 2 commits into
mainfrom
dario.castane/hungry-einstein-959818
Draft

feat(internal/log,telemetry): add SDK error-reporting API for Error Tracking#4997
darccio wants to merge 2 commits into
mainfrom
dario.castane/hungry-einstein-959818

Conversation

@darccio

@darccio darccio commented Jul 8, 2026

Copy link
Copy Markdown
Member

Summary

  • Auto-forward sink: hooks internal/log.Error so every non-rate-limited call is mirrored to instrumentation telemetry. The format string is the constant dedup key; variadic error args are scrubbed via NewSafeError; a redacted stack trace is always attached. No change to local log output.
  • Policy table (internal/telemetry/log/policy.go): single place to classify format templates as report / downgrade (→ warn) / exclude. Pre-seeded with agent-connectivity and user-misconfiguration noise.
  • Explicit helpers (internal/telemetry/log/helpers.go): ReportError(msg, err, opts...) for swallowed errors; ReportPanic(recovered, msg) for recover() sites. Both enforce the constant-message contract.
  • constantlogmsg analyzer (internal/telemetry/log/analyzer/): go vet-compatible pass that rejects non-constant message arguments. Standalone binary for go vet -vettool; make lint/errlog target added.

Test plan

  • go build ./... — no compile errors
  • go vet ./internal/log/... ./internal/telemetry/log/... — clean
  • go test ./internal/log/... ./internal/telemetry/log/... -count=1 -race — all green (17 new tests)
  • make lint/errlog — no violations in existing codebase
  • No import cycle introduced (internal/log does not import internal/telemetry/log)
  • Local log output unchanged (verified by TestErrorLocalOutputUnchanged)

Reviewer notes

  • The sink is registered via init() in forward.go; it is active whenever any code imports internal/telemetry/log. Existing uses of log.Error require no changes — the auto-forward handles them. Non-constant first arguments will be flagged by the new analyzer.
  • The policyTable in policy.go is intentionally sparse. Prompt 2 (migration) will populate it as call sites are classified.
  • The analyzer uses fake package paths in testdata (bypasses the internal import restriction) and a New([]FuncSpec) constructor for test-scoped configuration.

Related epic: APMLP-1503

…racking

Introduce a centralized, PII-safe pipeline that forwards SDK-originated
errors from `internal/log.Error` to instrumentation telemetry (→ Error
Tracking) without changing existing local log output.

## What changed

**Auto-forward sink** (`internal/log/log.go`):
- `SetErrorTelemetrySink(f)` installs a forwarding hook; called before
  the local aggregation lock so telemetry is notified on every
  non-rate-limited `Error` call.
- `atomic.Pointer` ensures lock-free concurrent access.

**Sink registration** (`internal/telemetry/log/forward.go`):
- `init()` registers `forwardError` with `log.SetErrorTelemetrySink`.
- Uses the raw format string as the constant telemetry message (never
  interpolated); type-switches args to attach only `error` values, each
  scrubbed through `NewSafeError`.
- Always attaches a redacted stacktrace via `WithStacktrace()`.

**Policy table** (`internal/telemetry/log/policy.go`):
- Single source of truth for per-template actions: `report` (default),
  `downgrade` (→ warn), or `exclude`.
- Pre-seeded with agent-connectivity and user-misconfiguration templates.

**Explicit helpers** (`internal/telemetry/log/helpers.go`):
- `ReportError(msg, err, opts...)` — for swallowed-error branches.
- `ReportPanic(recovered, msg)` — for `recover()` sites.
- Both enforce the constant-message contract and policy table.

**`constantlogmsg` analyzer** (`internal/telemetry/log/analyzer/`):
- `go/analysis` pass that rejects non-constant first arguments on
  `log.Error`, `log.Warn`, `ReportError`, and `ReportPanic`.
- Standalone `cmd/main.go` runnable as `go vet -vettool`.
- Makefile target `make lint/errlog`.

**`MIGRATION-CONTRACT.md`**: canonical before/after for each of the
five call-site patterns Prompt 2 (automated migration) will encounter.

No existing log output is changed. No import cycle is introduced.
All tests pass under `-race`.
var policyTable = map[string]policyAction{
// ── Agent / network connectivity ────────────────────────────────────────
// These reflect user environment issues, not SDK bugs.
"failure sending traces (attempt %d of %d): %v": policyExclude,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
File is not properly formatted (gofmt)

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 8, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 6 Pipeline jobs failed

Static Checks | check-docs   View in Datadog   GitHub Actions

Static Checks | check-format   View in Datadog   GitHub Actions

Static Checks | copyright   View in Datadog   GitHub Actions

View all 6 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 2 jobs - 2 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 86.11%
Overall Coverage: 62.78% (+11.35%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a9d8023 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jul 8, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-09 15:54:50

Comparing candidate commit a9d8023 in PR branch dario.castane/hungry-einstein-959818 with baseline commit bb7fa5b in branch main.

Found 5 performance improvements and 0 performance regressions! Performance is the same for 321 metrics, 0 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:BenchmarkOTLPProtoSize/1000spans

  • 🟩 execution_time [-22.836µs; -21.559µs] or [-6.733%; -6.357%]

scenario:BenchmarkOTLPProtoSize/100spans

  • 🟩 execution_time [-2.386µs; -2.261µs] or [-7.026%; -6.656%]

scenario:BenchmarkOTLPProtoSize/10spans

  • 🟩 execution_time [-253.586ns; -242.414ns] or [-7.273%; -6.953%]

scenario:BenchmarkOTLPProtoSize/1span

  • 🟩 execution_time [-38.200ns; -36.260ns] or [-9.001%; -8.545%]

scenario:BenchmarkPayloadVersions/simple_1000spans/v1.0

  • 🟩 allocated_mem [-1 bytes; -1 bytes] or [-2.857%; -2.857%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:BenchmarkOTLPTraceWriterFlush

@kakkoyun

kakkoyun commented Jul 9, 2026

Copy link
Copy Markdown
Member

We should remove these files and make the new analyzer includes the functionality https://github.com/DataDog/dd-trace-go/blob/main/rules/telemetry_rules.go and https://github.com/DataDog/dd-trace-go/blob/main/rules/logging_rules.go

… telemetrylog.{Debug,Warn,Error}

Extends DefaultFuncs to check all three telemetry log functions (and their
Logger method counterparts) in addition to the existing ReportError/ReportPanic
helpers. A single FuncSpec per name covers both call styles because resolveFunc
maps both pkg-level and method calls to the same (pkgPath, name) key via the
type checker.

Adds a skipPkgs mechanism (DefaultSkipPkgs, New variadic param) so the analyzer
does not flag the internal/telemetry/log package's own delegation calls — e.g.
func Error(message string,...){ defaultLogger.Load().Error(message,...) }.

Adds TestAnalyzerSkip to verify that listed packages are silently ignored even
when they contain variable-message calls.

This brings constantlogmsg to feature parity with the telemetryLogConstantMessage
ruleguard rule in rules/telemetry_rules.go, making it possible to drop that rule
once this analyzer is wired into all CI paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants