Skip to content

fix(orchestrion): allow the experimental span pool together with orchestrion GLS#4928

Draft
kakkoyun wants to merge 1 commit into
kakkoyun/orchestrion-gls-peek-skip-donefrom
kakkoyun/orchestrion-gls-remove-gate
Draft

fix(orchestrion): allow the experimental span pool together with orchestrion GLS#4928
kakkoyun wants to merge 1 commit into
kakkoyun/orchestrion-gls-peek-skip-donefrom
kakkoyun/orchestrion-gls-remove-gate

Conversation

@kakkoyun

@kakkoyun kakkoyun commented Jun 23, 2026

Copy link
Copy Markdown
Member

What

Final PR in the GLS span-pool stack. Removes the mutual-exclusion gate from #4891 so the experimental span pool (WithSpanPool(true) / DD_TRACER_EXPERIMENTAL_SPAN_POOL_ENABLED) is honored under orchestrion, and adds the coexistence regression test.

Why

With the reclaim signal decoupled from the recyclable span (PR 1) and Peek skipping finished entries (PR 2), the two reasons #4891 gated the features no longer hold: a recycled span can neither leak a stale GLS entry nor resurface as the active span. The gate (shouldDisableSpanPool, its newConfig call, and its unit test) can go.

How

  • Remove the gate and the shouldDisableSpanPool helper + test.
  • Add TestGLSNoHeapLeakWithSpanPool: enables the pool via WithSpanPool(true) and runs glsleak.MeasureLeakLiveInject — the realistic franz-go shape that injects each span while live and finishes it after the worker pushes it, respecting the span pool's "do not touch a span after Finish" contract. Run under -race in CI it guards both the leak and the span-pool-vs-GLS data races.
  • The existing finish-then-inject TestGLSNoHeapLeak / TestSpanGLS*CrossGoroutine stay as no-pool gates (their finish-before-inject order is a deliberate use-after-Finish that the pool legitimately recycles), now documented as such.

Validation

Built woven with orchestrion v1.10.0:

  • gls, gls-leak, os, gin gates pass (non-race).
  • gls-leak -race: TestGLSNoHeapLeak (no-pool) and TestGLSNoHeapLeakWithSpanPool (live-inject + WithSpanPool(true)) pass — flat retained-heap, no race.
  • gls -race (no-pool) passes; under forced pooling only the finish-then-inject (use-after-Finish) tests fail, as expected — every other gls test (double-finish, concurrency, all dyngo cross-goroutine) passes pooled.
  • apidiff incompatible clean; golangci-lint 0 issues.

Stack (merge in order)

  1. fix(orchestrion): decouple the GLS reclaim signal from the recyclable span #4926 — decouple the reclaim signal
  2. fix(orchestrion): skip finished entries in GLS Peek to avoid surfacing recycled spans #4927Peek skips finished entries
  3. fix(orchestrion): allow the experimental span pool together with orchestrion GLS #4928 (this PR) — remove the gate + coexistence regression test

Based on #4927 (which is based on #4926); review/merge those first.

Refs: #4891 · DataDog/orchestrion#782

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jun 23, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 7 Pipeline jobs failed

Orchestrion | Integration Test (ubuntu | oldstable | DRIVER)   View in Datadog   GitHub Actions

Orchestrion | Integration Test (ubuntu | oldstable | TOOLEXEC)   View in Datadog   GitHub Actions

Orchestrion | Integration Test (ubuntu | stable | DRIVER)   View in Datadog   GitHub Actions

View all 7 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 62.94% (+0.08%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fe8cf33 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 23, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-23 14:23:22

Comparing candidate commit fe8cf33 in PR branch kakkoyun/orchestrion-gls-remove-gate with baseline commit 2e35de7 in branch kakkoyun/orchestrion-gls-peek-skip-done.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 297 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:BenchmarkOTLPTraceWriterFlush

@kakkoyun kakkoyun force-pushed the kakkoyun/orchestrion-gls-peek-skip-done branch from c24407c to 2e35de7 Compare June 23, 2026 13:56
…estrion GLS

Final PR in the GLS span-pool stack. With the reclaim signal decoupled from the
recyclable span (cell-based, captured at push time) and Peek skipping finished
entries, the two reasons #4891 gated span pool and orchestrion GLS as mutually
exclusive no longer hold: a recycled span can neither leak a stale GLS entry nor
resurface as the active span. Remove the gate (the shouldDisableSpanPool helper,
its newConfig call, and its unit test) so WithSpanPool(true) /
DD_TRACER_EXPERIMENTAL_SPAN_POOL_ENABLED is honored under orchestrion.

Add TestGLSNoHeapLeakWithSpanPool as the coexistence regression gate. It enables
the pool via WithSpanPool(true) and runs glsleak.MeasureLeakLiveInject: the
realistic franz-go shape that injects each span while live and finishes it after
the worker pushes it, respecting the span pool's "do not touch a span after
Finish" contract. Run under -race in CI it guards both the leak and the
span-pool-vs-GLS data races. The existing finish-then-inject TestGLSNoHeapLeak
stays as the no-pool gate (its finish-before-inject order is a deliberate
use-after-Finish that the pool legitimately recycles, so it is not run pooled).

Refs: DataDog/orchestrion#782

Environment: Datadog workspace

Co-authored-by: Kemal Akkoyun <kakkoyun@users.noreply.github.com>
@kakkoyun kakkoyun force-pushed the kakkoyun/orchestrion-gls-remove-gate branch from b8baea9 to fe8cf33 Compare June 23, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant