Fix performance of tracing rate limiter#1530
Open
sergeymatov wants to merge 3 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces configurable rate limiting for tracing output by adding a tracing-throttle-based layer to tracectl, wiring it into dataplane startup, and exposing configuration via a new CLI argument in args.
Changes:
- Add
TracingRateLimitConfig/TracingControl::init_with_rate_limit()and initialize tracing with optional rate limiting. - Add
--tracing-rate-limit BURST:REPLENISH_PER_SECONDparsing and propagate it into the launch configuration. - Add the
tracing-throttledependency to the workspace andtracectl.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tracectl/src/lib.rs |
Re-exports rate-limit config type from control. |
tracectl/src/control.rs |
Adds rate-limit config + subscriber initialization changes, including throttling logic and tests. |
tracectl/Cargo.toml |
Adds tracing-throttle dependency. |
dataplane/src/main.rs |
Initializes tracing with optional rate limit from CLI args; adjusts default log level behavior. |
Cargo.toml |
Adds tracing-throttle to workspace dependencies. |
Cargo.lock |
Locks new tracing-throttle transitive dependency. |
args/src/lib.rs |
Adds TracingRateLimit type, CLI parsing/help, and config propagation + tests. |
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
7f3a6a7 to
0814455
Compare
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
0814455 to
0e90334
Compare
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
4 times, most recently
from
81f0207 to
65ae6b0
Compare
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
65ae6b0 to
510ff04
Compare
This reverts commit 75f964b. Signed-off-by: Sergey Matov <sergey.matov@githedgehog.com>
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
510ff04 to
c7c6cb5
Compare
sergeymatov
changed the title
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
3 times, most recently
from
dc1c83c to
55a1289
Compare
mvachhar
reviewed
May 19, 2026
Fredi-raspall
requested changes
May 21, 2026
Contributor
Fredi-raspall
left a comment
Fredi-raspall
left a comment
There was a problem hiding this comment.
Looks good to me overall, but I wish the PR included several changes.
- why two types for AtomicEnvFilter? Any chance a single type suffices?
- I'd remove the no longer used Result cases.
- I would like to see a test where a bunch of logs are produced and rate-limitted. There is a way to know if an event would be traced under a given config. Unsure if that is extensible when you have rate-limiting. It would be good to at least be able to see the effect.
- Super ideally: I wonder if two subscribers (one with rate limiting, one without) could be compared (profiled) to be able to see the impact in performance of the rate limiter. Unsure if this can be done in any meaningful / reliable way.
Avoid using RwLock to reduce complexity for EnvFilter. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Matov <sergey.matov@githedgehog.com>
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
55a1289 to
dcc9ef1
Compare
Contributor
Author
|
I will add test as separate commit. Thanks @Fredi-raspall |
Signed-off-by: Sergey Matov <sergey.matov@githedgehog.com>
Contributor
Author
|
@Fredi-raspall There is a simple test that measures emitted logs among throttled |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move to own atomic operations for per-level filter for callsites. Reloading of interest is done via ArcSwap rather then RwLock
Layered structure of throtteled/unthrotteled filters produces atomic reloading of
Interestfor each callsite.Layered<EnvFilter>::enable()invokes every time on each of tracing message, buttracing_subscriber::reloadpath that was previously used to check atomic state spawnedRwLock::read_contendedon each of such calls.Pyroscope flame graph analysis showed that those operations were causing extreme cost of CPU operations.
Proposed solution is to use
ArcSwapand precisely update callsite Interest for each of theLayer