Multiple outputs backpressure behavior – one output blocking others

[razbahri]

Hi,

I’m running Fluent Bit as a FireLens sidecar on ECS, replacing the CloudWatch Agent.

Setup

• tail input (stdout)
• Two outputs:
  • CloudWatch Logs
  • OpenAPM

Issue
When one output becomes unavailable:

• The buffer fills up
• Once limits are reached, Fluent Bit stops ingesting new logs
• This blocks both outputs, including the one that is still healthy

Understanding
From the docs, it seems Fluent Bit has a single shared buffer per input, and backpressure is applied at the input level.
So if one output is blocked and the buffer is full, all outputs are affected.

Is this correct?

Question

Is there any supported way to:

• Isolate buffering per output, or
• Allow healthy outputs to continue even if one output is unavailable?

Thanks.

[patrick-stephens]

Adjusting the retry limits on the output would likely prevent it blocking, I'm guessing at the moment you have defaults of retry forever on both outputs?

[patrick-stephens]

https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configure-retries

[Katarina1134]

Your understanding is correct — Fluent Bit uses a single shared chunk buffer per input, and backpressure propagates upstream when any output's retry queue fills. This is a known architectural constraint, not a misconfiguration.

The retry limit suggestion from @patrick-stephens helps prevent indefinite blocking, but it trades blocking for data loss — once retries are exhausted, those events are dropped. Whether that's acceptable depends on your requirements for CloudWatch vs OpenAPM.

Practical options for your FireLens setup:

1. Set asymmetric retry limits per output
If CloudWatch is your primary destination and OpenAPM is secondary/best-effort:

[OUTPUT]
    Name cloudwatch_logs
    Retry_Limit False    # retry forever for primary

[OUTPUT]
    Name http
    Match *
    # OpenAPM endpoint
    Retry_Limit 3        # drop after 3 retries, don't block primary

This lets OpenAPM fail fast while CloudWatch continues retrying.

2. Use separate Fluent Bit instances per output
With FireLens on ECS, you can run two sidecar containers — one routing to CloudWatch, one to OpenAPM — each with their own input and buffer. Complete isolation, no shared backpressure. The cost is doubled memory/CPU for the agent layer, but on ECS this is usually acceptable.

3. Add a filesystem buffer with output-level limits
If you want both outputs to continue independently without dropping:

[OUTPUT]
    Name cloudwatch_logs
    storage.type filesystem

[OUTPUT]
    Name http
    storage.type filesystem

With filesystem buffering, each output gets its own queue on disk. Backpressure still applies per-output rather than globally, reducing the chance of one slow output blocking the other. You'll need to tune storage.max_chunks_up to control memory usage.

The fundamental constraint
Fluent Bit's single-buffer-per-input design means true per-output isolation requires either separate processes or filesystem buffering with careful tuning. There's no configuration flag that gives you fully independent output queues within a single process.

For what it's worth, this shared-buffer backpressure coupling is one of the harder operational problems in C-based log agents — it's a consequence of the single-threaded event loop model. When building WarpParse we made per-sink queues independent by default, so a slow sink can only affect its own throughput. Happy to share more detail if useful.