Data loss in TCP -> File scenario when sender push rate exceeds engine processing capacity

[datasc31]

Description:
In the TCP -> File topology, data loss can be observed when the sender push rate significantly exceeds the engine’s actual processing capacity.
Under this condition, not all events sent by the sender are successfully ingested, parsed, and written to the output file.

Reproduction Scenario:

• Topology: TCP input -> parsing -> File output
• Sender configuration:
  • Push rate: ~900,000 events/sec
  • Total events sent: ~100,000 events
• Engine behavior:
  • Actual processing throughput is lower than the configured sender push rate.

Expected behavior:

• All 100,000 events should be received, parsed, and written to the output file, regardless of sender rate, as long as the sender completes transmission successfully.

Actual behavior:

• The output file contains fewer events than the number sent.
• Some events are missing, indicating data loss during ingestion or processing when the sender rate exceeds engine capacity.

Observations:

• Data loss occurs when the sender push rate is significantly higher than the engine’s sustainable processing throughput.
• The issue can also manifest when the sender rate is much higher than the total event count, causing the sender to finish very quickly.
• When the sender push rate is at or below the engine’s processing capacity, no data loss is observed.
• No explicit error messages or failures are reported during runtime.

Additional Notes:

• This issue was discovered during stability and correctness testing.
• It was not reproduced during standard performance benchmarking, where the sender push rate was configured below the engine’s processing capacity.
• The issue is reproducible with consistent results under the same configuration.