Skip to content

nsq_to_file: with a large max-in-flight some messages unprocessed during a SIGTERM#1521

Draft
danbf wants to merge 1 commit into
nsqio:masterfrom
danbf:fix-messages-lost-at-SIGTERM
Draft

nsq_to_file: with a large max-in-flight some messages unprocessed during a SIGTERM#1521
danbf wants to merge 1 commit into
nsqio:masterfrom
danbf:fix-messages-lost-at-SIGTERM

Conversation

@danbf

@danbf danbf commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Root cause: Go's select chooses randomly when multiple cases are ready simultaneously. At shutdown with 1000 max-in-flight:

The last handler goroutine sends its message to logChan (buffer=1, so it succeeds immediately and the goroutine returns)
go-nsq sees all handlers have returned → closes StopChan
Now both StopChan and logChan are ready at the same time
If select picks StopChan, the message in logChan is abandoned — never written to disk, never Finish()'d
The higher the max-in-flight, the more often this happens because there's more concurrency in the final draining.

Fix: The StopChan case now runs a non-blocking drain loop over logChan before setting sync = true. Since StopChan only closes after all handler goroutines return (and each goroutine sends exactly one message and returns), the buffer holds at most one message at that moment — but the loop is written generically with a labeled break for safety.

NoteL This may not be important as re-queuing should prevent messages from getting dropped.

@danbf danbf marked this pull request as draft April 14, 2026 16:34
@danbf danbf changed the title nsq_to_file: with a large max-in-flight some messages can be lost during a SIGTERM nsq_to_file: with a large max-in-flight some messages unprocessed during a SIGTERM Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant