fix: deadlock in DefaultServerDispatcher

[smulloni]

Proposed changes

This fixes a deadlock in DefaultServerDispatcher when many requests timeout concurrently, matching the symptoms reported in #358.

DefaultServerDispatcher shares its RWMutex with ServerState (via NewServerState(&d.mutex)). When more than 10 client request timeouts fire simultaneously, waitForTimeout goroutines acquire RLock and block sending to timerC (buffer=10). Meanwhile messagePump, the sole timerC reader, calls HasPendingRequest which needs Lock on the same mutex — forming a circular wait that freezes the entire server permanently.

Three fixes applied:

Give ServerState its own RWMutex instead of sharing the dispatcher's. This breaks the cross-concern locking dependency.

In waitForTimeout, release RLock before sending to timerC. Previously the lock was held via defer across the blocking send.

In messagePump's timeout handler, inline the request completion instead of calling CompleteRequest, which sends to readyForDispatch (buffer=1). Since messagePump is the sole reader of that channel, this was a self-deadlock path when processing consecutive timeouts.

Additionally, read-only ServerState/ClientState methods (HasPendingRequest, GetPendingRequest, DeletePendingRequest, HasPendingRequests) now use RLock instead of Lock, and GetClientState uses a read-then-upgrade pattern, reducing contention under load.

Includes a regression test that reliably triggers the deadlock on unfixed code (15 clients, 50ms timeout, expects all callbacks to fire within 2s — previously only 1/15 would fire before freezing).

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of
them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before
merging your code.

• I have read the CONTRIBUTING doc
• I have signed the CLA (glad to, where do I do this?)
• Lint and unit tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in downstream modules

Further comments

There are several PRs partially addressing this issue, but as far as we can tell this is comprehensive (and explains why the problem can be mitigated through horizontal scaling). It is currently running in production for us at GreenJoulez.