fix: prevent pipeline errors from killing Reactor subscription

[usmansaleem]

Summary

• Wrap each pipeline handler invocation in a try-catch so errors are contained at the handler level and never propagate to the Reactor stream
• Add catch(Throwable) to HandshakeMessagePacketHandler for defense-in-depth, matching the existing pattern in MessagePacketHandler

Root Cause

PipelineImpl chains handlers using doOnNext, but Reactor's onErrorContinue does not reliably work with doOnNext (a known Reactor limitation — it only supports operators like map/flatMap). When a StackOverflowError escaped a handler, it could intermittently terminate the Reactor subscription, causing the node to stop processing all subsequent incoming packets.

This caused the flaky timeout in DiscoveryIntegrationTest.shouldRecoverAfterErrorWhileDecodingInboundMessage().

Test plan

• shouldRecoverAfterErrorWhileDecodingInboundMessage passes
• Full test suite passes

---

Note

Medium Risk
Changes core packet processing flow by swallowing all Throwable from pipeline handlers; this reduces outages but could mask bugs and alter failure semantics/logging. Also adjusts handshake error handling to drop sessions on unexpected JVM errors.

Overview
Prevents individual EnvelopeHandler failures from terminating the Reactor pipeline by wrapping each doOnNext handler invocation in a try/catch (Throwable) and logging a warning instead of letting errors propagate.

Adds defense-in-depth in HandshakeMessagePacketHandler by catching unexpected Throwable during handshake processing, logging at warn, and marking the handshake/session as failed to ensure cleanup continues even on non-Exception errors.

^{Written by Cursor Bugbot for commit bc34525. This will update automatically on new commits. Configure here.}

[gfukushima]

lgtm

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.