Add UDP video protocol + client/server screen sharing support#7162
Draft
maxer137 wants to merge 9 commits intomumble-voip:masterfrom
Draft
Add UDP video protocol + client/server screen sharing support#7162maxer137 wants to merge 9 commits intomumble-voip:masterfrom
maxer137 wants to merge 9 commits intomumble-voip:masterfrom
Conversation
Introduces the MumbleUDP::Video protobuf message for carrying encoded video frame fragments over UDP. Adds the Video entry to the MUMBLE_ALL_UDP_MESSAGES macro and a corresponding VideoData struct in MumbleProtocol.h with fields for sender session, codec, resolution, fragmentation info, payload, and keyframe flag. Also adds the screen_sharing boolean field to MumbleProto::UserState (Mumble.proto) and the matching bScreenSharing member to the shared User data model, providing the state foundation needed by both server and client for screen sharing sessions. Stub switch cases for UDPMessageType::Video are added in ServerHandler and Server to keep the compiler happy; full decoding and relay follow in subsequent commits.
Introduce full support for decoding, handling, and relaying UDP video packets within the Mumble protocol stack. This change adds a VideoData structure and corresponding decoding path in UDPDecoder, including a new decodeVideo_protobuf implementation. Video packets are now parsed into a strongly typed structure with validation for resolution, fragment bounds, and payload presence. Protocol-related adjustments include: - Addition of UDPDecoder::getVideoData accessor - Introduction of decodeVideo_protobuf path - Strongly typed VideoData payload handling - Replacement of video codec field with enum type Also handles the screen_sharing UserState field server-side: when a client toggles screen_sharing, the server broadcasts a TextMessage to the channel announcing that the user started or stopped sharing, and updates bScreenSharing on the ServerUser.
…oss-platform foundation)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change introduces initial support for screen sharing in Mumble by extending the protocol, server, and client to handle a new UDP video message type.
The current implementation focuses on wiring and basic functionality.
I intend to keep this PR more as a tracker. Per component, I hope to make PRs and track them in this list.
Protocol
To ensure video streaming works, a video message type for the Mumble protocol is added.
This video message type contains codec, resolution, fragmentation metadata, the video payload, and a keyframe flag.
This probably should get expanded further to enable Scalable Video Coding (SVC) in the future.
Server
Add a simple relay that doesn't decode the video packets but simply passes on the video content to the other users in the channel. Decoding and encoding of video packets is done on client devices to reduce computational load on the server.
Ideally, the only computational load that gets added to the server is for deciding which packet needs to get sent to what user.
Client
Added a screen capture implementation. This screen capture implementation uses FFmpeg libraries to encode captured frames. As it stands, QT does not implement neat window picker implementations for video capture, especially for macOS and Wayland.
Build
FFmpeg libraries would become a new dependency for Mumble on the client.
Notes
This PR is kept as a draft and is an initial implementation and not considered complete.
UDP video handling is minimal and currently only supports the H.264 codec.
I intend this PR to keep track of progress on this larger feature and hope to stabilize parts of the PR step by step.
Fixes / relates to: #4150