Background
#2356 fixed the Safari crash where file publishing threw a generic decode error, and #2369 made it degrade gracefully: on browsers without HTMLMediaElement.captureStream (Safari/WebKit), a video file publishes video-only via a canvas.captureStream fallback, and its audio is dropped (with a warning). Audio-only files can't be published on Safari at all.
That's a stopgap, not the real fix. The whole file-publishing path in js/publish/src/source/file.ts (#decodeMedia) is built on a <video> element as an implicit demuxer/decoder, and then scrapes tracks back out via captureStream. That's why WebKit gaps (captureStream unimplemented; createMediaElementSource silent without a mic grant) leave us with no way to get file audio.
Proposal
Demux the file ourselves with MediaBunny (a WebCodecs-based container reader/writer) instead of routing through a media element. MediaBunny can read MP4/WebM/etc. and hand us EncodedVideoChunk / EncodedAudioChunk (or decoded frames), which we feed straight into the existing publish encode pipeline. This bypasses the <video> element, captureStream, and Web Audio entirely.
Benefits:
- Audio works on Safari (and everywhere), including audio-only files, since nothing depends on
captureStream or createMediaElementSource.
- True frame cadence / timestamps from the container, instead of resampling a playing
<video> onto a canvas at a fixed 30fps.
- Removes the browser-capability branching in
#decodeMedia and the canvas fallback.
Scope / open questions
- Pull MediaBunny in as a dependency of
@moq/publish (per the root CLAUDE.md, prefer a maintained crate/library over hand-rolling container parsing).
- Decide whether to hand the pipeline decoded frames or re-mux encoded chunks (ideally pass through encoded chunks when the codec already matches a supported publish config, to skip a decode+encode round-trip).
- Looping:
#decodeMedia currently sets video.loop = true; a demuxer-based path needs to loop by seeking back to the start.
- Feature-detect / fallback story: if MediaBunny doesn't support a given container, do we fall back to the current
<video> path or just error.
Once this lands, the captureStream feature-detect and the canvas fallback from #2369 can be removed.
Background
#2356 fixed the Safari crash where file publishing threw a generic decode error, and #2369 made it degrade gracefully: on browsers without
HTMLMediaElement.captureStream(Safari/WebKit), a video file publishes video-only via acanvas.captureStreamfallback, and its audio is dropped (with a warning). Audio-only files can't be published on Safari at all.That's a stopgap, not the real fix. The whole file-publishing path in
js/publish/src/source/file.ts(#decodeMedia) is built on a<video>element as an implicit demuxer/decoder, and then scrapes tracks back out viacaptureStream. That's why WebKit gaps (captureStreamunimplemented;createMediaElementSourcesilent without a mic grant) leave us with no way to get file audio.Proposal
Demux the file ourselves with MediaBunny (a WebCodecs-based container reader/writer) instead of routing through a media element. MediaBunny can read MP4/WebM/etc. and hand us
EncodedVideoChunk/EncodedAudioChunk(or decoded frames), which we feed straight into the existing publish encode pipeline. This bypasses the<video>element,captureStream, and Web Audio entirely.Benefits:
captureStreamorcreateMediaElementSource.<video>onto a canvas at a fixed 30fps.#decodeMediaand the canvas fallback.Scope / open questions
@moq/publish(per the root CLAUDE.md, prefer a maintained crate/library over hand-rolling container parsing).#decodeMediacurrently setsvideo.loop = true; a demuxer-based path needs to loop by seeking back to the start.<video>path or just error.Once this lands, the
captureStreamfeature-detect and the canvas fallback from #2369 can be removed.