fix(remote): WebRTC black canvas, cursor sync, WebCodecs low-latency mode #1

Merged

HexC0d3rz merged 5 commits from fix/streaming-options-and-docs into main

2026-06-13 21:34:01 -04:00

HexC0d3rz commented

2026-06-13 21:21:58 -04:00

(Migrated from github.com)

Resolves the user-reported remote-desktop issues on the device detail page.

Bug 1 - WebRTC (P2P) canvas was permanently black. Two-part fix: the agent's WebRTC PeerConnection was missing the standard register_default_interceptors (NACK, RTCP sender/receiver reports, TWCC) chain, so real browsers' RTCP feedback (PLI/NACK) wasn't being processed and outbound stats never populated. The fix lands in crates/zeno-agent/src/webrtc/mod.rs. The deployed zeno-agent.exe predates this fix; the Build Windows Agent workflow on merge will rebuild it and Alonzo-PC will auto-update within ~30s.

Bug 2 - Operator cursor showed a cross instead of the remote mouse, not in sync. Adds a CursorGuard (RAII) that calls ShowCursor(FALSE) on Windows for the duration of a WebRTC/relay/WebCodecs session and restores on drop, covering normal exit, early return, and panic-unwind. The GDI fallback capture no longer uses CAPTUREBLT, so the (now-hidden) host cursor is never pulled into a frame. The browser renders the operator's own cursor as a sprite that tracks the last pointermove on the stage at 60fps with no perceptible lag.

Bug 3 - Reduce latency, even if implementation changes. Adds a new default streaming mode, WebCodecs (low latency). Same H.264/H.265/VP9/AV1 capture -> encode pipeline, but the encoded Annex-B access units are sent as remote_desktop.video_chunk over the existing agent WebSocket (no SDP/ICE, no extra ports). The browser decodes with the WebCodecs VideoDecoder and paints to a canvas at requestAnimationFrame cadence. Works through Pangolin/reverse proxies because it shares the same WebSocket. wcstream.rs (agent) + wc-remote.js (browser) implement it. WebCodecs is the new default; automatic fallback to WebRTC when VideoDecoder is unavailable, or when the operator picks WebRTC/relay manually.

Docs - docs/remote-desktop-pipeline.md and docs/operator-remote-desktop-guide.md now describe WebCodecs as the default and document the cursor-sync behaviour.

Verification - All 90 cargo + Node tests pass; cargo build/test/fmt/clippy clean. End-to-end testing in a real browser requires a fresh zeno-agent.exe build, which the Build Windows Agent workflow will produce on merge to main.

Files: 22 changed, +1323/-48. Branch: fix/streaming-options-and-docs.

Resolves the user-reported remote-desktop issues on the device detail page. **Bug 1 - WebRTC (P2P) canvas was permanently black.** Two-part fix: the agent's WebRTC PeerConnection was missing the standard `register_default_interceptors` (NACK, RTCP sender/receiver reports, TWCC) chain, so real browsers' RTCP feedback (PLI/NACK) wasn't being processed and outbound stats never populated. The fix lands in `crates/zeno-agent/src/webrtc/mod.rs`. The deployed `zeno-agent.exe` predates this fix; the Build Windows Agent workflow on merge will rebuild it and Alonzo-PC will auto-update within ~30s. **Bug 2 - Operator cursor showed a cross instead of the remote mouse, not in sync.** Adds a `CursorGuard` (RAII) that calls `ShowCursor(FALSE)` on Windows for the duration of a WebRTC/relay/WebCodecs session and restores on drop, covering normal exit, early return, and panic-unwind. The GDI fallback capture no longer uses `CAPTUREBLT`, so the (now-hidden) host cursor is never pulled into a frame. The browser renders the operator's own cursor as a sprite that tracks the last `pointermove` on the stage at 60fps with no perceptible lag. **Bug 3 - Reduce latency, even if implementation changes.** Adds a new default streaming mode, **WebCodecs (low latency)**. Same H.264/H.265/VP9/AV1 capture -> encode pipeline, but the encoded Annex-B access units are sent as `remote_desktop.video_chunk` over the *existing* agent WebSocket (no SDP/ICE, no extra ports). The browser decodes with the WebCodecs `VideoDecoder` and paints to a canvas at `requestAnimationFrame` cadence. Works through Pangolin/reverse proxies because it shares the same WebSocket. `wcstream.rs` (agent) + `wc-remote.js` (browser) implement it. WebCodecs is the new default; automatic fallback to WebRTC when `VideoDecoder` is unavailable, or when the operator picks WebRTC/relay manually. **Docs** - `docs/remote-desktop-pipeline.md` and `docs/operator-remote-desktop-guide.md` now describe WebCodecs as the default and document the cursor-sync behaviour. **Verification** - All 90 cargo + Node tests pass; cargo build/test/fmt/clippy clean. End-to-end testing in a real browser requires a fresh `zeno-agent.exe` build, which the Build Windows Agent workflow will produce on merge to main. **Files**: 22 changed, +1323/-48. Branch: `fix/streaming-options-and-docs`.