feat(mediaplayer): split-stream, on-demand delivery, and optional page-URL resolution

[towneh]

Summary

Several related additions to the media player, plus an optional integration package, that together let it play high-resolution adaptive sources (YouTube/Twitch and similar) — not just the live broadcast URLs it handles today.

1. Split-stream playback. Adaptive sources above ~360p serve video and audio as separate streams. basis_media_open_dual runs a second demux thread for an audio-only URL into the same decoder, so both present in sync on one clock. Exposed via BasisMediaSource.AudioUri (null = single muxed stream, unchanged).

2. Live vs on-demand, auto-detected. BasisMediaSource.Delivery (Auto/Live/OnDemand) selects the playback clock. Auto resolves at open: a finite, byte-range-seekable HTTP body (known Content-Length + Accept-Ranges) or an HLS playlist with EXT-X-ENDLIST is treated as on-demand; an open-ended response or a non-HTTP transport is live. On-demand throttles delivery and presents on a fixed 1× clock with a compressed read-ahead buffer, so VOD that arrives faster than real time no longer fast-forwards — and the caller doesn't have to flag it.

3. Optional page-URL resolution. BasisMediaUrlRouter is a small seam the player consults before loading a URL. BasisMediaPlayerStreaming routes its StreamUrl through it: a directly-playable URL (transport scheme, or an HTTP URL with a media extension) loads straight through; a page URL (no media extension) goes to an installed resolver. With no resolver installed, a page URL reports that the resolver package is needed instead of failing silently.

The new com.basis.integration.ytdlp package registers a yt-dlp-based resolver into that seam — turning a YouTube/Twitch page URL into the playable stream(s) (a split avc1+mp4a pair for VOD, a single HLS stream for live).

4. Smooth live HLS. Live HLS (Twitch and similar) is paced by the engine's per-AU clock rather than a byte-rate token bucket, so variable-bitrate streams stay smooth through busy scenes while still presenting at — and converging to — the live edge.

5. Multiplayer-aware sync. BasisMediaPlayerNetworking syncs the page URL, not the resolved CDN URL (which is per-client and expiring), so every client resolves it independently; direct stream URLs sync as before.

Why a new seam (BasisMediaUrlRouter)

No existing player hook carries "resolve this URL before loading it," and the player package deliberately can't reference the integration package (removability — deleting the integration leaves the player intact). Dependency inversion via a registration seam is the only way to let the existing StreamUrl field steer page URLs without coupling the player to the resolver. The seam isn't an orphan extension point: its consumer (BasisMediaPlayerStreaming.TryResolveAndLoad) ships in this PR; the integration registers a second consumer. IsDirectlyPlayable lives on the player as the single classifier, and the resolver defers to it so the live-vs-resolve rule has one home.

Scope / prerequisite

• The dlp-native plugin (com.yewnyx.ytdlp) is intentionally not in this PR — how it enters the project is a separate vendoring decision, and it's a large binary package. The integration is gated #if BASIS_MEDIAPLAYER_EXISTS && YTDLP_EXISTS, so it compiles to nothing until that plugin is present. This PR is therefore safe to land without it — the resolver simply stays dormant until the plugin lands, then activates.
• Windows backend. Split-stream and the auto-detect run on the Windows (Media Foundation) decode path. Other platform backends are follow-ups.

Known limitations (documented in the package READMEs)

• Codec ceiling is H.264 + AAC, ~1080p — 4K YouTube is VP9/AV1-only, which the player can't decode, so resolution caps the chosen video at 1080p avc1.
• Steering gap: a direct HTTP stream with no file extension can't be told apart from a page URL; with a resolver installed it's sent to the resolver and fails rather than loading directly. Use a media extension or a transport scheme to avoid it.
• On-demand presents on a fixed internal buffer; BufferMilliseconds/BufferMode tune the live clock only.
• Progressive muxed VOD (a single demux thread) can intermittently starve audio; HLS and split-stream playback are clean

Demo

End-to-end: loading a range of sources through the same BasisMediaPlayerStreaming URL field — VRCDN live streams (direct), plus Twitch and YouTube (resolved via the yt-dlp integration) — each resolving and playing.

basis_pr871_video.mp4

3981a9f landing also fixes the above video's observed stutter on the initial A State Of Trance test HLS stream.

basis_pr871_hls_stutter_fix.mp4

What I need help with confirming?

Speaking to @dooly123 we need some consensus on how dlp-native gets embedded in to the client either dynamically vs static package, which needs comments from the maintainer (@yewnyx).

By that I mean either static — vendoring com.yewnyx.ytdlp (the native plugin plus its bundled CPython runtime) into the repo as an embedded package, so it ships with the client and works on a plain checkout, at the cost of a sizable binary blob in git — or dynamic, keeping those binaries out of the tree and pulling them at build/install time, which keeps the repo lean but adds a fetch step and a version-pinning/provenance story. It's largely a repo-weight vs build-pipeline trade-off, so we're after your steer on how you'd prefer the package to be consumed.

To be clear though, this isn't a blocker and there's no pressure on @yewnyx to land a particular answer quickly. The integration package is asmdef-guarded (#if BASIS_MEDIAPLAYER_EXISTS && YTDLP_EXISTS), so with dlp-native absent it compiles to nothing and the URL handling just falls back to the media player directly: a directly-playable URL loads exactly as it does today, and a YouTube/Twitch page URL surfaces a clear BasisDebug message that a resolver package is needed rather than failing silently or breaking the load path. So this PR is safe to land as-is with the resolver dormant — the embedding decision can be taken on its own timeline, and nothing here regresses without it.

Required checks

All boxes below must be ticked before this PR can merge. If a check is genuinely N/A, tick it anyway and explain under Notes.

• Tested — I built and ran this locally. The change works in the editor and (where relevant) in a built player.
• Transform access is combined and limited — In hot paths, transform reads/writes go through TransformAccessArray or are otherwise batched. I have not added per-frame transform.position / transform.rotation / transform.localPosition calls inside loops. Whenever I need both position and rotation, I use the combined APIs — SetPositionAndRotation / SetLocalPositionAndRotation for writes, GetPositionAndRotation / GetLocalPositionAndRotation for reads — instead of two separate property accesses; the combined call does one local-to-world matrix traversal instead of two.
• Addressables used for asset/memory loading — Any new asset loads go through Addressables. No new Resources.Load, no direct asset references that pull large content into memory on scene load.
• No new GetComponent / AddComponent where avoidable — Where unavoidable, the result is cached on a field, and any GetComponent<T> is replaced with TryGetComponent<T>(out var x) — bare GetComponent will be denied. TryGetComponent is the modern API (Unity 2019.2+) and skips the Editor-only GC allocation GetComponent causes when a component is missing: Unity wraps the null return in a managed "fake null" object so its overloaded == operator can still detect destroyed C++ objects, and constructing that wrapper allocates; TryGetComponent returns a bool plus out parameter and never builds the wrapper. None of these calls run inside Update, LateUpdate, FixedUpdate, jobs, or other per-frame code paths.
• Per-frame work is scheduled through BasisEventDriver — Any new per-frame work hooks into BasisEventDriver rather than adding standalone Update / LateUpdate / FixedUpdate callbacks on a MonoBehaviour.
• Anything added to BasisEventDriver is bulletproof, or guarded by try/catch — BasisEventDriver runs the single per-frame tick that drives the whole framework (network apply, local player sim, blendshapes, JigglePhysics, nameplates, and more) as one sequential chain. An unhandled exception anywhere in that chain aborts the rest of the tick, so every step after the throwing one is silently skipped for that frame. New work added to the driver must either be guaranteed not to throw, or be wrapped in a try/catch that contains the failure and surfaces it through BasisDebug — logged once / rate-limited, never every frame (see the existing HVRBasisBuiltInAddresses.Simulate() guard for the pattern). Expect this to be scrutinized closely in review.
• Considered jobification — I asked whether this work can be moved to a Unity Job (Burst-compiled where possible). If it can, it is. If it cannot, the reason is in Notes.
• No needless { get; set; } properties or access lockdowns — Public fields are fine; Basis is meant to be read and modified freely, so don't wall things off private/internal without a real reason. Don't wrap a field in { get; set; } when the accessors do nothing — property accessors have a real performance cost vs direct field access, and the lead maintainer prefers plain fields (or a method / setter-only property when only the setter needs logic) over a noop-getter pair. For .Instance singletons, callers reassigning Type.Instance is allowed; if that would break your code, log a warning or throw — don't block the assignment. Locking down access is not your call.
• Camera access goes through BasisLocalCameraDriver — Code that needs the local camera (transform, projection, rig data, etc.) pulls it from BasisLocalCameraDriver rather than looking one up itself. Don't roll a separate camera discovery path.
• Logging uses BasisDebug — All new logging calls go through BasisDebug.Log / BasisDebug.LogWarning / BasisDebug.LogError (with an appropriate LogTag) instead of UnityEngine.Debug.Log / Debug.LogWarning / Debug.LogError. BasisDebug routes through Basis's tagged, color-coded logger and respects the project-wide LoggingDisabled toggle so logging can be killed at runtime; bare Debug.Log calls bypass that and will be denied.
• No scene-wide discovery for dependencies — New code is architected so it does not need FindObjectOfType / FindObjectsOfType / GameObject.Find / FindGameObjectsWithTag to locate what it depends on. References are wired in — registered through an existing manager/driver, injected at init, or passed in by the caller — rather than discovered by scanning the scene at runtime. If a scene scan is genuinely unavoidable, justify it under Notes.
• No allocations in hot paths — Per-frame code (Update / LateUpdate / FixedUpdate, simulation loops, jobs, anything called once per frame or more) does not allocate. No new on reference types, no LINQ, no string concatenation/interpolation, no boxing, no foreach over interface-typed collections. Allocate once at init and reuse the buffer.
• No debugging in hot paths — No log calls of any kind on per-frame paths, including BasisDebug. Hot-path logging floods the console and incurs cost on every frame regardless of whether the message is filtered out downstream. If a hot-path log is needed while iterating, gate it behind #if UNITY_EDITOR and remove (or leave gated) before merge.
• Hot-path collection access is optimized — Cache .Count (lists) / .Length (arrays) into a local int before the loop instead of re-reading the property each iteration. Prefer T[] (with a separate length int when the array is over-sized) over List<T> where the data is hot — Unity's mono BCL doesn't expose CollectionsMarshal.AsSpan(List<T>), so a list can't be fed into Span<T> / unsafe paths cleanly. Where the perf justifies it, drop into Span<T> / ref locals / Unsafe.As / unsafe pointer code to skip bounds checks and copies, and call out the invariants you're relying on under Notes so reviewers can sanity-check them.

Testing details

Tick the platforms you actually tested on. Leave the rest unticked — these are informational and do not block merge.

• Windows
• Linux
• Android
• iOS
• macOS

Input / control mode coverage:

• Tested in VR (note headset under Notes)
• Tested in desktop / non-VR mode
• Tested with phone controls (mobile touch input)
• N/A — change does not touch player/XR/input code

Where applicable, confirm these flows still work after your changes:

• Hot-switching (desktop ↔ VR mode swap at runtime)
• Avatar swapping
• Server swapping (joining / leaving / changing servers)
• N/A — change does not touch any of the above

Notes

What "Tested" covers (Windows): In the editor — direct stream playback (RTSPT / fMP4 / TS / HLS) unchanged; a split avc1+mp4a VOD source playing in sync and paced (no fast-forward); auto-detect classifying a seekable HTTP body as on-demand and a live HLS stream as live; the StreamUrl field resolving real YouTube and Twitch links (and direct URLs bypassing the resolver); and the clear "resolver package needed" message when no resolver is installed. In a built player — live Twitch HLS playing smoothly at 1080p60 under the per-AU pacing, and the synced page URL resolving independently across two networked clients.

Most checklist rows are N/A for this change — it's load-path / native decode / integration code, with no per-frame gameplay work:

• No transform access, camera access, Addressables loads, or scene-wide discovery anywhere in the change.
• No new per-frame work, so nothing is added to BasisEventDriver and there are no hot paths — the "hot path" allocation/logging/collection rows don't apply. (The one cold-path delimiter array in IsDirectlyPlayable is hoisted to a static readonly field regardless.)
• No new GetComponent; BasisMediaPlayerStreaming uses the existing TryGetComponent.
• Jobification N/A — the work is socket I/O plus OS hardware decode on native threads, not parallelisable compute.
• Public fields (Delivery, BasisMediaUrlRouter.Resolver) are plain fields per house style; the resolver delegate is freely reassignable.

Prerequisite: the yt-dlp resolver only activates once the com.yewnyx.ytdlp plugin is present (separate PR). Until then this change is inert on that path and direct-URL playback is unaffected.

Draft status is about the dlp-native embedding decision above, not the player work: split-stream, VOD pacing, live-HLS pacing, page-URL resolution and the multiplayer page-URL sync are all validated in the editor and in a built player (incl. live Twitch and two-client sync). The integration stays inert until the plugin lands.