feat: slide the media timeline over the publisher's cache

[kixelated]

Summary

The media timeline was an append-only moq_json::stream, so it listed every segment ever published, including long-evicted ones. An HLS playlist built from it advertised segments that could no longer be fetched, and moq-hls papered over the misses with EXT-X-DISCONTINUITY.

This makes the timeline a sliding window over what the publisher can still serve, retracting a record once the groups behind it reach their track's latency_max.

Root cause

Two distinct bugs, and the second is the interesting one.

The track could not shrink. A moq_json::stream's whole log rides one group that is never rolled, and a consumer always starts at frame 0. Once moq-net evicted the earliest frames the track was bricked for late joiners (Error::Lagged), and with compression the retained suffix is undecodable anyway since its DEFLATE window depends on the evicted prefix. There was no shape in which a record could be dropped, so moq_json::window is a new mode: each group is self-contained (snapshot at frame 0, then {"append":V} / {"trim":N} operations) and rolls once the records trimmed since the snapshot outnumber those still in the window. Total bytes stay at roughly 2x an append-only log, old groups become disposable, and a late joiner reads only the newest.

latency_max was a knife edge. evict_expired dropped a group the instant its age exceeded latency_max, which is the same instant anything reasoning about the window would conclude the group was gone. A consumer that decides to fetch at that edge has its FETCH arrive a round trip later, by which time the media is already evicted. This is not fixable by retracting on eviction: an observed eviction is by definition after the fact, so the retraction is emitted when the content is already gone and then takes another propagation delay to arrive. The timeline has to lead the cache, which only a predictive trigger can do.

So latency_max becomes a floor rather than a deadline: age eviction now fires at latency_max + EVICT_GRACE (1s), and the timeline trims at latency_max exactly. The user-facing knob keeps its meaning (30s configured is 30s of timeline and 30s guaranteed fetchable) and the grace is invisible slack. It fixes the race for every reader of the window edge, not just the timeline.

Design notes for reviewers

• timeline::Producer::track takes the track::Producer, not its name. The retention is read from the same place the track declares it, so there is nothing for a caller to duplicate and get wrong. This is deliberate: an earlier attempt at a JS retention timer (bafc419a4, on the feat: trim the media timeline as segments leave the publisher's cache #2631 branch) was reverted precisely because it guessed DEFAULT_LATENCY_MAX_MS and silently mismatched any track published with a different latencyMax. That is an argument against a guessed constant, not against time-based trimming.
• Heterogeneous retention needs no rule. A record's deadline is the earliest reported + latency_max across the groups it names. A segment is fetched whole, so the first group to leave its cache breaks it as thoroughly as any other; a broadcast whose audio keeps 5s and video keeps 60s falls out of that without a special case.
• No timer on either side. The trim sweeps as segments are recorded, which is also when moq-net's commit_group runs evict_expired. The two stay in phase, and a publisher that stalls stops trimming and stops evicting together.
• The clock is web_async::time::Instant, which is what moq-net's cache pool already uses. std::time::Instant would drift from it under wasm and under a paused test clock. It is wall-clock rather than pts on purpose: eviction is wall-clock, so a file import racing pts ahead of real time must not trim content the cache is still holding in full.
• Fanout::trim leaves the wall-clock anchor alone, even when a retraction empties the replay history. The anchor maps the pts axis onto wall clock and a retraction says nothing about that axis; re-estimating would jump EXT-X-PROGRAM-DATE-TIME and DASH availabilityStartTime over content that never moved, and would do it exactly when the publisher is shedding media, which is when a fresh "this segment just ended" estimate is least true.

Relationship to #2631

Same diagnosis of the append-only problem and the same moq_json::window format, which is lifted from that branch unchanged. The trigger is what differs: #2631 retracts by observing eviction, via a new group::Demand primitive in moq-net and Group.Consumer.gone / Producer.evict / Track.Subscriber.tryNextGroup in js/net. That is exact but structurally too late, as above. Predicting from the declared window is both smaller and correctly ordered, so this drops group::Demand and the gone/evict surface entirely. tryNextGroup stays, since the window primitive itself needs it to reach the newest buffered group.

The tradeoff taken knowingly: the prediction is optimistic where observation was exact. A group dropped early by the cache pool's byte budget is still listed until its declared window elapses. That is a bounded error on a cache hint, it self-corrects at the next group's snapshot, and moq-hls keeps the discontinuity path as the genuine-miss fallback.

Public API changes

Added

• rs/moq-net: track::Producer::info(), the effective (post-clamp) track info.
• rs/moq-json: the window module (Producer, Consumer, Update, ProducerConfig, ConsumerConfig), plus an Error::Window variant (the enum is #[non_exhaustive]).
• rs/moq-mux: timeline::Update.
• js/json: the Window namespace, mirroring the above.
• js/net: Track.Producer.accepted (the synchronous peek at info()) and Track.Subscriber.tryNextGroup().

Changed

• rs/moq-mux: timeline::Consumer::{next, poll_next} yield Update<E> rather than Entry<E>, since an entry alone cannot express a retraction. Entry is unchanged and rides inside Update::Append. timeline::Producer::track and catalog::Producer::enroll take &moq_net::track::Producer rather than &str.
• js/hang: Container.Timeline.Producer.track takes the Moq.Track.Producer, matching.

Behavior

• moq_net::track::Info::latency_max is now the age a group is guaranteed to survive; eviction fires a short unspecified grace later. Mirrored in js/net's Track.Producer cache prune.

Wire format (breaking). The timeline track changes from an append-only stream to a sliding window. Nothing negotiates this: a new consumer against an old publisher errors rather than mis-parsing. That is acceptable here specifically because the catalog's timeline field is still being specified, so breaking it now costs nothing and breaking it later would cost a version.

Cross-Package Sync

• rs/hang catalog/container -> js/hang, drafts/: done. draft-lcurley-moq-hang's Timeline section specifies the window framing (snapshot, append/trim, group rolling) and a new Availability section covering retraction.
• rs/moq-net wire/API -> js/net, drafts/: done. draft-lcurley-moq-lite's Expiration section now states Publisher Max Latency as a floor with the round-trip margin, with a changelog bullet under moq-lite-06. The hang draft cites it rather than restating.
• doc/: no change. The timeline is not documented on the site and no CLI surface moved.

Test plan

• just check passes, including cargo doc with -D warnings.
• just drafts check (kramdown-rfc) passes on all 8 drafts. The HT characters warning on the hang draft is pre-existing: its schema blocks use tabs.
• New Rust coverage: a group surviving its own latency_max deadline and dying past the grace; a record retracted once its groups expire; a live consumer seeing the retraction with the oldest surviving segment named; the shortest retention bounding a multi-track record; and the load-bearing one, the_retraction_leads_the_eviction, which asserts the record is gone while a real FETCH on its group still succeeds. That test was mutation-checked by zeroing EVICT_GRACE, and fails without it. Plus moq-hls trims dropping segments the duration window would keep, never rewinding EXT-X-MEDIA-SEQUENCE, and emptying a window when trimmed past.
• New JS coverage: the 8 window tests from feat: trim the media timeline as segments leave the publisher's cache #2631 (two decode hand-written wire frames, so the decoder is checked against the byte shape rather than only against its own producer), plus retraction and shortest-retention tests mirroring the Rust ones.

Branch targeting

Targets dev: timeline::Consumer::next, timeline::Producer::track, and catalog::Producer::enroll all change signature, as does js/hang's Timeline.Producer.track.

(Written by Opus 5)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 98e3706dfb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Trim through expired records with nonmonotonic deadlines

When a short-retention track is enrolled after the timeline already contains records from a long-retention track, a newer record can have an earlier deadline than the record at the front of this deque. This prefix-only take_while then stops at the live older record and leaves the newer expired record advertised even after its media has been evicted. The mirrored JS loop has the same problem. Since the wire window can only trim from the head, sweep through the furthest expired record, even if that conservatively removes older records that remain fetchable. (Written by GPT-5.6 Sol)

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve the media sequence when trimming the last row

If a timeline trim removes every currently listed row without immediately appending another segment, this loop leaves rows empty and State::window falls back to sequence 0. The rendered playlist consequently rewinds EXT-X-MEDIA-SEQUENCE from its previous value to 0, despite the nearby invariant that this sequence must never move backwards. Retain the trim boundary as a sequence watermark so an empty live window still reports the next unavailable segment number. (Written by GPT-5.6 Sol)

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Document latencyMax as a guaranteed floor

This changes JS eviction to occur at latencyMax + EVICT_GRACE_MS, but the exported Info.latencyMax documentation still calls the value the maximum age before eviction. Consumers reading the generated API docs can therefore incorrectly treat it as a hard upper bound when sizing caches or reasoning about retention, while the Rust API and updated draft describe it as a guaranteed floor with unspecified extra retention. Update the public member documentation with the new semantics. (Written by GPT-5.6 Sol)

AGENTS.md reference: AGENTS.md:L86-L88

Useful? React with 👍 / 👎.