feat(net): route by cumulative cost on lite-06 announcements

[kixelated]

Routing today ranks candidate routes by hop count, which is blind to what a link costs and to which candidate already carries the media. Two relays in one datacenter each pull the same broadcast over a metered backbone, because the via-sibling route is +1 hop and shortest-hop routing never picks it. The same blindness applies to expensive work: a transcoder pool has no way to say "one of us is already transcoding this, route to it" versus "any of us could, pick one".

The cost model

Route.cost becomes the cumulative cost of the transfers (and work) a subscription via that route would newly cause, and rides lite-06 announcements as a single varint:

• The original publisher seeds its production cost: zero for a live publish, something large for a standby (a cold transcoder pool announces every broadcast it could serve at, say, 1000; the existing name+hop hash tie-break then elects the same standby on every node, spreading broadcasts across the pool).
• Each link adds its configured price as the announcement crosses it (new SETUP Cost parameter, id 0x4; the dialing side declares it and both ends charge the same amount; unpriced links cost 1, which reproduces hop counting exactly).
• The accumulation restarts at zero at any node actively carrying the broadcast: those upstream legs already exist and are not re-paid by one more subscriber. This one rule covers both relay cache dedup (a warm sibling beats a second metered fetch) and standby activation (the winning transcoder's cost drops to ~0 once it starts working and snaps back after it stops), with no application code.
• route_order keeps its exact prior shape (!announce, cost, hops.len(), hash). Pre-lite-06 peers carry no cost and rank exactly as before; equal-cost warm copies resolve by hop length, which also bounds same-datacenter chains to one hop.

"Actively carrying" is exposed as broadcast::Demand, the broadcast-level sibling of track::Demand: a watch-only weak handle from Producer::demand() with is_used() plus used()/unused() edges (and poll_* variants). Demand means subscriber interest: a subscribed spliced track on a relay front, or a pending request / consumed track on an ordinary broadcast (a publisher producing into the void is not a warm copy worth routing to). The consumer counts live on per-track channels whose flips never write the broadcast state, so Demand parks the waiter across every channel feeding the answer.

The announce loop waits on exactly the transition it cares about, watching unused while advertising zero and used while advertising cold, with no polling tick. Activation re-prices immediately; decay keeps a deliberate 5s linger (demand returning within the window cancels the restore), so viewer churn does not flap routing across the mesh.

The simultaneous-activation race

Two relays that independently pulled the same broadcast both advertise zero, each sees the other as cheaper than its own source, and re-parenting onto each other simultaneously leaves the broadcast with no upstream at all (the reflected-announce check then breaks the cycle, both revert, and the attraction repeats: a stall/resume oscillation at announce-RTT cadence).

The fix is a deterministic tie-break scoped to exactly that hazard. While a front is carrying, a strictly cheaper route whose announcing relay is itself carrying (it advertised zero from a chain of two or more hops; a single-hop chain is the original publisher, which can never adopt a route to its own broadcast) only displaces an announced incumbent if the peer's build-stable FNV key for the broadcast is below ours. Both sides compute the same pair, exactly one re-parents (seamlessly, at a group boundary via the existing splice), and the loser's restarted announcement carries the winner in its chain, so the reflected-announce check removes the residual attraction. Every other cheaper route, e.g. a forwarder path or an upstream that repriced itself down, is taken immediately. The pre-charge cost needed to recognize a warm peer is kept on the route as a crate-private advertised field.

OriginList remains the authority on loop freedom; the key only prevents the transient double-switch.

Config

Cluster peer URLs price their links with a ?cost=N query param, composing with static lists, gossip, and connect_api feeds:

[cluster]
connect = [
  "https://sibling.same-dc.example/?cost=0",
  "https://us-east.example.com/?cost=10",
]

The param is consumed locally (the value rides SETUP, never the URL); a garbage value is a startup error rather than a silent default. moq_net::Client::with_cost / moq_native::Client::with_cost are the API surface.

Cross-package sync

• js/net: mirrors the wire change (hasRouteCost, cost on ANNOUNCE_START/RESTART) with round-trip tests. The browser client does not charge links.
• drafts/draft-lcurley-moq-lite.md: Route Cost fields on ANNOUNCE_START/RESTART, the SETUP Cost parameter, the tie-break guidance, and changelog entries. Validated with just drafts check.
• doc/bin/relay/cluster.md: new Link costs section.
• Not run: just test smoke-full; the wire change is gated behind the opt-in moq-lite-06-wip version, which the smoke matrix does not negotiate.

Notes for review

• Targets dev per the wire-change rule: the feature rides moq-lite-06-wip (opt-in, still WIP), extending ANNOUNCE_START/RESTART and adding a SETUP parameter. All Rust/JS API changes are additive.
• Rebased from a main base onto dev: the one conflict was detach_source, where dev's graceful teardown gate and this PR's reselect(carrying) touch the same lines; the resolution keeps both. Everything else applied clean and the full suite passes on the dev base.
• Supersedes feat(moq-net): cost-based cache-aware routing (moq-lite-06 WIP) + route-cost I-D #2179; the base+transit split from that prototype collapsed into the single cumulative cost above (lexicographic policy-vs-distance turned out unnecessary once the cost is denominated in real units: prices sum).
• Follow-ups worth their own issues: an e2e two-relay test of the race itself, standby announce fan-out (N transcoders x M broadcasts) if it shows up in practice, and re-running reselect when a front's demand drains (today the next route event picks it up).

(written by Fable 5)

[sourcery-ai]

Sorry @kixelated, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Walkthrough

Adds per-link routing costs to MoQ-lite SETUP and ANNOUNCE messages with version-gated encoding and legacy defaults. Client and relay configuration now carry costs into subscriber route construction. Publishers track and recheck advertised costs, while broadcast activity affects outgoing cost. Route selection gains carrying-aware deterministic handover logic. Documentation and tests cover protocol compatibility, cost accumulation, saturation, URL parsing, and route updates.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: cumulative cost-based routing for lite-06 announcements.
Description check	✅ Passed	The description is directly aligned with the PR and explains the routing cost model, protocol changes, and related updates.

✨ Finishing Touches

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch claude/cdn-dynamic-routing-1b7d09

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

rs/moq-net/src/lite/publisher.rs (1)

434-446: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Optimize cost drift detection to avoid O(N) allocations.

Currently, the periodic cost recheck calls entry.consumer.route() for every watched broadcast, which acquires an RwLock read lock and clones the Vec-backed hop chain (OriginList).

Since the target cost is strictly 0 when actively carrying, and always equals the unchanging route.cost when idle (because any route change would have eagerly triggered poll_route_changed instead of reaching here), you can use the currently recorded sent.cost to safely skip fetching the route in the vast majority of cases.

⚡ Proposed fast-path optimization

 						// The advertised cost drifts without a route change when
 						// liveness flips; reuse the route path to re-send it.
 						if version.has_route_cost()
 							&& let Some(sent) = &entry.sent
 						{
+							// Fast path: avoid allocating and cloning the route's hop chain
+							// on every tick if the liveness-based cost hasn't changed.
+							let active = entry.consumer.is_active();
+							if active == (sent.cost == lite::RouteCost(0)) {
+								continue;
+							}
+
 							let route = entry.consumer.route();
 							if Self::outgoing_cost(version, &entry.consumer, &route) != sent.cost {
 								return Poll::Ready(Ok(Op::Route(suffix.clone(), Ok(route))));
 							}
 						}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rs/moq-net/src/lite/publisher.rs` around lines 434 - 446, Optimize the
cost-drift check in the periodic recheck block by using the recorded sent.cost
as a fast path: when it already represents the expected active-carrying cost of
0 or the unchanged idle route cost, skip entry.consumer.route() and its
allocation. Only fetch the route and recompute outgoing_cost when sent.cost
indicates a possible liveness-driven drift, while preserving the existing Route
response and route-change handling.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@rs/moq-net/src/lite/publisher.rs`:
- Around line 434-446: Optimize the cost-drift check in the periodic recheck
block by using the recorded sent.cost as a fast path: when it already represents
the expected active-carrying cost of 0 or the unchanged idle route cost, skip
entry.consumer.route() and its allocation. Only fetch the route and recompute
outgoing_cost when sent.cost indicates a possible liveness-driven drift, while
preserving the existing Route response and route-change handling.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 62339f02-de36-4911-aacc-e2d4261b9e89

📥 Commits

Reviewing files that changed from the base of the PR and between 3c07334 and d52e5d0.

📒 Files selected for processing (21)

• doc/bin/relay/cluster.md
• drafts/draft-lcurley-moq-lite.md
• js/net/src/lite/announce.test.ts
• js/net/src/lite/announce.ts
• js/net/src/lite/version.ts
• rs/kio/src/producer.rs
• rs/moq-native/src/client.rs
• rs/moq-net/src/client.rs
• rs/moq-net/src/lite/announce.rs
• rs/moq-net/src/lite/publisher.rs
• rs/moq-net/src/lite/session.rs
• rs/moq-net/src/lite/setup.rs
• rs/moq-net/src/lite/subscriber.rs
• rs/moq-net/src/lite/version.rs
• rs/moq-net/src/model/broadcast.rs
• rs/moq-net/src/model/origin.rs
• rs/moq-net/src/model/requests.rs
• rs/moq-net/src/model/resume.rs
• rs/moq-net/src/model/weak_cache.rs
• rs/moq-net/src/server.rs
• rs/moq-relay/src/cluster.rs

💤 Files with no reviewable changes (1)

• rs/moq-net/src/model/requests.rs

[coderabbitai]

🧹 Nitpick comments (1)

rs/moq-net/src/lite/publisher.rs (1)

398-745: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add regression coverage for the idle/linger repricing path.
This module has unrelated recv_next tests, but nothing that exercises Op::Idle, Op::Linger, demand dropping to zero, or demand returning before COST_LINGER expires.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rs/moq-net/src/lite/publisher.rs` around lines 398 - 745, Add regression
tests covering the publisher announce loop’s idle/linger behavior around
Op::Idle, Op::Linger, and COST_LINGER: verify a zero-demand transition starts
lingering without immediately repricing, demand returning before expiry cancels
the pending restore, and expiry reprices the route back to its cold cost. Reuse
the module’s existing publisher/recv_next test setup and assert the emitted
route updates and costs.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@rs/moq-net/src/lite/publisher.rs`:
- Around line 398-745: Add regression tests covering the publisher announce
loop’s idle/linger behavior around Op::Idle, Op::Linger, and COST_LINGER: verify
a zero-demand transition starts lingering without immediately repricing, demand
returning before expiry cancels the pending restore, and expiry reprices the
route back to its cold cost. Reuse the module’s existing publisher/recv_next
test setup and assert the emitted route updates and costs.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b87dcdcb-b2f4-4950-b62a-e6a38d9d6809

📥 Commits

Reviewing files that changed from the base of the PR and between d52e5d0 and ccd90e9.

📒 Files selected for processing (7)

• rs/kio/src/weak.rs
• rs/moq-net/src/lite/publisher.rs
• rs/moq-net/src/model/broadcast.rs
• rs/moq-net/src/model/origin.rs
• rs/moq-net/src/model/resume.rs
• rs/moq-net/src/model/track.rs
• rs/moq-net/src/model/weak_cache.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• rs/moq-net/src/model/resume.rs
• rs/moq-net/src/model/origin.rs

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The announce loop in lite::Publisher has grown quite complex with the added COST_LINGER and demand polling logic; consider extracting the demand/linger handling into a helper or two to keep the main loop’s control flow easier to follow and reason about.
• In BroadcastState::register_demand and Demand::poll_demand, demand registration currently walks all tracks on every poll; if broadcasts are expected to have many tracks, it may be worth noting or revisiting this for potential per-broadcast or per-track coalescing to avoid O(n) work on each wake.
• The handover key logic in FrontState::handover_allowed recalculates the FNV hash over self.path and the last hop on each call; if this ends up on a hot path, caching the self-origin key per broadcast (and reusing it in both tests and runtime code) could reduce repeated hashing.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The announce loop in `lite::Publisher` has grown quite complex with the added COST_LINGER and demand polling logic; consider extracting the demand/linger handling into a helper or two to keep the main loop’s control flow easier to follow and reason about.
- In `BroadcastState::register_demand` and `Demand::poll_demand`, demand registration currently walks all tracks on every poll; if broadcasts are expected to have many tracks, it may be worth noting or revisiting this for potential per-broadcast or per-track coalescing to avoid O(n) work on each wake.
- The handover key logic in `FrontState::handover_allowed` recalculates the FNV hash over `self.path` and the last hop on each call; if this ends up on a hot path, caching the self-origin key per broadcast (and reusing it in both tests and runtime code) could reduce repeated hashing.

## Individual Comments

### Comment 1
<location path="rs/moq-net/src/lite/publisher.rs" line_range="474" />
<code_context>
+					match entry.idle_at {
</code_context>
<issue_to_address>
**issue (bug_risk):** The cost restore is delayed for roughly twice COST_LINGER due to how `idle_at` and `fired` are combined.

Today `idle_at` is set to `Instant::now()` when demand drains, the loop computes `deadline = idle_at + COST_LINGER`, and `linger` sleeps until that deadline. When the sleep completes, `fired` is set to `Instant::now()` (≈ `deadline`). The restore guard is `Some(at) if fired.is_some_and(|now| now >= at + COST_LINGER)`, which means we actually wait until `now >= idle_at + 2 * COST_LINGER`. If the intended delay is a single `COST_LINGER`, the comparison should be against `at` (the deadline) instead of `at + COST_LINGER`, or `idle_at` should be initialized to the deadline so the extra `+ COST_LINGER` isn’t applied twice.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

issue (bug_risk): The cost restore is delayed for roughly twice COST_LINGER due to how idle_at and fired are combined.

Today idle_at is set to Instant::now() when demand drains, the loop computes deadline = idle_at + COST_LINGER, and linger sleeps until that deadline. When the sleep completes, fired is set to Instant::now() (≈ deadline). The restore guard is Some(at) if fired.is_some_and(|now| now >= at + COST_LINGER), which means we actually wait until now >= idle_at + 2 * COST_LINGER. If the intended delay is a single COST_LINGER, the comparison should be against at (the deadline) instead of at + COST_LINGER, or idle_at should be initialized to the deadline so the extra + COST_LINGER isn’t applied twice.