worldtree: multiplex per-zone source reads over a bounded connection …#40
Closed
sonkehahn-shopify wants to merge 1 commit into
Closed
Conversation
…pool The socket-served source path (Mode B, `tec --sha`) opened one worldtree control-socket connection per clean zone in `getZoneStorePath`. A broad evaluation — the Tartarus planning bundle over a large changeset, run with parallel-eval + `eval-cores 0` — fans out to hundreds of zones at once, so it opened >512 connections in a burst and tripped worldtreed's per-listener connection cap (`ServeConfig::max_connections`, 512). Past the cap the daemon accepts-then-drops, which the C++ client sees as `worldtree: read(): Connection reset by peer`, aborting the eval. A hard cap on concurrent connections cannot fix this: the planner needs more zones mounted simultaneously than any safe per-listener cap allows. The daemon already supports many ephemeral RO sessions per connection (its owned-ephemeral set is per-connection and is drained on disconnect), so instead multiplex the per-zone scoped sessions over a bounded pool of connections. - `WorldtreeConn` gains a ws-parametric API (`openScopedSession`, `closeSessionOn`, `readTreeOn`, `readBlobOn`) so one physical connection can host many independent per-zone sessions; the existing single-session methods (used by the control connection) are unchanged. - New `WorldtreeZoneSession` handle owns one zone's scoped session on a shared carrier connection and releases it (`close_ro`) when the zone's `WorldtreeSourceAccessor` — its sole owner — drops. - `EvalState::acquireWorldtreeZoneSession` draws a carrier round-robin from a lazily-grown pool bounded by the new `tectonix-worldtree-max-connections` setting (default 64, well under the daemon's 512 cap), and opens the zone's scoped session on it. `getZoneStorePath` uses it instead of one connection per zone. Per-zone confinement is unchanged: each zone still gets its own `scoped` session (the daemon enforces the cone per session), just hosted on a shared connection. Reads that land on the same carrier serialize on it; cross-zone parallelism now comes from the pool holding several carriers rather than one connection per zone. Co-authored-by: Sonke Hahn <sonke.hahn@shopify.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
superseded by #41, closing.
Motivation
worldtreedhas a connection limit (512) and for bigger evalstecnixruns over that limit. This PR multiplexes connections through a connection pool of size 64.Changes
The socket-served source path (Mode B,
tec --sha) opened a freshWorldtreeConnper clean zone inEvalState::getZoneStorePath, each hosting onescopedephemeral RO session. A hard cap on concurrent connections can't fix this — the planner needs more zones mounted at once than any safe per-listener cap allows. But the daemon already supports many ephemeral RO sessions per connection: itsowned_ephemeralset is per-connection and is drained on disconnect (wt-controlplane/src/rpc/server.rs). So this multiplexes the per-zone sessions over a bounded pool:WorldtreeConngains a ws-parametric API (openScopedSession,closeSessionOn,readTreeOn,readBlobOn) so one physical connection can host many independent per-zone sessions. The existing single-session methods (used by the singleton control connection) are untouched.WorldtreeZoneSessionhandle owns one zone's scoped session on a shared carrier connection and releases it (close_ro) when the zone'sWorldtreeSourceAccessor— its sole owner — drops.EvalState::acquireWorldtreeZoneSessiondraws a carrier round-robin from a lazily-grown pool bounded by the newtectonix-worldtree-max-connectionssetting (default 64, well under 512), and opens the zone's scoped session on it.getZoneStorePathuses it instead of one connection per zone.Design notes / tradeoffs:
scopedsession; the daemon enforces the cone per session, independent of which connection hosts it.acquireWorldtreeZoneSessionnever calls a carrier's client under the pool mutex — theopen_rohappens after the pool lock is released — so there is no pool-mutex/carrier-mutex inversion.connectWorldtree(), which throws on an unreachable daemon (no libgit2 fallback).