From fc1597d1bef0fc6bc03848635f6ad2eee6fbee95 Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 08:51:26 -0700 Subject: [PATCH 01/11] docs: design for absorbing offchain and solana into this repo --- .../2026-08-27-monorepo-migration-design.md | 333 ++++++++++++++++++ 1 file changed, 333 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-27-monorepo-migration-design.md diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md new file mode 100644 index 0000000000..18e2608751 --- /dev/null +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -0,0 +1,333 @@ +# Absorbing doublezero-offchain and doublezero-solana into this repo + +Target repo: malbeclabs/doublezero (this repo). Branch each step off `origin/main`. +Source repos: malbeclabs/doublezero-offchain, malbeclabs/doublezero-solana. +Prerequisite, done 2026-08-26: both source repos moved from the `doublezerofoundation` +org to `malbeclabs`, and every reference was repointed. The five pull requests that did it: +doublezero-offchain#412, doublezero-solana#125, doublezero-shreds#686, doublezero#4236, +docs#202. + +## Why + +Three problems, all of which come from the code living in three repos. + +1. **A change that spans the three repos takes three pull requests in a fixed order.** + It also takes a pin bump in each consumer. On 2026-08-26 this failed in a way that + was hard to read: `doublezero-shreds` pointed `doublezero-program-tools` at the new + org while the offchain revision it pinned still pointed at the old one. Cargo makes + a git dependency's URL part of the crate identity, so it built two copies of the + crate. The oracle then reported 194 errors of the form "no method named X", on types + that plainly had X. Nothing in the error text pointed at the URL. +2. **Three release setups.** Offchain runs five goreleaser configs and its own Cloudsmith + push. Solana runs none. This repo runs one per component already. Three repos means + three sets of Actions secrets, three rulesets, three CODEOWNERS files. +3. **Version skew has no single source of truth.** Ten git dependencies carry pins today. + Nothing checks that they agree. + +## Scope + +In scope: move `doublezero-offchain` and `doublezero-solana` into this repo, merge the +Cargo workspaces, and merge the release and CI setup. + +Out of scope, decided deliberately: + +- **`doublezero-shreds` stays a separate repo.** It is private and this repo is public. + It keeps git dependencies and coordinated pin bumps. It does get a large improvement: + three upstream pins collapse to one. See "What this does to shreds". +- **`network-shapley-rs` stays in the `doublezerofoundation` org.** It is public and it + works. Moving it is a separate job for another day. +- **Consolidating the two sentinel crates.** This spec renames a binary to stop a + collision. Whether the two crates should become one is a later question. +- **Any other malbeclabs repo.** The pattern here can be reused for a later wave. + +## The state we are starting from + +| | doublezero | offchain | solana | +| --- | --- | --- | --- | +| Rust crates | 22 | 16 | 6 | +| Rust lines | 199k | 52k | 21k | +| Go lines | 454k | 0 | 0 | +| Other | 2 Go modules | Elixir app + Rust NIF | none | +| Workflows | 49 | 8 | 3 | +| Toolchain | 1.97.1 | 1.92.0 | 1.91 | +| `solana-sdk` | 3.0.0 | 3.0.0 | 3.0.0 | +| `borsh` | 1.7.0 | 1.7.0 | 1.6.0 | + +The Solana crates already agree across all three repos. Only borsh and the toolchain +differ, and both move upward without a break. The usual reason a monorepo merge stalls +is absent here. + +Two facts about this repo make the merge cheaper than it looks: + +- `smartcontract/programs/rust-toolchain.toml` already pins `channel = "1.91"`, the same + channel doublezero-solana uses. Per-directory toolchain pinning is established practice + here. +- The root `Cargo.toml` already carries an `exclude` list for nested workspaces (the four + `generate-fixtures` directories). Nested workspaces are established practice too. + +## Decisions + +### D1. Two new top-level directories. Nothing existing moves. + +``` +doublezero/ +├── crates/ 22 crates, unchanged +├── smartcontract/programs/ DoubleZero Ledger programs, unchanged +├── client/ sdk/ e2e/ ... unchanged +│ +├── solana/ from doublezero-solana +│ ├── programs/passport +│ ├── programs/revenue-distribution +│ ├── crates/program-tools +│ └── mock/{swap-sol-2z,rewards-integration} +│ +└── offchain/ from doublezero-offchain + ├── crates/ 14 crates + └── scheduler/ Elixir app and its Rust NIF +``` + +This matches how the repo is already laid out: `client/`, `sdk/`, `smartcontract/`, +`controlplane/`, `e2e/` are all top-level trees named for what they hold. + +Rejected: flattening offchain's crates into `crates/` and solana's programs into +`smartcontract/programs/`. Three reasons. + +1. **The two program sets run on different chains.** This repo's own CLAUDE.md already + documents the split: the serviceability, telemetry, geolocation and record programs + deploy to the **DoubleZero Ledger** (`ledger_rpc_url`), while **Solana L1** + (`solana_l1_rpc_url`) is a separate network carrying the 2Z token. `passport` and + `revenue-distribution` deploy to Solana L1. The offchain scheduler's config carries + `DZ_LEDGER_RPC` and `SOLANA_RPC` as separate endpoints for the same reason. Putting + both sets in `smartcontract/programs/` would hide a distinction the codebase already + treats as load-bearing, and one the CLAUDE.md warns fails silently when confused: a + lookup against the wrong cluster does not error, the account simply is not there. +2. It puts 38 crates in one flat directory. +3. It forces the sentinel directory rename on day one. Keeping the trees separate makes + that collision disappear (`offchain/crates/sentinel` next to `crates/sentinel`). + +### D2. One workspace, with the two Solana programs held back until measured. + +After the workspace merge, the root workspace gains the 14 offchain crates, +`scheduler/native/scheduler_doublezero`, and `solana/crates/program-tools`. + +`passport` and `revenue-distribution` stay in `exclude` until we measure whether folding +them in changes the bytes they compile to. `doublezero-solana` builds them in Docker with +`cargo fetch --locked` and checks them against `programs/sha256sums_*.txt`. A shared +lockfile may resolve some dependency differently, which would change the artifact. + +Step 0 below settles this by measurement, not by argument. If the bytes do not change, +they join the root workspace. If they change, they stay excluded with their own lockfile, +and they keep needing a pin bump like today. + +The two `mock/` programs stay excluded. Offchain already excludes its own mock program, +so this follows the existing convention. + +### D3. Toolchain: root stays 1.97.1, programs pin 1.91. + +Add `solana/programs/rust-toolchain.toml` pinning 1.91, mirroring +`smartcontract/programs/rust-toolchain.toml`. + +Offchain's repo-wide 1.92.0 pin goes away and its crates build on 1.97.1. Expect new +clippy findings. That work belongs to step 2 and is not a surprise to discover later. + +borsh unifies on 1.7.0. + +### D4. Rename this repo's sentinel binary. + +Both repos have a `crates/sentinel`, and both declare `[[bin]] name = "doublezero-sentinel"`. +Two members of one workspace writing the same file into `target/release/` is a real +collision. + +| | this repo | offchain | +| --- | --- | --- | +| package | `doublezero-sentinel` | `doublezero-ledger-sentinel` | +| binary | `doublezero-sentinel` | `doublezero-sentinel` | +| released? | no | yes, as a Cloudsmith deb | +| used by | `e2e.yml` only | deployed | + +Rename this repo's binary to `dz-e2e-sentinel`. It has no external contract: no goreleaser +config, gated behind `required-features = ["server"]`, and built only into the +`ghcr.io/malbeclabs/dz-e2e/sentinel` image that the e2e shards consume. Offchain's keeps +its name because `doublezero-sentinel` is a deployed package name. + +The rename touches two files: `crates/sentinel/Cargo.toml` and +`e2e/docker/sentinel/Dockerfile`. + +### D5. Keep the imported tags, with a provenance prefix. + +`git filter-repo` carries tags by default. Import them with `--tag-rename` so offchain's +`contributor-rewards/v0.6.1` becomes `offchain/contributor-rewards/v0.6.1`, and solana's +`revenue-distribution/v0.3.7` becomes `solana/revenue-distribution/v0.3.7`. + +This keeps release history reachable and says where each tag came from. It also leaves the +unprefixed namespace free for releases built from this repo, so the existing workflow +triggers (`contributor-rewards/v*.*.*`) need no change and cannot match history by +accident. `git describe --tags` still finds the old tags. + +**This makes retiring the orphan `sentinel/v0.6.1` tag a prerequisite, not a tidy-up.** +That tag sits in this repo with no sentinel release behind it. Offchain's sentinel line +reaches `v0.2.6`, so its next release is `sentinel/v0.2.7`, below the orphan. Leave the +orphan in place and the sentinel version line reads backwards, and a later release at +v0.6.1 fails on an existing tag. + +## Sequencing + +One throwaway measurement, then four pull requests in this repo, then one follow-on +pull request in `doublezero-shreds`. + +### Step 0. Measure the program bytes. Throwaway. + +Do the whole merge locally. Add `passport` and `revenue-distribution` as root workspace +members. Rebuild through the existing path: + +```sh +make build-artifacts NETWORK=mainnet-beta +shasum -a 256 -c programs/sha256sums_mainnet_beta.txt +``` + +Repeat for `NETWORK=development`. Output is one fact that decides D2. Discard the work +either way. + +### Step 1. Import the code and the history. + +Run `git filter-repo --path-rename` on each source repo to relocate its tree, with +`--tag-rename` per D5. Add each as a remote and merge with +`--allow-unrelated-histories`. Fetch `main` only. Offchain alone has 599 refs and none of +the others are wanted. + +Add `offchain/` and `solana/` to the root `exclude` list wholesale. Both trees keep their +own `Cargo.toml` and `Cargo.lock` and build as nested workspaces, exactly as they do +today. + +Do not bring their `.github/workflows/` across yet. Eleven workflows firing on a repo +that does not expect them is noise. + +Gate: every existing job green, with no existing job definition touched. `git diff` +between each imported tree and its filtered source is empty. The diff is large and needs +no judgement to review. + +### Step 2. Merge the workspaces. + +This is the step that can hurt, and it hurts alone. + +- Root workspace absorbs the crates per D2. +- Delete the four nested `Cargo.toml` and `Cargo.lock` files. +- **Ten git dependencies become path dependencies.** Offchain's seven + `malbeclabs/doublezero` dependencies and its three `doublezero-solana` dependencies are + all same-repo now. The pinning that failed on 2026-08-26 stops existing. +- Apply D3 (toolchain, borsh) and D4 (sentinel rename). + +Gate: `cargo check --workspace`, `cargo clippy --all-targets`, the full test suite, e2e, +and the program checksum check. Also confirm the lockfile holds no crate twice: +`grep '^name = ' Cargo.lock | sort | uniq -d` should print nothing that was not already +duplicated before the merge. + +### Step 3. Merge release and CI. + +Fold 11 workflows into this repo's 49. This repo already runs one release workflow per +component, so offchain's five fit the existing shape. + +- Dedupe the overlaps: two `local-validator.yml`, three Rust CI configs (`rust.yml` twice + plus offchain's `ci.yml`), two `changelog-reminder.yml`. +- Add `erlef/setup-beam` for the Elixir scheduler. +- Copy offchain's three release secrets onto this repo: `CLOUDSMITH_TOKEN`, + `GORELEASER_KEY`, `SLACK_BOTS_WEBHOOK`. +- **Change `release.github.name` in the five goreleaser configs from `doublezero-offchain` + to `doublezero`.** The `owner` field was corrected during the org move. The `name` field + becomes wrong the moment the code lives here. +- Retire the orphan `sentinel/v0.6.1` tag per D5. + +Gate: push a throwaway release candidate tag on the smallest component. Confirm the +release lands under malbeclabs and the package reaches Cloudsmith. Config review proves +the target is named correctly. It does not prove the token can write there. + +### Step 4. Decommission the source repos. + +Archive both read-only with a README pointing here. Do not delete them. Their transfer +redirects, 11 forks, and old release download URLs all still resolve through them. + +Archiving also keeps shreds building on its current pins. An archived repo still serves +reads, so `malbeclabs/doublezero-solana` at tag `revenue-distribution/v0.3.7` keeps +resolving. The tag renaming in D5 applies to the copies imported here, not to the tags in +the archived repos. That decouples step 5 from step 4: shreds can be repointed when it +suits, not the same day. + +### Step 5. Repoint shreds. Separate repo, separate risk. + +`doublezero-shreds` consumes both source repos over git. Once they are archived it must +point at `malbeclabs/doublezero`. See below for why this needs care. + +## What this does to shreds + +Today shreds pins three repos at three refs: + +``` +shreds → malbeclabs/doublezero rev 8bb7900e 2 crates +shreds → malbeclabs/doublezero-offchain rev 0c84bdef 1 crate +shreds → malbeclabs/doublezero-solana tag revenue-distribution/v0.3.7 2 crates +``` + +Shreds declares `doublezero_sdk` at rev `8bb7900e` and also inherits it at tag +`client/v0.31.0` through offchain. Two URLs for one crate, so two copies are built. It +compiles today only because those types never meet. + +After this migration, everything shreds needs lives in `malbeclabs/doublezero`: the sdk, +serviceability, solana-client-tools, program-tools and revenue-distribution. One URL, one +ref, one copy of each crate. The duplicate cannot survive, because no second source +exists for it to come from. + +That holds on one condition. **Shreds must repoint every `malbeclabs/doublezero` +dependency to a single ref in one commit.** A half-repoint reproduces the 2026-08-26 +failure exactly. + +Give the shreds change a real check rather than "it builds": + +```sh +grep -oE 'git\+https://github.com/malbeclabs/doublezero[^"]*' Cargo.lock | sort -u # expect 1 line +grep -c 'name = "doublezero_sdk"' Cargo.lock # expect 1 +``` + +Keep that as a CI guard in shreds afterwards. It fails the moment someone reintroduces a +second ref, which is far cheaper than diagnosing 194 errors again. + +Shreds still needs coordinated pin bumps to pick up changes from this repo. One pin +instead of three is a real gain. Full atomic change only reaches code inside this repo. +That is the price of the public and private split, and we are paying it on purpose. + +## Rollback + +Steps 2, 3 and 4 revert cleanly. + +Step 2 reverts cleanly **because step 1 deliberately touches no build file.** Step 2 only +edits and deletes manifests, so reverting it restores the four nested `Cargo.toml` and +`Cargo.lock` files and leaves a repo that builds as it did after step 1. Step 3 is +workflows. Unarchiving a repo is a click. + +Step 1 is the one-way door. `git revert -m 1` removes the files, and the grafted history +stays in the graph unless `main` is rewritten. Treat merging step 1 as the commitment +point, not step 2. + +## Risks + +**The 68 branches.** `doublezero-shreds` has 68 remote branches carrying +`programs/Cargo.toml` with old org URLs. They are not on `main`. Each one merged after +the org move reintroduces an old URL. Harmless while the transfer redirects hold, and a +build failure once they do not. Rebasing them is shreds work, not this migration's, but it +interacts with step 5. + +**Clippy churn on the toolchain bump.** Offchain moves from 1.92.0 to 1.97.1 in step 2. +Volume is unknown until tried. Step 0 can measure it at the same time as the program +bytes, for free. + +**One lockfile, one resolution.** Today three lockfiles disagreeing is a visible signal. +After step 2 one lockfile resolves silently. The Solana crates agree today only because +of an exact pin (`solana-sdk = "=3.0"` in all three). Keep that pin. + +**The fixture generators float.** `sdk/revdist/testdata/fixtures/generate-fixtures` +declares its two `doublezero-solana` dependencies with no tag and no rev. Only the +lockfile pins them, so the next `cargo update` there moves them to whatever `main` holds. +Pin them during step 2. + +**`doublezero-solana` uses version ranges** (`>=2,<=3`) on its program-side crates. That +earns its keep for a library other repos consume. Inside one workspace it only produces +resolution nobody asked for. Convert to exact pins during step 2. From 7bc5b84c0519a0d1a9fca338e99c38eeeeb56c95 Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 08:54:20 -0700 Subject: [PATCH 02/11] docs: import tags unprefixed, retiring the one colliding orphan --- .../2026-08-27-monorepo-migration-design.md | 53 ++++++++++++------- 1 file changed, 34 insertions(+), 19 deletions(-) diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index 18e2608751..dd01c2c176 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -153,22 +153,38 @@ its name because `doublezero-sentinel` is a deployed package name. The rename touches two files: `crates/sentinel/Cargo.toml` and `e2e/docker/sentinel/Dockerfile`. -### D5. Keep the imported tags, with a provenance prefix. +### D5. Keep the imported tags, unprefixed. Retire one orphan first. -`git filter-repo` carries tags by default. Import them with `--tag-rename` so offchain's -`contributor-rewards/v0.6.1` becomes `offchain/contributor-rewards/v0.6.1`, and solana's -`revenue-distribution/v0.3.7` becomes `solana/revenue-distribution/v0.3.7`. +`git filter-repo` carries tags by default. Keep them: they make release history reachable +from this repo, and an unprefixed tag continues each component's version line, so a later +release of `contributor-rewards` picks up from `v0.6.1` rather than restarting. -This keeps release history reachable and says where each tag came from. It also leaves the -unprefixed namespace free for releases built from this repo, so the existing workflow -triggers (`contributor-rewards/v*.*.*`) need no change and cannot match history by -accident. `git describe --tags` still finds the old tags. +Unprefixed is safe because there is almost no overlap. Comparing every tag prefix across +the three repos, 33 in this repo, 14 in offchain, 2 in solana, gives exactly one collision: -**This makes retiring the orphan `sentinel/v0.6.1` tag a prerequisite, not a tidy-up.** -That tag sits in this repo with no sentinel release behind it. Offchain's sentinel line -reaches `v0.2.6`, so its next release is `sentinel/v0.2.7`, below the orphan. Leave the -orphan in place and the sentinel version line reads backwards, and a later release at -v0.6.1 fails on an existing tag. +``` +sentinel +``` + +**That collision resolves by deleting the orphan, which we want to do anyway.** This repo +holds a single `sentinel/v0.6.1` tag with no sentinel release behind it: no goreleaser +config, no published artifact. Offchain's sentinel line is the live one and reaches +`v0.2.6`, so its next release is `sentinel/v0.2.7`, below the orphan. Left in place the +version line reads backwards, and a later release at v0.6.1 fails on an existing tag. + +So: **delete `sentinel/v0.6.1` before the import**, then import every tag unprefixed. No +`--tag-rename`, no long names, and every component keeps one continuous version line. + +Check the collision set again immediately before importing, in case new tags have landed: + +```sh +comm -12 <(gh api 'repos/malbeclabs/doublezero-offchain/tags?per_page=100' --paginate \ + -q '.[].name' | sed -E 's|/v[0-9].*$||' | sort -u) \ + <(gh api 'repos/malbeclabs/doublezero/tags?per_page=100' --paginate \ + -q '.[].name' | sed -E 's|/v[0-9].*$||' | sort -u) +``` + +Anything it prints needs a decision before step 1 runs. ## Sequencing @@ -190,8 +206,9 @@ either way. ### Step 1. Import the code and the history. -Run `git filter-repo --path-rename` on each source repo to relocate its tree, with -`--tag-rename` per D5. Add each as a remote and merge with +First delete the orphan `sentinel/v0.6.1` tag per D5, then re-run the collision check +in D5 and resolve anything it prints. Then run `git filter-repo --path-rename` on each +source repo to relocate its tree, keeping tags unprefixed. Add each as a remote and merge with `--allow-unrelated-histories`. Fetch `main` only. Offchain alone has 599 refs and none of the others are wanted. @@ -235,7 +252,6 @@ component, so offchain's five fit the existing shape. - **Change `release.github.name` in the five goreleaser configs from `doublezero-offchain` to `doublezero`.** The `owner` field was corrected during the org move. The `name` field becomes wrong the moment the code lives here. -- Retire the orphan `sentinel/v0.6.1` tag per D5. Gate: push a throwaway release candidate tag on the smallest component. Confirm the release lands under malbeclabs and the package reaches Cloudsmith. Config review proves @@ -248,9 +264,8 @@ redirects, 11 forks, and old release download URLs all still resolve through the Archiving also keeps shreds building on its current pins. An archived repo still serves reads, so `malbeclabs/doublezero-solana` at tag `revenue-distribution/v0.3.7` keeps -resolving. The tag renaming in D5 applies to the copies imported here, not to the tags in -the archived repos. That decouples step 5 from step 4: shreds can be repointed when it -suits, not the same day. +resolving, and the same tag imported here points at the same commit. That decouples step 5 +from step 4: shreds can be repointed when it suits, not the same day. ### Step 5. Repoint shreds. Separate repo, separate risk. From d3d1cfd25803e5219c720e4cbb825c4028a28902 Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 09:08:40 -0700 Subject: [PATCH 03/11] docs: carry forward the edition trap and reward-drift warnings from infra#1515 --- .../2026-08-27-monorepo-migration-design.md | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index dd01c2c176..9d73555ada 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -40,6 +40,36 @@ Out of scope, decided deliberately: collision. Whether the two crates should become one is a later question. - **Any other malbeclabs repo.** The pattern here can be reused for a later wave. +## Prior art + +An earlier attempt is tracked in malbeclabs/infra#1952 (offchain into the monorepo) and +malbeclabs/infra#1515 (folding the `doublezero-solana` CLI into the main binary). Both are +open and neither binds this spec, but they were read before writing it and they changed it. + +Where they agree with this spec, independently: offchain lands under a **top-level +`offchain/`** rather than inside `crates/`, for the same reason (`crates/` is a flat bucket +of single crates; offchain is a multi-crate tree plus an Elixir component). The staged, +stop-anywhere shape is also the same. + +Where this spec differs: + +- **#1952 leaves `doublezero-solana` out of scope** as an external git dependency. This + spec brings it in. The objection at the time was partly a cross-org governance one, and + that is gone: both repos now sit in `malbeclabs`. +- **#1952 Phase 2a wanted the monorepo bumped to 1.92 plus a musl target.** Both are + obsolete. The monorepo is now on 1.97.1, so the toolchain moves the other way (D3), and + it already builds `x86_64-unknown-linux-musl` in `rust.yml`, `release.client.yml`, + `release.daily.yml` and `release.pipeline.validation.yml`. + +Two warnings from #1515 are carried into this spec, in D3 and in Risks. They were the most +valuable thing in either issue. + +**Out of scope but unblocked:** #1515 wants a `doublezero solana ` surface mounted in +the main binary. It notes that a shared `CliContext` only unifies inside one workspace, +which is why its plan needs reach-back and pin alignment. Step 2 of this spec removes that +constraint, so #1515's increments reduce to adding a path dependency and a subcommand. This +spec does not do that work. + ## The state we are starting from | | doublezero | offchain | solana | @@ -132,6 +162,18 @@ clippy findings. That work belongs to step 2 and is not a surprise to discover l borsh unifies on 1.7.0. +**The edition trap, from #1515.** This repo's `[workspace.package]` sets +`edition = "2021"`. Offchain's sets `"2024"`. All **14** offchain crates declare +`edition.workspace = true`, so folding them into the root workspace silently moves every +one of them from 2024 to 2021, and they fail to build. + +Fix: set `edition = "2024"` explicitly on all 14 crates in step 2. One line each. If this +repo later moves its workspace to 2024, those lines drop out. + +Rejected: bumping this repo's workspace edition to 2024 as part of the merge. It would +touch all 22 existing crates and put an unrelated migration inside the step that already +carries the most risk. + ### D4. Rename this repo's sentinel binary. Both repos have a `crates/sentinel`, and both declare `[[bin]] name = "doublezero-sentinel"`. @@ -334,6 +376,19 @@ interacts with step 5. Volume is unknown until tried. Step 0 can measure it at the same time as the program bytes, for free. +**Reward and debt output can change silently. This is the risk to take seriously.** +#1515 flags it: relocating `contributor-rewards` and `validator-debt` re-resolves their +maths in a different lockfile and feature context, and the output can change without +anything failing. Both crates move into the root workspace in step 2 of this spec. + +There are **no golden or snapshot tests** in either crate today, so nothing would catch it. +The failure mode is wrong reward or debt figures that build clean and pass every test. + +Mitigation, and it should land before step 2 rather than inside it: add golden tests to +both crates on `main` as they stand now. Fixed input, byte-identical output. Then step 2 +either keeps them green or names exactly what moved. Writing goldens against today's +behaviour is also useful on its own, whatever happens to this migration. + **One lockfile, one resolution.** Today three lockfiles disagreeing is a visible signal. After step 2 one lockfile resolves silently. The Solana crates agree today only because of an exact pin (`solana-sdk = "=3.0"` in all three). Keep that pin. From 1b5475c6527253e78bbcd9e233676732373a42ff Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 09:09:18 -0700 Subject: [PATCH 04/11] docs: map the superseded infra#1952 sub-issues --- .../2026-08-27-monorepo-migration-design.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index 9d73555ada..73b75feafc 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -62,7 +62,21 @@ Where this spec differs: `release.daily.yml` and `release.pipeline.validation.yml`. Two warnings from #1515 are carried into this spec, in D3 and in Risks. They were the most -valuable thing in either issue. +valuable thing in either issue. Neither issue stalled for a technical reason, so nothing in +them was invalidated by discovery. + +#1952 has five open, unassigned sub-issues. This is how they map, so nobody works from two +plans at once: + +| Issue | Fate | +| --- | --- | +| #1947 Phase 0, import under `offchain/` | Superseded by step 1, which also imports solana | +| #1948 Phase 1, flip git deps to path deps | Folded into step 2 | +| #1949 Phase 2a, toolchain 1.92 plus musl | **Obsolete.** Monorepo is on 1.97.1 and already builds musl | +| #1950 Phase 2b, unify the workspace | Becomes step 2 | +| #1951 Phase 3, distribute crates and mount verbs | **Still valid, out of scope here, unblocked by step 2** | + +Close #1947 through #1950 against this spec if it is accepted. Keep #1951 and #1515 open. **Out of scope but unblocked:** #1515 wants a `doublezero solana ` surface mounted in the main binary. It notes that a shared `CliContext` only unifies inside one workspace, From c303ceab0fd3d983a47f8c9323971d92625c2553 Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 09:09:44 -0700 Subject: [PATCH 05/11] docs: narrow the reward-drift risk to contributor-rewards --- .../2026-08-27-monorepo-migration-design.md | 30 +++++++++++++------ 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index 73b75feafc..6bcf0df6c8 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -390,18 +390,30 @@ interacts with step 5. Volume is unknown until tried. Step 0 can measure it at the same time as the program bytes, for free. -**Reward and debt output can change silently. This is the risk to take seriously.** -#1515 flags it: relocating `contributor-rewards` and `validator-debt` re-resolves their -maths in a different lockfile and feature context, and the output can change without -anything failing. Both crates move into the root workspace in step 2 of this spec. +**Contributor reward output can change silently. This is the risk to take seriously.** +#1515 flags it for both `contributor-rewards` and `validator-debt`: relocating them +re-resolves their maths in a different lockfile and feature context, and the output can +change without anything failing. Both move into the root workspace in step 2. -There are **no golden or snapshot tests** in either crate today, so nothing would catch it. -The failure mode is wrong reward or debt figures that build clean and pass every test. +Only `contributor-rewards` is load-bearing. Validator debt is no longer being collected, so +drift in `validator-debt` output changes no payment. It still has to compile and it still +releases, but it does not need a correctness gate. + +`contributor-rewards` decides what contributors are paid. It has **no golden or snapshot +tests** today, so nothing would catch a change. The failure mode is wrong reward figures +that build clean and pass every test. Mitigation, and it should land before step 2 rather than inside it: add golden tests to -both crates on `main` as they stand now. Fixed input, byte-identical output. Then step 2 -either keeps them green or names exactly what moved. Writing goldens against today's -behaviour is also useful on its own, whatever happens to this migration. +`contributor-rewards` on `main` as it stands now. Fixed input, byte-identical output. Then +step 2 either keeps them green or names exactly what moved. Writing goldens against today's +behaviour is worth doing on its own merits, whatever happens to this migration. + +**Is `validator-debt` worth migrating at all?** If it is not collecting and is not expected +to, moving it, releasing it and carrying it in the workspace is work spent on dormant code. +Deleting it, or archiving it in place, may be cheaper than migrating it. Worth a decision +before step 2 rather than after. Note it is still deployed: `malbeclabs/infra` runs it from +the offchain scheduler with its own AWS credentials, and it ships a Cloudsmith package, so +this is a real decision and not a formality. **One lockfile, one resolution.** Today three lockfiles disagreeing is a visible signal. After step 2 one lockfile resolves silently. The Solana crates agree today only because From 711b80456c9c9ff17328713e65f5e4ddd416b92e Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 09:12:44 -0700 Subject: [PATCH 06/11] docs: split the workspace step, add a goldens step, keep validator-debt --- .../2026-08-27-monorepo-migration-design.md | 191 +++++++++++------- 1 file changed, 115 insertions(+), 76 deletions(-) diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index 6bcf0df6c8..d795db7564 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -70,17 +70,17 @@ plans at once: | Issue | Fate | | --- | --- | -| #1947 Phase 0, import under `offchain/` | Superseded by step 1, which also imports solana | -| #1948 Phase 1, flip git deps to path deps | Folded into step 2 | +| #1947 Phase 0, import under `offchain/` | Superseded by step 2, which also imports solana | +| #1948 Phase 1, flip git deps to path deps | Becomes step 3 | | #1949 Phase 2a, toolchain 1.92 plus musl | **Obsolete.** Monorepo is on 1.97.1 and already builds musl | -| #1950 Phase 2b, unify the workspace | Becomes step 2 | -| #1951 Phase 3, distribute crates and mount verbs | **Still valid, out of scope here, unblocked by step 2** | +| #1950 Phase 2b, unify the workspace | Becomes step 4 | +| #1951 Phase 3, distribute crates and mount verbs | **Still valid, out of scope here, unblocked by step 4** | Close #1947 through #1950 against this spec if it is accepted. Keep #1951 and #1515 open. **Out of scope but unblocked:** #1515 wants a `doublezero solana ` surface mounted in the main binary. It notes that a shared `CliContext` only unifies inside one workspace, -which is why its plan needs reach-back and pin alignment. Step 2 of this spec removes that +which is why its plan needs reach-back and pin alignment. Step 4 of this spec removes that constraint, so #1515's increments reduce to adding a path dependency and a subcommand. This spec does not do that work. @@ -172,7 +172,7 @@ Add `solana/programs/rust-toolchain.toml` pinning 1.91, mirroring `smartcontract/programs/rust-toolchain.toml`. Offchain's repo-wide 1.92.0 pin goes away and its crates build on 1.97.1. Expect new -clippy findings. That work belongs to step 2 and is not a surprise to discover later. +clippy findings. That work belongs to step 4 and is not a surprise to discover later. borsh unifies on 1.7.0. @@ -181,7 +181,7 @@ borsh unifies on 1.7.0. `edition.workspace = true`, so folding them into the root workspace silently moves every one of them from 2024 to 2021, and they fail to build. -Fix: set `edition = "2024"` explicitly on all 14 crates in step 2. One line each. If this +Fix: set `edition = "2024"` explicitly on all 14 crates in step 4. One line each. If this repo later moves its workspace to 2024, those lines drop out. Rejected: bumping this repo's workspace edition to 2024 as part of the merge. It would @@ -240,12 +240,16 @@ comm -12 <(gh api 'repos/malbeclabs/doublezero-offchain/tags?per_page=100' --pag -q '.[].name' | sed -E 's|/v[0-9].*$||' | sort -u) ``` -Anything it prints needs a decision before step 1 runs. +Anything it prints needs a decision before step 2 runs. ## Sequencing -One throwaway measurement, then four pull requests in this repo, then one follow-on -pull request in `doublezero-shreds`. +One throwaway measurement, then six pull requests in this repo, then one follow-on pull +request in `doublezero-shreds`. + +The riskiest work is split across steps 3 and 4 so that flipping the dependencies and +merging the workspaces fail separately. Each step is revertable on its own except step 2, +which is the one-way door. ### Step 0. Measure the program bytes. Throwaway. @@ -257,14 +261,29 @@ make build-artifacts NETWORK=mainnet-beta shasum -a 256 -c programs/sha256sums_mainnet_beta.txt ``` -Repeat for `NETWORK=development`. Output is one fact that decides D2. Discard the work -either way. +Repeat for `NETWORK=development`. Output is one fact that decides D2. Measure the clippy +volume from the toolchain bump at the same time, since the tree is already there. Discard +the work either way. + +### Step 1. Golden tests for `contributor-rewards`. + +Land on `main`, before anything moves. Fixed input, byte-identical reward output. This crate +decides what contributors are paid and has no output test today, so step 4 currently has no +way to prove it changed nothing. + +This step stands on its own merits and is worth landing whether or not the rest of this +spec proceeds. -### Step 1. Import the code and the history. +`validator-debt` does not need the same gate. Debt is not being collected, so drift in its +output changes no payment. It still has to compile and release, and it stays in the tree. -First delete the orphan `sentinel/v0.6.1` tag per D5, then re-run the collision check -in D5 and resolve anything it prints. Then run `git filter-repo --path-rename` on each -source repo to relocate its tree, keeping tags unprefixed. Add each as a remote and merge with +Gate: the goldens pass on `main` and fail if a reward figure moves. + +### Step 2. Import the code and the history. One-way door. + +First delete the orphan `sentinel/v0.6.1` tag per D5, then re-run the collision check in D5 +and resolve anything it prints. Then run `git filter-repo --path-rename` on each source +repo to relocate its tree, keeping tags unprefixed. Add each as a remote and merge with `--allow-unrelated-histories`. Fetch `main` only. Offchain alone has 599 refs and none of the others are wanted. @@ -272,58 +291,81 @@ Add `offchain/` and `solana/` to the root `exclude` list wholesale. Both trees k own `Cargo.toml` and `Cargo.lock` and build as nested workspaces, exactly as they do today. -Do not bring their `.github/workflows/` across yet. Eleven workflows firing on a repo -that does not expect them is noise. +Do not bring their `.github/workflows/` across yet. Eleven workflows firing on a repo that +does not expect them is noise. + +Gate: every existing job green, with no existing job definition touched. `git diff` between +each imported tree and its filtered source is empty. The diff is large and needs no +judgement to review. + +### Step 3. Flip the git dependencies to path dependencies. -Gate: every existing job green, with no existing job definition touched. `git diff` -between each imported tree and its filtered source is empty. The diff is large and needs -no judgement to review. +Both trees are still excluded nested workspaces. A path dependency may point outside its +own workspace, so this works before the workspaces merge, and it delivers most of the value +on its own. -### Step 2. Merge the workspaces. +- Offchain's 7 `malbeclabs/doublezero` git deps become paths into `crates/`, `client/`, + `config/` and `smartcontract/`. +- Offchain's 3 `doublezero-solana` git deps become paths into `solana/`. +- **All ten pins stop existing.** The class of failure that produced 194 errors on + 2026-08-26 is gone from this repo at the end of this step. -This is the step that can hurt, and it hurts alone. +Nothing else changes. Toolchains, editions, borsh and the sentinel binary name are all +untouched, and both nested lockfiles stay in place. + +Gate: both nested workspaces build and test standalone. No `git+` source for +`malbeclabs/doublezero` or `malbeclabs/doublezero-solana` remains in either nested +lockfile. + +### Step 4. Merge the workspaces. + +Now a manifest consolidation plus the toolchain work, with the dependency flip already +proven by step 3. - Root workspace absorbs the crates per D2. -- Delete the four nested `Cargo.toml` and `Cargo.lock` files. -- **Ten git dependencies become path dependencies.** Offchain's seven - `malbeclabs/doublezero` dependencies and its three `doublezero-solana` dependencies are - all same-repo now. The pinning that failed on 2026-08-26 stops existing. -- Apply D3 (toolchain, borsh) and D4 (sentinel rename). +- Delete the four nested `Cargo.toml` and `Cargo.lock` files, and `offchain/rust-toolchain.toml`. + A `rust-toolchain.toml` is directory-scoped, so leaving it would give a different compiler + depending on which directory cargo was invoked from. +- Set `edition = "2024"` explicitly on all 14 offchain crates per D3. +- Offchain's crates move to 1.97.1; borsh unifies on 1.7.0; rename the sentinel binary + per D4. +- Pin the fixture generators and convert solana's `>=2,<=3` ranges to exact pins. Gate: `cargo check --workspace`, `cargo clippy --all-targets`, the full test suite, e2e, -and the program checksum check. Also confirm the lockfile holds no crate twice: -`grep '^name = ' Cargo.lock | sort | uniq -d` should print nothing that was not already -duplicated before the merge. +the program checksum check, and **the step 1 goldens still green**. Also confirm the +lockfile holds no crate twice: `grep '^name = ' Cargo.lock | sort | uniq -d` should print +nothing that was not already duplicated before the merge. -### Step 3. Merge release and CI. +### Step 5. Merge release and CI. Fold 11 workflows into this repo's 49. This repo already runs one release workflow per component, so offchain's five fit the existing shape. - Dedupe the overlaps: two `local-validator.yml`, three Rust CI configs (`rust.yml` twice plus offchain's `ci.yml`), two `changelog-reminder.yml`. -- Add `erlef/setup-beam` for the Elixir scheduler. +- Add `erlef/setup-beam` for the Elixir scheduler, with CI path-scoped to + `offchain/scheduler/**` so it does not run on unrelated changes. - Copy offchain's three release secrets onto this repo: `CLOUDSMITH_TOKEN`, `GORELEASER_KEY`, `SLACK_BOTS_WEBHOOK`. - **Change `release.github.name` in the five goreleaser configs from `doublezero-offchain` to `doublezero`.** The `owner` field was corrected during the org move. The `name` field becomes wrong the moment the code lives here. -Gate: push a throwaway release candidate tag on the smallest component. Confirm the -release lands under malbeclabs and the package reaches Cloudsmith. Config review proves -the target is named correctly. It does not prove the token can write there. +Gate: push a throwaway release candidate tag on the smallest component. Confirm the release +lands under malbeclabs and the package reaches Cloudsmith. Config review proves the target +is named correctly. It does not prove the token can write there. -### Step 4. Decommission the source repos. +### Step 6. Decommission the source repos. Archive both read-only with a README pointing here. Do not delete them. Their transfer redirects, 11 forks, and old release download URLs all still resolve through them. Archiving also keeps shreds building on its current pins. An archived repo still serves reads, so `malbeclabs/doublezero-solana` at tag `revenue-distribution/v0.3.7` keeps -resolving, and the same tag imported here points at the same commit. That decouples step 5 -from step 4: shreds can be repointed when it suits, not the same day. +resolving, and the same tag imported here points at the same commit. That decouples step 7 +from step 6: shreds can be repointed when it suits, not the same day. -### Step 5. Repoint shreds. Separate repo, separate risk. +### Step 7. Repoint shreds. Separate repo, separate risk. `doublezero-shreds` consumes both source repos over git. Once they are archived it must point at `malbeclabs/doublezero`. See below for why this needs care. @@ -367,16 +409,19 @@ That is the price of the public and private split, and we are paying it on purpo ## Rollback -Steps 2, 3 and 4 revert cleanly. +Every step reverts cleanly except step 2. + +Step 3 reverts to git dependencies. Step 4 reverts **because steps 2 and 3 leave the nested +manifests in place**: step 4 only edits and deletes manifests, so reverting it restores the +four nested `Cargo.toml` and `Cargo.lock` files and leaves a repo that builds as it did +after step 3. Step 5 is workflows. Unarchiving a repo is a click. -Step 2 reverts cleanly **because step 1 deliberately touches no build file.** Step 2 only -edits and deletes manifests, so reverting it restores the four nested `Cargo.toml` and -`Cargo.lock` files and leaves a repo that builds as it did after step 1. Step 3 is -workflows. Unarchiving a repo is a click. +Splitting the old single workspace step into steps 3 and 4 is what buys this. A revert of +step 4 no longer drags the dependency flip back with it. -Step 1 is the one-way door. `git revert -m 1` removes the files, and the grafted history -stays in the graph unless `main` is rewritten. Treat merging step 1 as the commitment -point, not step 2. +Step 2 is the one-way door. `git revert -m 1` removes the files, and the grafted history +stays in the graph unless `main` is rewritten. Treat merging step 2 as the commitment +point. ## Risks @@ -384,46 +429,40 @@ point, not step 2. `programs/Cargo.toml` with old org URLs. They are not on `main`. Each one merged after the org move reintroduces an old URL. Harmless while the transfer redirects hold, and a build failure once they do not. Rebasing them is shreds work, not this migration's, but it -interacts with step 5. +interacts with step 7. -**Clippy churn on the toolchain bump.** Offchain moves from 1.92.0 to 1.97.1 in step 2. +**Clippy churn on the toolchain bump.** Offchain moves from 1.92.0 to 1.97.1 in step 4. Volume is unknown until tried. Step 0 can measure it at the same time as the program bytes, for free. **Contributor reward output can change silently. This is the risk to take seriously.** -#1515 flags it for both `contributor-rewards` and `validator-debt`: relocating them -re-resolves their maths in a different lockfile and feature context, and the output can -change without anything failing. Both move into the root workspace in step 2. - -Only `contributor-rewards` is load-bearing. Validator debt is no longer being collected, so -drift in `validator-debt` output changes no payment. It still has to compile and it still -releases, but it does not need a correctness gate. - -`contributor-rewards` decides what contributors are paid. It has **no golden or snapshot -tests** today, so nothing would catch a change. The failure mode is wrong reward figures -that build clean and pass every test. - -Mitigation, and it should land before step 2 rather than inside it: add golden tests to -`contributor-rewards` on `main` as it stands now. Fixed input, byte-identical output. Then -step 2 either keeps them green or names exactly what moved. Writing goldens against today's -behaviour is worth doing on its own merits, whatever happens to this migration. - -**Is `validator-debt` worth migrating at all?** If it is not collecting and is not expected -to, moving it, releasing it and carrying it in the workspace is work spent on dormant code. -Deleting it, or archiving it in place, may be cheaper than migrating it. Worth a decision -before step 2 rather than after. Note it is still deployed: `malbeclabs/infra` runs it from -the offchain scheduler with its own AWS credentials, and it ships a Cloudsmith package, so -this is a real decision and not a formality. +#1515 flags it: relocating `contributor-rewards` re-resolves its maths in a different +lockfile and feature context, and the output can change without anything failing. It moves +into the root workspace in step 4. + +`contributor-rewards` decides what contributors are paid, and it has **no golden or +snapshot tests** today. The failure mode is wrong reward figures that build clean and pass +every test. + +This is why step 1 exists and why it comes first: write the goldens against today's +behaviour on `main`, before anything moves. Step 4 then either keeps them green or names +exactly what changed. + +**`validator-debt` stays, without a correctness gate.** It is not collecting today, so +drift in its output changes no payment. It is kept deliberately rather than dropped: it is +still deployed, `malbeclabs/infra` runs it from the offchain scheduler with its own AWS +credentials, and it ships a Cloudsmith package. It migrates and keeps releasing like +everything else. **One lockfile, one resolution.** Today three lockfiles disagreeing is a visible signal. -After step 2 one lockfile resolves silently. The Solana crates agree today only because +After step 4 one lockfile resolves silently. The Solana crates agree today only because of an exact pin (`solana-sdk = "=3.0"` in all three). Keep that pin. **The fixture generators float.** `sdk/revdist/testdata/fixtures/generate-fixtures` declares its two `doublezero-solana` dependencies with no tag and no rev. Only the lockfile pins them, so the next `cargo update` there moves them to whatever `main` holds. -Pin them during step 2. +Pin them during step 4. **`doublezero-solana` uses version ranges** (`>=2,<=3`) on its program-side crates. That earns its keep for a library other repos consume. Inside one workspace it only produces -resolution nobody asked for. Convert to exact pins during step 2. +resolution nobody asked for. Convert to exact pins during step 4. From bc6118269a768214bbbb0cc5788f6c847cdcaecb Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 09:23:05 -0700 Subject: [PATCH 07/11] docs: record step 0 results, hold the solana programs out of the workspace --- ...6-06-17-least-privilege-oracle-overview.md | 130 ++ .../plans/2026-03-26-latency-command-ux.md | 829 +++++++++ .../2026-04-03-status-multicast-groups.md | 419 +++++ ...09-startup-tunnel-endpoint-registration.md | 185 +++ ...-23-multicast-modify-without-disconnect.md | 1476 +++++++++++++++++ .../2026-03-26-latency-command-ux-design.md | 86 + ...26-04-03-status-multicast-groups-design.md | 79 + ...ticast-modify-without-disconnect-design.md | 112 ++ ...y-feed-oracle-permission-account-design.md | 160 ++ .../2026-08-27-monorepo-migration-design.md | 149 +- 10 files changed, 3588 insertions(+), 37 deletions(-) create mode 100644 docs/superpowers/2026-06-17-least-privilege-oracle-overview.md create mode 100644 docs/superpowers/plans/2026-03-26-latency-command-ux.md create mode 100644 docs/superpowers/plans/2026-04-03-status-multicast-groups.md create mode 100644 docs/superpowers/plans/2026-04-09-startup-tunnel-endpoint-registration.md create mode 100644 docs/superpowers/plans/2026-04-23-multicast-modify-without-disconnect.md create mode 100644 docs/superpowers/specs/2026-03-26-latency-command-ux-design.md create mode 100644 docs/superpowers/specs/2026-04-03-status-multicast-groups-design.md create mode 100644 docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md create mode 100644 docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md diff --git a/docs/superpowers/2026-06-17-least-privilege-oracle-overview.md b/docs/superpowers/2026-06-17-least-privilege-oracle-overview.md new file mode 100644 index 0000000000..186b8e04d1 --- /dev/null +++ b/docs/superpowers/2026-06-17-least-privilege-oracle-overview.md @@ -0,0 +1,130 @@ +# Least-privilege feed oracle — change overview + +**Status:** proposal, pending team consensus. No code written. +**Driver:** malbeclabs/infra#1652 (reframed below) · parent #1547 Generalized Seat Buying. +**Repos touched:** `malbeclabs/doublezero` (serviceability program) and `malbeclabs/doublezero-shreds` (oracle). +**Detailed Unit-1 spec:** `doublezero/docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md`. + +## TL;DR + +The shred/feed oracle currently sits in `GlobalState.foundation_allowlist`, which is effectively +**near-superuser** (governance, allowlist edits, authority rotation, etc.). It only needs to manage +access passes, multicast subscriptions, and the users it provisions. We want to drop it to +**least privilege** by giving it a per-key **Permission PDA** carrying exactly +`ACCESS_PASS_ADMIN | USER_ADMIN`, and teaching the seven handlers it calls to honor that Permission +account. The change is **additive** — no existing caller's authorization changes. It splits into three +units (program → oracle wiring → operational rollout), sequenced carefully to avoid an unauthorized +window. + +## Why not the two obvious options + +The original #1652 idea was to give the oracle the scoped `feed_authority` role. Two findings killed +the simple paths: + +- **`feed_authority` is incompatible with the oracle's validator-owned flow.** The access-pass + handlers enforce an "owns-only" gate — `feed_authority == payer && accesspass.owner != payer → deny` + — and that gate fires **even for foundation members**. Validator-owned passes are owned by the + *validator* (the oracle deliberately does not re-`set` an existing pass, so as not to clobber the + validator's settings). An oracle holding `feed_authority` would be blocked on every validator-owned + pass. This is exactly why `feed_authority` was cleared during the validator-owned rollout. +- **The `sentinel` slot is taken.** `sentinel_authority_pk` is a single-pubkey slot already wired as + the **billing sentinel** (writes tenant payment status after deduction) and the **suspend / + circuit-breaker** authority. The oracle can't co-occupy it, and merging the network kill-switch onto + the hot oracle key would be a poor separation of duties. + +→ The **Permission-account (PDA) bitmask** system is the right vehicle: per-key, no slot contention, +no owns-only gate, and far narrower than foundation. + +## The chosen approach (mechanism) + +The program already has a `Permission` PDA per pubkey (`get_permission_pda(program_id, key)`) with a +`u128` bitmask and an `authorize()` helper. But `authorize()` bundles a **legacy fallback** whose +composite differs from each handler's hand-written inline check, so routing existing callers through +it would silently change their authorities. Instead: + +1. Add a **new, no-legacy-fallback** helper, `authorize_permission_account_only(...)`, that validates + *only* a passed Permission PDA (correct PDA for the signer, program-owned, `Activated`, required bit + set). +2. In each of the seven handlers, **keep the existing inline check verbatim** and OR in + `perm_only(BIT)`. Existing callers are untouched; a new Permission-account path is added. +3. The Permission account is an **optional trailing account** per instruction — today's callers omit + it and keep working; the oracle passes it. + +The oracle's Permission bitmask is `ACCESS_PASS_ADMIN | USER_ADMIN`. + +## What changes, by unit + +### Unit 1 — serviceability program (`malbeclabs/doublezero`) + +The seven handlers the oracle uses, the authority each preserves, and the bit added: + +| Handler | Preserved inline authority | Bit added | +|---|---|---| +| SetAccessPass | foundation / sentinel / feed (owns-it) / tenant-admin / pass-owner | `ACCESS_PASS_ADMIN` | +| CloseAccessPass | foundation / feed (owns-it) | `ACCESS_PASS_ADMIN` | +| AddMulticastGroupSubAllowlist | mgroup-owner / sentinel / feed (owns-it) / foundation | `ACCESS_PASS_ADMIN` | +| RemoveMulticastGroupSubAllowlist | same as add | `ACCESS_PASS_ADMIN` | +| UpdateMulticastGroupRoles (subscribe/unsubscribe) | pass `user_payer` / foundation | `ACCESS_PASS_ADMIN` | +| DeleteUser | foundation / `user.owner` | `USER_ADMIN` | +| CreateSubscribeUser (owner override) | foundation / sentinel | `USER_ADMIN` | + +Plus: the `perm_only` helper, integration tests per handler (positive Permission path, negative +no-account/insufficient/suspended, regression on existing paths), an IDL bump (one optional trailing +account per instruction), and a `PERMISSION.md` update. **Self-contained and shippable on its own** — +the new path is dormant until a caller passes a Permission account. + +One structural note for reviewers: because the Permission account is a *trailing* account, the final +authorization decision moves to just after each handler's required-account reads (before any state +mutation). Reads are non-destructive, so this is safe; it's a small restructure, not a logic change. + +### Unit 2 — oracle wiring (`malbeclabs/doublezero-shreds`) + +`dz_ledger.rs` appends the oracle's Permission PDA `AccountMeta` to the seven instructions it builds. +No behavior change until Unit 1 is deployed; the oracle continues to work via foundation in the +meantime. + +### Unit 3 — operational rollout + +`CreatePermission` for the oracle key with `ACCESS_PASS_ADMIN | USER_ADMIN`, then remove the oracle +from `foundation_allowlist`. **Sequencing matters** (we've had unauthorized-window incidents from +out-of-order authority changes before): + +1. Deploy Unit 1 (handlers honor Permission accounts). +2. Deploy Unit 2 (oracle passes its Permission account). +3. `CreatePermission` for the oracle key; **verify** the oracle operates end-to-end via the Permission + path while still in foundation. +4. Only then remove the oracle from `foundation_allowlist`. + +Doing step 4 before 1–3 are confirmed leaves the oracle unauthorized. + +## Security review framing + +- **Additive / behavior-preserving:** every existing caller (foundation, `user.owner`, pass + `user_payer`, `feed_authority` owns-it, sentinel, mgroup-owner) keeps its exact current path. The + only new capability is "a key with a valid `Activated` Permission PDA bearing the required bit." +- **Net privilege reduction:** the oracle goes from foundation (≈ all admin bits, incl. allowlist + edits and authority rotation) to exactly `ACCESS_PASS_ADMIN | USER_ADMIN`. Blast radius on oracle-key + compromise shrinks accordingly. +- **No owns-only gate on the Permission path** — required so the oracle can manage both + connection-ticket passes (which it owns) and validator-owned passes (which it does not). + +## Open questions for the team + +1. **Bit for `UpdateMulticastGroupRoles`:** proposed `ACCESS_PASS_ADMIN` (it's access-pass-gated). If + the team prefers `MULTICAST_ADMIN`, the oracle's bitmask changes to add it. +2. **`USER_ADMIN` semantics:** our `delete`/`create` changes make a Permission-account `USER_ADMIN` + holder able to delete/create users without an owns-it restriction (matching the oracle's + cross-owner cleanup job). Confirm that's the intended meaning of the bit for permission holders. +3. **`RequirePermissionAccounts`:** out of scope here (we stay additive and leave legacy paths intact). + Worth a separate decision on whether/when to flip the program-wide flag that retires legacy auth. +4. **`CreateSubscribeUser`:** plan-phase item — confirm there's no second foundation-only gate beyond + the owner-override check. + +## Issue / PR map + +- **#1652** — to be reframed from "feed-authority delete" to this Permission-PDA approach. +- **#1547** (parent) — Generalized Seat Buying. +- **doublezero-shreds #501** (connection-ticket cap eviction + zero-connection close) — the feature + that exercises the oracle's delete/unsubscribe/close; it currently relies on the oracle's foundation + membership. This least-privilege work is its security follow-up; #501 does not depend on it (it ships + on the foundation path in the interim). diff --git a/docs/superpowers/plans/2026-03-26-latency-command-ux.md b/docs/superpowers/plans/2026-03-26-latency-command-ux.md new file mode 100644 index 0000000000..1bdc55e553 --- /dev/null +++ b/docs/superpowers/plans/2026-03-26-latency-command-ux.md @@ -0,0 +1,829 @@ +# Latency Command UX Improvements Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Fix the daemon probe race condition, expose readiness state, and add spinner + better error messages to the CLI latency command. + +**Architecture:** The daemon's `LatencyManager` gets a `devicesFetched` channel so the probe goroutine waits for device data before its first run, and an `atomic.Bool` to track probe readiness. `ServeLatency` wraps the response in `{"ready": bool, "results": [...]}`. The CLI deserializes this new format, adds a spinner, and uses the `ready` flag to show accurate progress messages. + +**Tech Stack:** Go (daemon), Rust (CLI), `indicatif` (spinner), `sync/atomic` (readiness flag) + +**Spec:** `docs/superpowers/specs/2026-03-26-latency-command-ux-design.md` + +--- + +### Task 1: Daemon — Add `devicesFetched` channel and `probeReady` flag to LatencyManager + +**Files:** +- Modify: `client/doublezerod/internal/latency/manager.go:192-220` (struct + constructor) +- Test: `client/doublezerod/internal/latency/manager_test.go` + +- [ ] **Step 1: Write the failing test — probe waits for device fetch** + +Add a new test to `manager_test.go` that verifies the probe goroutine does not run before the device cache is populated. Use a slow smart contract func that takes 500ms, and verify that the first probe sees the devices (not an empty cache). + +```go +func TestLatencyManager_ProbeWaitsForDeviceFetch(t *testing.T) { + probeTargets := make(chan []latency.ProbeTarget, 1) + + slowSmartContractFunc := func(ctx context.Context) (*latency.ContractData, error) { + time.Sleep(500 * time.Millisecond) + return &latency.ContractData{ + Devices: []serviceability.Device{ + { + AccountType: serviceability.DeviceType, + PublicIp: [4]uint8{127, 0, 0, 1}, + PubKey: [32]byte{1}, + Code: "dev01", + }, + }, + }, nil + } + + mockProber := func(ctx context.Context, target latency.ProbeTarget) latency.LatencyResult { + probeTargets <- []latency.ProbeTarget{target} + return latency.LatencyResult{ + Min: 1, + Max: 10, + Avg: 5, + Loss: 0, + Device: target.Device, + IP: target.IP, + Reachable: true, + } + } + + manager := latency.NewLatencyManager( + latency.WithSmartContractFunc(slowSmartContractFunc), + latency.WithProberFunc(mockProber), + latency.WithProbeInterval(30*time.Second), + latency.WithCacheUpdateInterval(30*time.Second), + ) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + + go func() { + _ = manager.Start(ctx) + }() + + // The probe should have received actual targets (not empty), meaning it waited for fetch + select { + case targets := <-probeTargets: + if len(targets) == 0 { + t.Fatal("probe ran with empty targets — did not wait for device fetch") + } + case <-time.After(3 * time.Second): + t.Fatal("timed out waiting for probe to run") + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `go test -run TestLatencyManager_ProbeWaitsForDeviceFetch -v -count=1 ./client/doublezerod/internal/latency/...` + +Expected: The test may pass or fail depending on timing. If it passes, the race is hard to reproduce deterministically — that's OK, the structural fix is still needed. Proceed to the implementation. + +- [ ] **Step 3: Add `devicesFetched` channel and `probeReady` flag to LatencyManager** + +In `manager.go`, add two fields to the `LatencyManager` struct: + +```go +type LatencyManager struct { + SmartContractFunc SmartContractorFunc + fetcher Fetcher + proberFunc ProberFunc + DeviceCache *DeviceCache + ResultsCache *LatencyResults + probeInterval time.Duration + cacheUpdateInterval time.Duration + metricsEnabled bool + probeTunnelEndpoints bool + devicesFetched chan struct{} // closed after first successful fetch + probeReady atomic.Bool // true after first probe completes +} +``` + +Add `"sync/atomic"` to imports. + +Update `NewLatencyManager` to initialize the channel: + +```go +func NewLatencyManager(options ...Option) *LatencyManager { + lm := &LatencyManager{ + DeviceCache: &DeviceCache{Devices: []serviceability.Device{}, Lock: sync.Mutex{}}, + ResultsCache: &LatencyResults{Results: []LatencyResult{}, Lock: sync.RWMutex{}}, + proberFunc: UdpPing, + probeInterval: 10 * time.Second, + cacheUpdateInterval: 300 * time.Second, + metricsEnabled: false, + devicesFetched: make(chan struct{}), + } + for _, o := range options { + o(lm) + } + return lm +} +``` + +Add a public getter for readiness: + +```go +func (l *LatencyManager) IsProbeReady() bool { + return l.probeReady.Load() +} +``` + +- [ ] **Step 4: Update `Start()` — fetch goroutine closes channel, probe goroutine waits** + +In the fetch goroutine, close `devicesFetched` after the first successful fetch: + +```go +go func() { + fetch := func() { + // ... existing fetch logic unchanged ... + } + // don't wait for first tick and populate cache + fetch() + // Signal that initial device data is available for probing + select { + case <-l.devicesFetched: + // already closed + default: + close(l.devicesFetched) + } + + ticker := time.NewTicker(l.cacheUpdateInterval) + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + fetch() + } + } +}() +``` + +In the probe goroutine, wait for `devicesFetched` before the first probe: + +```go +go func() { + probe := func() { + // ... existing probe logic unchanged ... + } + + // Wait for initial device fetch before first probe + select { + case <-l.devicesFetched: + case <-ctx.Done(): + return + } + + // don't wait for first tick to ping stuff + probe() + l.probeReady.Store(true) + + ticker := time.NewTicker(l.probeInterval) + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + probe() + } + } +}() +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `go test -run TestLatencyManager_ProbeWaitsForDeviceFetch -v -count=1 ./client/doublezerod/internal/latency/...` + +Expected: PASS + +- [ ] **Step 6: Run all existing latency tests to verify no regressions** + +Run: `go test -v -count=1 ./client/doublezerod/internal/latency/...` + +Expected: All tests pass. + +- [ ] **Step 7: Commit** + +```bash +git add client/doublezerod/internal/latency/manager.go client/doublezerod/internal/latency/manager_test.go +git commit -m "client: add devicesFetched channel and probeReady flag to LatencyManager" +``` + +--- + +### Task 2: Daemon — Update `ServeLatency` response to include readiness + +**Files:** +- Modify: `client/doublezerod/internal/latency/manager.go:386-394` (ServeLatency handler) +- Test: `client/doublezerod/internal/latency/manager_test.go` + +- [ ] **Step 1: Write the failing test — HTTP response includes ready field** + +Add a test that verifies the `/latency` HTTP response includes the `ready` and `results` fields. Add this after the existing `check_results_via_http_are_correct` test in `TestLatencyManager`: + +```go +func TestServeLatency_ResponseFormat(t *testing.T) { + manager := latency.NewLatencyManager( + latency.WithSmartContractFunc(func(context.Context) (*latency.ContractData, error) { + return &latency.ContractData{ + Devices: []serviceability.Device{ + { + AccountType: serviceability.DeviceType, + PublicIp: [4]uint8{127, 0, 0, 1}, + PubKey: [32]byte{1}, + Code: "dev01", + }, + }, + }, nil + }), + latency.WithProberFunc(func(ctx context.Context, target latency.ProbeTarget) latency.LatencyResult { + return latency.LatencyResult{ + Min: 1, Max: 10, Avg: 5, Loss: 0, + Device: target.Device, IP: target.IP, Reachable: true, + } + }), + latency.WithProbeInterval(30*time.Second), + latency.WithCacheUpdateInterval(30*time.Second), + ) + + // Before Start — probe not ready, no results + f, err := os.CreateTemp("/tmp", "doublezero-test.sock") + if err != nil { + t.Fatal(err) + } + defer os.Remove(f.Name()) + _ = unix.Unlink(f.Name()) + + lis, err := net.Listen("unix", f.Name()) + if err != nil { + t.Fatal(err) + } + mux := http.NewServeMux() + mux.HandleFunc("GET /latency", manager.ServeLatency) + server := http.Server{Handler: mux} + defer server.Close() + go func() { _ = server.Serve(lis) }() + + client := http.Client{ + Transport: &http.Transport{ + DialContext: func(_ context.Context, _, _ string) (net.Conn, error) { + return net.Dial("unix", f.Name()) + }, + }, + } + + // Test: before probing, ready should be false + resp, err := client.Get("http://localhost/latency") + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + buf, _ := io.ReadAll(resp.Body) + + var parsed struct { + Ready bool `json:"ready"` + Results []json.RawMessage `json:"results"` + } + if err := json.Unmarshal(buf, &parsed); err != nil { + t.Fatalf("failed to parse response as {ready, results}: %v\nbody: %s", err, buf) + } + if parsed.Ready { + t.Error("expected ready=false before probe has run") + } + + // Start the manager and wait for probe to complete + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + go func() { _ = manager.Start(ctx) }() + + // Poll until ready + deadline := time.Now().Add(3 * time.Second) + for time.Now().Before(deadline) { + if manager.IsProbeReady() { + break + } + time.Sleep(50 * time.Millisecond) + } + if !manager.IsProbeReady() { + t.Fatal("manager never became ready") + } + + // Test: after probing, ready should be true with results + resp2, err := client.Get("http://localhost/latency") + if err != nil { + t.Fatal(err) + } + defer resp2.Body.Close() + buf2, _ := io.ReadAll(resp2.Body) + + var parsed2 struct { + Ready bool `json:"ready"` + Results []json.RawMessage `json:"results"` + } + if err := json.Unmarshal(buf2, &parsed2); err != nil { + t.Fatalf("failed to parse response: %v\nbody: %s", err, buf2) + } + if !parsed2.Ready { + t.Error("expected ready=true after probe completed") + } + if len(parsed2.Results) == 0 { + t.Error("expected non-empty results after probe completed") + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `go test -run TestServeLatency_ResponseFormat -v -count=1 ./client/doublezerod/internal/latency/...` + +Expected: FAIL — the response is currently a bare JSON array, not `{"ready":..., "results":...}`. + +- [ ] **Step 3: Update `ServeLatency` to wrap response** + +In `manager.go`, replace the `ServeLatency` method: + +```go +// latencyResponse is the wire format for the /latency endpoint. +// This is internal to the daemon-CLI communication — not the user-facing output. +type latencyResponse struct { + Ready bool `json:"ready"` + Results *LatencyResults `json:"results"` +} + +func (l *LatencyManager) ServeLatency(w http.ResponseWriter, r *http.Request) { + resp := latencyResponse{ + Ready: l.probeReady.Load(), + Results: l.ResultsCache, + } + data, err := json.Marshal(resp) + if err != nil { + w.WriteHeader(http.StatusInternalServerError) + _, _ = fmt.Fprintf(w, "error generating latency: %v", err) + return + } + _, _ = w.Write(data) +} +``` + +- [ ] **Step 4: Update the existing `check_results_via_http_are_correct` test** + +The existing test in `TestLatencyManager` parses the response as `[]map[string]any`. Update it to expect the new wrapped format: + +```go +t.Run("check_results_via_http_are_correct", func(t *testing.T) { + req, err := http.NewRequest("GET", "http://localhost/latency", nil) + if err != nil { + t.Fatalf("error generating http request: %v", err) + } + resp, err := client.Do(req) + if err != nil { + t.Fatalf("error while making http request: %v", err) + } + defer resp.Body.Close() + + buf, _ := io.ReadAll(resp.Body) + var parsed struct { + Ready bool `json:"ready"` + Results []map[string]any `json:"results"` + } + if err := json.Unmarshal(buf, &parsed); err != nil { + t.Fatalf("error unmarshaling latency data: %v\nbody: %s", err, buf) + } + + if !parsed.Ready { + t.Error("expected ready=true") + } + + want := []map[string]any{ + { + "device_pk": base58.Encode(tests[0].DeviceCache[0].PubKey[:]), + "device_code": tests[0].DeviceCache[0].Code, + "device_ip": "127.0.0.1", + "min_latency_ns": float64(1), + "max_latency_ns": float64(10), + "avg_latency_ns": float64(5), + "loss_percentage": float64(0), + "reachable": true, + }, + } + + if diff := cmp.Diff(want, parsed.Results); diff != "" { + t.Errorf("LatencyResults mismatch (-want +got): %s\n", diff) + } +}) +``` + +- [ ] **Step 5: Run all latency tests to verify** + +Run: `go test -v -count=1 ./client/doublezerod/internal/latency/...` + +Expected: All tests pass. + +- [ ] **Step 6: Commit** + +```bash +git add client/doublezerod/internal/latency/manager.go client/doublezerod/internal/latency/manager_test.go +git commit -m "client: update ServeLatency to include readiness in response" +``` + +--- + +### Task 3: CLI — Update `ServiceController::latency()` to return `LatencyResponse` + +**Files:** +- Modify: `client/doublezero/src/servicecontroller.rs:15-30,167-179,233-250` + +- [ ] **Step 1: Add `LatencyResponse` struct** + +In `servicecontroller.rs`, add the new struct after the `LatencyRecord` definition (after line 50): + +```rust +#[derive(Deserialize, Debug)] +pub struct LatencyResponse { + pub ready: bool, + pub results: Vec, +} +``` + +- [ ] **Step 2: Update `ServiceController` trait and impl** + +Change the trait method signature at line 173: + +```rust +async fn latency(&self) -> eyre::Result; +``` + +Change the implementation at line 233-250: + +```rust +async fn latency(&self) -> eyre::Result { + let uri = Uri::new(&self.socket_path, "/latency").into(); + let client: Client> = + Client::builder(TokioExecutor::new()).build(UnixConnector); + let res = client + .get(uri) + .await + .map_err(|e| eyre!("Unable to connect to doublezero daemon: {e}"))?; + + let data = res + .into_body() + .collect() + .await + .map_err(|e| eyre!("Unable to read response body: {e}"))? + .to_bytes(); + + parse_daemon_response::(&data, "/latency") +} +``` + +- [ ] **Step 3: Update the mock expectation return type** + +The `#[automock]` macro will auto-generate the mock. But all existing test code that calls `expect_latency()` returns `Ok(vec![...])`. These need to return `Ok(LatencyResponse { ready: true, results: vec![...] })`. + +This is handled in Task 5 (updating `dzd_latency.rs`). + +- [ ] **Step 4: Verify it compiles (expect test failures from mock changes)** + +Run: `cargo check -p doublezero` + +Expected: Compilation errors in `dzd_latency.rs` tests where `expect_latency().returning(...)` returns the old type. That's expected — Task 5 fixes those. + +- [ ] **Step 5: Commit** + +```bash +git add client/doublezero/src/servicecontroller.rs +git commit -m "client: update ServiceController::latency() to return LatencyResponse" +``` + +--- + +### Task 4: CLI — Add spinner to latency command + +**Files:** +- Modify: `client/doublezero/src/command/latency.rs` + +- [ ] **Step 1: Add spinner to latency command** + +Replace the entire `latency.rs` file content: + +```rust +use crate::command::util; +use clap::Args; +use doublezero_cli::doublezerocommand::CliCommand; +use doublezero_sdk::commands::device::list::ListDeviceCommand; +use indicatif::{ProgressBar, ProgressStyle}; +use std::time::Duration; + +use crate::{ + dzd_latency::retrieve_latencies, requirements::check_doublezero, + servicecontroller::ServiceControllerImpl, +}; + +#[derive(Args, Debug)] +pub struct LatencyCliCommand { + /// Output as json + #[arg(long, default_value = "false")] + json: bool, +} + +impl LatencyCliCommand { + pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { + let controller = ServiceControllerImpl::new(None); + + let spinner = ProgressBar::new_spinner(); + spinner.set_style( + ProgressStyle::default_spinner() + .template("{spinner:.green} [{elapsed_precise}] {msg}") + .expect("Failed to set template") + .tick_strings(&["-", "\\", "|", "/"]), + ); + spinner.enable_steady_tick(Duration::from_millis(100)); + spinner.set_message("Checking daemon..."); + + check_doublezero(&controller, client, Some(&spinner)).await?; + + spinner.set_message("Fetching devices..."); + let devices = client.list_device(ListDeviceCommand)?; + + let latencies = + retrieve_latencies(&controller, &devices, false, Some(&spinner)).await?; + + spinner.finish_and_clear(); + util::show_output(latencies, self.json)?; + + Ok(()) + } +} +``` + +- [ ] **Step 2: Verify it compiles** + +Run: `cargo check -p doublezero` + +Expected: May still have errors from Task 3 mock changes — that's OK if Task 3 is committed but Task 5 isn't yet. + +- [ ] **Step 3: Commit** + +```bash +git add client/doublezero/src/command/latency.rs +git commit -m "client: add spinner to latency command" +``` + +--- + +### Task 5: CLI — Update `retrieve_latencies` to use readiness flag + +**Files:** +- Modify: `client/doublezero/src/dzd_latency.rs:68-124` +- Test: `client/doublezero/src/dzd_latency.rs` (test module) + +- [ ] **Step 1: Update `retrieve_latencies` to use `LatencyResponse`** + +Replace the `retrieve_latencies` function (lines 68-124): + +```rust +pub async fn retrieve_latencies( + controller: &T, + devices: &HashMap, + reachable_only: bool, + spinner: Option<&indicatif::ProgressBar>, +) -> eyre::Result> { + if let Some(spinner) = spinner { + spinner.set_message("Retrieving latency stats..."); + } + + let max_wait = Duration::from_secs(60); + let poll_interval = Duration::from_secs(1); + let start = std::time::Instant::now(); + + let mut latencies = loop { + let response = controller.latency().await.map_err(|e| eyre::eyre!(e))?; + + let mut results = response.results; + results.retain(|l| { + Pubkey::from_str(&l.device_pk) + .ok() + .and_then(|pubkey| devices.get(&pubkey)) + .map(|device| device.status == DeviceStatus::Activated) + .unwrap_or(false) + }); + + if reachable_only { + results.retain(|l| l.reachable); + } + + if !results.is_empty() { + break results; + } + + // Daemon is still warming up — poll with feedback + if !response.ready { + if start.elapsed() >= max_wait { + eyre::bail!( + "Timed out waiting for daemon to finish probing devices. \ + The daemon may still be starting up — try again in a few seconds." + ); + } + if let Some(spinner) = spinner { + spinner.set_message("Waiting for daemon to finish probing devices..."); + } + tokio::time::sleep(poll_interval).await; + continue; + } + + // Daemon is ready but no results — this is a real "no devices" situation + eyre::bail!("No activated devices found"); + }; + + latencies.sort_by(|a, b| { + let reachable_cmp = b.reachable.cmp(&a.reachable); + if reachable_cmp != std::cmp::Ordering::Equal { + return reachable_cmp; + } + a.avg_latency_ns + .partial_cmp(&b.avg_latency_ns) + .unwrap_or(std::cmp::Ordering::Equal) + }); + + Ok(latencies) +} +``` + +- [ ] **Step 2: Update all test mock expectations to return `LatencyResponse`** + +In the test module, add the import: + +```rust +use crate::servicecontroller::LatencyResponse; +``` + +Then update every `expect_latency().returning(...)` call. Each test that does: + +```rust +controller + .expect_latency() + .returning(move || Ok(latencies.clone())); +``` + +Must become: + +```rust +controller + .expect_latency() + .returning(move || Ok(LatencyResponse { ready: true, results: latencies.clone() })); +``` + +Apply this to all tests: +- `test_retrieve_latencies_filters_and_sorts` +- `test_best_latency_prefers_current_within_tolerance` +- `test_best_latency_selects_lowest` +- `test_best_latency_ignores_unreachable_devices` +- `test_best_latency_ignores_faster_devices_at_max_users` +- `test_best_latency_current_faster_but_at_max_users` +- `test_best_latency_excludes_ips` +- `test_best_latency_excludes_specific_ip` +- `test_best_latency_device_with_multiple_endpoints_not_excluded` +- `test_best_latency_device_all_endpoints_excluded` +- `test_best_latency_prefers_same_device_with_available_endpoint` + +- [ ] **Step 3: Add a new test for the "not ready" polling behavior** + +```rust +#[tokio::test] +async fn test_retrieve_latencies_waits_for_daemon_ready() { + let (pk1, dev1) = make_device(DeviceStatus::Activated, 0); + let mut devices = HashMap::new(); + devices.insert(pk1, dev1); + + let latencies = vec![make_latency(&pk1.to_string(), 10000000, true)]; + let call_count = std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0)); + let call_count_clone = call_count.clone(); + let latencies_clone = latencies.clone(); + + let mut controller = MockServiceController::new(); + controller.expect_latency().returning(move || { + let count = call_count_clone.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + if count < 2 { + // First two calls: not ready yet + Ok(LatencyResponse { + ready: false, + results: vec![], + }) + } else { + // Third call: ready with results + Ok(LatencyResponse { + ready: true, + results: latencies_clone.clone(), + }) + } + }); + + let result = retrieve_latencies(&controller, &devices, false, None) + .await + .unwrap(); + assert_eq!(result.len(), 1); + assert_eq!(result[0].device_pk, pk1.to_string()); + assert!(call_count.load(std::sync::atomic::Ordering::SeqCst) >= 3); +} +``` + +- [ ] **Step 4: Add a test for "ready but no devices" error** + +```rust +#[tokio::test] +async fn test_retrieve_latencies_ready_but_empty_returns_error() { + let devices = HashMap::new(); + + let mut controller = MockServiceController::new(); + controller.expect_latency().returning(move || { + Ok(LatencyResponse { + ready: true, + results: vec![], + }) + }); + + let result = retrieve_latencies(&controller, &devices, false, None).await; + assert!(result.is_err()); + assert_eq!( + result.unwrap_err().to_string(), + "No activated devices found" + ); +} +``` + +- [ ] **Step 5: Run all Rust tests** + +Run: `cargo test -p doublezero` + +Expected: All tests pass. + +- [ ] **Step 6: Remove unused `backon` import** + +The `retrieve_latencies` function no longer uses `backon`'s retry logic. Remove the import from the top of `dzd_latency.rs`: + +```rust +// Remove this line: +use backon::{ExponentialBuilder, Retryable}; +``` + +Also add the `std::time::Instant` import (used by the new polling loop, `Instant` was not previously imported): + +```rust +use std::{collections::HashMap, net::Ipv4Addr, str::FromStr, time::Duration}; +``` + +Note: `std::time::Instant` is used as `std::time::Instant::now()` in the function body, so the fully qualified path is fine without a dedicated import. Alternatively add it to the existing `use std::` line. + +- [ ] **Step 7: Run formatting** + +Run: `make rust-fmt` + +- [ ] **Step 8: Run all Rust tests** + +Run: `cargo test -p doublezero` + +Expected: All tests pass. + +- [ ] **Step 9: Commit** + +```bash +git add client/doublezero/src/dzd_latency.rs +git commit -m "client: update retrieve_latencies to use daemon readiness flag" +``` + +--- + +### Task 6: Final verification + +**Files:** None (verification only) + +- [ ] **Step 1: Run full Go test suite for the daemon** + +Run: `go test -v -count=1 ./client/doublezerod/internal/latency/...` + +Expected: All tests pass. + +- [ ] **Step 2: Run full Rust test suite for the CLI** + +Run: `cargo test -p doublezero` + +Expected: All tests pass. + +- [ ] **Step 3: Run lint for both** + +Run: `make rust-lint` and `make go-lint` + +Expected: No lint errors. + +- [ ] **Step 4: Run formatting for both** + +Run: `make rust-fmt` and `make go-fmt` + +Expected: No formatting changes needed. diff --git a/docs/superpowers/plans/2026-04-03-status-multicast-groups.md b/docs/superpowers/plans/2026-04-03-status-multicast-groups.md new file mode 100644 index 0000000000..81bf55211d --- /dev/null +++ b/docs/superpowers/plans/2026-04-03-status-multicast-groups.md @@ -0,0 +1,419 @@ +# Status Command: Multicast Group Memberships — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Show which multicast groups a user publishes to / subscribes to in the `doublezero status` command output. + +**Architecture:** Extend the Go daemon's `V2ServiceStatus` with a `multicast_groups` object containing `publisher` and `subscriber` string arrays (group codes). The `enrichStatuses()` function already has access to the matched user and onchain multicast group data — it just needs to resolve the user's pubkey vecs to group codes. The Rust CLI then deserializes and displays the new field. + +**Tech Stack:** Go (daemon), Rust (CLI), serde, tabled + +--- + +## File Structure + +| Action | File | Responsibility | +|--------|------|---------------| +| Modify | `client/doublezerod/internal/manager/http.go` | Add `MulticastGroups` struct and field to `V2ServiceStatus`; populate in `enrichStatuses()` | +| Modify | `client/doublezerod/internal/manager/reconciler_test.go` | Add multicast group assertions to `TestServeV2Status_Enrichment` | +| Modify | `client/doublezero/src/servicecontroller.rs` | Add `MulticastGroups` struct and field to `V2ServiceStatus` | +| Modify | `client/doublezero/src/command/status.rs` | Add multicast groups column to display; format as `P:code,S:code` | + +--- + +### Task 1: Go daemon — add MulticastGroups to V2ServiceStatus and populate in enrichStatuses + +**Files:** +- Modify: `client/doublezerod/internal/manager/http.go:17-26` (struct definitions) +- Modify: `client/doublezerod/internal/manager/http.go:226-367` (enrichStatuses) +- Modify: `client/doublezerod/internal/manager/reconciler_test.go:1205-1397` (TestServeV2Status_Enrichment) + +- [ ] **Step 1: Write the failing test** + +In `client/doublezerod/internal/manager/reconciler_test.go`, update the `wantService` struct and test cases in `TestServeV2Status_Enrichment` to assert on multicast groups. + +First, add `Code` to the existing `mcastGroup` fixture (around line 1218): + +```go +mcastGroup := serviceability.MulticastGroup{ + PubKey: mcastGroupPK, + MulticastIp: [4]uint8{239, 0, 0, 1}, + Code: "solana-ams", +} +``` + +Then add fields to the `wantService` struct (around line 1223): + +```go +type wantService struct { + userType string + currentDevice string + metro string + tenant string + hasDzIP bool + pubGroups []string + subGroups []string +} +``` + +Update each test case's `want` entries. For `ibrl_only`: +```go +want: []wantService{ + {userType: "IBRL", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: nil, subGroups: nil}, +}, +``` + +For `multicast_publisher`: +```go +want: []wantService{ + {userType: "Multicast", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: []string{"solana-ams"}, subGroups: nil}, +}, +``` + +For `multicast_subscriber`: +```go +want: []wantService{ + {userType: "Multicast", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: false, pubGroups: nil, subGroups: []string{"solana-ams"}}, +}, +``` + +For `ibrl_plus_multicast_subscriber`: +```go +want: []wantService{ + {userType: "IBRL", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: nil, subGroups: nil}, + {userType: "Multicast", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: false, pubGroups: nil, subGroups: []string{"solana-ams"}}, +}, +``` + +For `ibrl_plus_multicast_publisher`: +```go +want: []wantService{ + {userType: "IBRL", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: nil, subGroups: nil}, + {userType: "Multicast", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: []string{"solana-ams"}, subGroups: nil}, +}, +``` + +Add assertions at the end of the test loop (after the existing `hasDzIP` check, around line 1394): + +```go +if !slices.Equal(svc.MulticastGroups.Publisher, w.pubGroups) { + t.Errorf("[%s] expected pub groups %v, got %v", w.userType, w.pubGroups, svc.MulticastGroups.Publisher) +} +if !slices.Equal(svc.MulticastGroups.Subscriber, w.subGroups) { + t.Errorf("[%s] expected sub groups %v, got %v", w.userType, w.subGroups, svc.MulticastGroups.Subscriber) +} +``` + +Add `"slices"` to the import block if not already present. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd /Users/ben/src/malbec/doublezero && go test -run TestServeV2Status_Enrichment -v ./client/doublezerod/internal/manager/...` + +Expected: Compilation error — `V2ServiceStatus` has no `MulticastGroups` field. + +- [ ] **Step 3: Implement the Go daemon changes** + +In `client/doublezerod/internal/manager/http.go`, add the `MulticastGroups` struct and field. + +After the existing `V2ServiceStatus` struct (line 26), add: + +```go +// MulticastGroups contains the group codes a user publishes to and subscribes to. +type MulticastGroups struct { + Publisher []string `json:"publisher"` + Subscriber []string `json:"subscriber"` +} +``` + +Add the field to `V2ServiceStatus`: + +```go +type V2ServiceStatus struct { + *api.StatusResponse + CurrentDevice string `json:"current_device"` + CurrentDeviceRttNanoseconds int64 `json:"current_device_rtt_nanoseconds,omitempty"` + CurrentDeviceLossPercentage float64 `json:"current_device_loss_percentage,omitempty"` + LowestLatencyDevice string `json:"lowest_latency_device"` + Metro string `json:"metro"` + Tenant string `json:"tenant"` + MulticastGroups MulticastGroups `json:"multicast_groups"` +} +``` + +In `enrichStatuses()`, build a multicast group lookup map. After the existing `tenantsByPK` map (around line 266), add: + +```go +mcastGroupsByPK := make(map[[32]byte]serviceability.MulticastGroup, len(data.MulticastGroups)) +for _, mg := range data.MulticastGroups { + mcastGroupsByPK[mg.PubKey] = mg +} +``` + +After the tenant enrichment block (after line 356, before the lowest latency computation), add: + +```go +if matchedUser != nil { + for _, pk := range matchedUser.Publishers { + if mg, ok := mcastGroupsByPK[pk]; ok { + es.MulticastGroups.Publisher = append(es.MulticastGroups.Publisher, mg.Code) + } + } + for _, pk := range matchedUser.Subscribers { + if mg, ok := mcastGroupsByPK[pk]; ok { + es.MulticastGroups.Subscriber = append(es.MulticastGroups.Subscriber, mg.Code) + } + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd /Users/ben/src/malbec/doublezero && go test -run TestServeV2Status_Enrichment -v ./client/doublezerod/internal/manager/...` + +Expected: PASS + +- [ ] **Step 5: Run full Go test suite for the package** + +Run: `cd /Users/ben/src/malbec/doublezero && go test -v ./client/doublezerod/internal/manager/...` + +Expected: All tests pass. The new field serializes as empty arrays `{"publisher":null,"subscriber":null}` for non-multicast users, which is fine — the Rust side uses `#[serde(default)]`. + +- [ ] **Step 6: Commit** + +``` +client/daemon: add multicast groups to v2 status response +``` + +--- + +### Task 2: Rust CLI — deserialize and display multicast groups + +**Files:** +- Modify: `client/doublezero/src/servicecontroller.rs:149-161` (V2ServiceStatus struct) +- Modify: `client/doublezero/src/command/status.rs:20-36` (AppendedStatusResponse struct) +- Modify: `client/doublezero/src/command/status.rs:51-131` (command_impl) + +- [ ] **Step 1: Write the failing test** + +In `client/doublezero/src/command/status.rs`, add a new test for multicast group display. Add after the existing `test_status_command_multicast_subscriber` test (after line 354): + +```rust +#[tokio::test] +async fn test_status_command_multicast_groups_display() { + let mock_command = MockCliCommand::new(); + let mut mock_controller = MockServiceController::new(); + + mock_controller.expect_v2_status().returning(|| { + Ok(V2StatusResponse { + reconciler_enabled: true, + client_ip: String::new(), + network: "testnet".to_string(), + services: vec![V2ServiceStatus { + status: StatusResponse { + doublezero_status: DoubleZeroStatus { + session_status: "BGP Session Up".to_string(), + last_session_update: Some(1625247600), + }, + tunnel_name: Some("doublezero1".to_string()), + tunnel_src: Some("10.10.10.10".to_string()), + tunnel_dst: Some("5.6.7.8".to_string()), + doublezero_ip: None, + user_type: Some("Multicast".to_string()), + }, + current_device: "device1".to_string(), + lowest_latency_device: "device1".to_string(), + metro: "metro".to_string(), + tenant: String::new(), + multicast_groups: MulticastGroups { + publisher: vec!["solana-lv".to_string()], + subscriber: vec!["solana-ams".to_string()], + }, + }], + }) + }); + + let result = StatusCliCommand { json: true } + .command_impl(&mock_command, &mock_controller) + .await; + + assert!(result.is_ok()); + let result = result.unwrap(); + assert_eq!(result.len(), 1); + assert_eq!(result[0].multicast_groups, "P:solana-lv,S:solana-ams"); +} +``` + +Also add a test for backward compatibility (daemon doesn't send the field): + +```rust +#[test] +fn test_multicast_groups_serde_default() { + let json = r#"{ + "doublezero_status": {"session_status": "BGP Session Up", "last_session_update": null}, + "tunnel_name": null, "tunnel_src": null, "tunnel_dst": null, + "doublezero_ip": null, "user_type": "IBRL", + "current_device": "dz1", "lowest_latency_device": "dz1", + "metro": "ams", "tenant": "" + }"#; + let svc: V2ServiceStatus = serde_json::from_str(json).unwrap(); + assert!(svc.multicast_groups.publisher.is_empty()); + assert!(svc.multicast_groups.subscriber.is_empty()); +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd /Users/ben/src/malbec/doublezero && cargo test -p doublezero test_status_command_multicast_groups_display` + +Expected: Compilation error — `MulticastGroups` type doesn't exist, `V2ServiceStatus` has no `multicast_groups` field. + +- [ ] **Step 3: Add MulticastGroups struct to servicecontroller.rs** + +In `client/doublezero/src/servicecontroller.rs`, add the struct before `V2ServiceStatus` (before line 149): + +```rust +#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq)] +pub struct MulticastGroups { + #[serde(default)] + pub publisher: Vec, + #[serde(default)] + pub subscriber: Vec, +} +``` + +Add the field to `V2ServiceStatus` (after the `tenant` field): + +```rust +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct V2ServiceStatus { + #[serde(flatten)] + pub status: StatusResponse, + #[serde(default)] + pub current_device: String, + #[serde(default)] + pub lowest_latency_device: String, + #[serde(default)] + pub metro: String, + #[serde(default)] + pub tenant: String, + #[serde(default)] + pub multicast_groups: MulticastGroups, +} +``` + +- [ ] **Step 4: Add multicast_groups to AppendedStatusResponse and format in command_impl** + +In `client/doublezero/src/command/status.rs`, add the field to `AppendedStatusResponse`: + +```rust +#[derive(Tabled, Debug, Deserialize, Serialize)] +struct AppendedStatusResponse { + #[tabled(inline)] + response: StatusResponse, + #[tabled(rename = "Reconciler")] + reconciler_enabled: bool, + #[tabled(rename = "Tenant")] + tenant: String, + #[tabled(rename = "Current Device")] + current_device: String, + #[tabled(rename = "Lowest Latency Device")] + lowest_latency_device: String, + #[tabled(rename = "Metro")] + metro: String, + #[tabled(rename = "Network")] + network: String, + #[tabled(rename = "Multicast Groups")] + multicast_groups: String, +} +``` + +Add `use crate::servicecontroller::MulticastGroups;` to the imports at the top of the file (add to the existing `use crate::servicecontroller::{...}` import). + +Add a helper function to format multicast groups (before the `impl StatusCliCommand` block): + +```rust +fn format_multicast_groups(groups: &MulticastGroups) -> String { + let mut parts = Vec::new(); + for code in &groups.publisher { + parts.push(format!("P:{code}")); + } + for code in &groups.subscriber { + parts.push(format!("S:{code}")); + } + parts.join(",") +} +``` + +In `command_impl`, populate the new field when building each `AppendedStatusResponse`. In the empty-services branch (around line 62), add `multicast_groups: String::new()` to the struct literal. + +In the main loop (around line 119), add the field: + +```rust +responses.push(AppendedStatusResponse { + response: svc.status.clone(), + reconciler_enabled: v2_status.reconciler_enabled, + current_device, + lowest_latency_device, + metro, + network: network.clone(), + tenant: svc.tenant.clone(), + multicast_groups: format_multicast_groups(&svc.multicast_groups), +}); +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `cd /Users/ben/src/malbec/doublezero && cargo test -p doublezero test_status_command_multicast_groups_display test_multicast_groups_serde_default` + +Expected: PASS + +- [ ] **Step 6: Fix existing tests** + +The existing tests in `status.rs` need the new `multicast_groups` field added to `V2ServiceStatus` struct literals and `AppendedStatusResponse` assertions. For each existing test that constructs a `V2ServiceStatus`, add: + +```rust +multicast_groups: MulticastGroups::default(), +``` + +For tests using the `make_v2_service` helper, update the helper to include the field: + +```rust +fn make_v2_service( + // ... existing params ... +) -> V2ServiceStatus { + V2ServiceStatus { + status: StatusResponse { ... }, + current_device: current_device.to_string(), + lowest_latency_device: lowest_latency_device.to_string(), + metro: metro.to_string(), + tenant: tenant.to_string(), + multicast_groups: MulticastGroups::default(), + } +} +``` + +For tests that assert on `AppendedStatusResponse` fields (like `test_status_json_output_format`), add `multicast_groups: String::new()` to the struct literal and add a JSON field assertion: + +```rust +assert!( + status.get("multicast_groups").is_some(), + "Missing 'multicast_groups' field" +); +``` + +- [ ] **Step 7: Run full test suite** + +Run: `cd /Users/ben/src/malbec/doublezero && cargo test -p doublezero` + +Expected: All tests pass. + +- [ ] **Step 8: Format** + +Run: `cd /Users/ben/src/malbec/doublezero && make rust-fmt` + +- [ ] **Step 9: Commit** + +``` +client/cli: show multicast groups in status command +``` diff --git a/docs/superpowers/plans/2026-04-09-startup-tunnel-endpoint-registration.md b/docs/superpowers/plans/2026-04-09-startup-tunnel-endpoint-registration.md new file mode 100644 index 0000000000..1c74d7562c --- /dev/null +++ b/docs/superpowers/plans/2026-04-09-startup-tunnel-endpoint-registration.md @@ -0,0 +1,185 @@ +# Fix Startup Tunnel Endpoint Registration + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Prevent duplicate GRE tunnel pairs by registering implicit tunnel endpoints (device `public_ip`) during activator startup reconstruction. + +**Architecture:** The activator's `reserve_user_allocations` rebuilds in-memory state from existing onchain users at startup. It currently only registers explicit `tunnel_endpoint` values, missing users that fall back to the device's `public_ip`. This causes `get_available_tunnel_endpoint` to hand out already-in-use endpoints when a second user with the same `client_ip` arrives on the same device. + +**Tech Stack:** Rust (activator crate) + +--- + +### Task 1: Add regression test for startup tunnel endpoint registration + +**Files:** +- Modify: `activator/src/processor.rs:534-629` (test module, after existing `test_updating_user_allocations_must_be_reserved_at_startup`) + +- [ ] **Step 1: Write failing test** + +Add a test that creates two users with the same `client_ip` on the same device — the first with `tunnel_endpoint = UNSPECIFIED` (simulating an existing activated user that uses the device's `public_ip` implicitly). After `reserve_user_allocations`, call `get_available_tunnel_endpoint` for that `client_ip` and assert it does NOT return the device's `public_ip` (since it's already in use). + +```rust +/// Regression test: reserve_user_allocations must register implicit tunnel endpoints +/// (device public_ip) for users with unspecified tunnel_endpoint. Otherwise a second +/// user from the same client_ip gets the same endpoint, creating a duplicate GRE tunnel pair. +#[test] +fn test_implicit_tunnel_endpoint_reserved_at_startup() { + use crate::{ipblockallocator::IPBlockAllocator, states::devicestate::DeviceState}; + use doublezero_sdk::{ + AccountType, Device, DeviceStatus, DeviceType, User, UserStatus, UserType, + }; + use doublezero_serviceability::state::user::UserCYOA; + use std::net::Ipv4Addr; + + let device_pubkey = Pubkey::new_unique(); + let device = Device { + account_type: AccountType::Device, + owner: Pubkey::new_unique(), + index: 0, + reference_count: 0, + bump_seed: 0, + contributor_pk: Pubkey::new_unique(), + location_pk: Pubkey::new_unique(), + exchange_pk: Pubkey::new_unique(), + device_type: DeviceType::Hybrid, + public_ip: [88, 216, 220, 195].into(), + status: DeviceStatus::Activated, + metrics_publisher_pk: Pubkey::default(), + code: "TestDevice".to_string(), + dz_prefixes: "10.0.0.0/24".parse().unwrap(), + mgmt_vrf: "default".to_string(), + interfaces: vec![], + max_users: 255, + users_count: 0, + device_health: doublezero_serviceability::state::device::DeviceHealth::ReadyForUsers, + desired_status: + doublezero_serviceability::state::device::DeviceDesiredStatus::Activated, + unicast_users_count: 0, + multicast_subscribers_count: 0, + max_unicast_users: 0, + max_multicast_subscribers: 0, + reserved_seats: 0, + multicast_publishers_count: 0, + max_multicast_publishers: 0, + }; + + let client_ip: Ipv4Addr = [64, 130, 32, 201].into(); + + let mut device_map: DeviceMap = DeviceMap::new(); + device_map.insert(device_pubkey, DeviceState::new(&device)); + + // Existing activated user with unspecified tunnel_endpoint (uses device public_ip implicitly) + let existing_user = User { + account_type: AccountType::User, + owner: Pubkey::new_unique(), + index: 0, + bump_seed: 0, + user_type: UserType::IBRL, + tenant_pk: Pubkey::new_unique(), + device_pk: device_pubkey, + cyoa_type: UserCYOA::GREOverDIA, + client_ip, + dz_ip: [10, 0, 0, 1].into(), + tunnel_id: 509, + tunnel_net: "169.254.2.14/31".parse().unwrap(), + status: UserStatus::Activated, + publishers: vec![], + subscribers: vec![], + validator_pubkey: Pubkey::default(), + tunnel_endpoint: Ipv4Addr::UNSPECIFIED, + tunnel_flags: 0, + bgp_status: Default::default(), + last_bgp_up_at: 0, + last_bgp_reported_at: 0, + }; + + let mut users: HashMap = HashMap::new(); + users.insert(Pubkey::new_unique(), existing_user); + + let mut user_tunnel_ips = IPBlockAllocator::new("169.254.0.0/16".parse().unwrap()); + let mut publisher_dz_ips = IPBlockAllocator::new("148.51.120.0/21".parse().unwrap()); + + reserve_user_allocations(&users, &mut device_map, &mut user_tunnel_ips, &mut publisher_dz_ips) + .expect("reserve_user_allocations should succeed"); + + // A second user from the same client_ip should NOT get device public_ip + // because the first user is already using it + let device_state = device_map.get(&device_pubkey).unwrap(); + let next_endpoint = device_state.get_available_tunnel_endpoint(client_ip); + assert_eq!( + next_endpoint, None, + "BUG: get_available_tunnel_endpoint returned device public_ip which is already \ + in use by existing user with unspecified tunnel_endpoint — this creates a \ + duplicate GRE tunnel pair" + ); +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cargo test -p doublezero-activator test_implicit_tunnel_endpoint_reserved_at_startup` + +Expected: FAIL — the assertion fails because `get_available_tunnel_endpoint` returns `Some(88.216.220.195)` (the device's `public_ip`), since `reserve_user_allocations` didn't register it. + +### Task 2: Fix startup registration to include implicit endpoints + +**Files:** +- Modify: `activator/src/processor.rs:146-149` + +- [ ] **Step 3: Implement the fix** + +In `reserve_user_allocations`, change the tunnel endpoint registration to always register the effective endpoint — either the explicit `tunnel_endpoint` or the device's `public_ip`: + +```rust + // Register tunnel endpoint (explicit or implicit via device public_ip) + // so that get_available_tunnel_endpoint knows what's already in use. + let effective_endpoint = if user.has_tunnel_endpoint() { + user.tunnel_endpoint + } else { + device_state.device.public_ip + }; + device_state.register_tunnel_endpoint(user.client_ip, effective_endpoint); +``` + +This replaces the current code at lines 146-149: +```rust + // Register tunnel endpoint if set + if user.has_tunnel_endpoint() { + device_state.register_tunnel_endpoint(user.client_ip, user.tunnel_endpoint); + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cargo test -p doublezero-activator test_implicit_tunnel_endpoint_reserved_at_startup` + +Expected: PASS + +- [ ] **Step 5: Run full activator test suite** + +Run: `cargo test -p doublezero-activator` + +Expected: All tests pass. + +- [ ] **Step 6: Run formatting and lint** + +Run: `make rust-fmt && make rust-lint` + +Expected: Clean. + +- [ ] **Step 7: Commit** + +```bash +git add activator/src/processor.rs +git commit -m "activator: register implicit tunnel endpoints at startup + +When rebuilding in-memory state from existing onchain users, +register the device's public_ip as the effective tunnel endpoint +for users with unspecified tunnel_endpoint. Previously only +explicit tunnel endpoints were registered, allowing +get_available_tunnel_endpoint to hand out already-in-use +endpoints when a second user with the same client_ip arrived +on the same device — creating duplicate GRE tunnel pairs that +the controller had to deduplicate by dropping one tunnel." +``` diff --git a/docs/superpowers/plans/2026-04-23-multicast-modify-without-disconnect.md b/docs/superpowers/plans/2026-04-23-multicast-modify-without-disconnect.md new file mode 100644 index 0000000000..ec04c4df1d --- /dev/null +++ b/docs/superpowers/plans/2026-04-23-multicast-modify-without-disconnect.md @@ -0,0 +1,1476 @@ +# Multicast: modify subscriptions without disconnecting — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add four CLI subcommands under `doublezero multicast` — `subscribe`, `unsubscribe`, `publish`, `unpublish` — so a connected user can modify their multicast role set without running `disconnect`. + +**Architecture:** Extend the client CLI only. Each verb resolves group codes to pubkeys, loads the caller's existing Multicast user, and issues one `UpdateMulticastGroupRolesCommand` per group with the correct boolean flags (carrying through the other role's current value). The smartcontract is unchanged; the daemon reconciles multicast routes asynchronously on its next poll. + +**Tech Stack:** Rust (clap, eyre, mockall tests), Go (e2e tests via testcontainers + cEOS). + +**Spec:** `docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md` + +--- + +## File Structure + +| Action | File | Responsibility | +|--------|------|---------------| +| Modify | `client/doublezero/src/cli/multicast.rs` | Add four new `MulticastCommands` variants + arg structs | +| Create | `client/doublezero/src/command/multicast.rs` | Handlers for the four verbs + shared helpers | +| Modify | `client/doublezero/src/command/mod.rs` | Register new `multicast` module | +| Modify | `client/doublezero/src/main.rs` | Dispatch new variants under `Command::Multicast` | +| Modify | `e2e/main_test.go` | Add `TestDevnet` helpers for the four new verbs | +| Modify | `e2e/multicast_test.go` | Extend `TestE2E_Multicast` with unsubscribe/unpublish sub-tests | + +--- + +### Task 1: Extend `MulticastCommands` with four new variants + +**Files:** +- Modify: `client/doublezero/src/cli/multicast.rs` (entire file is 16 lines) + +- [ ] **Step 1: Replace the file with the expanded enum** + +Overwrite `client/doublezero/src/cli/multicast.rs` with: + +```rust +use clap::{Args, Subcommand}; + +use super::multicastgroup::MulticastGroupCliCommand; + +#[derive(Args, Debug)] +pub struct MulticastCliCommand { + #[command(subcommand)] + pub command: MulticastCommands, +} + +#[derive(Debug, Subcommand)] +pub enum MulticastCommands { + /// Manage multicast groups + #[clap()] + Group(MulticastGroupCliCommand), + /// Subscribe to one or more multicast groups (user must already be connected) + #[clap()] + Subscribe(MulticastSubscribeCliCommand), + /// Unsubscribe from one or more multicast groups + #[clap()] + Unsubscribe(MulticastUnsubscribeCliCommand), + /// Publish to one or more multicast groups (user must already be connected) + #[clap()] + Publish(MulticastPublishCliCommand), + /// Stop publishing to one or more multicast groups + #[clap()] + Unpublish(MulticastUnpublishCliCommand), +} + +#[derive(Args, Debug)] +pub struct MulticastSubscribeCliCommand { + /// Multicast group code(s) to subscribe to + #[arg(num_args = 1..)] + pub groups: Vec, +} + +#[derive(Args, Debug)] +pub struct MulticastUnsubscribeCliCommand { + /// Multicast group code(s) to unsubscribe from + #[arg(num_args = 1..)] + pub groups: Vec, +} + +#[derive(Args, Debug)] +pub struct MulticastPublishCliCommand { + /// Multicast group code(s) to publish to + #[arg(num_args = 1..)] + pub groups: Vec, +} + +#[derive(Args, Debug)] +pub struct MulticastUnpublishCliCommand { + /// Multicast group code(s) to stop publishing to + #[arg(num_args = 1..)] + pub groups: Vec, +} +``` + +- [ ] **Step 2: Verify clap parsing compiles and `--help` lists the new verbs** + +Run: + +```bash +cargo check -p doublezero +``` + +Expected: compiles (may warn about unused `MulticastSubscribeCliCommand` etc. until Task 7 — that's fine). + +- [ ] **Step 3: Commit** + +```bash +git add client/doublezero/src/cli/multicast.rs +git commit -m "client/doublezero: add multicast subscribe/unsubscribe/publish/unpublish CLI variants" +``` + +--- + +### Task 2: Create command module skeleton with shared helpers + +**Files:** +- Create: `client/doublezero/src/command/multicast.rs` +- Modify: `client/doublezero/src/command/mod.rs` + +- [ ] **Step 1: Add module registration** + +Edit `client/doublezero/src/command/mod.rs` to add `pub mod multicast;` alphabetically. The file should then read: + +```rust +pub mod connect; +pub mod disable; +pub mod disconnect; +pub mod enable; +pub mod helpers; +pub mod latency; +pub mod multicast; +pub mod routes; +pub mod status; +pub mod util; +``` + +- [ ] **Step 2: Write the failing tests** + +Create `client/doublezero/src/command/multicast.rs`: + +```rust +use std::net::Ipv4Addr; + +use doublezero_cli::doublezerocommand::CliCommand; +use doublezero_sdk::{ + commands::{ + multicastgroup::list::ListMulticastGroupCommand, user::list::ListUserCommand, + }, + User, UserType, +}; +use solana_sdk::pubkey::Pubkey; + +/// Resolve a list of multicast group codes to their on-chain pubkeys. +/// Errors on any unknown code, with no onchain writes. +pub(super) fn resolve_groups( + client: &dyn CliCommand, + codes: &[String], +) -> eyre::Result> { + let mcast_groups = client.list_multicastgroup(ListMulticastGroupCommand)?; + let mut out = Vec::with_capacity(codes.len()); + for code in codes { + let (pk, _) = mcast_groups + .iter() + .find(|(_, g)| g.code == *code) + .ok_or_else(|| eyre::eyre!("Multicast group not found: {code}"))?; + out.push((code.clone(), *pk)); + } + Ok(out) +} + +/// Load the Multicast user for the given client_ip. Errors if none exists. +pub(super) fn load_multicast_user( + client: &dyn CliCommand, + client_ip: Ipv4Addr, +) -> eyre::Result<(Pubkey, User)> { + let users = client.list_user(ListUserCommand)?; + users + .into_iter() + .find(|(_, u)| u.client_ip == client_ip && u.user_type == UserType::Multicast) + .ok_or_else(|| { + eyre::eyre!( + "No active multicast user for {client_ip}. \ + Run 'doublezero connect Multicast --publish/--subscribe ' first." + ) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use doublezero_cli::tests::utils::create_test_client; + use doublezero_sdk::{AccountType, MulticastGroup, MulticastGroupStatus, User, UserCYOA, UserStatus}; + use std::collections::HashMap; + + fn make_user(client_ip: Ipv4Addr, user_type: UserType) -> User { + User { + account_type: AccountType::User, + owner: Pubkey::new_unique(), + index: 0, + bump_seed: 0, + user_type, + tenant_pk: Pubkey::default(), + device_pk: Pubkey::default(), + cyoa_type: UserCYOA::None, + client_ip, + dz_ip: Ipv4Addr::UNSPECIFIED, + tunnel_id: 0, + tunnel_net: Default::default(), + status: UserStatus::Activated, + publishers: vec![], + subscribers: vec![], + validator_pubkey: Pubkey::default(), + tunnel_endpoint: Ipv4Addr::UNSPECIFIED, + tunnel_flags: 0, + bgp_status: Default::default(), + last_bgp_up_at: 0, + last_bgp_reported_at: 0, + } + } + + fn make_group(code: &str) -> MulticastGroup { + MulticastGroup { + account_type: AccountType::MulticastGroup, + owner: Pubkey::default(), + index: 0, + bump_seed: 0, + tenant_pk: Pubkey::default(), + code: code.to_string(), + max_bandwidth: 0, + status: MulticastGroupStatus::Activated, + multicast_ip: Ipv4Addr::UNSPECIFIED, + publisher_count: 0, + subscriber_count: 0, + } + } + + #[test] + fn resolve_groups_returns_pubkeys_in_order() { + let mut client = create_test_client(); + let g1_pk = Pubkey::new_unique(); + let g2_pk = Pubkey::new_unique(); + let mut groups = HashMap::new(); + groups.insert(g1_pk, make_group("g1")); + groups.insert(g2_pk, make_group("g2")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + let out = resolve_groups(&client, &["g2".into(), "g1".into()]).unwrap(); + assert_eq!(out, vec![("g2".into(), g2_pk), ("g1".into(), g1_pk)]); + } + + #[test] + fn resolve_groups_errors_on_unknown_code() { + let mut client = create_test_client(); + let g1_pk = Pubkey::new_unique(); + let mut groups = HashMap::new(); + groups.insert(g1_pk, make_group("g1")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + let err = resolve_groups(&client, &["nope".into()]).unwrap_err(); + assert!( + err.to_string().contains("Multicast group not found: nope"), + "unexpected error: {err}" + ); + } + + #[test] + fn load_multicast_user_finds_user_for_client_ip() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let mut client = create_test_client(); + let user_pk = Pubkey::new_unique(); + let user = make_user(ip, UserType::Multicast); + let mut users = HashMap::new(); + users.insert(user_pk, user.clone()); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let (pk, loaded) = load_multicast_user(&client, ip).unwrap(); + assert_eq!(pk, user_pk); + assert_eq!(loaded.client_ip, ip); + assert_eq!(loaded.user_type, UserType::Multicast); + } + + #[test] + fn load_multicast_user_errors_when_only_ibrl_user_exists() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let mut client = create_test_client(); + let mut users = HashMap::new(); + users.insert(Pubkey::new_unique(), make_user(ip, UserType::IBRL)); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let err = load_multicast_user(&client, ip).unwrap_err(); + assert!( + err.to_string().contains("No active multicast user"), + "unexpected error: {err}" + ); + } + + #[test] + fn load_multicast_user_errors_when_no_user_for_this_ip() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let other_ip = Ipv4Addr::new(10, 0, 0, 2); + let mut client = create_test_client(); + let mut users = HashMap::new(); + users.insert(Pubkey::new_unique(), make_user(other_ip, UserType::Multicast)); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let err = load_multicast_user(&client, ip).unwrap_err(); + assert!(err.to_string().contains("No active multicast user")); + } +} +``` + +- [ ] **Step 3: Run tests to verify they fail** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast +``` + +Expected: tests in this new module compile and run; they should pass immediately because the helpers are implemented. If any fail, fix before committing. + +- [ ] **Step 4: Run lint to confirm clean** + +Run: + +```bash +make rust-fmt && cargo clippy -p doublezero --all-targets -- -D warnings +``` + +Expected: no warnings/errors in the new file. + +- [ ] **Step 5: Commit** + +```bash +git add client/doublezero/src/command/mod.rs client/doublezero/src/command/multicast.rs +git commit -m "client/doublezero: add multicast command module with resolve_groups and load_multicast_user helpers" +``` + +--- + +### Task 3: Implement `unsubscribe` handler + +**Files:** +- Modify: `client/doublezero/src/command/multicast.rs` + +- [ ] **Step 1: Write the failing tests** + +Append inside the existing `#[cfg(test)] mod tests { ... }` block in `client/doublezero/src/command/multicast.rs`: + +```rust + // --- MulticastUnsubscribeCliCommand tests --- + + use crate::cli::multicast::MulticastUnsubscribeCliCommand; + use doublezero_sdk::commands::multicastgroup::subscribe::UpdateMulticastGroupRolesCommand; + + fn user_with_roles( + ip: Ipv4Addr, + publishers: Vec, + subscribers: Vec, + ) -> User { + let mut u = make_user(ip, UserType::Multicast); + u.publishers = publishers; + u.subscribers = subscribers; + u + } + + #[tokio::test] + async fn unsubscribe_removes_subscriber_role_and_preserves_publisher_role() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g_pk = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + // User is BOTH publisher and subscriber of g — unsubscribe must keep publisher=true. + let user = user_with_roles(ip, vec![g_pk], vec![g_pk]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g_pk, make_group("g")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client + .expect_update_multicastgroup_roles() + .withf(move |cmd: &UpdateMulticastGroupRolesCommand| { + cmd.user_pk == user_pk + && cmd.group_pk == g_pk + && cmd.client_ip == ip + && cmd.publisher // carry-through preserved + && !cmd.subscriber + }) + .once() + .returning(|_| Ok(solana_sdk::signature::Signature::default())); + + let cmd = MulticastUnsubscribeCliCommand { + groups: vec!["g".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } + + #[tokio::test] + async fn unsubscribe_skips_group_user_is_not_subscribed_to() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g_pk = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + // User has no sub roles — should no-op without an onchain call. + let user = user_with_roles(ip, vec![], vec![]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g_pk, make_group("g")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client.expect_update_multicastgroup_roles().never(); + + let cmd = MulticastUnsubscribeCliCommand { + groups: vec!["g".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } + + #[tokio::test] + async fn unsubscribe_errors_when_user_missing() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let mut client = create_test_client(); + client.expect_list_user().returning(|_| Ok(HashMap::new())); + + let cmd = MulticastUnsubscribeCliCommand { + groups: vec!["g".into()], + }; + let err = cmd.execute_inner(&client, ip).await.unwrap_err(); + assert!(err.to_string().contains("No active multicast user")); + } + + #[tokio::test] + async fn unsubscribe_errors_on_unknown_group_before_any_onchain_call() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let user_pk = Pubkey::new_unique(); + let mut client = create_test_client(); + + let user = user_with_roles(ip, vec![], vec![]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + client + .expect_list_multicastgroup() + .returning(|_| Ok(HashMap::new())); + client.expect_update_multicastgroup_roles().never(); + + let cmd = MulticastUnsubscribeCliCommand { + groups: vec!["unknown".into()], + }; + let err = cmd.execute_inner(&client, ip).await.unwrap_err(); + assert!(err.to_string().contains("Multicast group not found: unknown")); + } +``` + +- [ ] **Step 2: Run tests to verify they fail with "no method named `execute_inner`"** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast::tests::unsubscribe +``` + +Expected: compile error — `MulticastUnsubscribeCliCommand::execute_inner` does not exist. + +- [ ] **Step 3: Implement the handler** + +Append to `client/doublezero/src/command/multicast.rs` (above the `#[cfg(test)]` block): + +```rust +use doublezero_sdk::commands::multicastgroup::subscribe::UpdateMulticastGroupRolesCommand; + +use crate::{ + cli::multicast::MulticastUnsubscribeCliCommand, servicecontroller::ServiceControllerImpl, +}; +use doublezero_cli::helpers::init_command; +use indicatif::ProgressBar; + +impl MulticastUnsubscribeCliCommand { + pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { + let controller = ServiceControllerImpl::new(None); + let client_ip = crate::command::helpers::resolve_client_ip(&controller).await?; + self.execute_inner(client, client_ip).await + } + + /// Testable core: takes an already-resolved client_ip. + async fn execute_inner( + self, + client: &dyn CliCommand, + client_ip: Ipv4Addr, + ) -> eyre::Result<()> { + let spinner = init_command(2); + spinner.println(format!("⚡ Unsubscribing (client_ip: {client_ip})...")); + + let (user_pk, user) = load_multicast_user(client, client_ip)?; + let groups = resolve_groups(client, &self.groups)?; + spinner.inc(1); + + for (code, group_pk) in groups { + if !user.subscribers.contains(&group_pk) { + spinner.println(format!(" not subscribed to {code} — skipping")); + continue; + } + let carry_pub = user.publishers.contains(&group_pk); + client.update_multicastgroup_roles(UpdateMulticastGroupRolesCommand { + user_pk, + group_pk, + client_ip, + publisher: carry_pub, + subscriber: false, + })?; + spinner.println(format!(" unsubscribed from {code}")); + } + + finish_update(&spinner); + Ok(()) + } +} + +fn finish_update(spinner: &ProgressBar) { + spinner.println("✅ Updated. Routes will adjust shortly."); + spinner.finish_and_clear(); +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast::tests::unsubscribe +``` + +Expected: all four unsubscribe tests PASS. + +- [ ] **Step 5: Lint clean** + +Run: + +```bash +make rust-fmt && cargo clippy -p doublezero --all-targets -- -D warnings +``` + +Expected: no warnings. + +- [ ] **Step 6: Commit** + +```bash +git add client/doublezero/src/command/multicast.rs +git commit -m "client/doublezero: implement multicast unsubscribe command" +``` + +--- + +### Task 4: Implement `unpublish` handler with last-publisher warning + +**Files:** +- Modify: `client/doublezero/src/command/multicast.rs` + +- [ ] **Step 1: Write the failing tests** + +Append inside the existing `tests` module: + +```rust + // --- MulticastUnpublishCliCommand tests --- + + use crate::cli::multicast::MulticastUnpublishCliCommand; + + #[tokio::test] + async fn unpublish_removes_publisher_role_and_preserves_subscriber_role() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g1 = Pubkey::new_unique(); + let g2 = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + // Publisher of g1 & g2, subscriber of g1. Unpublish g1 must keep subscriber=true. + let user = user_with_roles(ip, vec![g1, g2], vec![g1]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g1, make_group("g1")); + groups.insert(g2, make_group("g2")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client + .expect_update_multicastgroup_roles() + .withf(move |cmd: &UpdateMulticastGroupRolesCommand| { + cmd.user_pk == user_pk + && cmd.group_pk == g1 + && !cmd.publisher + && cmd.subscriber // carry-through preserved + }) + .once() + .returning(|_| Ok(solana_sdk::signature::Signature::default())); + + let cmd = MulticastUnpublishCliCommand { + groups: vec!["g1".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } + + #[tokio::test] + async fn unpublish_skips_group_user_is_not_publishing_to() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g_pk = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + let user = user_with_roles(ip, vec![], vec![]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g_pk, make_group("g")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client.expect_update_multicastgroup_roles().never(); + + let cmd = MulticastUnpublishCliCommand { + groups: vec!["g".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } + + #[tokio::test] + async fn unpublish_last_publisher_still_issues_onchain_call() { + // The CLI prints a warning but does not block. + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g_pk = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + let user = user_with_roles(ip, vec![g_pk], vec![]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g_pk, make_group("g")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client + .expect_update_multicastgroup_roles() + .once() + .returning(|_| Ok(solana_sdk::signature::Signature::default())); + + let cmd = MulticastUnpublishCliCommand { + groups: vec!["g".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } + + #[tokio::test] + async fn unpublish_of_nonlast_publisher_does_not_claim_last() { + // Would_empty_publishers logic: user has two, remove one — NOT last. + // Regression check: the helper should return false. + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g1 = Pubkey::new_unique(); + let g2 = Pubkey::new_unique(); + + let user = user_with_roles(ip, vec![g1, g2], vec![]); + let would_empty = super::would_empty_publishers(&user, &[g1]); + assert!(!would_empty); + + let would_empty_all = super::would_empty_publishers(&user, &[g1, g2]); + assert!(would_empty_all); + } +``` + +- [ ] **Step 2: Run tests to verify failure** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast::tests::unpublish +``` + +Expected: compile errors — `MulticastUnpublishCliCommand::execute_inner` and `would_empty_publishers` do not exist. + +- [ ] **Step 3: Implement the handler and helper** + +Append to `client/doublezero/src/command/multicast.rs` (above the `#[cfg(test)]` block): + +```rust +use crate::cli::multicast::MulticastUnpublishCliCommand; + +/// Returns true when removing `to_remove` publisher roles from `user` would leave +/// `user.publishers` empty (and the user currently has at least one publisher role). +pub(super) fn would_empty_publishers(user: &User, to_remove: &[Pubkey]) -> bool { + if user.publishers.is_empty() { + return false; + } + let remaining = user + .publishers + .iter() + .filter(|p| !to_remove.contains(p)) + .count(); + remaining == 0 +} + +impl MulticastUnpublishCliCommand { + pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { + let controller = ServiceControllerImpl::new(None); + let client_ip = crate::command::helpers::resolve_client_ip(&controller).await?; + self.execute_inner(client, client_ip).await + } + + async fn execute_inner( + self, + client: &dyn CliCommand, + client_ip: Ipv4Addr, + ) -> eyre::Result<()> { + let spinner = init_command(2); + spinner.println(format!("⚡ Unpublishing (client_ip: {client_ip})...")); + + let (user_pk, user) = load_multicast_user(client, client_ip)?; + let groups = resolve_groups(client, &self.groups)?; + spinner.inc(1); + + // Figure out which of the requested groups the user is actually publishing to. + let effective_removals: Vec = groups + .iter() + .map(|(_, pk)| *pk) + .filter(|pk| user.publishers.contains(pk)) + .collect(); + + if would_empty_publishers(&user, &effective_removals) { + spinner.println( + "⚠️ This removes your last publisher role. In legacy-allocation \ + environments the service may briefly reprovision while the network \ + reallocates.", + ); + } + + for (code, group_pk) in groups { + if !user.publishers.contains(&group_pk) { + spinner.println(format!(" not publishing to {code} — skipping")); + continue; + } + let carry_sub = user.subscribers.contains(&group_pk); + client.update_multicastgroup_roles(UpdateMulticastGroupRolesCommand { + user_pk, + group_pk, + client_ip, + publisher: false, + subscriber: carry_sub, + })?; + spinner.println(format!(" unpublished from {code}")); + } + + finish_update(&spinner); + Ok(()) + } +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast::tests::unpublish +``` + +Expected: all four unpublish tests PASS. + +- [ ] **Step 5: Lint clean** + +Run: + +```bash +make rust-fmt && cargo clippy -p doublezero --all-targets -- -D warnings +``` + +Expected: no warnings. + +- [ ] **Step 6: Commit** + +```bash +git add client/doublezero/src/command/multicast.rs +git commit -m "client/doublezero: implement multicast unpublish command with last-publisher warning" +``` + +--- + +### Task 5: Implement `subscribe` handler + +**Files:** +- Modify: `client/doublezero/src/command/multicast.rs` + +- [ ] **Step 1: Write the failing tests** + +Append inside the existing `tests` module: + +```rust + // --- MulticastSubscribeCliCommand tests --- + + use crate::cli::multicast::MulticastSubscribeCliCommand; + + #[tokio::test] + async fn subscribe_adds_subscriber_role_and_preserves_publisher_role() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g_pk = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + // User is already a publisher of g — subscribing must keep publisher=true. + let user = user_with_roles(ip, vec![g_pk], vec![]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g_pk, make_group("g")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client + .expect_update_multicastgroup_roles() + .withf(move |cmd: &UpdateMulticastGroupRolesCommand| { + cmd.user_pk == user_pk + && cmd.group_pk == g_pk + && cmd.publisher // carry-through preserved + && cmd.subscriber + }) + .once() + .returning(|_| Ok(solana_sdk::signature::Signature::default())); + + let cmd = MulticastSubscribeCliCommand { + groups: vec!["g".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } + + #[tokio::test] + async fn subscribe_skips_already_subscribed_group() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g_pk = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + let user = user_with_roles(ip, vec![], vec![g_pk]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g_pk, make_group("g")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client.expect_update_multicastgroup_roles().never(); + + let cmd = MulticastSubscribeCliCommand { + groups: vec!["g".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } +``` + +- [ ] **Step 2: Run tests to verify failure** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast::tests::subscribe +``` + +Expected: compile error — `MulticastSubscribeCliCommand::execute_inner` does not exist. + +- [ ] **Step 3: Implement the handler** + +Append to `client/doublezero/src/command/multicast.rs` (above the `#[cfg(test)]` block): + +```rust +use crate::cli::multicast::MulticastSubscribeCliCommand; + +impl MulticastSubscribeCliCommand { + pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { + let controller = ServiceControllerImpl::new(None); + let client_ip = crate::command::helpers::resolve_client_ip(&controller).await?; + self.execute_inner(client, client_ip).await + } + + async fn execute_inner( + self, + client: &dyn CliCommand, + client_ip: Ipv4Addr, + ) -> eyre::Result<()> { + let spinner = init_command(2); + spinner.println(format!("⚡ Subscribing (client_ip: {client_ip})...")); + + let (user_pk, user) = load_multicast_user(client, client_ip)?; + let groups = resolve_groups(client, &self.groups)?; + spinner.inc(1); + + for (code, group_pk) in groups { + if user.subscribers.contains(&group_pk) { + spinner.println(format!(" already subscribed to {code} — skipping")); + continue; + } + let carry_pub = user.publishers.contains(&group_pk); + client.update_multicastgroup_roles(UpdateMulticastGroupRolesCommand { + user_pk, + group_pk, + client_ip, + publisher: carry_pub, + subscriber: true, + })?; + spinner.println(format!(" subscribed to {code}")); + } + + finish_update(&spinner); + Ok(()) + } +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast::tests::subscribe +``` + +Expected: both subscribe tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add client/doublezero/src/command/multicast.rs +git commit -m "client/doublezero: implement multicast subscribe command" +``` + +--- + +### Task 6: Implement `publish` handler + +**Files:** +- Modify: `client/doublezero/src/command/multicast.rs` + +- [ ] **Step 1: Write the failing tests** + +Append inside the existing `tests` module: + +```rust + // --- MulticastPublishCliCommand tests --- + + use crate::cli::multicast::MulticastPublishCliCommand; + + #[tokio::test] + async fn publish_adds_publisher_role_and_preserves_subscriber_role() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g_pk = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + // User is already a subscriber of g — publishing must keep subscriber=true. + let user = user_with_roles(ip, vec![], vec![g_pk]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g_pk, make_group("g")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client + .expect_update_multicastgroup_roles() + .withf(move |cmd: &UpdateMulticastGroupRolesCommand| { + cmd.user_pk == user_pk + && cmd.group_pk == g_pk + && cmd.publisher + && cmd.subscriber // carry-through preserved + }) + .once() + .returning(|_| Ok(solana_sdk::signature::Signature::default())); + + let cmd = MulticastPublishCliCommand { + groups: vec!["g".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } + + #[tokio::test] + async fn publish_skips_already_published_group() { + let ip = Ipv4Addr::new(10, 0, 0, 1); + let g_pk = Pubkey::new_unique(); + let user_pk = Pubkey::new_unique(); + + let mut client = create_test_client(); + let user = user_with_roles(ip, vec![g_pk], vec![]); + let mut users = HashMap::new(); + users.insert(user_pk, user); + client + .expect_list_user() + .returning(move |_| Ok(users.clone())); + + let mut groups = HashMap::new(); + groups.insert(g_pk, make_group("g")); + client + .expect_list_multicastgroup() + .returning(move |_| Ok(groups.clone())); + + client.expect_update_multicastgroup_roles().never(); + + let cmd = MulticastPublishCliCommand { + groups: vec!["g".into()], + }; + cmd.execute_inner(&client, ip).await.unwrap(); + } +``` + +- [ ] **Step 2: Run tests to verify failure** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast::tests::publish +``` + +Expected: compile error — `MulticastPublishCliCommand::execute_inner` does not exist. + +- [ ] **Step 3: Implement the handler** + +Append to `client/doublezero/src/command/multicast.rs` (above the `#[cfg(test)]` block): + +```rust +use crate::cli::multicast::MulticastPublishCliCommand; + +impl MulticastPublishCliCommand { + pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { + let controller = ServiceControllerImpl::new(None); + let client_ip = crate::command::helpers::resolve_client_ip(&controller).await?; + self.execute_inner(client, client_ip).await + } + + async fn execute_inner( + self, + client: &dyn CliCommand, + client_ip: Ipv4Addr, + ) -> eyre::Result<()> { + let spinner = init_command(2); + spinner.println(format!("⚡ Publishing (client_ip: {client_ip})...")); + + let (user_pk, user) = load_multicast_user(client, client_ip)?; + let groups = resolve_groups(client, &self.groups)?; + spinner.inc(1); + + for (code, group_pk) in groups { + if user.publishers.contains(&group_pk) { + spinner.println(format!(" already publishing to {code} — skipping")); + continue; + } + let carry_sub = user.subscribers.contains(&group_pk); + client.update_multicastgroup_roles(UpdateMulticastGroupRolesCommand { + user_pk, + group_pk, + client_ip, + publisher: true, + subscriber: carry_sub, + })?; + spinner.println(format!(" publishing to {code}")); + } + + finish_update(&spinner); + Ok(()) + } +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: + +```bash +cargo test -p doublezero --lib command::multicast +``` + +Expected: all multicast command tests PASS (subscribe, unsubscribe, publish, unpublish, helpers). + +- [ ] **Step 5: Lint clean** + +Run: + +```bash +make rust-fmt && cargo clippy -p doublezero --all-targets -- -D warnings +``` + +Expected: no warnings. + +- [ ] **Step 6: Commit** + +```bash +git add client/doublezero/src/command/multicast.rs +git commit -m "client/doublezero: implement multicast publish command" +``` + +--- + +### Task 7: Wire up dispatch in main.rs + +**Files:** +- Modify: `client/doublezero/src/main.rs:265-313` (the existing `Command::Multicast` match arm) + +- [ ] **Step 1: Extend the match arm** + +Find the existing `Command::Multicast(args) => match args.command { ... }` block (around line 265). The existing inner `match` has one arm for `MulticastCommands::Group(...)`. Add four new arms for the new variants *before* the closing `},` of the outer match arm: + +Change: + +```rust + Command::Multicast(args) => match args.command { + cli::multicast::MulticastCommands::Group(args) => match args.command { + // ... existing group subcommand handling ... + }, + }, +``` + +To: + +```rust + Command::Multicast(args) => match args.command { + cli::multicast::MulticastCommands::Group(args) => match args.command { + // ... existing group subcommand handling unchanged ... + }, + cli::multicast::MulticastCommands::Subscribe(args) => args.execute(&client).await, + cli::multicast::MulticastCommands::Unsubscribe(args) => args.execute(&client).await, + cli::multicast::MulticastCommands::Publish(args) => args.execute(&client).await, + cli::multicast::MulticastCommands::Unpublish(args) => args.execute(&client).await, + }, +``` + +- [ ] **Step 2: Build and verify the CLI help text** + +Run: + +```bash +cargo build -p doublezero && \ + ./target/debug/doublezero multicast --help +``` + +Expected: the `--help` output lists `group`, `subscribe`, `unsubscribe`, `publish`, `unpublish` subcommands. + +- [ ] **Step 3: Run full workspace build and lint** + +Run: + +```bash +make rust-lint +``` + +Expected: no errors or warnings. + +- [ ] **Step 4: Commit** + +```bash +git add client/doublezero/src/main.rs +git commit -m "client/doublezero: wire up multicast subscribe/unsubscribe/publish/unpublish dispatch" +``` + +--- + +### Task 8: Add E2E helper methods for the four verbs + +**Files:** +- Modify: `e2e/main_test.go` (after the existing `AddMulticastSubscriberGroupSkipAccessPass` helper around line 551-559) + +- [ ] **Step 1: Add the four helpers** + +Insert after the existing `AddMulticastSubscriberGroupSkipAccessPass` function (around line 559), before `DisconnectMulticastSubscriber`: + +```go +// SubscribeMulticastGroup adds subscriber role(s) to an already-connected multicast user. +func (dn *TestDevnet) SubscribeMulticastGroup(t *testing.T, client *devnet.Client, multicastGroupCodes ...string) { + dn.log.Debug("==> Subscribing to multicast groups", "clientIP", client.CYOANetworkIP, "groups", multicastGroupCodes) + + groupArgs := strings.Join(multicastGroupCodes, " ") + _, err := client.Exec(t.Context(), []string{"bash", "-c", "doublezero multicast subscribe " + groupArgs}) + require.NoError(t, err, "failed to subscribe to multicast groups") + + dn.log.Debug("--> Subscribed to multicast groups") +} + +// UnsubscribeMulticastGroup removes subscriber role(s) from an already-connected multicast user. +func (dn *TestDevnet) UnsubscribeMulticastGroup(t *testing.T, client *devnet.Client, multicastGroupCodes ...string) { + dn.log.Debug("==> Unsubscribing from multicast groups", "clientIP", client.CYOANetworkIP, "groups", multicastGroupCodes) + + groupArgs := strings.Join(multicastGroupCodes, " ") + _, err := client.Exec(t.Context(), []string{"bash", "-c", "doublezero multicast unsubscribe " + groupArgs}) + require.NoError(t, err, "failed to unsubscribe from multicast groups") + + dn.log.Debug("--> Unsubscribed from multicast groups") +} + +// PublishMulticastGroup adds publisher role(s) to an already-connected multicast user. +func (dn *TestDevnet) PublishMulticastGroup(t *testing.T, client *devnet.Client, multicastGroupCodes ...string) { + dn.log.Debug("==> Publishing to multicast groups", "clientIP", client.CYOANetworkIP, "groups", multicastGroupCodes) + + groupArgs := strings.Join(multicastGroupCodes, " ") + _, err := client.Exec(t.Context(), []string{"bash", "-c", "doublezero multicast publish " + groupArgs}) + require.NoError(t, err, "failed to publish to multicast groups") + + dn.log.Debug("--> Published to multicast groups") +} + +// UnpublishMulticastGroup removes publisher role(s) from an already-connected multicast user. +func (dn *TestDevnet) UnpublishMulticastGroup(t *testing.T, client *devnet.Client, multicastGroupCodes ...string) { + dn.log.Debug("==> Unpublishing from multicast groups", "clientIP", client.CYOANetworkIP, "groups", multicastGroupCodes) + + groupArgs := strings.Join(multicastGroupCodes, " ") + _, err := client.Exec(t.Context(), []string{"bash", "-c", "doublezero multicast unpublish " + groupArgs}) + require.NoError(t, err, "failed to unpublish from multicast groups") + + dn.log.Debug("--> Unpublished from multicast groups") +} +``` + +- [ ] **Step 2: Verify Go build with e2e tag** + +Run: + +```bash +go build -tags e2e ./e2e/... +``` + +Expected: compiles cleanly. + +- [ ] **Step 3: Commit** + +```bash +git add e2e/main_test.go +git commit -m "e2e: add multicast subscribe/unsubscribe/publish/unpublish test helpers" +``` + +--- + +### Task 9: Extend `TestE2E_Multicast` with unsubscribe/unpublish sub-tests + +**Files:** +- Modify: `e2e/multicast_test.go` (add a new sub-test block after the existing `connect` block around lines 194-226) + +**Context:** The existing `TestE2E_Multicast` connects a publisher to `mg01`, adds `mg02` incrementally, connects a subscriber to `mg01`, adds `mg02` incrementally, and then disconnects both. We'll insert a new sub-test between `connect` and `disconnect` that removes *one of two* roles (so we don't trip the last-publisher `Updating` teardown path) and verifies state. + +- [ ] **Step 1: Write the failing sub-test** + +In `e2e/multicast_test.go`, locate the `if !t.Run("disconnect", ...)` block (around line 228). Insert a new `modify_roles` sub-test *before* it: + +```go + if !t.Run("modify_roles", func(t *testing.T) { + // doublezero user list renders multicast memberships in a column named `groups` + // with each entry prefixed "P:" (publisher) or "S:" (subscriber). + // See smartcontract/cli/src/user/list.rs::format_multicast_group_names. + + subscriberRow := func() map[string]string { + out, err := dn.Manager.Exec(t.Context(), []string{"doublezero", "user", "list"}) + if err != nil { + return nil + } + for _, row := range fixtures.ParseCLITable(out) { + if row["user_type"] == "Multicast" && row["client_ip"] == subscriberClient.CYOANetworkIP { + return row + } + } + return nil + } + publisherRow := func() map[string]string { + out, err := dn.Manager.Exec(t.Context(), []string{"doublezero", "user", "list"}) + if err != nil { + return nil + } + for _, row := range fixtures.ParseCLITable(out) { + if row["user_type"] == "Multicast" && row["client_ip"] == publisherClient.CYOANetworkIP { + return row + } + } + return nil + } + + // Unsubscribe the subscriber from mg01 (keeps it subscribed to mg02). + tdn.UnsubscribeMulticastGroup(t, subscriberClient, "mg01") + + require.Eventually(t, func() bool { + row := subscriberRow() + if row == nil { + return false + } + groups := row["groups"] + return !strings.Contains(groups, "S:mg01") && strings.Contains(groups, "S:mg02") + }, 60*time.Second, 2*time.Second, "subscriber should be unsubscribed from mg01 but still subscribed to mg02") + + // Unpublish the publisher from mg01 (keeps it publishing to mg02). + tdn.UnpublishMulticastGroup(t, publisherClient, "mg01") + + require.Eventually(t, func() bool { + row := publisherRow() + if row == nil { + return false + } + groups := row["groups"] + return !strings.Contains(groups, "P:mg01") && strings.Contains(groups, "P:mg02") + }, 60*time.Second, 2*time.Second, "publisher should be unpublished from mg01 but still publishing to mg02") + + // Tunnels should still be up — key assertion: no disconnect required. + err := publisherClient.WaitForTunnelUp(t.Context(), 30*time.Second) + require.NoError(t, err, "publisher tunnel should remain up after unpublish") + err = subscriberClient.WaitForTunnelUp(t.Context(), 30*time.Second) + require.NoError(t, err, "subscriber tunnel should remain up after unsubscribe") + + // Restore pre-test state so the disconnect sub-test exercises the same two-group + // teardown it did before. + tdn.SubscribeMulticastGroup(t, subscriberClient, "mg01") + tdn.PublishMulticastGroup(t, publisherClient, "mg01") + + require.Eventually(t, func() bool { + pub := publisherRow() + sub := subscriberRow() + if pub == nil || sub == nil { + return false + } + return strings.Contains(pub["groups"], "P:mg01") && + strings.Contains(pub["groups"], "P:mg02") && + strings.Contains(sub["groups"], "S:mg01") && + strings.Contains(sub["groups"], "S:mg02") + }, 60*time.Second, 2*time.Second, "roles should be restored for both clients") + }) { + t.Fail() + return + } +``` + +- [ ] **Step 2: Run the e2e test** + +Run (requires sudo on Linux for libpcap; see CLAUDE.md): + +```bash +make e2e-test RUN=TestE2E_Multicast +``` + +Expected: the new `modify_roles` sub-test passes. The existing `connect` and `disconnect` sub-tests still pass unchanged. + +If the test fails on the `multicast_pubs` / `multicast_subs` column names, fix per the note above and re-run. + +- [ ] **Step 3: Run Go lint and formatter** + +Run: + +```bash +make go-lint && make go-fmt +``` + +Expected: no errors. + +- [ ] **Step 4: Commit** + +```bash +git add e2e/multicast_test.go +git commit -m "e2e: verify multicast role modification without disconnect" +``` + +--- + +### Task 10: Final verification across the whole workspace + +- [ ] **Step 1: Full Rust test suite** + +Run: + +```bash +make rust-test +``` + +Expected: all tests pass. + +- [ ] **Step 2: Full Rust lint** + +Run: + +```bash +make rust-lint +``` + +Expected: no warnings. + +- [ ] **Step 3: Full Go lint and build** + +Run: + +```bash +make go-lint +``` + +Expected: no warnings. + +- [ ] **Step 4: Manual devnet sanity check (optional, documented)** + +On a local devnet: + +```bash +dev/dzctl destroy -y && dev/dzctl build # only if no devnet running +# Connect a multicast user to two groups via connect: +docker exec dz-local-client- doublezero connect Multicast --subscribe mg01 mg02 + +# Remove one via the new verb: +docker exec dz-local-client- doublezero multicast unsubscribe mg01 + +# Verify onchain state: +docker exec dz-local-manager doublezero user list +# → the user should show subscribers = [mg02] only, and the tunnel should still be up. + +# Verify daemon status: +docker exec dz-local-client- doublezero status +# → session_status should remain "established". +``` + +--- + +## Known limitation (spec-documented) + +`doublezero multicast unpublish ` in legacy-allocation environments will still cause a brief service reprovision because the smartcontract sets `UserStatus::Updating` and the daemon's reconciler tears the service down. The CLI warns when this would happen. Environments using onchain allocation are unaffected. This is intentionally out of scope for this plan. diff --git a/docs/superpowers/specs/2026-03-26-latency-command-ux-design.md b/docs/superpowers/specs/2026-03-26-latency-command-ux-design.md new file mode 100644 index 0000000000..e6566aca5a --- /dev/null +++ b/docs/superpowers/specs/2026-03-26-latency-command-ux-design.md @@ -0,0 +1,86 @@ +# Latency Command UX Improvements + +## Problem + +The `doublezero latency` CLI command has three issues: + +1. **Race condition in daemon:** The device fetch goroutine and probe goroutine start concurrently on daemon startup. The first `probe()` often reads an empty `DeviceCache`, probes nothing, and caches empty results. Users must wait for the next probe tick (default 30s) before data appears. + +2. **No progress feedback:** The latency command passes `None` for the spinner, so the user sees a blank terminal during what can be a 30+ second wait. + +3. **Misleading error:** When retries exhaust, the user sees "No devices found" — which conflates "daemon hasn't finished probing yet" (transient) with "no activated devices exist" (permanent). + +## Changes + +### Daemon: Fix probe sequencing and expose readiness + +**File:** `client/doublezerod/internal/latency/manager.go` + +Add to `LatencyManager`: +- `devicesFetched chan struct{}` — created in `NewLatencyManager`, closed after first successful `fetch()` populates `DeviceCache` +- `probeReady atomic.Bool` — set to `true` after first `probe()` completes and writes to `ResultsCache` + +In `Start()`: +- The fetch goroutine closes `devicesFetched` after the first successful `fetch()` call +- The probe goroutine waits on `<-l.devicesFetched` before running the first `probe()` +- After the first `probe()` writes results, set `l.probeReady.Store(true)` + +Update `ServeLatency` response format from a bare `[]LatencyResult` to: + +```json +{ + "ready": false, + "results": [] +} +``` + +Where `ready` reflects `probeReady.Load()`. + +### CLI: Spinner and improved feedback + +**File:** `client/doublezero/src/servicecontroller.rs` + +Add a new response struct for the latency endpoint: + +```rust +struct LatencyResponse { + ready: bool, + results: Vec, +} +``` + +Update `ServiceController::latency()` to return `LatencyResponse` instead of `Vec`. + +**File:** `client/doublezero/src/command/latency.rs` + +Create a spinner in `execute()` and pass it to `check_doublezero()` and `retrieve_latencies()`. + +**File:** `client/doublezero/src/dzd_latency.rs` + +Update `retrieve_latencies()` retry logic: +- When `ready: false`: retry with 1s interval, spinner shows "Waiting for daemon to finish probing devices..." +- When `ready: true` but results empty: stop retrying immediately, return clear error "No activated devices found" +- When `ready: true` with results: return normally + +Remove exponential backoff for the "not ready" case — a simple 1s poll is appropriate since we're waiting for a one-time daemon initialization, not recovering from an error. + +Keep the existing retry with backoff for transient errors from the daemon endpoint itself (connection refused, etc.). + +## Files Modified + +| File | Change | +|------|--------| +| `client/doublezerod/internal/latency/manager.go` | Add `devicesFetched` channel, `probeReady` atomic, sequencing, new response format | +| `client/doublezero/src/servicecontroller.rs` | New `LatencyResponse` struct, update `latency()` return type | +| `client/doublezero/src/command/latency.rs` | Add spinner creation and passing | +| `client/doublezero/src/dzd_latency.rs` | Update retry logic to use `ready` field, improve error messages | + +## User-facing output unchanged + +The `{ready, results}` wrapper is only the internal daemon-to-CLI wire format over the Unix socket. The user-facing output from `doublezero latency` (table or `--json`) remains `Vec` unchanged. + +## Not in scope + +- Changing probe intervals or timeouts +- Concurrent device fetch + latency fetch in the CLI (they're already fast once the daemon is ready) +- Streaming/incremental display of results diff --git a/docs/superpowers/specs/2026-04-03-status-multicast-groups-design.md b/docs/superpowers/specs/2026-04-03-status-multicast-groups-design.md new file mode 100644 index 0000000000..7aa81dcee7 --- /dev/null +++ b/docs/superpowers/specs/2026-04-03-status-multicast-groups-design.md @@ -0,0 +1,79 @@ +# Status Command: Show Multicast Group Memberships + +## Goal + +Show which multicast groups a user is subscribed to and/or publishing to in the `doublezero status` command output. + +## Current Behavior + +The `doublezero status` command shows tunnel status, device info, metro, network, and tenant. For multicast user types, there is no indication of which groups the user belongs to or their role (publisher vs subscriber). + +## Design + +### Data flow + +The daemon already fetches full onchain program data during reconciliation, including `MulticastGroup` accounts (with group codes) and `User` accounts (with `Publishers` and `Subscribers` pubkey vecs). The multicast group information just needs to be threaded through the v2 status response. + +### Go daemon changes (`client/doublezerod/`) + +Add a `MulticastGroups` struct and a `multicast_groups` field to `V2ServiceStatus`: + +```go +type MulticastGroups struct { + Publisher []string `json:"publisher"` + Subscriber []string `json:"subscriber"` +} + +type V2ServiceStatus struct { + *api.StatusResponse + // ... existing fields ... + MulticastGroups MulticastGroups `json:"multicast_groups"` +} +``` + +In `enrichStatuses()`, for each service, resolve the matched user's `Publishers` and `Subscribers` pubkey vecs against the fetched `MulticastGroup` map to populate the group code lists. Non-multicast users get empty lists. + +### Rust CLI changes (`client/doublezero/`) + +Extend `V2ServiceStatus` in `servicecontroller.rs`: + +```rust +#[derive(Serialize, Deserialize, Debug, Clone, Default)] +pub struct MulticastGroups { + #[serde(default)] + pub publisher: Vec, + #[serde(default)] + pub subscriber: Vec, +} +``` + +Add `multicast_groups: MulticastGroups` to `V2ServiceStatus` (with `#[serde(default)]` for backward compatibility with older daemons). + +In `status.rs`, add a "Multicast Groups" column to `AppendedStatusResponse` that formats as `P:code1,S:code2` in table mode. JSON mode includes the structured `multicast_groups` object. + +### Output examples + +**Table mode:** +``` +| Session Status | ... | Multicast Groups | +|----------------|-----|--------------------------| +| BGP Session Up | ... | P:solana-lv,S:solana-ams | +``` + +**JSON mode:** +```json +{ + "multicast_groups": { + "publisher": ["solana-lv"], + "subscriber": ["solana-ams"] + } +} +``` + +For non-multicast users, the column is empty and both JSON arrays are `[]`. + +## Testing + +- Unit tests in `status.rs` for the new field formatting (table and JSON) +- Unit test for backward compatibility when daemon omits the field (serde default) +- Verify `enrichStatuses` correctly resolves publisher/subscriber pubkeys to group codes in Go tests diff --git a/docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md b/docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md new file mode 100644 index 0000000000..1d4822d092 --- /dev/null +++ b/docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md @@ -0,0 +1,112 @@ +# Multicast: modify subscriptions and publishers without disconnecting + +**Status:** Design approved, pending implementation plan. +**Date:** 2026-04-23. + +## Problem + +A connected multicast user cannot modify their set of publisher or subscriber group memberships without running `doublezero disconnect` (which deletes the user account) and then reconnecting with the new desired set. + +Today, `doublezero connect Multicast --publish --subscribe ` handles first-time connection and is purely additive — it will add new groups to an existing connected user, but it never removes. There is no CLI path to drop a subscriber role, drop a publisher role, or otherwise narrow the active role set. Users work around this by disconnecting and reconnecting. + +## Scope + +In scope: + +- Add four new CLI subcommands under `doublezero multicast` so users can add or remove publisher/subscriber roles on an already-connected multicast user without running `disconnect`. +- Extend the existing multicast e2e suite to cover the new commands. + +Out of scope: + +- Smartcontract changes. The existing `UpdateMulticastGroupRoles` instruction already supports add and remove via its `publisher` / `subscriber` boolean flags, and is what `connect` uses today. +- Daemon changes. In particular, the legacy-allocation codepath in the smartcontract transitions a user's status to `Updating` when they gain their first publisher or lose their last publisher, and the daemon's reconciler treats `Updating` as "not provisioned" and tears the service down. This teardown behavior is a known limitation and is not fixed here. Environments using onchain allocation already avoid `Updating` for these transitions. +- Making `connect` declarative / non-additive. `connect` stays as-is. + +## Commands + +All four commands live under the existing `doublezero multicast` namespace (which currently only exposes administrative `group` subcommands). Each takes one or more group codes. + +``` +doublezero multicast subscribe [ ...] # add subscriber role(s) +doublezero multicast unsubscribe [ ...] # remove subscriber role(s) +doublezero multicast publish [ ...] # add publisher role(s) +doublezero multicast unpublish [ ...] # remove publisher role(s) +``` + +`subscribe` and `publish` overlap functionally with `doublezero connect Multicast --subscribe/--publish` when a user is already connected. They are added for interface symmetry with the `un-` variants and to offer a shorter invocation for the common "already connected, adjust my roles" case. + +## Behavior + +**Precondition.** A Multicast user must already exist for the caller's `client_ip`. If none, the command fails with: + +``` +No active multicast user for . Run 'doublezero connect Multicast --publish/--subscribe ' first. +``` + +These commands never create users — that remains the responsibility of `connect`. + +**Group code resolution.** Group codes are resolved to pubkeys via `ListMulticastGroupCommand`, matching the pattern used by `connect::execute_multicast`. Any unknown code fails the whole command before any onchain call is issued. + +**Onchain call.** For each resolved group, one `UpdateMulticastGroupRolesCommand` is issued. The onchain instruction is a per-flag "set" (passing `publisher: false` removes the publisher role for that group; passing `publisher: true` adds it), so each verb carries the *other* role's current value through unchanged to avoid clobbering it: + +| Verb | `publisher` flag | `subscriber` flag | +| ------------- | ------------------------------ | ------------------------------ | +| `subscribe` | `user.publishers.contains(g)` | `true` | +| `unsubscribe` | `user.publishers.contains(g)` | `false` | +| `publish` | `true` | `user.subscribers.contains(g)` | +| `unpublish` | `false` | `user.subscribers.contains(g)` | + +**No-op handling.** If a group is already in the requested state for the given verb (e.g. `subscribe` on a group the user is already subscribed to, or `unpublish` on a group the user does not publish to), the CLI logs a skip message (`already subscribed to `, `not publishing to `, etc.) and moves on without issuing an onchain call for that group. The command as a whole still succeeds. + +**Last-publisher unpublish warning.** When an `unpublish` would empty `user.publishers`, the CLI prints a warning that the service may briefly reprovision — this is the legacy-allocation limitation described above. The command proceeds. No interactive prompt; the warning is informational. + +**Daemon reconciliation.** No explicit daemon notify. The daemon's reconciler polls onchain state and will add or drop multicast routes automatically. The CLI prints `Updated. Routes will adjust shortly.` and exits. + +## Implementation layout + +### New and modified files + +- **`client/doublezero/src/cli/multicast.rs`** — extend `MulticastCommands` with four new variants: `Subscribe`, `Unsubscribe`, `Publish`, `Unpublish`. Each takes an args struct with one or more group codes. The existing `Group(MulticastGroupCliCommand)` variant is unchanged. +- **`client/doublezero/src/command/multicast.rs`** *(new)* — one handler per verb, sharing helpers: + - `resolve_groups(client, codes) -> Vec` — mirrors the group-code resolution in `connect::execute_multicast`. + - `load_multicast_user(client, client_ip) -> (Pubkey, User)` — finds the single Multicast user for the caller's `client_ip`, returning the precondition error when absent. + - `apply_role_change(client, user_pk, user, group_pk, publisher, subscriber)` — builds the `UpdateMulticastGroupRolesCommand` with the correct carry-through of the opposite role, handles the no-op skip log, emits the last-publisher warning when applicable. + - Each of the four verb handlers: resolve groups, load user, iterate groups calling `apply_role_change` with the flag pattern from the table above, print the completion message. +- **`client/doublezero/src/main.rs`** — dispatch the four new `MulticastCommands` variants to the new handlers, following the existing `Command::Connect(args) => args.execute(&client).await` pattern. + +No SDK changes — `UpdateMulticastGroupRolesCommand` already exists and is what `connect` invokes for additive role changes today. + +Handlers live in `client/doublezero/src/command/` (not `smartcontract/cli/src/`) because they depend on client-side `client_ip` discovery and user lookup, matching where `connect.rs` and `disconnect.rs` already sit. The administrative `multicast group *` commands remain in the smartcontract CLI crate. + +## Testing + +### Unit tests (Rust) + +Colocated with the new command handlers in `client/doublezero/src/command/multicast.rs` (or a dedicated test module), using the same mocking patterns as the existing `connect` tests: + +- Happy-path for each verb: resolves groups, calls `UpdateMulticastGroupRolesCommand` with the correct flag pair (including correct carry-through of the opposite role), succeeds. +- Missing multicast user → precondition error with the documented message. +- Unknown group code → error before any onchain call is issued. +- No-op: `subscribe` on a group the user is already subscribed to skips; same for `publish`. `unsubscribe` and `unpublish` on groups the user is not in also skip. +- Role carry-through: `unsubscribe` on a group where the user is *also* a publisher keeps the publisher role (verify the outgoing `UpdateMulticastGroupRolesCommand` has `publisher: true, subscriber: false`). +- Last-publisher `unpublish`: warning is emitted and the command still proceeds. + +### E2E test + +Extend the existing multicast e2e suite in `e2e/` (e.g. `multicast_test.go` or a new sibling file under `e2e/internal/qa/client_multicast.go`) to exercise the full flow end-to-end. At minimum: + +1. Connect a client with `--subscribe groupA`. +2. Run `multicast subscribe groupB`; assert the onchain user now has `subscribers = [A, B]` and the client has multicast routes for both groups. +3. Run `multicast unsubscribe groupA`; assert `subscribers = [B]` and routes for A are gone. +4. Run `multicast publish groupC`; assert publisher role is added (onchain allocation path; no teardown). +5. Run `multicast unpublish groupC`; assert publisher role is removed. Note the expected `Updating` teardown in legacy-allocation environments and confirm the onchain-allocation path does not tear down. + +Mirror whichever helpers in `e2e/internal/qa/client_multicast.go` are most convenient (e.g. add `SubscribeMulticast`, `UnsubscribeMulticast`, etc. alongside the existing `ConnectUserMulticast_*` helpers). + +### Manual local-devnet verification + +Document in the PR's Testing Verification section: on `dev/dzctl` local devnet, connect a client with `--subscribe groupA`, then exercise each new verb and inspect state with `doublezero user list` (onchain) and `doublezero status` (daemon) to confirm the expected transitions. + +## Known limitation (documented) + +In legacy-allocation environments, running `doublezero multicast unpublish ` still causes a brief service reprovision because the smartcontract transitions the user to `Updating` and the daemon's reconciler tears the service down. Onchain-allocation environments already avoid this. Fixing this requires either daemon or smartcontract changes and is deliberately out of scope for this work. diff --git a/docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md b/docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md new file mode 100644 index 0000000000..c194e83c3c --- /dev/null +++ b/docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md @@ -0,0 +1,160 @@ +# Permission-account authorization for the feed oracle (serviceability) + +Issue: malbeclabs/infra#1652 (reframed — see below). Parent: #1547 Generalized Seat Buying. +Target repo: malbeclabs/doublezero (this repo, serviceability program). Branch off `origin/main`. + +## Why this reframes #1652 + +#1652 proposed giving the shred oracle a scoped `feed_authority` delete authority (mirroring the +`feed_authority` owns-it scoping on set/close). Two findings make that the wrong vehicle: + +1. **`feed_authority`'s owns-only gate is incompatible with the oracle's validator-owned flow.** + `set`/`close`/`add`/`remove` deny when `feed_authority == payer && accesspass.owner != payer` — + and that check fires even for a caller who is *also* in `foundation_allowlist`. Validator-owned + access passes are owned by the validator (the oracle skips `set_access_pass` on an existing pass + precisely so it does not clobber the validator's settings), so an oracle holding `feed_authority` + would be blocked on every validator-owned pass. This is exactly why the validator-owned rollout + cleared `feed_authority` and put the oracle in `foundation_allowlist`. +2. **The `sentinel_authority_pk` slot is occupied.** It is wired as the billing sentinel + (`processors/tenant/update_payment_status.rs`, "used by billing sentinel after deduction") and the + suspend/circuit-breaker authority. It is a single-pubkey slot, so the oracle cannot co-occupy it, + and folding the oracle into it would put billing + the network kill-switch on the hot oracle key. + +The least-privilege path that avoids both problems is the program's **Permission-account** system: +grant the oracle a per-key `Permission` PDA carrying exactly the bits it needs, and make the handlers +it calls honor that Permission account. No single-role slot contention, no owns-only gate, far +narrower than `foundation_allowlist` (no governance / allowlist-edit / authority-rotation bits). + +## Scope of this spec (Unit 1 of 3) + +This spec covers ONLY the serviceability program change. It is self-contained and testable on its own +— the new Permission-account path is dormant until a caller actually passes a Permission account, so +nothing changes for existing callers. Two follow-up units are out of scope here: + +- **Unit 2 (oracle wiring, repo `malbeclabs/doublezero-shreds`):** `dz_ledger.rs` appends the oracle's + Permission PDA `AccountMeta` to the relevant instructions. +- **Unit 3 (operational rollout):** `CreatePermission` for the oracle's key with + `ACCESS_PASS_ADMIN | USER_ADMIN`, then remove the oracle from `foundation_allowlist`. + +## Goal / acceptance + +A key that signs while passing its own `Permission` PDA (status `Activated`) bearing +`ACCESS_PASS_ADMIN | USER_ADMIN` can perform all seven serviceability operations the oracle uses, +**without** `foundation_allowlist` membership and **without** any single-role slot. Every existing +caller's authorization is unchanged (additive only). + +## Design + +### 1. New helper: `authorize_permission_account_only` (authorize.rs) + +`authorize()` already supports a Permission-account path, but it bundles a *legacy fallback* +(`check_legacy_any`) whose composite differs from each handler's hand-written inline check (e.g. +`delete` is `foundation || user.owner`, but `USER_ADMIN` legacy is `foundation || activator`). Routing +existing callers through `authorize()` would therefore silently change their authorities. So we add a +**permission-account-only** helper with no legacy fallback, to be OR'd with each handler's existing +inline check: + +```rust +/// Permission-account-only authorization (no legacy GlobalState fallback). +/// +/// Reads the next account from `accounts_iter` as the payer's optional trailing +/// `Permission` PDA: +/// - No further account present -> Ok(false) (caller falls back to its own inline checks) +/// - Present, is the payer's Permission PDA, program-owned, status == Activated, +/// and `permissions & any_of_flags != 0` -> Ok(true) +/// - Present, is the payer's Permission PDA, program-owned, Activated, but no +/// required bit set -> Ok(false) +/// - Present but NOT the payer's Permission PDA, or not program-owned -> Err +/// (a malformed/incorrect account was supplied) +/// +/// Unlike `authorize`, this never consults `check_legacy_any`, so OR-ing it into a +/// handler's existing inline check cannot widen the legacy authorities. +pub fn authorize_permission_account_only<'a, 'b: 'a, I>( + program_id: &Pubkey, + accounts_iter: &mut I, + payer_key: &Pubkey, + any_of_flags: u128, +) -> Result +where + I: Iterator>; +``` + +It reuses the same validation `authorize()`'s new path performs (PDA == `get_permission_pda(program_id, +payer_key)`, `owner == program_id`, deserialize `Permission`, `status == Activated`). The only +behavioral differences from `authorize()`'s new path: a missing-bit result returns `Ok(false)` rather +than `Err` (so the handler's own inline OR decides the final verdict), and an absent account returns +`Ok(false)` rather than entering the legacy branch. + +### 2. Integration pattern for each handler + +The Permission account is an **optional trailing account** (appended after each instruction's current +last account), so today's callers — which pass no such account — are unaffected and keep hitting their +inline checks. Because the helper reads the *next* account from the iterator, the authorization +decision must be made **after all of the handler's required accounts have been read** (the iterator +must be positioned at the trailing slot) and **before any state mutation**. Per handler: + +1. Compute `inline_ok` from the already-read accounts using the handler's *existing* inline check, + verbatim (do not change it). +2. Finish reading the handler's required accounts (so the iterator reaches the trailing slot). +3. Before mutating any account: `let authorized = inline_ok || authorize_permission_account_only(program_id, accounts_iter, payer, BIT)?;` and reject with the handler's existing error if `!authorized`. + +Handlers today reject early (right after reading `globalstate`); this moves the *final* rejection to +just after the required-account reads, before mutations. No account read is destructive, so deferring +the rejection is safe. The existing `feed_authority` owns-only restrictions stay exactly as they are — +they gate only the `feed_authority` path, never the Permission-account path (so the oracle's +Permission account is not owns-gated, which is required for validator-owned passes it does not own). + +### 3. The seven handlers and their required bit + +| Handler (file) | Existing inline authority (preserve verbatim) | Bit added via `perm_only` | +|---|---|---| +| `SetAccessPass` (`accesspass/set.rs`) | `foundation \|\| sentinel \|\| feed_authority \|\| tenant_admin \|\| accesspass.owner==payer` (+ feed owns-it) | `ACCESS_PASS_ADMIN` | +| `CloseAccessPass` (`accesspass/close.rs`) | `foundation \|\| feed_authority` (+ feed owns-it) | `ACCESS_PASS_ADMIN` | +| `AddMulticastGroupSubAllowlist` (`multicastgroup/allowlist/subscriber/add.rs`) | `mgroup.owner==payer \|\| sentinel \|\| feed_authority \|\| foundation` (+ feed owns-it) | `ACCESS_PASS_ADMIN` | +| `RemoveMulticastGroupSubAllowlist` (`multicastgroup/allowlist/subscriber/remove.rs`) | same as add | `ACCESS_PASS_ADMIN` | +| `UpdateMulticastGroupRoles` (`multicastgroup/subscribe.rs`) | `accesspass.user_payer==payer \|\| foundation` | `ACCESS_PASS_ADMIN` | +| `DeleteUser` (`user/delete.rs`) | `foundation \|\| user.owner==payer` | `USER_ADMIN` | +| `CreateSubscribeUser` owner_override (`user/create_core.rs`) | `foundation \|\| sentinel` (to set `owner != payer`) | `USER_ADMIN` | + +The oracle's `Permission` bitmask is therefore `ACCESS_PASS_ADMIN | USER_ADMIN`. + +> Verification item for the plan: confirm `CreateSubscribeUser` has no *other* foundation-only gate +> beyond the `owner_override` check (the oracle creates users with `owner = validator`). If a second +> gate exists, it needs the same `perm_only` OR. + +### 4. Tests (`programs/doublezero-serviceability/tests/`, solana-program-test) + +Reuse the `permission_test.rs` harness (`CreatePermission` + `get_permission_pda`) and the +per-handler test helpers. For each of the seven handlers: + +- **Positive:** a signer who is NOT foundation/sentinel/feed/owner, but passes an `Activated` + Permission PDA with the required bit, succeeds. For `delete`/`close`, use a pass/user owned by a + *different* key to prove the Permission path has no owns-it restriction. +- **Negative (no account):** same signer, no Permission account passed → the handler's existing error + (unchanged legacy behavior). +- **Negative (insufficient/suspended):** Permission PDA present but missing the bit, or `Suspended` → + rejected. +- **Regression:** the existing foundation / `user.owner` / `user_payer` / feed-authority-owns-it paths + still succeed/deny exactly as before (one representative assertion each). + +Pin exact error variants (`NotAllowed` / `Unauthorized` as each handler currently returns; the new +helper's malformed-account case returns `InvalidArgument`/`InvalidAccountData`). Add a focused unit +test for `authorize_permission_account_only` covering each branch of its contract. + +### 5. IDL / SDK / docs + +- Each of the seven instructions gains one **optional trailing account** (the payer's Permission PDA). + Update the program IDL accordingly. Existing instruction builders that omit it stay valid. +- Update `PERMISSION.md` to record that these seven handlers now honor a Permission account bearing + `ACCESS_PASS_ADMIN` / `USER_ADMIN` (as listed above), in addition to their legacy authorities. +- No instruction *data* (args) change; no discriminator change. + +## Out of scope + +- Oracle wiring (Unit 2) and the operational rollout (Unit 3: create the oracle's Permission PDA; + remove it from `foundation_allowlist`). +- Setting the `RequirePermissionAccounts` feature flag (the program-wide switch that disables the + legacy path). This change is purely additive; the legacy paths remain for all existing callers. +- Converting these handlers to call `authorize()` wholesale (would change legacy composites). We add + the narrow `perm_only` OR instead. +- Any change to `feed_authority`'s owns-only gates or the sentinel/billing wiring. diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index d795db7564..b2fa2314ef 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -120,12 +120,15 @@ doublezero/ ├── client/ sdk/ e2e/ ... unchanged │ ├── solana/ from doublezero-solana +│ │ NESTED WORKSPACE, excluded from root (D2) +│ │ keeps its own Cargo.lock + rust-toolchain.toml │ ├── programs/passport │ ├── programs/revenue-distribution │ ├── crates/program-tools │ └── mock/{swap-sol-2z,rewards-integration} │ └── offchain/ from doublezero-offchain + │ root workspace members after step 4 ├── crates/ 14 crates └── scheduler/ Elixir app and its Rust NIF ``` @@ -149,30 +152,60 @@ Rejected: flattening offchain's crates into `crates/` and solana's programs into 3. It forces the sentinel directory rename on day one. Keeping the trees separate makes that collision disappear (`offchain/crates/sentinel` next to `crates/sentinel`). -### D2. One workspace, with the two Solana programs held back until measured. +### D2. Offchain joins the root workspace. The `solana/` tree stays nested. Measured. -After the workspace merge, the root workspace gains the 14 offchain crates, -`scheduler/native/scheduler_doublezero`, and `solana/crates/program-tools`. +**This was open pending step 0. Step 0 has run and the answer is no: the two Solana +programs must not join the root workspace.** -`passport` and `revenue-distribution` stay in `exclude` until we measure whether folding -them in changes the bytes they compile to. `doublezero-solana` builds them in Docker with -`cargo fetch --locked` and checks them against `programs/sha256sums_*.txt`. A shared -lockfile may resolve some dependency differently, which would change the artifact. +The programs' build closure is 96 crates. **62 of them resolve to a different version in +this repo's lockfile.** A sample: -Step 0 below settles this by measurement, not by argument. If the bytes do not change, -they join the root workspace. If they change, they stay excluded with their own lockfile, -and they keep needing a pin bump like today. - -The two `mock/` programs stay excluded. Offchain already excludes its own mock program, -so this follows the existing convention. - -### D3. Toolchain: root stays 1.97.1, programs pin 1.91. - -Add `solana/programs/rust-toolchain.toml` pinning 1.91, mirroring +| crate | in solana | in this repo | +| --- | --- | --- | +| `borsh` | 1.6.0 | 1.7.0 | +| `bytemuck` | 1.24.0 | 1.25.0 | +| `solana-instruction` | 3.1.0 | 3.4.0 | +| `solana-clock` | 3.0.0 | 3.1.0 | +| `solana-account-info` | 3.1.0 | 3.1.1 | +| `solana-fee-calculator` | 3.0.0 | 3.2.1 | + +`borsh` and `bytemuck` are the serialization and zero-copy layout crates the account +structs are built on. A shared lockfile changes the compiled bytes, so +`programs/sha256sums_*.txt` would no longer verify and the deployed artifacts would stop +being reproducible from this repo. + +So the end state is: + +- **`offchain/` joins the root workspace.** Its 14 crates plus + `scheduler/native/scheduler_doublezero` become members. +- **`solana/` stays a nested excluded workspace**, keeping its own `Cargo.toml`, + `Cargo.lock` and `rust-toolchain.toml`. That covers `programs/passport`, + `programs/revenue-distribution`, `crates/program-tools` and both `mock/` programs. +- **Offchain reaches `program-tools` by path** into that excluded tree, rather than by git + tag. `program-tools` compiles twice, once per lockfile, exactly as it does today. + +**This does not cost the atomic-change goal.** The pin disappears either way. A change +touching `program-tools` and offchain together is still one commit in one repo, reviewed +and merged atomically. Only dependency *resolution* stays separate, which is the whole +point of holding the programs back. + +It also makes step 4 smaller than planned: it merges offchain only. + +Re-measure if this repo's lockfile ever converges on solana's versions, using the method in +step 0. Nothing about this is permanent. + +### D3. Toolchain: root stays 1.97.1, the Solana programs keep 1.91. + +Nothing to add. `solana/` stays a nested workspace per D2, so its existing repo-wide +`rust-toolchain.toml` (channel 1.91) carries over and keeps applying to that tree. That is +the same directory-scoped pinning this repo already uses at `smartcontract/programs/rust-toolchain.toml`. -Offchain's repo-wide 1.92.0 pin goes away and its crates build on 1.97.1. Expect new -clippy findings. That work belongs to step 4 and is not a surprise to discover later. +Offchain's repo-wide 1.92.0 pin goes away and its crates build on 1.97.1. **Measured in +step 0: it compiles clean with 15 clippy warnings**, all trivial and mostly `--fix`-able +(`useless_conversion`, `useless_borrows_in_formatting`, `unnecessary_sort_by`, +`explicit_counter_loop`, `collapsible_match`). This was expected to be the painful part of +step 4. It is not. borsh unifies on 1.7.0. @@ -242,6 +275,22 @@ comm -12 <(gh api 'repos/malbeclabs/doublezero-offchain/tags?per_page=100' --pag Anything it prints needs a decision before step 2 runs. +## Two cargo behaviours this plan depends on + +Both were tested with a minimal reproduction rather than reasoned about, because the design +rests on them. + +1. **A root workspace can path-depend into an excluded nested workspace.** So offchain, as + a root member, can depend on `solana/crates/program-tools` by path while `solana/` keeps + its own workspace and lockfile. Verified: builds clean. +2. **A git dependency can reach a package in the repo's `exclude` list.** Cargo scans the + repository for the named package; workspace membership does not gate git dependencies. + Verified: builds clean. + +The second one matters for shreds. Holding `revenue-distribution` out of the root workspace +does **not** stop shreds pulling it from `malbeclabs/doublezero` by git, so the single-source +story in "What this does to shreds" survives D2 intact. + ## Sequencing One throwaway measurement, then six pull requests in this repo, then one follow-on pull @@ -251,19 +300,33 @@ The riskiest work is split across steps 3 and 4 so that flipping the dependencie merging the workspaces fail separately. Each step is revertable on its own except step 2, which is the one-way door. -### Step 0. Measure the program bytes. Throwaway. +### Step 0. Measure whether a shared lockfile changes the program bytes. DONE. + +**Method.** Rebuilding the artifacts is the wrong test, and the obvious version of it gives +a false negative: this repo has no `build-artifacts` target, and solana's Docker path runs +`cargo fetch --locked` against solana's *own* lockfile, so it would report no change however +the merged workspace resolved. -Do the whole merge locally. Add `passport` and `revenue-distribution` as root workspace -members. Rebuild through the existing path: +The bytes can only change if the resolved dependency versions change, so compare resolution +directly. Take the programs' normal-dependency closure, then compare each crate's resolved +version against this repo's lockfile: ```sh -make build-artifacts NETWORK=mainnet-beta -shasum -a 256 -c programs/sha256sums_mainnet_beta.txt +cargo tree --locked -p doublezero-passport -e normal --prefix none --features entrypoint \ + | sed 's/ (\*)$//' | awk 'NF{print $1" "$2}' | sort -u ``` -Repeat for `NETWORK=development`. Output is one fact that decides D2. Measure the clippy -volume from the toolchain bump at the same time, since the tree is already there. Discard -the work either way. +Cheaper than a build, exact, and it names *which* crates move rather than just saying the +hash differs. + +**Result.** 96 crates in the closure, **62 resolve differently**. See D2. The programs stay +excluded. + +**Also measured, since the tree was already built:** offchain on 1.97.1 compiles clean with +15 trivial clippy warnings. See D3. + +**Prerequisite this surfaced:** `git-filter-repo` is not installed. Step 2 needs it. +Step 0 did not, because dependency resolution depends on manifests and paths, not history. ### Step 1. Golden tests for `contributor-rewards`. @@ -296,7 +359,7 @@ does not expect them is noise. Gate: every existing job green, with no existing job definition touched. `git diff` between each imported tree and its filtered source is empty. The diff is large and needs no -judgement to review. +judgment to review. ### Step 3. Flip the git dependencies to path dependencies. @@ -322,19 +385,31 @@ lockfile. Now a manifest consolidation plus the toolchain work, with the dependency flip already proven by step 3. -- Root workspace absorbs the crates per D2. -- Delete the four nested `Cargo.toml` and `Cargo.lock` files, and `offchain/rust-toolchain.toml`. +- Root workspace absorbs **offchain's** crates per D2. +- Delete `offchain/Cargo.toml`, `offchain/Cargo.lock` and `offchain/rust-toolchain.toml`. A `rust-toolchain.toml` is directory-scoped, so leaving it would give a different compiler depending on which directory cargo was invoked from. +- **Keep `solana/Cargo.toml`, `solana/Cargo.lock` and `solana/rust-toolchain.toml`.** The + programs stay excluded per D2 and need their own locked build for the checksum gate. Do + not delete them. - Set `edition = "2024"` explicitly on all 14 offchain crates per D3. - Offchain's crates move to 1.97.1; borsh unifies on 1.7.0; rename the sentinel binary per D4. - Pin the fixture generators and convert solana's `>=2,<=3` ranges to exact pins. Gate: `cargo check --workspace`, `cargo clippy --all-targets`, the full test suite, e2e, -the program checksum check, and **the step 1 goldens still green**. Also confirm the -lockfile holds no crate twice: `grep '^name = ' Cargo.lock | sort | uniq -d` should print -nothing that was not already duplicated before the merge. +the program checksum check from inside `solana/`, and **the step 1 goldens still green**. + +Two lockfile gates. A bare duplicate-name check is useless here: `cargo tree -d --workspace` +already reports 140 duplicate groups on `main` today, which are normal. + +```sh +# no internal git sources should remain; all are path deps after step 3 +grep -c 'source = "git+https://github.com/malbeclabs' Cargo.lock # expect 0 + +# duplicate groups must not increase against the pre-merge baseline +cargo tree -d --workspace --locked | grep -cE '^[a-z0-9_-]+ v' # baseline was 140 +``` ### Step 5. Merge release and CI. @@ -413,7 +488,7 @@ Every step reverts cleanly except step 2. Step 3 reverts to git dependencies. Step 4 reverts **because steps 2 and 3 leave the nested manifests in place**: step 4 only edits and deletes manifests, so reverting it restores the -four nested `Cargo.toml` and `Cargo.lock` files and leaves a repo that builds as it did +`offchain/Cargo.toml` and `offchain/Cargo.lock` and leaves a repo that builds as it did after step 3. Step 5 is workflows. Unarchiving a repo is a click. Splitting the old single workspace step into steps 3 and 4 is what buys this. A revert of @@ -431,9 +506,9 @@ the org move reintroduces an old URL. Harmless while the transfer redirects hold build failure once they do not. Rebasing them is shreds work, not this migration's, but it interacts with step 7. -**Clippy churn on the toolchain bump.** Offchain moves from 1.92.0 to 1.97.1 in step 4. -Volume is unknown until tried. Step 0 can measure it at the same time as the program -bytes, for free. +**Clippy churn on the toolchain bump. Measured and small.** Offchain moves from 1.92.0 to +1.97.1 in step 4. Step 0 ran it: clean compile, 15 trivial warnings, mostly `--fix`-able. +This risk is closed. **Contributor reward output can change silently. This is the risk to take seriously.** #1515 flags it: relocating `contributor-rewards` re-resolves its maths in a different From 2d7999f58795250f8de6717a8b226b0690c7b4d5 Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 09:31:00 -0700 Subject: [PATCH 08/11] docs: replace byte-identical with a tolerance gate for the reward goldens --- .../2026-08-27-monorepo-migration-design.md | 35 +++++++++++++++++-- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index b2fa2314ef..b886a567b7 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -330,9 +330,33 @@ Step 0 did not, because dependency resolution depends on manifests and paths, no ### Step 1. Golden tests for `contributor-rewards`. -Land on `main`, before anything moves. Fixed input, byte-identical reward output. This crate -decides what contributors are paid and has no output test today, so step 4 currently has no -way to prove it changed nothing. +Land on offchain's `main`, before anything moves. This crate decides what contributors are +paid and has no output test today, so step 4 currently has no way to prove it changed +nothing. + +Planned in detail at `docs/superpowers/plans/2026-08-27-contributor-rewards-goldens.md` in +`malbeclabs/doublezero-offchain`. + +**Not byte-identical output.** An earlier draft of this spec said that, and it is the wrong +gate for `f64`. The computation is deterministic on one machine (demands grouped in a +`BTreeMap`, the `rayon` `par_iter().map()` collecting back into a `BTreeMap`, `BTreeMap` +throughout the aggregator), but bit-identical floating point across architectures is not +guaranteed. Goldens generated on arm64 could differ in the last bit from x86_64 CI, giving a +permanently red test that everyone learns to ignore. That is worse than no test. + +The gate is instead: + +- **exact** on the operator set, the operator ordering, the city set, the per-city operator + ordering, and all counts, which carry no float risk +- **`1e-12` relative tolerance** on every reward value and proportion + +A dependency change that alters reward maths moves values far more than `1e-12`. A +last-bit difference across architectures does not. + +The seam is `compute_shapley_values`, the deepest point the pipeline reaches before it turns +async and needs RPC. The crate already has the fixtures for it: a real 5.3 MB +`testnet_snapshot.json`, its leader schedule, and a deterministic settings factory in +`tests/common/mod.rs`. This step stands on its own merits and is worth landing whether or not the rest of this spec proceeds. @@ -523,6 +547,11 @@ This is why step 1 exists and why it comes first: write the goldens against toda behaviour on `main`, before anything moves. Step 4 then either keeps them green or names exactly what changed. +Scope limit worth knowing: the goldens stop at `compute_shapley_values`. They do not cover +`try_distribute_epoch_rewards`, the merkle root construction, or anything else past that +point, because those are async and need RPC. If step 4 somehow affects those, the goldens +will not catch it. + **`validator-debt` stays, without a correctness gate.** It is not collecting today, so drift in its output changes no payment. It is kept deliberately rather than dropped: it is still deployed, `malbeclabs/infra` runs it from the offchain scheduler with its own AWS From 068e4a1688ebacead7b601a0ab93d3048e94a234 Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 09:47:43 -0700 Subject: [PATCH 09/11] docs: defer the reward goldens to infra#2392 and record what step 4 loses --- .../2026-08-27-monorepo-migration-design.md | 79 +++++++++---------- 1 file changed, 39 insertions(+), 40 deletions(-) diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index b886a567b7..fb87c67f37 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -328,43 +328,42 @@ excluded. **Prerequisite this surfaced:** `git-filter-repo` is not installed. Step 2 needs it. Step 0 did not, because dependency resolution depends on manifests and paths, not history. -### Step 1. Golden tests for `contributor-rewards`. - -Land on offchain's `main`, before anything moves. This crate decides what contributors are -paid and has no output test today, so step 4 currently has no way to prove it changed -nothing. - -Planned in detail at `docs/superpowers/plans/2026-08-27-contributor-rewards-goldens.md` in -`malbeclabs/doublezero-offchain`. - -**Not byte-identical output.** An earlier draft of this spec said that, and it is the wrong -gate for `f64`. The computation is deterministic on one machine (demands grouped in a -`BTreeMap`, the `rayon` `par_iter().map()` collecting back into a `BTreeMap`, `BTreeMap` -throughout the aggregator), but bit-identical floating point across architectures is not -guaranteed. Goldens generated on arm64 could differ in the last bit from x86_64 CI, giving a -permanently red test that everyone learns to ignore. That is worse than no test. - -The gate is instead: - -- **exact** on the operator set, the operator ordering, the city set, the per-city operator - ordering, and all counts, which carry no float risk -- **`1e-12` relative tolerance** on every reward value and proportion - -A dependency change that alters reward maths moves values far more than `1e-12`. A -last-bit difference across architectures does not. - -The seam is `compute_shapley_values`, the deepest point the pipeline reaches before it turns -async and needs RPC. The crate already has the fixtures for it: a real 5.3 MB -`testnet_snapshot.json`, its leader schedule, and a deterministic settings factory in -`tests/common/mod.rs`. - -This step stands on its own merits and is worth landing whether or not the rest of this -spec proceeds. - -`validator-debt` does not need the same gate. Debt is not being collected, so drift in its -output changes no payment. It still has to compile and release, and it stays in the tree. - -Gate: the goldens pass on `main` and fail if a reward figure moves. +### Step 1. Golden tests for `contributor-rewards`. DEFERRED to malbeclabs/infra#2392. + +**Attempted 2026-08-27. The committed fixture cannot support this test.** + +A golden test was written and it works, but driven from +`crates/contributor-rewards/tests/testnet_snapshot.json` it produces **0.0 for every +operator in every one of 8 cities**. Diagnosed rather than assumed: + +- The Shapley machinery is fine. `evaluator.rs`'s own + `test_aggregated_proportions_sum_to_one` asserts nonzero proportions against synthetic + inputs. +- The input assembly was correct, matching `PreparedData::from_snapshot` builder for + builder. +- The links carry real data. `test_pvt_links.rs` pins real latencies from the same + snapshot (154.520, 5.804, 67.249, 68.448, 98.787 ms). + +The zero is genuine output. At 67 to 154 ms those private links are no better than public +internet on those routes, so the private network adds no value and every operator's +marginal contribution is legitimately zero. The fixture is simply too thin: 4 contributors, +9 devices, 9 links, 2 operators reaching the output. + +Pinning those zeros would give a test that passes forever, detects nothing, and reads as +coverage. Worse than no test. + +Building a fixture that produces nonzero rewards is tracked in **malbeclabs/infra#2392**. +The raw material exists (`dry-run-output/mn-beta-epoch-12{7,8,9}-snapshot.json`, 14 +contributors, 96 devices, 166 links) but the files are 100 MB and gitignored, so one has to +be trimmed down to a committable size. They are `CompleteSnapshot` shaped, so a trimmed +fixture feeds `PreparedData::from_snapshot` directly, exercising the production entry point +rather than reassembling inputs by hand. + +**What this costs step 4.** Step 4 was gated on these goldens. Deferring them means +**nothing proves that a shared lockfile leaves reward figures unchanged**, which is the +single risk this spec's Risks section calls out as the one to take seriously. Either +infra#2392 lands before step 4, or step 4 proceeds knowing that reward drift would go +undetected. That should be a decision on the record, not a silent omission. ### Step 2. Import the code and the history. One-way door. @@ -543,9 +542,9 @@ into the root workspace in step 4. snapshot tests** today. The failure mode is wrong reward figures that build clean and pass every test. -This is why step 1 exists and why it comes first: write the goldens against today's -behaviour on `main`, before anything moves. Step 4 then either keeps them green or names -exactly what changed. +Step 1 was meant to answer this, and it is now deferred to malbeclabs/infra#2392 because no +committed fixture produces nonzero rewards. Until that lands, this risk is unmitigated and +step 4 has no way to prove it changed nothing. Scope limit worth knowing: the goldens stop at `compute_shapley_values`. They do not cover `try_distribute_epoch_rewards`, the merkle root construction, or anything else past that From 1c3574d398750e552a2ffb2eca62ababc7695702 Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 09:48:16 -0700 Subject: [PATCH 10/11] docs: build the reward fixture now rather than deferring step 1 --- .../2026-08-27-monorepo-migration-design.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md index fb87c67f37..f16181f1b0 100644 --- a/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md +++ b/docs/superpowers/specs/2026-08-27-monorepo-migration-design.md @@ -328,7 +328,7 @@ excluded. **Prerequisite this surfaced:** `git-filter-repo` is not installed. Step 2 needs it. Step 0 did not, because dependency resolution depends on manifests and paths, not history. -### Step 1. Golden tests for `contributor-rewards`. DEFERRED to malbeclabs/infra#2392. +### Step 1. Golden tests for `contributor-rewards`. IN PROGRESS, tracked by malbeclabs/infra#2392. **Attempted 2026-08-27. The committed fixture cannot support this test.** @@ -359,11 +359,10 @@ be trimmed down to a committable size. They are `CompleteSnapshot` shaped, so a fixture feeds `PreparedData::from_snapshot` directly, exercising the production entry point rather than reassembling inputs by hand. -**What this costs step 4.** Step 4 was gated on these goldens. Deferring them means -**nothing proves that a shared lockfile leaves reward figures unchanged**, which is the -single risk this spec's Risks section calls out as the one to take seriously. Either -infra#2392 lands before step 4, or step 4 proceeds knowing that reward drift would go -undetected. That should be a decision on the record, not a silent omission. +**Decision: build the fixture now, before step 2.** Step 4 is gated on these goldens, and +without them nothing proves that a shared lockfile leaves reward figures unchanged, which +is the one risk this spec's Risks section calls out as serious. So infra#2392 is being done +now rather than deferred, and step 1 keeps its place at the front of the sequence. ### Step 2. Import the code and the history. One-way door. @@ -542,9 +541,10 @@ into the root workspace in step 4. snapshot tests** today. The failure mode is wrong reward figures that build clean and pass every test. -Step 1 was meant to answer this, and it is now deferred to malbeclabs/infra#2392 because no -committed fixture produces nonzero rewards. Until that lands, this risk is unmitigated and -step 4 has no way to prove it changed nothing. +This is why step 1 exists and why it comes first. It turned out to need a fixture that does +not exist yet (malbeclabs/infra#2392), so building that fixture is part of step 1 rather +than a reason to skip it. Step 4 then either keeps the goldens green or names exactly what +changed. Scope limit worth knowing: the goldens stop at `compute_shapley_values`. They do not cover `try_distribute_epoch_rewards`, the merkle root construction, or anything else past that From e768af5a50083331346a0eb629131cce085d5be9 Mon Sep 17 00:00:00 2001 From: Ben Marx Date: Thu, 27 Aug 2026 11:16:42 -0700 Subject: [PATCH 11/11] docs: untrack the working documents swept in by a stray git add -A Only the monorepo migration design belongs on this branch. Nine other spec and plan documents under docs/superpowers were untracked working files and were committed by accident. They stay on disk, they are just no longer tracked. --- ...6-06-17-least-privilege-oracle-overview.md | 130 -- .../plans/2026-03-26-latency-command-ux.md | 829 --------- .../2026-04-03-status-multicast-groups.md | 419 ----- ...09-startup-tunnel-endpoint-registration.md | 185 --- ...-23-multicast-modify-without-disconnect.md | 1476 ----------------- .../2026-03-26-latency-command-ux-design.md | 86 - ...26-04-03-status-multicast-groups-design.md | 79 - ...ticast-modify-without-disconnect-design.md | 112 -- ...y-feed-oracle-permission-account-design.md | 160 -- 9 files changed, 3476 deletions(-) delete mode 100644 docs/superpowers/2026-06-17-least-privilege-oracle-overview.md delete mode 100644 docs/superpowers/plans/2026-03-26-latency-command-ux.md delete mode 100644 docs/superpowers/plans/2026-04-03-status-multicast-groups.md delete mode 100644 docs/superpowers/plans/2026-04-09-startup-tunnel-endpoint-registration.md delete mode 100644 docs/superpowers/plans/2026-04-23-multicast-modify-without-disconnect.md delete mode 100644 docs/superpowers/specs/2026-03-26-latency-command-ux-design.md delete mode 100644 docs/superpowers/specs/2026-04-03-status-multicast-groups-design.md delete mode 100644 docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md delete mode 100644 docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md diff --git a/docs/superpowers/2026-06-17-least-privilege-oracle-overview.md b/docs/superpowers/2026-06-17-least-privilege-oracle-overview.md deleted file mode 100644 index 186b8e04d1..0000000000 --- a/docs/superpowers/2026-06-17-least-privilege-oracle-overview.md +++ /dev/null @@ -1,130 +0,0 @@ -# Least-privilege feed oracle — change overview - -**Status:** proposal, pending team consensus. No code written. -**Driver:** malbeclabs/infra#1652 (reframed below) · parent #1547 Generalized Seat Buying. -**Repos touched:** `malbeclabs/doublezero` (serviceability program) and `malbeclabs/doublezero-shreds` (oracle). -**Detailed Unit-1 spec:** `doublezero/docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md`. - -## TL;DR - -The shred/feed oracle currently sits in `GlobalState.foundation_allowlist`, which is effectively -**near-superuser** (governance, allowlist edits, authority rotation, etc.). It only needs to manage -access passes, multicast subscriptions, and the users it provisions. We want to drop it to -**least privilege** by giving it a per-key **Permission PDA** carrying exactly -`ACCESS_PASS_ADMIN | USER_ADMIN`, and teaching the seven handlers it calls to honor that Permission -account. The change is **additive** — no existing caller's authorization changes. It splits into three -units (program → oracle wiring → operational rollout), sequenced carefully to avoid an unauthorized -window. - -## Why not the two obvious options - -The original #1652 idea was to give the oracle the scoped `feed_authority` role. Two findings killed -the simple paths: - -- **`feed_authority` is incompatible with the oracle's validator-owned flow.** The access-pass - handlers enforce an "owns-only" gate — `feed_authority == payer && accesspass.owner != payer → deny` - — and that gate fires **even for foundation members**. Validator-owned passes are owned by the - *validator* (the oracle deliberately does not re-`set` an existing pass, so as not to clobber the - validator's settings). An oracle holding `feed_authority` would be blocked on every validator-owned - pass. This is exactly why `feed_authority` was cleared during the validator-owned rollout. -- **The `sentinel` slot is taken.** `sentinel_authority_pk` is a single-pubkey slot already wired as - the **billing sentinel** (writes tenant payment status after deduction) and the **suspend / - circuit-breaker** authority. The oracle can't co-occupy it, and merging the network kill-switch onto - the hot oracle key would be a poor separation of duties. - -→ The **Permission-account (PDA) bitmask** system is the right vehicle: per-key, no slot contention, -no owns-only gate, and far narrower than foundation. - -## The chosen approach (mechanism) - -The program already has a `Permission` PDA per pubkey (`get_permission_pda(program_id, key)`) with a -`u128` bitmask and an `authorize()` helper. But `authorize()` bundles a **legacy fallback** whose -composite differs from each handler's hand-written inline check, so routing existing callers through -it would silently change their authorities. Instead: - -1. Add a **new, no-legacy-fallback** helper, `authorize_permission_account_only(...)`, that validates - *only* a passed Permission PDA (correct PDA for the signer, program-owned, `Activated`, required bit - set). -2. In each of the seven handlers, **keep the existing inline check verbatim** and OR in - `perm_only(BIT)`. Existing callers are untouched; a new Permission-account path is added. -3. The Permission account is an **optional trailing account** per instruction — today's callers omit - it and keep working; the oracle passes it. - -The oracle's Permission bitmask is `ACCESS_PASS_ADMIN | USER_ADMIN`. - -## What changes, by unit - -### Unit 1 — serviceability program (`malbeclabs/doublezero`) - -The seven handlers the oracle uses, the authority each preserves, and the bit added: - -| Handler | Preserved inline authority | Bit added | -|---|---|---| -| SetAccessPass | foundation / sentinel / feed (owns-it) / tenant-admin / pass-owner | `ACCESS_PASS_ADMIN` | -| CloseAccessPass | foundation / feed (owns-it) | `ACCESS_PASS_ADMIN` | -| AddMulticastGroupSubAllowlist | mgroup-owner / sentinel / feed (owns-it) / foundation | `ACCESS_PASS_ADMIN` | -| RemoveMulticastGroupSubAllowlist | same as add | `ACCESS_PASS_ADMIN` | -| UpdateMulticastGroupRoles (subscribe/unsubscribe) | pass `user_payer` / foundation | `ACCESS_PASS_ADMIN` | -| DeleteUser | foundation / `user.owner` | `USER_ADMIN` | -| CreateSubscribeUser (owner override) | foundation / sentinel | `USER_ADMIN` | - -Plus: the `perm_only` helper, integration tests per handler (positive Permission path, negative -no-account/insufficient/suspended, regression on existing paths), an IDL bump (one optional trailing -account per instruction), and a `PERMISSION.md` update. **Self-contained and shippable on its own** — -the new path is dormant until a caller passes a Permission account. - -One structural note for reviewers: because the Permission account is a *trailing* account, the final -authorization decision moves to just after each handler's required-account reads (before any state -mutation). Reads are non-destructive, so this is safe; it's a small restructure, not a logic change. - -### Unit 2 — oracle wiring (`malbeclabs/doublezero-shreds`) - -`dz_ledger.rs` appends the oracle's Permission PDA `AccountMeta` to the seven instructions it builds. -No behavior change until Unit 1 is deployed; the oracle continues to work via foundation in the -meantime. - -### Unit 3 — operational rollout - -`CreatePermission` for the oracle key with `ACCESS_PASS_ADMIN | USER_ADMIN`, then remove the oracle -from `foundation_allowlist`. **Sequencing matters** (we've had unauthorized-window incidents from -out-of-order authority changes before): - -1. Deploy Unit 1 (handlers honor Permission accounts). -2. Deploy Unit 2 (oracle passes its Permission account). -3. `CreatePermission` for the oracle key; **verify** the oracle operates end-to-end via the Permission - path while still in foundation. -4. Only then remove the oracle from `foundation_allowlist`. - -Doing step 4 before 1–3 are confirmed leaves the oracle unauthorized. - -## Security review framing - -- **Additive / behavior-preserving:** every existing caller (foundation, `user.owner`, pass - `user_payer`, `feed_authority` owns-it, sentinel, mgroup-owner) keeps its exact current path. The - only new capability is "a key with a valid `Activated` Permission PDA bearing the required bit." -- **Net privilege reduction:** the oracle goes from foundation (≈ all admin bits, incl. allowlist - edits and authority rotation) to exactly `ACCESS_PASS_ADMIN | USER_ADMIN`. Blast radius on oracle-key - compromise shrinks accordingly. -- **No owns-only gate on the Permission path** — required so the oracle can manage both - connection-ticket passes (which it owns) and validator-owned passes (which it does not). - -## Open questions for the team - -1. **Bit for `UpdateMulticastGroupRoles`:** proposed `ACCESS_PASS_ADMIN` (it's access-pass-gated). If - the team prefers `MULTICAST_ADMIN`, the oracle's bitmask changes to add it. -2. **`USER_ADMIN` semantics:** our `delete`/`create` changes make a Permission-account `USER_ADMIN` - holder able to delete/create users without an owns-it restriction (matching the oracle's - cross-owner cleanup job). Confirm that's the intended meaning of the bit for permission holders. -3. **`RequirePermissionAccounts`:** out of scope here (we stay additive and leave legacy paths intact). - Worth a separate decision on whether/when to flip the program-wide flag that retires legacy auth. -4. **`CreateSubscribeUser`:** plan-phase item — confirm there's no second foundation-only gate beyond - the owner-override check. - -## Issue / PR map - -- **#1652** — to be reframed from "feed-authority delete" to this Permission-PDA approach. -- **#1547** (parent) — Generalized Seat Buying. -- **doublezero-shreds #501** (connection-ticket cap eviction + zero-connection close) — the feature - that exercises the oracle's delete/unsubscribe/close; it currently relies on the oracle's foundation - membership. This least-privilege work is its security follow-up; #501 does not depend on it (it ships - on the foundation path in the interim). diff --git a/docs/superpowers/plans/2026-03-26-latency-command-ux.md b/docs/superpowers/plans/2026-03-26-latency-command-ux.md deleted file mode 100644 index 1bdc55e553..0000000000 --- a/docs/superpowers/plans/2026-03-26-latency-command-ux.md +++ /dev/null @@ -1,829 +0,0 @@ -# Latency Command UX Improvements Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Fix the daemon probe race condition, expose readiness state, and add spinner + better error messages to the CLI latency command. - -**Architecture:** The daemon's `LatencyManager` gets a `devicesFetched` channel so the probe goroutine waits for device data before its first run, and an `atomic.Bool` to track probe readiness. `ServeLatency` wraps the response in `{"ready": bool, "results": [...]}`. The CLI deserializes this new format, adds a spinner, and uses the `ready` flag to show accurate progress messages. - -**Tech Stack:** Go (daemon), Rust (CLI), `indicatif` (spinner), `sync/atomic` (readiness flag) - -**Spec:** `docs/superpowers/specs/2026-03-26-latency-command-ux-design.md` - ---- - -### Task 1: Daemon — Add `devicesFetched` channel and `probeReady` flag to LatencyManager - -**Files:** -- Modify: `client/doublezerod/internal/latency/manager.go:192-220` (struct + constructor) -- Test: `client/doublezerod/internal/latency/manager_test.go` - -- [ ] **Step 1: Write the failing test — probe waits for device fetch** - -Add a new test to `manager_test.go` that verifies the probe goroutine does not run before the device cache is populated. Use a slow smart contract func that takes 500ms, and verify that the first probe sees the devices (not an empty cache). - -```go -func TestLatencyManager_ProbeWaitsForDeviceFetch(t *testing.T) { - probeTargets := make(chan []latency.ProbeTarget, 1) - - slowSmartContractFunc := func(ctx context.Context) (*latency.ContractData, error) { - time.Sleep(500 * time.Millisecond) - return &latency.ContractData{ - Devices: []serviceability.Device{ - { - AccountType: serviceability.DeviceType, - PublicIp: [4]uint8{127, 0, 0, 1}, - PubKey: [32]byte{1}, - Code: "dev01", - }, - }, - }, nil - } - - mockProber := func(ctx context.Context, target latency.ProbeTarget) latency.LatencyResult { - probeTargets <- []latency.ProbeTarget{target} - return latency.LatencyResult{ - Min: 1, - Max: 10, - Avg: 5, - Loss: 0, - Device: target.Device, - IP: target.IP, - Reachable: true, - } - } - - manager := latency.NewLatencyManager( - latency.WithSmartContractFunc(slowSmartContractFunc), - latency.WithProberFunc(mockProber), - latency.WithProbeInterval(30*time.Second), - latency.WithCacheUpdateInterval(30*time.Second), - ) - - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - - go func() { - _ = manager.Start(ctx) - }() - - // The probe should have received actual targets (not empty), meaning it waited for fetch - select { - case targets := <-probeTargets: - if len(targets) == 0 { - t.Fatal("probe ran with empty targets — did not wait for device fetch") - } - case <-time.After(3 * time.Second): - t.Fatal("timed out waiting for probe to run") - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `go test -run TestLatencyManager_ProbeWaitsForDeviceFetch -v -count=1 ./client/doublezerod/internal/latency/...` - -Expected: The test may pass or fail depending on timing. If it passes, the race is hard to reproduce deterministically — that's OK, the structural fix is still needed. Proceed to the implementation. - -- [ ] **Step 3: Add `devicesFetched` channel and `probeReady` flag to LatencyManager** - -In `manager.go`, add two fields to the `LatencyManager` struct: - -```go -type LatencyManager struct { - SmartContractFunc SmartContractorFunc - fetcher Fetcher - proberFunc ProberFunc - DeviceCache *DeviceCache - ResultsCache *LatencyResults - probeInterval time.Duration - cacheUpdateInterval time.Duration - metricsEnabled bool - probeTunnelEndpoints bool - devicesFetched chan struct{} // closed after first successful fetch - probeReady atomic.Bool // true after first probe completes -} -``` - -Add `"sync/atomic"` to imports. - -Update `NewLatencyManager` to initialize the channel: - -```go -func NewLatencyManager(options ...Option) *LatencyManager { - lm := &LatencyManager{ - DeviceCache: &DeviceCache{Devices: []serviceability.Device{}, Lock: sync.Mutex{}}, - ResultsCache: &LatencyResults{Results: []LatencyResult{}, Lock: sync.RWMutex{}}, - proberFunc: UdpPing, - probeInterval: 10 * time.Second, - cacheUpdateInterval: 300 * time.Second, - metricsEnabled: false, - devicesFetched: make(chan struct{}), - } - for _, o := range options { - o(lm) - } - return lm -} -``` - -Add a public getter for readiness: - -```go -func (l *LatencyManager) IsProbeReady() bool { - return l.probeReady.Load() -} -``` - -- [ ] **Step 4: Update `Start()` — fetch goroutine closes channel, probe goroutine waits** - -In the fetch goroutine, close `devicesFetched` after the first successful fetch: - -```go -go func() { - fetch := func() { - // ... existing fetch logic unchanged ... - } - // don't wait for first tick and populate cache - fetch() - // Signal that initial device data is available for probing - select { - case <-l.devicesFetched: - // already closed - default: - close(l.devicesFetched) - } - - ticker := time.NewTicker(l.cacheUpdateInterval) - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - fetch() - } - } -}() -``` - -In the probe goroutine, wait for `devicesFetched` before the first probe: - -```go -go func() { - probe := func() { - // ... existing probe logic unchanged ... - } - - // Wait for initial device fetch before first probe - select { - case <-l.devicesFetched: - case <-ctx.Done(): - return - } - - // don't wait for first tick to ping stuff - probe() - l.probeReady.Store(true) - - ticker := time.NewTicker(l.probeInterval) - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - probe() - } - } -}() -``` - -- [ ] **Step 5: Run the test to verify it passes** - -Run: `go test -run TestLatencyManager_ProbeWaitsForDeviceFetch -v -count=1 ./client/doublezerod/internal/latency/...` - -Expected: PASS - -- [ ] **Step 6: Run all existing latency tests to verify no regressions** - -Run: `go test -v -count=1 ./client/doublezerod/internal/latency/...` - -Expected: All tests pass. - -- [ ] **Step 7: Commit** - -```bash -git add client/doublezerod/internal/latency/manager.go client/doublezerod/internal/latency/manager_test.go -git commit -m "client: add devicesFetched channel and probeReady flag to LatencyManager" -``` - ---- - -### Task 2: Daemon — Update `ServeLatency` response to include readiness - -**Files:** -- Modify: `client/doublezerod/internal/latency/manager.go:386-394` (ServeLatency handler) -- Test: `client/doublezerod/internal/latency/manager_test.go` - -- [ ] **Step 1: Write the failing test — HTTP response includes ready field** - -Add a test that verifies the `/latency` HTTP response includes the `ready` and `results` fields. Add this after the existing `check_results_via_http_are_correct` test in `TestLatencyManager`: - -```go -func TestServeLatency_ResponseFormat(t *testing.T) { - manager := latency.NewLatencyManager( - latency.WithSmartContractFunc(func(context.Context) (*latency.ContractData, error) { - return &latency.ContractData{ - Devices: []serviceability.Device{ - { - AccountType: serviceability.DeviceType, - PublicIp: [4]uint8{127, 0, 0, 1}, - PubKey: [32]byte{1}, - Code: "dev01", - }, - }, - }, nil - }), - latency.WithProberFunc(func(ctx context.Context, target latency.ProbeTarget) latency.LatencyResult { - return latency.LatencyResult{ - Min: 1, Max: 10, Avg: 5, Loss: 0, - Device: target.Device, IP: target.IP, Reachable: true, - } - }), - latency.WithProbeInterval(30*time.Second), - latency.WithCacheUpdateInterval(30*time.Second), - ) - - // Before Start — probe not ready, no results - f, err := os.CreateTemp("/tmp", "doublezero-test.sock") - if err != nil { - t.Fatal(err) - } - defer os.Remove(f.Name()) - _ = unix.Unlink(f.Name()) - - lis, err := net.Listen("unix", f.Name()) - if err != nil { - t.Fatal(err) - } - mux := http.NewServeMux() - mux.HandleFunc("GET /latency", manager.ServeLatency) - server := http.Server{Handler: mux} - defer server.Close() - go func() { _ = server.Serve(lis) }() - - client := http.Client{ - Transport: &http.Transport{ - DialContext: func(_ context.Context, _, _ string) (net.Conn, error) { - return net.Dial("unix", f.Name()) - }, - }, - } - - // Test: before probing, ready should be false - resp, err := client.Get("http://localhost/latency") - if err != nil { - t.Fatal(err) - } - defer resp.Body.Close() - buf, _ := io.ReadAll(resp.Body) - - var parsed struct { - Ready bool `json:"ready"` - Results []json.RawMessage `json:"results"` - } - if err := json.Unmarshal(buf, &parsed); err != nil { - t.Fatalf("failed to parse response as {ready, results}: %v\nbody: %s", err, buf) - } - if parsed.Ready { - t.Error("expected ready=false before probe has run") - } - - // Start the manager and wait for probe to complete - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - go func() { _ = manager.Start(ctx) }() - - // Poll until ready - deadline := time.Now().Add(3 * time.Second) - for time.Now().Before(deadline) { - if manager.IsProbeReady() { - break - } - time.Sleep(50 * time.Millisecond) - } - if !manager.IsProbeReady() { - t.Fatal("manager never became ready") - } - - // Test: after probing, ready should be true with results - resp2, err := client.Get("http://localhost/latency") - if err != nil { - t.Fatal(err) - } - defer resp2.Body.Close() - buf2, _ := io.ReadAll(resp2.Body) - - var parsed2 struct { - Ready bool `json:"ready"` - Results []json.RawMessage `json:"results"` - } - if err := json.Unmarshal(buf2, &parsed2); err != nil { - t.Fatalf("failed to parse response: %v\nbody: %s", err, buf2) - } - if !parsed2.Ready { - t.Error("expected ready=true after probe completed") - } - if len(parsed2.Results) == 0 { - t.Error("expected non-empty results after probe completed") - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `go test -run TestServeLatency_ResponseFormat -v -count=1 ./client/doublezerod/internal/latency/...` - -Expected: FAIL — the response is currently a bare JSON array, not `{"ready":..., "results":...}`. - -- [ ] **Step 3: Update `ServeLatency` to wrap response** - -In `manager.go`, replace the `ServeLatency` method: - -```go -// latencyResponse is the wire format for the /latency endpoint. -// This is internal to the daemon-CLI communication — not the user-facing output. -type latencyResponse struct { - Ready bool `json:"ready"` - Results *LatencyResults `json:"results"` -} - -func (l *LatencyManager) ServeLatency(w http.ResponseWriter, r *http.Request) { - resp := latencyResponse{ - Ready: l.probeReady.Load(), - Results: l.ResultsCache, - } - data, err := json.Marshal(resp) - if err != nil { - w.WriteHeader(http.StatusInternalServerError) - _, _ = fmt.Fprintf(w, "error generating latency: %v", err) - return - } - _, _ = w.Write(data) -} -``` - -- [ ] **Step 4: Update the existing `check_results_via_http_are_correct` test** - -The existing test in `TestLatencyManager` parses the response as `[]map[string]any`. Update it to expect the new wrapped format: - -```go -t.Run("check_results_via_http_are_correct", func(t *testing.T) { - req, err := http.NewRequest("GET", "http://localhost/latency", nil) - if err != nil { - t.Fatalf("error generating http request: %v", err) - } - resp, err := client.Do(req) - if err != nil { - t.Fatalf("error while making http request: %v", err) - } - defer resp.Body.Close() - - buf, _ := io.ReadAll(resp.Body) - var parsed struct { - Ready bool `json:"ready"` - Results []map[string]any `json:"results"` - } - if err := json.Unmarshal(buf, &parsed); err != nil { - t.Fatalf("error unmarshaling latency data: %v\nbody: %s", err, buf) - } - - if !parsed.Ready { - t.Error("expected ready=true") - } - - want := []map[string]any{ - { - "device_pk": base58.Encode(tests[0].DeviceCache[0].PubKey[:]), - "device_code": tests[0].DeviceCache[0].Code, - "device_ip": "127.0.0.1", - "min_latency_ns": float64(1), - "max_latency_ns": float64(10), - "avg_latency_ns": float64(5), - "loss_percentage": float64(0), - "reachable": true, - }, - } - - if diff := cmp.Diff(want, parsed.Results); diff != "" { - t.Errorf("LatencyResults mismatch (-want +got): %s\n", diff) - } -}) -``` - -- [ ] **Step 5: Run all latency tests to verify** - -Run: `go test -v -count=1 ./client/doublezerod/internal/latency/...` - -Expected: All tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add client/doublezerod/internal/latency/manager.go client/doublezerod/internal/latency/manager_test.go -git commit -m "client: update ServeLatency to include readiness in response" -``` - ---- - -### Task 3: CLI — Update `ServiceController::latency()` to return `LatencyResponse` - -**Files:** -- Modify: `client/doublezero/src/servicecontroller.rs:15-30,167-179,233-250` - -- [ ] **Step 1: Add `LatencyResponse` struct** - -In `servicecontroller.rs`, add the new struct after the `LatencyRecord` definition (after line 50): - -```rust -#[derive(Deserialize, Debug)] -pub struct LatencyResponse { - pub ready: bool, - pub results: Vec, -} -``` - -- [ ] **Step 2: Update `ServiceController` trait and impl** - -Change the trait method signature at line 173: - -```rust -async fn latency(&self) -> eyre::Result; -``` - -Change the implementation at line 233-250: - -```rust -async fn latency(&self) -> eyre::Result { - let uri = Uri::new(&self.socket_path, "/latency").into(); - let client: Client> = - Client::builder(TokioExecutor::new()).build(UnixConnector); - let res = client - .get(uri) - .await - .map_err(|e| eyre!("Unable to connect to doublezero daemon: {e}"))?; - - let data = res - .into_body() - .collect() - .await - .map_err(|e| eyre!("Unable to read response body: {e}"))? - .to_bytes(); - - parse_daemon_response::(&data, "/latency") -} -``` - -- [ ] **Step 3: Update the mock expectation return type** - -The `#[automock]` macro will auto-generate the mock. But all existing test code that calls `expect_latency()` returns `Ok(vec![...])`. These need to return `Ok(LatencyResponse { ready: true, results: vec![...] })`. - -This is handled in Task 5 (updating `dzd_latency.rs`). - -- [ ] **Step 4: Verify it compiles (expect test failures from mock changes)** - -Run: `cargo check -p doublezero` - -Expected: Compilation errors in `dzd_latency.rs` tests where `expect_latency().returning(...)` returns the old type. That's expected — Task 5 fixes those. - -- [ ] **Step 5: Commit** - -```bash -git add client/doublezero/src/servicecontroller.rs -git commit -m "client: update ServiceController::latency() to return LatencyResponse" -``` - ---- - -### Task 4: CLI — Add spinner to latency command - -**Files:** -- Modify: `client/doublezero/src/command/latency.rs` - -- [ ] **Step 1: Add spinner to latency command** - -Replace the entire `latency.rs` file content: - -```rust -use crate::command::util; -use clap::Args; -use doublezero_cli::doublezerocommand::CliCommand; -use doublezero_sdk::commands::device::list::ListDeviceCommand; -use indicatif::{ProgressBar, ProgressStyle}; -use std::time::Duration; - -use crate::{ - dzd_latency::retrieve_latencies, requirements::check_doublezero, - servicecontroller::ServiceControllerImpl, -}; - -#[derive(Args, Debug)] -pub struct LatencyCliCommand { - /// Output as json - #[arg(long, default_value = "false")] - json: bool, -} - -impl LatencyCliCommand { - pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { - let controller = ServiceControllerImpl::new(None); - - let spinner = ProgressBar::new_spinner(); - spinner.set_style( - ProgressStyle::default_spinner() - .template("{spinner:.green} [{elapsed_precise}] {msg}") - .expect("Failed to set template") - .tick_strings(&["-", "\\", "|", "/"]), - ); - spinner.enable_steady_tick(Duration::from_millis(100)); - spinner.set_message("Checking daemon..."); - - check_doublezero(&controller, client, Some(&spinner)).await?; - - spinner.set_message("Fetching devices..."); - let devices = client.list_device(ListDeviceCommand)?; - - let latencies = - retrieve_latencies(&controller, &devices, false, Some(&spinner)).await?; - - spinner.finish_and_clear(); - util::show_output(latencies, self.json)?; - - Ok(()) - } -} -``` - -- [ ] **Step 2: Verify it compiles** - -Run: `cargo check -p doublezero` - -Expected: May still have errors from Task 3 mock changes — that's OK if Task 3 is committed but Task 5 isn't yet. - -- [ ] **Step 3: Commit** - -```bash -git add client/doublezero/src/command/latency.rs -git commit -m "client: add spinner to latency command" -``` - ---- - -### Task 5: CLI — Update `retrieve_latencies` to use readiness flag - -**Files:** -- Modify: `client/doublezero/src/dzd_latency.rs:68-124` -- Test: `client/doublezero/src/dzd_latency.rs` (test module) - -- [ ] **Step 1: Update `retrieve_latencies` to use `LatencyResponse`** - -Replace the `retrieve_latencies` function (lines 68-124): - -```rust -pub async fn retrieve_latencies( - controller: &T, - devices: &HashMap, - reachable_only: bool, - spinner: Option<&indicatif::ProgressBar>, -) -> eyre::Result> { - if let Some(spinner) = spinner { - spinner.set_message("Retrieving latency stats..."); - } - - let max_wait = Duration::from_secs(60); - let poll_interval = Duration::from_secs(1); - let start = std::time::Instant::now(); - - let mut latencies = loop { - let response = controller.latency().await.map_err(|e| eyre::eyre!(e))?; - - let mut results = response.results; - results.retain(|l| { - Pubkey::from_str(&l.device_pk) - .ok() - .and_then(|pubkey| devices.get(&pubkey)) - .map(|device| device.status == DeviceStatus::Activated) - .unwrap_or(false) - }); - - if reachable_only { - results.retain(|l| l.reachable); - } - - if !results.is_empty() { - break results; - } - - // Daemon is still warming up — poll with feedback - if !response.ready { - if start.elapsed() >= max_wait { - eyre::bail!( - "Timed out waiting for daemon to finish probing devices. \ - The daemon may still be starting up — try again in a few seconds." - ); - } - if let Some(spinner) = spinner { - spinner.set_message("Waiting for daemon to finish probing devices..."); - } - tokio::time::sleep(poll_interval).await; - continue; - } - - // Daemon is ready but no results — this is a real "no devices" situation - eyre::bail!("No activated devices found"); - }; - - latencies.sort_by(|a, b| { - let reachable_cmp = b.reachable.cmp(&a.reachable); - if reachable_cmp != std::cmp::Ordering::Equal { - return reachable_cmp; - } - a.avg_latency_ns - .partial_cmp(&b.avg_latency_ns) - .unwrap_or(std::cmp::Ordering::Equal) - }); - - Ok(latencies) -} -``` - -- [ ] **Step 2: Update all test mock expectations to return `LatencyResponse`** - -In the test module, add the import: - -```rust -use crate::servicecontroller::LatencyResponse; -``` - -Then update every `expect_latency().returning(...)` call. Each test that does: - -```rust -controller - .expect_latency() - .returning(move || Ok(latencies.clone())); -``` - -Must become: - -```rust -controller - .expect_latency() - .returning(move || Ok(LatencyResponse { ready: true, results: latencies.clone() })); -``` - -Apply this to all tests: -- `test_retrieve_latencies_filters_and_sorts` -- `test_best_latency_prefers_current_within_tolerance` -- `test_best_latency_selects_lowest` -- `test_best_latency_ignores_unreachable_devices` -- `test_best_latency_ignores_faster_devices_at_max_users` -- `test_best_latency_current_faster_but_at_max_users` -- `test_best_latency_excludes_ips` -- `test_best_latency_excludes_specific_ip` -- `test_best_latency_device_with_multiple_endpoints_not_excluded` -- `test_best_latency_device_all_endpoints_excluded` -- `test_best_latency_prefers_same_device_with_available_endpoint` - -- [ ] **Step 3: Add a new test for the "not ready" polling behavior** - -```rust -#[tokio::test] -async fn test_retrieve_latencies_waits_for_daemon_ready() { - let (pk1, dev1) = make_device(DeviceStatus::Activated, 0); - let mut devices = HashMap::new(); - devices.insert(pk1, dev1); - - let latencies = vec![make_latency(&pk1.to_string(), 10000000, true)]; - let call_count = std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0)); - let call_count_clone = call_count.clone(); - let latencies_clone = latencies.clone(); - - let mut controller = MockServiceController::new(); - controller.expect_latency().returning(move || { - let count = call_count_clone.fetch_add(1, std::sync::atomic::Ordering::SeqCst); - if count < 2 { - // First two calls: not ready yet - Ok(LatencyResponse { - ready: false, - results: vec![], - }) - } else { - // Third call: ready with results - Ok(LatencyResponse { - ready: true, - results: latencies_clone.clone(), - }) - } - }); - - let result = retrieve_latencies(&controller, &devices, false, None) - .await - .unwrap(); - assert_eq!(result.len(), 1); - assert_eq!(result[0].device_pk, pk1.to_string()); - assert!(call_count.load(std::sync::atomic::Ordering::SeqCst) >= 3); -} -``` - -- [ ] **Step 4: Add a test for "ready but no devices" error** - -```rust -#[tokio::test] -async fn test_retrieve_latencies_ready_but_empty_returns_error() { - let devices = HashMap::new(); - - let mut controller = MockServiceController::new(); - controller.expect_latency().returning(move || { - Ok(LatencyResponse { - ready: true, - results: vec![], - }) - }); - - let result = retrieve_latencies(&controller, &devices, false, None).await; - assert!(result.is_err()); - assert_eq!( - result.unwrap_err().to_string(), - "No activated devices found" - ); -} -``` - -- [ ] **Step 5: Run all Rust tests** - -Run: `cargo test -p doublezero` - -Expected: All tests pass. - -- [ ] **Step 6: Remove unused `backon` import** - -The `retrieve_latencies` function no longer uses `backon`'s retry logic. Remove the import from the top of `dzd_latency.rs`: - -```rust -// Remove this line: -use backon::{ExponentialBuilder, Retryable}; -``` - -Also add the `std::time::Instant` import (used by the new polling loop, `Instant` was not previously imported): - -```rust -use std::{collections::HashMap, net::Ipv4Addr, str::FromStr, time::Duration}; -``` - -Note: `std::time::Instant` is used as `std::time::Instant::now()` in the function body, so the fully qualified path is fine without a dedicated import. Alternatively add it to the existing `use std::` line. - -- [ ] **Step 7: Run formatting** - -Run: `make rust-fmt` - -- [ ] **Step 8: Run all Rust tests** - -Run: `cargo test -p doublezero` - -Expected: All tests pass. - -- [ ] **Step 9: Commit** - -```bash -git add client/doublezero/src/dzd_latency.rs -git commit -m "client: update retrieve_latencies to use daemon readiness flag" -``` - ---- - -### Task 6: Final verification - -**Files:** None (verification only) - -- [ ] **Step 1: Run full Go test suite for the daemon** - -Run: `go test -v -count=1 ./client/doublezerod/internal/latency/...` - -Expected: All tests pass. - -- [ ] **Step 2: Run full Rust test suite for the CLI** - -Run: `cargo test -p doublezero` - -Expected: All tests pass. - -- [ ] **Step 3: Run lint for both** - -Run: `make rust-lint` and `make go-lint` - -Expected: No lint errors. - -- [ ] **Step 4: Run formatting for both** - -Run: `make rust-fmt` and `make go-fmt` - -Expected: No formatting changes needed. diff --git a/docs/superpowers/plans/2026-04-03-status-multicast-groups.md b/docs/superpowers/plans/2026-04-03-status-multicast-groups.md deleted file mode 100644 index 81bf55211d..0000000000 --- a/docs/superpowers/plans/2026-04-03-status-multicast-groups.md +++ /dev/null @@ -1,419 +0,0 @@ -# Status Command: Multicast Group Memberships — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Show which multicast groups a user publishes to / subscribes to in the `doublezero status` command output. - -**Architecture:** Extend the Go daemon's `V2ServiceStatus` with a `multicast_groups` object containing `publisher` and `subscriber` string arrays (group codes). The `enrichStatuses()` function already has access to the matched user and onchain multicast group data — it just needs to resolve the user's pubkey vecs to group codes. The Rust CLI then deserializes and displays the new field. - -**Tech Stack:** Go (daemon), Rust (CLI), serde, tabled - ---- - -## File Structure - -| Action | File | Responsibility | -|--------|------|---------------| -| Modify | `client/doublezerod/internal/manager/http.go` | Add `MulticastGroups` struct and field to `V2ServiceStatus`; populate in `enrichStatuses()` | -| Modify | `client/doublezerod/internal/manager/reconciler_test.go` | Add multicast group assertions to `TestServeV2Status_Enrichment` | -| Modify | `client/doublezero/src/servicecontroller.rs` | Add `MulticastGroups` struct and field to `V2ServiceStatus` | -| Modify | `client/doublezero/src/command/status.rs` | Add multicast groups column to display; format as `P:code,S:code` | - ---- - -### Task 1: Go daemon — add MulticastGroups to V2ServiceStatus and populate in enrichStatuses - -**Files:** -- Modify: `client/doublezerod/internal/manager/http.go:17-26` (struct definitions) -- Modify: `client/doublezerod/internal/manager/http.go:226-367` (enrichStatuses) -- Modify: `client/doublezerod/internal/manager/reconciler_test.go:1205-1397` (TestServeV2Status_Enrichment) - -- [ ] **Step 1: Write the failing test** - -In `client/doublezerod/internal/manager/reconciler_test.go`, update the `wantService` struct and test cases in `TestServeV2Status_Enrichment` to assert on multicast groups. - -First, add `Code` to the existing `mcastGroup` fixture (around line 1218): - -```go -mcastGroup := serviceability.MulticastGroup{ - PubKey: mcastGroupPK, - MulticastIp: [4]uint8{239, 0, 0, 1}, - Code: "solana-ams", -} -``` - -Then add fields to the `wantService` struct (around line 1223): - -```go -type wantService struct { - userType string - currentDevice string - metro string - tenant string - hasDzIP bool - pubGroups []string - subGroups []string -} -``` - -Update each test case's `want` entries. For `ibrl_only`: -```go -want: []wantService{ - {userType: "IBRL", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: nil, subGroups: nil}, -}, -``` - -For `multicast_publisher`: -```go -want: []wantService{ - {userType: "Multicast", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: []string{"solana-ams"}, subGroups: nil}, -}, -``` - -For `multicast_subscriber`: -```go -want: []wantService{ - {userType: "Multicast", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: false, pubGroups: nil, subGroups: []string{"solana-ams"}}, -}, -``` - -For `ibrl_plus_multicast_subscriber`: -```go -want: []wantService{ - {userType: "IBRL", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: nil, subGroups: nil}, - {userType: "Multicast", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: false, pubGroups: nil, subGroups: []string{"solana-ams"}}, -}, -``` - -For `ibrl_plus_multicast_publisher`: -```go -want: []wantService{ - {userType: "IBRL", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: nil, subGroups: nil}, - {userType: "Multicast", currentDevice: "dz1", metro: "Amsterdam", tenant: "acme", hasDzIP: true, pubGroups: []string{"solana-ams"}, subGroups: nil}, -}, -``` - -Add assertions at the end of the test loop (after the existing `hasDzIP` check, around line 1394): - -```go -if !slices.Equal(svc.MulticastGroups.Publisher, w.pubGroups) { - t.Errorf("[%s] expected pub groups %v, got %v", w.userType, w.pubGroups, svc.MulticastGroups.Publisher) -} -if !slices.Equal(svc.MulticastGroups.Subscriber, w.subGroups) { - t.Errorf("[%s] expected sub groups %v, got %v", w.userType, w.subGroups, svc.MulticastGroups.Subscriber) -} -``` - -Add `"slices"` to the import block if not already present. - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd /Users/ben/src/malbec/doublezero && go test -run TestServeV2Status_Enrichment -v ./client/doublezerod/internal/manager/...` - -Expected: Compilation error — `V2ServiceStatus` has no `MulticastGroups` field. - -- [ ] **Step 3: Implement the Go daemon changes** - -In `client/doublezerod/internal/manager/http.go`, add the `MulticastGroups` struct and field. - -After the existing `V2ServiceStatus` struct (line 26), add: - -```go -// MulticastGroups contains the group codes a user publishes to and subscribes to. -type MulticastGroups struct { - Publisher []string `json:"publisher"` - Subscriber []string `json:"subscriber"` -} -``` - -Add the field to `V2ServiceStatus`: - -```go -type V2ServiceStatus struct { - *api.StatusResponse - CurrentDevice string `json:"current_device"` - CurrentDeviceRttNanoseconds int64 `json:"current_device_rtt_nanoseconds,omitempty"` - CurrentDeviceLossPercentage float64 `json:"current_device_loss_percentage,omitempty"` - LowestLatencyDevice string `json:"lowest_latency_device"` - Metro string `json:"metro"` - Tenant string `json:"tenant"` - MulticastGroups MulticastGroups `json:"multicast_groups"` -} -``` - -In `enrichStatuses()`, build a multicast group lookup map. After the existing `tenantsByPK` map (around line 266), add: - -```go -mcastGroupsByPK := make(map[[32]byte]serviceability.MulticastGroup, len(data.MulticastGroups)) -for _, mg := range data.MulticastGroups { - mcastGroupsByPK[mg.PubKey] = mg -} -``` - -After the tenant enrichment block (after line 356, before the lowest latency computation), add: - -```go -if matchedUser != nil { - for _, pk := range matchedUser.Publishers { - if mg, ok := mcastGroupsByPK[pk]; ok { - es.MulticastGroups.Publisher = append(es.MulticastGroups.Publisher, mg.Code) - } - } - for _, pk := range matchedUser.Subscribers { - if mg, ok := mcastGroupsByPK[pk]; ok { - es.MulticastGroups.Subscriber = append(es.MulticastGroups.Subscriber, mg.Code) - } - } -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd /Users/ben/src/malbec/doublezero && go test -run TestServeV2Status_Enrichment -v ./client/doublezerod/internal/manager/...` - -Expected: PASS - -- [ ] **Step 5: Run full Go test suite for the package** - -Run: `cd /Users/ben/src/malbec/doublezero && go test -v ./client/doublezerod/internal/manager/...` - -Expected: All tests pass. The new field serializes as empty arrays `{"publisher":null,"subscriber":null}` for non-multicast users, which is fine — the Rust side uses `#[serde(default)]`. - -- [ ] **Step 6: Commit** - -``` -client/daemon: add multicast groups to v2 status response -``` - ---- - -### Task 2: Rust CLI — deserialize and display multicast groups - -**Files:** -- Modify: `client/doublezero/src/servicecontroller.rs:149-161` (V2ServiceStatus struct) -- Modify: `client/doublezero/src/command/status.rs:20-36` (AppendedStatusResponse struct) -- Modify: `client/doublezero/src/command/status.rs:51-131` (command_impl) - -- [ ] **Step 1: Write the failing test** - -In `client/doublezero/src/command/status.rs`, add a new test for multicast group display. Add after the existing `test_status_command_multicast_subscriber` test (after line 354): - -```rust -#[tokio::test] -async fn test_status_command_multicast_groups_display() { - let mock_command = MockCliCommand::new(); - let mut mock_controller = MockServiceController::new(); - - mock_controller.expect_v2_status().returning(|| { - Ok(V2StatusResponse { - reconciler_enabled: true, - client_ip: String::new(), - network: "testnet".to_string(), - services: vec![V2ServiceStatus { - status: StatusResponse { - doublezero_status: DoubleZeroStatus { - session_status: "BGP Session Up".to_string(), - last_session_update: Some(1625247600), - }, - tunnel_name: Some("doublezero1".to_string()), - tunnel_src: Some("10.10.10.10".to_string()), - tunnel_dst: Some("5.6.7.8".to_string()), - doublezero_ip: None, - user_type: Some("Multicast".to_string()), - }, - current_device: "device1".to_string(), - lowest_latency_device: "device1".to_string(), - metro: "metro".to_string(), - tenant: String::new(), - multicast_groups: MulticastGroups { - publisher: vec!["solana-lv".to_string()], - subscriber: vec!["solana-ams".to_string()], - }, - }], - }) - }); - - let result = StatusCliCommand { json: true } - .command_impl(&mock_command, &mock_controller) - .await; - - assert!(result.is_ok()); - let result = result.unwrap(); - assert_eq!(result.len(), 1); - assert_eq!(result[0].multicast_groups, "P:solana-lv,S:solana-ams"); -} -``` - -Also add a test for backward compatibility (daemon doesn't send the field): - -```rust -#[test] -fn test_multicast_groups_serde_default() { - let json = r#"{ - "doublezero_status": {"session_status": "BGP Session Up", "last_session_update": null}, - "tunnel_name": null, "tunnel_src": null, "tunnel_dst": null, - "doublezero_ip": null, "user_type": "IBRL", - "current_device": "dz1", "lowest_latency_device": "dz1", - "metro": "ams", "tenant": "" - }"#; - let svc: V2ServiceStatus = serde_json::from_str(json).unwrap(); - assert!(svc.multicast_groups.publisher.is_empty()); - assert!(svc.multicast_groups.subscriber.is_empty()); -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd /Users/ben/src/malbec/doublezero && cargo test -p doublezero test_status_command_multicast_groups_display` - -Expected: Compilation error — `MulticastGroups` type doesn't exist, `V2ServiceStatus` has no `multicast_groups` field. - -- [ ] **Step 3: Add MulticastGroups struct to servicecontroller.rs** - -In `client/doublezero/src/servicecontroller.rs`, add the struct before `V2ServiceStatus` (before line 149): - -```rust -#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq)] -pub struct MulticastGroups { - #[serde(default)] - pub publisher: Vec, - #[serde(default)] - pub subscriber: Vec, -} -``` - -Add the field to `V2ServiceStatus` (after the `tenant` field): - -```rust -#[derive(Serialize, Deserialize, Debug, Clone)] -pub struct V2ServiceStatus { - #[serde(flatten)] - pub status: StatusResponse, - #[serde(default)] - pub current_device: String, - #[serde(default)] - pub lowest_latency_device: String, - #[serde(default)] - pub metro: String, - #[serde(default)] - pub tenant: String, - #[serde(default)] - pub multicast_groups: MulticastGroups, -} -``` - -- [ ] **Step 4: Add multicast_groups to AppendedStatusResponse and format in command_impl** - -In `client/doublezero/src/command/status.rs`, add the field to `AppendedStatusResponse`: - -```rust -#[derive(Tabled, Debug, Deserialize, Serialize)] -struct AppendedStatusResponse { - #[tabled(inline)] - response: StatusResponse, - #[tabled(rename = "Reconciler")] - reconciler_enabled: bool, - #[tabled(rename = "Tenant")] - tenant: String, - #[tabled(rename = "Current Device")] - current_device: String, - #[tabled(rename = "Lowest Latency Device")] - lowest_latency_device: String, - #[tabled(rename = "Metro")] - metro: String, - #[tabled(rename = "Network")] - network: String, - #[tabled(rename = "Multicast Groups")] - multicast_groups: String, -} -``` - -Add `use crate::servicecontroller::MulticastGroups;` to the imports at the top of the file (add to the existing `use crate::servicecontroller::{...}` import). - -Add a helper function to format multicast groups (before the `impl StatusCliCommand` block): - -```rust -fn format_multicast_groups(groups: &MulticastGroups) -> String { - let mut parts = Vec::new(); - for code in &groups.publisher { - parts.push(format!("P:{code}")); - } - for code in &groups.subscriber { - parts.push(format!("S:{code}")); - } - parts.join(",") -} -``` - -In `command_impl`, populate the new field when building each `AppendedStatusResponse`. In the empty-services branch (around line 62), add `multicast_groups: String::new()` to the struct literal. - -In the main loop (around line 119), add the field: - -```rust -responses.push(AppendedStatusResponse { - response: svc.status.clone(), - reconciler_enabled: v2_status.reconciler_enabled, - current_device, - lowest_latency_device, - metro, - network: network.clone(), - tenant: svc.tenant.clone(), - multicast_groups: format_multicast_groups(&svc.multicast_groups), -}); -``` - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `cd /Users/ben/src/malbec/doublezero && cargo test -p doublezero test_status_command_multicast_groups_display test_multicast_groups_serde_default` - -Expected: PASS - -- [ ] **Step 6: Fix existing tests** - -The existing tests in `status.rs` need the new `multicast_groups` field added to `V2ServiceStatus` struct literals and `AppendedStatusResponse` assertions. For each existing test that constructs a `V2ServiceStatus`, add: - -```rust -multicast_groups: MulticastGroups::default(), -``` - -For tests using the `make_v2_service` helper, update the helper to include the field: - -```rust -fn make_v2_service( - // ... existing params ... -) -> V2ServiceStatus { - V2ServiceStatus { - status: StatusResponse { ... }, - current_device: current_device.to_string(), - lowest_latency_device: lowest_latency_device.to_string(), - metro: metro.to_string(), - tenant: tenant.to_string(), - multicast_groups: MulticastGroups::default(), - } -} -``` - -For tests that assert on `AppendedStatusResponse` fields (like `test_status_json_output_format`), add `multicast_groups: String::new()` to the struct literal and add a JSON field assertion: - -```rust -assert!( - status.get("multicast_groups").is_some(), - "Missing 'multicast_groups' field" -); -``` - -- [ ] **Step 7: Run full test suite** - -Run: `cd /Users/ben/src/malbec/doublezero && cargo test -p doublezero` - -Expected: All tests pass. - -- [ ] **Step 8: Format** - -Run: `cd /Users/ben/src/malbec/doublezero && make rust-fmt` - -- [ ] **Step 9: Commit** - -``` -client/cli: show multicast groups in status command -``` diff --git a/docs/superpowers/plans/2026-04-09-startup-tunnel-endpoint-registration.md b/docs/superpowers/plans/2026-04-09-startup-tunnel-endpoint-registration.md deleted file mode 100644 index 1c74d7562c..0000000000 --- a/docs/superpowers/plans/2026-04-09-startup-tunnel-endpoint-registration.md +++ /dev/null @@ -1,185 +0,0 @@ -# Fix Startup Tunnel Endpoint Registration - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Prevent duplicate GRE tunnel pairs by registering implicit tunnel endpoints (device `public_ip`) during activator startup reconstruction. - -**Architecture:** The activator's `reserve_user_allocations` rebuilds in-memory state from existing onchain users at startup. It currently only registers explicit `tunnel_endpoint` values, missing users that fall back to the device's `public_ip`. This causes `get_available_tunnel_endpoint` to hand out already-in-use endpoints when a second user with the same `client_ip` arrives on the same device. - -**Tech Stack:** Rust (activator crate) - ---- - -### Task 1: Add regression test for startup tunnel endpoint registration - -**Files:** -- Modify: `activator/src/processor.rs:534-629` (test module, after existing `test_updating_user_allocations_must_be_reserved_at_startup`) - -- [ ] **Step 1: Write failing test** - -Add a test that creates two users with the same `client_ip` on the same device — the first with `tunnel_endpoint = UNSPECIFIED` (simulating an existing activated user that uses the device's `public_ip` implicitly). After `reserve_user_allocations`, call `get_available_tunnel_endpoint` for that `client_ip` and assert it does NOT return the device's `public_ip` (since it's already in use). - -```rust -/// Regression test: reserve_user_allocations must register implicit tunnel endpoints -/// (device public_ip) for users with unspecified tunnel_endpoint. Otherwise a second -/// user from the same client_ip gets the same endpoint, creating a duplicate GRE tunnel pair. -#[test] -fn test_implicit_tunnel_endpoint_reserved_at_startup() { - use crate::{ipblockallocator::IPBlockAllocator, states::devicestate::DeviceState}; - use doublezero_sdk::{ - AccountType, Device, DeviceStatus, DeviceType, User, UserStatus, UserType, - }; - use doublezero_serviceability::state::user::UserCYOA; - use std::net::Ipv4Addr; - - let device_pubkey = Pubkey::new_unique(); - let device = Device { - account_type: AccountType::Device, - owner: Pubkey::new_unique(), - index: 0, - reference_count: 0, - bump_seed: 0, - contributor_pk: Pubkey::new_unique(), - location_pk: Pubkey::new_unique(), - exchange_pk: Pubkey::new_unique(), - device_type: DeviceType::Hybrid, - public_ip: [88, 216, 220, 195].into(), - status: DeviceStatus::Activated, - metrics_publisher_pk: Pubkey::default(), - code: "TestDevice".to_string(), - dz_prefixes: "10.0.0.0/24".parse().unwrap(), - mgmt_vrf: "default".to_string(), - interfaces: vec![], - max_users: 255, - users_count: 0, - device_health: doublezero_serviceability::state::device::DeviceHealth::ReadyForUsers, - desired_status: - doublezero_serviceability::state::device::DeviceDesiredStatus::Activated, - unicast_users_count: 0, - multicast_subscribers_count: 0, - max_unicast_users: 0, - max_multicast_subscribers: 0, - reserved_seats: 0, - multicast_publishers_count: 0, - max_multicast_publishers: 0, - }; - - let client_ip: Ipv4Addr = [64, 130, 32, 201].into(); - - let mut device_map: DeviceMap = DeviceMap::new(); - device_map.insert(device_pubkey, DeviceState::new(&device)); - - // Existing activated user with unspecified tunnel_endpoint (uses device public_ip implicitly) - let existing_user = User { - account_type: AccountType::User, - owner: Pubkey::new_unique(), - index: 0, - bump_seed: 0, - user_type: UserType::IBRL, - tenant_pk: Pubkey::new_unique(), - device_pk: device_pubkey, - cyoa_type: UserCYOA::GREOverDIA, - client_ip, - dz_ip: [10, 0, 0, 1].into(), - tunnel_id: 509, - tunnel_net: "169.254.2.14/31".parse().unwrap(), - status: UserStatus::Activated, - publishers: vec![], - subscribers: vec![], - validator_pubkey: Pubkey::default(), - tunnel_endpoint: Ipv4Addr::UNSPECIFIED, - tunnel_flags: 0, - bgp_status: Default::default(), - last_bgp_up_at: 0, - last_bgp_reported_at: 0, - }; - - let mut users: HashMap = HashMap::new(); - users.insert(Pubkey::new_unique(), existing_user); - - let mut user_tunnel_ips = IPBlockAllocator::new("169.254.0.0/16".parse().unwrap()); - let mut publisher_dz_ips = IPBlockAllocator::new("148.51.120.0/21".parse().unwrap()); - - reserve_user_allocations(&users, &mut device_map, &mut user_tunnel_ips, &mut publisher_dz_ips) - .expect("reserve_user_allocations should succeed"); - - // A second user from the same client_ip should NOT get device public_ip - // because the first user is already using it - let device_state = device_map.get(&device_pubkey).unwrap(); - let next_endpoint = device_state.get_available_tunnel_endpoint(client_ip); - assert_eq!( - next_endpoint, None, - "BUG: get_available_tunnel_endpoint returned device public_ip which is already \ - in use by existing user with unspecified tunnel_endpoint — this creates a \ - duplicate GRE tunnel pair" - ); -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cargo test -p doublezero-activator test_implicit_tunnel_endpoint_reserved_at_startup` - -Expected: FAIL — the assertion fails because `get_available_tunnel_endpoint` returns `Some(88.216.220.195)` (the device's `public_ip`), since `reserve_user_allocations` didn't register it. - -### Task 2: Fix startup registration to include implicit endpoints - -**Files:** -- Modify: `activator/src/processor.rs:146-149` - -- [ ] **Step 3: Implement the fix** - -In `reserve_user_allocations`, change the tunnel endpoint registration to always register the effective endpoint — either the explicit `tunnel_endpoint` or the device's `public_ip`: - -```rust - // Register tunnel endpoint (explicit or implicit via device public_ip) - // so that get_available_tunnel_endpoint knows what's already in use. - let effective_endpoint = if user.has_tunnel_endpoint() { - user.tunnel_endpoint - } else { - device_state.device.public_ip - }; - device_state.register_tunnel_endpoint(user.client_ip, effective_endpoint); -``` - -This replaces the current code at lines 146-149: -```rust - // Register tunnel endpoint if set - if user.has_tunnel_endpoint() { - device_state.register_tunnel_endpoint(user.client_ip, user.tunnel_endpoint); - } -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cargo test -p doublezero-activator test_implicit_tunnel_endpoint_reserved_at_startup` - -Expected: PASS - -- [ ] **Step 5: Run full activator test suite** - -Run: `cargo test -p doublezero-activator` - -Expected: All tests pass. - -- [ ] **Step 6: Run formatting and lint** - -Run: `make rust-fmt && make rust-lint` - -Expected: Clean. - -- [ ] **Step 7: Commit** - -```bash -git add activator/src/processor.rs -git commit -m "activator: register implicit tunnel endpoints at startup - -When rebuilding in-memory state from existing onchain users, -register the device's public_ip as the effective tunnel endpoint -for users with unspecified tunnel_endpoint. Previously only -explicit tunnel endpoints were registered, allowing -get_available_tunnel_endpoint to hand out already-in-use -endpoints when a second user with the same client_ip arrived -on the same device — creating duplicate GRE tunnel pairs that -the controller had to deduplicate by dropping one tunnel." -``` diff --git a/docs/superpowers/plans/2026-04-23-multicast-modify-without-disconnect.md b/docs/superpowers/plans/2026-04-23-multicast-modify-without-disconnect.md deleted file mode 100644 index ec04c4df1d..0000000000 --- a/docs/superpowers/plans/2026-04-23-multicast-modify-without-disconnect.md +++ /dev/null @@ -1,1476 +0,0 @@ -# Multicast: modify subscriptions without disconnecting — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add four CLI subcommands under `doublezero multicast` — `subscribe`, `unsubscribe`, `publish`, `unpublish` — so a connected user can modify their multicast role set without running `disconnect`. - -**Architecture:** Extend the client CLI only. Each verb resolves group codes to pubkeys, loads the caller's existing Multicast user, and issues one `UpdateMulticastGroupRolesCommand` per group with the correct boolean flags (carrying through the other role's current value). The smartcontract is unchanged; the daemon reconciles multicast routes asynchronously on its next poll. - -**Tech Stack:** Rust (clap, eyre, mockall tests), Go (e2e tests via testcontainers + cEOS). - -**Spec:** `docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md` - ---- - -## File Structure - -| Action | File | Responsibility | -|--------|------|---------------| -| Modify | `client/doublezero/src/cli/multicast.rs` | Add four new `MulticastCommands` variants + arg structs | -| Create | `client/doublezero/src/command/multicast.rs` | Handlers for the four verbs + shared helpers | -| Modify | `client/doublezero/src/command/mod.rs` | Register new `multicast` module | -| Modify | `client/doublezero/src/main.rs` | Dispatch new variants under `Command::Multicast` | -| Modify | `e2e/main_test.go` | Add `TestDevnet` helpers for the four new verbs | -| Modify | `e2e/multicast_test.go` | Extend `TestE2E_Multicast` with unsubscribe/unpublish sub-tests | - ---- - -### Task 1: Extend `MulticastCommands` with four new variants - -**Files:** -- Modify: `client/doublezero/src/cli/multicast.rs` (entire file is 16 lines) - -- [ ] **Step 1: Replace the file with the expanded enum** - -Overwrite `client/doublezero/src/cli/multicast.rs` with: - -```rust -use clap::{Args, Subcommand}; - -use super::multicastgroup::MulticastGroupCliCommand; - -#[derive(Args, Debug)] -pub struct MulticastCliCommand { - #[command(subcommand)] - pub command: MulticastCommands, -} - -#[derive(Debug, Subcommand)] -pub enum MulticastCommands { - /// Manage multicast groups - #[clap()] - Group(MulticastGroupCliCommand), - /// Subscribe to one or more multicast groups (user must already be connected) - #[clap()] - Subscribe(MulticastSubscribeCliCommand), - /// Unsubscribe from one or more multicast groups - #[clap()] - Unsubscribe(MulticastUnsubscribeCliCommand), - /// Publish to one or more multicast groups (user must already be connected) - #[clap()] - Publish(MulticastPublishCliCommand), - /// Stop publishing to one or more multicast groups - #[clap()] - Unpublish(MulticastUnpublishCliCommand), -} - -#[derive(Args, Debug)] -pub struct MulticastSubscribeCliCommand { - /// Multicast group code(s) to subscribe to - #[arg(num_args = 1..)] - pub groups: Vec, -} - -#[derive(Args, Debug)] -pub struct MulticastUnsubscribeCliCommand { - /// Multicast group code(s) to unsubscribe from - #[arg(num_args = 1..)] - pub groups: Vec, -} - -#[derive(Args, Debug)] -pub struct MulticastPublishCliCommand { - /// Multicast group code(s) to publish to - #[arg(num_args = 1..)] - pub groups: Vec, -} - -#[derive(Args, Debug)] -pub struct MulticastUnpublishCliCommand { - /// Multicast group code(s) to stop publishing to - #[arg(num_args = 1..)] - pub groups: Vec, -} -``` - -- [ ] **Step 2: Verify clap parsing compiles and `--help` lists the new verbs** - -Run: - -```bash -cargo check -p doublezero -``` - -Expected: compiles (may warn about unused `MulticastSubscribeCliCommand` etc. until Task 7 — that's fine). - -- [ ] **Step 3: Commit** - -```bash -git add client/doublezero/src/cli/multicast.rs -git commit -m "client/doublezero: add multicast subscribe/unsubscribe/publish/unpublish CLI variants" -``` - ---- - -### Task 2: Create command module skeleton with shared helpers - -**Files:** -- Create: `client/doublezero/src/command/multicast.rs` -- Modify: `client/doublezero/src/command/mod.rs` - -- [ ] **Step 1: Add module registration** - -Edit `client/doublezero/src/command/mod.rs` to add `pub mod multicast;` alphabetically. The file should then read: - -```rust -pub mod connect; -pub mod disable; -pub mod disconnect; -pub mod enable; -pub mod helpers; -pub mod latency; -pub mod multicast; -pub mod routes; -pub mod status; -pub mod util; -``` - -- [ ] **Step 2: Write the failing tests** - -Create `client/doublezero/src/command/multicast.rs`: - -```rust -use std::net::Ipv4Addr; - -use doublezero_cli::doublezerocommand::CliCommand; -use doublezero_sdk::{ - commands::{ - multicastgroup::list::ListMulticastGroupCommand, user::list::ListUserCommand, - }, - User, UserType, -}; -use solana_sdk::pubkey::Pubkey; - -/// Resolve a list of multicast group codes to their on-chain pubkeys. -/// Errors on any unknown code, with no onchain writes. -pub(super) fn resolve_groups( - client: &dyn CliCommand, - codes: &[String], -) -> eyre::Result> { - let mcast_groups = client.list_multicastgroup(ListMulticastGroupCommand)?; - let mut out = Vec::with_capacity(codes.len()); - for code in codes { - let (pk, _) = mcast_groups - .iter() - .find(|(_, g)| g.code == *code) - .ok_or_else(|| eyre::eyre!("Multicast group not found: {code}"))?; - out.push((code.clone(), *pk)); - } - Ok(out) -} - -/// Load the Multicast user for the given client_ip. Errors if none exists. -pub(super) fn load_multicast_user( - client: &dyn CliCommand, - client_ip: Ipv4Addr, -) -> eyre::Result<(Pubkey, User)> { - let users = client.list_user(ListUserCommand)?; - users - .into_iter() - .find(|(_, u)| u.client_ip == client_ip && u.user_type == UserType::Multicast) - .ok_or_else(|| { - eyre::eyre!( - "No active multicast user for {client_ip}. \ - Run 'doublezero connect Multicast --publish/--subscribe ' first." - ) - }) -} - -#[cfg(test)] -mod tests { - use super::*; - use doublezero_cli::tests::utils::create_test_client; - use doublezero_sdk::{AccountType, MulticastGroup, MulticastGroupStatus, User, UserCYOA, UserStatus}; - use std::collections::HashMap; - - fn make_user(client_ip: Ipv4Addr, user_type: UserType) -> User { - User { - account_type: AccountType::User, - owner: Pubkey::new_unique(), - index: 0, - bump_seed: 0, - user_type, - tenant_pk: Pubkey::default(), - device_pk: Pubkey::default(), - cyoa_type: UserCYOA::None, - client_ip, - dz_ip: Ipv4Addr::UNSPECIFIED, - tunnel_id: 0, - tunnel_net: Default::default(), - status: UserStatus::Activated, - publishers: vec![], - subscribers: vec![], - validator_pubkey: Pubkey::default(), - tunnel_endpoint: Ipv4Addr::UNSPECIFIED, - tunnel_flags: 0, - bgp_status: Default::default(), - last_bgp_up_at: 0, - last_bgp_reported_at: 0, - } - } - - fn make_group(code: &str) -> MulticastGroup { - MulticastGroup { - account_type: AccountType::MulticastGroup, - owner: Pubkey::default(), - index: 0, - bump_seed: 0, - tenant_pk: Pubkey::default(), - code: code.to_string(), - max_bandwidth: 0, - status: MulticastGroupStatus::Activated, - multicast_ip: Ipv4Addr::UNSPECIFIED, - publisher_count: 0, - subscriber_count: 0, - } - } - - #[test] - fn resolve_groups_returns_pubkeys_in_order() { - let mut client = create_test_client(); - let g1_pk = Pubkey::new_unique(); - let g2_pk = Pubkey::new_unique(); - let mut groups = HashMap::new(); - groups.insert(g1_pk, make_group("g1")); - groups.insert(g2_pk, make_group("g2")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - let out = resolve_groups(&client, &["g2".into(), "g1".into()]).unwrap(); - assert_eq!(out, vec![("g2".into(), g2_pk), ("g1".into(), g1_pk)]); - } - - #[test] - fn resolve_groups_errors_on_unknown_code() { - let mut client = create_test_client(); - let g1_pk = Pubkey::new_unique(); - let mut groups = HashMap::new(); - groups.insert(g1_pk, make_group("g1")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - let err = resolve_groups(&client, &["nope".into()]).unwrap_err(); - assert!( - err.to_string().contains("Multicast group not found: nope"), - "unexpected error: {err}" - ); - } - - #[test] - fn load_multicast_user_finds_user_for_client_ip() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let mut client = create_test_client(); - let user_pk = Pubkey::new_unique(); - let user = make_user(ip, UserType::Multicast); - let mut users = HashMap::new(); - users.insert(user_pk, user.clone()); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let (pk, loaded) = load_multicast_user(&client, ip).unwrap(); - assert_eq!(pk, user_pk); - assert_eq!(loaded.client_ip, ip); - assert_eq!(loaded.user_type, UserType::Multicast); - } - - #[test] - fn load_multicast_user_errors_when_only_ibrl_user_exists() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let mut client = create_test_client(); - let mut users = HashMap::new(); - users.insert(Pubkey::new_unique(), make_user(ip, UserType::IBRL)); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let err = load_multicast_user(&client, ip).unwrap_err(); - assert!( - err.to_string().contains("No active multicast user"), - "unexpected error: {err}" - ); - } - - #[test] - fn load_multicast_user_errors_when_no_user_for_this_ip() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let other_ip = Ipv4Addr::new(10, 0, 0, 2); - let mut client = create_test_client(); - let mut users = HashMap::new(); - users.insert(Pubkey::new_unique(), make_user(other_ip, UserType::Multicast)); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let err = load_multicast_user(&client, ip).unwrap_err(); - assert!(err.to_string().contains("No active multicast user")); - } -} -``` - -- [ ] **Step 3: Run tests to verify they fail** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast -``` - -Expected: tests in this new module compile and run; they should pass immediately because the helpers are implemented. If any fail, fix before committing. - -- [ ] **Step 4: Run lint to confirm clean** - -Run: - -```bash -make rust-fmt && cargo clippy -p doublezero --all-targets -- -D warnings -``` - -Expected: no warnings/errors in the new file. - -- [ ] **Step 5: Commit** - -```bash -git add client/doublezero/src/command/mod.rs client/doublezero/src/command/multicast.rs -git commit -m "client/doublezero: add multicast command module with resolve_groups and load_multicast_user helpers" -``` - ---- - -### Task 3: Implement `unsubscribe` handler - -**Files:** -- Modify: `client/doublezero/src/command/multicast.rs` - -- [ ] **Step 1: Write the failing tests** - -Append inside the existing `#[cfg(test)] mod tests { ... }` block in `client/doublezero/src/command/multicast.rs`: - -```rust - // --- MulticastUnsubscribeCliCommand tests --- - - use crate::cli::multicast::MulticastUnsubscribeCliCommand; - use doublezero_sdk::commands::multicastgroup::subscribe::UpdateMulticastGroupRolesCommand; - - fn user_with_roles( - ip: Ipv4Addr, - publishers: Vec, - subscribers: Vec, - ) -> User { - let mut u = make_user(ip, UserType::Multicast); - u.publishers = publishers; - u.subscribers = subscribers; - u - } - - #[tokio::test] - async fn unsubscribe_removes_subscriber_role_and_preserves_publisher_role() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g_pk = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - // User is BOTH publisher and subscriber of g — unsubscribe must keep publisher=true. - let user = user_with_roles(ip, vec![g_pk], vec![g_pk]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g_pk, make_group("g")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client - .expect_update_multicastgroup_roles() - .withf(move |cmd: &UpdateMulticastGroupRolesCommand| { - cmd.user_pk == user_pk - && cmd.group_pk == g_pk - && cmd.client_ip == ip - && cmd.publisher // carry-through preserved - && !cmd.subscriber - }) - .once() - .returning(|_| Ok(solana_sdk::signature::Signature::default())); - - let cmd = MulticastUnsubscribeCliCommand { - groups: vec!["g".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } - - #[tokio::test] - async fn unsubscribe_skips_group_user_is_not_subscribed_to() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g_pk = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - // User has no sub roles — should no-op without an onchain call. - let user = user_with_roles(ip, vec![], vec![]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g_pk, make_group("g")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client.expect_update_multicastgroup_roles().never(); - - let cmd = MulticastUnsubscribeCliCommand { - groups: vec!["g".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } - - #[tokio::test] - async fn unsubscribe_errors_when_user_missing() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let mut client = create_test_client(); - client.expect_list_user().returning(|_| Ok(HashMap::new())); - - let cmd = MulticastUnsubscribeCliCommand { - groups: vec!["g".into()], - }; - let err = cmd.execute_inner(&client, ip).await.unwrap_err(); - assert!(err.to_string().contains("No active multicast user")); - } - - #[tokio::test] - async fn unsubscribe_errors_on_unknown_group_before_any_onchain_call() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let user_pk = Pubkey::new_unique(); - let mut client = create_test_client(); - - let user = user_with_roles(ip, vec![], vec![]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - client - .expect_list_multicastgroup() - .returning(|_| Ok(HashMap::new())); - client.expect_update_multicastgroup_roles().never(); - - let cmd = MulticastUnsubscribeCliCommand { - groups: vec!["unknown".into()], - }; - let err = cmd.execute_inner(&client, ip).await.unwrap_err(); - assert!(err.to_string().contains("Multicast group not found: unknown")); - } -``` - -- [ ] **Step 2: Run tests to verify they fail with "no method named `execute_inner`"** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast::tests::unsubscribe -``` - -Expected: compile error — `MulticastUnsubscribeCliCommand::execute_inner` does not exist. - -- [ ] **Step 3: Implement the handler** - -Append to `client/doublezero/src/command/multicast.rs` (above the `#[cfg(test)]` block): - -```rust -use doublezero_sdk::commands::multicastgroup::subscribe::UpdateMulticastGroupRolesCommand; - -use crate::{ - cli::multicast::MulticastUnsubscribeCliCommand, servicecontroller::ServiceControllerImpl, -}; -use doublezero_cli::helpers::init_command; -use indicatif::ProgressBar; - -impl MulticastUnsubscribeCliCommand { - pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { - let controller = ServiceControllerImpl::new(None); - let client_ip = crate::command::helpers::resolve_client_ip(&controller).await?; - self.execute_inner(client, client_ip).await - } - - /// Testable core: takes an already-resolved client_ip. - async fn execute_inner( - self, - client: &dyn CliCommand, - client_ip: Ipv4Addr, - ) -> eyre::Result<()> { - let spinner = init_command(2); - spinner.println(format!("⚡ Unsubscribing (client_ip: {client_ip})...")); - - let (user_pk, user) = load_multicast_user(client, client_ip)?; - let groups = resolve_groups(client, &self.groups)?; - spinner.inc(1); - - for (code, group_pk) in groups { - if !user.subscribers.contains(&group_pk) { - spinner.println(format!(" not subscribed to {code} — skipping")); - continue; - } - let carry_pub = user.publishers.contains(&group_pk); - client.update_multicastgroup_roles(UpdateMulticastGroupRolesCommand { - user_pk, - group_pk, - client_ip, - publisher: carry_pub, - subscriber: false, - })?; - spinner.println(format!(" unsubscribed from {code}")); - } - - finish_update(&spinner); - Ok(()) - } -} - -fn finish_update(spinner: &ProgressBar) { - spinner.println("✅ Updated. Routes will adjust shortly."); - spinner.finish_and_clear(); -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast::tests::unsubscribe -``` - -Expected: all four unsubscribe tests PASS. - -- [ ] **Step 5: Lint clean** - -Run: - -```bash -make rust-fmt && cargo clippy -p doublezero --all-targets -- -D warnings -``` - -Expected: no warnings. - -- [ ] **Step 6: Commit** - -```bash -git add client/doublezero/src/command/multicast.rs -git commit -m "client/doublezero: implement multicast unsubscribe command" -``` - ---- - -### Task 4: Implement `unpublish` handler with last-publisher warning - -**Files:** -- Modify: `client/doublezero/src/command/multicast.rs` - -- [ ] **Step 1: Write the failing tests** - -Append inside the existing `tests` module: - -```rust - // --- MulticastUnpublishCliCommand tests --- - - use crate::cli::multicast::MulticastUnpublishCliCommand; - - #[tokio::test] - async fn unpublish_removes_publisher_role_and_preserves_subscriber_role() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g1 = Pubkey::new_unique(); - let g2 = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - // Publisher of g1 & g2, subscriber of g1. Unpublish g1 must keep subscriber=true. - let user = user_with_roles(ip, vec![g1, g2], vec![g1]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g1, make_group("g1")); - groups.insert(g2, make_group("g2")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client - .expect_update_multicastgroup_roles() - .withf(move |cmd: &UpdateMulticastGroupRolesCommand| { - cmd.user_pk == user_pk - && cmd.group_pk == g1 - && !cmd.publisher - && cmd.subscriber // carry-through preserved - }) - .once() - .returning(|_| Ok(solana_sdk::signature::Signature::default())); - - let cmd = MulticastUnpublishCliCommand { - groups: vec!["g1".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } - - #[tokio::test] - async fn unpublish_skips_group_user_is_not_publishing_to() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g_pk = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - let user = user_with_roles(ip, vec![], vec![]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g_pk, make_group("g")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client.expect_update_multicastgroup_roles().never(); - - let cmd = MulticastUnpublishCliCommand { - groups: vec!["g".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } - - #[tokio::test] - async fn unpublish_last_publisher_still_issues_onchain_call() { - // The CLI prints a warning but does not block. - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g_pk = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - let user = user_with_roles(ip, vec![g_pk], vec![]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g_pk, make_group("g")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client - .expect_update_multicastgroup_roles() - .once() - .returning(|_| Ok(solana_sdk::signature::Signature::default())); - - let cmd = MulticastUnpublishCliCommand { - groups: vec!["g".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } - - #[tokio::test] - async fn unpublish_of_nonlast_publisher_does_not_claim_last() { - // Would_empty_publishers logic: user has two, remove one — NOT last. - // Regression check: the helper should return false. - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g1 = Pubkey::new_unique(); - let g2 = Pubkey::new_unique(); - - let user = user_with_roles(ip, vec![g1, g2], vec![]); - let would_empty = super::would_empty_publishers(&user, &[g1]); - assert!(!would_empty); - - let would_empty_all = super::would_empty_publishers(&user, &[g1, g2]); - assert!(would_empty_all); - } -``` - -- [ ] **Step 2: Run tests to verify failure** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast::tests::unpublish -``` - -Expected: compile errors — `MulticastUnpublishCliCommand::execute_inner` and `would_empty_publishers` do not exist. - -- [ ] **Step 3: Implement the handler and helper** - -Append to `client/doublezero/src/command/multicast.rs` (above the `#[cfg(test)]` block): - -```rust -use crate::cli::multicast::MulticastUnpublishCliCommand; - -/// Returns true when removing `to_remove` publisher roles from `user` would leave -/// `user.publishers` empty (and the user currently has at least one publisher role). -pub(super) fn would_empty_publishers(user: &User, to_remove: &[Pubkey]) -> bool { - if user.publishers.is_empty() { - return false; - } - let remaining = user - .publishers - .iter() - .filter(|p| !to_remove.contains(p)) - .count(); - remaining == 0 -} - -impl MulticastUnpublishCliCommand { - pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { - let controller = ServiceControllerImpl::new(None); - let client_ip = crate::command::helpers::resolve_client_ip(&controller).await?; - self.execute_inner(client, client_ip).await - } - - async fn execute_inner( - self, - client: &dyn CliCommand, - client_ip: Ipv4Addr, - ) -> eyre::Result<()> { - let spinner = init_command(2); - spinner.println(format!("⚡ Unpublishing (client_ip: {client_ip})...")); - - let (user_pk, user) = load_multicast_user(client, client_ip)?; - let groups = resolve_groups(client, &self.groups)?; - spinner.inc(1); - - // Figure out which of the requested groups the user is actually publishing to. - let effective_removals: Vec = groups - .iter() - .map(|(_, pk)| *pk) - .filter(|pk| user.publishers.contains(pk)) - .collect(); - - if would_empty_publishers(&user, &effective_removals) { - spinner.println( - "⚠️ This removes your last publisher role. In legacy-allocation \ - environments the service may briefly reprovision while the network \ - reallocates.", - ); - } - - for (code, group_pk) in groups { - if !user.publishers.contains(&group_pk) { - spinner.println(format!(" not publishing to {code} — skipping")); - continue; - } - let carry_sub = user.subscribers.contains(&group_pk); - client.update_multicastgroup_roles(UpdateMulticastGroupRolesCommand { - user_pk, - group_pk, - client_ip, - publisher: false, - subscriber: carry_sub, - })?; - spinner.println(format!(" unpublished from {code}")); - } - - finish_update(&spinner); - Ok(()) - } -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast::tests::unpublish -``` - -Expected: all four unpublish tests PASS. - -- [ ] **Step 5: Lint clean** - -Run: - -```bash -make rust-fmt && cargo clippy -p doublezero --all-targets -- -D warnings -``` - -Expected: no warnings. - -- [ ] **Step 6: Commit** - -```bash -git add client/doublezero/src/command/multicast.rs -git commit -m "client/doublezero: implement multicast unpublish command with last-publisher warning" -``` - ---- - -### Task 5: Implement `subscribe` handler - -**Files:** -- Modify: `client/doublezero/src/command/multicast.rs` - -- [ ] **Step 1: Write the failing tests** - -Append inside the existing `tests` module: - -```rust - // --- MulticastSubscribeCliCommand tests --- - - use crate::cli::multicast::MulticastSubscribeCliCommand; - - #[tokio::test] - async fn subscribe_adds_subscriber_role_and_preserves_publisher_role() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g_pk = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - // User is already a publisher of g — subscribing must keep publisher=true. - let user = user_with_roles(ip, vec![g_pk], vec![]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g_pk, make_group("g")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client - .expect_update_multicastgroup_roles() - .withf(move |cmd: &UpdateMulticastGroupRolesCommand| { - cmd.user_pk == user_pk - && cmd.group_pk == g_pk - && cmd.publisher // carry-through preserved - && cmd.subscriber - }) - .once() - .returning(|_| Ok(solana_sdk::signature::Signature::default())); - - let cmd = MulticastSubscribeCliCommand { - groups: vec!["g".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } - - #[tokio::test] - async fn subscribe_skips_already_subscribed_group() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g_pk = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - let user = user_with_roles(ip, vec![], vec![g_pk]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g_pk, make_group("g")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client.expect_update_multicastgroup_roles().never(); - - let cmd = MulticastSubscribeCliCommand { - groups: vec!["g".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } -``` - -- [ ] **Step 2: Run tests to verify failure** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast::tests::subscribe -``` - -Expected: compile error — `MulticastSubscribeCliCommand::execute_inner` does not exist. - -- [ ] **Step 3: Implement the handler** - -Append to `client/doublezero/src/command/multicast.rs` (above the `#[cfg(test)]` block): - -```rust -use crate::cli::multicast::MulticastSubscribeCliCommand; - -impl MulticastSubscribeCliCommand { - pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { - let controller = ServiceControllerImpl::new(None); - let client_ip = crate::command::helpers::resolve_client_ip(&controller).await?; - self.execute_inner(client, client_ip).await - } - - async fn execute_inner( - self, - client: &dyn CliCommand, - client_ip: Ipv4Addr, - ) -> eyre::Result<()> { - let spinner = init_command(2); - spinner.println(format!("⚡ Subscribing (client_ip: {client_ip})...")); - - let (user_pk, user) = load_multicast_user(client, client_ip)?; - let groups = resolve_groups(client, &self.groups)?; - spinner.inc(1); - - for (code, group_pk) in groups { - if user.subscribers.contains(&group_pk) { - spinner.println(format!(" already subscribed to {code} — skipping")); - continue; - } - let carry_pub = user.publishers.contains(&group_pk); - client.update_multicastgroup_roles(UpdateMulticastGroupRolesCommand { - user_pk, - group_pk, - client_ip, - publisher: carry_pub, - subscriber: true, - })?; - spinner.println(format!(" subscribed to {code}")); - } - - finish_update(&spinner); - Ok(()) - } -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast::tests::subscribe -``` - -Expected: both subscribe tests PASS. - -- [ ] **Step 5: Commit** - -```bash -git add client/doublezero/src/command/multicast.rs -git commit -m "client/doublezero: implement multicast subscribe command" -``` - ---- - -### Task 6: Implement `publish` handler - -**Files:** -- Modify: `client/doublezero/src/command/multicast.rs` - -- [ ] **Step 1: Write the failing tests** - -Append inside the existing `tests` module: - -```rust - // --- MulticastPublishCliCommand tests --- - - use crate::cli::multicast::MulticastPublishCliCommand; - - #[tokio::test] - async fn publish_adds_publisher_role_and_preserves_subscriber_role() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g_pk = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - // User is already a subscriber of g — publishing must keep subscriber=true. - let user = user_with_roles(ip, vec![], vec![g_pk]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g_pk, make_group("g")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client - .expect_update_multicastgroup_roles() - .withf(move |cmd: &UpdateMulticastGroupRolesCommand| { - cmd.user_pk == user_pk - && cmd.group_pk == g_pk - && cmd.publisher - && cmd.subscriber // carry-through preserved - }) - .once() - .returning(|_| Ok(solana_sdk::signature::Signature::default())); - - let cmd = MulticastPublishCliCommand { - groups: vec!["g".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } - - #[tokio::test] - async fn publish_skips_already_published_group() { - let ip = Ipv4Addr::new(10, 0, 0, 1); - let g_pk = Pubkey::new_unique(); - let user_pk = Pubkey::new_unique(); - - let mut client = create_test_client(); - let user = user_with_roles(ip, vec![g_pk], vec![]); - let mut users = HashMap::new(); - users.insert(user_pk, user); - client - .expect_list_user() - .returning(move |_| Ok(users.clone())); - - let mut groups = HashMap::new(); - groups.insert(g_pk, make_group("g")); - client - .expect_list_multicastgroup() - .returning(move |_| Ok(groups.clone())); - - client.expect_update_multicastgroup_roles().never(); - - let cmd = MulticastPublishCliCommand { - groups: vec!["g".into()], - }; - cmd.execute_inner(&client, ip).await.unwrap(); - } -``` - -- [ ] **Step 2: Run tests to verify failure** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast::tests::publish -``` - -Expected: compile error — `MulticastPublishCliCommand::execute_inner` does not exist. - -- [ ] **Step 3: Implement the handler** - -Append to `client/doublezero/src/command/multicast.rs` (above the `#[cfg(test)]` block): - -```rust -use crate::cli::multicast::MulticastPublishCliCommand; - -impl MulticastPublishCliCommand { - pub async fn execute(self, client: &dyn CliCommand) -> eyre::Result<()> { - let controller = ServiceControllerImpl::new(None); - let client_ip = crate::command::helpers::resolve_client_ip(&controller).await?; - self.execute_inner(client, client_ip).await - } - - async fn execute_inner( - self, - client: &dyn CliCommand, - client_ip: Ipv4Addr, - ) -> eyre::Result<()> { - let spinner = init_command(2); - spinner.println(format!("⚡ Publishing (client_ip: {client_ip})...")); - - let (user_pk, user) = load_multicast_user(client, client_ip)?; - let groups = resolve_groups(client, &self.groups)?; - spinner.inc(1); - - for (code, group_pk) in groups { - if user.publishers.contains(&group_pk) { - spinner.println(format!(" already publishing to {code} — skipping")); - continue; - } - let carry_sub = user.subscribers.contains(&group_pk); - client.update_multicastgroup_roles(UpdateMulticastGroupRolesCommand { - user_pk, - group_pk, - client_ip, - publisher: true, - subscriber: carry_sub, - })?; - spinner.println(format!(" publishing to {code}")); - } - - finish_update(&spinner); - Ok(()) - } -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: - -```bash -cargo test -p doublezero --lib command::multicast -``` - -Expected: all multicast command tests PASS (subscribe, unsubscribe, publish, unpublish, helpers). - -- [ ] **Step 5: Lint clean** - -Run: - -```bash -make rust-fmt && cargo clippy -p doublezero --all-targets -- -D warnings -``` - -Expected: no warnings. - -- [ ] **Step 6: Commit** - -```bash -git add client/doublezero/src/command/multicast.rs -git commit -m "client/doublezero: implement multicast publish command" -``` - ---- - -### Task 7: Wire up dispatch in main.rs - -**Files:** -- Modify: `client/doublezero/src/main.rs:265-313` (the existing `Command::Multicast` match arm) - -- [ ] **Step 1: Extend the match arm** - -Find the existing `Command::Multicast(args) => match args.command { ... }` block (around line 265). The existing inner `match` has one arm for `MulticastCommands::Group(...)`. Add four new arms for the new variants *before* the closing `},` of the outer match arm: - -Change: - -```rust - Command::Multicast(args) => match args.command { - cli::multicast::MulticastCommands::Group(args) => match args.command { - // ... existing group subcommand handling ... - }, - }, -``` - -To: - -```rust - Command::Multicast(args) => match args.command { - cli::multicast::MulticastCommands::Group(args) => match args.command { - // ... existing group subcommand handling unchanged ... - }, - cli::multicast::MulticastCommands::Subscribe(args) => args.execute(&client).await, - cli::multicast::MulticastCommands::Unsubscribe(args) => args.execute(&client).await, - cli::multicast::MulticastCommands::Publish(args) => args.execute(&client).await, - cli::multicast::MulticastCommands::Unpublish(args) => args.execute(&client).await, - }, -``` - -- [ ] **Step 2: Build and verify the CLI help text** - -Run: - -```bash -cargo build -p doublezero && \ - ./target/debug/doublezero multicast --help -``` - -Expected: the `--help` output lists `group`, `subscribe`, `unsubscribe`, `publish`, `unpublish` subcommands. - -- [ ] **Step 3: Run full workspace build and lint** - -Run: - -```bash -make rust-lint -``` - -Expected: no errors or warnings. - -- [ ] **Step 4: Commit** - -```bash -git add client/doublezero/src/main.rs -git commit -m "client/doublezero: wire up multicast subscribe/unsubscribe/publish/unpublish dispatch" -``` - ---- - -### Task 8: Add E2E helper methods for the four verbs - -**Files:** -- Modify: `e2e/main_test.go` (after the existing `AddMulticastSubscriberGroupSkipAccessPass` helper around line 551-559) - -- [ ] **Step 1: Add the four helpers** - -Insert after the existing `AddMulticastSubscriberGroupSkipAccessPass` function (around line 559), before `DisconnectMulticastSubscriber`: - -```go -// SubscribeMulticastGroup adds subscriber role(s) to an already-connected multicast user. -func (dn *TestDevnet) SubscribeMulticastGroup(t *testing.T, client *devnet.Client, multicastGroupCodes ...string) { - dn.log.Debug("==> Subscribing to multicast groups", "clientIP", client.CYOANetworkIP, "groups", multicastGroupCodes) - - groupArgs := strings.Join(multicastGroupCodes, " ") - _, err := client.Exec(t.Context(), []string{"bash", "-c", "doublezero multicast subscribe " + groupArgs}) - require.NoError(t, err, "failed to subscribe to multicast groups") - - dn.log.Debug("--> Subscribed to multicast groups") -} - -// UnsubscribeMulticastGroup removes subscriber role(s) from an already-connected multicast user. -func (dn *TestDevnet) UnsubscribeMulticastGroup(t *testing.T, client *devnet.Client, multicastGroupCodes ...string) { - dn.log.Debug("==> Unsubscribing from multicast groups", "clientIP", client.CYOANetworkIP, "groups", multicastGroupCodes) - - groupArgs := strings.Join(multicastGroupCodes, " ") - _, err := client.Exec(t.Context(), []string{"bash", "-c", "doublezero multicast unsubscribe " + groupArgs}) - require.NoError(t, err, "failed to unsubscribe from multicast groups") - - dn.log.Debug("--> Unsubscribed from multicast groups") -} - -// PublishMulticastGroup adds publisher role(s) to an already-connected multicast user. -func (dn *TestDevnet) PublishMulticastGroup(t *testing.T, client *devnet.Client, multicastGroupCodes ...string) { - dn.log.Debug("==> Publishing to multicast groups", "clientIP", client.CYOANetworkIP, "groups", multicastGroupCodes) - - groupArgs := strings.Join(multicastGroupCodes, " ") - _, err := client.Exec(t.Context(), []string{"bash", "-c", "doublezero multicast publish " + groupArgs}) - require.NoError(t, err, "failed to publish to multicast groups") - - dn.log.Debug("--> Published to multicast groups") -} - -// UnpublishMulticastGroup removes publisher role(s) from an already-connected multicast user. -func (dn *TestDevnet) UnpublishMulticastGroup(t *testing.T, client *devnet.Client, multicastGroupCodes ...string) { - dn.log.Debug("==> Unpublishing from multicast groups", "clientIP", client.CYOANetworkIP, "groups", multicastGroupCodes) - - groupArgs := strings.Join(multicastGroupCodes, " ") - _, err := client.Exec(t.Context(), []string{"bash", "-c", "doublezero multicast unpublish " + groupArgs}) - require.NoError(t, err, "failed to unpublish from multicast groups") - - dn.log.Debug("--> Unpublished from multicast groups") -} -``` - -- [ ] **Step 2: Verify Go build with e2e tag** - -Run: - -```bash -go build -tags e2e ./e2e/... -``` - -Expected: compiles cleanly. - -- [ ] **Step 3: Commit** - -```bash -git add e2e/main_test.go -git commit -m "e2e: add multicast subscribe/unsubscribe/publish/unpublish test helpers" -``` - ---- - -### Task 9: Extend `TestE2E_Multicast` with unsubscribe/unpublish sub-tests - -**Files:** -- Modify: `e2e/multicast_test.go` (add a new sub-test block after the existing `connect` block around lines 194-226) - -**Context:** The existing `TestE2E_Multicast` connects a publisher to `mg01`, adds `mg02` incrementally, connects a subscriber to `mg01`, adds `mg02` incrementally, and then disconnects both. We'll insert a new sub-test between `connect` and `disconnect` that removes *one of two* roles (so we don't trip the last-publisher `Updating` teardown path) and verifies state. - -- [ ] **Step 1: Write the failing sub-test** - -In `e2e/multicast_test.go`, locate the `if !t.Run("disconnect", ...)` block (around line 228). Insert a new `modify_roles` sub-test *before* it: - -```go - if !t.Run("modify_roles", func(t *testing.T) { - // doublezero user list renders multicast memberships in a column named `groups` - // with each entry prefixed "P:" (publisher) or "S:" (subscriber). - // See smartcontract/cli/src/user/list.rs::format_multicast_group_names. - - subscriberRow := func() map[string]string { - out, err := dn.Manager.Exec(t.Context(), []string{"doublezero", "user", "list"}) - if err != nil { - return nil - } - for _, row := range fixtures.ParseCLITable(out) { - if row["user_type"] == "Multicast" && row["client_ip"] == subscriberClient.CYOANetworkIP { - return row - } - } - return nil - } - publisherRow := func() map[string]string { - out, err := dn.Manager.Exec(t.Context(), []string{"doublezero", "user", "list"}) - if err != nil { - return nil - } - for _, row := range fixtures.ParseCLITable(out) { - if row["user_type"] == "Multicast" && row["client_ip"] == publisherClient.CYOANetworkIP { - return row - } - } - return nil - } - - // Unsubscribe the subscriber from mg01 (keeps it subscribed to mg02). - tdn.UnsubscribeMulticastGroup(t, subscriberClient, "mg01") - - require.Eventually(t, func() bool { - row := subscriberRow() - if row == nil { - return false - } - groups := row["groups"] - return !strings.Contains(groups, "S:mg01") && strings.Contains(groups, "S:mg02") - }, 60*time.Second, 2*time.Second, "subscriber should be unsubscribed from mg01 but still subscribed to mg02") - - // Unpublish the publisher from mg01 (keeps it publishing to mg02). - tdn.UnpublishMulticastGroup(t, publisherClient, "mg01") - - require.Eventually(t, func() bool { - row := publisherRow() - if row == nil { - return false - } - groups := row["groups"] - return !strings.Contains(groups, "P:mg01") && strings.Contains(groups, "P:mg02") - }, 60*time.Second, 2*time.Second, "publisher should be unpublished from mg01 but still publishing to mg02") - - // Tunnels should still be up — key assertion: no disconnect required. - err := publisherClient.WaitForTunnelUp(t.Context(), 30*time.Second) - require.NoError(t, err, "publisher tunnel should remain up after unpublish") - err = subscriberClient.WaitForTunnelUp(t.Context(), 30*time.Second) - require.NoError(t, err, "subscriber tunnel should remain up after unsubscribe") - - // Restore pre-test state so the disconnect sub-test exercises the same two-group - // teardown it did before. - tdn.SubscribeMulticastGroup(t, subscriberClient, "mg01") - tdn.PublishMulticastGroup(t, publisherClient, "mg01") - - require.Eventually(t, func() bool { - pub := publisherRow() - sub := subscriberRow() - if pub == nil || sub == nil { - return false - } - return strings.Contains(pub["groups"], "P:mg01") && - strings.Contains(pub["groups"], "P:mg02") && - strings.Contains(sub["groups"], "S:mg01") && - strings.Contains(sub["groups"], "S:mg02") - }, 60*time.Second, 2*time.Second, "roles should be restored for both clients") - }) { - t.Fail() - return - } -``` - -- [ ] **Step 2: Run the e2e test** - -Run (requires sudo on Linux for libpcap; see CLAUDE.md): - -```bash -make e2e-test RUN=TestE2E_Multicast -``` - -Expected: the new `modify_roles` sub-test passes. The existing `connect` and `disconnect` sub-tests still pass unchanged. - -If the test fails on the `multicast_pubs` / `multicast_subs` column names, fix per the note above and re-run. - -- [ ] **Step 3: Run Go lint and formatter** - -Run: - -```bash -make go-lint && make go-fmt -``` - -Expected: no errors. - -- [ ] **Step 4: Commit** - -```bash -git add e2e/multicast_test.go -git commit -m "e2e: verify multicast role modification without disconnect" -``` - ---- - -### Task 10: Final verification across the whole workspace - -- [ ] **Step 1: Full Rust test suite** - -Run: - -```bash -make rust-test -``` - -Expected: all tests pass. - -- [ ] **Step 2: Full Rust lint** - -Run: - -```bash -make rust-lint -``` - -Expected: no warnings. - -- [ ] **Step 3: Full Go lint and build** - -Run: - -```bash -make go-lint -``` - -Expected: no warnings. - -- [ ] **Step 4: Manual devnet sanity check (optional, documented)** - -On a local devnet: - -```bash -dev/dzctl destroy -y && dev/dzctl build # only if no devnet running -# Connect a multicast user to two groups via connect: -docker exec dz-local-client- doublezero connect Multicast --subscribe mg01 mg02 - -# Remove one via the new verb: -docker exec dz-local-client- doublezero multicast unsubscribe mg01 - -# Verify onchain state: -docker exec dz-local-manager doublezero user list -# → the user should show subscribers = [mg02] only, and the tunnel should still be up. - -# Verify daemon status: -docker exec dz-local-client- doublezero status -# → session_status should remain "established". -``` - ---- - -## Known limitation (spec-documented) - -`doublezero multicast unpublish ` in legacy-allocation environments will still cause a brief service reprovision because the smartcontract sets `UserStatus::Updating` and the daemon's reconciler tears the service down. The CLI warns when this would happen. Environments using onchain allocation are unaffected. This is intentionally out of scope for this plan. diff --git a/docs/superpowers/specs/2026-03-26-latency-command-ux-design.md b/docs/superpowers/specs/2026-03-26-latency-command-ux-design.md deleted file mode 100644 index e6566aca5a..0000000000 --- a/docs/superpowers/specs/2026-03-26-latency-command-ux-design.md +++ /dev/null @@ -1,86 +0,0 @@ -# Latency Command UX Improvements - -## Problem - -The `doublezero latency` CLI command has three issues: - -1. **Race condition in daemon:** The device fetch goroutine and probe goroutine start concurrently on daemon startup. The first `probe()` often reads an empty `DeviceCache`, probes nothing, and caches empty results. Users must wait for the next probe tick (default 30s) before data appears. - -2. **No progress feedback:** The latency command passes `None` for the spinner, so the user sees a blank terminal during what can be a 30+ second wait. - -3. **Misleading error:** When retries exhaust, the user sees "No devices found" — which conflates "daemon hasn't finished probing yet" (transient) with "no activated devices exist" (permanent). - -## Changes - -### Daemon: Fix probe sequencing and expose readiness - -**File:** `client/doublezerod/internal/latency/manager.go` - -Add to `LatencyManager`: -- `devicesFetched chan struct{}` — created in `NewLatencyManager`, closed after first successful `fetch()` populates `DeviceCache` -- `probeReady atomic.Bool` — set to `true` after first `probe()` completes and writes to `ResultsCache` - -In `Start()`: -- The fetch goroutine closes `devicesFetched` after the first successful `fetch()` call -- The probe goroutine waits on `<-l.devicesFetched` before running the first `probe()` -- After the first `probe()` writes results, set `l.probeReady.Store(true)` - -Update `ServeLatency` response format from a bare `[]LatencyResult` to: - -```json -{ - "ready": false, - "results": [] -} -``` - -Where `ready` reflects `probeReady.Load()`. - -### CLI: Spinner and improved feedback - -**File:** `client/doublezero/src/servicecontroller.rs` - -Add a new response struct for the latency endpoint: - -```rust -struct LatencyResponse { - ready: bool, - results: Vec, -} -``` - -Update `ServiceController::latency()` to return `LatencyResponse` instead of `Vec`. - -**File:** `client/doublezero/src/command/latency.rs` - -Create a spinner in `execute()` and pass it to `check_doublezero()` and `retrieve_latencies()`. - -**File:** `client/doublezero/src/dzd_latency.rs` - -Update `retrieve_latencies()` retry logic: -- When `ready: false`: retry with 1s interval, spinner shows "Waiting for daemon to finish probing devices..." -- When `ready: true` but results empty: stop retrying immediately, return clear error "No activated devices found" -- When `ready: true` with results: return normally - -Remove exponential backoff for the "not ready" case — a simple 1s poll is appropriate since we're waiting for a one-time daemon initialization, not recovering from an error. - -Keep the existing retry with backoff for transient errors from the daemon endpoint itself (connection refused, etc.). - -## Files Modified - -| File | Change | -|------|--------| -| `client/doublezerod/internal/latency/manager.go` | Add `devicesFetched` channel, `probeReady` atomic, sequencing, new response format | -| `client/doublezero/src/servicecontroller.rs` | New `LatencyResponse` struct, update `latency()` return type | -| `client/doublezero/src/command/latency.rs` | Add spinner creation and passing | -| `client/doublezero/src/dzd_latency.rs` | Update retry logic to use `ready` field, improve error messages | - -## User-facing output unchanged - -The `{ready, results}` wrapper is only the internal daemon-to-CLI wire format over the Unix socket. The user-facing output from `doublezero latency` (table or `--json`) remains `Vec` unchanged. - -## Not in scope - -- Changing probe intervals or timeouts -- Concurrent device fetch + latency fetch in the CLI (they're already fast once the daemon is ready) -- Streaming/incremental display of results diff --git a/docs/superpowers/specs/2026-04-03-status-multicast-groups-design.md b/docs/superpowers/specs/2026-04-03-status-multicast-groups-design.md deleted file mode 100644 index 7aa81dcee7..0000000000 --- a/docs/superpowers/specs/2026-04-03-status-multicast-groups-design.md +++ /dev/null @@ -1,79 +0,0 @@ -# Status Command: Show Multicast Group Memberships - -## Goal - -Show which multicast groups a user is subscribed to and/or publishing to in the `doublezero status` command output. - -## Current Behavior - -The `doublezero status` command shows tunnel status, device info, metro, network, and tenant. For multicast user types, there is no indication of which groups the user belongs to or their role (publisher vs subscriber). - -## Design - -### Data flow - -The daemon already fetches full onchain program data during reconciliation, including `MulticastGroup` accounts (with group codes) and `User` accounts (with `Publishers` and `Subscribers` pubkey vecs). The multicast group information just needs to be threaded through the v2 status response. - -### Go daemon changes (`client/doublezerod/`) - -Add a `MulticastGroups` struct and a `multicast_groups` field to `V2ServiceStatus`: - -```go -type MulticastGroups struct { - Publisher []string `json:"publisher"` - Subscriber []string `json:"subscriber"` -} - -type V2ServiceStatus struct { - *api.StatusResponse - // ... existing fields ... - MulticastGroups MulticastGroups `json:"multicast_groups"` -} -``` - -In `enrichStatuses()`, for each service, resolve the matched user's `Publishers` and `Subscribers` pubkey vecs against the fetched `MulticastGroup` map to populate the group code lists. Non-multicast users get empty lists. - -### Rust CLI changes (`client/doublezero/`) - -Extend `V2ServiceStatus` in `servicecontroller.rs`: - -```rust -#[derive(Serialize, Deserialize, Debug, Clone, Default)] -pub struct MulticastGroups { - #[serde(default)] - pub publisher: Vec, - #[serde(default)] - pub subscriber: Vec, -} -``` - -Add `multicast_groups: MulticastGroups` to `V2ServiceStatus` (with `#[serde(default)]` for backward compatibility with older daemons). - -In `status.rs`, add a "Multicast Groups" column to `AppendedStatusResponse` that formats as `P:code1,S:code2` in table mode. JSON mode includes the structured `multicast_groups` object. - -### Output examples - -**Table mode:** -``` -| Session Status | ... | Multicast Groups | -|----------------|-----|--------------------------| -| BGP Session Up | ... | P:solana-lv,S:solana-ams | -``` - -**JSON mode:** -```json -{ - "multicast_groups": { - "publisher": ["solana-lv"], - "subscriber": ["solana-ams"] - } -} -``` - -For non-multicast users, the column is empty and both JSON arrays are `[]`. - -## Testing - -- Unit tests in `status.rs` for the new field formatting (table and JSON) -- Unit test for backward compatibility when daemon omits the field (serde default) -- Verify `enrichStatuses` correctly resolves publisher/subscriber pubkeys to group codes in Go tests diff --git a/docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md b/docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md deleted file mode 100644 index 1d4822d092..0000000000 --- a/docs/superpowers/specs/2026-04-23-multicast-modify-without-disconnect-design.md +++ /dev/null @@ -1,112 +0,0 @@ -# Multicast: modify subscriptions and publishers without disconnecting - -**Status:** Design approved, pending implementation plan. -**Date:** 2026-04-23. - -## Problem - -A connected multicast user cannot modify their set of publisher or subscriber group memberships without running `doublezero disconnect` (which deletes the user account) and then reconnecting with the new desired set. - -Today, `doublezero connect Multicast --publish --subscribe ` handles first-time connection and is purely additive — it will add new groups to an existing connected user, but it never removes. There is no CLI path to drop a subscriber role, drop a publisher role, or otherwise narrow the active role set. Users work around this by disconnecting and reconnecting. - -## Scope - -In scope: - -- Add four new CLI subcommands under `doublezero multicast` so users can add or remove publisher/subscriber roles on an already-connected multicast user without running `disconnect`. -- Extend the existing multicast e2e suite to cover the new commands. - -Out of scope: - -- Smartcontract changes. The existing `UpdateMulticastGroupRoles` instruction already supports add and remove via its `publisher` / `subscriber` boolean flags, and is what `connect` uses today. -- Daemon changes. In particular, the legacy-allocation codepath in the smartcontract transitions a user's status to `Updating` when they gain their first publisher or lose their last publisher, and the daemon's reconciler treats `Updating` as "not provisioned" and tears the service down. This teardown behavior is a known limitation and is not fixed here. Environments using onchain allocation already avoid `Updating` for these transitions. -- Making `connect` declarative / non-additive. `connect` stays as-is. - -## Commands - -All four commands live under the existing `doublezero multicast` namespace (which currently only exposes administrative `group` subcommands). Each takes one or more group codes. - -``` -doublezero multicast subscribe [ ...] # add subscriber role(s) -doublezero multicast unsubscribe [ ...] # remove subscriber role(s) -doublezero multicast publish [ ...] # add publisher role(s) -doublezero multicast unpublish [ ...] # remove publisher role(s) -``` - -`subscribe` and `publish` overlap functionally with `doublezero connect Multicast --subscribe/--publish` when a user is already connected. They are added for interface symmetry with the `un-` variants and to offer a shorter invocation for the common "already connected, adjust my roles" case. - -## Behavior - -**Precondition.** A Multicast user must already exist for the caller's `client_ip`. If none, the command fails with: - -``` -No active multicast user for . Run 'doublezero connect Multicast --publish/--subscribe ' first. -``` - -These commands never create users — that remains the responsibility of `connect`. - -**Group code resolution.** Group codes are resolved to pubkeys via `ListMulticastGroupCommand`, matching the pattern used by `connect::execute_multicast`. Any unknown code fails the whole command before any onchain call is issued. - -**Onchain call.** For each resolved group, one `UpdateMulticastGroupRolesCommand` is issued. The onchain instruction is a per-flag "set" (passing `publisher: false` removes the publisher role for that group; passing `publisher: true` adds it), so each verb carries the *other* role's current value through unchanged to avoid clobbering it: - -| Verb | `publisher` flag | `subscriber` flag | -| ------------- | ------------------------------ | ------------------------------ | -| `subscribe` | `user.publishers.contains(g)` | `true` | -| `unsubscribe` | `user.publishers.contains(g)` | `false` | -| `publish` | `true` | `user.subscribers.contains(g)` | -| `unpublish` | `false` | `user.subscribers.contains(g)` | - -**No-op handling.** If a group is already in the requested state for the given verb (e.g. `subscribe` on a group the user is already subscribed to, or `unpublish` on a group the user does not publish to), the CLI logs a skip message (`already subscribed to `, `not publishing to `, etc.) and moves on without issuing an onchain call for that group. The command as a whole still succeeds. - -**Last-publisher unpublish warning.** When an `unpublish` would empty `user.publishers`, the CLI prints a warning that the service may briefly reprovision — this is the legacy-allocation limitation described above. The command proceeds. No interactive prompt; the warning is informational. - -**Daemon reconciliation.** No explicit daemon notify. The daemon's reconciler polls onchain state and will add or drop multicast routes automatically. The CLI prints `Updated. Routes will adjust shortly.` and exits. - -## Implementation layout - -### New and modified files - -- **`client/doublezero/src/cli/multicast.rs`** — extend `MulticastCommands` with four new variants: `Subscribe`, `Unsubscribe`, `Publish`, `Unpublish`. Each takes an args struct with one or more group codes. The existing `Group(MulticastGroupCliCommand)` variant is unchanged. -- **`client/doublezero/src/command/multicast.rs`** *(new)* — one handler per verb, sharing helpers: - - `resolve_groups(client, codes) -> Vec` — mirrors the group-code resolution in `connect::execute_multicast`. - - `load_multicast_user(client, client_ip) -> (Pubkey, User)` — finds the single Multicast user for the caller's `client_ip`, returning the precondition error when absent. - - `apply_role_change(client, user_pk, user, group_pk, publisher, subscriber)` — builds the `UpdateMulticastGroupRolesCommand` with the correct carry-through of the opposite role, handles the no-op skip log, emits the last-publisher warning when applicable. - - Each of the four verb handlers: resolve groups, load user, iterate groups calling `apply_role_change` with the flag pattern from the table above, print the completion message. -- **`client/doublezero/src/main.rs`** — dispatch the four new `MulticastCommands` variants to the new handlers, following the existing `Command::Connect(args) => args.execute(&client).await` pattern. - -No SDK changes — `UpdateMulticastGroupRolesCommand` already exists and is what `connect` invokes for additive role changes today. - -Handlers live in `client/doublezero/src/command/` (not `smartcontract/cli/src/`) because they depend on client-side `client_ip` discovery and user lookup, matching where `connect.rs` and `disconnect.rs` already sit. The administrative `multicast group *` commands remain in the smartcontract CLI crate. - -## Testing - -### Unit tests (Rust) - -Colocated with the new command handlers in `client/doublezero/src/command/multicast.rs` (or a dedicated test module), using the same mocking patterns as the existing `connect` tests: - -- Happy-path for each verb: resolves groups, calls `UpdateMulticastGroupRolesCommand` with the correct flag pair (including correct carry-through of the opposite role), succeeds. -- Missing multicast user → precondition error with the documented message. -- Unknown group code → error before any onchain call is issued. -- No-op: `subscribe` on a group the user is already subscribed to skips; same for `publish`. `unsubscribe` and `unpublish` on groups the user is not in also skip. -- Role carry-through: `unsubscribe` on a group where the user is *also* a publisher keeps the publisher role (verify the outgoing `UpdateMulticastGroupRolesCommand` has `publisher: true, subscriber: false`). -- Last-publisher `unpublish`: warning is emitted and the command still proceeds. - -### E2E test - -Extend the existing multicast e2e suite in `e2e/` (e.g. `multicast_test.go` or a new sibling file under `e2e/internal/qa/client_multicast.go`) to exercise the full flow end-to-end. At minimum: - -1. Connect a client with `--subscribe groupA`. -2. Run `multicast subscribe groupB`; assert the onchain user now has `subscribers = [A, B]` and the client has multicast routes for both groups. -3. Run `multicast unsubscribe groupA`; assert `subscribers = [B]` and routes for A are gone. -4. Run `multicast publish groupC`; assert publisher role is added (onchain allocation path; no teardown). -5. Run `multicast unpublish groupC`; assert publisher role is removed. Note the expected `Updating` teardown in legacy-allocation environments and confirm the onchain-allocation path does not tear down. - -Mirror whichever helpers in `e2e/internal/qa/client_multicast.go` are most convenient (e.g. add `SubscribeMulticast`, `UnsubscribeMulticast`, etc. alongside the existing `ConnectUserMulticast_*` helpers). - -### Manual local-devnet verification - -Document in the PR's Testing Verification section: on `dev/dzctl` local devnet, connect a client with `--subscribe groupA`, then exercise each new verb and inspect state with `doublezero user list` (onchain) and `doublezero status` (daemon) to confirm the expected transitions. - -## Known limitation (documented) - -In legacy-allocation environments, running `doublezero multicast unpublish ` still causes a brief service reprovision because the smartcontract transitions the user to `Updating` and the daemon's reconciler tears the service down. Onchain-allocation environments already avoid this. Fixing this requires either daemon or smartcontract changes and is deliberately out of scope for this work. diff --git a/docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md b/docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md deleted file mode 100644 index c194e83c3c..0000000000 --- a/docs/superpowers/specs/2026-06-17-serviceability-feed-oracle-permission-account-design.md +++ /dev/null @@ -1,160 +0,0 @@ -# Permission-account authorization for the feed oracle (serviceability) - -Issue: malbeclabs/infra#1652 (reframed — see below). Parent: #1547 Generalized Seat Buying. -Target repo: malbeclabs/doublezero (this repo, serviceability program). Branch off `origin/main`. - -## Why this reframes #1652 - -#1652 proposed giving the shred oracle a scoped `feed_authority` delete authority (mirroring the -`feed_authority` owns-it scoping on set/close). Two findings make that the wrong vehicle: - -1. **`feed_authority`'s owns-only gate is incompatible with the oracle's validator-owned flow.** - `set`/`close`/`add`/`remove` deny when `feed_authority == payer && accesspass.owner != payer` — - and that check fires even for a caller who is *also* in `foundation_allowlist`. Validator-owned - access passes are owned by the validator (the oracle skips `set_access_pass` on an existing pass - precisely so it does not clobber the validator's settings), so an oracle holding `feed_authority` - would be blocked on every validator-owned pass. This is exactly why the validator-owned rollout - cleared `feed_authority` and put the oracle in `foundation_allowlist`. -2. **The `sentinel_authority_pk` slot is occupied.** It is wired as the billing sentinel - (`processors/tenant/update_payment_status.rs`, "used by billing sentinel after deduction") and the - suspend/circuit-breaker authority. It is a single-pubkey slot, so the oracle cannot co-occupy it, - and folding the oracle into it would put billing + the network kill-switch on the hot oracle key. - -The least-privilege path that avoids both problems is the program's **Permission-account** system: -grant the oracle a per-key `Permission` PDA carrying exactly the bits it needs, and make the handlers -it calls honor that Permission account. No single-role slot contention, no owns-only gate, far -narrower than `foundation_allowlist` (no governance / allowlist-edit / authority-rotation bits). - -## Scope of this spec (Unit 1 of 3) - -This spec covers ONLY the serviceability program change. It is self-contained and testable on its own -— the new Permission-account path is dormant until a caller actually passes a Permission account, so -nothing changes for existing callers. Two follow-up units are out of scope here: - -- **Unit 2 (oracle wiring, repo `malbeclabs/doublezero-shreds`):** `dz_ledger.rs` appends the oracle's - Permission PDA `AccountMeta` to the relevant instructions. -- **Unit 3 (operational rollout):** `CreatePermission` for the oracle's key with - `ACCESS_PASS_ADMIN | USER_ADMIN`, then remove the oracle from `foundation_allowlist`. - -## Goal / acceptance - -A key that signs while passing its own `Permission` PDA (status `Activated`) bearing -`ACCESS_PASS_ADMIN | USER_ADMIN` can perform all seven serviceability operations the oracle uses, -**without** `foundation_allowlist` membership and **without** any single-role slot. Every existing -caller's authorization is unchanged (additive only). - -## Design - -### 1. New helper: `authorize_permission_account_only` (authorize.rs) - -`authorize()` already supports a Permission-account path, but it bundles a *legacy fallback* -(`check_legacy_any`) whose composite differs from each handler's hand-written inline check (e.g. -`delete` is `foundation || user.owner`, but `USER_ADMIN` legacy is `foundation || activator`). Routing -existing callers through `authorize()` would therefore silently change their authorities. So we add a -**permission-account-only** helper with no legacy fallback, to be OR'd with each handler's existing -inline check: - -```rust -/// Permission-account-only authorization (no legacy GlobalState fallback). -/// -/// Reads the next account from `accounts_iter` as the payer's optional trailing -/// `Permission` PDA: -/// - No further account present -> Ok(false) (caller falls back to its own inline checks) -/// - Present, is the payer's Permission PDA, program-owned, status == Activated, -/// and `permissions & any_of_flags != 0` -> Ok(true) -/// - Present, is the payer's Permission PDA, program-owned, Activated, but no -/// required bit set -> Ok(false) -/// - Present but NOT the payer's Permission PDA, or not program-owned -> Err -/// (a malformed/incorrect account was supplied) -/// -/// Unlike `authorize`, this never consults `check_legacy_any`, so OR-ing it into a -/// handler's existing inline check cannot widen the legacy authorities. -pub fn authorize_permission_account_only<'a, 'b: 'a, I>( - program_id: &Pubkey, - accounts_iter: &mut I, - payer_key: &Pubkey, - any_of_flags: u128, -) -> Result -where - I: Iterator>; -``` - -It reuses the same validation `authorize()`'s new path performs (PDA == `get_permission_pda(program_id, -payer_key)`, `owner == program_id`, deserialize `Permission`, `status == Activated`). The only -behavioral differences from `authorize()`'s new path: a missing-bit result returns `Ok(false)` rather -than `Err` (so the handler's own inline OR decides the final verdict), and an absent account returns -`Ok(false)` rather than entering the legacy branch. - -### 2. Integration pattern for each handler - -The Permission account is an **optional trailing account** (appended after each instruction's current -last account), so today's callers — which pass no such account — are unaffected and keep hitting their -inline checks. Because the helper reads the *next* account from the iterator, the authorization -decision must be made **after all of the handler's required accounts have been read** (the iterator -must be positioned at the trailing slot) and **before any state mutation**. Per handler: - -1. Compute `inline_ok` from the already-read accounts using the handler's *existing* inline check, - verbatim (do not change it). -2. Finish reading the handler's required accounts (so the iterator reaches the trailing slot). -3. Before mutating any account: `let authorized = inline_ok || authorize_permission_account_only(program_id, accounts_iter, payer, BIT)?;` and reject with the handler's existing error if `!authorized`. - -Handlers today reject early (right after reading `globalstate`); this moves the *final* rejection to -just after the required-account reads, before mutations. No account read is destructive, so deferring -the rejection is safe. The existing `feed_authority` owns-only restrictions stay exactly as they are — -they gate only the `feed_authority` path, never the Permission-account path (so the oracle's -Permission account is not owns-gated, which is required for validator-owned passes it does not own). - -### 3. The seven handlers and their required bit - -| Handler (file) | Existing inline authority (preserve verbatim) | Bit added via `perm_only` | -|---|---|---| -| `SetAccessPass` (`accesspass/set.rs`) | `foundation \|\| sentinel \|\| feed_authority \|\| tenant_admin \|\| accesspass.owner==payer` (+ feed owns-it) | `ACCESS_PASS_ADMIN` | -| `CloseAccessPass` (`accesspass/close.rs`) | `foundation \|\| feed_authority` (+ feed owns-it) | `ACCESS_PASS_ADMIN` | -| `AddMulticastGroupSubAllowlist` (`multicastgroup/allowlist/subscriber/add.rs`) | `mgroup.owner==payer \|\| sentinel \|\| feed_authority \|\| foundation` (+ feed owns-it) | `ACCESS_PASS_ADMIN` | -| `RemoveMulticastGroupSubAllowlist` (`multicastgroup/allowlist/subscriber/remove.rs`) | same as add | `ACCESS_PASS_ADMIN` | -| `UpdateMulticastGroupRoles` (`multicastgroup/subscribe.rs`) | `accesspass.user_payer==payer \|\| foundation` | `ACCESS_PASS_ADMIN` | -| `DeleteUser` (`user/delete.rs`) | `foundation \|\| user.owner==payer` | `USER_ADMIN` | -| `CreateSubscribeUser` owner_override (`user/create_core.rs`) | `foundation \|\| sentinel` (to set `owner != payer`) | `USER_ADMIN` | - -The oracle's `Permission` bitmask is therefore `ACCESS_PASS_ADMIN | USER_ADMIN`. - -> Verification item for the plan: confirm `CreateSubscribeUser` has no *other* foundation-only gate -> beyond the `owner_override` check (the oracle creates users with `owner = validator`). If a second -> gate exists, it needs the same `perm_only` OR. - -### 4. Tests (`programs/doublezero-serviceability/tests/`, solana-program-test) - -Reuse the `permission_test.rs` harness (`CreatePermission` + `get_permission_pda`) and the -per-handler test helpers. For each of the seven handlers: - -- **Positive:** a signer who is NOT foundation/sentinel/feed/owner, but passes an `Activated` - Permission PDA with the required bit, succeeds. For `delete`/`close`, use a pass/user owned by a - *different* key to prove the Permission path has no owns-it restriction. -- **Negative (no account):** same signer, no Permission account passed → the handler's existing error - (unchanged legacy behavior). -- **Negative (insufficient/suspended):** Permission PDA present but missing the bit, or `Suspended` → - rejected. -- **Regression:** the existing foundation / `user.owner` / `user_payer` / feed-authority-owns-it paths - still succeed/deny exactly as before (one representative assertion each). - -Pin exact error variants (`NotAllowed` / `Unauthorized` as each handler currently returns; the new -helper's malformed-account case returns `InvalidArgument`/`InvalidAccountData`). Add a focused unit -test for `authorize_permission_account_only` covering each branch of its contract. - -### 5. IDL / SDK / docs - -- Each of the seven instructions gains one **optional trailing account** (the payer's Permission PDA). - Update the program IDL accordingly. Existing instruction builders that omit it stay valid. -- Update `PERMISSION.md` to record that these seven handlers now honor a Permission account bearing - `ACCESS_PASS_ADMIN` / `USER_ADMIN` (as listed above), in addition to their legacy authorities. -- No instruction *data* (args) change; no discriminator change. - -## Out of scope - -- Oracle wiring (Unit 2) and the operational rollout (Unit 3: create the oracle's Permission PDA; - remove it from `foundation_allowlist`). -- Setting the `RequirePermissionAccounts` feature flag (the program-wide switch that disables the - legacy path). This change is purely additive; the legacy paths remain for all existing callers. -- Converting these handlers to call `authorize()` wholesale (would change legacy composites). We add - the narrow `perm_only` OR instead. -- Any change to `feed_authority`'s owns-only gates or the sentinel/billing wiring.