[DON'T MERGE] feat(drive-abci,dashmate): seed Orchard shielded pool at SDK_TEST_DATA devnet genesis#3732
Draft
shumkov wants to merge 2 commits into
Draft
[DON'T MERGE] feat(drive-abci,dashmate): seed Orchard shielded pool at SDK_TEST_DATA devnet genesis#3732shumkov wants to merge 2 commits into
shumkov wants to merge 2 commits into
Conversation
…A devnet genesis
Pre-populates the shielded pool with 500_000 Orchard notes (8 owned by two
deterministic test wallets) when a local devnet binary is built with
`--cfg=create_sdk_test_data`. Closes the gap for benchmarking wallet sync at
scale without paying per-note Halo 2 proof time at chain bring-up.
Seeder (drive-abci):
- `create_data_for_shielded_pool` runs inside `create_sdk_test_data` and
emits 500k `ShieldedPoolOperationType::InsertNote` ops through the
production `commitment_tree_insert_op` path. GroveDB's
`preprocess_commitment_tree_ops` batches them into a single
Sinsemilla-frontier load / `append_with_mem_buffer` loop / Merk
propagation.
- Two-tier note generator: filler (random valid Pallas-base `cmx` + 216
bytes of opaque ciphertext) + owned (real `Note::from_parts` encrypted
to a fixed ZIP-32-derived address via `OrchardNoteEncryption<DashMemo>`).
- Genesis-time anchor recording at `block_height=1` matches production's
end-of-block-1 anchor — single recorded anchor suffices for spends via
the wallet's one-checkpoint-at-post-sync-tree-size invariant.
- Determinism: single `StdRng::seed_from_u64(0xDEAD_BEEF)` threaded
through every loop; `seed_a`/`seed_b` test wallets derived via
`SpendingKey::from_zip32_seed(seed, coin_type=1, account=0)`.
dashmate config plumbing:
- New `buildArgs: Record<string,string>` field on `dockerBuild` schema —
generic per-image build-arg map. Dashmate becomes the single source of
truth for `SDK_TEST_DATA` and `CARGO_BUILD_PROFILE`; shell-env
passthrough is dropped.
- `scripts/setup_local_network.sh` writes
`buildArgs.SDK_TEST_DATA="true"` + `CARGO_BUILD_PROFILE="release"` to
each `local_N` after `dashmate setup local`. Release profile is
mandatory — debug-Sinsemilla pushes InitChain past tenderdash's timeout.
(Marked TODO/temporary in the script — removable once Option B
precomputed-snapshot lands, or N drops low enough for debug seeding.)
- `generateEnvsFactory` flattens both `platform.drive.abci.docker.build.buildArgs`
and `platform.dapi.rsDapi.docker.build.buildArgs` into the
docker-compose env so `${KEY}` substitution in the compose `build.args`
blocks picks them up.
`dashmate config set` bug fix:
- The old `config.get(path)` pre-check rejected legal sets of new keys
inside `additionalProperties: <schema>` maps (e.g. `…buildArgs.X`).
Replaced with `Config.isSchemaPathAllowed(path)` which walks the JSON
schema descending through `properties`, `additionalProperties` value
schemas, and `$ref` references. 15 unit tests pin the walker.
Tests:
- 23 in-process tests in `rs-drive-abci`: wallet derivation, note
generator (filler + owned + ρ uniqueness + ciphertext layout +
determinism + per-wallet decrypt + cross-wallet privacy + aggregate
balance), Drive-level integration (count + anchor + cross-platform
byte-identical determinism).
- 15 dashmate unit tests for the schema walker.
- One `#[ignore]`-d functional test in `rs-platform-wallet` that drives
the full `PlatformWalletManager → bind_shielded → coordinator.sync →
shielded_balances` flow against a live SDK_TEST_DATA devnet.
Cost (release profile, 500_000 notes, Apple Silicon Docker Desktop):
~3h 41m wall-clock for the seeder. CPU work is ~95s; the rest is
GroveDB writes through the macOS Docker VM. See
`docs/shielded-seeder-performance.md` for the breakdown and the
Option-B follow-up.
Refs #3714.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…t InitChain
Cuts SDK_TEST_DATA shielded-pool seeding from ~3h41m (Docker on macOS at
N=500k) or ~65 min (native release at N=500k) to ~134 ms at InitChain by
moving the seed work to a one-shot bake during docker image build and
loading the result via `IngestExternalFile` + parent-Merk leaf patch.
End-to-end proven on local devnet: bake-inside-Dockerfile produces a
snapshot file with `combined_root=b682f38442c8...8144e3c3` byte-for-byte
identical to a local macOS bake; InitChain applies it in 134 ms; wallet-A
sync against the snapshot-loaded chain recovers the expected 400 000
balance (4 owned × 100 000) — proving proof verification works against
snapshot-loaded state.
Components
----------
- `packages/rs-drive-abci/src/shielded_snapshot/mod.rs` (NEW):
- `dump_shielded_subtree(grove, w)` — writes one SST per CF + header +
blake3 checksum.
- `apply_shielded_snapshot(grove, r, txn)` — validates header,
`ingest_subtree_sst`, cross-validates `combined_root` against the
reconstructed CommitmentTree, patches parent Merk leaf via
`replace_commitment_tree_subtree_root`. No fallback — any mismatch
is FATAL so the operator notices.
- `drive-abci snapshot-bake --out <path>` subcommand:
- Self-contained: opens fresh tempdir, runs `create_genesis_state`
(which under `cfg(create_sdk_test_data)` seeds the shielded pool),
then dumps the resulting subtree. Uses a NoopCoreRPC stub because
genesis doesn't talk to Core.
- `DRIVE_SHIELDED_SNAPSHOT` env var read in `create_data_for_shielded_pool`:
takes the snapshot fast-path when set, runs the runtime seeder
otherwise. Failure during apply is fatal (no silent fallback).
- Dockerfile: new `bake-shielded-snapshot` stage runs `drive-abci
snapshot-bake` against an in-container tempdir when `SDK_TEST_DATA=true`,
embeds the snapshot at `/opt/dashmate/snapshots/shielded-pool.snap` in
the runtime image, sets `ENV DRIVE_SHIELDED_SNAPSHOT=...`.
- `docs/genesis-snapshot-design.md` — design doc covering format,
cross-validation, threat model, compatibility policy.
Side-effect changes
-------------------
- `ShieldedSeedConfig::sdk_test_data().total_notes`: 500_000 → 5_000 for
fast iteration while the snapshot path is fresh. Bump back when needed.
- `scripts/setup_local_network.sh`: `CARGO_BUILD_PROFILE` set to `dev`
for fast docker rebuilds during snapshot dev. Flip to `release` when
going back to large N for stress testing.
- grovedb dep rev bumped to `04f2d4243872b65fbec33650e15d85571df385e1`
(branch `feat/snapshot-apply-public-api` — DON'T MERGE; adds three
public methods: `ingest_subtree_sst`, `replace_commitment_tree_subtree_root`,
`raw_storage`).
- New deps in `rs-drive-abci`: `rocksdb` (SstFileWriter), `blake3`
(snapshot checksum). Plus `grovedb-path` + `grovedb-storage` as dev-deps
for the data-location test.
Tests
-----
- `dump_only_default_and_aux_cfs_under_shielded_subtree_prefix` — pins
the CF layout the dump enumerates (default CF only, contrary to design's
original §3 guess of default + aux).
- `snapshot_dump_apply_preserves_anchor` — in-process roundtrip; A seeds,
dumps, B applies, asserts anchors match byte-for-byte.
- `bench_native_seed_full` — bake-feasibility benchmark; measures
N=500k seed in release locally.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
shumkov
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the gap from #3714 for benchmarking wallet sync at scale. When a local devnet's drive-abci is built with
--cfg=create_sdk_test_data,InitChainnow pre-populates the Orchard shielded pool with 500_000 notes (8 owned across two deterministic ZIP-32 test wallets) before block 1 is proposed. No per-note Halo 2 proving at bring-up; deterministic root hash for byte-identical re-runs.Adopts Option A from the design audit: seeder runs through the production
commitment_tree_insert_oppath, batched into a singleapply_drive_operationscall so GroveDB'spreprocess_commitment_tree_opscollapses 500k inserts into one Sinsemilla-frontier load +append_with_mem_bufferloop + one Merk propagation.What's in scope
Drive-abci seeder
create_data_for_shielded_poolinsidecreate_sdk_test_data(regtest-only runtime check kept).cmx, opaque 216-byte ciphertext) + owned (realNote::from_partsencrypted to a fixed wallet viaOrchardNoteEncryption<DashMemo>).block_height = 1(matches production's first end-of-block anchor).StdRngthreaded through every randomness call (seed_a = [0x73;32],seed_b = [0x74;32], derived viaSpendingKey::from_zip32_seed(seed, coin_type=1, account=0)).orchard+zip32deps onrs-drive-abci(gated for use only inside the cfg).dashmate generic
buildArgsconfigdockerBuildschema:buildArgs: Record<string,string>. Replaces ad-hoc shell-env passthrough as the single source of truth for image build args.scripts/setup_local_network.shwritesbuildArgs.SDK_TEST_DATA = "true"+buildArgs.CARGO_BUILD_PROFILE = "release"to eachlocal_Nafterdashmate setup local. Marked TODO/temporary — release profile is mandatory while the seeder is slow.generateEnvsFactoryforwards both drive-abci and rs-dapi build-args into compose's env so${KEY}substitution inbuild.args:resolves.dashmate config setbug fixconfig.get(path)rejected legal sets of new keys underadditionalProperties: <schema>maps. Replaced withConfig.isSchemaPathAllowed(path)— a schema walker that descendsproperties,additionalPropertiesvalue schemas, and$ref. 15 unit tests pin it.What's NOT in scope (follow-ups)
docs/shielded-seeder-performance.mdfor the breakdown and removal-conditions of the TODO.Drive::validate_anchor_existsvia constructed but unsubmitted spend) — design doc §9 acceptance, deemed tautological given current sync test (we drop it here, can revisit).SEED_A/SEED_B— currently duplicated between drive-abci side and the platform-wallet functional test with// keep in synccomment.Test plan
cargo test -p drive-abci --lib 'create_genesis_state::test::shielded'withRUSTFLAGS='--cfg create_sdk_test_data --cfg tokio_unstable'): 23/23 passing — wallet derivation, generator (filler + owned + ρ uniqueness + ciphertext + determinism + per-wallet decrypt + cross-wallet privacy + aggregate balance), Drive integration (count + anchor + cross-platform byte-identical determinism).yarn mocha test/unit/config/Config.spec.js): 15/15 passing.PlatformWalletManager → bind_shielded → coordinator.sync → shielded_balancesrecovers400_000per wallet against an SDK_TEST_DATA-seeded chain (earlier run at N=500). Functional test ships inpackages/rs-platform-wallet/tests/shielded_sync.rsas#[ignore]-d; run withcargo test -p platform-wallet --test shielded_sync --features shielded -- --ignored --nocaptureafteryarn startbrings a seeded devnet up.🤖 Generated with Claude Code