Skip to content

Improve large validator batches, update Node to 24.22.3, disable builder config - #448

Merged
3alpha merged 8 commits into
mainfrom
fix/large-validator-batches
Nov 16, 2026
Merged

3alpha merged 8 commits into
mainfrom
fix/large-validator-batches

Conversation

@3alpha

@3alpha 3alpha commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Summary

Improves the experience for large validator batches, fixes issues found while reviewing last month's changes, updates Node for the November 2026 security release, and switches off the builder configuration feature for now. Validator-client writes and fee-recipient checks keep the behavior they have on main; the intentional behavior changes are listed below.

Changes

Importing large batches

  • Batches: the UI imports keystores in batches of up to 20, sized by exact encoded bytes including each key's slashing history, so every call stays well inside the 120s RPC timeout and the payload limit. Each batch carries only its own keys' slashing protection history.
  • Too large: a key that cannot fit one request on its own is reported as an error and the import continues.
  • Progress: the dialog shows live progress, counts (imported, already present, failed, not attempted) and a "show only problems" filter. Users can stop after the current batch and retry only the validators that were not imported; keys that already reached the signer come back as "already present."
  • Form: keystore cards render 25 at a time, with a count, "Remove all", and an alert pointing to keystores that still need a password, protocol or fee recipient.
  • Signer timeout: the signer import request waits up to 10 minutes, because dropping it does not stop the signer, it only loses the result.

Listing large sets of validators

  • Client reads: each list request reads signer and validator client membership live, and fee recipients with getFeeRecipient(), within a 10s budget and 8 parallel reads. Reads cut off by the deadline are marked detailsPending; the read cursor advances on dispatch, so follow-up requests continue past keys that hang. Failed reads are reported in detailsError instead of reusing an earlier result.
  • Fee check cache: matching checks are cached for 60s and used only alongside a current membership read. Mismatches are dropped when a validator change is announced, reads that overlap a change are not cached, and a key absent from the client loses its cached check.
  • Beacon details: a new BeaconStatusStore fetches index and withdrawal credentials with chunked bulk requests (500 keys) on its own cron job, separate from reconciliation and its lock, and looks up new keys when the database changes. Indexes are saved to the database, so the Brain API's format=index list includes validators imported since the performance cron was disabled.
  • Live updates: the Brain emits a validatorsChanged Socket.IO event after validator changes (debounced 500ms). Notifications are held while an operation holds the validator lock and sent once it finishes, so other UIs refresh after clients are configured.
  • UI: grid rows are keyed by pubkey and dialogs receive the selected validators, so a refresh can no longer move a selection onto another validator. The list refreshes on push, on reconnect and when a dialog closes; it requests again only while reads were cut off (one request at a time, stopping with Retry after 3 rounds without progress); a failed request shows an error with Retry instead of polling.
  • Smooth oracle: queried only when the validator set or indexes change, and not again after a failed call.

Review fixes

  • Lock waits: UI and launchpad import, update and delete wait at most 30s for the validator operation lock, then fail with "busy" and change nothing. The launchpad API returns 503 for busy.
  • Pubkey casing: updates and builder config reads match stored pubkeys regardless of hex casing.
  • Reload robustness: reload no longer aborts when the signer or validator client returns fewer results than keys sent.
  • Exits: validator indexes are looked up with chunked bulk requests instead of one request per selected validator.
  • Launchpad fee recipients: the routes read stored fee recipients directly instead of listing every validator with client calls; responses are unchanged.

Node security release

  • Node 24.22.1 → 24.22.3 in Dockerfile, Dockerfile.brain.dev, .nvmrc, CI and engines, for the November 2026 security releases (16 CVEs, 7 High, in HTTP/2, TLS, HTTPS, zlib and the permission model). The Brain does not use HTTP/2, TLS servers or the permission model; its only TLS use is outgoing https requests to the validator client and signer.

Builder configuration disabled

  • Flag: featureFlags.builderConfig = false in packages/common/src/featureFlags.ts.
  • What is off: the UI hides builder settings, the builder RPCs and builderConfig in updates are rejected, reload does not replay stored overrides, and BUILDER_IDS is ignored without consuming its one-time import.
  • What is kept: stored builder settings stay in the database, so re-enabling needs no migration.

Behavior changes to be aware of

  • Lock wait limit: the 30s limit rejects operations still queued for the lock; it does not cancel operations that have already started.
  • Signer import timeout: at 10 minutes it can keep the lock held after the UI's 2-minute RPC timeout, so a UI timeout still means "outcome uncertain."
  • Beacon traffic: listing and exits use bulk POST /eth/v1/beacon/states/head/validators requests instead of per-validator lookups; the beacon job runs at startup (reads only) and every minute. Reconciliation keeps its existing schedule and makes no beacon calls.
  • Validator-client writes: unchanged from main; they stay sequential.
  • Builder configuration: disabled, which changes validator-client API traffic for nodes that used it.

Testing

  • Install, yarn lint, yarn format:check, yarn build and yarn test all pass (common 13, brain 107 unit + 9 e2e, UI 59).
  • E2E (packages/brain/test/e2e): the real UI server, RPC handler, database, lock, reload job and API clients run against fake web3signer, validator client and beacon node servers, driven over Socket.IO with the UI's own RpcClient and batch import code. Scenarios: batched import of 120 keystores, a failed batch with retry, per-key signer rejections, live pushes to a second UI, a final push after slow validator-client writes, 600 validators listed with beacon details arriving by push, unreachable beacon node and validator client reported as errors and recovering, reload repairing a client that lost its keys, and builder configuration staying disabled.
  • Unit: bounded and resumable list reads, fee check caching and invalidation, beacon chunking, index persistence and failure handling, batch sizing with slashing history, exit bulk lookups, the lock and notification holds.
  • The production image builds on node:24.22.3-alpine3.24 (arm64, locally).
  • Not yet run against a real signer, validator client and beacon node, and React components have no tests. Recommended before merge: import a few hundred keystores on Hoodi and watch progress, stop and retry, and the list filling in.

UI and API validator operations now give up before the RPC timeout instead of
running after the caller has stopped waiting. Updates and builder config reads
resolve stored pubkeys regardless of hex casing, and reload tolerates a short
signer delete response.
Import runs in batches of 20 with live progress, stop and retry; each batch
carries only its own slashing history. The Brain pushes a validatorsChanged
event after every database write so open lists refresh themselves, and the
list keeps refreshing while rows are marked detailsPending, without blanking
the table. Fee checks are cached like beacon data so large lists converge.
Signer imports get a 10 minute timeout, fee recipients are applied with
bounded concurrency on import and reload, and the import form pages its
keystore cards and points at files that still need input.
Covers the HTTP/2, TLS, HTTPS, zlib and permission model fixes published on
2026-11-11 (https://nodejs.org/en/blog/vulnerability/november-2026-security-releases).
A failed call left the subscription map unset, so every live refresh round
while validator details were loading queried the oracle again.
Gate the Gloas builder feature behind featureFlags.builderConfig (off): the UI
hides builder settings, builder RPCs and update fields are rejected, reload no
longer replays stored overrides, and BUILDER_IDS is ignored without consuming
its one-time import. Stored builder settings are kept so the feature can be
re-enabled without migration.
@3alpha
3alpha requested a review from a team as a code owner November 16, 2026 09:07
Run the real UI server, RPC handler, database, lock, reload and API clients
against fake web3signer, validator client and beacon node HTTP servers, and
drive them over Socket.IO with the UI's own RpcClient and batch import code.
Covers batched import of 120 keys, a failed batch with retry, per-key signer
rejections, live validatorsChanged pushes, progressive list loading past the
10s deadline, and builder configuration staying disabled. yarn test now runs
them in CI.
- Capture the previous rows before scheduling the selection update, so a
  live refresh cannot move the selection onto a different validator.
- Size import batches by exact encoded bytes, including each key's slashing
  history; report a key too large for any request instead of stopping.
- Hold change notifications while a validator operation runs and send one
  when it finishes, so other UIs refresh after clients are configured.
  Reload also announces validator-client membership repairs.
…n store

Restore the validator-client behavior from main: sequential writes in import,
update, delete cleanup and reconciliation, fee-recipient correctness from
getFeeRecipient() reads, and live signer/validator membership reads on every
list request, reported as unavailable when they fail.

- getValidators reads membership and fee recipients within a 10s budget and
  8 parallel reads. Cut-off reads are pending and the cursor advances on
  dispatch, so follow-up requests continue past hung keys. Matching fee checks
  are cached for 60s only alongside a current membership read; mismatches are
  dropped on change notifications, reads overlapping a change are not cached,
  and keys absent from the client lose their cached check.
- BeaconStatusStore holds beacon details only, fetched in chunked bulk
  requests by its own cron job (separate from reconciliation and its lock),
  looked up for new keys on database changes, and persisted as indexes.
- Exits look up indexes with chunked bulk requests.
- The UI keys grid rows by pubkey and passes selected validators to dialogs;
  it requests the list again only while reads were cut off (bounded,
  coalesced), waits for pushes for beacon details, and shows errors with
  Retry instead of polling.
- Launchpad fee-recipient routes read stored values directly; busy lock
  errors return 503.
@3alpha
3alpha merged commit 47b31f4 into main Nov 16, 2026
6 checks passed
@3alpha
3alpha deleted the fix/large-validator-batches branch November 16, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant