Skip to content

Keep reload restoring fee recipients, refresh UIs after repairs, show import and delete progress - #449

Merged
3alpha merged 2 commits into
mainfrom
fix/reload-recovery
Nov 16, 2026
Merged

3alpha merged 2 commits into
mainfrom
fix/reload-recovery

Conversation

@3alpha

@3alpha 3alpha commented Nov 16, 2026

Copy link
Copy Markdown
Member

Summary

Fixes two reconciliation gaps found by a recovery audit of 0.2.0-rc.1, and improves progress feedback for large imports and deletes based on testing rc.1 on a Hoodi node with 300 keystores.

Changes

Reload keeps configuring validators when cleanup fails (445bf93)

  • Problem (older code): a failure while deleting obsolete validator-client keys aborted the whole reload pass, so fee recipients were never restored. This repeated on every pass while the deletion kept failing.
  • Fix: signer cleanup, adding keys to the validator client and removing obsolete keys log their failure and the pass continues to fee recipients. Failed deletions stay in the key lists and are retried on the next pass.
  • Still stops the pass: the signer upcheck and key listing (the signer is authoritative, so a bad listing must not prune anything), database pruning, and listing the validator client's keys.

Fee-only repairs refresh idle UIs (445bf93)

  • Problem (introduced in Improve large validator batches, update Node to 24.22.3, disable builder config #448): reload only announced changes to validator-client keys. A repaired fee recipient changes no database record, so a UI that last showed a mismatch kept showing it.
  • Fix: the validator list remembers when its last response showed a problem (a fee mismatch, a failed read, or reads cut off by the deadline). A completed reload pass then announces itself, so UIs list again and re-check fee recipients by GET. A clean pass announces nothing, so idle UIs are not made to re-read every fee recipient each minute; a persisting problem is announced after each pass.
  • Unchanged: the read state moved to validatorList/readState.ts so reload can use it. A matching fee check is still cached for up to 60s.

Batch progress for imports and deletes (b42e613)

  • Imports: while running, validators show as "importing" (the batch in flight) or "queued", not "not attempted"; that status now only appears once a run stops or fails. The progress line names the batch in flight and the bar shows it as a moving buffer section, so the dialog visibly works during the roughly 10s each batch takes.
  • Deletes: run in batches of 25 instead of one request, with the same progress view, "Stop after this batch", and "Retry N not deleted". Each batch's slashing protection export is merged into one download, which stays available after a stop or failure and across retries; an export that cannot be merged is offered as its own file rather than dropped. The confirmation lists the first 10 pubkeys and a count.
  • Shared code: import and delete use one batch runner (batchRunner.ts) and one progress view (BatchProgressView.tsx).

Testing

  • yarn lint, yarn format:check, yarn build and yarn test pass (common 13, brain 111 unit + 10 e2e, UI 65).
  • Reload: unit tests show fee recipients are restored on every pass while removing obsolete keys keeps failing (and the removal is retried), when adding keys fails, and when signer cleanup fails (database records kept). A signer that cannot be listed still stops the pass. The first three fail on the previous code.
  • Idle UI (e2e): the validator client's fee recipient changes behind the Brain's back, a UI shows the mismatch and idles, reload repairs it, a push arrives and the next list is correct; a later clean pass sends no push. Without the fix the test times out waiting for the push.
  • Batches: unit tests cover queued and in-flight states during import, delete planning, aligned results, stop, a failed batch, retry positions and slashing export merging.
  • Live rc.1 baseline (Hoodi, Lighthouse, Geth, web3signer): 300 keystores import in 15 batches of about 10.4s each (157s in total, decryption-bound); listing 301 validators takes about 300ms. This PR's UI has not been run on that node yet; it is worth checking import and delete progress there once it is in an rc.

Signer cleanup and adding or removing validator-client keys no longer abort
the reload pass, so fee recipients are restored even when cleanup keeps
failing; failed removals stay listed and are retried. The signer listing,
database pruning and validator-client listing still stop the pass.

When the last validator list showed a problem (a fee mismatch, a failed or
unfinished read), a completed reload pass now announces itself, so idle UIs
list again and re-check fee recipients by GET. Clean passes stay silent.
Imports report validators as importing or queued while running and only
call them not attempted once a run stops; the progress bar shows the batch
in flight. Deletes run in batches of 25 with the same progress view, stop
and retry, and merge every batch's slashing protection export into one
download without dropping any. Import and delete share one batch runner and
one progress view.
@3alpha
3alpha requested a review from a team as a code owner November 16, 2026 11:59
@3alpha
3alpha merged commit 6925786 into main Nov 16, 2026
6 checks passed
@3alpha
3alpha deleted the fix/reload-recovery branch November 16, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant