Skip to content

startup: decouple precache and mempool sync from server startup#218

Draft
DeviaVir wants to merge 1 commit into
new-indexfrom
startup-decouple-mempool-precache
Draft

startup: decouple precache and mempool sync from server startup#218
DeviaVir wants to merge 1 commit into
new-indexfrom
startup-decouple-mempool-precache

Conversation

@DeviaVir

@DeviaVir DeviaVir commented Jun 12, 2026

Copy link
Copy Markdown

Problem

After a restart, electrs refuses to listen on any port until it has serially completed the popular-scripts precache and a full initial mempool sync over JSONRPC. With a congested mainnet mempool this keeps an instance unreachable for 15-30+ minutes, even though the chain index is fully usable within seconds of opening the database.

Measured on a production mainnet instance (2026-06-11): ~17 minutes from process start to first accepted connection, of which DB open + chain catch-up was 17s, precache 17s, and the initial mempool sync 15.2 min (including a bitcoind disconnected while receiving mid-load).

Recent perf work (e.g. running parallel RPC requests on the rpc_threads pool) makes the mempool sync faster, but doesn't change the ordering: however quick the sync gets, the servers still won't accept a single connection until it finishes, and the precache still runs serially before it. This PR removes the waiting and adds the health signal needed to do that safely.

Changes

  1. Precache runs in a background thread. It is pure cache warming and never affects correctness; the scripthash file is still read upfront so a bad path fails fast. In practice it gets a head start while the mempool syncs, so it's typically warm before traffic arrives.
  2. New flag --serve-during-mempool-sync (default off → zero behavior change). When enabled, the REST/Electrum servers start before the initial mempool sync. Chain/block/tx/address queries are fully correct during the sync; mempool-derived data (unconfirmed history entries, outspends, /mempool*, Electrum subscription status) is incomplete until the first full sync. Fee estimates are unaffected (they come from bitcoind's estimatesmartfee).
  3. New endpoint GET /health/ready{"chain_synced":true,"mempool_synced":bool}, HTTP 200 once the initial mempool sync has completed, 503 before. Cache-Control: no-cache.

Operational coupling (important)

Enabling --serve-during-mempool-sync while health checks are plain TCP checks would route traffic to instances with an incomplete mempool view. Rollout order per deployment:

  1. Ship this version (flag off), no-op apart from the background precache.
  2. Switch readiness/health checks from TCP to GET /health/ready on the HTTP port.
  3. Add --serve-during-mempool-sync to the start arguments.

End state: the process accepts connections ~30s after start, health checks still flip to healthy only when the mempool is synced (same externally visible behavior as today), and operators gain visibility into which startup phase an instance is in instead of staring at a closed port. Deployments fronted by redundant instances can later opt to gate health on chain_synced only and accept a briefly stale mempool to cut restart downtime to ~1 minute.

Notes for reviewers

  • Mempool::update already returns Ok(true) only after a full consistent snapshot; the synced flag is set at both success returns and is monotonic.
  • During the initial sync the final add() holds the mempool write lock for the bulk apply; early-served requests touching the mempool can briefly block. Traffic is unaffected when health checks gate on /health/ready.
  • Tested: cargo check --all-targets (both feature sets), cargo test --lib, CI integration suites green.

@DeviaVir DeviaVir force-pushed the startup-decouple-mempool-precache branch from 7d872cc to 7a82462 Compare June 12, 2026 07:28
After a pod/node restart, electrs refused to listen on any port until it
had serially completed (1) the popular-scripts precache and (2) a full
initial mempool sync over JSONRPC. On a congested mainnet mempool this
kept instances unready for 15-30+ minutes even though the chain index
was fully usable within seconds (measured: 17 min total, of which 15 min
was mempool sync).

- Run --precache-scripts in a background thread; it is pure cache
  warming and never affects correctness. The file is still read upfront
  to fail fast on a bad path.
- Add --serve-during-mempool-sync (default off, no behavior change):
  start the REST/Electrum servers before the initial mempool sync.
  Chain-based queries are fully correct during the sync; mempool-derived
  data is incomplete until the first full sync completes.
- Add GET /health/ready returning {chain_synced, mempool_synced} with
  200 once the initial mempool sync has completed and 503 before that,
  so load balancers / readiness probes can keep early-serving instances
  out of rotation. Deployments enabling --serve-during-mempool-sync MUST
  switch readiness probes from a TCP check to this endpoint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@DeviaVir DeviaVir force-pushed the startup-decouple-mempool-precache branch from 7a82462 to e9f4b3b Compare June 12, 2026 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant