Non-validator rejects fresh block batches with "Bad block proposer proposer=X expected=Y

[thetraderhub]

Non-validator rejects first client-block batch with Bad block proposer after successful fresh bootstrap

We are trying to run a Mainnet Hyperliquid non-validating node in Tokyo for low-latency read-only market-data research.

Peering and snapshot bootstrap now work reliably, but the node repeatedly rejects received client blocks with:

Bad block proposer proposer=X expected=Y

This reproduces across multiple peers, fresh snapshots, and a completely clean state using the current official signed binary.

Environment

• AWS region: ap-northeast-1 (Tokyo)

• Instance: r6a.4xlarge

• OS: Ubuntu 24.04.4 LTS

• Kernel: 7.0.0-1012-aws

• Mainnet visor.json: {"chain":"Mainnet"}

• Ports 4001 / 4002 publicly reachable and confirmed externally

• hl-visor SHA256:
  89338207c793...212e31

• Visor signature verified with:
  Hyperliquid <notices@hyperfoundation.org>

• Child binary fetched automatically by visor:

  • binary_index: 109
  • commit/version:
    ed713469199d55a6d03bb5c54746a035e078d2ed
  • build timestamp:
    2026-09-05 16:23:30 +0800
  • uncommitted=false

Command:

run-non-validator --write-trades --write-raw-book-diffs --batch-by-block --disable-output-file-buffering

Gossip configuration contains current gossipRootIps from the API plus known community peers, with try_new_peers=true.

What works

• Snapshot peers accept the node.
• Fresh snapshots download successfully.
• Block peers accept the node.
• Client-block batches are returned within approximately one second.
• One block was successfully applied on one run.

What fails

The received block batch is then rejected by the local node because the proposer in the block does not match the proposer expected by its local state.

This has reproduced:

• across 4 block peers;
• across 3 fresh snapshots;
• after fully deleting both ~/hl/hyperliquid_data and ~/hl/data;
• after re-downloading the official visor and child binary.

Example from a fully clean restart:

• snapshot height: 1143650000
• snapshot serialized: 2026-09-11 11:45:05.2 UTC
• peer: 180.189.55.18
• received block batch:
  1444474889-1444474988
• received approximately:
  2026-09-11 11:45:06.5 UTC
• received proposer:
  0xb796a00b6e50c3dd46e43346c921fe8e146f4e06
• expected proposer:
  0x4a900e9266b00e3d5310e09bb49520b80f2bd417

The block is rejected immediately.

The snapshot ABCI-height numbering (~1.14365e9) and client-block numbering (~1.44447e9) differ significantly. We do not know whether that is expected or whether the local proposer set is being reconstructed from an incompatible epoch/state.

Reproduction history

The same proposer-mismatch condition was observed with peers including:

• 64.31.51.137
• 54.168.210.170
• 95.216.201.187
• 54.250.245.62
• 180.189.55.18

Fresh snapshots and different block peers produce the same failure.

Additional observations

We also see:

• many Peer full responses before a usable block peer becomes available;
• occasional abci_stream ... timed out;
• failed to get new peers between attempts.

However, those do not appear to explain the proposer failure because the node successfully downloads a fresh snapshot and receives the client blocks before rejecting them locally.

Question

Why would a freshly bootstrapped Mainnet non-validating node using the current official signed binary expect a different proposer for the first received client-block batch?

Is there:

• a required bootstrap/state-reset step we are missing;
• a known binary/snapshot incompatibility;
• a proposer-set/epoch-sync issue;
• or a newer node/visor configuration required for non-validator catch-up?

We have preserved:

• full journal;
• extracted failure log;
• snapshot/block/proposer table;
• binary hashes and versions.

Happy to provide both logs.

This node is strictly read-only; no trading keys or accounts are involved.

node_failure_evidence_20260911_1137.log