Repository navigation
perf(blockchain): import blocks and envelopes before the engine answers - #773
MegaRedHand wants to merge 3 commits into
Conversation
The chain actor waited for every engine_newPayload inside the import, so the head moved only after EL + state transition and the actor sat idle for the EL round trip, handling nothing else. Gloas envelopes did the same. A pre-gloas block that is an optimistic candidate now imports as optimistic as soon as its consensus checks pass, and a gloas envelope applies with its payload NOT_VALIDATED. The newPayload question goes out first, from a tokio task that reports back with a NewPayloadAnswered message, and the answer is applied then as optimistic-sync.md applies a verdict. Optimism already gates attestation data, so validators still attest only to validated payloads. - Payloads are sent parent-before-child, at most 4 in flight, and forkchoiceUpdated never names a payload whose request is queued or in flight: the Engine API orders only forkchoiceUpdated, the client's connection pool can reorder requests, and a payload whose parent the EL has not seen is answered SYNCING and may start a peer sync. - At most 128 payloads wait for an answer; past that the import is held until answers free room, so catch-up cannot run unboundedly ahead. - No answer after the retry ladder keeps the block optimistic and asks again each slot. optimistic-sync.md says such a block MUST NOT be applied; it already is, so this is documented as a deviation. - An envelope answered INVALID with a null latestValidHash (how Engine API V2+ reports a block hash mismatch) is taken back out and only that copy refused, matching Lighthouse and Prysm, which reject the envelope without condemning the payload. Blocks built on a FULL payload wait for its answer, as they do in both clients, so nothing is built on an envelope that gets taken back out. - A VALID answer for the current head sends `head` again without the optimistic flag, for validator clients that react to head events. Non-candidate blocks (merge transition) and servers without a mailbox (offline replay) still wait for the answer inline.
Skipping forkchoiceUpdated while the head's newPayload was queued or in flight starved the execution client of a head for the whole of a catch-up: parent ordering sends a linear chain one payload at a time, so the newest imported block, the head, always waits behind its parent's request. The call now names the newest payload on the head's chain the client has been asked about, walking the waiting payloads' parent hashes, and is sent again when the head's own answer arrives. Also forgets the waiting asks of payloads a condemned gloas payload took with it, names the limits by their constants in the docs, lists the new metrics in docs/metrics.md, and covers a child that arrives after its parent's envelope is applied but before the answer.
…rs (#773) The chain actor waited for every engine_newPayload inside the import, so the head moved only after EL + state transition and the actor sat idle for the EL round trip, handling nothing else. Gloas envelopes did the same. A pre-gloas block that is an optimistic candidate now imports as optimistic as soon as its consensus checks pass, and a gloas envelope applies with its payload NOT_VALIDATED. The newPayload question goes out first, from a tokio task that reports back with a NewPayloadAnswered message, and the answer is applied then as optimistic-sync.md applies a verdict. Optimism already gates attestation data, so validators still attest only to validated payloads. - Payloads are sent parent-before-child, at most 4 in flight, and forkchoiceUpdated names the newest payload on the head's chain whose request is not queued or in flight (walking the waiting payloads' parent hashes), sent again when the head's own answer arrives, so a catch-up still feeds the EL a head. - At most 128 payloads wait for an answer; past that the import is held until answers free room, so catch-up cannot run unboundedly ahead. - No answer after the retry ladder keeps the block optimistic and asks again each slot. optimistic-sync.md says such a block MUST NOT be applied; it already is, so this is documented as a deviation. - An envelope answered INVALID with a null latestValidHash (how Engine API V2+ reports a block hash mismatch) is taken back out and only that copy refused, matching Lighthouse and Prysm. Blocks built on a FULL payload wait for its answer, so nothing is built on an envelope that gets taken back out. - A VALID answer for the current head sends `head` again without the optimistic flag, for validator clients that react to head events. Non-candidate blocks (merge transition) and servers without a mailbox (offline replay, the import benchmark) still wait for the answer inline. On nightly, that inline wait is #755's: the request goes out from a task and the block's state transition runs meanwhile, so the inline path keeps its overlap, speculation metrics and EngineAnswer telemetry rows, and #773's own inline helper (ask_payload_inline) is dropped. The engine section of an import that asks without waiting is a recorded zero. An applied envelope still publishes #748's execution_payload and execution_payload_available events, now before its answer. Answers that arrive later write no el_sent/el_verdict telemetry rows (#765): those come only from an import that waited. Cherry-picked from perf/async-new-payload at 33d1969 (f23010e, 33d1969). Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB
…rs (#773): write the engine rows and attestable when the answer is applied fix(telemetry): time a block's attestable and engine rows by its answer With #773 a block that may be imported optimistically no longer waits for engine_newPayload, so #765's block_events rows lost the execution client: el_sent and el_verdict came only from an import that waited (none on the live node since), and attestable, written at the head recomputation after the cascade, now marked a head the block was optimistic in, which validators cannot attest to. On beacon-eth4 that cut received-to-attestable from a 75 ms median to 40 ms without any block becoming attestable sooner, so the Silver comparison read faster than it was. - Each pre-gloas ask records the import's source and when its request went out; the request task stamps when its answer came back. - Applying the answer writes el_sent and el_verdict at those instants, and attestable at the moment a VALID (or NOT_REQUIRED) answer marks the block validated. The head recomputation leaves out the attestable of a block whose answer is still out. The answer is handled only after the import that asked, head recomputation and forkchoiceUpdated included, so the applied moment is always the later one; attestable - el_verdict is the time the answer waited in the mailbox. - No answer writes nothing (the redrive's answer does, with its own el_sent); an answer for a block that left fork choice writes nothing; a SYNCING/ACCEPTED or INVALID answer writes el_sent and el_verdict only. Envelopes write no block rows, as before. docs/telemetry.md says when each row is written on the asynchronous path. Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB
…ne answers (#773)" Temporarily takes #773 (0b62d9a) and its follow-up (4117c5a) back out of nightly while it is reworked. With #773 a pre-gloas block entered fork choice before engine_newPayload answered, so for the length of the round trip the head was a block the execution client had not validated. The Beacon API reported it as such: the `block` and first `head`/`head_v2` events of every Hoodi block carried `execution_optimistic: true`, and REST answers about the head did too. Every other client imports only after the answer, so validator clients read an optimistic head as a syncing execution client; Vero closes its event stream on one and reconnects ten seconds later, which on ethlambda-3's Hoodi beacon (nightly 7f57b45) was every block, for all six validator nodes reading it. #773 comes back once a block becomes visible to fork choice and the API only with its answer in hand, as before it.
Reverted from
|
Why
The chain actor awaited every
engine_newPayloadinside the block import. The head moved only after the EL round trip plus the state transition, and the actor handled nothing else (gossip, ticks, other imports) while it waited. Gloas envelopes did the same.What
A pre-gloas block that is an optimistic candidate now imports as optimistic as soon as its consensus checks pass, and a gloas envelope applies with its payload
NOT_VALIDATED. ThenewPayloadquestion goes out first, from a tokio task that reports back with aNewPayloadAnsweredmessage. The answer is applied then, the wayoptimistic-sync.mdapplies a verdict. Optimism already gates attestation data (503), so validators still attest only to validated payloads.VALIDheadsent again without the optimistic flagVALID, held children releasedSYNCING/ACCEPTEDNOT_VALIDATED, held children releasedINVALIDwith a hashinvalidate_block_payload, shared with the FCU path)condemn_payload_chain)INVALIDwith no hashrevert_envelope): only that copy is refusedRules (module docs in
crates/blockchain/src/beacon_new_payload.rs):MAX_NEW_PAYLOADS_IN_FLIGHT(4) at once. The Engine API orders onlyforkchoiceUpdated(common.md), the engine client's connection pool can reorder requests, and a payload whose parent the EL hasn't seen is answeredSYNCINGand may start a peer sync.forkchoiceUpdatednames the newest payload on the head's chain whose request is not queued or in flight, and is sent again when the head's answer arrives. Skipping the call instead would starve the EL of a head for the whole of a catch-up, where the newest block's request always waits behind its parent's.MAX_PAYLOADS_AWAITING_VERDICT(128) payloads wait. Past that the import is held (a pre-gloas block is written as a pending block; an envelope waits inawaiting_engine) until answers free room.get_parent_import_status) and Prysm (ParentPayloadReady) hold the same blocks, since their FULL node exists only after the answer.engine_newPayloadV2a block hash mismatch is answeredINVALIDwith a nulllatestValidHash(INVALID_BLOCK_HASHis "supplanted byINVALID", execution-apisshanghai.md). Lighthouse (unstable25b9b75) and Prysm (developbf0db8c) both reject only the envelope there and condemn nothing. This PR matches that by reverting the applied envelope. The code comment calling this an ethrex quirk is corrected.Spec deviation
optimistic-sync.mdsays a block the EL gave no answer for MUST NOT be applied to fork choice. Here it already is, so it stays optimistic and is asked again each slot. This is documented indocs/spec_deviations.md("Payloads are applied before the execution client answers"), which replaces the gloas-only entry.Relation to other PRs
process_block.Testing
ethlambda-blockchain --lib --features beacon-spec-tests(with consensus-spec-tests v1.7.0-beta.2): 353 passed. New tests cover:forkchoice_head, the bound;VALID/SYNCING/INVALID/no answer/stale);INVALID; condemnation on a real-hashINVALID; no answer then a re-ask.cargo test --workspace --lib --bins,make lint.#[started]hook that records the actor's reference, and dispatch ofNewPayloadAnsweredthrough the real mailbox). Every test feeds answers through a channel. A follower run is the end-to-end check.New metrics:
lean_engine_new_payload_in_flight,lean_engine_payloads_awaiting_verdict,lean_engine_new_payload_answers_total{kind,outcome}(indocs/metrics.md).