fix(core,eth): fix nonce too low and wrong intermediate roots in debug tracing

[gzliudan]

Proposed changes

Debug tracing rebuilds the pre-state of a transaction, and of a whole block, by re-executing the transactions that come before it. Those replays called core.ApplyMessage, which knows nothing about the routing block processing applies. While the XDCX receiver fork is active (mainnet TIPXDCXBlock 38,383,838 … TIPXDCXReceiverDisableBlock 80,370,900), transactions to the system addresses 0x91/0x92/0x93/0x94 are handled by ApplyEmptyTransaction: no EVM execution, no nonce check, no nonce increment and no state change. A plain ApplyMessage replay executes them as ordinary EVM transactions and bumps the sender nonce, so every following transaction of the same sender is replayed against a nonce the chain never had. That replay also credited the block fee to the zero address instead of the coinbase owner and skipped the historical balance bypass.

The block-level APIs made a second, related mistake. IntermediateRoots skipped those transactions with tx.IsSkipNonceTransaction(), a condition that does not consult the fork at all. Outside the fork window block processing does execute them and does bump the nonce, so dropping them loses exactly the nonce increment the following transaction relies on; IntermediateRoots then hits the error and silently truncates, returning the roots collected so far. Inside the window the skip keeps the state right but leaves one root missing. The same unconditional skip in traceBlock and in the state feeder of the JS tracer path — the one that makes debug_traceBlock* fail with nonce too high outside the window and leave a null hole inside it — is a separate change, submitted as #2581.

Symptoms

• nonce too low — debug_traceTransaction on any transaction that follows a system-address transaction of the same sender, while the fork is active. stateAtTransaction never skipped, so it bumped a nonce block processing leaves alone. Apothem block 48,667,667 (0x2e69c13, transactions 5–7), mainnet block 39,083,312 (0x2545d30, transaction 5).
• One root fewer than the block has transactions from debug_intermediateRoots, and one null hole next to it in debug_traceBlockByNumber / ByHash / traceBlock (that side is fix(eth/tracers): fix nonce too high and null entries in block traces #2581), whenever the fork is active.
• The last debug_intermediateRoots root diverges from the block state root whenever the block charges a non-zero fee, because the replay credited it to the zero address instead of the coinbase owner (mainnet block 107,084,658).

Fix

Three commits: one per defect, plus the shared routing and the replay entry point they need.

1. refactor(core): ApplyTransactionWithEVM decided the routing inline, so every replay caller had to re-derive it and each copy drifted. The decision moves into routeTransaction, the per-transaction finalisation into finaliseTxState and the sign-transaction nonce handling into applySignTransactionNonce, so a replay can follow exactly the same path. No behaviour change.
2. fix(eth): stateAtTransaction, the replay behind debug_traceTransaction and debug_traceCall, goes through the shared entry point instead of ApplyMessage.
3. fix(eth/tracers): IntermediateRoots replays through the same entry point and stops skipping the transactions sent to the system addresses, so every transaction gets a root and the roots match block processing.

core.ApplyTransactionForReplay follows exactly the routing of ApplyTransactionWithEVM, records the same log through the same addNonEVMTxLog helper for a transaction the EVM never executes, and skips the receipt and its bloom, which a replay does not need. That log is not a receipt artefact: StateDB.AddLog advances the block-wide log count every later log takes its index from, so a replay that dropped it would make callTracer report indexes the chain never had. It refuses an EVM that carries a tracer (it never fires OnTxStart/OnTxEnd) or a state that is not a *state.StateDB, with sentinel errors the callers can tell apart from a transaction failure; IntermediateRoots reports them instead of returning the roots it has collected so far.

Upstream

No upstream fix to port: geth has no non-EVM transaction concept, so neither the routing, the unconditional skips nor the trailing ApplyMessage have a geth counterpart.

Tests

• core/state_processor_test.go — TestApplyTransactionForReplayKeepsTheNonEVMTxLog: drives the replay and block processing of a transaction to a system address (the trading state address with the receiver fork active, and the block signers address) and compares the logs they record, so the replay cannot lose the log that advances the block-wide log count the following transactions index their logs with; without it the replay records no log where block processing records one.
• eth/state_accessor_test.go — TestStateAtTransactionReplayKeepsNonceLessSenderNonce: replays the block behind the issue (first transaction to the trading state address, receiver fork active from genesis) and asserts the sender nonce after the replay is still 0; the old replay left it at 1, which is what made the following transaction fail with nonce too low.
• eth/tracers/api_test.go — TestTraceTransactionSkipNonceTransactions: drives debug_traceTransaction for the skip-nonce transaction itself and for the follower, over both receiver fork settings.
• eth/tracers/api_test.go — TestIntermediateRootsMatchesBlockProcessing: compares the roots against a replay through core.ApplyTransactionWithEVM, pins the last root to the block root after block finalisation and fails when the coinbase owner fee is dropped; the base has no intermediate-roots test at all.

End-to-end verification

The branch binary was run against archive nodes of both live networks, on the same data directory and the same node as the baseline binary (dev-upgrade @ cdce8fc5c), and the tracing RPCs were compared on the same blocks.

The probes below were run before the two eth/tracers commits moved to #2584. They exercise the replay and the intermediate roots, which this branch still carries unchanged.

Mainnet block 37,849,457 (0x2418971, 204 transactions, outside the receiver fork window, contains sign transactions):

probe	baseline	this branch
debug_intermediateRoots	94 roots for a 204-transaction block (truncated at the failure)	204 roots, last root == block stateRoot

Mainnet block 39,083,312 (0x2545d30, 14 transactions, inside the window; tx 0 goes to 0x92, tx 5 to 0x90 from the same sender with the same nonce) and Apothem block 48,667,667 (0x2e69c13, 8 transactions; tx 0 to 0x92, tx 5 to 0x89 from the same sender with the same nonce):

probe	baseline	this branch
debug_traceTransaction (the follower)	tracing failed: nonce too low	ok
debug_intermediateRoots	13 / 7 roots	14 / 8 roots, last root == block stateRoot

Mainnet block 107,084,658 (0x661FB72, 6 transactions, after the receiver-disable fork, no system-address transaction, baseFeePerGas 12.5 Gwei): only debug_intermediateRoots differs — the baseline last root is not the block stateRoot, this branch's is.

Manual test plan: start a node on that data directory with --rpcapi debug and call debug_traceTransaction on 0x2545d30 and 0x2e69c13, and debug_intermediateRoots on 0x2418971, 0x2545d30 and 0x2e69c13; the baseline binary fails every one of them and this branch passes them.

Regression: on both networks this branch imported testnet and mainnet segments normally, with no bad block, no panic and no error attributable to the change.

Types of changes

• fix: A bug fix
• refactor: A code change that neither fixes a bug nor adds a feature
• test: Adding missing tests or correcting existing tests
• build / ci / chore / docs / feat / perf / revert / style

Impacted Components

• Geth
• Not sure (the changes are the debug JSON-RPC surface plus a behaviour-preserving refactor of the block processing routing in core/state_processor.go)
• Consensus
• Account
• Network
• Smart Contract
• External components

Checklist

• This PR has sufficient test coverage (unit/integration test)
• Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet (see "End-to-end verification")
• Tested the backwards compatibility — the change only affects replays; block processing keeps the same conditions, finalisation and nonce handling, and no API, RPC method or state format changes
• Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs — not done for this branch. It was run against live testnet and mainnet archive nodes instead (segments imported, no bad block, no panic, no new error), and the tracing RPCs were compared block by block against the baseline binary.
• Tested with XDC nodes running this version co-exist with those running the previous version — the branch ran on the live networks next to the rest of the network without being dropped
• Relevant documentation has been updated as part of this PR

Relation to other work

#2584 carries the two eth/tracers commits this one used to contain — callTracer shadowing the real top-level frame of a system-address transaction, and flatCallTracer failing the whole call with invalid number of calls on a block that carries a non-EVM transaction — so the two defects of the debug tracing surface are reviewable on their own. The flatCallTracer row that used to be in the table above moved there with its fix. The two branches touch disjoint files.

#2581 fixes the unconditional skip in traceBlock and in the state feeder of the JS tracer path — the nonce too high and the null hole in debug_traceBlock* — and carries the skipNonceForkCases / newSkipNonceBackend fixtures and the two block-level tests. This PR carries its own copy of those fixtures, because TestTraceTransactionSkipNonceTransactions builds on them; the second of the two PRs to merge drops that copy, which is the only textual overlap between them.

#2578 fixes the give-up paths of the same stateAtTransaction function whose replay this PR rewrites. Both branches touch eth/state_accessor.go; merging #2578 first keeps this one a trivial rebase.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: c6b367e7-40ac-4fe9-b8d4-39b6c0dcff26

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

🟡 Changes recommended

Replay still diverges from canonical processing for synthetic log indexes and sequential TRC21 fee-capacity accounting.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Fixes debug tracing replay so transaction routing and trace frames match canonical block processing.

Changes:

• Adds a shared routing-aware replay path.
• Corrects synthetic versus real call frames.
• Adds regression coverage for nonce, root, and tracer behavior.

File summaries

File	Description
eth/tracers/native/call.go	Selects real or synthetic call frames correctly.
eth/tracers/native/call_test.go	Tests real-frame preservation.
eth/tracers/native/call_flat.go	Supports synthetic non-EVM frames.
eth/tracers/native/call_flat_test.go	Tests flat tracer frame handling.
eth/tracers/internal/tracetest/non_evm_trace_test.go	Tests routing through production processing.
eth/tracers/api.go	Replays block state through shared routing.
eth/tracers/api_test.go	Adds block, transaction, and root regressions.
eth/state_accessor.go	Uses routing-aware transaction replay.
eth/state_accessor_test.go	Tests nonce preservation during replay.
core/state_processor.go	Centralizes routing and adds replay processing.

Review details

Suppressed comments (2)

eth/tracers/internal/tracetest/non_evm_trace_test.go:74

• The test only sends to TradingStateAddrBinary and toggles the receiver fork, so it never exercises routeSign/ApplySignTransaction. Add a case using BlockSignersBinary with TIPSigningBlock active; this is especially important because the PR's reported flatCallTracer failure involved a sign transaction and the new replay branch has distinct nonce and log behavior.

	key, _ := crypto.GenerateKey()
	from := crypto.PubkeyToAddress(key.PublicKey)
	to := common.TradingStateAddrBinary
	blockNumber := common.Big1

eth/state_accessor.go:266

• These replay-error returns discard the release callback obtained from StateAtBlock(..., readOnly=true), so repeated failed traces can retain live trie references. This leak predates this replay change and is already addressed by the linked #2578; ensure that dependency lands before this branch is merged, or include the same deferred release guard here.

		if err := core.ApplyTransactionForReplay(msg, new(core.GasPool).AddGas(tx.Gas()), block.Number(), tx, evm, balance); err != nil {
			// An EVM this replay cannot use is a problem of this caller, not of the
			// transaction: report it as it is instead of blaming the transaction.
			if errors.Is(err, core.ErrReplayTracingEVM) || errors.Is(err, core.ErrReplayStateType) {
				return nil, vm.BlockContext{}, nil, nil, err
			}
			return nil, vm.BlockContext{}, nil, nil, fmt.Errorf("transaction %#x failed: %v", tx.Hash(), err)

• Files reviewed: 10/10 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Pre-Byzantium replay redundantly hashes trie state for every transaction, adding avoidable cost to historical tracing.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 2
• Review effort level: Balanced