Duplicate blocks in cache silently block subgraph indexing

[madumas]

Summary

Duplicate blocks accumulating in the block cache (chainN.blocks) can cause subgraphs to stop indexing. The subgraph remains health=healthy and active=true but processes 0 blocks/s. The only symptom is an increasing blocks_behind count.

The error is only visible at DEBUG level:

Block stream produced a non-fatal error, error: block stream error Block 14500860 does not contain hash

This repeats every ~7 minutes indefinitely. The subgraph never fails or enters an unhealthy state — it just stops advancing.

Root cause

The block ingestor stores both branches of a reorg in the block cache without cleaning up orphaned entries. Under certain conditions (not fully characterized — it does not happen at every duplicate block), a subgraph hits a block where the hash it expects no longer resolves, causing the "does not contain hash" error loop.

It is unclear what exact combination of factors triggers the issue — the presence of duplicate blocks is a necessary condition, but there may be additional factors (e.g., specific reorg depth, timing of block ingestion vs subgraph processing).

Impact observed (production, ~250 subgraphs)

Chain	Duplicate blocks	Range checked	Ratio
moonbeam	2,400+	16,000 blocks	~15%
mainnet (Ethereum PoS)	1,382	full cache	—

• Blocked subgraphs can show very high reorg_count (779,742 for the moonbeam case)
• The issue is silent — no WARN/ERROR at default log level, health stays healthy
• Observed on chains using RPC polling (not firehose)

Workaround

Remove duplicate blocks from the cache and perform a short rewind:

-- Find duplicate blocks
SELECT number, count(*) FROM chainN.blocks
GROUP BY number HAVING count(*) > 1;

# Remove duplicates (or truncate the whole chain cache)
graphman chain check-blocks <chain> by-range -f <from> -t <to> --delete-duplicates
# or: graphman chain truncate <chain>

# Then rewind the stuck subgraph with the correct block hash
graphman rewind --force --block-number <N> --block-hash <correct_hash> <deployment>

Related

• [Bug] Graph-node save incorrect block cache during reorgs #4792 — reports the same duplicate block storage issue

Environment

• graph-node v0.41.1