`graphman copy`: since v0.44.0 the destination gets the layout's GiST/`id` indexes regardless of the source's index set (#5140 / #5425)

[madumas]

Summary

Since PR #6434 (in v0.44.0), graphman copy builds the destination's index DDL from the default layout; before it, the copy path passed the source's actual index list (index_def) down to the DDL. The source's index list is only consulted afterwards, and only for indexes that pass CreateIndex::to_postpone() (BTree, key ≠ id alone, attr_ prefix). Every other index — the GiST (col, block_range) indexes on reference columns, the BTree on id, and any manually created index — is now created from the layout (or not created at all, for manual ones), blind to what the source carries.

As far as we can read the code, this undoes requirement 1 of #5140 ("Create the indexes based on the index creation DDL of the source, not based on the default"), which #5425 implemented. #6434 does not mention either, and #5140 is still open. We may be missing a reason for the change — if it is intentional, a note in the changelog would help operators who slim down large deployments before copying them.

For an indexer that slims down large production deployments with graphman index drop and then copies them out to rebalance shards, a copy now comes back with all the dropped indexes — in our case +1.35 TB on a 1.6 TB deployment.

Observed (v0.45.0, GRAPH_POSTPONE_ATTRIBUTE_INDEX_CREATION=true on every node incl. the one running graphman)

Uniswap V3 subgraph on bsc (QmUwBMokciw5TY5kHYtqMXXuZdymDy2knYpRxHwqiy85od, swap = 1.61 G rows; sgd numbers below are placeholders), copied shard-to-shard with graphman copy create -o 200 <sgd> <shard> <node>. pg_index on both sides after the copy finished:

	source sgd1001	copy sgd1002
indexes, whole schema	155 — 584 GB	192 — 1 930 GB
swap	3 — 290 GB (swap_pkey, swap_id_block_range_excl, swap_block_range_closed)	9 — 1 431 GB
transaction	4 — 232 GB	5 — 394 GB

Indexes present on the copy and absent from the source, swap table:

attr_11_0_swap_id            btree (id)                        213 GB
attr_11_1_swap_transaction   gist  (transaction, block_range)  241 GB
attr_11_3_swap_pool          gist  (pool, block_range)         231 GB
attr_11_4_swap_token_0       gist  (token_0, block_range)      228 GB
attr_11_5_swap_token_1       gist  (token_1, block_range)      229 GB

Same pattern on transaction (attr_8_0_transaction_id, 163 GB), burn, mint, position_snapshot. The ~10 other BTree attribute indexes of swap that the source also lacks were correctly not recreated — i.e. the env var did its job for the indexes to_postpone() covers, and only those.

Other copies made by the same procedure (sources with a full default index set) came out identical to their source, as expected under either behaviour. We have not re-run a copy of a slimmed-down source under a pre-v0.44 build to compare side by side; from our experience operating v0.4x before the upgrade, copies did not come back with the full default index set.

Minimal controlled reproduction (v0.45.0)

Small deployment on arbitrum-one (QmR3U5haiQg5ymFW9okrkbNBBiRzYJR8kB5Qh9AhFQK5Z5, 228 MB, paused) whose source sgd1003 had every non-primary index dropped — 19 indexes left, all *_pkey (vid) / *_id_key (id). graphman copy create -o 200 sgd1003 <shard> <node> with GRAPH_POSTPONE_ATTRIBUTE_INDEX_CREATION=true in graphman's environment. The copy sgd1004, once finished, has 54 indexes (+35, −0):

category	count	examples
time-travel defaults (*_id_block_range_excl, brin_*, *_block_range_closed, *_block) — created under any version	26	—
attr_N_0_<table>_id BTree (id) — not postponed (lone id key)	6	attr_10_0_transaction_id, attr_9_0_user_token_id, …
GiST (reference, block_range) — not postponed (not BTree)	3	attr_10_1_transaction_from, attr_10_2_transaction_to, attr_9_1_user_token_user
any other BTree attribute index ((col), (col, "block$") on immutable tables) — postponed, hence taken from the source	0	—

I.e. exactly the split to_postpone() makes: everything it covers follows the source (nothing), everything it does not cover comes from the layout.

Side effects beyond disk: the copy's catch-up is slowed by maintaining 9 indexes per swap insert instead of 3, and the destination shard crossed its disk threshold while the copy was still catching up.

Code

All snippets and line numbers below are from the upstream tags v0.42.1 and v0.45.0 of this repository.

v0.42.1 — store/postgres/src/relational/ddl.rs (Table::as_ddl):

if index_def.is_some() && ENV_VARS.postpone_attribute_index_creation {
    // copy/graft: replay the SOURCE's own CREATE INDEX statements
    let arr = index_def.unwrap()
        .indexes_for_table(&self.nsp, &self.name.to_string(), self, false, false, false)?;
    for (_, sql) in arr { writeln!(out, "{};", sql)? }
} else {
    self.create_attribute_indexes(out)?;
    self.create_aggregate_indexes(schema, out)?;
}

index_def came from src_store.load_indexes(src) in SubgraphStore::copy_deployment (subgraph_store.rs:908), through DeploymentStore::create_deployment (deployment_store.rs:179, whose comment reads: "Parameter index_def is used to copy over the definition of the indexes from the source subgraph to the destination one"). indexes_for_table(.., postponed=false, ..) replayed the non-postponed source indexes (GiST, id, manual) at creation, and copy.rs:1236 replayed the postponed ones (postponed=true) once the data was in. Reading this, the destination should have ended up with the source's set (when the env var was set — without it, the else branch built the full default set).

v0.45.0 — same function, after commit 2ef92e7 ("store: Use Table::indexes() for all index creation", part of #6434):

for idx in self.indexes(schema)? {          // time-travel + attribute + aggregate, from the LAYOUT
    if !idx.to_postpone() {
        writeln!(out, "{};", creat.to_sql(&idx)?)?;
    }
}

and copy.rs:306-340 (create_indexes) only recreates index_list.indexes_for_table(&table.dst).filter(|idx| idx.to_postpone()). to_postpone() (index.rs:641) is false for anything non-BTree, for a lone id key, and for any name not prefixed attr_ — so for all of those, the source list is never read.

The PR description frames this as "Remove the index_def: Option<IndexList> parameter threading and simplify callers across copy.rs, prune.rs, and deployment_store.rs". There is no discussion of the copy-from-source behaviour, no review comment on it, and #5140 / #5425 are not referenced. (v0.44.0 additionally shipped the env var inert, so a copy made under v0.44.0 got the entire default set.)

Expected

What #5140 asked for and #5425 implemented, as stated on that issue (2024-04-11):

For copies and grafts, we do not want to create the indexes that we create by default for new subgraphs. Instead, we should look at the source of the graft/copy, and recreate all the indexes that actually exist on the source

Concretely: the destination of a copy would carry the source's index set — the non-postponed part (GiST, id, manual indexes) at creation, the postponed BTree part at the end — plus the layout's indexes for columns that do not exist in the source (grafts). The CreateIndex consolidation of #6434 looks orthogonal to this; what seems missing is for Table::as_ddl (or its copy-path caller) to consult the source's IndexList again for non-postponed indexes, as the pre-#6434 code did. If that is not the intended direction, a way to opt out of the layout's GiST/id indexes on copy would serve the same operational need.

Environment

• graph-node v0.45.0, previously v0.42.1
• GRAPH_POSTPONE_ATTRIBUTE_INDEX_CREATION=true on all nodes and on the graphman host
• PostgreSQL shards; copy via graphman copy create

Happy to provide the full pg_index listing of both sides or test a patch.