fix(merge-insert): detect concurrent inserted join keys

[lance-gatefixer]

Summary

• persist inserted-key Bloom filters for merge-insert join columns even without schema-level primary-key metadata
• carry the same filter through scalar-indexed merge execution
• retry stale overlapping inserts so concurrent merge-inserts do not duplicate new keys

Root cause

Inserted-key conflict metadata was emitted only when the join columns exactly matched the dataset unenforced primary key. Explicit merge keys on ordinary schemas therefore produced filterless update transactions, which conflict resolution treated as disjoint when both writers added only new fragments.

Validation

• cargo fmt --all
• cargo clippy --all --tests --benches -- -D warnings
• focused merge-insert concurrency, transaction-filter, indexed-path, append, and update conflict tests

Fixes #8949

[lance-gatekeeper]

🟡 Gate recommendation: maintainer decision required.

The defect reported in #8949 is real, but this changes an intentional transaction contract. #5633 scoped inserted-key conflict detection to schema primary keys and treated duplicate inserts on ordinary ON columns as valid snapshot-isolation behavior. When the same generalization was proposed in #6018, maintainer feedback required a separate specification change and community vote; the corresponding #6052 closed without acceptance.

Maintainers need to choose whether naming ON columns should itself opt every merge into key-uniqueness conflict serialization—including conservative retries when concurrent merges use different key sets—or whether stronger semantics must remain tied to primary-key metadata or an explicit opt-in. Please record that specification decision first; the implementation can then be assessed against it, including total conflict-key encoding for every supported non-null join type.

[lance-gatefixer]

No code change made: this review requires a maintainer specification decision before the implementation can be revised. PR #5633 established primary-key-only inserted-key conflict detection and treated duplicates on ordinary ON columns as valid snapshot-isolation behavior; the proposed generalization in #6018/#6052 was routed to a specification vote and closed without acceptance. Once maintainers choose implicit ON-column serialization versus primary-key or explicit-opt-in semantics, the implementation and total key encoding can be aligned to that contract.