Skip to content

fix(db): preserve optimistic collection order - #1839

Closed
KyleAMathews wants to merge 1 commit into
codex/wave1-core-reconciliationfrom
codex/wave2-collection-publication-1792
Closed

KyleAMathews wants to merge 1 commit into
codex/wave1-core-reconciliationfrom
codex/wave2-collection-publication-1792

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Important

This is stacked on #1835. Its base must remain codex/wave1-core-reconciliation at 7a21b713eaa17fc555db98fe20dcccf27bb520ee until that PR lands; do not merge this PR first.

Preserves a Collection's configured order while optimistic inserts or updates are visible. Every existing Collection iteration surface now observes the same correctly ordered optimistic overlay, with no new API or scheduling behavior.

Root cause

CollectionStateManager.keys() yielded all authoritative keys first and appended optimistic-only keys afterward. Because values(), entries(), the iterator, state, toArray, and currentStateAsChanges() derive their order from keys(), a pending row could appear out of the Collection's declared comparator order. Optimistic updates also shadowed their authoritative row without moving to the position of their visible optimistic value.

Approach

  • Keep the existing no-comparator behavior and zero-upsert fast path.
  • Sort only the visible optimistic upserts with the configured comparator.
  • Preserve the comparator's direct nonzero result; use the established compareKeys ordering only for 0, -0, or NaN ties.
  • Lazily merge those optimistic entries with the already sorted authoritative entries, suppressing optimistic deletes and shadowed authoritative keys.

This is O(S + U log U) time with O(U) temporary space for S synced rows and U optimistic upserts.

Key invariants

  • Pending inserts and moved updates occupy the position of their visible values on every Collection iteration surface.
  • Deleted rows stay hidden and a shadowed key appears exactly once.
  • Equal comparator positions retain the existing deterministic key tie-break.
  • Collections without a comparator retain append-after-authoritative optimistic insertion order.
  • Reentrant sync/truncate publication advances layout state only when the visible ordered overlay actually moves.

Non-goals

Trade-offs and shipped weight

A full visible-state sort would be shorter but would sort all S + U rows and allocate the complete visible state on every ordered read. The local two-way merge retains the authoritative structure's existing order and limits sorting/allocation to pending upserts. Throwing was rejected because #1792 is an existing-contract defect with a deterministic repair.

Exact comparison against the stacked base 7a21b713:

Artifact Baseline Candidate Delta
Normal ESM JavaScript 781,329 B 782,810 B +1,481 B
Normal CJS JavaScript 804,286 B 805,768 B +1,482 B
Minified ESM JavaScript 559,390 B 560,087 B +697 B
Minified CJS JavaScript 372,884 B 373,369 B +485 B
Minified ESM gzip (-9 -n) 165,581 B 165,830 B +249 B
Minified CJS gzip (-9 -n) 127,524 B 127,737 B +213 B
npm tarball 1,564,217 B 1,566,515 B +2,298 B
npm unpacked package 6,766,354 B 6,774,150 B +7,796 B

The oracle, documentation, and test-only cleanup add exactly 0 B of production JavaScript. Registering the oracle adds 46 raw bytes to the shipped package.json; that manifest change is included in the package totals above.

Verification

On exact base 7a21b713, the permanent same-path oracle deterministically produced expected [a,b,c], actual [a,c,b], first divergence index 1, while its path and declared-order checker controls stayed green.

On final head 82d443c3:

  • Visible-order oracle: 13/13.
  • Focused Collection, optimistic-history, reconciliation, comparator-law, deterministic-order, and live-query collateral: 303/303.
  • test:oracles registration: the selected campaign executes the new oracle successfully.
  • Hostile controls kill append-after-authoritative, stale optimistic-update position, missing or wrong-sign key ties, key-order override of -7/+11, optimistic-delete placeholder, full visible-state sort, missing optimistic-side sort, quadratic optimistic comparisons, and duplicate merged keys.
  • Focused ESLint, Prettier, TypeScript, normal build, minified build, and diff checks: clean.
pnpm --filter @tanstack/db exec vitest run \
  tests/collection-visible-order-oracle.test.ts \
  tests/collection-sync-reentrancy.test.ts \
  tests/comparison.test.ts \
  tests/comparison.property.test.ts

pnpm --filter @tanstack/db exec tsc -p tsconfig.json --noEmit
pnpm --filter @tanstack/db build
pnpm --filter @tanstack/db build:minified

The two ignored React Native examples still emit the baseline-only missing expo/tsconfig.base warning; tests and builds complete successfully.

Files changed

  • packages/db/src/collection/state.ts: ordered optimistic-overlay merge.
  • packages/db/tests/collection-visible-order-oracle.test.ts: public-surface law, tie cases, complexity checks, and hostile-control witnesses.
  • packages/db/tests/collection-sync-reentrancy.test.ts: existing reentrant publication expectation aligned with visible order.
  • packages/db/package.json: register the dedicated oracle in test:oracles.
  • docs/contributing/oracle-coverage.md: document the declared-order law and explicit limits.
  • .changeset/fix-optimistic-collection-order.md: patch release note.

Provenance and credit

No material code was copied from an unmerged proposal, so no additional coauthor trailer is warranted.


Fixes #1792

Summary by CodeRabbit

  • Bug Fixes

    • Optimistic inserts and updates now preserve a collection’s configured sort order.
    • Collection views remain consistent while pending changes are visible, including iteration and array-based results.
    • Visible ordering is restored correctly when optimistic changes are rolled back or rejected.
  • Tests

    • Added coverage for ordering across optimistic inserts, updates, deletes, concurrent changes, and rollback scenarios.

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 5c3d3344-1bcd-4e93-b8c5-f9d4a46ca622

📥 Commits

Reviewing files that changed from the base of the PR and between 7a21b71 and 82d443c.

📒 Files selected for processing (6)
  • .changeset/fix-optimistic-collection-order.md
  • docs/contributing/oracle-coverage.md
  • packages/db/package.json
  • packages/db/src/collection/state.ts
  • packages/db/tests/collection-sync-reentrancy.test.ts
  • packages/db/tests/collection-visible-order-oracle.test.ts

Included review availability: Your plan provides up to 8 included reviews per hour; 2 remain after this review.


📝 Walkthrough

Walkthrough

The collection now merges optimistic rows with synced rows using the configured comparator. Key comparison resolves comparator ties. Tests cover visible iteration surfaces, rollback, rejection, confirmation, and unchanged public layouts.

Changes

Optimistic collection ordering

Layer / File(s) Summary
Ordered optimistic overlay
packages/db/src/collection/state.ts
keys() merges synced and optimistic keys in comparator order. It skips optimistic deletes and shadowed synced rows. compareKeys resolves ties.
Visible-order validation
packages/db/tests/collection-visible-order-oracle.test.ts, packages/db/tests/collection-sync-reentrancy.test.ts
Tests cover all collection iteration surfaces, optimistic inserts, updates, deletes, rollback, rejection, confirmation, comparator ties, comparison bounds, and preserved public layouts.
Release and oracle wiring
.changeset/fix-optimistic-collection-order.md, packages/db/package.json, docs/contributing/oracle-coverage.md
The patch changeset records the fix. The oracle test runs through test:oracles and appears in coverage documentation.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant Transaction
  participant CollectionStateManager
  participant Collection
  participant Persistence
  Transaction->>CollectionStateManager: apply optimistic mutation
  CollectionStateManager->>Collection: expose comparator-ordered rows
  Collection->>Transaction: return ordered iteration surfaces
  Transaction->>Persistence: await write result
  Persistence->>CollectionStateManager: confirm, rollback, or reject
  CollectionStateManager->>Collection: update the visible ordered view
Loading

Merge Risk: ⚪ Minimal · up to 82d44

The optimistic overlay preserves configured collection ordering across authoritative and pending rows. No concrete merge-blocking issue remains.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: preserving Collection order during optimistic updates.
Description check ✅ Passed The description is detailed and relevant. It explains the change, motivation, approach, scope, testing, release impact, and changed files. It does not use the exact template headings or mark the check…
Linked Issues check ✅ Passed Issue #1792 requires comparator-configured Collections to place pending optimistic rows at their declared sorted positions. The PR updates CollectionStateManager.keys() to merge authoritative and op…
Out of Scope Changes check ✅ Passed The changes stay connected to issue #1792. The state change implements the Collection-level ordering behavior. The oracle test, test registration, coverage documentation, changeset, and updated reentr…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 3…
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/wave2-collection-publication-1792

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@KyleAMathews

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@pkg-pr-new

pkg-pr-new Bot commented Sep 16, 2026

Copy link
Copy Markdown
More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@1839

@tanstack/browser-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1839

@tanstack/capacitor-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1839

@tanstack/cloudflare-durable-objects-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1839

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@1839

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@1839

@tanstack/db-sqlite-persistence-core

npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1839

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@1839

@tanstack/electron-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1839

@tanstack/expo-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1839

@tanstack/node-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1839

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@1839

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1839

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@1839

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@1839

@tanstack/react-native-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1839

@tanstack/react-router-with-db

npm i https://pkg.pr.new/@tanstack/react-router-with-db@1839

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1839

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@1839

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@1839

@tanstack/tauri-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1839

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1839

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@1839

commit: 82d443c

@github-actions

Copy link
Copy Markdown
Contributor

Size Change: +251 B (+0.15%)

Total Size: 165 kB

📦 View Changed
Filename Size Change
packages/db/dist/esm/collection/state.js 6.78 kB +251 B (+3.85%)
ℹ️ View Unchanged
Filename Size
packages/db/dist/esm/client.js 3.66 kB
packages/db/dist/esm/collection-options.js 236 B
packages/db/dist/esm/collection/change-events.js 1.44 kB
packages/db/dist/esm/collection/changes.js 2.24 kB
packages/db/dist/esm/collection/cleanup-queue.js 794 B
packages/db/dist/esm/collection/events.js 481 B
packages/db/dist/esm/collection/index.js 4.58 kB
packages/db/dist/esm/collection/indexes.js 1.99 kB
packages/db/dist/esm/collection/lifecycle.js 2.15 kB
packages/db/dist/esm/collection/mutations.js 2.53 kB
packages/db/dist/esm/collection/subscription.js 8.72 kB
packages/db/dist/esm/collection/sync.js 4.62 kB
packages/db/dist/esm/collection/transaction-metadata.js 144 B
packages/db/dist/esm/deferred.js 207 B
packages/db/dist/esm/errors.js 5.26 kB
packages/db/dist/esm/event-emitter.js 964 B
packages/db/dist/esm/index.js 3.68 kB
packages/db/dist/esm/indexes/auto-index.js 829 B
packages/db/dist/esm/indexes/base-index.js 1.14 kB
packages/db/dist/esm/indexes/basic-index.js 2.07 kB
packages/db/dist/esm/indexes/btree-index.js 2.26 kB
packages/db/dist/esm/indexes/index-registry.js 820 B
packages/db/dist/esm/indexes/reverse-index.js 376 B
packages/db/dist/esm/live-query-adapter.js 318 B
packages/db/dist/esm/live-query-observer.js 3.69 kB
packages/db/dist/esm/live-query-options.js 702 B
packages/db/dist/esm/live-query-window-controller.js 4.36 kB
packages/db/dist/esm/local-only.js 975 B
packages/db/dist/esm/local-storage.js 2.15 kB
packages/db/dist/esm/optimistic-action.js 359 B
packages/db/dist/esm/paced-mutations.js 496 B
packages/db/dist/esm/proxy.js 3.32 kB
packages/db/dist/esm/query/builder/functions.js 1.47 kB
packages/db/dist/esm/query/builder/index.js 6.69 kB
packages/db/dist/esm/query/builder/query-ir.js 116 B
packages/db/dist/esm/query/builder/ref-proxy.js 1.24 kB
packages/db/dist/esm/query/compiler/evaluators.js 1.92 kB
packages/db/dist/esm/query/compiler/expressions.js 560 B
packages/db/dist/esm/query/compiler/group-by.js 4.13 kB
packages/db/dist/esm/query/compiler/index.js 9.06 kB
packages/db/dist/esm/query/compiler/joins.js 3 kB
packages/db/dist/esm/query/compiler/lazy-targets.js 1.1 kB
packages/db/dist/esm/query/compiler/order-by.js 1.91 kB
packages/db/dist/esm/query/compiler/parent-routes.js 319 B
packages/db/dist/esm/query/compiler/route-metadata.js 1.24 kB
packages/db/dist/esm/query/compiler/select.js 1.58 kB
packages/db/dist/esm/query/effect.js 4.6 kB
packages/db/dist/esm/query/equality-value-identity.js 591 B
packages/db/dist/esm/query/expression-helpers.js 1.43 kB
packages/db/dist/esm/query/ir-stable-identity.js 4.04 kB
packages/db/dist/esm/query/ir.js 1.59 kB
packages/db/dist/esm/query/live-query-collection.js 391 B
packages/db/dist/esm/query/live/bucket-facade-adapter.js 2.73 kB
packages/db/dist/esm/query/live/collection-config-builder.js 6.97 kB
packages/db/dist/esm/query/live/collection-registry.js 264 B
packages/db/dist/esm/query/live/collection-subscriber.js 2.25 kB
packages/db/dist/esm/query/live/internal.js 145 B
packages/db/dist/esm/query/live/materialized-pipeline.js 2.32 kB
packages/db/dist/esm/query/live/ordered-source-loader.js 3.14 kB
packages/db/dist/esm/query/live/subset-demand-controller.js 1.26 kB
packages/db/dist/esm/query/live/utils.js 1.14 kB
packages/db/dist/esm/query/optimizer.js 2.91 kB
packages/db/dist/esm/query/query-once.js 359 B
packages/db/dist/esm/query/runtime-reference-identity.js 572 B
packages/db/dist/esm/query/subset-dedupe.js 486 B
packages/db/dist/esm/scheduler.js 1.34 kB
packages/db/dist/esm/SortedMap.js 1.3 kB
packages/db/dist/esm/strategies/debounceStrategy.js 247 B
packages/db/dist/esm/strategies/queueStrategy.js 428 B
packages/db/dist/esm/strategies/throttleStrategy.js 246 B
packages/db/dist/esm/transactions.js 3.72 kB
packages/db/dist/esm/utils.js 1.01 kB
packages/db/dist/esm/utils/array-utils.js 270 B
packages/db/dist/esm/utils/browser-polyfills.js 304 B
packages/db/dist/esm/utils/btree.js 4.51 kB
packages/db/dist/esm/utils/callbacks.js 174 B
packages/db/dist/esm/utils/comparison.js 1.49 kB
packages/db/dist/esm/utils/cursor.js 676 B
packages/db/dist/esm/utils/error.js 167 B
packages/db/dist/esm/utils/get-or-create.js 155 B
packages/db/dist/esm/utils/index-optimization.js 2.42 kB
packages/db/dist/esm/utils/type-guards.js 230 B
packages/db/dist/esm/utils/uuid.js 449 B
packages/db/dist/esm/virtual-props.js 360 B

compressed-size-action::db-package-size

@github-actions

Copy link
Copy Markdown
Contributor

Size Change: 0 B

Total Size: 7.34 kB

ℹ️ View Unchanged
Filename Size
packages/react-db/dist/esm/DbProvider.js 317 B
packages/react-db/dist/esm/HydrationBoundary.js 263 B
packages/react-db/dist/esm/index.js 330 B
packages/react-db/dist/esm/live-query-internals.js 282 B
packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.9 kB
packages/react-db/dist/esm/useLiveQuery.js 2.68 kB
packages/react-db/dist/esm/useLiveQueryEffect.js 355 B
packages/react-db/dist/esm/useLiveSuspenseQuery.js 812 B
packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

@KyleAMathews

Copy link
Copy Markdown
Collaborator Author

Closing after scope/value review. This changes optimistic ordering for direct iteration of sorted Collections, but it does not affect normal live-query ordering or the offline-transactions executor. The use case is too narrow to justify adding this behavior and maintenance surface now. The branch is retained if concrete demand appears later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant