Repository navigation
Find moves in make_patch without rewriting all later operations - #208
Merged
Merged
Conversation
make_patch took about 150 s to diff list(range(10000)) against its reverse, and 10 s for a random permutation of 3000 items. Profiling showed the time in DiffBuilder._adjust_following: whenever a 'remove' and a later 'add' (or the other way round) became a 'move', all operations after the earlier one were walked and their array indices rewritten, which is quadratic in the number of operations. Operations now refer to array items by _ArrayItem objects instead of indices. These do not change when a move replaces an earlier operation, so nothing has to be rewritten. Each array keeps all items it has during the diff in one order, by integer labels, and the items it has at the moment; the indices that the moves need are found with bisect. When the diff is done, the operations are replayed once to get their indices. The order puts an inserted item directly after the item before it, so before removed items. This is how _adjust_following placed them, and the patches are the same as before: they were compared with the previous version on about 170000 random and Hypothesis generated pairs of documents. take_index scanned all stored unhashable values (lists and objects) for one equal to the given one. They are now grouped by a hashable key that equal values share, and only values in the same group are compared. The two cases above now take 0.45 s and 0.08 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RE4gSqjC7bhmzDr3325NKC
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The core diff and move-indexing algorithm is substantially rewritten and warrants final human validation despite strong regression coverage.
0 open findings
What changed in this PR
Optimizes make_patch by replacing repeated index rewrites with stable array-item references and replaying operations once.
Changes:
- Tracks array items with ordered labels and resolves indices during replay.
- Groups unhashable values by structural keys for faster move matching.
- Adds correctness and performance regression tests for large and nested moves.
| File | Description |
|---|---|
jsonpatch.py |
Implements stable array locations, operation replay, and grouped unhashable lookup. |
tests.py |
Covers index-sensitive, nested, large-list, and unhashable-value moves. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Conflicts in jsonpatch.py: - store_index/take_index: master's _move_key, which keys moves by the JSON serialization of values, already avoids scanning unhashable values one by one, so it replaces the _hashable grouping of this branch, which is dropped along with index_storage2. - Helpers: master's _positional_cost and _sorted_members are kept next to the _ArrayItem classes of this branch. test_moves_change_indices_in_nested_list gets an input for which the list alignment from master still pairs up the list items by position, so a move in the outer array still shifts operations in the nested one. Its expected patch is the one master gives. The duplicate import of random in tests.py is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RE4gSqjC7bhmzDr3325NKC
Conflict in tests.py: both sides added tests to OptimizationTests at the same place; all of them are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RE4gSqjC7bhmzDr3325NKC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
make_patch took about 150 s to diff list(range(10000)) against its
reverse, and 10 s for a random permutation of 3000 items. Profiling
showed the time in DiffBuilder._adjust_following: whenever a 'remove'
and a later 'add' (or the other way round) became a 'move', all
operations after the earlier one were walked and their array indices
rewritten, which is quadratic in the number of operations.
Operations now refer to array items by _ArrayItem objects instead of
indices. These do not change when a move replaces an earlier operation,
so nothing has to be rewritten. Each array keeps all items it has during
the diff in one order, by integer labels, and the items it has at the
moment; the indices that the moves need are found with bisect. When the
diff is done, the operations are replayed once to get their indices.
The order puts an inserted item directly after the item before it, so
before removed items. This is how _adjust_following placed them, and
the patches are the same as before: they were compared with the
previous version on about 170000 random and Hypothesis generated pairs
of documents.
take_index scanned all stored unhashable values (lists and objects) for
one equal to the given one. They are now grouped by a hashable key that
equal values share, and only values in the same group are compared.
The two cases above now take 0.45 s and 0.08 s.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01RE4gSqjC7bhmzDr3325NKC