feat: add iter()/aiter() for lazy filter-based key iteration - #682
Open
Aryan-Pardeshi wants to merge 3 commits into
Open
feat: add iter()/aiter() for lazy filter-based key iteration#682Aryan-Pardeshi wants to merge 3 commits into
Aryan-Pardeshi wants to merge 3 commits into
Conversation
…ted __iter__/__aiter__, restore conftest
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 441a51d. Configure here.
…H+LIMIT FT.SEARCH + LIMIT is capped by MAXSEARCHRESULTS and non-deterministic without a unique sort, which is exactly the large-index case this API targets. The repo already has _iter_keys_by_filter for this reason -- it pages with FT.AGGREGATE ... WITHCURSOR and always releases the cursor. Delegate to it instead of reimplementing offset-based paging. Caught by Cursor Bugbot on review, verified against the existing _iter_keys_by_filter docstring and callers (drop_by_filter, update_by_filter).
Author
|
Good catch — pushed a fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #489
Adds
iter()andaiter()for lazy, filter-based iteration over the keys in an index.Neither existing API covers this:
paginate()yields full document records rather than keys, andscan_by_pattern()works on raw Redis key patterns and materialises a list. For index maintenance over a large index you want keys only, streamed, and selectable by filter expression.Both take an optional
FilterExpression(defaulting to match-all) and abatch_sizedefaulting toDEFAULT_BULK_BATCH_SIZE, page through_querywithreturn_fields=["id"], and yield keys one at a time without ever building the full list. The async version mirrors the sync one exactly.tests/integration/test_index_iteration.pycovers full iteration, filtered iteration, laziness (the first key arrives without draining the index), and abatch_sizesmaller than the document count so the paging loop is actually exercised rather than short-circuiting on a single batch. 7 passed against Redis in Docker.Two things I want to flag rather than have you find:
itershadows the builtin as a method name. That is what the issue asked for, but if you would rather haveiter_keys/aiter_keysI will rename without argument.I did not add
__iter__/__aiter__dunders. Making aSearchIndexdirectly iterable would meanlist(index)silently issues a full paged scan against Redis, which felt like a surprising thing to attach to a plainforloop. Easy to add if you want it.Note
Low Risk
Additive public API on top of an existing bulk helper; no changes to auth, persistence, or bulk mutation semantics beyond new iteration entry points.
Overview
Adds
iter()onSearchIndexandaiter()onAsyncSearchIndexso callers can walk document Redis keys (not full records) with an optionalFilterExpressionand configurablebatch_size, defaulting to match-all when the filter is omitted.Both methods are thin wrappers around the existing
_iter_keys_by_filterpath (FT.AGGREGATE+WITHCURSOR), so iteration avoidsMAXSEARCHRESULTScaps that affectFT.SEARCH+LIMIT, with the same de-duplication and memory characteristics documented on that helper.Integration tests in
test_index_iteration.pycover full and filtered scans, lazy consumption, and paging whenbatch_sizeis smaller than the index size for sync and async.Reviewed by Cursor Bugbot for commit 69dd491. Bugbot is set up for automated code reviews on this repo. Configure here.