Skip to content

Python(fix): fix get data pagaination and add progress bars#678

Merged
alexluck-sift merged 7 commits into
mainfrom
al/python/feat/get-data-progress
Jul 10, 2026
Merged

Python(fix): fix get data pagaination and add progress bars#678
alexluck-sift merged 7 commits into
mainfrom
al/python/feat/get-data-progress

Conversation

@alexluck-sift

@alexluck-sift alexluck-sift commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator
Screen.Recording.2026-07-10.at.10.40.23.mov

Summary

Adds progress bars to get_data and archive unzip, and fixes a pagination defect in get_data where the limit argument was silently ignored.

Progress bars

get_data shows a two-line bar: the top line fills by channel (3/8), and the second line tracks points loaded and throughput live as each page arrives, e.g. 1,024,000 pts · 218,400 pts/s · voltage, current (+3 more).
On completion the second line becomes a past-tense summary, e.g. Loaded 716,834 points (avg 211,921/s).
Point totals include cache hits, while the rate covers only fetched points, so a warm call reads Loaded 716,834 points from cache with no rate and a mixed call reads Loaded N points: F fetched (avg …/s), C cached.
Archive extraction shows unzip progress. Both no-op when stdout cannot back a bar (e.g. a detached process or pythonw), default on for sync callers and off for async, and are overridable per call or globally via sift_client.config.show_progress.

get_data pagination fix

While adding the bar I found get_data paging one proto at a time and ignoring limit. limit is documented as a maximum data-point count, but it was passed to the shared paginator as a maximum number of returned protos. Each page returns exactly one proto holding up to page_size points, so the loop counted protos, never bounded points, and paged to the end of the run regardless of limit. The fix routes the point budget through page_size and stops after ceil(limit / page_size) pages.

Request Wire calls Max points/channel Wall time
limit=1000, before ~800 1,000,000 (limit ignored) tens of seconds
limit=1000, after 8 1,000 0.40s
limit=100, after 8 100 0.43s
limit=50000, after 8 50,000 1.98s
no limit (full export) 1,108 1,000,000 ~145s, unchanged

limit now bounds points per channel and each channel resolves in a single round-trip. The unbounded export path still pages to exhaustion via the token.

Deserialize once

To count points for the progress line without parsing the wire payload twice, each page is deserialized once in the paginator and handed to _merge_pages as frames, rather than parsed once to count and again to merge.

Tests

67 data tests pass. New: 5 for the limit path (single-page budget, ceil pages, no-limit exhaustion, over-budget termination, zero budget) and 3 for the progress line (live points total, fetched/cached split, all-cached shows no rate). The two core limit tests fail against the old proto-counting logic and pass against the fix. The _merge_pages tests were updated to pass pre-deserialized frames for the single-parse change.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Python docs preview: https://sift-stack.github.io/sift/python/pr-678/

Deployed from fd91fd2. The link may take up to a minute to become live as GitHub Pages propagates.

Comment thread python/lib/sift_client/_internal/low_level_wrappers/data.py Outdated
Comment thread python/lib/sift_client/_internal/low_level_wrappers/data.py Outdated
@alexluck-sift
alexluck-sift requested a review from ian-sift July 10, 2026 18:37
@alexluck-sift alexluck-sift changed the title Python(feat): add progress bars to get_data and unzip Python(fix): fix get data pagaination and add progress bars Jul 10, 2026
@alexluck-sift
alexluck-sift merged commit a963444 into main Jul 10, 2026
26 checks passed
@alexluck-sift
alexluck-sift deleted the al/python/feat/get-data-progress branch July 10, 2026 20:26
alexluck-sift pushed a commit that referenced this pull request Jul 11, 2026
Bump version to 0.19.0 and update the changelog for the PRs merged
since v0.18.0: ABAC attributes (#646), get_data cache improvements
(#650), pytest docstring cleaning (#658), ULog data imports (#663),
access control API grouping (#666), report templates (#672),
download_file extra headers (#677), and the get_data pagination fix
plus progress bars (#678).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y5U49rrpvQm5bPbvEjR4xr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants