Python(fix): fix get data pagaination and add progress bars#678
Merged
Conversation
Contributor
|
Python docs preview: https://sift-stack.github.io/sift/python/pr-678/ Deployed from |
ian-sift
reviewed
Jul 10, 2026
ian-sift
reviewed
Jul 10, 2026
ian-sift
approved these changes
Jul 10, 2026
alexluck-sift
pushed a commit
that referenced
this pull request
Jul 11, 2026
Bump version to 0.19.0 and update the changelog for the PRs merged since v0.18.0: ABAC attributes (#646), get_data cache improvements (#650), pytest docstring cleaning (#658), ULog data imports (#663), access control API grouping (#666), report templates (#672), download_file extra headers (#677), and the get_data pagination fix plus progress bars (#678). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y5U49rrpvQm5bPbvEjR4xr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Screen.Recording.2026-07-10.at.10.40.23.mov
Summary
Adds progress bars to
get_dataand archive unzip, and fixes a pagination defect inget_datawhere thelimitargument was silently ignored.Progress bars
get_datashows a two-line bar: the top line fills by channel (3/8), and the second line tracks points loaded and throughput live as each page arrives, e.g.1,024,000 pts · 218,400 pts/s · voltage, current (+3 more).On completion the second line becomes a past-tense summary, e.g.
Loaded 716,834 points (avg 211,921/s).Point totals include cache hits, while the rate covers only fetched points, so a warm call reads
Loaded 716,834 points from cachewith no rate and a mixed call readsLoaded N points: F fetched (avg …/s), C cached.Archive extraction shows unzip progress. Both no-op when stdout cannot back a bar (e.g. a detached process or
pythonw), default on for sync callers and off for async, and are overridable per call or globally viasift_client.config.show_progress.get_data pagination fix
While adding the bar I found
get_datapaging one proto at a time and ignoringlimit.limitis documented as a maximum data-point count, but it was passed to the shared paginator as a maximum number of returned protos. Each page returns exactly one proto holding up topage_sizepoints, so the loop counted protos, never bounded points, and paged to the end of the run regardless oflimit. The fix routes the point budget throughpage_sizeand stops afterceil(limit / page_size)pages.limit=1000, beforelimit=1000, afterlimit=100, afterlimit=50000, afterlimitnow bounds points per channel and each channel resolves in a single round-trip. The unbounded export path still pages to exhaustion via the token.Deserialize once
To count points for the progress line without parsing the wire payload twice, each page is deserialized once in the paginator and handed to
_merge_pagesas frames, rather than parsed once to count and again to merge.Tests
67 data tests pass. New: 5 for the limit path (single-page budget, ceil pages, no-limit exhaustion, over-budget termination, zero budget) and 3 for the progress line (live points total, fetched/cached split, all-cached shows no rate). The two core limit tests fail against the old proto-counting logic and pass against the fix. The
_merge_pagestests were updated to pass pre-deserialized frames for the single-parse change.