Skip to content

Fix offset auto-pagination skipping every other page#127

Open
IlyaasK wants to merge 2 commits into
mainfrom
fix-offset-pagination-page-skip
Open

Fix offset auto-pagination skipping every other page#127
IlyaasK wants to merge 2 commits into
mainfrom
fix-offset-pagination-page-skip

Conversation

@IlyaasK

@IlyaasK IlyaasK commented Jun 12, 2026

Copy link
Copy Markdown

Bug

OffsetPagination.nextPageRequestOptions() computed the next request offset as next_offset + items.length. But X-Next-Offset already holds the offset where the next page starts (the API sets it to offset + limit), so every iteration skipped a full page: with 250 items at limit 100, for await yields items 0–99 and 200–249 — 100–199 silently vanish. X-Has-More still terminates the loop, so nothing ever errored.

Same bug class as kernel/kernel#2401 investigated; the Python SDK has the identical flaw (PR incoming), while the Go SDK uses the header directly and was always correct.

Fix

Use X-Next-Offset verbatim as the next request's offset, and stop paginating when the header is absent. Termination via X-Has-More: false and empty pages is unchanged.

Why a hand-written commit to generated code

Stainless's hosted generator is winding down post-acquisition, so a template-level fix is not coming. While the service still runs, this commit rides Stainless's custom-code three-way merge; after sunset it is simply the code. tests/pagination.test.ts pins the arithmetic and all three termination conditions.

Note for kernel/kernel#2401

That PR's stainless.yaml remap (X-Offset count-start semantics) must not merge: it fixes nothing that this commit doesn't, and it breaks the Go SDK (single-page truncation via its next != 0 sentinel).

🤖 Generated with Claude Code


Note

Medium Risk
Changes core list pagination used across the SDK; incorrect offsets previously caused silent data loss, and the new error when headers disagree may surface misconfigured API responses.

Overview
Fixes offset auto-pagination so the next request uses X-Next-Offset as-is instead of next_offset + current page length, which was advancing by two pages per step and dropping every other page during for await over list endpoints.

When X-Next-Offset is missing, pagination now stops (or returns null from nextPageRequestOptions); if X-Has-More: true without that header, hasNextPage() throws KernelError so results are not silently truncated. Existing stop conditions (X-Has-More: false, empty pages) are unchanged.

Adds tests/pagination.test.ts to lock in the corrected offset, termination behavior, and the error path.

Reviewed by Cursor Bugbot for commit 994749d. Bugbot is set up for automated code reviews on this repo. Configure here.

OffsetPagination.nextPageRequestOptions() requested the next page at
next_offset + items.length. The X-Next-Offset header already holds the
offset where the next page starts (the API computes offset + limit), so
adding the current page length skipped a full page per iteration: with
250 items at limit 100, iteration returned items 0-99 and 200-249 and
silently dropped 100-199. X-Has-More still terminated the loop, hiding
the data loss.

Use the header value directly, and stop paginating when it is absent.

Hand-maintained fix: with Stainless's hosted generator winding down, the
template-level fix is not coming; this commit and its test are the
canonical behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@firetiger-agent

Copy link
Copy Markdown

Created a monitoring plan for this PR.

What this PR does: Fixes a silent data-loss bug where iterating any paginated list endpoint (proxies, browsers, connections, profiles, etc.) with the Node.js SDK skipped every other page — so a 250-item result set yielded only 200 items with no error raised.

Intended effect:

  • GET list endpoint completeness: no direct telemetry signal per-request, but confirmed working if GET /proxies?offset=100 etc. traffic appears after SDK upgrade; pre-deploy baseline for GET /proxies is ~235 req/hr, ~120/hr for GET /auth/connections
  • Zero new 5xx errors on list endpoints: baseline is 0 5xx on all GET list endpoints in the last 24h; confirmed if this remains 0 post-release

Risks:

  • Rate limit surge — users iterating large lists for the first time may hit per-org caps; alert if 429 responses appear on GET /proxies, GET /auth/connections, or any other list endpoint (zero pre-deploy baseline)
  • GET list traffic spike — expected moderate increase, but alert if any list endpoint exceeds 5× its hourly baseline sustained for 2+ hours (e.g. /proxies >1,175/hr)
  • Runaway pagination loop — a regression in termination logic could cause infinite GET requests with monotonically increasing offsets; alert if any single org generates sustained same-path GET traffic with incrementing offset params

Status updates will be posted automatically on this PR as monitoring progresses.

View monitor

…Next-Offset

Review follow-up: stopping silently on an absent header is correct for
real last pages, but if a server ever reports more results without
saying where they start, truncating silently reproduces the data-loss
failure mode this branch exists to fix. Throw instead.

Also pins the 0 next-offset sentinel the API emits on last pages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant