fix(skills): apply changes after a none intent, and retry recoverable failures indefinitely - #131
Conversation
… failures indefinitely - A `none` intent now leaves the FDv2 reader expecting changes, as the base SDK's `ChangeSetBuilder.expect_changes()` does. Previously a put-object or delete-object following `none` on the same stream was dropped while the following payload-transferred still advanced the basis, so a skill revoked after a routine reconnect kept being served. - Remove `max_consecutive_failures`. Recoverable failures are retried on the capped backoff for as long as the store runs; only a fatal status stops delivery. Clamp the backoff exponent so a long outage cannot overflow it. - README: scope the "revoked SKILL.md leaves disk" claim to "*", and say what an explicit skill list does instead. Correct the _resolve_requests docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jeffdupont
left a comment
There was a problem hiding this comment.
Follow-up to my GA review of #87. I reviewed f6bb491 against its base, xie/agent-skills at c547be2, not main. make test: 2166 passed, 11 skipped, exit 0. make typecheck is clean. I also reverted each fix on its own and re-ran the new tests. Without the none change, all three none tests fail. Without the exponent clamp, the backoff test fails with OverflowError. With the base branch's skills_fdv2.py put back, the no-option test and both retry-indefinitely tests fail. So the new tests do catch the bugs they're meant to.
Where the four #87 blockers stand:
- (1) Changes after
none: resolved. This now matches the LaunchDarkly Python SDK'sexpect_changes(). - (3) Giving up after 10 failures: resolved. Removing
max_consecutive_failuresis not a breaking change. It never reachedmain(git greponorigin/mainfinds neither it norFDv2SkillStore), sofix(skills)without a!is right. - (2) Skill-key grammar: not touched. Still open. Per your reply on #87, it needs an SDK-or-API decision, and it stays on the GA list until that's made.
- (4) Revocation for explicit lists: docs only. I think that's acceptable for 1.0. What 1.0 freezes is the conservative behaviour: absent keys keep their files and report an
error. The likely fixes are additive and can ship in a minor: letwatch_skillstake a callable that returns the current refs, or add an opt-in prune. The one change that would be hard to make after 1.0 is making explicit lists prune absent keys by default, because 1.x would then delete files that 1.0 kept. If we never want that default, docs-only is fine. A few unscoped claims are left; details inline.
One thing to settle before merge:
- The backoff options aren't validated, and without the budget, a zero or negative value now spins forever. With a fake requester that always fails:
initial_backoff=0made 773,754 attempts in one second;max_backoff=0andmax_backoff=-5made about 775k each.failedstayedNone. On the base branch, the same probe stops after 11 attempts withfailedset. Against the real service, that's a fleet hammering LaunchDarkly over a config typo. JS #108 does the same (about 860 attempts/s). I'd validate both as positive and finite, aspoll_intervalalready is, and add that to the spec. Inline.
Smaller notes:
- Fleet behaviour in an outage is fine for a hard outage. The cap is 30s with subtractive 50% jitter, the same as the LaunchDarkly Python SDK's FDv2 streaming (
MAX_RETRY_DELAY = 30,JITTER_RATIO = 0.5,ldclient/impl/datasourcev2/streaming.py:55-57). I found two gaps, both older than this PR, but they now last for the whole outage instead of 10 attempts. First,Retry-Afteris used without jitter (skills_fdv2.py:1651-1655), so a fleet toldRetry-After: 30reconnects at the same moment. Second, backoff resets on everynone, while the base SDK resets only after a connection has stayed open 60s (BACKOFF_RESET_INTERVAL = 60,streaming.py:56,118). A server that answersnoneand then drops each connection gets reconnects at about 1/s indefinitely. My probe with default backoff saw 27 connections in 20s. Both are spec questions. - The clamp is correct at the boundaries. Attempts up to 1 give the base delay, and attempt 63 and above hit the clamp.
float(2**62)is exact, and attempt 10,000 gives exactly 30.0. - An oversized response is now retried forever. A body over
MAX_RESPONSE_BYTES(64 MiB) is recoverable, so the store downloads up to 64 MiB again every 15-30s and never setsfailed. The rewritten test attest_skills_fdv2.py:2777now asserts exactly that. Before, the budget turned this intofailed. It's unlikely, but it fails the same way every time, so it could be fatal. I haven't checked how large a real payload can get. - Outage visibility.
failedno longer signals an outage; onlyconnection_failuresandlast_errordo. A last-success timestamp onStoreDiagnosticswould help health checks. That's additive, so it can come after 1.0. - A stale comment at
skills_fdv2.py:1711-1712still mentions "exhausting its budget". #108 updated the same comment. - Test parity with #108. JS has tests that Python doesn't, and spec #36 asks for two of them:
waitForSkillsrunning to its timeout during an outage, and fail/succeed/fail retrying at the first backoff step. JS also replaced the two 400-budget tests with "a 400 after a recoverable failure still repairs" and "the second 400 still stops"; this PR deletes them with no replacement. connection_failuresmeans different things in the two languages. Python counts agoodbyeafter a complete answer as a failure: it setslast_error, andtest_skills_fdv2.py:2406-2408allows the count to read 1. JS exempts it (reachedServer, jsskills-fdv2.ts:1810). This predates the PR, but the counter is now the main outage signal, so the two should agree and the spec should say which way.- Spec: launchdarkly/ai-sdks-monorepo#36 matches both implementations on the two behaviour changes, the absence-by-name assertion and the finite-backoff assertion. I'd add four points there: the backoff options must be positive and finite; whether a
goodbyeafter an answer counts towardconnection_failures; when backoff resets; and whetherRetry-Aftergets jitter.
JS counterpart: launchdarkly/js-ai-sdk#108 (same findings, noted there).
Review follow-ups on the unbounded-retry change: - Validate initial_backoff and max_backoff: positive, finite, and initial <= max. Without a failure bound they are the only limit on the retry loop, and zero reconnected ~775k times a second. - Reset the backoff delay only after a stream stays open 60s, as the base SDKs do, or after a completed poll. connection_failures still resets on a commit or a none intent. A server that answers and drops is now backed off instead of reconnected about once a second. - A response over MAX_RESPONSE_BYTES is fatal, like 422, instead of being re-downloaded on every backoff step forever. - A goodbye after a completed exchange is not counted or reported as a failure, matching the JS SDK; the reader logs goodbyes at debug. - Tests matching the JS suite: wait_for_skills runs to its timeout during an outage, and the two 400-repair cases. - Docs: scope the skills_watch docstring and agents.md §6 to "*", and fix a stale "budget" comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. The items in the review body:
|
… failures indefinitely (#108) JS counterpart of launchdarkly/python-ai-sdk#131, from review comments on launchdarkly/python-ai-sdk#87 that also apply to #71. Targets `xie/agent-skills-feature-ac9ac7`. ## Changes - **Apply objects that follow a `none` intent** ([r4167784120](launchdarkly/python-ai-sdk#87 (comment))). `serverIntent()` now sets the intent to `xfer-changes` on `none`, as js-core's `protocolHandler.ts` does. Before, a `put-object` / `delete-object` after `none` went to `ignoreUnderUnknownIntent()` and was dropped while the next `payload-transferred` still advanced the basis, so a skill revoked after a routine reconnect kept being served. - **Remove `maxConsecutiveFailures`; retry recoverable failures indefinitely** ([r4167784131](launchdarkly/python-ai-sdk#87 (comment))). Only fatal statuses stop delivery and set `failed`. The 400-retried-once repair is unchanged. - **Clamp the backoff exponent.** `2 ** n` reaches `Infinity` at large attempt numbers, and with a zero `initialBackoffMs` that gave `0 * Infinity = NaN`. - **Scope the on-disk revocation claims to `'*'`** ([r4167784141](launchdarkly/python-ai-sdk#87 (comment)), docs only), in the README, `agents.md` and the `skills-watch.ts` header. Also fixes the `resolveRequests` doc and `agents.md` §4b, which said an unresolved reference makes a run incomplete. ## Tests - New: put and delete after `none` (reader level), and a listener getting the tombstone on a stream. All three fail with the fix reverted. - Budget tests rewritten: retried well past 10 failures (poll and stream) with last known good still served; `waitForSkills` runs to its timeout during an outage; fails/succeeds/fails reports 1; an intent-then-drop repeated 5 times reads `connectionFailures === 5`; backoff at attempt 10,000 is finite; `maxConsecutiveFailures` is absent (source check plus runtime). - `yarn test` (1177 passed, 10 skipped), `yarn typecheck`, `yarn code:check` all pass. Spec: launchdarkly/ai-sdks-monorepo#36 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Aligns **FDv2 skill delivery** with the Python SDK: fixes post-`none` updates, changes retry/fatal semantics, and tightens docs around disk revocation. > > **Protocol:** After a `none` server intent, the reader now treats the connection as **`xfer-changes`** so later `put-object` / `delete-object` events apply instead of being ignored while the basis still advances—fixing revocations (and puts) after a routine reconnect. > > **Retry policy:** **`maxConsecutiveFailures` is removed**; recoverable errors retry for the store’s lifetime and only **fatal** conditions (401/403/422, oversize bodies/events past **64 Mi**, etc.) set `failed` and stop delivery. **`connectionFailures`** remains a diagnostic counter; **backoff** uses a separate **`backoffAttempt`** that grows on every reconnect and resets after a stream stays open **60s** or a poll completes. Constructor now validates **`initialBackoffMs` / `maxBackoffMs`** (positive, finite, ordered). > > **Docs / reconcile semantics:** README, `agents.md`, and `skills-watch` clarify that **automatic on-disk revocation via `watchSkills` applies to `'*'`**, not an explicit skill list; `skills-fs` docs distinguish **incomplete retrieval** (no store, uninitialized, timeout) from **`absent`** references. > > **Tests:** Extensive `skills-fdv2` updates for indefinite retry, backoff behavior, fatal oversize handling, and §3.25 protocol cases. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1c8b75c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
Resolves conflicts with #129 (a 304 confirms a payload rather than establishing one): - FDv2SkillStore._apply keeps this branch's recycled= flag on the recoverable error and takes the base's new boolean return, which _poll_once uses to decide when to adopt an etag. - The over-cap poll test keeps this branch's version: an oversized response is now fatal, so the base's retry-then-recover variant no longer applies. The recovery path still queues a full payload before start(), so it does not rely on a 304 to release the waiter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses three review comments on #87. Targets
xie/agent-skills.Changes
noneintent (r4167784120). Afternone, the reader now expects changes, as the base SDK'sChangeSetBuilder.expect_changes()does. Before, aput-object/delete-objectafternoneon the same stream was dropped while the followingpayload-transferredstill advanced the basis. A skill revoked after a routine reconnect kept being served, and reconnecting didn't fix it.max_consecutive_failures; retry recoverable failures indefinitely (r4167784131). Only fatal statuses (401/403/404/422 etc., a second 400) stop delivery and setfailed. The 400-retried-once repair is unchanged.float(2 ** n)in_backoff_delayraisesOverflowErrorpast ~1024 consecutive failures. The old budget kept the count from getting there; with unlimited retries a long outage (~8.5 h at the default cap) would have crashed the delivery thread."*"(r4167784141, docs only). With an explicit list, anabsentskill stays requested with anerroraction and isn't pruned, and the watcher doesn't see config changes. Also fixes the_resolve_requestsdocstring, which saidabsentmakes a run incomplete.Tests
none, delete afternone(reader level), and a revocation after a reconnect answerednone(end to end on the stream). All three fail with the fix reverted.max_consecutive_failuresis absent by name; backoff at attempt 10,000 is finite. Tests that used a small budget only to stop the store now use a fatal status. Two tests about the budget's 400 exemption were deleted.make test(2166 passed, 11 skipped),make lint,make format-check,make typecheckall pass. Ran the affected tests 10x with no flakes.The same changes for JS: launchdarkly/js-ai-sdk#108.
Spec: launchdarkly/ai-sdks-monorepo#36
🤖 Generated with Claude Code
Note
Overview
FDv2SkillStoredelivery behavior is tightened so revocations and outages match the base SDKs and review feedback on #87.After a
server-intentwithnone, the protocol reader now expects laterput-object/delete-objectevents on the same stream (aligned withChangeSetBuilder.expect_changes()). Previously those edits could be ignored whilepayload-transferredstill advanced the basis, so a skill revoked right after a routine reconnect could keep being served.Recoverable transport failures retry for the life of the store —
max_consecutive_failuresis removed from the public API. Only fatal outcomes (401/403/404/422, a second 400, oversize bodies/events via new_ResponseTooLargeError, etc.) setfailedand stop delivery;start()clears terminal state and resets backoff. Backoff uses a separate attempt counter (reset after a completed poll or a stream held ≥60s), routine post-answergoodbyedisconnects are markedrecycledand not counted as failures, and the backoff exponent is clamped so unbounded retries cannot overflow.Docs and reconcile semantics clarify that disk revocation via
watch_skillsrequires"*"; explicit lists keep absent skills aserroractions without pruning._resolve_requestsno longer treatsabsentas an incomplete run for prune suppression.Reviewed by Cursor Bugbot for commit cc31b69. Bugbot is set up for automated code reviews on this repo. Configure here.