Skip to content

fix(update): preserve the agent pin on a failed resolution, and close a JIT-upgrade race - #711

Merged
defangdevs merged 1 commit into
masterfrom
fix/harness-upgrade-jit-pin-races
Sep 16, 2026
Merged

defangdevs merged 1 commit into
masterfrom
fix/harness-upgrade-jit-pin-races

Conversation

@defangdevs

Copy link
Copy Markdown
Owner

Summary

Two findings CodeRabbit raised on PR #710, real but unrelated to #707 (that
PR's branch was briefly based on stale, pre-squash-merge history that made
these look like part of its diff - they were not, and the branch has since
been corrected to a clean rebase onto master).

  • bin/agentbox: resolve_agent_pin() returned the mutable channel URL
    itself when resolution failed (network error, or a redirect that landed
    somewhere unexpected). The caller then wrote that straight into the pin
    file via write_agent_pin(), silently downgrading a good immutable pin to
    the mutable channel URL on nothing worse than a transient blip - and that
    write survives an otherwise-successful update, since it isn't part of what
    a failed update rolls back. Now returns None on failure, and the caller
    skips write_agent_pin() entirely, leaving the existing pin alone
    (matching what the OSError handler right next to it already did for a
    write that failed outright).

  • modules/src/lib/agents.sh: agent_upgrade() read the recorded pin
    and decided "already at this box's pin" before acquiring the per-harness
    flock. Two concurrent upgrades for the same agent (the update service
    loops over every configured user; a person can also run this by hand at
    the same moment) could both pass that check before either took the lock.
    If the second one's own nix profile add then failed, its recovery path
    restored what it still believed was the previous pin - silently undoing
    the first process's already-successful upgrade. Moved the pin read to
    right after the lock is acquired, matching how agent_install() already
    re-checks its own condition once it holds the lock ("The winner of the
    race installed it while we waited; nothing to do.").

Both are narrow, no-design-trade-off fixes, so this goes straight to a PR
rather than an issue per this repo's AGENTS.md.

A third finding on the same PR - SuccessExitStatus=0 1 on
agent-box-harness-upgrade@.service masking a genuine upgrade failure as
success - is not included here: exit 1 is also what the unit returns for
the documented "commonest reason: a box is offline" case that
SuccessExitStatus was added to tolerate, so narrowing it to 0 would
regress that intent and start flagging offline boxes as failed. Distinguishing
"offline/transient" from "genuinely broken" needs its own exit code, which is
a real design decision - tracking that separately rather than rushing it here.

Changes

  • bin/agentbox: resolve_agent_pin() returns None on failure instead of
    the mutable ref; the caller only calls write_agent_pin() when it got a
    real pin back.
  • modules/src/lib/agents.sh: move the pin read and "already at pin" check
    in agent_upgrade() to after the flock is acquired.
  • modules/agent-box.nix: regenerated (python3 bin/assemble-module.py).
  • tests/golden/vm/payloads/{agent-box-harness,agent-box-supervisor}/...:
    regenerated (nix run .#update-golden).

Test plan

  • bash tests/test-jit-agents.sh modules/src/lib/agents.sh - all
    assertions pass, no regressions (including the existing "an upgrade at the
    same pin runs no nix at all" and rollback-on-failed-add cases).
  • python3 -m py_compile bin/agentbox
  • python3 scripts/check_backend_parity.py - OK, every divergence
    declared.
  • nix build .#checks.aarch64-linux.{module-generated-up-to-date,golden-snapshot,one-spec-both-backends,multi-user,backend-parity,jit-agents,runtime-profile,module-single-file,phantom-unit-overrides} -
    all green (runtime-profile also confirms bin/agentbox still lints clean
    under writePython3Bin's flake8 pass).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MB26GQRXxhfGYoxWTsUp6M

… a JIT-upgrade race

Two findings CodeRabbit raised on PR #710 (real, but unrelated to #707,
since that PR's branch had been based on a stale pre-squash-merge
history that made them look like part of its diff - they are not; the
branch has since been corrected):

- bin/agentbox's resolve_agent_pin() returned the mutable channel URL
  itself on a failed resolution (network error, or a redirect that did
  not land on a nixpkgs release). The caller wrote that straight into
  the pin file, silently downgrading a good immutable pin to the mutable
  channel URL on nothing worse than a transient blip - and that write
  survives an otherwise-successful update, since it is not part of what
  a failed update rolls back. resolve_agent_pin() now returns None on
  failure and the caller skips write_agent_pin() entirely, leaving the
  existing pin alone (matching what the OSError handler already did for
  a write that failed outright).

- modules/src/lib/agents.sh's agent_upgrade() read the recorded pin and
  decided "already at this box's pin" BEFORE acquiring the per-harness
  flock, not after. Two concurrent upgrades for the same agent (the
  update service loops over every user; a person can run this by hand at
  the same moment) could both pass that check before either took the
  lock; if the second one's own `nix profile add` then failed, its
  recovery path restored what it still believed was the previous pin,
  silently undoing the first process's already-successful upgrade.
  Moved the pin read to right after the lock is acquired, matching how
  agent_install() already re-checks its own condition once it holds the
  lock ("The winner of the race installed it while we waited; nothing to
  do.").

Both are one-line-of-logic fixes with no design trade-off, so this goes
straight to a PR rather than an issue, per AGENTS.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MB26GQRXxhfGYoxWTsUp6M
@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 37 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: dd4c22b3-8104-41c4-9383-2e0423327b40

📥 Commits

Reviewing files that changed from the base of the PR and between 7f08f20 and d2ad16d.

📒 Files selected for processing (5)
  • bin/agentbox
  • modules/agent-box.nix
  • modules/src/lib/agents.sh
  • tests/golden/vm/payloads/agent-box-harness/bin/agent-box-harness
  • tests/golden/vm/payloads/agent-box-supervisor/bin/agent-box-supervisor

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@defangdevs
defangdevs merged commit ac4a56e into master Sep 16, 2026
8 checks passed
@defangdevs
defangdevs deleted the fix/harness-upgrade-jit-pin-races branch September 16, 2026 01:49
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Agent-Box Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant