fix(apple): key the toolchain fingerprint memo on the resolved xcode-select dir

[thymikee]

Summary

toolchainFingerprintCacheKey keyed the toolchain fingerprint memo on DEVELOPER_DIR alone, so every default xcode-select selection shared one entry. A daemon that answered a cache decision before someone ran sudo xcode-select -s kept answering with the Xcode the host had left. Fixes #3300

• The key is now the resolved developer dir plus an identity for the bundle standing in it. The dir comes from /var/db/xcode_select_link (or the client's DEVELOPER_DIR) resolved through realpath, so a version manager swapping the bundle behind a stable path, or a client naming a symlinked Xcode, lands on the right entry. xcode-select -p (trimmed) is the fallback when the link cannot be read; a selection neither can name leaves the read with no key, so it skips the memo rather than sharing one.
• The bundle half is the ino and integer mtimeMs of <bundle>/Contents/version.plist, the file xcodebuild -version reads. An update that replaces the bundle in place keeps the path, and a hit renews its entry, so without this the replaced build would be served for as long as the daemon stayed busy.
• The TTL renews on a hit, as it did before this PR, which is what apple: the toolchain fingerprint memo answers a stale default xcode-select selection for its whole TTL #3300 asked to keep. It is safe now because the key covers both ways the selection can change: a moved path and a replaced bundle. An entry nothing reads still expires.
• The failure report reads only the memo. It reuses the key halves the last real read built, so it never resolves, states, or spawns anything — it has no budget and no deadline. Client-named answers are retained per name in a TTL memo, because a daemon serves request-scoped DEVELOPER_DIRs concurrently and one global record would let the last read erase what an earlier client needs to report on. Cancellation is observed after the key's two filesystem reads, so a warm hit cannot hide an abort.

5 files (2 production, 3 test). Related: #3449, whose Xcode-too-old reports should name the toolchain a decision actually read; not implemented here.

Cost of the new key resolution

Measured at b3f7e6562 here over 20 reads each, warm memo:

	median
realpath of /var/db/xcode_select_link	0.021 ms
statSync of the bundle version file	0.003 ms
key resolution, both reads	0.024 ms
xcode-select -p spawn (the fallback only)	3.88 ms
computeRunnerSourceFingerprint (same call, already unbounded sync fs)	1.35 ms
whole resolveExpectedRunnerCacheMetadata, warm	1.27 ms

A warm read spawns nothing, which is also what makes renewal affordable: the key costs 1.9% of the call it belongs to, and the xcode-select -p it replaces on that path costs two orders of magnitude more than the two reads that replaced it.

Validation

pnpm check:affected --run --head d33b8049e → passed, exactCommit: true, checkoutChanged: false, receipt .tmp/validation/run-O0GjyD/receipt.json. Coverage, swift-runner and replay lanes are GitHub-authoritative and not claimed here.

Rebased onto 4c1c78696; git range-diff 40b16020b..b3f7e6562 4c1c78696..d33b8049e shows all seven commits unchanged, so the evidence below keeps its original SHA attribution. Upstream touched none of the five files this PR changes.

pnpm exec vitest run --project apple-runner <probe+metadata suites>:

base 4c1c78696	head d33b8049e
21 failed / 54 passed	75 passed

Mutations, each planted in the source at b3f7e6562 (content-identical here) and measured against that suite in this checkout: keying on the dir as named instead of its realpath (20 failures), dropping the selection-link read (5), probing xcode-select before reading the link (5), dropping the bundle identity from the key (2), removing TTL renewal (1), removing the cancellation check after the key reads (1), sending the failure report back through realpath (1), collapsing client answers into one slot (1), and keying a client's DEVELOPER_DIR unresolved (1).

All Smoke Tests lanes pass at d33b8049e — iOS run, plus Linux, Android and macOS, and every other check at the head is green. This is the lane that reaches the changed route on real toolchain state: the job's own prepare ios-runner returned "cache": "exact" with "buildMs": 0 (the toolchain fingerprint resolved and matched a cached build), then "connectMs": 3387 and "healthCheckMs": 20064 against a live runner reporting "currentUptimeMs": 1653725 — so a runner process was up in the simulator, and 01-settings.ad replayed open/appstate/click with a runner session prewarmed ("runnerPrewarmKind":"session", 29.3 s). The key came from the host's default selection: DEVELOPER_DIR appears in none of the step env dumps in that log, so the read goes through /var/db/xcode_select_link (falling back to xcode-select -p) exactly as in the fix, and the builds resolve to /Applications/Xcode_26.6.app. The earlier smoke failures were upstream — e24dc4568 called linux.gap after 4db1b7d67 removed the helper, which crashed the coverage declarations at import on every PR, and #3740 fixed it.

Live run

This host has one Xcode (/Applications/Xcode-27.1.0.app) and no passwordless sudo, so sudo xcode-select -s is not available here — the two-Xcode switch is still open with @thymikee. Everything below runs the production route (resolveExpectedRunnerCacheMetadata → requireRunnerToolchainFingerprint) against the real root-owned /var/db/xcode_select_link and real xcodebuild/xcrun output. A second toolchain is an APFS clone of the Xcode with ProductBuildVersion edited, so the substitution is only in how the selection is moved, never in what the probes read. All three "after" transcripts were re-captured at b3f7e6562; each "before" is the same script at the head that lacked the fix.

Warm, and unchanged on a repeated read (real host selection, no substitution at all):

xcode-select -p / readlink / realpath   /Applications/Xcode-27.1.0.app/Contents/Developer  (all three agree)
cold read   xcodeBuildVersion 27A9275  sdkBuildVersion 24A94232  …/derived/ios-simulator/cache-2d8626f9b665c837  132.8 ms
repeat read xcodeBuildVersion unchanged: true   derived path unchanged: true   median of 20: 1.58 ms
one Xcode by two paths (DEVELOPER_DIR alias → same real dir): memo hit, 1.53 ms, derived path unchanged

The reviewer's xcodes case — DEVELOPER_DIR names one stable path and the symlink behind it is swapped for another working Xcode. "Before" is head 3e61534, which you reviewed. The DEVELOPER_DIR string is identical before and after in both columns; only what the key uses differs.

                              head b3f7e6562              head 3e61534 (before the fix)
  before swap  build          27A9275                     27A9275
  after swap   key follows    /private/tmp/…Xcode-Clone   the raw DEVELOPER_DIR string, unchanged
               build          99B999   ← moved            27A9275   ← stale
               derived path   …cache-7cdf9ec1ec611bc8     …cache-2d8626f9b665c837  (unchanged)
               read cost      249 ms (re-probes)          2.39 ms (memo hit)
               stale answer   no                          YES

An in-place bundle replacement at an unchanged path, steady hits across the swap — this is what the bundle identity buys, and "before" is head 2953a77, which keyed on the dir alone:

                                head b3f7e6562                                  head 2953a77 (before the fix)
  read 3 (hit, renews)   27A9275  ino=153118599  1.7 ms                 read 3 (hit, renews)   27A9275  ino=152595979
  >> bundle replaced in place, DEVELOPER_DIR and realpath unchanged
  read 4 (after)         88C888   ino=153119192  92.8 ms  ← re-probes   read 4 (after)         27A9275  ino=152597930  2.2 ms (hit)
  read 5 (hit, renews)   88C888   …cache-78dc058b17eac9ba               read 5 (hit, renews)   27A9275  …cache-2d8626f9b665c837
  stale answer served    no                                             stale answer served    YES

Each substitution is an honest limit: they move the selection through a symlink or replace the bundle marker, not through sudo xcode-select -s, so the root-owned link itself is exercised only by unit tests against a scripted link plus the fact that the read is the same realpath. A two-Xcode host with sudo would close that end to end.

Review threads

All seven Cubic threads are resolved and answered: the report pointer on a hit, the "throw or skip" and two overstated comments, renewal against an in-place bundle replacement (r4297381441), the fs-bound reasoning on r4297381453, the missing-dir wording, and per-client retention of the reported answer (r4297748734).

🤖 Generated with Claude Code

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	5.32 MB	5.32 MB	+1.1 kB
Package (unpacked)	5.32 MB	5.32 MB	+1.1 kB
Package (download)	1.60 MB	1.60 MB	+397 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	30.2 ms	29.7 ms	-0.6 ms
CLI --help	82.0 ms	84.9 ms	+2.8 ms

[cubic-dev-ai]

All reported issues were addressed across 4 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[thymikee]

The code change in 3e61534 looks right, but the PR needs one live run before merge. I did not run the tests; I read the diff.

Every new test replaces runCmdSync, so no run uses the real xcode-select -p output (trailing newline, symlink path, CLT fallback). Nothing shows a real xcode-select -s changing xcodeBuildVersion or the derived path in runner-toolchain-probe.ts. The PR body says "No device run", but the changed route is resolveReusableRunnerSession in runner-session.ts:483, which decides whether a retained simulator runner is reused. docs/agents/pull-requests.md:15 asks for proof that the changed route ran live. A real host can key differently from the mocks. For example, a /Applications/Xcode.app symlink swapped by a tool like xcodes leaves xcode-select -p unchanged, so the key does not move until the TTL ends. Please attach one run on a host with two Xcodes: warm the fingerprint through a runner command on a simulator (or resolveExpectedRunnerCacheMetadata), run sudo xcode-select -s <other Xcode>, and repeat the same command. The output should show a changed xcodeBuildVersion and derived path, and unchanged results on a repeated read with the same selection. Please add the per-call wall time of the new xcode-select -p.

Not blocking, take or leave: a warm read used to spawn nothing, and now every read without DEVELOPER_DIR runs a synchronous xcode-select -p (about 6 ms per spawn on my host, measured over 20 spawns, which says nothing about a loaded CI host), so please state the cost in the PR or read the /var/db/xcode_select_link symlink first with the subprocess as fallback; and the TTL no longer renews on a hit, which issue #3300 asked to keep, so please say in the PR that this is a deliberate second change, or restore renewal since the key already covers the switch.

The three cubic-dev-ai threads (#3726 (comment), #3726 (comment), #3726 (comment)) are fixed at this head, so please resolve them.

CI shows 16 checks passing and Smoke Tests still running. I did not read that job, so I do not know if it starts an iOS runner and so reaches the real xcode-select -p; if it does and fails, the failure is related to this change. Before merge, the live switch run needs to be attached and Smoke Tests needs to finish.

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[thymikee]

The earlier findings on 3e61534 are partly addressed at 2953a77: the memo key now comes from the resolved xcode-select directory. One finding remains. The earlier review asked for one live run on a host with two Xcodes, and this head still has none. The new key source is the realpath of /var/db/xcode_select_link (https://github.com/callstack/agent-device/blob/2953a77/packages/platform-apple/src/runner/runner-toolchain-probe.ts#L209). Every test fakes it by mocking fs.realpathSync for that path (https://github.com/callstack/agent-device/blob/2953a77/packages/platform-apple/src/runner/apple-toolchain-fixtures.ts#L70-L83). No run reads the real root-owned link after a real sudo xcode-select -s. This matters because resolveReusableRunnerSession uses this key to decide whether to reuse a retained simulator runner. Please attach one live run on a host with two Xcodes. Warm the fingerprint through a runner command on a simulator, run sudo xcode-select -s <other Xcode>, and repeat the same command. The output must show a changed xcodeBuildVersion and derived path after the switch, no value-probe re-run on a repeated read with an unchanged selection, and the wall time of key resolution per read.

Not blocking, and you can take or leave it: the PR body still describes 3e61534 (it says the key comes from xcode-select -p on every read and that hits no longer extend the TTL), while at this head the link read comes first and hits renew the entry, so please update the Summary bullets; XCODE_SELECT_LINK_PATH is also exported only so the suite can match it.

The open inline threads still apply in part. Two hold: the renewal on every hit can serve an old build after an in-place Xcode update at the same real path, and rewording the comments settles it (#3726 (comment)); and the "no key" test wording is inaccurate (#3726 (comment)). One does not apply, so you can resolve it: realpathSync reads a local root-owned link, and any stall would also stall xcodebuild in the same read (#3726 (comment)).

I did not run the tests, so I did not confirm that the new tests fail on 3e61534 or on main. I also did not measure realpath against the spawn cost, and I could not check whether every macOS or Xcode version writes that link. When it is missing, the read falls back to xcode-select -p.

The 11 checks that are not passing are queued or still running for the latest push, so none has failed yet. Coverage and Integration cover the runner suites this diff touches. Before merge, the live run with a real sudo xcode-select -s through a simulator runner command is still needed.

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[thymikee]

Status at 0b90479, taking your 17:40 comment point by point.

Renewal serving an old build after an in-place update (r4297381441) — you were right, and it needed more than rewording. Fixed at the key: it now also carries the ino and integer mtimeMs of <bundle>/Contents/version.plist, the file xcodebuild -version reads. An update replacing the bundle in place moves that identity, so the key moves and the entry under the old key expires once nothing renews it. a bundle replaced in place at the same path moves the key, plus a live run: steady hits, marker replaced mid-run, build 27A9275 → 88C888 at 188 ms, where 2953a77 keeps answering 27A9275 from a 2 ms hit. Both transcripts are in the body.

"no key" test wording (r4297381459) — reworded to what the helper guarantees.

realpathSync bound (r4297381453) — resolved per your call, with two additions: cancellation is observed after the key's two reads, so a warm hit cannot answer across an abort; and the failure report now touches no filesystem at all, reusing the key halves a real read already built.

XCODE_SELECT_LINK_PATH exported only for the suite — taken. It sits in apple-runner-platform.ts beside COLD_TOOLCHAIN_PROBE_TIMEOUT_MS, which is the module's stated home for a constant both Apple toolchain probers read.

Summary bullets describing 3e61534 — rewritten. They now say the link read comes first and that hits renew.

The live two-Xcode run — still open on this host: one Xcode, no passwordless sudo. The body now carries three live transcripts on the production route against the real root-owned link and real xcodebuild/xcrun, each with its fail-before at the head you reviewed, and each labelled as to what is substituted: warm read unchanged (1.37 ms median of 20), the xcodes symlink swap (27A9275 → 99B999, derived path moves), and the in-place bundle replacement. Per-read key-resolution wall time is in the cost table: 0.067 ms for both reads, ~3% of the 2.12 ms warm call, with the xcode-select -p spawn at 3.86–6.11 ms measured for comparison. As you noted, the link itself is still only exercised by unit tests against a scripted link plus the fact that the read is the same realpath — a host with two Xcodes and sudo would close that end to end, and I'd rather attach that than claim it.

On your last paragraph: agreed, and nothing here depends on the link existing — a host that lacks it takes the xcode-select -p path, which is the shape 2953a77 had for every read.

Gate: pnpm check:affected --run --head 0b9047976 → passed, exactCommit: true, receipt .tmp/validation/run-0pjMSA/receipt.json. Suite at head 74 passed, base 40b16020b 21 failed / 53 passed, with eight mutations listed in the body. All six Cubic threads resolved.

[thymikee]

Round 4 in b3f7e6562, and one thing about CI you should not read as this PR.

Per-client report retention (r4297748734) — the reviewer is right, and it was a defect in my own round-3 design: I replaced the report's fs work with a single global, so the last client-named DEVELOPER_DIR read erased what an earlier client needed while its fingerprint was still cached. Answers are retained per name now, in a TtlMemo on the same 10-minute clock as the fingerprints they point at, so a client dir whose reads stopped drops out no later than its entry. each client-named selection keeps its own report across another client read fails with expected undefined to be '15.1' against the one-slot shape.

Gate: pnpm check:affected --run --head b3f7e6562 → passed, exactCommit: true, receipt .tmp/validation/run-tIIfv3/receipt.json. Suite 75 passed at head, base 40b16020b 21 failed / 54 passed. Nine mutations, each detected (20/5/5/2/1/1/1/1/1). All three live transcripts were re-captured at b3f7e6562, with their fail-before pairs; the body's cost table is now a fresh median-of-20 at this head (key 0.024 ms, 1.9% of the warm call).

Smoke Tests is red for everyone, not just here. At b3f7e6562 Android fails on Run Android emulator catalog coverage contract and iOS on Run fixture-backed iOS simulator E2E smoke; Linux passes. On main's own tip (0149438fa) macOS fails on Run macOS command coverage contract and iOS was cancelled. Every one dies at import with TypeError: linux.gap is not a function at test/integration/command-coverage/declarations.ts:558, while loading coverage declarations, before any device work starts, so the runner path is never executed. e24dc4568 added that linux.gap(...) call and 4db1b7d67 removed the helper it calls — this branch touches neither file, and a merge takes main's two files verbatim, so no PR can pass that lane until 5f60c69d6 lands. Importing that module locally succeeds at this head.

That does bear on your live-run requirement, so to be explicit about what is and is not on the table here: the iOS lane did run green on this branch at 3e61534, before main broke, and it reaches the probe path. What is still missing is a host with two Xcodes and sudo xcode-select -s, which this machine cannot provide. My substitute runs stand where they were: selection moved by symlink, bundle replaced in place, real link, real xcodebuild/xcrun.

[thymikee]

Closing out your 16:39 list at d33b8049e (rebased on main after #3740; range-diff shows all seven commits unchanged).

Done: /var/db/xcode_select_link is read first with xcode-select -p as fallback, so a warm read spawns nothing; renewal is back and now safe, since the key also carries the bundle's ino + mtimeMs; the per-read cost is in the body (0.024 ms for both key reads, 1.9% of the warm call); the three Cubic threads are resolved; and Smoke Tests finished — all four lanes pass, and the iOS job does start a runner: prepare ios-runner returned "cache": "exact" with "buildMs": 0, then connected to a simulator runner and health-checked it (runner-reported uptime 27 min), and 01-settings.ad replayed through it. No DEVELOPER_DIR is set anywhere in that job, so that key came from the host's default selection.

Not done here: the live sudo xcode-select -s switch on a host with two Xcodes. This machine has one Xcode and no passwordless sudo, so I cannot run it honestly. What's in the body instead is labelled as a substitute, and says what stands in — the selection is moved by swapping an xcodes-style symlink and by replacing an APFS-cloned bundle in place, never by faking what the probes read; xcodebuild/xcrun and the real root-owned link are untouched. Give me a two-Xcode host and I'll attach that transcript.