[Jackmeson1]

Summary

A Commit can stall after publication when an open orphan's lower source belongs to the captured generation. The mounted reproductions create files since the previous Commit, hold them open and unlink them after capture. On the base revision, the Commit did not return and readdir and create did not answer.

Known install and definite failed-capture resolution queued orphan-source wakes while the Commit's captured reader still held the generation. Each woken orphan parked again and restored its wait entry, keeping the wake pending. The owner drained that maintenance before returning the job's result, while the Commit could release its reader only afterward.

This change makes parking and waking use the same predicate: an orphan must wait while the capture has not settled its lower generation or a captured reader still holds it.

Repro

The issue's standalone orphan_wake_repro.rs, saved at engine/crates/layerfs-overlay/tests/orphan_wake_repro.rs, uses the existing payload_support module. It writes a file in the captured generation, acquires a captured reader, unlinks the file with a descriptor open, then installs the capture. Maintenance does not go idle within 256 turns on caebd268d; it does with this change.

The new mounted_orphan_commit test reaches the product control Commit through a real mount. An external bash holds three files open. The real History provider publishes, the external History wrapper holds the Commit before local install, the files are unlinked, and the Commit is released.

On caebd268d, the mounted test reports:

the first Commit did not return within 5s; the Workspace answers: Err("readdir: no answer (killed; leader reaped: true); create: no answer (killed; leader reaped: false)")

Closing the descriptors was not tried as a way out of the stall. Failure cleanup is not that experiment.

Root cause

The following source references are at caebd268d, under engine/crates/:

• layerfs-overlay/src/lifetime/generation.rs:24-25: install calls wake_orphan_sources; definite failed-capture resolution and closed-capture release call it too.
• layerfs-overlay/src/maintenance/source_wait.rs:9-22: a generation-wait entry queues a WAKE_ORPHAN item without checking whether the generation remains awaited.
• source_wait.rs:23-42: the wake removes entries from the indexed in-memory wait map and makes their orphan items ready. The wake finishes only after finding an empty batch.
• maintenance/orphan.rs:237-242: migration finds the captured generation held, restores its wait entry and parks again.
• layerfs-store-service/src/overlay/owner.rs:401-403,413-474: OwnerClient::maintain drains ready work before the job returns.
• layerfs-store-service/src/store/captured.rs:82-103,190-202: the captured reader is released after the driver's outcome, so it cannot end the wait while the install job remains in that drain.

The mounted failures establish that readdir and create also did not answer. I have not established their precise starvation mechanism.

Fix

• Add orphan_source_awaited(gen) in maintenance/source_wait.rs: the current capture owns the generation, or a captured reader holds it.
• Use that predicate in migrate_orphan, replacing the same inline check.
• Return from wake_orphan_sources without queueing an orphan-source wake while the predicate holds.
• Document the rule in a new current section of core/docs/architecture/35-shared-physical-capacity.md.

The wait ends when both ownership conditions clear. If settlement happens first, the final reader release reaches the wake through wake_generation. If the reader releases first, later settlement reaches the same check. Settlement includes known install, definite failed-capture resolution and release of a closed capture.

Captured-reader lifetime, wake_generation, wake stepping, generation retirement and OwnerClient::maintain are unchanged. The owner still drains until no work is ready. This fixes the cycle; it does not bound foreground maintenance generally or fix the large failed-Commit fold.

Tests

Outside-sandbox validation was performed on vm2: Linux aarch64, Ubuntu 24.04, kernel 6.8.0-147, Rust 1.85.1, --locked, Disposable profile and LAYERFS_CONSTRUCTION_WORKERS=1.

The suite comparison used caebd268d plus the two new regression-test files as the base, and the final five-file change as the fix. Source fingerprints in the retained receipts match f160d6e80.

Each suite package was built first with cargo +1.85.1 test --locked -p <crate> --no-run. Its binaries were copied under /tmp and executed separately as root, with LAYERFS_CONSTRUCTION_WORKERS=1, timeout 120 and --test-threads=1. The standalone issue reproduction was added temporarily, built first and executed separately under timeout 30; it needs neither root nor FUSE.

Test	Base	Fix
layerfs-overlay orphan_wake: install, failed capture, early reader release, no-reader control, no-descriptor control	3 failed, 2 passed	5 passed
layerfs-daemon mounted_orphan_commit	Failed in 9.04 s	Passed in 0.03 s
Issue's standalone orphan_wake_repro	Failed	Passed

The three overlay failures on base report:

install while the reader holds the generation: the drain does not end
failed capture while the reader holds the generation: the drain does not end
a release before the install: the drain does not end

The overlay tests also check retained captured bytes, wake progress and reclamation. Multiple-reader and closed-capture orphan combinations are not additional cases in the new test file.

The mounted test bounds mount-dependent observations from Ready through its unmount attempt. Scripts lead their own process groups and enter the mount inside bash. Owner/control calls run on worker threads with channel deadlines. Both Commits are observed for five seconds; only a handle already observed finished is joined. The product unmount is observed for ten seconds.

Timed-out scripts and probes signal their own groups and allow up to one second to reap each leader. If the experiment fails, Scene::abandon then attempts one lazy detach of its own mount, observing it for three seconds and allowing up to one further second to reap a killed unmount leader. It prints whether the mount entry remains, then kills the retained holder group and allows one second to reap its leader.

Cleanup is deliberately incomplete. Retained worker/Commit threads, the Harness and its serial gate remain until process exit; the fixture directory remains on disk for external cleanup. A killed helper blocked on the mount may outlive its reap budget. A failed lazy detach can leave a mount entry: process exit closes the test's FUSE connection but is not proof of detachment. The test never releases the captured reader early, retries the Commit or manipulates another FUSE connection. It is the only test in its binary. Bring-up and clean Harness shutdown retain the harness's existing lifecycle behavior and the invocation's external timeout.

In the retained base execution, the guard reported the mount entry gone and the holder leader reaped. A separate post-exit inspection found no matching mount or helper processes before external cleanup.

Whole suite-binary results, summed as passed / failed / ignored:

Crate	Binaries	Base	Fix
layerfs-overlay	17	76 / 3 / 1	79 / 0 / 1
layerfs-store-service	4	3 / 0 / 0	3 / 0 / 0
layerfs-daemon	50	127 / 2 / 5	128 / 1 / 5
layerfs-workspace	31	176 / 0 / 0	176 / 0 / 0
layerfs-fuse	5	8 / 0 / 0	8 / 0 / 0

The remaining daemon failure is forced_unavailable, at its precondition: the control filesystem must not be mounted here. vm2 has fusectl mounted; the same test fails on base. The additional base daemon failure is mounted_orphan_commit.

Ignored on both sides, requiring external setup:

• layerfs-overlay: allocation_filesystem (1).
• layerfs-daemon: complete_installed_roots (2), host_handoff (1), mounted_sqlite_full (2).

The throwaway unlink_during_commit race harness is not included in the tree. It creates 4,000 Workspace files, holds 100 open, and begins unlinking all names at delays of 0, 5 or 20 ms after Commit starts. Each run used timeout 60.

On caebd268d, all three runs reached the harness's 25-second Commit observation limit; readdir and create did not answer within five seconds. The root stat answered at delay 0. In that run, the harness had not exited after 52 seconds and its own FUSE connection was aborted externally.

With this change, the first Commit returned Committed in 549, 532 and 502 ms at those delays. These are diagnostic observations from one run per delay, not a performance claim. After unlinking completed, the harness committed again and compared the live and published trees with mtime excluded:

live paths=401 published paths=401 diffs(no mtime)=0

The following Commit returned UpToDate.

An earlier delay-0 execution on an earlier build returned:

FAILED phase: Construct, error: Content { error: InvalidRecord("binding kind") ... }

That is a separate unresolved failure: the Commit returned rather than stalling. This change does not address it. It did not recur in the three final runs, and the deterministic mounted regression unlinks only after publication, so it does not exercise that earlier construction window.

Checks on the final source:

• cargo +1.85.1 fmt --manifest-path Cargo.toml --all -- --check: clean.
• cargo +1.85.1 clippy --manifest-path Cargo.toml --locked -p layerfs-overlay -p layerfs-store-service -p layerfs-daemon --all-targets -- -D warnings: clean.
• cargo +1.85.1 check --manifest-path Cargo.toml --locked --workspace --exclude layerfs-node --all-targets: clean.
• python3 -B tools/check_product_boundary.py --skip-sdk: PASS, 818 production files; Node and JavaScript source scanning excluded.
• python3 -B -m unittest discover -s tools -p 'test_*.py': 155 passed.

Not run: layerfs-api test execution, macOS and Windows execution. The workspace check compiled layerfs-api. Durable: NOT_RUN — disabled by owner until explicit reauthorization.

Production LOC: 102,200 -> 102,203 (delta +3)

Counted with python3 -B tools/production_loc.py --root <tree> --detail on a git archive of parent caebd268d and the final source tree. Engine: 61,313 -> 61,316; layerfs-overlay: 9,730 -> 9,733. Core 38,241, API 2,646 and layerfs-store-service 10,032 are unchanged. Reference absent (retired R9c; last counted 65,417 at tree 498dd1917812). Counter blob 3341e4d0362c822b6d9a08419c432942b3a13800 is identical in both snapshots. Tests and documentation are excluded.

Fixes #474

AI disclosure: this was prepared with AI assistance (Claude Opus 5.5).

🤖 Generated with Claude Code

https://claude.ai/code/session_019TTGAv6fe859ebbCLLfLHP