Skip to content

simplify(automation): hold one lifetime file lease per task key - #4542

Merged
ScriptedAlchemy merged 7 commits into
masterfrom
simplify/automation-task-lease
Oct 26, 2026
Merged

ScriptedAlchemy merged 7 commits into
masterfrom
simplify/automation-task-lease

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Refs #4426. Parent link: none (stack S3 link 1, base master).

Problem

Automation task exclusivity was a second locking protocol built on top of a file lock: a published <key>.lock file with random ownership token, staging file, hard-link publication, PID/start-time/age reclamation, and a destructor that retried durable unlinks with sleeps.

Fix

AutomationTaskLock is now one nonblocking exclusive FileLease on a stable <key>.lease inode, held for the task and retained-settlement lifetime (the retained settlement guard already owns the lock value). The lease file is never unlinked, so exclusion cannot split across inodes. Deleted: token generation, hard-link publication, staging ownership, uncertain-publication states, age/PID reclamation, destructor unlink/retry/fsync, stale_lock_secs() and the job stale-lock constant, and the acquire now_secs/stale_after_secs parameters.

Transition from the old format

A live pre-lease holder (<key>.lock naming a live pid whose start time matches) still keeps the key. Artifacts of a confirmed-dead or pid-reused owner are retired (.lock and .lock.lock). Unparseable or unverifiable ownership is a typed error, never treated as abandoned.

Not done (needs a settings-contract decision)

The stale_lock_secs setting (domain AutomationTaskSettingsV1, deny_unknown_fields, CLI flags, generated SDK) is still accepted and validated but no longer read. Removing it breaks persisted settings without a migration, so it is left for a follow-up.

Verification (Bazel, nonzero counts)

  • //crates/tracedecay-automation-runtime:unit_test: 432+ passed, including new tests: one-winner under 16 concurrent acquirers, independent keys, release admits next, lease file/inode survives release and is observed by a foreign open file description, no-runtime release, final-symlink rejection, live/dead/reused/unverifiable pre-lease owners.
  • //crates/tracedecay:automation_runner_test: PASSED. //crates/tracedecay-daemon-service:unit_test: PASSED.
  • //crates/tracedecay:unit_test: the lock-touching tests (shutdown_waits_for_accepted_curator_settlement_and_lock_release, partial_replay_reuses_prior_scheduler_skip_without_current_publication) pass; an_in_flight_files_resource_observes_transport_cancellation and independent_reads_complete_out_of_order_with_exact_ids failed only in the full run under load average ~45 and pass alone.
  • bazel build --config=clippy for automation-runtime, daemon-service and tracedecay: clean.

🤖 Generated with Claude Code

Replace the token-file task-lock protocol (random ownership token,
hard-link publication, staging files, PID/age reclamation, retrying
destructor unlink) with a nonblocking exclusive FileLease on a stable
`<key>.lease` inode held for the task and retained-settlement lifetime.
The lease file is never unlinked, so exclusion cannot split across inodes.

A live pre-lease holder (`<key>.lock` naming a live pid) still keeps the
key; dead or pid-reused owners' artifacts are retired and unverifiable
ownership is a typed error. The stale-lock timeout no longer applies.

Refs #4426

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 2f00b0e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

Fleet logs show five complete 48,000-function generations take 53–65 seconds each. The default medium deadline kills the final refresh near 300 seconds despite steady successful publication. Declare the measured aggregate workload through the existing Cargo-to-Bazel size metadata, preserving every refresh, memory bound, and assertion.

The current #4542 run independently confirmed the same limit: 170/171 tests passed; complete generations published at 62, 123, 184, and 245 seconds before the final refresh was interrupted at 300.1 seconds with nominal memory usage.

Failure evidence: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/40248084891/job/121393230765
ScriptedAlchemy added a commit that referenced this pull request Oct 26, 2026
Restack unique #4551 commits onto the rewritten #4542 tip, including the
inactive-job isolation, canonical disabled-backend fixtures, and refreshed
schedule description.
Resolve .github/workflows/ci.yml to master's ci-windows configuration,
which already carries this branch's --jobs=2 Windows concurrency bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ScriptedAlchemy added a commit that referenced this pull request Oct 26, 2026
@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

PASS+NOTES. Independent runtime verification of exact head 2f00b0ebd5565657d7309e0235f5989e270c6555 targeting master.

Fresh macOS checks used scripts/require-exact-test.sh bazel test //crates/tracedecay-automation-runtime:unit_test, separately for each full automation::scheduler::tests:: name, with --test_arg=--exact --nocache_test_results --test_output=all. Each invocation exited 0 and reported 1 passed; 0 failed; 436 filtered out.

  • old_and_new_processes_exclude_publication_and_execution_until_release. Bazel invocation 04fbe0ce-09b0-4877-ba4e-4731612399aa.
  • cancellation_releases_both_task_coordination_locks. Invocation a2953036-c9a4-4fe4-b49b-6e3006c0d7e2.
  • same_key_admits_one_holder_until_release_and_other_keys_are_independent. Invocation 185f4fe8-594b-4a1a-a926-a05ea9b937ca.
  • dead_pre_lease_owner_is_retired_without_replacing_coordination. Invocation 97d369ef-356e-43d8-b8fc-7828e486b55e.

Source review confirms that both stable coordination files remain held through retained settlement. Dead-owner retirement removes only the owner record. The cross-process test directly checks the pre-publication gap, old-versus-new exclusion, explicit release, process death, and stable legacy coordination identity.

Evidence limits. The legacy child replays the shipped protocol, checked against master source, rather than launching a published older binary. PID-identity and symlink denial received source review only. This is not a full journey or Windows runtime verdict. No source edits, assertion changes, or merges were made. Net CI changes are empty against master. The separate CLI test-size adjustment preserves the workload and assertions; its archived failure log confirms the 300.1-second deadline. The known unused stale_lock_secs setting is not a new blocker from this review. Branch protection and merge decisions remain with the parent.

@ScriptedAlchemy
ScriptedAlchemy merged commit 89d72ec into master Oct 26, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant