Repository navigation
Workers & Ghouls — durable service jobs and recovery #22
Description
Outcome
One durable service job settles or recovers with explicit attempt identity and ownership.
Next planning checkpoint: Init. This is the earliest scope/review conversation, not an assignment or a promise to finish the whole outcome in that week. Selected work starts at #52.
Remaining acceptance
- Define and persist one ServiceJobAttempt with exact request, owner, idempotency and executor identity before submission.
- Wire one controlled adapter through submit → Graph park → terminal result → one resume; keep live handles out of checkpoints.
- Verify duplicate delivery, cancellation and crash around submission against PostgreSQL; reconcile uncertain outcomes without blindly launching again.
- Document worker concurrency and shutdown limits; a queued Ghoul is not an isolated process.
- Prove the bounded progress invariant for the chosen adapter: commit the durable wait/continuation before parking, release local capability grants, leases and run-scoped context before waiting, fence duplicate/stale claims, refresh or expire the declared heartbeat, and never let a parked Run retain a resource required by the work it awaits.
- Bound claim, retry, reconciliation and cancellation paths. A failed worker attempt does not silently choose another adapter or road; an ambiguous submitted effect keeps its identity and reconciles, while a changed executor or payload requires a newly admitted attempt.
Evidence and handoff
Start from current State of Work and its linked source/tests and owning ADRs. The linked original acceptance remains binding unless a named replacement outcome records its disposition. Detailed scope, negative cases and dated campaign evidence are preserved in preserved scope and evidence; historical evidence does not establish current qualification on its own. Any accepted requirement not selected for this profile needs an explicit route in #62 before this issue closes.
Choose one small implementation or verification slice when this outcome is admitted. Record the responsible contributor, exact profile/revision, observed result, failure/unknown boundary and next checkpoint. The founder reviews and approves all code for this campaign; merge and promotion follow CONTRIBUTING.
Activity
Correlate delegated observations with retained requests
zenithruneblade commentedon Nov 6, 2026
Planning reset — 6 November 2026. The original acceptance below remains binding unless a named replacement outcome records its disposition. This is a preserved historical body, not a fresh capability/qualification claim.
Original issue scope and dated evidence before the planning reset
Connect one bounded service operation to durable job state, Graph waiting and restart recovery.
Starting point: Current delivery: in-process Run workers already work. The generic service-attempt path is still designed.
First working slice
Connections: Builds on #8 and #18. Reuses existing Run delivery; complements #19 without replacing its peer-task contract.
Later: More service adapters, durable resource reservations and separately admitted multi-process workers.
References: ADR 14 · ADR 24.
Persistence foundation: #47 owns the core SQL/Alembic contract; new service-attempt records and their state transitions remain in this issue.
Contribution entry: Define one service-attempt state transition with crash/replay fixtures.
Open decision: Select the first controlled service adapter and its reconciliation semantics.
Verification: Real PostgreSQL at the transaction/recovery boundary, including duplicate claims, heartbeat expiry, cancellation, park/commit crash windows and a wait-for fixture proving that parked work holds no needed lease. This is a bounded progress claim, not a universal proof that arbitrary future workflows cannot deadlock. #62 owns the composed advanced review. Shared contribution and touched-docs rules: #52.
Delivery review — 12 October 2026
Published source: a913bff.
Current in-process Run workers have exact claim/park fencing, bounded owner recovery, cancellation and terminal-evidence repair. This campaign adds per-row cleanup-failure progress plus real PostgreSQL retained-debt recovery. The generic persisted ServiceJobAttempt, controlled submit/park/adopt adapter, heartbeat semantics and PostgreSQL crash-window/lease-progress receipt in this issue remain designed; existing Runs and the inert AgentJob path do not satisfy them.
Delivery evidence · Source/test 1