Skip to content

Recover object-store readiness failures - #41

Merged
jeregrine merged 2 commits into
mainfrom
fix/object-store-readiness-transport-error
Oct 12, 2026
Merged

jeregrine merged 2 commits into
mainfrom
fix/object-store-readiness-transport-error

Conversation

@jeregrine

@jeregrine jeregrine commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Summary

DurableServer startup treated bucket readiness as only success or already-created, so a Req transport error escaped as a CaseClauseError. Retrying the startup then exposed the same contract mismatch as a backend-readiness match failure.

Retry idempotent bucket creation for transport failures, preserve exhausted errors, and report expected readiness failures as :backend_unavailable. Invalid backend return shapes still raise as contract violations.

CI's distributed placement test also requires EPMD, but the runner did not start it. Start EPMD explicitly before the test suite instead of invalidating the test on :nodistribution.

Evidence

  • Before: {:error, %Req.TransportError{reason: binary()}} crashed ensure_bucket_exists/1.
  • After: an ambiguous create retries and resolves an already-created bucket through 409; exhausted transport errors remain explicit, typed startup failures.
  • CI before: both matrix jobs invalidated BalancedPlacementTest because Node.start/2 could not reach EPMD.
  • CI after locally: the exact formatting, warnings-as-errors compile, and warnings-as-errors test sequence passes with EPMD started; 452 tests pass with 37 excluded.

Merge Danger

Blast radius: ObjectStore-backed supervisor startup readiness and CI test setup. A startup can make up to two additional idempotent bucket-create attempts for transport failures and the transient HTTP failures Req already retried. CI now runs one EPMD daemon for distributed tests.

There is no durable-state schema or stored-data change. A rollback restores the prior startup failure behavior; mixed versions do not share runtime state for this check.

@jeregrine
jeregrine merged commit 7702bd4 into main Oct 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant