Timeouts that stop running steps, on_timeout: :fail fix, versioning phase 1

agents/ROADMAP.md

@@ -1,7 +1,7 @@
 # Durable — Roadmap
 
 **Last Updated:** 2026-10-23
-**Current release:** `durable` 1.0.0 and `durable_dashboard` 1.0.0 on Hex (2026-10-21)
+**Current release:** `durable` 1.1.0 (2026-10-23) and `durable_dashboard` 1.0.0 (2026-10-21) on Hex
 **Reference:** See `arch.md` for technical architecture, `WORKPLAN_ARCHIVED.md` for historical detail
 
 ---
@@ -50,20 +50,17 @@
 | [M05](milestones/M05-message-bus.md) | Message Bus | Partial | `Durable.PubSub` lifecycle events shipped. Adapter behaviour / pg_notify not built → backlog |
 | [M06](milestones/M06-phoenix-dashboard.md) | Phoenix Dashboard | Done | Shipped as its own package |
 | [M07](milestones/M07-scalability-adapters.md) | Scalability Adapters | Backlog | No demand for Redis yet |
+| [M08](milestones/M08-workflow-versioning.md) | Workflow Versioning | In Progress | Phase 1 (fail on a removed step) done; phase 2 (`version:` pinning) waits on the doc's open questions |
 
 ---
 
 ## Next Up
 
 In priority order:
 
-1. **Correctness follow-ups from v1:**
-   - The workflow deadline can't interrupt a step that's still running; it's only checked between steps.
-   - `wait_for_input` with `on_timeout: :fail` ends the run as `cancelled`, not `failed`. Fixing it changes behaviour, so ship it in a minor release with a changelog note.
-2. **Workflow versioning.** Needs a milestone doc. Changing a deployed workflow while runs are still in flight is the main open problem for production users.
-3. **DSL gaps:** waits inside `parallel` blocks, an `each`/foreach primitive (each iteration needs its own context, see `Durable.DSL.Step` docs), decisions inside `parallel`.
-4. **Testing follow-up:** wait timeouts can't be exercised by `Durable.Testing.drain/2` (the timeout worker only runs with the queue on). A helper that fires a run's pending timeouts would close it.
-5. **Formatter follow-up to #10:** the repo's own `durable/.formatter.exs` only *exports* `locals_without_parens`, so internal code and tests still get parens. Also add `:durable` to `import_deps` in `durable_dashboard` and `examples/phoenix_demo`.
+1. **Workflow versioning, phase 2** ([M08](milestones/M08-workflow-versioning.md)). Pin each run to the `version:` it started on. Needs the doc's three open questions answered first. Adds a migration.
+2. **DSL gaps:** waits inside `parallel` blocks, an `each`/foreach primitive (each iteration needs its own context, see `Durable.DSL.Step` docs), decisions inside `parallel`.
+3. **Testing follow-up:** wait timeouts can't be exercised by `Durable.Testing.drain/2` (the timeout worker only runs with the queue on). A helper that fires a run's pending timeouts would close it.
 
 ---
 
@@ -86,9 +83,10 @@ In priority order:
 2. Decisions are not supported inside parallel blocks
 3. No backward jumps in decision steps (forward-only by design)
 4. Context is single-level atomized (top-level keys only)
-5. No workflow versioning
+5. No workflow version pinning (M08)
 6. No foreach/loop DSL primitive (use Elixir's `Enum` functions)
 7. A resumed wait re-runs the step body from the top; side effects before the wait must be idempotent
 8. Only one `sleep/1` per step body
-9. The workflow deadline is checked between steps and can't interrupt a running step
-10. `Durable.Testing.drain/2` doesn't fire wait timeouts
+9. `Durable.Testing.drain/2` doesn't fire wait timeouts
+10. A step with a `timeout:`, or in a workflow with one, runs in its own process (`self()` isn't the worker)
+11. In-flight runs resume on the deployed definition; a removed current step fails the run (versioning: M08)

agents/milestones/M08-workflow-versioning.md

@@ -0,0 +1,158 @@
+# M08: Workflow Versioning
+
+**Status:** In Progress (phase 1 done, phase 2 needs the open questions answered)
+**Priority:** High
+**Effort:** 1 week
+**Dependencies:** None
+
+## Motivation
+
+A deploy changes the workflow code under runs that are still in flight. Today a
+run resumes against whatever definition is deployed, matched to its position by
+step name. Whether that's safe depends on the change:
+
+| Change | In-flight run today |
+|--------|---------------------|
+| Edit a step body | Runs the new body. Usually what you want (bug fixes). |
+| Add a step after the run's position | Runs it. |
+| Add a step before the run's position | Skips it. |
+| Remove a later step | Skips it. |
+| Rename or remove the step the run stopped at | **Was:** marked `completed`, every step after skipped, no error. **Now (phase 1):** fails with `definition_changed`. |
+| Change a `parallel`/`branch` block's steps | The block's id is a hash of its shape, so its step names change: same as a rename. |
+| A step expects context an earlier *new* step writes | Crashes at runtime, retried, fails. |
+
+The silent `completed` was a data-loss bug and is fixed. The rest needs a way to
+keep old runs on the old definition until they finish, which is what this
+milestone adds.
+
+## Scope
+
+### In Scope
+
+- Phase 1 (done): a run whose current step no longer exists fails with
+  `%{type: "definition_changed"}` and no compensations, so it stays retryable
+  once the step is restored.
+- Phase 2: explicit versions. Several definitions of one workflow side by side;
+  each run pinned to the version it started on.
+- A way to see which versions still have runs in flight, so old code can be
+  deleted safely.
+
+### Out of Scope
+
+- Replay-based patching (Temporal's `patched?`). Durable checkpoints steps
+  rather than replaying history, so there's nothing to keep deterministic.
+- Automatic versions from a structural hash. Every change would make a new
+  version, including the safe ones above, and old definitions still have to
+  stay in the code to be run.
+- Migrating a run from one version to another (moving its position by hand).
+
+## Architecture & Design
+
+### API
+
+```elixir
+defmodule MyApp.OrderWorkflow do
+  use Durable
+
+  # Runs started before v2 shipped finish on this. Delete it once
+  # `mix durable.status` shows none left.
+  workflow "order", version: 1 do
+    step :reserve, ...
+    step :approve, ...
+    step :ship, ...
+  end
+
+  workflow "order", version: 2 do
+    step :reserve, ...
+    step :review, ...
+    step :ship, ...
+  end
+end
+```
+
+- `version:` is a positive integer workflow option, default `1`. Every workflow
+  written so far is version 1, which matches every existing run.
+- Two `workflow` blocks with the same name and version fail the compile.
+- `Durable.start/3`, `call_workflow`, and schedules start the highest version.
+  `Durable.start(mod, input, version: 1)` picks one explicitly (for tests of the
+  old path).
+- `workflow_version/0` (in `Durable.Context`) returns the run's version, so one
+  definition can branch on it for small differences instead of being copied.
+
+### Storage
+
+- New column `workflow_executions.workflow_version integer NOT NULL DEFAULT 1`
+  (a new migration, shipped through `mix durable.gen.upgrade`). The default
+  covers existing rows.
+- Index `(workflow_module, workflow_name, workflow_version) WHERE status IN
+  (pending, running, waiting)` for the in-flight report.
+
+### Resolution
+
+- `__workflow_definition__(name)` keeps returning the latest version (the
+  dashboard and `Durable.Scheduler` rely on it);
+  `__workflow_definition__(name, version)` returns one version, and
+  `__workflow_versions__(name)` lists them.
+- The executor resolves `(module, name, version)`. A version that's gone fails
+  the run with `definition_changed`, like a missing step.
+- Parallel children copy the parent's version. A `call_workflow` child is a new
+  run and starts the callee's latest version.
+
+### Visibility
+
+- `Durable.Query.list_executions(version: 1)`.
+- `mix durable.status` shows in-flight runs per version.
+- Dashboard: the version on the run page; per-version in-flight counts on the
+  workflow page.
+- Telemetry: `:workflow_version` in run metadata (adding keys is allowed by the
+  telemetry contract).
+
+## Implementation Plan
+
+1. ~~Phase 1: fail on a missing current step~~ (done in this branch).
+2. DSL: `version:` option, per-version definition functions, duplicate check,
+   `__workflow_versions__/1`.
+3. Migration, schema, executor resolution, start/call/schedule pick latest,
+   parallel children inherit.
+4. `workflow_version/0`, query filter, `mix durable.status`, telemetry.
+5. Dashboard: version on the run page and in-flight counts per version.
+6. Guide: "Changing workflows" — which changes are safe in place, when to add a
+   version, how to retire one.
+
+## Testing Strategy
+
+- Two versions in one module: new runs start v2, a v1 run resumes and finishes
+  on v1.
+- A run whose version was deleted fails with `definition_changed`; restoring it
+  and `Durable.retry/2` finishes the run.
+- Parallel children run the parent's version.
+- Existing rows (no version set) resolve as version 1.
+- Duplicate `name`/`version` fails the compile.
+
+## Acceptance Criteria
+
+- [x] A deploy that removes a run's current step fails the run loudly and
+      recoverably instead of completing it.
+- [ ] A run always finishes on the version it started on, while that version is
+      in the code.
+- [ ] Operators can see which versions still have runs before deleting one.
+- [ ] Upgrading from 1.x needs only `mix durable.gen.upgrade` and a migrate.
+
+## Open Questions
+
+1. **API shape:** several `workflow "order", version: N` blocks in one module
+   (proposed), or one module per version (`MyApp.Order.V1`) tied together by a
+   name?
+2. **Release:** the new column means users must run a migration. Ship as 1.2.0
+   with an upgrade note, as the migration tooling was built for, or hold for
+   2.0?
+3. **Missing version:** fail the run (proposed), or fall back to the latest
+   version when the run's current step exists there too?
+
+## References
+
+- AWS Step Functions versions and aliases: executions are pinned to the version
+  they started on.
+- Restate deployments: old deployments keep serving in-flight invocations until
+  they drain.
+- `test/durable/definition_change_test.exs`: the phase 1 behaviour.

durable/.formatter.exs

@@ -1,21 +1,22 @@
 # Used by "mix format"
+locals_without_parens = [
+  # Workflow DSL
+  workflow: 2,
+  workflow: 3,
+  # Step DSL
+  branch: 2,
+  compensate: 2,
+  compensate: 3,
+  decision: 2,
+  decision: 3,
+  parallel: 1,
+  parallel: 2,
+  step: 2,
+  step: 3
+]
+
 [
   inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"],
-  export: [
-    locals_without_parens: [
-      # Workflow DSL
-      workflow: 2,
-      workflow: 3,
-      # Step DSL
-      branch: 2,
-      compensate: 2,
-      compensate: 3,
-      decision: 2,
-      decision: 3,
-      parallel: 1,
-      parallel: 2,
-      step: 2,
-      step: 3
-    ]
-  ]
+  locals_without_parens: locals_without_parens,
+  export: [locals_without_parens: locals_without_parens]
 ]

durable/README.md

@@ -497,9 +497,9 @@ workflow "process_order", timeout: hours(2), queue: :orders do
 ```
 
 - `:timeout` — how long a run may take from when it's due to start, waits and
-  sleeps included. A run past it fails with a `"timeout"` error before its next
-  step and runs its compensations; a waiting run is woken to do so (within the
-  timeout worker's one-minute sweep). It can't interrupt a step that's running.
+  sleeps included. A run past it fails with a `"timeout"` error and runs its
+  compensations: a step still running is stopped, and a waiting run is woken
+  to fail (within the timeout worker's one-minute sweep).
 - `:queue` — the queue runs start on unless `Durable.start/3` names one.
 
 Unknown options fail the compile.

durable/guides/waiting.md

@@ -252,6 +252,7 @@ end
 | `:metadata` | Additional data for UI |
 | `:timeout` | Max wait time |
 | `:timeout_value` | Value on timeout |
+| `:on_timeout` | `:resume` (default) returns `:timeout_value`; `:fail` fails the run and runs its compensations |
 | `:fields` | Field definitions for forms |
 | `:choices` | Options for choice inputs |
 

durable/lib/durable/dsl/step.ex

@@ -37,9 +37,13 @@ defmodule Durable.DSL.Step do
   - `:retry` - Retry configuration
     - `:max_attempts` - Maximum retry attempts (default: 1)
     - `:backoff` - Backoff strategy: `:exponential`, `:linear`, `:constant`
-    - `:base` - Base for backoff calculation (default: 2)
-    - `:max_backoff` - Maximum backoff in ms (default: 3600000)
-  - `:timeout` - Step timeout in milliseconds
+    - `:base` - Base delay in ms (default: 1000)
+    - `:max_backoff` - Maximum backoff in ms (default: 30000)
+  - `:timeout` - How long one attempt may run, in milliseconds. A slower
+    attempt is stopped and fails with a `"timeout"` error, which is retried
+    like any error. A step with a timeout, or in a workflow with one, runs in
+    a separate process: its body sees the same context and captured logs,
+    but `self()` is not the worker.
   - `:compensate` - Name of compensation function for saga pattern
   - `:queue` - Override queue for this step
 
@@ -89,7 +93,7 @@ defmodule Durable.DSL.Step do
         type: :step,
         module: __MODULE__,
         body_fn: &(__MODULE__.unquote(func_name) / 1),
-        opts: unquote(Macro.escape(normalized_opts))
+        opts: unquote(quote_opts(normalized_opts))
       }
     end
   end
@@ -113,6 +117,14 @@ defmodule Durable.DSL.Step do
 
   defp normalize_retry_opts(opts), do: opts
 
+  # Option values are code: splice them in rather than escape them, so
+  # `timeout: seconds(30)` is a number in the workflow module, not the quoted
+  # call.
+  defp quote_opts(opts) when is_map(opts),
+    do: {:%{}, [], Enum.map(opts, fn {key, value} -> {key, quote_opts(value)} end)}
+
+  defp quote_opts(value), do: value
+
   @doc """
   Defines a decision step for conditional branching.
 
@@ -165,7 +177,7 @@ defmodule Durable.DSL.Step do
         type: :decision,
         module: __MODULE__,
         body_fn: &(__MODULE__.unquote(func_name) / 1),
-        opts: unquote(Macro.escape(normalized_opts))
+        opts: unquote(quote_opts(normalized_opts))
       }
     end
   end
@@ -227,7 +239,7 @@ defmodule Durable.DSL.Step do
         name: unquote(name),
         module: __MODULE__,
         body_fn: &(__MODULE__.unquote(func_name) / 1),
-        opts: unquote(Macro.escape(normalized_opts))
+        opts: unquote(quote_opts(normalized_opts))
       }
     end
   end
@@ -299,7 +311,7 @@ defmodule Durable.DSL.Step do
             type: :step,
             module: __MODULE__,
             body_fn: &(__MODULE__.unquote(func_name) / 1),
-            opts: unquote(Macro.escape(step_opts))
+            opts: unquote(quote_opts(step_opts))
           }
         end
       end)
@@ -515,7 +527,7 @@ defmodule Durable.DSL.Step do
             type: :step,
             module: __MODULE__,
             body_fn: &(__MODULE__.unquote(func_name) / 1),
-            opts: unquote(Macro.escape(step_opts))
+            opts: unquote(quote_opts(step_opts))
           }
         end
       end)

durable/lib/durable/dsl/workflow.ex

@@ -22,9 +22,10 @@ defmodule Durable.DSL.Workflow do
 
   - `:timeout` - How long a run may take, in milliseconds, from when it is due
     to start — waits and sleeps included. A run past it fails with a
-    `"timeout"` error before its next step, running compensations like any
-    failure; a waiting run is woken to do so. The deadline is checked between
-    steps, so it can't interrupt a step that is still running.
+    `"timeout"` error, running compensations like any failure: a step still
+    running is stopped (and not retried), and a waiting run is woken to fail.
+    Steps in a run with a timeout run in their own process; see the step
+    `:timeout` option.
   - `:queue` - The queue runs start on unless `Durable.start/3` names one
 
   """