Repository navigation
Conversation
Postgres 18 images keep data under /var/lib/postgresql/<major>/, so the compose volume mounts one level up, under a new name so an older-major volume can't block startup.
Two migrations: - v20261016000000 — run triggers (trigger, trigger_ref, retried_at), the queues table for pausing, and the nodes table for the node registry. - v20261016000001 — run deadlines (deadline_at), per-minute queue_samples, and the activity table for operations Durable performs on its own. A compensated run now also gets completed_at, so durations and retention treat every finished run alike.
Durable.Telemetry documents every event Durable emits, offers Telemetry.Metrics definitions (when :telemetry_metrics is installed) and an optional default logger: one line per finished run and failed step. The default logger tags its lines durable_internal: true, and step log capture skips them: telemetry handlers run in the step's process, so they were being stored as the step's own logs.
Block ids prefix the names of the steps inside them, and those names are stored on runs, so a suspended run resumes by name. The ids came from :erlang.unique_integer/1 and changed on every compile, which broke resumes across deploys. They are now derived from the block's shape (clause patterns and step names), bumped past any identical block earlier in the module.
Operator controls - Durable.retry/2 resumes a failed run from its failed step with a fresh attempt budget; Durable.rerun/2 starts a new run with the same (or edited) input, linked back by trigger: :rerun. Runs record what started them (api, schedule, parent, rerun). - Cancelling uses a status compare-and-set, so a cancel can't race a finishing step. - Queues can be paused and resumed across every node (Durable.Queue.Control). Workflow options - Options are evaluated at compile time; unknown ones fail the compile (:max_retries is gone). queue: is honoured. - timeout: is enforced: deadline_at is set when the run is created and checked before each step, failing the run through the normal path so compensations run. The timeout worker wakes waiting runs past it. Retry resets the deadline. Observability - Durable.Stats: run and step statistics, timelines, queue history. - Durable.Activity: a log of what Durable did on its own — recovered runs, fenced workers, failed acks, queue pauses, schedules turned off, pruning. - Node registry (heartbeats), queue sampler (per-minute backlog) and pruner (retention), started by the supervisor. - Query filters for trigger, trigger_ref, id prefix and more. Errors - Exceptions, exits and throws are stored as type, a readable message and the stack (file and line per frame) through one helper, for steps, compensations, the worker and a branch's on: function. Exits were stored without a stack and a branch's on: kept only the message. Scheduler - Turning a schedule back on crashed: whole-second next-run times went into a microsecond column. Times are normalised. - Scheduler.API.upcoming_runs/2 lists the next fire times.
Documents workflow options (timeout, queue), what a failed step stores for raised, exited and returned errors, and the operator, stats and telemetry APIs.
…ry bar Every page reworked against DESIGN.md (tokens, type, status semantics, component primitives), in dark and light. - Overview: KPI strip, runs/failure/duration charts, failing workflows, slowest steps, queues and a live activity feed. - Runs: saved views, a query bar that holds every filter as a tag with typing suggestions (QuerySuggest), fixed-width columns, a "new runs" pill instead of rows shifting, a floating bulk bar; throttled refresh. - Run page: header, banner and facts; one step inspector for the Trace and Graph views (error with a folded, readable stack; input, output, logs, attempts), linkable through ?step=; Logs, Context (engine keys folded away) and Family views. - Workflows and a workflow's page: per-workflow stats and a structure map with traffic, give-ups and p95 per step. - Inbox (inputs, events, sleeping runs), Queues & nodes (pause/resume, backlog history, operations feed), Schedules (on/off, a page per schedule), Telemetry and Settings. - One styled tooltip for the whole dashboard (tooltip.ts, data-tip), replacing every native title, with chart data-point tooltips. - ⌘K finds pages, saved views, workflows, schedules and any run by id. - Access levels (viewer/operator/admin) via a resolver, with an audit telemetry event per action; Inbox badge in the sidebar. Depends on the unreleased durable core, so durable is a path dependency on this branch; switch back to Hex with the core release.
- A parallel block ends in a join ("all 3 done") and a branch is one
gateway with an edge per clause, instead of wiring every child to every
clause (N × M labelled edges).
- The branch marks the clause it took; in a finished run, steps that never
ran are dashed and say "Not run". A step whose execution belongs to a
child run shows as a child-run card.
- Cards show the status dot and "Completed · 147 ms", with a tooltip of
the outcome, attempts, start time and error; the inspected card is
ringed, and a step that never ran isn't clickable. Clause labels sit on
the edge's last run into its target.
- The canvas is sized to the graph, never zooms out past 75% (wider graphs
open at their start), and has a legend; the minimap is gone.
Adds the v20261016000000/1 durable migrations and updates the demo workflows and pages for the v1 core (evaluated workflow options, triggers, new query helpers).
- DurableResolver makes the demo user an admin of the dashboard. - Telemetry: Durable's default logger, an audit log of dashboard actions, and Durable.Telemetry.metrics/0 in the demo's metrics. - Phoenix LiveDashboard at /dev/dashboard reports them (Durable tab). - phoenix_live_view ~> 1.2: 1.1.20 produced Elixir 1.20 "||" warnings.
Core - A resumed run no longer overwrites started_at. It was reset on every execution, so after a wait the run's duration covered only the last stretch, and queue wait (ready → started) counted the whole wait for input or events as time in the queue. Dashboard - Run duration is labelled start to finish, waits included. - The run header's children menu names each child's step from its own execution, like the Family tab; a finished parallel block drops the parent's __parallel_children, so it read "(no step)". - The Runs started tooltip says what it counts: top-level runs only. - A Runs selection starts over on a new page, view or filter; bulk actions only ever acted on the rows on screen.
DurableDashboard.LiveCase mounts the real LiveViews through a test router against a Durable instance on TestRepo (the compose Postgres, database durable_dashboard_test, created and migrated by test_helper). Runs come from DurableDashboard.Fixtures — real workflows executed inline: completed, failed, rolled back, waiting on input or an event, and a parallel + branch family. Covers the Runs list (query bar, saved views, bulk actions, the new-runs pill), a run's page (every view, the inspector, the graph, rerun / retry / cancel), Overview, Workflows, Inbox, Queues, Schedules, Telemetry, Settings, the command palette, and access levels enforced server-side with the audit event. A test showing an unfixed bug is tagged :known_bug and excluded. CI's dashboard job gets a Postgres service.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The v1 work: operator controls, observability and workflow timeouts in the engine,
and a dashboard redesign built on them, with page tests for every screen.
13 commits, ordered so each one builds on its own (core and dashboard commits were
compiled and tested one by one).
Engine
Durable.retry/2resumes a failed run from its failed stepwith a fresh attempt budget;
Durable.rerun/2starts a new run with the same (oredited) input, linked back. Runs record what started them (
trigger: api, schedule,parent, rerun). Cancel uses a status compare-and-set. Queues can be paused and
resumed across every node.
timeout:is enforced (a run past its deadline fails before its next step, throughthe normal path, so compensations run; waiting runs are woken and failed).
queue:is honoured.
Durable.Telemetry(documented events,Telemetry.Metricsdefinitions, an optional default logger),
Durable.Stats,Durable.Activity(whatDurable did on its own), a node registry, a per-minute queue sampler and a pruner.
stack (file and line per frame) for steps, compensations, the worker and a branch's
on:.compile, which broke resuming suspended runs after a deploy.
started_at; it was overwritten on every resume,which skewed durations and queue wait.
Dashboard
Every page reworked against
DESIGN.md, in dark and light:Tests:
DurableDashboard.LiveCasemounts the real LiveViews against a Durableinstance on Postgres, with runs from real workflows executed inline. 237 dashboard
tests, including access enforced server-side. The dashboard CI job gets a Postgres
service.
Demo
Adopts the v1 schema; adds a resolver (the demo user is an admin), telemetry (Durable's
logger, the dashboard audit log, Durable's metrics) and LiveDashboard at
/dev/dashboard.Moves to
phoenix_live_view ~> 1.2.Notes
core APIs that aren't on Hex yet. Switch back to Hex with the core release.
docker compose up -d(port 54321).on_timeout: :failstill ends its run ascancelled.Test plan
durable:mix test(483),mix credo --strict,mix compile --warnings-as-errorsdurable_dashboard:mix test(237, three runs in a row),tsc --noEmit,biome check