Skip to content

feat: v1 — operator controls, observability and dashboard redesign - #11

Merged
kasvith merged 13 commits into
mainfrom
feat/v1
Oct 18, 2026
Merged

kasvith merged 13 commits into
mainfrom
feat/v1

Conversation

@kasvith

@kasvith kasvith commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Summary

The v1 work: operator controls, observability and workflow timeouts in the engine,
and a dashboard redesign built on them, with page tests for every screen.
13 commits, ordered so each one builds on its own (core and dashboard commits were
compiled and tested one by one).

Engine

  • Operator controls — Durable.retry/2 resumes a failed run from its failed step
    with a fresh attempt budget; Durable.rerun/2 starts a new run with the same (or
    edited) input, linked back. Runs record what started them (trigger: api, schedule,
    parent, rerun). Cancel uses a status compare-and-set. Queues can be paused and
    resumed across every node.
  • Workflow options — evaluated at compile time; unknown options fail the compile.
    timeout: is enforced (a run past its deadline fails before its next step, through
    the normal path, so compensations run; waiting runs are woken and failed). queue:
    is honoured.
  • Observability — Durable.Telemetry (documented events, Telemetry.Metrics
    definitions, an optional default logger), Durable.Stats, Durable.Activity (what
    Durable did on its own), a node registry, a per-minute queue sampler and a pruner.
  • Errors — exceptions, exits and throws are stored as type, readable message and
    stack (file and line per frame) for steps, compensations, the worker and a branch's
    on:.
  • Fixes
    • Parallel and branch block ids are stable across compiles; they changed on every
      compile, which broke resuming suspended runs after a deploy.
    • Re-enabling a schedule no longer crashes.
    • A resumed run keeps its first started_at; it was overwritten on every resume,
      which skewed durations and queue wait.
    • Durable's own log lines no longer land in a step's captured logs.

⚠️ Requires migrations. v20261016000000_add_operator_controls and
v20261016000001_add_deadlines_and_history. Run Durable.Migration.up() (the demo's
upgrade migrations are included as examples).

⚠️ Behaviour changes.

  • Unknown workflow options now fail the compile (:max_retries is gone).
  • timeout: is enforced.
  • A run's started_at is its first start, not its last resume.

Dashboard

Every page reworked against DESIGN.md, in dark and light:

  • Overview: KPIs, charts, failing workflows, slowest steps and a live feed.
  • Runs:
    • A query bar that holds every filter as a tag, with typing suggestions.
    • Saved views, and a bulk bar that floats over the list.
    • New runs wait behind a pill instead of shifting the rows.
  • A run's page:
    • Trace waterfall, and a graph whose parallel blocks end in a join node and whose branches have gateway nodes.
    • One step inspector shared by both views, showing the error with a readable stack.
    • Logs, Context and Family views.
  • Workflows: a page per workflow with a structure map.
  • Operations:
    • Inbox, Queues & nodes (pause and resume, backlog history), and Schedules (on/off, a page per schedule).
    • Telemetry and Settings pages.
  • Across the dashboard:
    • One styled tooltip, including chart data points.
    • ⌘K finds pages, saved views, workflows, schedules and any run by id.
    • Access levels (viewer, operator, admin) via a resolver, with an audit telemetry event per action.

Tests: DurableDashboard.LiveCase mounts the real LiveViews against a Durable
instance on Postgres, with runs from real workflows executed inline. 237 dashboard
tests, including access enforced server-side. The dashboard CI job gets a Postgres
service.

Demo

Adopts the v1 schema; adds a resolver (the demo user is an admin), telemetry (Durable's
logger, the dashboard audit log, Durable's metrics) and LiveDashboard at /dev/dashboard.
Moves to phoenix_live_view ~> 1.2.

Notes

  • The dashboard depends on durable as a path dependency on this branch: it needs
    core APIs that aren't on Hex yet. Switch back to Hex with the core release.
  • Dashboard tests now need Postgres locally: docker compose up -d (port 54321).
  • Known follow-ups:
    • A workflow's deadline can't interrupt a step that's still running.
    • An input with on_timeout: :fail still ends its run as cancelled.
    • The graph doesn't draw compensation steps.

Test plan

  • durable: mix test (483), mix credo --strict, mix compile --warnings-as-errors
  • durable_dashboard: mix test (237, three runs in a row), tsc --noEmit, biome check
  • Every page and run view loads (HTTP sweep of 58 URLs against the demo)
  • Walked every page in the browser, dark and light; exercised rerun, queue pause/resume, an inbox approval and schedule on/off

Postgres 18 images keep data under /var/lib/postgresql/<major>/, so the
compose volume mounts one level up, under a new name so an older-major
volume can't block startup.
Two migrations:

- v20261016000000 — run triggers (trigger, trigger_ref, retried_at), the
  queues table for pausing, and the nodes table for the node registry.
- v20261016000001 — run deadlines (deadline_at), per-minute queue_samples,
  and the activity table for operations Durable performs on its own.

A compensated run now also gets completed_at, so durations and retention
treat every finished run alike.
Durable.Telemetry documents every event Durable emits, offers
Telemetry.Metrics definitions (when :telemetry_metrics is installed) and an
optional default logger: one line per finished run and failed step.

The default logger tags its lines durable_internal: true, and step log
capture skips them: telemetry handlers run in the step's process, so they
were being stored as the step's own logs.
Block ids prefix the names of the steps inside them, and those names are
stored on runs, so a suspended run resumes by name. The ids came from
:erlang.unique_integer/1 and changed on every compile, which broke resumes
across deploys. They are now derived from the block's shape (clause
patterns and step names), bumped past any identical block earlier in the
module.
Operator controls
- Durable.retry/2 resumes a failed run from its failed step with a fresh
  attempt budget; Durable.rerun/2 starts a new run with the same (or edited)
  input, linked back by trigger: :rerun. Runs record what started them
  (api, schedule, parent, rerun).
- Cancelling uses a status compare-and-set, so a cancel can't race a
  finishing step.
- Queues can be paused and resumed across every node (Durable.Queue.Control).

Workflow options
- Options are evaluated at compile time; unknown ones fail the compile
  (:max_retries is gone). queue: is honoured.
- timeout: is enforced: deadline_at is set when the run is created and
  checked before each step, failing the run through the normal path so
  compensations run. The timeout worker wakes waiting runs past it. Retry
  resets the deadline.

Observability
- Durable.Stats: run and step statistics, timelines, queue history.
- Durable.Activity: a log of what Durable did on its own — recovered runs,
  fenced workers, failed acks, queue pauses, schedules turned off, pruning.
- Node registry (heartbeats), queue sampler (per-minute backlog) and
  pruner (retention), started by the supervisor.
- Query filters for trigger, trigger_ref, id prefix and more.

Errors
- Exceptions, exits and throws are stored as type, a readable message and
  the stack (file and line per frame) through one helper, for steps,
  compensations, the worker and a branch's on: function. Exits were stored
  without a stack and a branch's on: kept only the message.

Scheduler
- Turning a schedule back on crashed: whole-second next-run times went into
  a microsecond column. Times are normalised.
- Scheduler.API.upcoming_runs/2 lists the next fire times.
Documents workflow options (timeout, queue), what a failed step stores for
raised, exited and returned errors, and the operator, stats and telemetry
APIs.
…ry bar

Every page reworked against DESIGN.md (tokens, type, status semantics,
component primitives), in dark and light.

- Overview: KPI strip, runs/failure/duration charts, failing workflows,
  slowest steps, queues and a live activity feed.
- Runs: saved views, a query bar that holds every filter as a tag with
  typing suggestions (QuerySuggest), fixed-width columns, a "new runs"
  pill instead of rows shifting, a floating bulk bar; throttled refresh.
- Run page: header, banner and facts; one step inspector for the Trace and
  Graph views (error with a folded, readable stack; input, output, logs,
  attempts), linkable through ?step=; Logs, Context (engine keys folded
  away) and Family views.
- Workflows and a workflow's page: per-workflow stats and a structure map
  with traffic, give-ups and p95 per step.
- Inbox (inputs, events, sleeping runs), Queues & nodes (pause/resume,
  backlog history, operations feed), Schedules (on/off, a page per
  schedule), Telemetry and Settings.
- One styled tooltip for the whole dashboard (tooltip.ts, data-tip),
  replacing every native title, with chart data-point tooltips.
- ⌘K finds pages, saved views, workflows, schedules and any run by id.
- Access levels (viewer/operator/admin) via a resolver, with an audit
  telemetry event per action; Inbox badge in the sidebar.

Depends on the unreleased durable core, so durable is a path dependency
on this branch; switch back to Hex with the core release.
- A parallel block ends in a join ("all 3 done") and a branch is one
  gateway with an edge per clause, instead of wiring every child to every
  clause (N × M labelled edges).
- The branch marks the clause it took; in a finished run, steps that never
  ran are dashed and say "Not run". A step whose execution belongs to a
  child run shows as a child-run card.
- Cards show the status dot and "Completed · 147 ms", with a tooltip of
  the outcome, attempts, start time and error; the inspected card is
  ringed, and a step that never ran isn't clickable. Clause labels sit on
  the edge's last run into its target.
- The canvas is sized to the graph, never zooms out past 75% (wider graphs
  open at their start), and has a legend; the minimap is gone.
Adds the v20261016000000/1 durable migrations and updates the demo
workflows and pages for the v1 core (evaluated workflow options, triggers,
new query helpers).
- DurableResolver makes the demo user an admin of the dashboard.
- Telemetry: Durable's default logger, an audit log of dashboard actions,
  and Durable.Telemetry.metrics/0 in the demo's metrics.
- Phoenix LiveDashboard at /dev/dashboard reports them (Durable tab).
- phoenix_live_view ~> 1.2: 1.1.20 produced Elixir 1.20 "||" warnings.
Core
- A resumed run no longer overwrites started_at. It was reset on every
  execution, so after a wait the run's duration covered only the last
  stretch, and queue wait (ready → started) counted the whole wait for
  input or events as time in the queue.

Dashboard
- Run duration is labelled start to finish, waits included.
- The run header's children menu names each child's step from its own
  execution, like the Family tab; a finished parallel block drops the
  parent's __parallel_children, so it read "(no step)".
- The Runs started tooltip says what it counts: top-level runs only.
- A Runs selection starts over on a new page, view or filter; bulk actions
  only ever acted on the rows on screen.
DurableDashboard.LiveCase mounts the real LiveViews through a test router
against a Durable instance on TestRepo (the compose Postgres, database
durable_dashboard_test, created and migrated by test_helper). Runs come from
DurableDashboard.Fixtures — real workflows executed inline: completed,
failed, rolled back, waiting on input or an event, and a parallel + branch
family.

Covers the Runs list (query bar, saved views, bulk actions, the new-runs
pill), a run's page (every view, the inspector, the graph, rerun / retry /
cancel), Overview, Workflows, Inbox, Queues, Schedules, Telemetry, Settings,
the command palette, and access levels enforced server-side with the audit
event. A test showing an unfixed bug is tagged :known_bug and excluded.

CI's dashboard job gets a Postgres service.
@kasvith
kasvith merged commit 0028417 into main Oct 18, 2026
4 checks passed
@kasvith
kasvith deleted the feat/v1 branch October 18, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant