Skip to content

feat(dashboard): show delayed node eligibility time - #862

Open
breken-ai wants to merge 1 commit into
FailproofAI:mainfrom
breken-ai:feat/dashboard-delayed-node-eligibility
Open

feat(dashboard): show delayed node eligibility time#862
breken-ai wants to merge 1 commit into
FailproofAI:mainfrom
breken-ai:feat/dashboard-delayed-node-eligibility

Conversation

@breken-ai

Copy link
Copy Markdown

Closes #611

What goes wrong

When a node is delayed (ReQueueAfterSignal / RequeueAtSignal, a retry-policy backoff, or a graph start_delay) the state manager stores the moment it becomes eligible in State.enqueue_after (epoch milliseconds; enqueue_states picks states with enqueue_after <= now). That value never reaches the dashboard: NodeRunDetailsResponse does not carry it, so the Node Details modal shows a CREATED node with no hint of why it is not running or when it may.

What changes

state-manager

  • NodeRunDetailsResponse.enqueue_after: Optional[int] = None (epoch ms), described as the stored eligibility time and explicitly not a guaranteed execution time.
  • get_node_run_details passes state.enqueue_after through verbatim (stored value, not recomputed).

dashboard

  • NodeRunDetailsResponse.enqueue_after?: number | null in the client types.
  • New pure helper src/lib/nodeEligibility.ts: getNodeEligibility(details, nowMs) returns scheduled (future) or waiting (past-due, still not picked up) only for CREATED states with a usable timestamp; null otherwise. formatEligibilityTime renders the instant in the viewer's locale/timezone with a zone label. Wire representation stays epoch ms; nothing localized is ever sent back.
  • NodeDetailsModal shows, under Status & Timestamps:
    • Eligible after: <local time TZ> — "Delayed by a start delay, retry policy or requeue signal. A worker can pick this node up any time after this moment; it is not a guaranteed start time."
    • Eligible since: <local time TZ> — "Eligible to run and waiting for a worker to pick it up."
      The clock reading is paired with the details object it was taken for, so a node opened long after page load is never classified against a stale reading. A 5 s interval flips scheduledwaiting on time; the timeout and interval are cleared whenever details change or the modal unmounts.
  • npm test script (node --test, zero new dependencies) for the helper; allowImportingTsExtensions in tsconfig.json (valid because the project is noEmit) so the test can import ./nodeEligibility.ts the way Node's type-stripping loader requires.
  • Docs: one paragraph in docs/docs/exosphere/dashboard.md and a feature bullet in dashboard/README.md.

What deliberately does not change

  • No scheduler/dispatch logic: enqueue_states, the requeue signals and the retry path are untouched. This is observability only.
  • No new persisted fields; enqueue_after already exists on every State document.
  • Nothing is shown for QUEUED/EXECUTED/SUCCESS/ERRORED/… states, so a node that was already picked up never carries a stale "waiting" message.
  • Records or servers without the field (undefined/null/0) show nothing — no Invalid Date, no 1970 epoch.
  • Existing tests were not modified; the existing MagicMock fixtures still pass because the field is optional.

Tests

  • state-manager: tests/unit/controller/test_get_node_run_details.py +2 (exposes_stored_enqueue_after, response_enqueue_after_is_optional). Full suite 639 → 641 passed.
  • dashboard: src/lib/nodeEligibility.test.ts 0 → 7 passed (fake clock via injected nowMs, no real timers): future delayed node, past-due node, boundary at == now (matches the server's <=), missing/null/0/NaN field, every non-CREATED status, null/undefined details, and the same instant rendered in UTC / Asia/Tokyo / America/Los_Angeles across a day boundary.

Both new tests fail on main (AttributeError: 'NodeRunDetailsResponse' object has no attribute 'enqueue_after'; ERR_MODULE_NOT_FOUND for the helper).

Evidence record

Repository / issue: FailproofAI/runtime — #611 "feat: Show next run time of state"
Upstream SHA / patch SHA: 2db4b154c8bb020051f6ba349ed922f8479eef13 / fc5e6c9ef845f6deea883756a5cdceffcfcaca32
Issue ownership and competing PR result (2026-09-12T21:49:28Z UTC):
  state=open, assignees=[], updated_at=2026-01-03T14:58:35Z; only comment is the CodeRabbit plan bot;
  cross-references: #616 (state querying, unrelated feature issue);
  gh search prs "611" -> none; gh search prs "enqueue_after next run time" -> none;
  open PRs (37) -> none touch node run details / enqueue_after.

Reproducer command and red result:
  state-manager$ uv run pytest tests/unit/controller/test_get_node_run_details.py -q
    -> 2 failed, 6 passed  (AttributeError: 'NodeRunDetailsResponse' object has no attribute 'enqueue_after')
  dashboard$ node --test src/lib/nodeEligibility.test.ts
    -> fail 1  (ERR_MODULE_NOT_FOUND ./nodeEligibility)

Patched command and green result:
  state-manager$ uv run pytest tests/ -q          -> 641 passed (baseline on main: 639 passed)
  dashboard$ npm test                             -> tests 7, pass 7, fail 0

Full checks executed (all local, macOS, Python 3.12 via uv, Node 25.6.1):
  state-manager$ uv sync --group dev
  state-manager$ MONGO_URI=mongodb://admin:password@localhost:27017 MONGO_DATABASE_NAME=test_exosphere_state_manager \
      STATE_MANAGER_SECRET=test-secret-key SECRETS_ENCRYPTION_KEY=<CI value> uv run pytest tests/ -q
      (mongo:7 in Docker with the CI credentials)          -> 641 passed
  state-manager$ uv run ruff check <3 changed .py files>   -> All checks passed!
  dashboard$ npm ci
  dashboard$ npx tsc --noEmit -p tsconfig.json             -> exit 0
  dashboard$ npm test                                      -> 7/7
  dashboard$ npx next build                                -> ✓ Compiled successfully, exit 0
  dashboard$ eslint on the 4 touched files with typescript-eslint recommended + react-hooks recommended
      (ad-hoc flat config, not committed)                  -> 0 findings on changed lines; the 2 reported
      errors (react-hooks/set-state-in-effect) are on pre-existing lines of NodeDetailsModal.tsx.

Checks not executed and why:
  dashboard$ npm run lint  -> cannot run on upstream main: `next lint` was removed in Next 16
      ("Invalid project directory provided ... /dashboard/lint"), and `eslint src` with the repo's
      eslint.config.mjs fails at config load (circular structure in FlatCompat + eslint-config-next 16;
      eslint-config-next's bundled eslint-plugin-react also calls context.getFilename, removed in ESLint 10).
      Both are pre-existing and unrelated to this change; not fixed here to keep the PR in scope.
  No browser-driven test of the modal: the dashboard has no component test runner; the display rule is
  isolated in the pure helper and covered there. Verified visually by reading the JSX only.

https://claude.ai/code/session_01DkE5qM85ht9Uoq3aAqcKf7

A node delayed by ReQueueAfterSignal/RequeueAtSignal, a retry-policy
backoff or a graph start_delay stores the instant it becomes eligible in
State.enqueue_after (epoch ms, picked up when enqueue_after <= now), but
NodeRunDetailsResponse never carried it, so the Node Details modal showed
a CREATED node with no explanation of the delay.

Expose the stored enqueue_after as an optional field on the node run
details response (passed through, not recomputed) and render it in the
modal only for CREATED states: "Eligible after" while the instant is in
the future, "Eligible since" once it has passed and the node is still
waiting for a worker. The copy states that this is an eligibility time,
not a guaranteed start time. Older records/clients without the field see
nothing (no Invalid Date, no epoch zero); already queued or finished
nodes carry no waiting message.

The wire representation stays epoch milliseconds; formatting to the
viewer's locale/timezone happens in a pure helper covered by fake-clock
tests (node --test, no new dependencies). The modal pairs each clock
reading with the details it was taken for and clears its timers when the
details change or it unmounts. Scheduler dispatch logic is untouched.

Closes FailproofAI#611

Claude-Session: https://claude.ai/code/session_01DkE5qM85ht9Uoq3aAqcKf7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Show next run time of state

1 participant