Skip to content

feat(flow): add flow run with phase/task selection via onlyFlowTaskIds (#725) - #732

Draft
padak wants to merge 2 commits into
mainfrom
claude/issue-725-flow-run
Draft

feat(flow): add flow run with phase/task selection via onlyFlowTaskIds (#725)#732
padak wants to merge 2 commits into
mainfrom
claude/issue-725-flow-run

Conversation

@padak

@padak padak commented Sep 2, 2026

Copy link
Copy Markdown
Member

What

Adds kbagent flow run — run a keboola.flow flow, optionally only part of it.

kbagent flow run --project P --flow-id ID
    [--from-phase PHASE_ID | --only-task TASK_ID ...]
    [--dry-run] [--wait] [--timeout N] [--branch ID]
    [--poll-strategy exponential|fixed] [--log-tail-lines N]
  • no selector — the ordinary full run, identical to job run --component-id keboola.flow.
  • --from-phase P — P's tasks plus every task in every phase reachable from P via next[].goto (BFS over the phase graph). Disabled tasks in scope are skipped silently and reported under skipped_disabled_task_ids.
  • --only-task ID (repeatable) — an explicit allowlist. An unknown or disabled id is a hard error, not a silent skip: the caller named it. Returned in flow order, deduplicated.
  • --dry-run — print the resolved selection, create no job. Requires a selector (a full run has nothing to preview).

Mirrored over kbagent serve as POST /flows/{project}/{config_id}/run.

Why

Flow execution had no phase or task awareness: every run started at phase 1 and re-executed the whole graph. Re-testing one downstream phase of a 16-phase production flow meant re-running ~50 unrelated tasks, or running components individually and bypassing the flow engine. onlyFlowTaskIds is a documented POST /jobs field (keboola/job-queue apps/public-api/docs/swagger.yaml) that kbagent simply never sent — which is why partial runs were UI-only.

Important: a selected run ignores the flow's conditions

The issue left two questions open. Both are settled by keboola/job-queue-daemon docs/flow-documentation.md, and the answer changes what this feature is good for:

A flow job created with onlyFlowTaskIds runs only the selected tasks and ignores the configured conditions.

Concretely, in a selected run: the phase graph is linearized into one sequential chain of synthetic unconditional selected-run transitions; phases kept only to route to a selected task run empty; a failing task does not stop the chain; the flow's final status is that of the last phase in the chain — so a partial run can report success with a failed task inside it; and flow variables from unselected variable tasks are not merged.

So this cannot be used to verify that new next[].condition logic evaluates correctly — the conditions are exactly what a selected run discards, and no API does that short of a full run (a dev branch keeps that cheap). It re-runs part of a flow, which is still the thing the UI's partial run does and the CLI could not.

Rather than leave that as a trap, it is stated in the command's --dry-run output and human output, in every JSON payload as conditions_evaluated: false, and in gotchas.md + keboola-expert.md.

kbagent also pre-empts the daemon's own user-errors locally, before a job exists — unknown phase id (listing the valid ones), a --from-phase unreachable from the entry phase, an unknown or disabled --only-task id, and an empty selection all exit 2 with a structured INVALID_ARGUMENT.

How

Three layers, no duplicated machinery:

  • client/queue.pycreate_job(only_flow_task_ids=...)body["onlyFlowTaskIds"]. The deprecated onlyOrchestrationTaskIds alias is deliberately not sent (the API rejects the two together and mirrors this one back under both keys anyway).
  • services/flow_validation.py — new pure reachable_phases(phases, start_id); find_unreachable_phases now delegates to it.
  • services/flow_service.pyresolve_flow_task_ids() does the resolution and the validation.
  • services/job_service.pyrun_job(only_flow_task_ids=...) passthrough, so the existing polling / log-tail / timeout-kill behaviour is reused unchanged rather than reimplemented in the flow service.
  • commands/_flow_run.py — the command, in a private module mounted flat onto flow_app, because commands/flow.py is at the 800-code-line commands ceiling (same pattern as _flow_triggers.py).

Docs updated across every silent-drift surface in CLAUDE.md convention #17: CLAUDE.md command list, context.py AGENT_CONTEXT, commands-reference.md, gotchas.md (tagged vNEXT), keboola-expert.md (tool-selection row + inline gotcha; 58.4 KB, cap 70 KB), and the generated SKILL.md.

Per CONTRIBUTING.md this feature PR does not bump the version or add a changelog.py entry; the vNEXT tags are resolved by the release PR.

Testing

  • make check — green (lint, format, ty, skill, version, version-gates, command-sync, endpoints, changelog, error-codes, sentinel-guards, loc, 6439 tests).
  • 27 new tests in tests/test_flow_run.py: reachable_phases (forward-only, forks, flow-order-not-visit-order, cycles, unknown start); resolution rules (downstream expansion, silent disabled skip, conditions_evaluated: false, unknown phase, unreachable phase, all-disabled selection, flow-order + dedup for --only-task, unknown/disabled id rejection, exactly-one-selector); the Queue body field (sent, alias absent, omitted when unset); and CLI wiring (both selectors rejected, --dry-run without a selector rejected, --dry-run creates no job, resolved ids reach the job service, no-selector path never fetches the flow body, error→exit-code mapping, flow.run is a write operation).
  • Four existing create_job call assertions in tests/test_services.py updated for the new keyword.
  • Not exercised against a live project: a partial run mutates production data, and the API semantics were verified against the Queue API swagger and the daemon's own documentation instead. Worth one live --dry-run → real run on a dev branch before this leaves draft.

Fixes #725

…Ids (#725)

Running a flow meant `job run --component-id keboola.flow`, which has no
phase or task awareness: every run starts at phase 1 and re-executes the
whole graph. Re-testing one downstream phase of a 16-phase production flow
therefore meant re-running ~50 unrelated tasks, or running the components
individually and bypassing the flow engine entirely.

`onlyFlowTaskIds` is a documented `POST /jobs` field (keboola/job-queue
`apps/public-api/docs/swagger.yaml`) that kbagent never sent, which is why
partial runs were UI-only. `flow run` sends it, and resolves a *phase* into
task ids for callers who think in phases:

- `--from-phase P` -- P's tasks plus every task in every phase reachable via
  `next[].goto` (BFS); disabled tasks in scope are skipped silently.
- `--only-task ID` (repeatable) -- explicit allowlist; an unknown *or
  disabled* id is a hard error, because the caller named it.
- `--dry-run` -- print the resolved selection, create no job. Requires a
  selector: a full run has nothing to preview.
- No selector -- the ordinary full run, unchanged.

Semantics verified against keboola/job-queue-daemon
`docs/flow-documentation.md`, which settles both open questions in the
issue: a selected run **ignores every configured condition**. The daemon
linearizes the phase graph into one chain of synthetic unconditional
`selected-run` transitions, routing-only phases run empty, a failing task
does not stop the chain, and the flow's final status is the last phase's.
So this re-runs part of a flow; it cannot verify new `next[].condition`
logic -- nothing short of a full run does. Every payload carries
`conditions_evaluated: false`, and the command, gotchas.md and
keboola-expert.md all say so explicitly.

kbagent pre-empts the daemon's user-errors locally, before a job exists:
unknown phase id (listing the valid ones), unreachable `--from-phase`,
unknown/disabled `--only-task` id, and an empty selection all exit 2.

Layers: `create_job(only_flow_task_ids=...)` on the Queue client;
`FlowService.resolve_flow_task_ids()` for resolution (reusing a new
`reachable_phases()` in flow_validation.py, which `find_unreachable_phases`
now delegates to); `JobService.run_job(only_flow_task_ids=...)` passthrough
so all the polling/log-tail/timeout-kill machinery is reused unchanged.
The command lives in `commands/_flow_run.py` because `commands/flow.py` is
at the 800-code-line ceiling, mounted flat like `_flow_triggers.py`.

Also mirrored as `POST /flows/{project}/{config_id}/run` over `kbagent serve`.

27 new tests: reachability, resolution rules, the Queue body field, and the
CLI wiring including --dry-run and the mutually-exclusive selectors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flow run: no way to trigger a Keboola Flow from a specific phase onward (onlyFlowTaskIds exists in job responses but is never set)

1 participant