feat(flow): add flow run with phase/task selection via onlyFlowTaskIds (#725) - #732
Draft
padak wants to merge 2 commits into
Draft
feat(flow): add flow run with phase/task selection via onlyFlowTaskIds (#725)#732padak wants to merge 2 commits into
flow run with phase/task selection via onlyFlowTaskIds (#725)#732padak wants to merge 2 commits into
Conversation
…Ids (#725) Running a flow meant `job run --component-id keboola.flow`, which has no phase or task awareness: every run starts at phase 1 and re-executes the whole graph. Re-testing one downstream phase of a 16-phase production flow therefore meant re-running ~50 unrelated tasks, or running the components individually and bypassing the flow engine entirely. `onlyFlowTaskIds` is a documented `POST /jobs` field (keboola/job-queue `apps/public-api/docs/swagger.yaml`) that kbagent never sent, which is why partial runs were UI-only. `flow run` sends it, and resolves a *phase* into task ids for callers who think in phases: - `--from-phase P` -- P's tasks plus every task in every phase reachable via `next[].goto` (BFS); disabled tasks in scope are skipped silently. - `--only-task ID` (repeatable) -- explicit allowlist; an unknown *or disabled* id is a hard error, because the caller named it. - `--dry-run` -- print the resolved selection, create no job. Requires a selector: a full run has nothing to preview. - No selector -- the ordinary full run, unchanged. Semantics verified against keboola/job-queue-daemon `docs/flow-documentation.md`, which settles both open questions in the issue: a selected run **ignores every configured condition**. The daemon linearizes the phase graph into one chain of synthetic unconditional `selected-run` transitions, routing-only phases run empty, a failing task does not stop the chain, and the flow's final status is the last phase's. So this re-runs part of a flow; it cannot verify new `next[].condition` logic -- nothing short of a full run does. Every payload carries `conditions_evaluated: false`, and the command, gotchas.md and keboola-expert.md all say so explicitly. kbagent pre-empts the daemon's user-errors locally, before a job exists: unknown phase id (listing the valid ones), unreachable `--from-phase`, unknown/disabled `--only-task` id, and an empty selection all exit 2. Layers: `create_job(only_flow_task_ids=...)` on the Queue client; `FlowService.resolve_flow_task_ids()` for resolution (reusing a new `reachable_phases()` in flow_validation.py, which `find_unreachable_phases` now delegates to); `JobService.run_job(only_flow_task_ids=...)` passthrough so all the polling/log-tail/timeout-kill machinery is reused unchanged. The command lives in `commands/_flow_run.py` because `commands/flow.py` is at the 800-code-line ceiling, mounted flat like `_flow_triggers.py`. Also mirrored as `POST /flows/{project}/{config_id}/run` over `kbagent serve`. 27 new tests: reachability, resolution rules, the Queue body field, and the CLI wiring including --dry-run and the mutually-exclusive selectors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
kbagent flow run— run akeboola.flowflow, optionally only part of it.job run --component-id keboola.flow.--from-phase P— P's tasks plus every task in every phase reachable from P vianext[].goto(BFS over the phase graph). Disabled tasks in scope are skipped silently and reported underskipped_disabled_task_ids.--only-task ID(repeatable) — an explicit allowlist. An unknown or disabled id is a hard error, not a silent skip: the caller named it. Returned in flow order, deduplicated.--dry-run— print the resolved selection, create no job. Requires a selector (a full run has nothing to preview).Mirrored over
kbagent serveasPOST /flows/{project}/{config_id}/run.Why
Flow execution had no phase or task awareness: every run started at phase 1 and re-executed the whole graph. Re-testing one downstream phase of a 16-phase production flow meant re-running ~50 unrelated tasks, or running components individually and bypassing the flow engine.
onlyFlowTaskIdsis a documentedPOST /jobsfield (keboola/job-queueapps/public-api/docs/swagger.yaml) that kbagent simply never sent — which is why partial runs were UI-only.Important: a selected run ignores the flow's conditions
The issue left two questions open. Both are settled by
keboola/job-queue-daemondocs/flow-documentation.md, and the answer changes what this feature is good for:Concretely, in a selected run: the phase graph is linearized into one sequential chain of synthetic unconditional
selected-runtransitions; phases kept only to route to a selected task run empty; a failing task does not stop the chain; the flow's final status is that of the last phase in the chain — so a partial run can reportsuccesswith a failed task inside it; and flow variables from unselected variable tasks are not merged.So this cannot be used to verify that new
next[].conditionlogic evaluates correctly — the conditions are exactly what a selected run discards, and no API does that short of a full run (a dev branch keeps that cheap). It re-runs part of a flow, which is still the thing the UI's partial run does and the CLI could not.Rather than leave that as a trap, it is stated in the command's
--dry-runoutput and human output, in every JSON payload asconditions_evaluated: false, and ingotchas.md+keboola-expert.md.kbagent also pre-empts the daemon's own user-errors locally, before a job exists — unknown phase id (listing the valid ones), a
--from-phaseunreachable from the entry phase, an unknown or disabled--only-taskid, and an empty selection all exit 2 with a structuredINVALID_ARGUMENT.How
Three layers, no duplicated machinery:
client/queue.py—create_job(only_flow_task_ids=...)→body["onlyFlowTaskIds"]. The deprecatedonlyOrchestrationTaskIdsalias is deliberately not sent (the API rejects the two together and mirrors this one back under both keys anyway).services/flow_validation.py— new purereachable_phases(phases, start_id);find_unreachable_phasesnow delegates to it.services/flow_service.py—resolve_flow_task_ids()does the resolution and the validation.services/job_service.py—run_job(only_flow_task_ids=...)passthrough, so the existing polling / log-tail / timeout-kill behaviour is reused unchanged rather than reimplemented in the flow service.commands/_flow_run.py— the command, in a private module mounted flat ontoflow_app, becausecommands/flow.pyis at the 800-code-line commands ceiling (same pattern as_flow_triggers.py).Docs updated across every silent-drift surface in CLAUDE.md convention #17: CLAUDE.md command list,
context.pyAGENT_CONTEXT,commands-reference.md,gotchas.md(taggedvNEXT),keboola-expert.md(tool-selection row + inline gotcha; 58.4 KB, cap 70 KB), and the generatedSKILL.md.Per CONTRIBUTING.md this feature PR does not bump the version or add a
changelog.pyentry; thevNEXTtags are resolved by the release PR.Testing
make check— green (lint, format, ty, skill, version, version-gates, command-sync, endpoints, changelog, error-codes, sentinel-guards, loc, 6439 tests).tests/test_flow_run.py:reachable_phases(forward-only, forks, flow-order-not-visit-order, cycles, unknown start); resolution rules (downstream expansion, silent disabled skip,conditions_evaluated: false, unknown phase, unreachable phase, all-disabled selection, flow-order + dedup for--only-task, unknown/disabled id rejection, exactly-one-selector); the Queue body field (sent, alias absent, omitted when unset); and CLI wiring (both selectors rejected,--dry-runwithout a selector rejected,--dry-runcreates no job, resolved ids reach the job service, no-selector path never fetches the flow body, error→exit-code mapping,flow.runis awriteoperation).create_jobcall assertions intests/test_services.pyupdated for the new keyword.--dry-run→ real run on a dev branch before this leaves draft.Fixes #725