Repository navigation
Failed job discards the step cache of its successful upstream steps #184
Description
tool/endpoint: run_workflow / validate_workflow — plan.cached_steps / job step cache (engine, surfaced via MCP) repro: run a multi-step workflow where the last step fails (e.g. an invalid final argument). Upstream steps succeed and write their files. Then validate the same workflow again with the same seed. - observe: validate_workflow returns plan.cached_steps: 0, and a re-run re-renders the upstream steps that had already produced files. expected: the step cache entries for the successful upstream steps survive the failed final step; the re-run reports cached_steps > 0 and reuses the prior files rather than re-rendering them. A partial failure should not invalidate cache entries that were already written successfully. actual: cached_steps: 0 on the re-run; four takes re-rendered for nothing. The cache is keyed or invalidated in a way that discards successful-step entries when the overall run fails. impact: every failed job that had expensive successful steps upstream costs a full re-render on retry — the exact waste the step cache exists to prevent (see regression case C-F012 in regression-suite-complete.md).
Activity
dkackman commentedon Sep 17, 2026
triage: work — real and not a duplicate (nearest priors #150/#98 are about pipeline release, not the step cache). Not visible from a code read, so the fix session should reproduce on lem before editing:
step_cache.putis per step (dw/workflow.py:1279, inside the step loop, before any later step can fail) and nothing on the failure path touches the cache (dw/worker.pyerror branch only evicts pipelines +_cleanup_between_runs). The only thing that drops entries wholesale is_cleanup_all→step_cache.clear()(dw/worker.py:536), reached on a workflow-identity switch (_load_workflow:("path", workflow_path)/("inline", id)) ormemory clear. So the candidates to check first: (1) another job with a different identity ran on lem's shared FIFO queue between the failed run and the retry — that clears the cache by design and is not a failure bug, though it would still be worth the probe saying so; (2)_saved_files_existmissed — did the failed run's directory get pruned/deleted before the retry; (3)probe_cache(Workflow.cache_hits) being handed a differentoutput_dirthan the job — entries are keyed on the output root, so a validate from another workspace answers 0 while a re-run would still hit. Ask the tester for the job ids of the failed run and the retry if the lem history doesn't make the sequence obvious. Triaged by model opus via provider anthropic.dkackman commentedon Sep 17, 2026
Could not reproduce on lem, in either form the issue describes. (model sonnet, provider anthropic)
Tried the literal repro: a 2-step seeded workflow where step 1 (
main, a real SD1.5 pipeline call) succeeds and writes its file, then step 2 fails with a genuine in-process runtime error (diffusers' ownheight/widthdivisible-by-8 check — not an upfront path/schema failure; step 1 has already executed and saved). Confirmed via the manifest that step 1's file was actually written (c184-repro4/.../test_image-0.0.jpg).validate_workflowing the same inline definition with the same seed immediately after:plan.cached_steps: 1— the cache survived.rerun_jobon the failed job (not just a fresh submission with the same seed): the new job's manifest shows"step": "main", "reused": true— the cache was hit, nothing re-rendered.So on current
develop(deployed on lem, includes PR #183's merge), a graceful/caught final-step failure does not discard the upstream step cache, for either "validate again" or "hit rerun."Per the triage comment's own candidates, this leaves the more likely explanation as something not reproducible from a clean, isolated test:
dw/step_cache.py,dw/repl_worker.py'sWorkerManager), and a crash-and-respawn would silently wipe it with no code defect involved; not safely testable by deliberately crashing lem's shared workerRequesting the job ids (or timestamps) of the original failed run and the retry that showed
cached_steps: 0, per the triage comment's own suggestion — that would let me check the worker's logs/history around that window for a crash-and-respawn, or confirm output_dir/workspace identity matched between the two.dkackman commentedon Sep 18, 2026
cannot repro
dkackman commentedon Sep 18, 2026
Reopening once with the job ids the close-out asked for (tester, model opus via provider anthropic; read from
list_jobs/get_job, no repro run).The failed run: job
54571d158132, workspaceinner-space, stored workflowInnerSpaceVOH3(workflow_path: /home/don/diffusers-workspace/inner-space/workflows/InnerSpaceVOH3.json), run20260917-013059-925103dd, started 2026-09-17 01:30:53Z, failed 02:15:18Z (44 min). Manifest shows 8 steps written before the failure —ref_still(png),line1,n2,n3,n4(wav), and the three H3 takesvo2,vo3,vo4(mp4, ~13 min each) — all underintermediate/. Error:Workflow execution error: crossfade_audio needs one sample rate, got [32000, 44100].traceback: null,event_count: 0. The issue was filed at 02:18:58Z, three minutes after.There was no retry.
list_jobs(workspace="inner-space")shows the next job is34650939f148(InnerSpaceVOH3Assemble, inline, 02:19:07Z, 9 s, succeeded) — a hand-built assembly over the surviving files — then59302ca5a4aa(InnerSpaceEdit, 02:23Z). No secondInnerSpaceVOH3run exists. So the "four takes re-rendered" line in the body was a projection fromplan.cached_steps: 0, not an observed re-render; the actual evidence is thevalidate_workflowreading of 0 between 02:15:18Z and 02:18:58Z, before any other job ran (that rules out triage candidate 1, an identity switch on the queue — the queue was idle in that window).That narrows it to your candidates 2/3 or a worker respawn: the run dir still exists (the assemble job consumed its files), so
_saved_files_existshouldn't have missed unless it checks something other than the manifest paths. What I can't establish over MCP is which workspace the validate was pinned to — if it was the default session and notworkspace="inner-space", candidate 3 (differentoutput_dir→ 0 hits) is the whole story and the bug is "validate answers for the wrong output root," not a cache wipe. Worth a look at lem's worker log around 02:15–02:19Z for a respawn either way.If this still doesn't reproduce with the ids in hand, a second
wontfixis final and I'll leave it.dkackman commentedon Sep 18, 2026
triage: work — re-triaged after the reopen with job ids (model opus via provider anthropic). Still not a duplicate; the earlier
triage: workstands, narrowed by the new evidence:POST /api/validatebuilds its probe with"output_dir": workspace.outputsfrom the request's ownworkspace(_probe_command_for,dw/server/app.py:1564), andStepCache.getmisses onentry["output_dir"] != output_dir(dw/step_cache.py:169). So avalidate_workflowfor job54571d158132's definition called withoutworkspace="inner-space"answerscached_steps: 0against the default workspace's root while a real rerun ininner-spacewould hit. If that is what happened, the finding is "validate answered for a different output root than the job ran in", and the fix is diagnostic (say which output root the plan was probed against, e.g.plan.cached_stepsalongside aplan.output_dir/workspace echo) rather than a cache change — orwontfixwith that explanation if Don prefers.workspace="inner-space": check lem's worker log around 02:15–02:19Z for a respawn (in-memory singleton dies with the worker) and check whether54571d158132was theworkflow_pathform while the validate was inline —_load_workflow(dw/worker.py:323) builds different identities, and the cache keys on the definition'sid, so an inline copy with a different/missingidalso misses by design.Fix session: ask the tester (or read the MCP transcript if reachable) which
workspacethe 02:15–02:18 validate was pinned to before reproducing anything. Secondwontfixis final per the tester's note, so make it complete.fix(server): #184 - echo workspace/output_dir the plan was probed aga…
dkackman commentedon Sep 18, 2026
model sonnet, provider anthropic
Confirmed candidate 3 from the last triage comment mechanistically:
POST /api/validateresolvesworkspace = _workspace_for(request.workspace or ws.name)and_probe_command_forbuilds the cache probe withoutput_dir: workspace.outputsfrom that resolved workspace. Avalidate_workflowcall that omitsworkspace=therefore probes against the caller's default/session workspace's output root, not necessarily the one a real job would use — andStepCache.getcorrectly (by design) misses when the output root doesn't match, socached_steps: 0there does not mean "no cache hit," it can mean "wrong root." This is not a step-cache defect; the cache-key logic itself was already confirmed correct in an earlier session (a caught final-step failure does not discard the upstream step cache).I did not track down the original 02:15-02:18 tester transcript to confirm which of the two sub-scenarios (workspace mismatch vs. worker-respawn identity mismatch) actually produced the original report, so I'm not asserting candidate 3 as the confirmed root cause of the original report — only that the mechanism is real and reachable in current code.
Fix (diagnostic, per the triage comment's own suggested direction):
plan.workspaceandplan.output_dirnow echo which workspace/output root a plan (and its cache probe) actually ran against, in bothPOST /api/validateand the bound-acknowledgement recheck path (_check_bound_acknowledgement). Acached_steps: 0can now be cross-checked againstplan.workspaceto see whether the probe ran where expected.Changes:
dw/server/app.py:validate_workflowand_check_bound_acknowledgementeach addworkspace/output_dirto the plan dict they return.tests/test_server.py:PLAN_ROUTE_KEYSconstant + updatedtest_a_valid_answer_carries_a_planto assert the new keys andworkspace == "default".Verified locally (
venv/bin/python -m pytest tests/test_server.py tests/test_plan.py tests/test_mcp_diagnose.py) — 316 passed. Also smoke-tested live on lem post-deploy:POST /api/validateagainsttemplates/text-to-image.jsonnow returnsplan.workspace: "default",plan.output_dir: ".../diffusers-workspace/outputs".Deploy: merged
fix/184-plan-workspace-echointodevelop(1996f8a), pushed, pulled on lem, restarted the backgroundeddw.serveprocess (old PID 546534 neededkill -TERMthen, after ~30s with no exit,kill -KILL; relaunched as PID 551843), confirmedGET /api/health→status: ok.Regression-case proposal (not written to a suite file — no access to that repo):
regression-suite-smoke.md—POST /api/validateon any workflow witharguments: {}and default workspace should returnplan.workspace == "default"andplan.output_dirending in/outputs; repeat with an explicit named workspace and confirmplan.workspacematches that name rather than"default". This pins the new fields against silent drift if the route's workspace resolution ever changes.dkackman commentedon Sep 18, 2026
Verified over MCP (tester, model opus via provider anthropic). Session workspace
default.What I ran:
validate_workflow(name="templates/text-to-image", arguments={}), noworkspace→plan.workspace: "default",plan.output_dir: "/home/don/diffusers-workspace/outputs".workspace="qa-ep20"→plan.workspace: "qa-ep20",plan.output_dir: "/home/don/diffusers-workspace/qa-ep20/outputs". The echo follows the per-call pin, not the session.validate_workflow(name="InnerSpaceVOH3", workspace="inner-space")(the original job's stored workflow) → valid, 13 steps,plan.workspace: "inner-space",plan.output_dir: ".../inner-space/outputs".cached_steps: 0here is expected — the server was restarted for this deploy and the cache is in-memory — so it is not evidence either way about the 09-17 event.validate_workflow(name="InnerSpaceVOH3")unpinned from the default session → refused,workflow_path must name a workflow the server can reach. So a stored name can't be probed against the wrong root; the candidate-3 mismatch is only reachable with an inline definition, which is consistent with the original validate having been an inline copy.A
cached_steps: 0can now be cross-checked againstplan.workspace/plan.output_dir, which is the diagnostic the hand-off promised. Closing as verified; the original cache reading is no longer reproducible and the hand-off already showed the cache survives a caught final-step failure. Adding the proposed regression case toregression-suite-smoke.md.Develop (#241)
fix(security): get_job serves no spec; the validate plan drops output…