Carry pod_template override through map_task serialization#3428
Conversation
88a1cd2 to
d9e2fd3
Compare
map_task(...).with_overrides(pod_template=PodTemplate(...)) was being
silently dropped at serialization. The override is accepted at workflow
build time and stored on the Node, but get_serializable_array_node_map_task
never read node._pod_template — TaskNodeOverrides was emitted with only
resources/extended_resources/container_image, so the registered task spec
had an empty overrides {} block.
Same class of bug as #6463 / PR flyteorg#3270 fixed for regular tasks, but in
the array-node serialization path.
The map-task path needs two implementation tweaks vs. the regular-task
mirror:
* Use entity.get_container() (not _get_container) so prepare_target()
substitutes the map-task command (pyflyte-map-execute) into the
container args before the override pod spec is built.
* For fast-serialization, prefix container args post-build (matching
get_serializable_task) instead of swapping command_fn via
_fast_serialize_command_fn — the latter wraps the inherited
pyflyte-execute default, which is wrong for map_task.
Adds a parametrized regression test covering both fast-registration
enabled and disabled.
Fixes flyteorg/flyte#7076
Signed-off-by: 1fanwang <1fannnw@gmail.com>
d9e2fd3 to
d2edd5d
Compare
pingsutw
left a comment
There was a problem hiding this comment.
Thank you for fixing it. LGTM
| @task | ||
| def t(a: int) -> str: |
There was a problem hiding this comment.
Could we also add another test that t has a podTemplate config and we use with_overrides() to override it?
@task(pod_template=base_pod_template)
def t(a: int) -> str:
return str(a)There was a problem hiding this comment.
Done in 9dde146 — added test_map_task_with_pod_template_override_replaces_task_pod_template covering the case where @task(pod_template=base_pod_template) is then overridden via map_task(t)(...).with_overrides(pod_template=override_pod_template).
While writing the test I caught a real bug: when the underlying task already has pod_template set, entity.get_container(settings) returns None (it intentionally defers to get_k8s_pod), and _serialize_pod_spec would dereference primary_container.image on None. Fixed in the same commit by falling back to the inner task's _get_container under prepare_target() so the override pod spec still resolves image/command/args. Test asserts the override's OVERRIDE_KEY env wins over the base.
…_template When the underlying @task has pod_template set, get_container() returns None. Fall back to the inner task's _get_container under prepare_target() so the override pod spec can still resolve image/command/args. Add a parametrized test exercising this scenario. Signed-off-by: 1fanwang <1fannnw@gmail.com>
9dde146 to
52a63ba
Compare
|
Congrats on merging your first pull request! 🎉 |
Tracking issue
Fixes flyteorg/flyte#7076
Why are the changes needed?
map_task(...).with_overrides(pod_template=PodTemplate(...))silently drops the override at serialization. The override is accepted at workflow build time and stored on theNode(node._pod_template), butget_serializable_array_node_map_taskinflytekit/tools/translator.pynever reads that field — it buildsTaskNodeOverrideswith onlyresources,extended_resources, andcontainer_image. The user thinks the override is applied, but the registered task spec has an emptyoverrides {}block.This is the same class of bug as #6463, which #3270 fixed for the regular-task path of
get_serializable_node. The array-node path has the same gap in a different function.What changes were proposed in this pull request?
flytekit/tools/translator.py—get_serializable_array_node_map_task:node._pod_template, build the override pod spec via_serialize_pod_spec, and includepod_templatein the emittedTaskNodeOverrides.entity.get_container(settings)(not_get_container) soprepare_target()substitutes the map-task command (pyflyte-map-execute) into the container args before the pod spec is built.ArrayNodeMapTaskextendsPythonTaskdirectly (notPythonAutoContainerTask), so_get_containerisn't on it anyway.get_serializable_taskattranslator.py:170) instead of swappingcommand_fnvia_fast_serialize_command_fn. The latter wrapstask.get_default_command(settings), which onArrayNodeMapTaskreturns the inheritedpyflyte-execute— wrong shape for a map task.tests/flytekit/unit/test_translator.py:test_map_task_with_pod_template_overridecovering bothfast_registration_enabled=TrueandFalse. Mirrors the structure of the existingtest_task_with_pod_template_overridefrom Fix issue where pod template override pod spec was missing #3270.How was this patch tested?
tests/flytekit/unit/test_translator.py(11 tests) passes.tests/flytekit/unit/core/test_array_node_map_task.py(46 tests)tests/flytekit/unit/core/test_map_task.py(23 tests)tests/flytekit/unit/core/test_node_creation.py(26 tests)Check all the applicable boxes
with_overrides(pod_template=...)formap_taskis what this PR makes work.)