Skip to content

Commit d96adb6

Browse files
lukasbindreiterlukasbindreiter
authored
Document optional subtasks (#106)
1 parent df73b60 commit d96adb6

24 files changed

Lines changed: 1449 additions & 190 deletions

‎.github/styles/config/vocabularies/docs/accept.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,4 +70,4 @@ reprojected
7070
(?i)geospatial
7171
(?i)cartesian
7272
(?i)OAuth
73-
73+
(?i)Subtrees?

‎api-reference/go/workflows/Jobs.Query.mdx‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,15 @@ The output sequence can be transformed into a slice of Job using [Collect](/api-
3333
Specify the automation id for which data should be queried.
3434
Only jobs that were created by the specified automation will be returned.
3535
</ParamField>
36+
<ParamField path="WithJobStates(states ...workflows.JobState)">
37+
Filter jobs by their state. Only jobs in any of the given states will be returned.
38+
</ParamField>
39+
<ParamField path="WithName(name string)">
40+
Filter jobs by name. Only jobs with a matching name will be returned.
41+
</ParamField>
42+
<ParamField path="WithTaskStates(states ...workflows.TaskState)">
43+
Filter jobs by the states of their tasks. Only jobs that have at least one task in any of the given states will be returned. Useful for finding jobs with [optional](/workflows/concepts/tasks#optional-tasks) task failures.
44+
</ParamField>
3645

3746
## Returns
3847

‎api-reference/go/workflows/SubmitSubtask.mdx‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ This function is intended to be used in tasks.
3636
<ParamField path="WithMaxRetries(maxRetries int64)" default="0">
3737
Set the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtask in case it fails
3838
</ParamField>
39+
<ParamField path="WithOptional()" default="false">
40+
Mark the subtask as [optional](/workflows/concepts/tasks#optional-tasks). An optional subtask will not fail the job if it fails. Tasks that depend on it will still execute even if it failed.
41+
</ParamField>
3942

4043
## Returns
4144

‎api-reference/go/workflows/SubmitSubtasks.mdx‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ This function is intended to be used in tasks.
3636
<ParamField path="WithMaxRetries(maxRetries int64)" default="0">
3737
Set the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtasks in case it fails
3838
</ParamField>
39+
<ParamField path="WithOptional()" default="false">
40+
Mark the subtasks as [optional](/workflows/concepts/tasks#optional-tasks). Optional subtasks will not fail the job if they fail. Tasks that depend on them will still execute even if they failed.
41+
</ParamField>
3942

4043
## Returns
4144

‎api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,8 @@ icon: laptop-code
88
def Client.create_or_update_dataset(
99
kind: DatasetKind,
1010
code_name: str,
11-
fields: list[FieldDict],
12-
*,
1311
name: str | None = None,
12+
custom_fields: list[FieldDict],
1413
) -> Dataset
1514
```
1615

@@ -24,12 +23,12 @@ Create a dataset.
2423
<ParamField path="code_name" type="str">
2524
The code name of the dataset
2625
</ParamField>
27-
<ParamField path="fields" type="list[FieldDict]">
28-
The fields of the dataset
29-
</ParamField>
3026
<ParamField path="name" type="str | None">
3127
The name of the dataset
3228
</ParamField>
29+
<ParamField path="custom_fields" type="list[FieldDict]">
30+
The fields of the dataset
31+
</ParamField>
3332

3433
## Dataset kinds
3534

@@ -98,9 +97,10 @@ The created dataset object.
9897
```python Python
9998
from shapely import Geometry
10099

101-
dataset = client.create_dataset(
100+
dataset = client.create_or_update_dataset(
102101
DatasetKind.SPATIOTEMPORAL,
103102
"my_catalog",
103+
"My personal catalog",
104104
[
105105
{
106106
"name": "field1",
@@ -116,8 +116,7 @@ dataset = client.create_dataset(
116116
"description": "Field 3",
117117
"example_value": "Value 3",
118118
},
119-
],
120-
name="My personal catalog",
119+
],
121120
)
122121
```
123122
</RequestExample>

‎api-reference/python/tilebox.workflows/ExecutionContext.submit_subtask.mdx‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@ def ExecutionContext.submit_subtask(
88
task: Task,
99
depends_on: list[FutureTask] = None,
1010
cluster: str | None = None,
11-
max_retries: int = 0
11+
max_retries: int = 0,
12+
optional: bool = False
1213
) -> FutureTask
1314
```
1415

@@ -32,6 +33,10 @@ Submit a [subtask](/workflows/concepts/tasks#subtasks-and-task-composition) from
3233
Specify the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtask in case of failure. The default is 0.
3334
</ParamField>
3435

36+
<ParamField path="optional" type="bool">
37+
Whether the subtask is [optional](/workflows/concepts/tasks#optional-tasks). If `True`, the subtask will not fail the job if it fails. Tasks that depend on this task will still execute even if this task failed. The default is `False`.
38+
</ParamField>
39+
3540
## Returns
3641

3742
A `FutureTask` object that represents the submitted subtask. Can be used to set up dependencies between tasks.
@@ -51,6 +56,10 @@ flaky_task = context.submit_subtask(
5156
MyFlakyTask(),
5257
max_retries=5
5358
)
59+
optional_task = context.submit_subtask(
60+
MyOptionalTask(),
61+
optional=True
62+
)
5463
```
5564
</RequestExample>
5665

‎api-reference/python/tilebox.workflows/ExecutionContext.submit_subtasks.mdx‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
---
2+
title: Context.submit_subtasks
3+
icon: folder-gear
4+
---
5+
6+
```python
7+
def ExecutionContext.submit_subtasks(
8+
tasks: Sequence[Task],
9+
depends_on: list[FutureTask] = None,
10+
cluster: str | None = None,
11+
max_retries: int = 0,
12+
optional: bool = False
13+
) -> list[FutureTask]
14+
```
15+
16+
Submit multiple [subtasks](/workflows/concepts/tasks#subtasks-and-task-composition) from a currently executing task. Same as [submit_subtask](/api-reference/python/tilebox.workflows/ExecutionContext.submit_subtask), but accepts a sequence of tasks.
17+
18+
## Parameters
19+
20+
<ParamField path="tasks" type="Sequence[Task]">
21+
The tasks to submit as subtasks.
22+
</ParamField>
23+
24+
<ParamField path="depends_on" type="list[FutureTask]">
25+
An optional list of tasks already submitted within the same context that the subtasks depend on.
26+
</ParamField>
27+
28+
<ParamField path="cluster" type="str">
29+
An optional [cluster slug](/workflows/concepts/clusters#managing-clusters) for running the subtasks. If not provided, the subtasks run on the same cluster as the parent task.
30+
</ParamField>
31+
32+
<ParamField path="max_retries" type="int">
33+
Specify the maximum number of [retries](/workflows/concepts/tasks#retry-handling) for the subtasks in case of failure. The default is 0.
34+
</ParamField>
35+
36+
<ParamField path="optional" type="bool">
37+
Whether the subtasks are [optional](/workflows/concepts/tasks#optional-tasks). If `True`, the subtasks will not fail the job if they fail. Tasks that depend on them will still execute even if they failed. The default is `False`.
38+
</ParamField>
39+
40+
## Returns
41+
42+
A list of `FutureTask` objects representing the submitted subtasks. Can be used to set up dependencies between tasks.
43+
44+
<RequestExample>
45+
```python Python
46+
# within the execute method of a Task:
47+
subtasks = context.submit_subtasks([
48+
MySubtask(i) for i in range(10)
49+
])
50+
dependent_subtask = context.submit_subtask(
51+
MyOtherSubtask(), depends_on=subtasks
52+
)
53+
```
54+
</RequestExample>

‎api-reference/python/tilebox.workflows/JobClient.query.mdx‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ icon: diagram-project
77
def JobClient.query(
88
temporal_extent: TimeIntervalLike | IDIntervalLike,
99
automation_id: UUID | None = None,
10+
job_states: JobState | list[JobState] | None = None,
11+
name: str | None = None,
12+
task_states: TaskState | list[TaskState] | None = None,
1013
) -> list[Job]
1114
```
1215

@@ -30,6 +33,15 @@ Query jobs in the specified interval.
3033
<ParamField path="automation_id" type="UUID | None">
3134
The automation id to filter jobs by. If not provided, jobs from all automations are returned.
3235
</ParamField>
36+
<ParamField path="job_states" type="JobState | list[JobState] | None">
37+
A job state or list of job states to filter by. If specified, only jobs in any of the given states are returned.
38+
</ParamField>
39+
<ParamField path="name" type="str | None">
40+
A name to filter jobs by. If specified, only jobs with a matching name are returned. The name is matched using case-insensitive fuzzy search.
41+
</ParamField>
42+
<ParamField path="task_states" type="TaskState | list[TaskState] | None">
43+
A task state or list of task states to filter jobs by. If specified, only jobs that have at least one task in any of the given states are returned. Useful for finding jobs with [optional](/workflows/concepts/tasks#optional-tasks) task failures, e.g. `task_states=TaskState.FAILED_OPTIONAL`.
44+
</ParamField>
3345

3446
## Returns
3547

‎assets/changelog/2026-02-optional-tasks.png‎

830 KB
Loading

‎assets/workflows/diagrams/optional-subtasks.d2‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
vars: {
2+
d2-config: {
3+
layout-engine: elk
4+
}
5+
}
6+
7+
title: {
8+
label: optional-subtask
9+
class: diagram-title
10+
}
11+
12+
root: {
13+
label: "RootTask"
14+
class: computed
15+
}
16+
17+
required-task: {
18+
label: "Required Task"
19+
class: computed
20+
}
21+
root -> required-task: {class: subtask-edge}
22+
23+
risky-task: {
24+
label: "Risky Task\n(optional)"
25+
class: [optional; failed]
26+
}
27+
root -> risky-task: {class: subtask-edge}
28+
29+
final-task: {
30+
label: "Final Task"
31+
class: computed
32+
}
33+
required-task <- final-task: {class: dependency-edge}
34+
risky-task <- final-task: {class: dependency-edge}
35+
root -> final-task: {class: subtask-edge}

0 commit comments

Comments
 (0)