Search before asking
What happened
When a running pipeline is interrupted mid-subtask (host/Docker crash or restart) and DevLake restarts with RESUME_PIPELINES=true, the interrupted subtask is correctly re-queued and resumes — but a GraphQL collector subtask (e.g. Collect Job Runs in the github_graphql plugin) restarts pagination from the very first page, discarding all progress made before the crash.
Observed on my instance: Collect Job Runs had progressDetail.finishedRecords: 990 before the container restarted. After restart, the same subtask resumed but finishedRecords reset to 1 and began re-fetching from page 1.
What do you expect to happen
Either:
- the collector persists the last-completed page's cursor (per subtask/params) so a resumed run continues from there instead of re-fetching everything, or
- if that's out of scope, the docs/UI make it clear that GraphQL collector subtasks are not crash-resumable, so a mid-collection crash is known to mean starting that subtask over (as opposed to
RESUME_PIPELINES implying full resumability).
How to reproduce
- Start a Blueprint that includes a
github_graphql collect task on a repo with a large amount of CI history (many workflow/job runs), ideally with fullSync: true and a timeAfter far enough back that the subtask runs for several minutes.
- While
Collect Job Runs (or any other GraphQL-paginated subtask) is mid-flight, restart the devlake container (e.g. docker compose restart devlake, or in my case the whole Docker runtime restarted).
- With
RESUME_PIPELINES=true (default), the pipeline auto-resumes the pending task.
- Observe
progressDetail.finishedRecords for that subtask resets to a low number instead of continuing from where it left off.
Anything else
Root cause (from reading the source): SubtaskStateManager (backend/helpers/pluginhelper/api/subtask_state_manager.go) only persists subtask-level state (PrevStartedAt, TimeAfter) to _devlake_subtask_states. The GraphQL pagination cursor (GraphqlRequestData.Pager.SkipCursor, built inside InitGraphQLCollector's BuildQuery closure) lives only in process memory and is never written to the database, so while the subtask itself resumes, the page position inside it does not.
This happens every time regardless of repo, as long as the crash lands mid-subtask on a GraphQL-paginated collector with more than one page of data.
Version
v1.0.3-beta17
Are you willing to submit PR?
Code of Conduct
Search before asking
What happened
When a running pipeline is interrupted mid-subtask (host/Docker crash or restart) and DevLake restarts with
RESUME_PIPELINES=true, the interrupted subtask is correctly re-queued and resumes — but a GraphQL collector subtask (e.g.Collect Job Runsin thegithub_graphqlplugin) restarts pagination from the very first page, discarding all progress made before the crash.Observed on my instance:
Collect Job RunshadprogressDetail.finishedRecords: 990before the container restarted. After restart, the same subtask resumed butfinishedRecordsreset to1and began re-fetching from page 1.What do you expect to happen
Either:
RESUME_PIPELINESimplying full resumability).How to reproduce
github_graphqlcollect task on a repo with a large amount of CI history (many workflow/job runs), ideally withfullSync: trueand atimeAfterfar enough back that the subtask runs for several minutes.Collect Job Runs(or any other GraphQL-paginated subtask) is mid-flight, restart thedevlakecontainer (e.g.docker compose restart devlake, or in my case the whole Docker runtime restarted).RESUME_PIPELINES=true(default), the pipeline auto-resumes the pending task.progressDetail.finishedRecordsfor that subtask resets to a low number instead of continuing from where it left off.Anything else
Root cause (from reading the source):
SubtaskStateManager(backend/helpers/pluginhelper/api/subtask_state_manager.go) only persists subtask-level state (PrevStartedAt,TimeAfter) to_devlake_subtask_states. The GraphQL pagination cursor (GraphqlRequestData.Pager.SkipCursor, built insideInitGraphQLCollector'sBuildQueryclosure) lives only in process memory and is never written to the database, so while the subtask itself resumes, the page position inside it does not.This happens every time regardless of repo, as long as the crash lands mid-subtask on a GraphQL-paginated collector with more than one page of data.
Version
v1.0.3-beta17
Are you willing to submit PR?
Code of Conduct