Skip to content

Commit dbf9b4e

Browse files
authored
fix(core,cli): register tasks loaded via dynamic import during run (#3688)
On a warm worker process, a task whose `task()` definition is loaded via `await import(...)` from inside another task's `run()` could end up permanently missing from the catalog: the `task()` call fired with no `_currentFileContext` set, `registerTaskMetadata` silently returned, and Node's ESM module cache then blocked the worker's setContext + re-import recovery from ever firing the call again. Subsequent runs of that task on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the process hit `maxExecutionsPerProcess` and exited. All five of these had to coincide on the same worker for the bug to surface: 1. `processKeepAlive` enabled (so catalog state survives across runs). 2. A `run()` function (or lifecycle hook) does `await import(...)`. 3. The import's transitive static graph reaches a `task()` / `schemaTask()` call. 4. The task containing the dynamic import is the **first** task to run on a given warm worker process — so the dropped `task()` calls fire on this process for the first time, are silently dropped, and Node's module cache locks the wrong outcome in. 5. A subsequent run for one of the dropped task ids lands on the same warm worker before it recycles. The runtime workers now set a sentinel file context (`<no-context>`) around the `executor.execute(...)` call, so `task()` invocations firing during a run register normally. The catalog detects the sentinel and emits a one-time `console.warn` per task id so the pattern stays visible without spamming. The indexer never sets this context, so deploy-time behavior is unchanged. Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`. Verified end-to-end against a deployed image with firestarter warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded and the warning surfaces in the parent's run trace.
1 parent acfba02 commit dbf9b4e

9 files changed

Lines changed: 259 additions & 5 deletions

File tree

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
"@trigger.dev/core": patch
3+
"trigger.dev": patch
4+
---
5+
6+
Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via `await import(...)` from inside another task's `run()`. The runtime workers now register such tasks with a sentinel file context, and the catalog logs a one-time warning per task id.

packages/cli-v3/src/entryPoints/dev-run-worker.ts

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ import {
4747
SharedRuntimeManager,
4848
OtelTaskLogger,
4949
populateEnv,
50+
NO_FILE_CONTEXT,
5051
StandardLifecycleHooksManager,
5152
StandardLocalsManager,
5253
StandardMetadataManager,
@@ -501,8 +502,8 @@ const zodIpc = new ZodIpcConnection({
501502
async () => {
502503
const beforeImport = performance.now();
503504
resourceCatalog.setCurrentFileContext(
504-
taskManifest.entryPoint,
505-
taskManifest.filePath
505+
taskManifest.filePath,
506+
taskManifest.entryPoint
506507
);
507508

508509
// Load init file if it exists
@@ -610,6 +611,12 @@ const zodIpc = new ZodIpcConnection({
610611

611612
const signal = AbortSignal.any([_cancelController.signal, timeoutController.signal]);
612613

614+
// Sentinel context so `task()` calls firing during run / lifecycle
615+
// hooks (e.g. via `await import(...)` of a module containing a task
616+
// definition) register normally instead of being silently dropped.
617+
// Cleared in the surrounding finally below.
618+
resourceCatalog.setCurrentFileContext(NO_FILE_CONTEXT, NO_FILE_CONTEXT);
619+
613620
const { result } = await executor.execute(execution, ctx, signal);
614621

615622
if (_isRunning && !_isCancelled) {
@@ -628,6 +635,7 @@ const zodIpc = new ZodIpcConnection({
628635
}
629636
} finally {
630637
standardHeartbeatsManager.stopHeartbeat();
638+
resourceCatalog.clearCurrentFileContext();
631639

632640
_execution = undefined;
633641
_isRunning = false;

packages/cli-v3/src/entryPoints/managed-run-worker.ts

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ import {
4747
OtelTaskLogger,
4848
populateEnv,
4949
ProdUsageManager,
50+
NO_FILE_CONTEXT,
5051
StandardLifecycleHooksManager,
5152
StandardLocalsManager,
5253
StandardMetadataManager,
@@ -490,8 +491,8 @@ const zodIpc = new ZodIpcConnection({
490491
async () => {
491492
const beforeImport = performance.now();
492493
resourceCatalog.setCurrentFileContext(
493-
taskManifest.entryPoint,
494-
taskManifest.filePath
494+
taskManifest.filePath,
495+
taskManifest.entryPoint
495496
);
496497

497498
// Load init file if it exists
@@ -595,6 +596,12 @@ const zodIpc = new ZodIpcConnection({
595596

596597
const signal = AbortSignal.any([_cancelController.signal, timeoutController.signal]);
597598

599+
// Sentinel context so `task()` calls firing during run / lifecycle
600+
// hooks (e.g. via `await import(...)` of a module containing a task
601+
// definition) register normally instead of being silently dropped.
602+
// Cleared in the surrounding finally below.
603+
resourceCatalog.setCurrentFileContext(NO_FILE_CONTEXT, NO_FILE_CONTEXT);
604+
598605
const { result } = await executor.execute(execution, ctx, signal);
599606

600607
if (_isRunning && !_isCancelled) {
@@ -613,6 +620,7 @@ const zodIpc = new ZodIpcConnection({
613620
}
614621
} finally {
615622
standardHeartbeatsManager.stopHeartbeat();
623+
resourceCatalog.clearCurrentFileContext();
616624

617625
_execution = undefined;
618626
_isRunning = false;

packages/core/src/v3/resource-catalog/standardResourceCatalog.ts

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,18 @@ import {
1212
import { PromptMetadataWithFunctions, TaskMetadataWithFunctions, TaskSchema } from "../types/index.js";
1313
import { ResourceCatalog } from "./catalog.js";
1414

15+
/**
16+
* Sentinel file-context value the runtime workers set around task execution
17+
* (via `TaskExecutor.execute`) so that `task()` calls firing during a run —
18+
* e.g. as a side effect of `await import(...)` of a module containing a
19+
* task definition — register normally instead of hitting the silent-drop
20+
* guard in `registerTaskMetadata`. The catalog uses this exact string to
21+
* detect "registered during execution" and emit a one-time warning per
22+
* task id. The indexer never sets this context, so its behavior is
23+
* unchanged.
24+
*/
25+
export const NO_FILE_CONTEXT = "<no-context>";
26+
1527
export class StandardResourceCatalog implements ResourceCatalog {
1628
private _taskSchemas: Map<string, TaskSchema> = new Map();
1729
private _taskMetadata: Map<string, TaskMetadata> = new Map();
@@ -25,6 +37,7 @@ export class StandardResourceCatalog implements ResourceCatalog {
2537
private _queueMetadata: Map<string, QueueManifest> = new Map();
2638
private _skillMetadata: Map<string, SkillMetadata> = new Map();
2739
private _skillFileMetadata: Map<string, TaskFileMetadata> = new Map();
40+
private _sentinelContextWarned: Set<string> = new Set();
2841

2942
setCurrentFileContext(filePath: string, entryPoint: string) {
3043
this._currentFileContext = { filePath, entryPoint };
@@ -77,6 +90,20 @@ export class StandardResourceCatalog implements ResourceCatalog {
7790
return;
7891
}
7992

93+
// When the current context is the sentinel set by TaskExecutor around a
94+
// run, the task() call fired during execution — most commonly via a
95+
// dynamic import inside another task's run(). Warn once per task id so
96+
// the pattern stays visible.
97+
if (
98+
this._currentFileContext.filePath === NO_FILE_CONTEXT &&
99+
!this._sentinelContextWarned.has(task.id)
100+
) {
101+
this._sentinelContextWarned.add(task.id);
102+
console.warn(
103+
`[trigger.dev] task "${task.id}" was registered via dynamic import during another task's run(); move to a static import if you notice any issues.`
104+
);
105+
}
106+
80107
this._taskFileMetadata.set(task.id, {
81108
...this._currentFileContext,
82109
});

packages/core/src/v3/workers/index.ts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,10 @@ export {
1010
recordSpanException,
1111
carrierFromContext,
1212
} from "../otel/index.js";
13-
export { StandardResourceCatalog } from "../resource-catalog/standardResourceCatalog.js";
13+
export {
14+
StandardResourceCatalog,
15+
NO_FILE_CONTEXT,
16+
} from "../resource-catalog/standardResourceCatalog.js";
1417
export {
1518
TaskContextSpanProcessor,
1619
TaskContextLogProcessor,
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
// Fixture mimicking a task entrypoint file: top-level code calls into the
2+
// catalog (the same way `task()` / `schemaTask()` does via
3+
// `registerTaskMetadata`).
4+
//
5+
// Loaded via `await import()` from inside a test that simulates the worker
6+
// running a task. The point is to exercise top-level evaluation through Node's
7+
// ESM module loader so the module-cache semantics are real.
8+
9+
const register = globalThis.__catalogRegisterTaskMetadata;
10+
if (typeof register === "function") {
11+
register({
12+
id: "lazy-task",
13+
fns: {
14+
run: async () => "ok",
15+
},
16+
});
17+
}
18+
19+
export const lazyTask = { id: "lazy-task" };
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
import { afterEach, describe, expect, it, vi } from "vitest";
2+
import {
3+
NO_FILE_CONTEXT,
4+
StandardResourceCatalog,
5+
} from "../src/v3/resource-catalog/standardResourceCatalog.js";
6+
7+
// Regression tests for COULD_NOT_FIND_EXECUTOR on warm worker processes when
8+
// a task's `task()` / `schemaTask()` call is evaluated during another task's
9+
// execution (e.g. as a side effect of `await import(...)` of a module that
10+
// contains a task definition).
11+
//
12+
// Production throw site:
13+
// - managed-run-worker.ts:566 (post-wrap)
14+
// - dev-run-worker.ts:578 (post-wrap)
15+
// Pre-fix symptom: `resourceCatalog.getTask(execution.task.id)` returned
16+
// undefined even after the worker re-imported the task entrypoint.
17+
//
18+
// Pre-fix mechanism: `registerTaskMetadata` silently returned when
19+
// `_currentFileContext` was unset. Any `task()` call firing during a
20+
// running task's run() / lifecycle hooks (directly, or transitively via a
21+
// dynamic import) hit the silent guard. Node's ESM module cache then
22+
// prevented recovery — the worker's setContext + re-import fallback didn't
23+
// re-evaluate the module body, so the `task()` call never fired again.
24+
//
25+
// Post-fix: the runtime workers wrap their `executor.execute(...)` call with
26+
// `setCurrentFileContext(NO_FILE_CONTEXT, NO_FILE_CONTEXT)` so any `task()`
27+
// call firing during execution registers normally with sentinel file
28+
// metadata. The catalog detects the sentinel and emits a one-time warning
29+
// per task id to keep the bundle-shape pattern visible. The indexer never
30+
// sets this sentinel context — its behavior is unchanged.
31+
32+
describe("StandardResourceCatalog — runtime registration via sentinel context", () => {
33+
afterEach(() => {
34+
delete (globalThis as { __catalogRegisterTaskMetadata?: unknown })
35+
.__catalogRegisterTaskMetadata;
36+
vi.restoreAllMocks();
37+
});
38+
39+
it("silently drops registration when no context is set (indexer's invariant)", () => {
40+
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
41+
const catalog = new StandardResourceCatalog();
42+
43+
catalog.registerTaskMetadata({
44+
id: "no-context-task",
45+
fns: { run: async () => "ok" },
46+
});
47+
48+
expect(catalog.getTask("no-context-task")).toBeUndefined();
49+
expect(warn).not.toHaveBeenCalled();
50+
});
51+
52+
it(
53+
"registers normally and warns once when the sentinel context is set " +
54+
"(simulates the worker's executor wrap)",
55+
() => {
56+
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
57+
const catalog = new StandardResourceCatalog();
58+
59+
catalog.setCurrentFileContext(NO_FILE_CONTEXT, NO_FILE_CONTEXT);
60+
catalog.registerTaskMetadata({
61+
id: "lazy-task",
62+
fns: { run: async () => "ok" },
63+
});
64+
catalog.clearCurrentFileContext();
65+
66+
const registered = catalog.getTask("lazy-task");
67+
expect(registered).toBeDefined();
68+
expect(registered?.id).toBe("lazy-task");
69+
expect(registered?.filePath).toBe(NO_FILE_CONTEXT);
70+
expect(registered?.entryPoint).toBe(NO_FILE_CONTEXT);
71+
expect(warn).toHaveBeenCalledTimes(1);
72+
expect(warn.mock.calls[0]?.[0]).toContain("lazy-task");
73+
}
74+
);
75+
76+
it(
77+
"warm-start path: a task whose top-level definition fires during a " +
78+
"dynamic import inside the sentinel wrap remains findable; the " +
79+
"worker's setContext + re-import fallback (managed-run-worker.ts:482) " +
80+
"is not needed",
81+
async () => {
82+
vi.spyOn(console, "warn").mockImplementation(() => {});
83+
const catalog = new StandardResourceCatalog();
84+
85+
(globalThis as { __catalogRegisterTaskMetadata?: unknown })
86+
.__catalogRegisterTaskMetadata = (
87+
task: Parameters<StandardResourceCatalog["registerTaskMetadata"]>[0]
88+
) => {
89+
catalog.registerTaskMetadata(task);
90+
};
91+
92+
// Simulate the worker wrap: setContext(NO_FILE_CONTEXT) → run user code
93+
// (which does a dynamic import) → clearContext.
94+
catalog.setCurrentFileContext(NO_FILE_CONTEXT, NO_FILE_CONTEXT);
95+
await import("./fixtures/dynamic-task-module.mjs");
96+
catalog.clearCurrentFileContext();
97+
98+
const registered = catalog.getTask("lazy-task");
99+
expect(registered).toBeDefined();
100+
expect(registered?.filePath).toBe(NO_FILE_CONTEXT);
101+
}
102+
);
103+
104+
it("warns at most once per task id under the sentinel context", () => {
105+
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
106+
const catalog = new StandardResourceCatalog();
107+
108+
catalog.setCurrentFileContext(NO_FILE_CONTEXT, NO_FILE_CONTEXT);
109+
110+
const register = (id: string) =>
111+
catalog.registerTaskMetadata({
112+
id,
113+
fns: { run: async () => "ok" },
114+
});
115+
116+
register("task-a");
117+
register("task-a");
118+
register("task-a");
119+
expect(warn).toHaveBeenCalledTimes(1);
120+
121+
register("task-b");
122+
expect(warn).toHaveBeenCalledTimes(2);
123+
124+
catalog.clearCurrentFileContext();
125+
});
126+
127+
it(
128+
"control: real file context registers without firing the sentinel warning",
129+
async () => {
130+
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
131+
const catalog = new StandardResourceCatalog();
132+
133+
(globalThis as { __catalogRegisterTaskMetadata?: unknown })
134+
.__catalogRegisterTaskMetadata = (
135+
task: Parameters<StandardResourceCatalog["registerTaskMetadata"]>[0]
136+
) => {
137+
catalog.registerTaskMetadata(task);
138+
};
139+
140+
catalog.setCurrentFileContext(
141+
"/app/dist/lazy-task.entry.mjs",
142+
"src/tasks/lazy-task.ts"
143+
);
144+
await import("./fixtures/dynamic-task-module.mjs?control");
145+
catalog.clearCurrentFileContext();
146+
147+
const task = catalog.getTask("lazy-task");
148+
expect(task).toBeDefined();
149+
expect(task?.filePath).toBe("/app/dist/lazy-task.entry.mjs");
150+
expect(task?.entryPoint).toBe("src/tasks/lazy-task.ts");
151+
expect(warn).not.toHaveBeenCalled();
152+
}
153+
);
154+
});
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import { task } from "@trigger.dev/sdk";
2+
3+
// Defined in a module that's loaded via `await import(...)` from the parent
4+
// task's run() function. Pre-fix: the task() call below fires while
5+
// `_currentFileContext` is unset, so the registration is silently dropped.
6+
// Post-fix: registered with sentinel file metadata + console.warn fires once.
7+
export const lazyChildTask = task({
8+
id: "lazy-child-task",
9+
run: async (payload: { value: string }) => {
10+
return { received: payload.value };
11+
},
12+
});
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
import { logger, task } from "@trigger.dev/sdk";
2+
3+
// Triggers the dynamic-import silent-drop path. The child task's `task()`
4+
// definition lives in a module loaded via `await import(...)` inside this
5+
// parent's run() — so its registration would land outside the worker's
6+
// cold-load context window.
7+
export const dynamicImportReproParent = task({
8+
id: "dynamic-import-repro-parent",
9+
run: async () => {
10+
logger.info("parent: about to dynamically import child task module");
11+
const { lazyChildTask } = await import("./dynamicImportReproChild.js");
12+
logger.info("parent: import complete, triggering child");
13+
const handle = await lazyChildTask.trigger({ value: "hello from parent" });
14+
logger.info("parent: child triggered", { childRunId: handle.id });
15+
return { childRunId: handle.id };
16+
},
17+
});

0 commit comments

Comments
 (0)