diff --git a/.changeset/llm-stream-stall-watchdog.md b/.changeset/llm-stream-stall-watchdog.md new file mode 100644 index 0000000000..cb2281ba8a --- /dev/null +++ b/.changeset/llm-stream-stall-watchdog.md @@ -0,0 +1,5 @@ +--- +"@moonshot-ai/kimi-code": patch +--- + +Fix model requests hanging indefinitely when a provider response stream stalls; stalled requests are now detected and retried automatically with a bounded budget. Retrying a failed step no longer leaves the previous attempt's partial output stuck as a running step in the transcript. diff --git a/docs/en/configuration/config-files.md b/docs/en/configuration/config-files.md index e8a969fbbc..327c970532 100644 --- a/docs/en/configuration/config-files.md +++ b/docs/en/configuration/config-files.md @@ -283,17 +283,20 @@ Configuration errors fail loudly instead of falling back silently: session creat ## `loop_control` -`loop_control` governs the step count limit, the per-step attempt limit, and the threshold that triggers automatic context compaction in the Agent execution loop. +`loop_control` governs the step count limit, the per-step attempt limits, the stall detection timeouts for model requests, and the threshold that triggers automatic context compaction in the Agent execution loop. | Field | Type | Default | Description | | --- | --- | --- | --- | | `max_steps_per_turn` | `integer` | — | Maximum steps per turn; unset or `0` means unlimited | | `max_attempts_per_step` | `integer` | `10` | Maximum total attempts for a failing step, including the initial attempt | +| `first_output_timeout_ms` | `integer` | `180000` | Maximum time in milliseconds to wait for the first output of a model request; a request that still has no output past this limit is treated as stalled and retried. `0` disables the check | +| `stream_idle_timeout_ms` | `integer` | `120000` | Maximum time in milliseconds a streaming model response may produce no new output before it is treated as stalled and retried. `0` disables the check | +| `max_stall_attempts_per_step` | `integer` | `3` | Maximum total attempts for a step whose model request keeps stalling, including the initial attempt | | `reserved_context_size` | `integer` | — | Number of tokens reserved for model output; automatic compaction is triggered when the remaining context window falls below this value | -`max_steps_per_turn` can be overridden by the `KIMI_LOOP_MAX_STEPS_PER_TURN` environment variable, and `max_attempts_per_step` by `KIMI_LOOP_MAX_ATTEMPTS_PER_STEP`; both take higher priority than the config file. The former `KIMI_LOOP_MAX_RETRIES_PER_STEP` variable is deprecated but still honored (with a startup warning) when the new one is unset. +`max_steps_per_turn` can be overridden by the `KIMI_LOOP_MAX_STEPS_PER_TURN` environment variable, and `max_attempts_per_step` by `KIMI_LOOP_MAX_ATTEMPTS_PER_STEP`; both take higher priority than the config file. The same applies to `first_output_timeout_ms` (`KIMI_LOOP_FIRST_OUTPUT_TIMEOUT_MS`), `stream_idle_timeout_ms` (`KIMI_LOOP_STREAM_IDLE_TIMEOUT_MS`), and `max_stall_attempts_per_step` (`KIMI_LOOP_MAX_STALL_ATTEMPTS_PER_STEP`). The former `KIMI_LOOP_MAX_RETRIES_PER_STEP` variable is deprecated but still honored (with a startup warning) when the new one is unset. -Retries only apply to transient failures — connection errors, timeouts, HTTP 429 rate limits, and 5xx server errors. A 429 caused by an exhausted quota or insufficient account balance is not retried and fails immediately, since it cannot succeed until the account is recharged. +Retries only apply to transient failures — connection errors, timeouts, HTTP 429 rate limits, and 5xx server errors. A 429 caused by an exhausted quota or insufficient account balance is not retried and fails immediately, since it cannot succeed until the account is recharged. A model request that stops making progress — no first output within `first_output_timeout_ms`, or a stream that stays idle beyond `stream_idle_timeout_ms` — fails as a stall and is retried with the separate `max_stall_attempts_per_step` budget instead of `max_attempts_per_step`. ## `token_counting` diff --git a/docs/en/configuration/env-vars.md b/docs/en/configuration/env-vars.md index fb3fb86ee3..603fd8b979 100644 --- a/docs/en/configuration/env-vars.md +++ b/docs/en/configuration/env-vars.md @@ -138,6 +138,9 @@ Switches that control the behavior of subsystems such as telemetry, background t | `KIMI_MCP_TOOL_TIMEOUT_MS` | Global default single tool-call timeout (ms) for all MCP servers; takes higher priority than `[mcp] tool_timeout_ms` in `config.toml`, but a per-server `toolTimeoutMs` in `mcp.json` still wins (default `60000`) | Integer from `1` to `2147483647`; invalid values are ignored | | `KIMI_LOOP_MAX_STEPS_PER_TURN` | Maximum Agent steps per turn; takes higher priority than `[loop_control] max_steps_per_turn` in `config.toml` (unset or `0` means unlimited) | Non-negative integer; invalid values are ignored | | `KIMI_LOOP_MAX_ATTEMPTS_PER_STEP` | Maximum total attempts for a failing step (including the initial attempt); takes higher priority than `[loop_control] max_attempts_per_step` in `config.toml` (default `10`). The deprecated `KIMI_LOOP_MAX_RETRIES_PER_STEP` is still honored with a warning when this variable is unset | Non-negative integer; invalid values are ignored | +| `KIMI_LOOP_FIRST_OUTPUT_TIMEOUT_MS` | Maximum time in milliseconds to wait for the first output of a model request before treating it as stalled; takes higher priority than `[loop_control] first_output_timeout_ms` in `config.toml` (default `180000`; `0` disables the check) | Non-negative integer; invalid values are ignored | +| `KIMI_LOOP_STREAM_IDLE_TIMEOUT_MS` | Maximum time in milliseconds a streaming model response may produce no new output before being treated as stalled; takes higher priority than `[loop_control] stream_idle_timeout_ms` in `config.toml` (default `120000`; `0` disables the check) | Non-negative integer; invalid values are ignored | +| `KIMI_LOOP_MAX_STALL_ATTEMPTS_PER_STEP` | Maximum total attempts for a step whose model request keeps stalling (including the initial attempt); takes higher priority than `[loop_control] max_stall_attempts_per_step` in `config.toml` (default `3`) | Non-negative integer; invalid values are ignored | | `KIMI_TOKEN_COUNTING_STRATEGY` | Which context token count is reported externally (the context-size display); takes higher priority than `[token_counting] strategy` in `config.toml` (default `measured+estimated`) | `measured+estimated`, `measured`, `estimated` (case-insensitive); invalid values are ignored | | `KIMI_WEB_SEARCH_BASE_URL` | API URL of the web search (`WebSearch`) service; takes higher priority than `[services.moonshot_search] base_url` in `config.toml`, and enables the service without that config section. Persisted credentials and custom headers are not forwarded to an env-selected endpoint | Non-blank string; blank values are ignored | | `KIMI_WEB_SEARCH_API_KEY` | API key of the web search (`WebSearch`) service; replaces both the configured API key and OAuth credential when set | Non-blank string; blank values are ignored | diff --git a/docs/zh/configuration/config-files.md b/docs/zh/configuration/config-files.md index f272fb3a48..c2e51cd82d 100644 --- a/docs/zh/configuration/config-files.md +++ b/docs/zh/configuration/config-files.md @@ -282,17 +282,20 @@ k3-max = "同一模型的 max Thinking 档位。适合最难的子任务。" ## `loop_control` -`loop_control` 控制 Agent 执行循环的步数上限、单步尝试次数上限,以及触发上下文自动压缩的阈值。 +`loop_control` 控制 Agent 执行循环的步数上限、单步尝试次数上限、模型请求的停滞检测超时,以及触发上下文自动压缩的阈值。 | 字段 | 类型 | 默认值 | 说明 | | --- | --- | --- | --- | | `max_steps_per_turn` | `integer` | — | 单轮最大步数;不设或设为 `0` 则无上限 | | `max_attempts_per_step` | `integer` | `10` | 单步失败后的最大总尝试次数(含首次尝试) | +| `first_output_timeout_ms` | `integer` | `180000` | 等待模型请求首个输出的最长时间(毫秒);超过此时间仍无任何输出的请求会被判定为停滞并重试。设为 `0` 关闭该检测 | +| `stream_idle_timeout_ms` | `integer` | `120000` | 流式模型响应允许没有任何新输出的最长时间(毫秒),超时即判定为停滞并重试。设为 `0` 关闭该检测 | +| `max_stall_attempts_per_step` | `integer` | `3` | 模型请求持续停滞时单步的最大总尝试次数(含首次尝试) | | `reserved_context_size` | `integer` | — | 预留给模型输出的 token 数;上下文窗口剩余量低于此值时触发自动压缩 | -`max_steps_per_turn` 可被环境变量 `KIMI_LOOP_MAX_STEPS_PER_TURN` 覆盖,`max_attempts_per_step` 可被 `KIMI_LOOP_MAX_ATTEMPTS_PER_STEP` 覆盖,优先级均高于配置文件。旧的 `KIMI_LOOP_MAX_RETRIES_PER_STEP` 已废弃,但在新变量未设置时仍生效(启动时会给出警告)。 +`max_steps_per_turn` 可被环境变量 `KIMI_LOOP_MAX_STEPS_PER_TURN` 覆盖,`max_attempts_per_step` 可被 `KIMI_LOOP_MAX_ATTEMPTS_PER_STEP` 覆盖,优先级均高于配置文件。`first_output_timeout_ms`、`stream_idle_timeout_ms`、`max_stall_attempts_per_step` 同理,分别对应 `KIMI_LOOP_FIRST_OUTPUT_TIMEOUT_MS`、`KIMI_LOOP_STREAM_IDLE_TIMEOUT_MS`、`KIMI_LOOP_MAX_STALL_ATTEMPTS_PER_STEP`。旧的 `KIMI_LOOP_MAX_RETRIES_PER_STEP` 已废弃,但在新变量未设置时仍生效(启动时会给出警告)。 -重试仅针对瞬时故障——连接错误、超时、HTTP 429 限流和 5xx 服务端错误。账户额度耗尽或余额不足导致的 429 不会重试,会立即失败:在充值之前重试不可能成功。 +重试仅针对瞬时故障——连接错误、超时、HTTP 429 限流和 5xx 服务端错误。账户额度耗尽或余额不足导致的 429 不会重试,会立即失败:在充值之前重试不可能成功。模型请求停止推进——超过 `first_output_timeout_ms` 仍无首个输出,或流式响应超过 `stream_idle_timeout_ms` 没有新输出——会按停滞失败,并使用独立的 `max_stall_attempts_per_step` 预算重试,而不是 `max_attempts_per_step`。 ## `token_counting` diff --git a/docs/zh/configuration/env-vars.md b/docs/zh/configuration/env-vars.md index a31507fe25..14828c3e42 100644 --- a/docs/zh/configuration/env-vars.md +++ b/docs/zh/configuration/env-vars.md @@ -138,6 +138,9 @@ kimi | `KIMI_MCP_TOOL_TIMEOUT_MS` | 所有 MCP server 的全局默认单次工具调用超时(毫秒);优先级高于 `config.toml` 的 `[mcp] tool_timeout_ms`,但低于 `mcp.json` 中单个 server 的 `toolTimeoutMs`(默认 `60000`) | `1` 到 `2147483647` 的整数;非法值被忽略 | | `KIMI_LOOP_MAX_STEPS_PER_TURN` | Agent 单轮最大步数;优先级高于 `config.toml` 的 `[loop_control] max_steps_per_turn`(不设或 `0` 表示无上限) | 非负整数;非法值被忽略 | | `KIMI_LOOP_MAX_ATTEMPTS_PER_STEP` | 单步失败后的最大总尝试次数(含首次尝试);优先级高于 `config.toml` 的 `[loop_control] max_attempts_per_step`(默认 `10`)。旧的 `KIMI_LOOP_MAX_RETRIES_PER_STEP` 已废弃,但在本变量未设置时仍生效并给出警告 | 非负整数;非法值被忽略 | +| `KIMI_LOOP_FIRST_OUTPUT_TIMEOUT_MS` | 等待模型请求首个输出的最长时间(毫秒),超时即判定为停滞;优先级高于 `config.toml` 的 `[loop_control] first_output_timeout_ms`(默认 `180000`;`0` 表示关闭该检测) | 非负整数;非法值被忽略 | +| `KIMI_LOOP_STREAM_IDLE_TIMEOUT_MS` | 流式模型响应允许没有任何新输出的最长时间(毫秒),超时即判定为停滞;优先级高于 `config.toml` 的 `[loop_control] stream_idle_timeout_ms`(默认 `120000`;`0` 表示关闭该检测) | 非负整数;非法值被忽略 | +| `KIMI_LOOP_MAX_STALL_ATTEMPTS_PER_STEP` | 模型请求持续停滞时单步的最大总尝试次数(含首次尝试);优先级高于 `config.toml` 的 `[loop_control] max_stall_attempts_per_step`(默认 `3`) | 非负整数;非法值被忽略 | | `KIMI_TOKEN_COUNTING_STRATEGY` | 对外上报的上下文 token 计数(上下文大小显示);优先级高于 `config.toml` 的 `[token_counting] strategy`(默认 `measured+estimated`) | `measured+estimated`、`measured`、`estimated`(不区分大小写);非法值被忽略 | | `KIMI_WEB_SEARCH_BASE_URL` | 网页搜索(`WebSearch`)服务的 API URL;优先级高于 `config.toml` 的 `[services.moonshot_search] base_url`,未写配置段时也可启用服务。文件中持久化的凭据和自定义 header 不会发送到环境变量指定的端点 | 非空字符串;空白值被忽略 | | `KIMI_WEB_SEARCH_API_KEY` | 网页搜索(`WebSearch`)服务的 API 密钥;设置后同时替换配置中的 API 密钥和 OAuth 凭据 | 非空字符串;空白值被忽略 | diff --git a/packages/agent-core-v2/docs/config-manifest.toml b/packages/agent-core-v2/docs/config-manifest.toml index 99b6a14c4b..ecb480c2a6 100644 --- a/packages/agent-core-v2/docs/config-manifest.toml +++ b/packages/agent-core-v2/docs/config-manifest.toml @@ -186,6 +186,9 @@ extra_skill_dirs = [] # env: # max_steps_per_turn <- KIMI_LOOP_MAX_STEPS_PER_TURN (custom parse) # max_attempts_per_step <- KIMI_LOOP_MAX_ATTEMPTS_PER_STEP (custom parse; deprecated fallback KIMI_LOOP_MAX_RETRIES_PER_STEP) +# first_output_timeout_ms <- KIMI_LOOP_FIRST_OUTPUT_TIMEOUT_MS (custom parse) +# stream_idle_timeout_ms <- KIMI_LOOP_STREAM_IDLE_TIMEOUT_MS (custom parse) +# max_stall_attempts_per_step <- KIMI_LOOP_MAX_STALL_ATTEMPTS_PER_STEP (custom parse) # ########################################################################## [loop_control] @@ -194,6 +197,9 @@ extra_skill_dirs = [] # max_ralph_iterations: integer # reserved_context_size: integer # compaction_trigger_ratio: number +# first_output_timeout_ms: integer +# stream_idle_timeout_ms: integer +# max_stall_attempts_per_step: integer # ########################################################################## # mcp diff --git a/packages/agent-core-v2/docs/state-manifest.d.ts b/packages/agent-core-v2/docs/state-manifest.d.ts index 9cefad33ca..2dcb32d751 100644 --- a/packages/agent-core-v2/docs/state-manifest.d.ts +++ b/packages/agent-core-v2/docs/state-manifest.d.ts @@ -27,7 +27,7 @@ // references become '(circular)', and class instances collapse to a '(ClassName)' // marker — the wire shape of an entry is the JSON projection of the type here. // -// Index (App: 0 keys · Workspace: 6 keys · Session: 18 keys · Agent: 98 keys) +// Index (App: 0 keys · Workspace: 6 keys · Session: 18 keys · Agent: 99 keys) // App // Workspace // workspaceDirs.ephemeralDirs src/workspace/workspaceDirs/workspaceDirsService.ts @@ -127,6 +127,7 @@ // shellCommand.tasks src/agent/shellCommand/shellCommandService.ts // skill src/agent/skill/skillOps.ts // stepRetry.failedAttempts src/agent/stepRetry/stepRetryService.ts +// stepRetry.failedStallAttempts src/agent/stepRetry/stepRetryService.ts // stepRetry.lastFailedDriverId src/agent/stepRetry/stepRetryService.ts // swarm src/features/swarm/swarmOps.ts // task src/agent/task/taskOps.ts @@ -1286,6 +1287,8 @@ export interface AgentStateSnapshot { readonly usedContextTokens?: number; readonly maxContextTokens?: number; readonly onTraceId?: (traceId: string | null) => void; + readonly firstOutputTimeoutMs?: number; + readonly streamIdleTimeoutMs?: number; }; readonly systemPrompt: string; }>; @@ -1422,6 +1425,7 @@ export interface AgentStateSnapshot { 'skill': null; // src/agent/stepRetry/stepRetryService.ts 'stepRetry.failedAttempts': number; + 'stepRetry.failedStallAttempts': number; 'stepRetry.lastFailedDriverId': string | undefined; // src/agent/task/taskOps.ts // replayable · durable — folds: TaskStarted, TaskTerminated diff --git a/packages/agent-core-v2/src/_base/utils/abort.ts b/packages/agent-core-v2/src/_base/utils/abort.ts index 7662966b21..fe288ad5d5 100644 --- a/packages/agent-core-v2/src/_base/utils/abort.ts +++ b/packages/agent-core-v2/src/_base/utils/abort.ts @@ -1,3 +1,5 @@ +import { MAX_TIMER_DELAY_MS, systemTimeoutScheduler, type TimeoutScheduler } from './timer'; + export function abortError(message = 'Aborted'): Error { const error = new Error(message); error.name = 'AbortError'; @@ -69,6 +71,13 @@ export interface DeadlineAbortSignal { readonly clear: () => void; } +export interface IdleTimeoutAbortSignal { + readonly signal: AbortSignal; + readonly idleTimedOut: () => boolean; + readonly touch: (timeoutMs?: number) => void; + readonly clear: () => void; +} + export function createDeadlineAbortSignal( source: AbortSignal, timeoutMs: number, @@ -91,3 +100,44 @@ export function createDeadlineAbortSignal( }, }; } + +export function createIdleTimeoutAbortSignal( + source: AbortSignal | undefined, + timeoutMs: number, + scheduler: TimeoutScheduler = systemTimeoutScheduler, +): IdleTimeoutAbortSignal { + const controller = new AbortController(); + const unlinkAbortSignal = + source === undefined ? undefined : linkAbortSignal(source, controller); + let didIdleTimeout = false; + let cleared = false; + let currentTimeoutMs = timeoutMs; + let timeout: ReturnType | undefined; + + const arm = () => { + if (timeout !== undefined) scheduler.clear(timeout); + timeout = undefined; + if (cleared || didIdleTimeout || controller.signal.aborted || currentTimeoutMs <= 0) return; + timeout = scheduler.set(() => { + timeout = undefined; + didIdleTimeout = true; + controller.abort(abortError()); + }, Math.min(currentTimeoutMs, MAX_TIMER_DELAY_MS)); + }; + arm(); + + return { + signal: controller.signal, + idleTimedOut: () => didIdleTimeout, + touch: (nextTimeoutMs?: number) => { + if (nextTimeoutMs !== undefined) currentTimeoutMs = nextTimeoutMs; + arm(); + }, + clear: () => { + cleared = true; + if (timeout !== undefined) scheduler.clear(timeout); + timeout = undefined; + unlinkAbortSignal?.(); + }, + }; +} diff --git a/packages/agent-core-v2/src/_base/utils/timer.ts b/packages/agent-core-v2/src/_base/utils/timer.ts index f6eee24962..71a4dcf515 100644 --- a/packages/agent-core-v2/src/_base/utils/timer.ts +++ b/packages/agent-core-v2/src/_base/utils/timer.ts @@ -9,6 +9,20 @@ export function setClampedTimeout( return setTimeout(callback, Math.min(timeoutMs, MAX_TIMER_DELAY_MS)); } +export interface TimeoutScheduler { + now(): number; + set(callback: () => void, timeoutMs: number): ReturnType; + clear(handle: ReturnType): void; +} + +export const systemTimeoutScheduler: TimeoutScheduler = { + now: () => Date.now(), + set: (callback, timeoutMs) => setTimeout(callback, timeoutMs), + clear: (handle) => { + clearTimeout(handle); + }, +}; + export interface IntervalTimerOptions { readonly unref?: boolean; } diff --git a/packages/agent-core-v2/src/agent/llmRequester/llmRequesterService.ts b/packages/agent-core-v2/src/agent/llmRequester/llmRequesterService.ts index 050b65e749..c3b8926e4e 100644 --- a/packages/agent-core-v2/src/agent/llmRequester/llmRequesterService.ts +++ b/packages/agent-core-v2/src/agent/llmRequester/llmRequesterService.ts @@ -10,6 +10,12 @@ import { } from '#/agent/contextProjector/contextProjector'; import { IAgentTokenCountingService } from '#/agent/tokenCounting/tokenCounting'; import { IAgentProfileService, type ProfileModelContext } from '#/agent/profile/profile'; +import { + DEFAULT_FIRST_OUTPUT_TIMEOUT_MS, + DEFAULT_STREAM_IDLE_TIMEOUT_MS, + LOOP_CONTROL_SECTION, + type LoopControl, +} from '#/agent/loop/configSection'; import { IAgentStateService } from '#/agent/state/agentState'; import { IAgentToolRegistryService } from '#/agent/toolRegistry/toolRegistry'; import { IAgentToolSelectService } from '#/agent/toolSelect/toolSelect'; @@ -589,12 +595,20 @@ export class AgentLLMRequesterService implements IAgentLLMRequesterService { : undefined, }); const requester = this.modelCatalog.getRequester(resolved.modelAlias); + const loopControl = this.config.get(LOOP_CONTROL_SECTION); const messages = overrides.messages ?? this.context.get(); return { requester, model: requester.model, - params: { ...baseParams, ...budgetParams }, + params: { + ...baseParams, + ...budgetParams, + firstOutputTimeoutMs: + loopControl?.firstOutputTimeoutMs ?? DEFAULT_FIRST_OUTPUT_TIMEOUT_MS, + streamIdleTimeoutMs: + loopControl?.streamIdleTimeoutMs ?? DEFAULT_STREAM_IDLE_TIMEOUT_MS, + }, modelAlias: resolved.modelAlias, thinkingEffort: resolved.thinkingLevel, systemPrompt: overrides.systemPrompt ?? turnConfig?.systemPrompt ?? this.profile.getSystemPrompt(), diff --git a/packages/agent-core-v2/src/agent/loop/configSection.ts b/packages/agent-core-v2/src/agent/loop/configSection.ts index b04d4e4cc4..30c8742ddd 100644 --- a/packages/agent-core-v2/src/agent/loop/configSection.ts +++ b/packages/agent-core-v2/src/agent/loop/configSection.ts @@ -3,6 +3,7 @@ import { z } from 'zod'; import { type EnvBindings, envBindings, stripEnvBoundFields } from '#/app/config/config'; import { registerConfigSection } from '#/app/config/configSectionContributions'; import { plainObjectToToml } from '#/app/config/toml'; +import { MAX_TIMER_DELAY_MS } from '#/_base/utils/timer'; export const LOOP_CONTROL_SECTION = 'loopControl'; @@ -10,6 +11,13 @@ export const LOOP_MAX_STEPS_PER_TURN_ENV = 'KIMI_LOOP_MAX_STEPS_PER_TURN'; export const LOOP_MAX_ATTEMPTS_PER_STEP_ENV = 'KIMI_LOOP_MAX_ATTEMPTS_PER_STEP'; /** Deprecated former name of {@link LOOP_MAX_ATTEMPTS_PER_STEP_ENV}. */ export const LOOP_MAX_RETRIES_PER_STEP_ENV = 'KIMI_LOOP_MAX_RETRIES_PER_STEP'; +export const LOOP_FIRST_OUTPUT_TIMEOUT_MS_ENV = 'KIMI_LOOP_FIRST_OUTPUT_TIMEOUT_MS'; +export const LOOP_STREAM_IDLE_TIMEOUT_MS_ENV = 'KIMI_LOOP_STREAM_IDLE_TIMEOUT_MS'; +export const LOOP_MAX_STALL_ATTEMPTS_PER_STEP_ENV = 'KIMI_LOOP_MAX_STALL_ATTEMPTS_PER_STEP'; + +export const DEFAULT_FIRST_OUTPUT_TIMEOUT_MS = 180_000; +export const DEFAULT_STREAM_IDLE_TIMEOUT_MS = 120_000; +export const DEFAULT_MAX_STALL_ATTEMPTS_PER_STEP = 3; export const LoopControlSchema = z.object({ maxStepsPerTurn: z.number().int().min(0).optional(), @@ -17,17 +25,25 @@ export const LoopControlSchema = z.object({ maxRalphIterations: z.number().int().min(-1).optional(), reservedContextSize: z.number().int().min(0).optional(), compactionTriggerRatio: z.number().min(0.5).max(0.99).optional(), + firstOutputTimeoutMs: z.number().int().min(0).max(MAX_TIMER_DELAY_MS).optional(), + streamIdleTimeoutMs: z.number().int().min(0).max(MAX_TIMER_DELAY_MS).optional(), + maxStallAttemptsPerStep: z.number().int().min(0).optional(), }); export type LoopControl = z.infer; -function parseNonNegativeInt(raw: string): number | undefined { +function parseNonNegativeInt(raw: string, max?: number): number | undefined { const value = raw.trim(); if (value.length === 0 || !/^\d+$/.test(value)) return undefined; const parsed = Number(value); - return Number.isInteger(parsed) && parsed >= 0 ? parsed : undefined; + return Number.isInteger(parsed) && parsed >= 0 && (max === undefined || parsed <= max) + ? parsed + : undefined; } +const parseTimerDelayMs = (raw: string): number | undefined => + parseNonNegativeInt(raw, MAX_TIMER_DELAY_MS); + export const loopControlEnvBindings: EnvBindings = envBindings(LoopControlSchema, { maxStepsPerTurn: { env: LOOP_MAX_STEPS_PER_TURN_ENV, parse: parseNonNegativeInt }, maxAttemptsPerStep: { @@ -35,6 +51,12 @@ export const loopControlEnvBindings: EnvBindings = envBindings(Loop deprecatedEnv: LOOP_MAX_RETRIES_PER_STEP_ENV, parse: parseNonNegativeInt, }, + firstOutputTimeoutMs: { env: LOOP_FIRST_OUTPUT_TIMEOUT_MS_ENV, parse: parseTimerDelayMs }, + streamIdleTimeoutMs: { env: LOOP_STREAM_IDLE_TIMEOUT_MS_ENV, parse: parseTimerDelayMs }, + maxStallAttemptsPerStep: { + env: LOOP_MAX_STALL_ATTEMPTS_PER_STEP_ENV, + parse: parseNonNegativeInt, + }, }); export const stripLoopControlEnv = stripEnvBoundFields(loopControlEnvBindings); diff --git a/packages/agent-core-v2/src/agent/stepRetry/stepRetryService.ts b/packages/agent-core-v2/src/agent/stepRetry/stepRetryService.ts index 8ab985ddfe..c6cf3bd9d4 100644 --- a/packages/agent-core-v2/src/agent/stepRetry/stepRetryService.ts +++ b/packages/agent-core-v2/src/agent/stepRetry/stepRetryService.ts @@ -10,7 +10,7 @@ import { retryErrorFields, sleepForRetry, } from '#/_base/utils/retry'; -import { isRetryableGenerateError } from '#/kosong/contract/errors'; +import { isRetryableGenerateError, LLMStreamStalledError } from '#/kosong/contract/errors'; import { IConfigService } from '#/app/config/config'; import { IEventBus } from '#/app/event/eventBus'; import { Event2 } from '#/app/event/event2'; @@ -19,7 +19,11 @@ import { IAgentLoopService, type LoopErrorContext, } from '#/agent/loop/loop'; -import { LOOP_CONTROL_SECTION, type LoopControl } from '#/agent/loop/configSection'; +import { + DEFAULT_MAX_STALL_ATTEMPTS_PER_STEP, + LOOP_CONTROL_SECTION, + type LoopControl, +} from '#/agent/loop/configSection'; import { TurnStarted } from '#/agent/loop/turnEvents'; import { IAgentStateService } from '#/agent/state/agentState'; import { IEventDispatcher } from '#/state/eventDispatcher'; @@ -53,6 +57,10 @@ export const stepRetryFailedAttemptsKey = defineState( 'stepRetry.failedAttempts', () => 0, ); +export const stepRetryFailedStallAttemptsKey = defineState( + 'stepRetry.failedStallAttempts', + () => 0, +); export class AgentStepRetryService extends Disposable implements IAgentStepRetryService { declare readonly _serviceBrand: undefined; @@ -67,6 +75,7 @@ export class AgentStepRetryService extends Disposable implements IAgentStepRetry super(); this.states.contributeState(stepRetryLastFailedDriverIdKey); this.states.contributeState(stepRetryFailedAttemptsKey); + this.states.contributeState(stepRetryFailedStallAttemptsKey); this._register( this.loopService.registerLoopErrorHandler({ id: 'step-retry', @@ -99,9 +108,18 @@ export class AgentStepRetryService extends Disposable implements IAgentStepRetry this.states.set(stepRetryFailedAttemptsKey, value); } + private get failedStallAttempts(): number { + return this.states.get(stepRetryFailedStallAttemptsKey); + } + + private set failedStallAttempts(value: number) { + this.states.set(stepRetryFailedStallAttemptsKey, value); + } + private resetAttempts(): void { this.lastFailedDriverId = undefined; this.failedAttempts = 0; + this.failedStallAttempts = 0; } private async recover(context: LoopErrorContext): Promise { @@ -111,30 +129,38 @@ export class AgentStepRetryService extends Disposable implements IAgentStepRetry if (this.lastFailedDriverId !== driver.id) { this.lastFailedDriverId = driver.id; this.failedAttempts = 0; + this.failedStallAttempts = 0; } this.failedAttempts += 1; - const maxAttempts = Math.max( - this.config.get(LOOP_CONTROL_SECTION)?.maxAttemptsPerStep ?? - DEFAULT_MAX_RETRY_ATTEMPTS, - 1, - ); - if (this.failedAttempts >= maxAttempts) { + const error = unwrapErrorCause(context.error); + const loopControl = this.config.get(LOOP_CONTROL_SECTION); + const maxAttempts = Math.max(loopControl?.maxAttemptsPerStep ?? DEFAULT_MAX_RETRY_ATTEMPTS, 1); + let failedAttempt = this.failedAttempts; + let attemptBudget = maxAttempts; + if (error instanceof LLMStreamStalledError) { + this.failedStallAttempts += 1; + failedAttempt = this.failedStallAttempts; + attemptBudget = Math.max( + loopControl?.maxStallAttemptsPerStep ?? DEFAULT_MAX_STALL_ATTEMPTS_PER_STEP, + 1, + ); + } + if (this.failedAttempts >= maxAttempts || failedAttempt >= attemptBudget) { this.resetAttempts(); return false; } - const error = unwrapErrorCause(context.error); const delayMs = - readRetryAfterMs(error) ?? retryBackoffDelays(maxAttempts)[this.failedAttempts - 1] ?? 0; + readRetryAfterMs(error) ?? retryBackoffDelays(failedAttempt + 1)[failedAttempt - 1] ?? 0; void this.dispatcher.dispatch( new TurnStepRetrying({ turnId: context.turnId, step: context.step, stepId: context.stepId, - failedAttempt: this.failedAttempts, - nextAttempt: this.failedAttempts + 1, - maxAttempts, + failedAttempt, + nextAttempt: failedAttempt + 1, + maxAttempts: attemptBudget, delayMs, ...retryErrorFields(error), }), diff --git a/packages/agent-core-v2/src/kosong/contract/errors.ts b/packages/agent-core-v2/src/kosong/contract/errors.ts index e2db1f28a3..331c7e0e40 100644 --- a/packages/agent-core-v2/src/kosong/contract/errors.ts +++ b/packages/agent-core-v2/src/kosong/contract/errors.ts @@ -59,12 +59,37 @@ export class VideoUploadUnsupportedError extends ChatProviderError { } export class APITimeoutError extends ChatProviderError { - constructor(message: string) { - super(message, PROVIDER_CONNECTION_ERROR_CODE); + constructor(message: string, options?: Error2Options) { + super(message, PROVIDER_CONNECTION_ERROR_CODE, options); this.name = 'APITimeoutError'; } } +export type LLMStreamStallPhase = 'waiting_first_output' | 'streaming'; + +export class LLMStreamStalledError extends APITimeoutError { + readonly phase: LLMStreamStallPhase; + readonly elapsedMs: number; + readonly idleMs: number; + + constructor( + message: string, + options: { + readonly phase: LLMStreamStallPhase; + readonly elapsedMs: number; + readonly idleMs: number; + }, + ) { + super(message, { + details: { phase: options.phase, elapsedMs: options.elapsedMs, idleMs: options.idleMs }, + }); + this.name = 'LLMStreamStalledError'; + this.phase = options.phase; + this.elapsedMs = options.elapsedMs; + this.idleMs = options.idleMs; + } +} + export class APIStatusError extends ChatProviderError { readonly statusCode: number; readonly requestId: string | null; diff --git a/packages/agent-core-v2/src/kosong/model/modelRequester.ts b/packages/agent-core-v2/src/kosong/model/modelRequester.ts index c3cafa5710..0558114c0d 100644 --- a/packages/agent-core-v2/src/kosong/model/modelRequester.ts +++ b/packages/agent-core-v2/src/kosong/model/modelRequester.ts @@ -49,6 +49,8 @@ export interface ModelRequestParams { readonly usedContextTokens?: number; readonly maxContextTokens?: number; readonly onTraceId?: (traceId: string | null) => void; + readonly firstOutputTimeoutMs?: number; + readonly streamIdleTimeoutMs?: number; } export interface ModelRequester { diff --git a/packages/agent-core-v2/src/kosong/model/modelRequesterImpl.ts b/packages/agent-core-v2/src/kosong/model/modelRequesterImpl.ts index 2cc41870ed..68ebd2f5ca 100644 --- a/packages/agent-core-v2/src/kosong/model/modelRequesterImpl.ts +++ b/packages/agent-core-v2/src/kosong/model/modelRequesterImpl.ts @@ -1,6 +1,14 @@ import { AsyncEventQueue } from '#/_base/asyncEventQueue'; +import { createIdleTimeoutAbortSignal } from '#/_base/utils/abort'; +import { systemTimeoutScheduler, type TimeoutScheduler } from '#/_base/utils/timer'; import type { VideoURLPart } from '#/kosong/contract/message'; -import { APIStatusError, isAbortError, VideoUploadUnsupportedError } from '#/kosong/contract/errors'; +import { + APIStatusError, + isAbortError, + LLMStreamStalledError, + VideoUploadUnsupportedError, + type LLMStreamStallPhase, +} from '#/kosong/contract/errors'; import { generate, type GenerateResult } from '#/kosong/contract/generate'; import type { ChatProvider, @@ -27,6 +35,7 @@ export class ModelRequesterImpl implements ModelRequester { constructor( readonly model: Model, private readonly protocolRegistry: IProtocolAdapterRegistry, + private readonly scheduler: TimeoutScheduler = systemTimeoutScheduler, ) {} private resolveChatProvider(): ChatProvider { @@ -81,7 +90,7 @@ export class ModelRequesterImpl implements ModelRequester { signal?.throwIfAborted(); const provider = this.resolveChatProvider(); - let requestStartedAt = Date.now(); + let requestStartedAt = this.scheduler.now(); let requestSentAt: number | undefined; let firstChunkAt: number | undefined; let streamEndedAt: number | undefined; @@ -99,36 +108,76 @@ export class ModelRequesterImpl implements ModelRequester { usedContextTokens: params?.usedContextTokens, maxContextTokens: params?.maxContextTokens, onRequestStart: () => { - requestStartedAt = Date.now(); + requestStartedAt = this.scheduler.now(); }, onRequestSent: () => { - requestSentAt = Date.now(); + requestSentAt = this.scheduler.now(); }, onStreamEnd: (stats) => { - streamEndedAt = Date.now(); + streamEndedAt = this.scheduler.now(); decodeStats = stats; }, onTraceId: params?.onTraceId, responseFormat: input.responseFormat, }; + const firstOutputTimeoutMs = params?.firstOutputTimeoutMs ?? 0; + const streamIdleTimeoutMs = params?.streamIdleTimeoutMs ?? 0; + const stallWatchdogEnabled = firstOutputTimeoutMs > 0 || streamIdleTimeoutMs > 0; + let result: GenerateResult; try { - result = await this.runWithAuthRefresh((auth) => { - requestStartedAt = Date.now(); - return generate( - provider, - input.systemPrompt, - [...input.tools], - [...input.messages], - { - onMessagePart: (part) => { - firstChunkAt ??= Date.now(); - queue.push({ type: 'part', part }); + result = await this.runWithAuthRefresh(async (auth) => { + requestStartedAt = this.scheduler.now(); + const watchdog = stallWatchdogEnabled + ? createIdleTimeoutAbortSignal(signal, firstOutputTimeoutMs, this.scheduler) + : undefined; + let stallPhase: LLMStreamStallPhase = 'waiting_first_output'; + let lastActivityAt = this.scheduler.now(); + try { + return await generate( + provider, + input.systemPrompt, + [...input.tools], + [...input.messages], + { + onMessagePart: (part) => { + firstChunkAt ??= this.scheduler.now(); + stallPhase = 'streaming'; + lastActivityAt = this.scheduler.now(); + watchdog?.touch(streamIdleTimeoutMs); + queue.push({ type: 'part', part }); + }, + }, + { + ...options, + auth, + signal: watchdog?.signal ?? signal, + onRequestSent: () => { + requestSentAt = this.scheduler.now(); + lastActivityAt = this.scheduler.now(); + watchdog?.touch(); + }, }, - }, - { ...options, auth }, - ); + ); + } catch (error) { + if (watchdog?.idleTimedOut() === true && signal?.aborted !== true && isAbortError(error)) { + const stalledAt = this.scheduler.now(); + throw new LLMStreamStalledError( + `LLM stream stalled while ${stallPhase === 'waiting_first_output' ? 'waiting for the first output' : 'streaming'}: ` + + `no output for ${String(stalledAt - lastActivityAt)}ms ` + + `(elapsed ${String(stalledAt - requestStartedAt)}ms, provider: ${provider.name}, model: ${provider.modelName}).`, + { + phase: stallPhase, + elapsedMs: stalledAt - requestStartedAt, + idleMs: stalledAt - lastActivityAt, + }, + ); + } + throw error; + } finally { + watchdog?.clear(); + } }); } catch (error) { if (isAbortError(error) || signal?.aborted === true) throw error; diff --git a/packages/agent-core-v2/src/kosong/provider/bases/google-genai/google-genai.ts b/packages/agent-core-v2/src/kosong/provider/bases/google-genai/google-genai.ts index b6bc1d5c0b..898138067e 100644 --- a/packages/agent-core-v2/src/kosong/provider/bases/google-genai/google-genai.ts +++ b/packages/agent-core-v2/src/kosong/provider/bases/google-genai/google-genai.ts @@ -1,5 +1,6 @@ import { ApiError as GoogleApiError, GoogleGenAI as GenAIClient } from '@google/genai'; +import { isAbortError } from '#/_base/utils/abort'; import { APIConnectionError, APITimeoutError, @@ -190,6 +191,10 @@ function createAbortError(): DOMException { return new DOMException('The operation was aborted.', 'AbortError'); } +function isAborted(signal: AbortSignal | undefined): boolean { + return signal?.aborted === true; +} + async function abortPromise(signal: AbortSignal | undefined): Promise { if (signal === undefined) { return new Promise(() => {}); @@ -594,8 +599,14 @@ export class GoogleGenAIStreamedMessage implements StreamedMessage { response: AsyncIterable>, signal?: AbortSignal, ): AsyncGenerator { + const iterator = response[Symbol.asyncIterator](); + const aborted = abortPromise(signal); + void aborted.catch(() => {}); try { - for await (const chunk of response) { + for (;;) { + const result = await Promise.race([iterator.next(), aborted]); + if (result.done === true) return; + const chunk = result.value; this._throwIfAborted(signal); this._extractUsage(chunk); this._extractId(chunk); @@ -606,10 +617,14 @@ export class GoogleGenAIStreamedMessage implements StreamedMessage { } } } catch (error: unknown) { - if (error instanceof DOMException && error.name === 'AbortError') { - throw error; + if (signal?.aborted === true || isAbortError(error)) { + throw createAbortError(); } throw convertGoogleGenAIError(error); + } finally { + try { + void iterator.return?.()?.catch(() => {}); + } catch {} } } } @@ -780,6 +795,7 @@ export class GoogleGenAIChatProvider implements ChatProvider { const config: Record = { ...kwargs, systemInstruction: systemPrompt, + abortSignal: options?.signal, ...(tools.length > 0 ? { tools: tools.map((t) => toolToGoogleGenAI(t)) } : {}), }; applyResponseFormat(config, options?.responseFormat); @@ -816,8 +832,8 @@ export class GoogleGenAIChatProvider implements ChatProvider { options?.signal, ); } catch (error: unknown) { - if (error instanceof DOMException && error.name === 'AbortError') { - throw error; + if (isAborted(options?.signal) || isAbortError(error)) { + throw createAbortError(); } throw convertGoogleGenAIError(error); } diff --git a/packages/agent-core-v2/test/_base/utils/abort.test.ts b/packages/agent-core-v2/test/_base/utils/abort.test.ts index 310e637896..f8823e8dcb 100644 --- a/packages/agent-core-v2/test/_base/utils/abort.test.ts +++ b/packages/agent-core-v2/test/_base/utils/abort.test.ts @@ -3,10 +3,14 @@ import { describe, expect, it } from 'vitest'; import { abortError, abortable, + createIdleTimeoutAbortSignal, isAbortError, isUserCancellation, userCancellationReason, } from '#/_base/utils/abort'; +import { MAX_TIMER_DELAY_MS } from '#/_base/utils/timer'; + +import { ManualTimeoutScheduler } from './stubs'; describe('userCancellationReason', () => { it('is recognised as a deliberate user cancellation', () => { @@ -71,3 +75,146 @@ describe('abortable', () => { }); }); }); + +describe('createIdleTimeoutAbortSignal', () => { + it('aborts the synthesized signal once the idle timeout elapses', async () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, 1000, scheduler); + + expect(scheduler.scheduledTimeoutMs()).toBe(1000); + await scheduler.advance(999); + expect(idle.signal.aborted).toBe(false); + await scheduler.advance(1); + + expect(idle.signal.aborted).toBe(true); + expect(idle.idleTimedOut()).toBe(true); + expect(isAbortError(idle.signal.reason)).toBe(true); + idle.clear(); + }); + + it('restarts the countdown on every touch', async () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, 1000, scheduler); + + await scheduler.advance(900); + idle.touch(); + await scheduler.advance(900); + expect(idle.signal.aborted).toBe(false); + await scheduler.advance(100); + + expect(idle.signal.aborted).toBe(true); + expect(idle.idleTimedOut()).toBe(true); + }); + + it('switches the countdown duration when touch passes one', async () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, 10_000, scheduler); + + idle.touch(100); + expect(scheduler.scheduledTimeoutMs()).toBe(100); + await scheduler.advance(100); + + expect(idle.signal.aborted).toBe(true); + expect(idle.idleTimedOut()).toBe(true); + }); + + it('disarms when touch passes a non-positive duration', async () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, 1000, scheduler); + + idle.touch(0); + + expect(scheduler.size).toBe(0); + await scheduler.advance(60_000); + expect(idle.signal.aborted).toBe(false); + idle.clear(); + }); + + it('propagates a parent abort with its reason and stays distinguishable from an idle timeout', () => { + const scheduler = new ManualTimeoutScheduler(); + const controller = new AbortController(); + const idle = createIdleTimeoutAbortSignal(controller.signal, 1000, scheduler); + const reason = userCancellationReason(); + + controller.abort(reason); + + expect(idle.signal.aborted).toBe(true); + expect(idle.signal.reason).toBe(reason); + expect(idle.idleTimedOut()).toBe(false); + idle.clear(); + }); + + it('stops the timer and unlinks the parent on clear', () => { + const scheduler = new ManualTimeoutScheduler(); + const controller = new AbortController(); + const idle = createIdleTimeoutAbortSignal(controller.signal, 1000, scheduler); + expect(scheduler.size).toBe(1); + + idle.clear(); + + expect(scheduler.size).toBe(0); + controller.abort(); + expect(idle.signal.aborted).toBe(false); + }); + + it('never arms when created with a non-positive timeout', async () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, 0, scheduler); + + expect(scheduler.size).toBe(0); + idle.touch(); + expect(scheduler.size).toBe(0); + await scheduler.advance(60_000); + expect(idle.signal.aborted).toBe(false); + idle.clear(); + }); + + it('ignores touches after clear', () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, 1000, scheduler); + idle.clear(); + + idle.touch(); + + expect(scheduler.size).toBe(0); + expect(idle.signal.aborted).toBe(false); + }); + + it('does not re-arm once the idle timeout has fired', async () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, 1000, scheduler); + await scheduler.advance(1000); + expect(idle.idleTimedOut()).toBe(true); + + idle.touch(); + + expect(scheduler.size).toBe(0); + }); + + it('clamps an excessive timeout to the timer ceiling instead of firing immediately', async () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, Number.MAX_SAFE_INTEGER, scheduler); + + expect(scheduler.scheduledTimeoutMs()).toBe(MAX_TIMER_DELAY_MS); + await scheduler.advance(MAX_TIMER_DELAY_MS - 1); + expect(idle.signal.aborted).toBe(false); + await scheduler.advance(1); + + expect(idle.signal.aborted).toBe(true); + expect(idle.idleTimedOut()).toBe(true); + idle.clear(); + }); + + it('clamps an excessive timeout passed to touch', async () => { + const scheduler = new ManualTimeoutScheduler(); + const idle = createIdleTimeoutAbortSignal(undefined, 1000, scheduler); + + idle.touch(Number.MAX_SAFE_INTEGER); + + expect(scheduler.scheduledTimeoutMs()).toBe(MAX_TIMER_DELAY_MS); + await scheduler.advance(60_000); + expect(idle.signal.aborted).toBe(false); + expect(scheduler.size).toBe(1); + idle.clear(); + }); +}); diff --git a/packages/agent-core-v2/test/_base/utils/stubs.ts b/packages/agent-core-v2/test/_base/utils/stubs.ts new file mode 100644 index 0000000000..712ff8709e --- /dev/null +++ b/packages/agent-core-v2/test/_base/utils/stubs.ts @@ -0,0 +1,66 @@ +import type { TimeoutScheduler } from '#/_base/utils/timer'; + +interface ManualEntry { + readonly handle: object; + readonly callback: () => void; + readonly timeoutMs: number; + readonly dueAt: number; + readonly seq: number; +} + +export class ManualTimeoutScheduler implements TimeoutScheduler { + private currentTime = 0; + private nextSeq = 0; + private readonly entries: ManualEntry[] = []; + + now(): number { + return this.currentTime; + } + + set(callback: () => void, timeoutMs: number): ReturnType { + const handle = {}; + this.entries.push({ + handle, + callback, + timeoutMs, + dueAt: this.currentTime + timeoutMs, + seq: ++this.nextSeq, + }); + return handle as unknown as ReturnType; + } + + clear(handle: ReturnType): void { + const index = this.entries.findIndex((entry) => entry.handle === handle); + if (index !== -1) this.entries.splice(index, 1); + } + + get size(): number { + return this.entries.length; + } + + scheduledTimeoutMs(index = 0): number | undefined { + return this.entries[index]?.timeoutMs; + } + + async advance(ms: number): Promise { + const target = this.currentTime + ms; + for (;;) { + const due = this.entries + .filter((entry) => entry.dueAt <= target) + .toSorted((a, b) => a.dueAt - b.dueAt || a.seq - b.seq)[0]; + if (due === undefined) break; + this.entries.splice(this.entries.indexOf(due), 1); + this.currentTime = due.dueAt; + due.callback(); + await this.flushMicrotasks(); + } + this.currentTime = target; + await this.flushMicrotasks(); + } + + private async flushMicrotasks(): Promise { + await new Promise((resolve) => { + setImmediate(resolve); + }); + } +} diff --git a/packages/agent-core-v2/test/agent/llmRequester/llmRequester.test.ts b/packages/agent-core-v2/test/agent/llmRequester/llmRequester.test.ts index 20d773fbdb..43bd097d55 100644 --- a/packages/agent-core-v2/test/agent/llmRequester/llmRequester.test.ts +++ b/packages/agent-core-v2/test/agent/llmRequester/llmRequester.test.ts @@ -1,4 +1,10 @@ -import { APIConnectionError, APIStatusError } from '#/kosong/contract/errors'; +import { + APIConnectionError, + APIStatusError, + createAbortError, + isAbortError, + LLMStreamStalledError, +} from '#/kosong/contract/errors'; import { TOOL_SELECT_FLAG_ENV } from '#/agent/toolSelect/flag'; import { type StreamedMessagePart } from '#/kosong/contract/message'; import type { Tool } from '#/kosong/contract/tool'; @@ -635,6 +641,90 @@ describe('LLMRequester service migration coverage', () => { }); }); + describe('stream stall watchdog', () => { + let ctx: TestAgentContext; + + afterEach(async () => { + vi.useRealTimers(); + try { + await ctx.expectResumeMatches(); + } finally { + await ctx.dispose(); + } + }); + + it('fails a hung request with LLMStreamStalledError once the configured first-output budget elapses', async () => { + vi.useFakeTimers(); + ctx = createTestAgent( + llmGenerateServices( + async (_provider, _systemPrompt, _tools, _messages, _callbacks, options) => { + await new Promise((_resolve, reject) => { + options?.signal?.addEventListener('abort', () => reject(createAbortError()), { + once: true, + }); + }); + throw new Error('unreachable'); + }, + ), + { + initialConfig: { + loopControl: { firstOutputTimeoutMs: 1000, streamIdleTimeoutMs: 500 }, + }, + }, + ); + const llmRequester = ctx.get(IAgentLLMRequesterService); + + const failurePromise = llmRequester.request().catch((error: unknown) => error); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(1000); + const failure = (await failurePromise) as LLMStreamStalledError; + + expect(failure).toBeInstanceOf(LLMStreamStalledError); + expect(failure.phase).toBe('waiting_first_output'); + expect(failure.name).not.toBe('AbortError'); + }); + + it('leaves a hung request under user control when the configured budgets are zero', async () => { + vi.useFakeTimers(); + ctx = createTestAgent( + llmGenerateServices( + async (_provider, _systemPrompt, _tools, _messages, _callbacks, options) => { + await new Promise((_resolve, reject) => { + options?.signal?.addEventListener('abort', () => reject(createAbortError()), { + once: true, + }); + }); + throw new Error('unreachable'); + }, + ), + { + initialConfig: { + loopControl: { firstOutputTimeoutMs: 0, streamIdleTimeoutMs: 0 }, + }, + }, + ); + const llmRequester = ctx.get(IAgentLLMRequesterService); + const controller = new AbortController(); + + let settled = false; + const failurePromise = llmRequester + .request(undefined, undefined, controller.signal) + .catch((error: unknown) => error) + .then((error: unknown) => { + settled = true; + return error; + }); + + await vi.advanceTimersByTimeAsync(600_000); + expect(settled).toBe(false); + + controller.abort(); + const failure = await failurePromise; + expect(isAbortError(failure)).toBe(true); + expect(failure).not.toBeInstanceOf(LLMStreamStalledError); + }); + }); + }); type ProtocolEvent = Extract< diff --git a/packages/agent-core-v2/test/agent/stepRetry/stepRetry.test.ts b/packages/agent-core-v2/test/agent/stepRetry/stepRetry.test.ts index 7ed2b87446..b44e14265f 100644 --- a/packages/agent-core-v2/test/agent/stepRetry/stepRetry.test.ts +++ b/packages/agent-core-v2/test/agent/stepRetry/stepRetry.test.ts @@ -4,6 +4,7 @@ import { APIConnectionError, APIProviderRateLimitError, APIStatusError, + LLMStreamStalledError, } from '#/kosong/contract/errors'; import { emptyUsage } from '#/kosong/contract/usage'; import { IEventBus } from '#/app/event/eventBus'; @@ -214,6 +215,203 @@ describe('stepRetry plugin', () => { expect(rpcEvents('turn.step.retrying')).toEqual([]); }); + it('caps stream stall retries at the stall budget instead of maxAttemptsPerStep', async () => { + vi.useFakeTimers(); + let calls = 0; + ctx = createTestAgent( + llmGenerateServices(async () => { + calls += 1; + throw new LLMStreamStalledError('stalled', { + phase: 'waiting_first_output', + elapsedMs: 180_000, + idleMs: 180_000, + }); + }), + ); + + const result = await runTurn(1); + + expect(result.type).toBe('failed'); + expect(calls).toBe(3); + expect(rpcEvents('turn.step.retrying')).toHaveLength(2); + expect(rpcEvents('turn.step.retrying')[0]?.args).toMatchObject({ + maxAttempts: 3, + errorName: 'LLMStreamStalledError', + }); + }); + + it('honors loop_control.max_stall_attempts_per_step', async () => { + vi.useFakeTimers(); + let calls = 0; + ctx = createTestAgent( + llmGenerateServices(async () => { + calls += 1; + throw new LLMStreamStalledError('stalled', { + phase: 'streaming', + elapsedMs: 60_000, + idleMs: 60_000, + }); + }), + { initialConfig: { loopControl: { maxStallAttemptsPerStep: 1 } } }, + ); + + const result = await runTurn(1); + + expect(result.type).toBe('failed'); + expect(calls).toBe(1); + expect(rpcEvents('turn.step.retrying')).toEqual([]); + }); + + it('gives stream stalls a fresh budget after other retryable failures', async () => { + vi.useFakeTimers(); + let calls = 0; + ctx = createTestAgent( + llmGenerateServices(async () => { + calls += 1; + if (calls <= 2) throw new APIConnectionError('terminated'); + throw new LLMStreamStalledError('stalled', { + phase: 'streaming', + elapsedMs: 60_000, + idleMs: 60_000, + }); + }), + ); + + const result = await runTurn(1); + + expect(result.type).toBe('failed'); + expect(calls).toBe(5); + const retrying = rpcEvents('turn.step.retrying'); + expect(retrying).toHaveLength(4); + expect(retrying[0]?.args).toMatchObject({ + failedAttempt: 1, + maxAttempts: 10, + errorName: 'APIConnectionError', + }); + expect(retrying[2]?.args).toMatchObject({ + failedAttempt: 1, + maxAttempts: 3, + errorName: 'LLMStreamStalledError', + }); + expect(retrying[3]?.args).toMatchObject({ + failedAttempt: 2, + maxAttempts: 3, + errorName: 'LLMStreamStalledError', + }); + }); + + it('still counts stall attempts toward the total attempt budget', async () => { + vi.useFakeTimers(); + let calls = 0; + ctx = createTestAgent( + llmGenerateServices(async () => { + calls += 1; + if (calls <= 9) throw new APIConnectionError('terminated'); + throw new LLMStreamStalledError('stalled', { + phase: 'streaming', + elapsedMs: 60_000, + idleMs: 60_000, + }); + }), + ); + + const result = await runTurn(1); + + expect(result.type).toBe('failed'); + expect(calls).toBe(10); + const retrying = rpcEvents('turn.step.retrying'); + expect(retrying).toHaveLength(9); + expect(retrying[8]?.args).toMatchObject({ + failedAttempt: 9, + maxAttempts: 10, + errorName: 'APIConnectionError', + }); + }); + + it('retries a stall with a huge stall budget while computing only the needed delay', async () => { + vi.useFakeTimers(); + let calls = 0; + ctx = createTestAgent( + llmGenerateServices(async () => { + calls += 1; + if (calls === 1) { + throw new LLMStreamStalledError('stalled', { + phase: 'streaming', + elapsedMs: 60_000, + idleMs: 60_000, + }); + } + return { + id: 'huge-stall-budget-response', + message: { + role: 'assistant', + content: [{ type: 'text', text: 'recovered' }], + toolCalls: [], + }, + usage: emptyUsage(), + finishReason: 'completed', + rawFinishReason: 'stop', + }; + }), + { initialConfig: { loopControl: { maxStallAttemptsPerStep: 1_000_000_000 } } }, + ); + + const result = await runTurn(1); + + expect(result).toEqual({ type: 'completed', steps: 2, truncated: false }); + expect(calls).toBe(2); + const retrying = rpcEvents('turn.step.retrying'); + expect(retrying).toHaveLength(1); + expect(retrying[0]?.args).toMatchObject({ + failedAttempt: 1, + nextAttempt: 2, + maxAttempts: 1_000_000_000, + errorName: 'LLMStreamStalledError', + }); + const delayMs = (retrying[0]?.args as { delayMs: number }).delayMs; + expect(delayMs).toBeGreaterThanOrEqual(500); + expect(delayMs).toBeLessThanOrEqual(625); + }); + + it('retries a generic error with a huge attempts budget while computing only the needed delay', async () => { + vi.useFakeTimers(); + let calls = 0; + ctx = createTestAgent( + llmGenerateServices(async () => { + calls += 1; + if (calls === 1) throw new APIConnectionError('terminated'); + return { + id: 'huge-attempts-budget-response', + message: { + role: 'assistant', + content: [{ type: 'text', text: 'recovered' }], + toolCalls: [], + }, + usage: emptyUsage(), + finishReason: 'completed', + rawFinishReason: 'stop', + }; + }), + { initialConfig: { loopControl: { maxAttemptsPerStep: 1_000_000_000 } } }, + ); + + const result = await runTurn(1); + + expect(result).toEqual({ type: 'completed', steps: 2, truncated: false }); + expect(calls).toBe(2); + const retrying = rpcEvents('turn.step.retrying'); + expect(retrying).toHaveLength(1); + expect(retrying[0]?.args).toMatchObject({ + failedAttempt: 1, + nextAttempt: 2, + maxAttempts: 1_000_000_000, + errorName: 'APIConnectionError', + }); + const delayMs = (retrying[0]?.args as { delayMs: number }).delayMs; + expect(delayMs).toBeGreaterThanOrEqual(500); + expect(delayMs).toBeLessThanOrEqual(625); + }); + it('starts a fresh attempt budget on the next turn', async () => { vi.useFakeTimers(); let calls = 0; diff --git a/packages/agent-core-v2/test/app/config/config.test.ts b/packages/agent-core-v2/test/app/config/config.test.ts index 7dabf6d2f5..e2e5b5919f 100644 --- a/packages/agent-core-v2/test/app/config/config.test.ts +++ b/packages/agent-core-v2/test/app/config/config.test.ts @@ -55,9 +55,12 @@ import { import '#/agent/loop/configSection'; import { LOOP_CONTROL_SECTION, + LOOP_FIRST_OUTPUT_TIMEOUT_MS_ENV, LOOP_MAX_ATTEMPTS_PER_STEP_ENV, LOOP_MAX_RETRIES_PER_STEP_ENV, + LOOP_MAX_STALL_ATTEMPTS_PER_STEP_ENV, LOOP_MAX_STEPS_PER_TURN_ENV, + LOOP_STREAM_IDLE_TIMEOUT_MS_ENV, type LoopControl, } from '#/agent/loop/configSection'; import { @@ -893,6 +896,56 @@ describe('loopControl config section', () => { ).toEqual({ maxStepsPerTurn: 100, maxAttemptsPerStep: 3 }); expect(() => registry.validate(LOOP_CONTROL_SECTION, { maxStepsPerTurn: -1 })).toThrow(); expect(() => registry.validate(LOOP_CONTROL_SECTION, { maxAttemptsPerStep: 1.5 })).toThrow(); + expect(() => registry.validate(LOOP_CONTROL_SECTION, { streamIdleTimeoutMs: -1 })).toThrow(); + expect( + registry.validate(LOOP_CONTROL_SECTION, { + firstOutputTimeoutMs: 0, + streamIdleTimeoutMs: 1500, + maxStallAttemptsPerStep: 2, + }), + ).toEqual({ firstOutputTimeoutMs: 0, streamIdleTimeoutMs: 1500, maxStallAttemptsPerStep: 2 }); + }); + + it('rejects timeout values beyond the timer ceiling', () => { + const registry = new ConfigRegistry(); + + expect( + registry.validate(LOOP_CONTROL_SECTION, { streamIdleTimeoutMs: 2_147_483_647 }), + ).toEqual({ streamIdleTimeoutMs: 2_147_483_647 }); + expect(() => + registry.validate(LOOP_CONTROL_SECTION, { firstOutputTimeoutMs: 2_147_483_648 }), + ).toThrow(); + expect(() => + registry.validate(LOOP_CONTROL_SECTION, { streamIdleTimeoutMs: Number.MAX_SAFE_INTEGER }), + ).toThrow(); + }); + + it('ignores env timeout values beyond the timer ceiling without dropping other fields', async () => { + const env: Record = { + [LOOP_MAX_STEPS_PER_TURN_ENV]: '100', + [LOOP_FIRST_OUTPUT_TIMEOUT_MS_ENV]: '2147483648', + [LOOP_STREAM_IDLE_TIMEOUT_MS_ENV]: '99999999999999999999', + }; + const disposables = new DisposableStore(); + const ix = disposables.add(new TestInstantiationService()); + ix.stub(ILogService, stubLog()); + ix.stub(IBootstrapService, stubBootstrap('/tmp/kimi-cfg', env)); + ix.stub(IFileSystemStorageService, new InMemoryStorageService()); + ix.set(IAtomicTomlDocumentStore, new SyncDescriptor(TomlAtomicDocumentStore)); + ix.set(IConfigRegistry, new SyncDescriptor(ConfigRegistry)); + ix.set(IConfigService, new SyncDescriptor(ConfigService)); + const config = ix.get(IConfigService); + await config.ready; + + expect(config.get(LOOP_CONTROL_SECTION)).toEqual({ maxStepsPerTurn: 100 }); + + env[LOOP_STREAM_IDLE_TIMEOUT_MS_ENV] = '2147483647'; + expect(config.get(LOOP_CONTROL_SECTION)).toEqual({ + maxStepsPerTurn: 100, + streamIdleTimeoutMs: 2_147_483_647, + }); + + disposables.dispose(); }); it('re-applies loopControl env bindings on every get() and ignores invalid env', async () => { @@ -921,6 +974,17 @@ describe('loopControl config section', () => { maxAttemptsPerStep: 3, }); + env[LOOP_FIRST_OUTPUT_TIMEOUT_MS_ENV] = '0'; + env[LOOP_STREAM_IDLE_TIMEOUT_MS_ENV] = '1500'; + env[LOOP_MAX_STALL_ATTEMPTS_PER_STEP_ENV] = '2'; + expect(config.get(LOOP_CONTROL_SECTION)).toEqual({ + maxStepsPerTurn: 100, + maxAttemptsPerStep: 3, + firstOutputTimeoutMs: 0, + streamIdleTimeoutMs: 1500, + maxStallAttemptsPerStep: 2, + }); + env[LOOP_MAX_STEPS_PER_TURN_ENV] = '50'; expect(config.get(LOOP_CONTROL_SECTION).maxStepsPerTurn).toBe(50); diff --git a/packages/agent-core-v2/test/kosong/model/modelRequester.test.ts b/packages/agent-core-v2/test/kosong/model/modelRequester.test.ts index 22c5fc5022..4054c8618d 100644 --- a/packages/agent-core-v2/test/kosong/model/modelRequester.test.ts +++ b/packages/agent-core-v2/test/kosong/model/modelRequester.test.ts @@ -1,7 +1,14 @@ import { describe, expect, it } from 'vitest'; import { isError2 } from '#/_base/errors/errors'; -import { APIStatusError, createAbortError } from '#/kosong/contract/errors'; +import { + APIStatusError, + APITimeoutError, + createAbortError, + isAbortError, + isRetryableGenerateError, + LLMStreamStalledError, +} from '#/kosong/contract/errors'; import type { Message, StreamedMessagePart } from '#/kosong/contract/message'; import type { ChatProvider, @@ -12,11 +19,14 @@ import type { Tool } from '#/kosong/contract/tool'; import { emptyUsage, type TokenUsage } from '#/kosong/contract/usage'; import { ProtocolErrors } from '#/kosong/protocol/errors'; import type { IProtocolAdapterRegistry } from '#/kosong/protocol/protocol'; +import { GoogleGenAIChatProvider } from '#/kosong/provider/bases/google-genai/google-genai'; import type { Model } from '#/kosong/model/catalog'; import type { ModelRequestEvent } from '#/kosong/model/modelRequester'; import { effectiveMaxCompletionTokens } from '#/kosong/model/modelRequester'; import { buildStreamTiming, ModelRequesterImpl } from '#/kosong/model/modelRequesterImpl'; +import { ManualTimeoutScheduler } from '../../_base/utils/stubs'; + class FakeChatProvider implements ChatProvider { readonly name = 'fake-base'; readonly modelName = 'fake-model'; @@ -31,7 +41,7 @@ class FakeChatProvider implements ChatProvider { options?: GenerateOptions; }> = []; - handler: (callIndex: number) => Promise = () => + handler: (callIndex: number, options?: GenerateOptions) => Promise = () => Promise.resolve(streamOf([{ type: 'text', text: 'hello' }])); async generate( @@ -43,7 +53,7 @@ class FakeChatProvider implements ChatProvider { this.calls.push({ systemPrompt, tools, history, options }); options?.onRequestStart?.(); options?.onRequestSent?.(); - const stream = await this.handler(this.calls.length - 1); + const stream = await this.handler(this.calls.length - 1, options); return stream; } } @@ -289,6 +299,281 @@ describe('ModelRequesterImpl request execution', () => { }); }); +describe('ModelRequesterImpl stream stall watchdog', () => { + function neverYieldingStream(signal: AbortSignal | undefined): StreamedMessage { + return { + id: 'msg-stall', + usage: emptyUsage(), + finishReason: null, + rawFinishReason: null, + traceId: null, + [Symbol.asyncIterator]() { + return { + next: () => + new Promise>((_resolve, reject) => { + signal?.addEventListener('abort', () => reject(createAbortError()), { + once: true, + }); + }), + }; + }, + }; + } + + function stallingAfterStream( + scheduler: ManualTimeoutScheduler, + steps: readonly { readonly part: StreamedMessagePart; readonly delayMs?: number }[], + signal: AbortSignal | undefined, + ): StreamedMessage { + return { + id: 'msg-stall', + usage: emptyUsage(), + finishReason: null, + rawFinishReason: null, + traceId: null, + async *[Symbol.asyncIterator]() { + for (const step of steps) { + const delayMs = step.delayMs; + if (delayMs !== undefined) { + await new Promise((resolve) => { + scheduler.set(resolve, delayMs); + }); + } + yield step.part; + } + await new Promise((_resolve, reject) => { + signal?.addEventListener('abort', () => reject(createAbortError()), { once: true }); + }); + }, + }; + } + + it('fails with LLMStreamStalledError when the first output never arrives', async () => { + const scheduler = new ManualTimeoutScheduler(); + const provider = new FakeChatProvider(); + provider.handler = (_callIndex, options) => + Promise.resolve(neverYieldingStream(options?.signal)); + const requester = new ModelRequesterImpl( + modelWith(staticAuth()), + registryReturning(provider), + scheduler, + ); + + const failurePromise = collect( + requester.request(INPUT, undefined, { + firstOutputTimeoutMs: 1000, + streamIdleTimeoutMs: 500, + }), + ).catch((error: unknown) => error); + await scheduler.advance(0); + expect(scheduler.size).toBe(1); + + await scheduler.advance(1000); + const failure = (await failurePromise) as LLMStreamStalledError; + + expect(failure).toBeInstanceOf(LLMStreamStalledError); + expect(failure).toBeInstanceOf(APITimeoutError); + expect(failure).toMatchObject({ + name: 'LLMStreamStalledError', + phase: 'waiting_first_output', + code: ProtocolErrors.codes.PROVIDER_CONNECTION_ERROR, + }); + expect(failure.name).not.toBe('AbortError'); + expect(failure.elapsedMs).toBeGreaterThanOrEqual(1000); + expect(failure.idleMs).toBeGreaterThanOrEqual(1000); + expect(isRetryableGenerateError(failure)).toBe(true); + expect(scheduler.size).toBe(0); + }); + + it('cancels the first-output budget on the first part and resets the stream idle budget on every part', async () => { + const scheduler = new ManualTimeoutScheduler(); + const provider = new FakeChatProvider(); + provider.handler = (_callIndex, options) => + Promise.resolve( + stallingAfterStream( + scheduler, + [ + { part: { type: 'text', text: 'one' } }, + { part: { type: 'text', text: 'two' }, delayMs: 400 }, + ], + options?.signal, + ), + ); + const requester = new ModelRequesterImpl( + modelWith(staticAuth()), + registryReturning(provider), + scheduler, + ); + + let settled = false; + const failurePromise = collect( + requester.request(INPUT, undefined, { + firstOutputTimeoutMs: 300, + streamIdleTimeoutMs: 500, + }), + ).catch((error: unknown) => error); + void failurePromise.then(() => { + settled = true; + }); + await scheduler.advance(0); + + await scheduler.advance(399); + expect(settled).toBe(false); + await scheduler.advance(1); + expect(settled).toBe(false); + await scheduler.advance(499); + expect(settled).toBe(false); + + await scheduler.advance(1); + const failure = (await failurePromise) as LLMStreamStalledError; + + expect(settled).toBe(true); + expect(failure).toBeInstanceOf(LLMStreamStalledError); + expect(failure.phase).toBe('streaming'); + expect(failure.name).not.toBe('AbortError'); + expect(isRetryableGenerateError(failure)).toBe(true); + expect(scheduler.size).toBe(0); + }); + + it('keeps the original cancellation when the upstream signal aborts mid-stall', async () => { + const scheduler = new ManualTimeoutScheduler(); + const provider = new FakeChatProvider(); + provider.handler = (_callIndex, options) => + Promise.resolve(neverYieldingStream(options?.signal)); + const requester = new ModelRequesterImpl( + modelWith(staticAuth()), + registryReturning(provider), + scheduler, + ); + const controller = new AbortController(); + + const failurePromise = collect( + requester.request(INPUT, controller.signal, { + firstOutputTimeoutMs: 1000, + streamIdleTimeoutMs: 500, + }), + ).catch((error: unknown) => error); + await scheduler.advance(0); + + controller.abort(); + const failure = await failurePromise; + + expect(isAbortError(failure)).toBe(true); + expect(failure).not.toBeInstanceOf(LLMStreamStalledError); + expect(scheduler.size).toBe(0); + }); + + it('clears the watchdog timers after a successful request', async () => { + const scheduler = new ManualTimeoutScheduler(); + const provider = new FakeChatProvider(); + const requester = new ModelRequesterImpl( + modelWith(staticAuth()), + registryReturning(provider), + scheduler, + ); + + await collect( + requester.request(INPUT, undefined, { + firstOutputTimeoutMs: 1000, + streamIdleTimeoutMs: 500, + }), + ); + + expect(scheduler.size).toBe(0); + }); + + it('clears the watchdog timers after a provider failure', async () => { + const scheduler = new ManualTimeoutScheduler(); + const provider = new FakeChatProvider(); + provider.handler = () => Promise.reject(new APIStatusError(500, 'boom')); + const requester = new ModelRequesterImpl( + modelWith(staticAuth()), + registryReturning(provider), + scheduler, + ); + + const failure = await collect( + requester.request(INPUT, undefined, { + firstOutputTimeoutMs: 1000, + streamIdleTimeoutMs: 500, + }), + ).catch((error: unknown) => error); + + expect((failure as { code: string }).code).toBe(ProtocolErrors.codes.PROVIDER_API_ERROR); + expect(scheduler.size).toBe(0); + }); + + it('stays fully passive and passes the upstream signal through when both budgets are zero', async () => { + const scheduler = new ManualTimeoutScheduler(); + const provider = new FakeChatProvider(); + provider.handler = (_callIndex, options) => + Promise.resolve(neverYieldingStream(options?.signal)); + const requester = new ModelRequesterImpl( + modelWith(staticAuth()), + registryReturning(provider), + scheduler, + ); + const controller = new AbortController(); + + const failurePromise = collect( + requester.request(INPUT, controller.signal, { + firstOutputTimeoutMs: 0, + streamIdleTimeoutMs: 0, + }), + ).catch((error: unknown) => error); + await scheduler.advance(0); + + expect(scheduler.size).toBe(0); + expect(provider.calls[0]?.options?.signal).toBe(controller.signal); + + await scheduler.advance(600_000); + expect(scheduler.size).toBe(0); + + controller.abort(); + const failure = await failurePromise; + expect(isAbortError(failure)).toBe(true); + expect(failure).not.toBeInstanceOf(LLMStreamStalledError); + }); + + it('detects a stall behind the google-genai streamed message', async () => { + const scheduler = new ManualTimeoutScheduler(); + const provider = new GoogleGenAIChatProvider({ + model: 'gemini-2.5-flash', + apiKey: 'sk-probe', + stream: true, + }); + const client = Reflect.get(provider, '_client') as { + models: { generateContentStream: unknown }; + }; + client.models.generateContentStream = () => + Promise.resolve( + (async function* () { + await new Promise(() => {}); + yield {}; + })(), + ); + const requester = new ModelRequesterImpl( + modelWith(staticAuth()), + registryReturning(provider), + scheduler, + ); + + const failurePromise = collect( + requester.request(INPUT, undefined, { + firstOutputTimeoutMs: 1000, + streamIdleTimeoutMs: 500, + }), + ).catch((error: unknown) => error); + await scheduler.advance(0); + await scheduler.advance(1000); + const failure = (await failurePromise) as LLMStreamStalledError; + + expect(failure).toBeInstanceOf(LLMStreamStalledError); + expect(failure.phase).toBe('waiting_first_output'); + expect(failure.name).not.toBe('AbortError'); + }); +}); + describe('effectiveMaxCompletionTokens', () => { it('reads the folded budget back from the params', () => { expect(effectiveMaxCompletionTokens(undefined)).toBeUndefined(); diff --git a/packages/agent-core-v2/test/kosong/provider/composition.test.ts b/packages/agent-core-v2/test/kosong/provider/composition.test.ts index 5198b65613..acad7b152e 100644 --- a/packages/agent-core-v2/test/kosong/provider/composition.test.ts +++ b/packages/agent-core-v2/test/kosong/provider/composition.test.ts @@ -12,7 +12,7 @@ import { APIStatusError, isRetryableGenerateError, } from '#/kosong/contract/errors'; -import type { Message } from '#/kosong/contract/message'; +import type { Message, StreamedMessagePart } from '#/kosong/contract/message'; import type { ChatProvider, GenerateOptions, @@ -25,7 +25,10 @@ import { resolveDefaultMaxTokens, } from '#/kosong/provider/bases/anthropic/anthropic'; import '#/kosong/provider/bases/google-genai/index'; -import { GoogleGenAIChatProvider } from '#/kosong/provider/bases/google-genai/google-genai'; +import { + GoogleGenAIChatProvider, + GoogleGenAIStreamedMessage, +} from '#/kosong/provider/bases/google-genai/google-genai'; import '#/kosong/provider/bases/openai/index'; import { OpenAIResponsesChatProvider } from '#/kosong/provider/bases/openai/openai-responses'; import { OpenAILegacyChatProvider } from '#/kosong/provider/bases/openai/openai-legacy'; @@ -364,6 +367,210 @@ describe('google-genai vertex mode (providerOptions)', () => { }); }); +describe('GoogleGenAIStreamedMessage stream consumption', () => { + it('rejects with AbortError when the signal aborts while the provider iterator never yields', async () => { + const stuck: AsyncIterable> = { + [Symbol.asyncIterator]() { + return { + next: () => new Promise>>(() => {}), + }; + }, + }; + const controller = new AbortController(); + const message = new GoogleGenAIStreamedMessage(stuck, true, controller.signal); + + const failurePromise = (async () => { + for await (const part of message) void part; + })().then( + () => null, + (error: unknown) => error, + ); + controller.abort(); + + const failure = await failurePromise; + expect(failure).toBeInstanceOf(DOMException); + expect((failure as DOMException).name).toBe('AbortError'); + }); + + it('converts a healthy stream end to end', async () => { + const chunks: AsyncIterable> = { + async *[Symbol.asyncIterator]() { + yield { + candidates: [ + { + content: { parts: [{ text: 'Hello' }], role: 'model' }, + finishReason: 'STOP', + }, + ], + usageMetadata: { promptTokenCount: 3, candidatesTokenCount: 1, totalTokenCount: 4 }, + responseId: 'resp-1', + }; + }, + }; + const controller = new AbortController(); + const message = new GoogleGenAIStreamedMessage(chunks, true, controller.signal); + + const parts: StreamedMessagePart[] = []; + for await (const part of message) parts.push(part); + + expect(parts).toEqual([{ type: 'text', text: 'Hello' }]); + expect(message.id).toBe('resp-1'); + expect(message.finishReason).toBe('completed'); + expect(message.usage?.output).toBe(1); + }); +}); + +describe('GoogleGenAIChatProvider abort signal wiring', () => { + function stubGenerateContentStream( + provider: ChatProvider, + impl: (params: Record) => Promise>>, + ): void { + const client = sdkClient(provider) as { models: { generateContentStream: unknown } }; + client.models.generateContentStream = vi.fn().mockImplementation(impl); + } + + async function* googleChunkStream(): AsyncGenerator> { + yield { + candidates: [ + { content: { parts: [{ text: 'Hello' }], role: 'model' }, finishReason: 'STOP' }, + ], + usageMetadata: { promptTokenCount: 3, candidatesTokenCount: 1, totalTokenCount: 4 }, + responseId: 'resp-1', + }; + } + + it('passes the abort signal to the streaming transport config', async () => { + const provider = new GoogleGenAIChatProvider({ model: 'gemini-2.5-flash', apiKey: 'sk-probe' }); + let captured: Record | undefined; + stubGenerateContentStream(provider, (params) => { + captured = params; + return Promise.resolve(googleChunkStream()); + }); + const controller = new AbortController(); + + await drain(await provider.generate('', [], PROBE_HISTORY, { signal: controller.signal })); + + expect(captured).toBeDefined(); + expect((captured?.['config'] as Record)['abortSignal']).toBe( + controller.signal, + ); + }); + + it('passes the abort signal to the non-streaming transport config', async () => { + const provider = new GoogleGenAIChatProvider({ + model: 'gemini-2.5-flash', + apiKey: 'sk-probe', + stream: false, + }); + const controller = new AbortController(); + + const body = await captureGoogleBody(provider, { signal: controller.signal }); + + expect((body['config'] as Record)['abortSignal']).toBe(controller.signal); + }); + + it('cancels the pending transport read when the signal aborts mid-stream', async () => { + const provider = new GoogleGenAIChatProvider({ model: 'gemini-2.5-flash', apiKey: 'sk-probe' }); + const controller = new AbortController(); + let transportSignal: AbortSignal | undefined; + let transportReadCancelled = false; + let reachPendingRead!: () => void; + const pendingRead = new Promise((resolve) => { + reachPendingRead = resolve; + }); + stubGenerateContentStream(provider, (params) => { + transportSignal = (params['config'] as Record)['abortSignal'] as AbortSignal; + const signal = transportSignal; + return Promise.resolve({ + [Symbol.asyncIterator]() { + return { + next: () => + new Promise>>((_, reject) => { + signal.addEventListener( + 'abort', + () => { + transportReadCancelled = true; + reject(new DOMException('The operation was aborted.', 'AbortError')); + }, + { once: true }, + ); + reachPendingRead(); + }), + }; + }, + }); + }); + + const message = await provider.generate('', [], PROBE_HISTORY, { + signal: controller.signal, + }); + const failurePromise = drain(message).then( + () => null, + (error: unknown) => error, + ); + await pendingRead; + controller.abort(); + + const failure = await failurePromise; + expect(failure).toBeInstanceOf(DOMException); + expect((failure as DOMException).name).toBe('AbortError'); + expect(transportSignal).toBe(controller.signal); + expect(transportReadCancelled).toBe(true); + }); + + it('normalizes a non-DOMException transport error to AbortError once the signal has aborted', async () => { + const provider = new GoogleGenAIChatProvider({ model: 'gemini-2.5-flash', apiKey: 'sk-probe' }); + const controller = new AbortController(); + stubGenerateContentStream(provider, () => + Promise.resolve({ + [Symbol.asyncIterator]() { + return { + next: () => + Promise.reject>>(new Error('socket hang up')), + }; + }, + }), + ); + + const message = await provider.generate('', [], PROBE_HISTORY, { + signal: controller.signal, + }); + controller.abort(); + + const failure = await drain(message).then( + () => null, + (error: unknown) => error, + ); + expect(failure).toBeInstanceOf(DOMException); + expect((failure as DOMException).name).toBe('AbortError'); + }); + + it('still converts a mid-stream transport failure when the signal is not aborted', async () => { + const provider = new GoogleGenAIChatProvider({ model: 'gemini-2.5-flash', apiKey: 'sk-probe' }); + stubGenerateContentStream(provider, () => + Promise.resolve({ + [Symbol.asyncIterator]() { + return { + next: () => + Promise.reject>>(new Error('fetch failed')), + }; + }, + }), + ); + const controller = new AbortController(); + + const message = await provider.generate('', [], PROBE_HISTORY, { + signal: controller.signal, + }); + const failure = await drain(message).then( + () => null, + (error: unknown) => error, + ); + + expect(failure).toBeInstanceOf(APIConnectionError); + }); +}); + describe('resolveProviderEndpoint', () => { it('resolves the kimi endpoint chain from process.env', () => { process.env['KIMI_API_KEY'] = 'sk-kimi-env'; diff --git a/packages/kap-server/src/services/transcript/coreEventMap.ts b/packages/kap-server/src/services/transcript/coreEventMap.ts index ed740453fe..9b765fdb9a 100644 --- a/packages/kap-server/src/services/transcript/coreEventMap.ts +++ b/packages/kap-server/src/services/transcript/coreEventMap.ts @@ -517,10 +517,13 @@ export class AgentTranscriptProjector { } /** - * `turn.step.retrying` — a claimed provider failure is being retried on the - * same step. The step stays 'running' with the retry detail on the header; - * the terminal step upsert simply carries no `retry`, which clears it - * (step.upsert replaces the whole header). + * `turn.step.retrying` — a claimed provider failure is being retried. The + * engine runs the retry as a fresh step ordinal, so this event is the failed + * attempt's terminal signal: converge its open frames and mark the step + * interrupted, keeping the retry detail on the header as historical context + * (which attempt failed, what is scheduled next). Without the terminal + * upsert the attempt would linger as a forever-running step beside the + * retried output. */ private onStepRetrying(event: { turnId: number; @@ -534,6 +537,7 @@ export class AgentTranscriptProjector { statusCode?: number; }): TranscriptOperation[] { const ops: TranscriptOperation[] = []; + this.flushOpenFrames(ops); const turnId = `t${event.turnId}`; const stepId = `${turnId}.${event.step}`; const prev = this.currentStep?.stepId === stepId ? this.currentStep : undefined; @@ -542,8 +546,11 @@ export class AgentTranscriptProjector { stepId, turnId, ordinal: event.step, - state: 'running', + state: 'interrupted', startedAt: prev?.startedAt, + endedAt: nowIso(), + endReason: 'error', + endMessage: event.errorMessage, retry: { failedAttempt: event.failedAttempt, nextAttempt: event.nextAttempt, diff --git a/packages/kap-server/test/services/transcript.test.ts b/packages/kap-server/test/services/transcript.test.ts index 81ed9a8bed..9c62f2ac74 100644 --- a/packages/kap-server/test/services/transcript.test.ts +++ b/packages/kap-server/test/services/transcript.test.ts @@ -579,14 +579,16 @@ describe('AgentTranscriptProjector', () => { expect(step.endMessage).toBe('user cancelled'); }); - it('sets retry on turn.step.retrying and clears it at the terminal upsert', () => { + it('terminates the failed attempt on turn.step.retrying and runs the retry as a fresh step', () => { const projector = new AgentTranscriptProjector('main'); const tx = new AgentTranscript('main'); const feed = (event: ProjectorBusEvent): void => void tx.apply(projector.map(event)); - const step = (): TranscriptTurn['steps'][number] => turnOps('t1', tx.getItems()).steps[0]!; + const steps = (): readonly TranscriptTurn['steps'][number][] => + turnOps('t1', tx.getItems()).steps; feed(ev({ type: 'turn.started', turnId: 1, origin: { kind: 'user' } })); feed(ev({ type: 'turn.step.started', turnId: 1, step: 1 })); + feed(ev({ type: 'assistant.delta', turnId: 1, delta: 'partial out' })); feed( ev({ type: 'turn.step.retrying', @@ -596,26 +598,44 @@ describe('AgentTranscriptProjector', () => { nextAttempt: 2, maxAttempts: 3, delayMs: 2000, - errorName: 'ProviderRateLimitError', - errorMessage: '429 too many requests', - statusCode: 429, + errorName: 'LLMStreamStalledError', + errorMessage: 'no output for 120000ms', }), ); - expect(step().state).toBe('running'); - expect(step().retry).toEqual({ + const failed = steps()[0]!; + expect(failed.state).toBe('interrupted'); + expect(failed.endReason).toBe('error'); + expect(failed.endMessage).toBe('no output for 120000ms'); + expect(failed.retry).toEqual({ failedAttempt: 1, nextAttempt: 2, maxAttempts: 3, delayMs: 2000, - errorName: 'ProviderRateLimitError', - errorMessage: '429 too many requests', - statusCode: 429, + errorName: 'LLMStreamStalledError', + errorMessage: 'no output for 120000ms', + statusCode: undefined, }); + expect(failed.frames).toEqual([ + expect.objectContaining({ kind: 'text', text: 'partial out' }), + ]); - feed(ev({ type: 'turn.step.completed', turnId: 1, step: 1 })); - expect(step().state).toBe('completed'); - expect(step().retry).toBeUndefined(); + feed(ev({ type: 'turn.step.started', turnId: 1, step: 2 })); + feed(ev({ type: 'assistant.delta', turnId: 1, delta: 'recovered' })); + feed(ev({ type: 'turn.step.completed', turnId: 1, step: 2 })); + + expect(steps()).toHaveLength(2); + expect(steps()[0]).toMatchObject({ + state: 'interrupted', + endReason: 'error', + retry: { failedAttempt: 1, nextAttempt: 2 }, + }); + expect(steps()[1]).toMatchObject({ state: 'completed', ordinal: 2 }); + expect(steps()[1]!.retry).toBeUndefined(); + expect(steps()[1]!.frames).toEqual([ + expect.objectContaining({ kind: 'text', text: 'recovered' }), + ]); + expect(steps().every((step) => step.state !== 'running')).toBe(true); }); it('fills durationMs / error / accumulated step usage on turn.ended', () => { diff --git a/packages/transcript/src/model/turn.ts b/packages/transcript/src/model/turn.ts index 8efda19aab..ad960d0e3c 100644 --- a/packages/transcript/src/model/turn.ts +++ b/packages/transcript/src/model/turn.ts @@ -47,9 +47,11 @@ export interface StepTiming { } /** - * A retry in flight on a running step. Set while retrying; the step's - * terminal upsert simply carries no `retry`, which clears it (step.upsert - * replaces the whole header). + * Retry detail of a failed step attempt. The engine runs each retry as a + * fresh step ordinal, so `turn.step.retrying` is the failed attempt's + * terminal signal: the projection marks that step interrupted and keeps this + * annotation on its header as historical detail (which attempt failed, what + * is scheduled next). */ export interface StepRetry { readonly failedAttempt: number;