Skip to content

fix(studio): guard generated CreateAgentToolset against deepcopy on pinned runtimes - #1115

Open
jdp-just-does-projects wants to merge 1 commit into
volcengine:mainfrom
jdp-just-does-projects:fix/generated-agent-runtime-deepcopy-guard
Open

jdp-just-does-projects wants to merge 1 commit into
volcengine:mainfrom
jdp-just-does-projects:fix/generated-agent-runtime-deepcopy-guard

Conversation

@jdp-just-does-projects

Copy link
Copy Markdown

Summary / 摘要

English: Add a __deepcopy__ guard to the CreateAgentToolset subclass that Studio writes into generated projects (quick_mode_compat.py), so a deployed agent no longer fails with TypeError: cannot pickle '_thread.RLock' object even when its pinned veadk-python predates #1113.

中文: 为 Studio 生成项目中 quick_mode_compat.py 里的 CreateAgentToolset 子类增加 __deepcopy__ 保护,使部署后的 Agent 即使固定安装的是 #1113 之前的 veadk-python 版本,也不会再报 TypeError: cannot pickle '_thread.RLock' object

Related / 相关:#1113 (merged), #1114 (open), #1087 (open)


English

Problem

A Studio-built agent with dynamic agent delegation enabled, deployed to an AgentKit runtime with APMPlus tracing (ENABLE_APMPLUS=true, which the deploy path sets by default), fails on its first message, before any model call:

TypeError: cannot pickle '_thread.RLock' object

The Studio UI shows this with the generic "check network settings such as the shared public egress" hint, which is misleading: the failure happens inside the agent process.

Root cause

  1. RetryingLiteLlm.generate_content_async snapshots the request with copy.deepcopy(llm_request) (releases up to and including 1.1.12).
  2. LlmRequest.tools_dict holds collect_resources / create_agents, which are bound methods of CreateAgentToolset.
  3. The toolset references its bootstrap agents. With APMPlus on, those agents hold an OpentelemetryTracer, whose span processor owns a lock.
  4. Deep-copying the request recursively reaches that lock and raises.

Object path found while reproducing:

llm_request.tools_dict['collect_resources'].func.__self__
  ._bootstrap_agents[0][0].tracers[0]
  ._global_tracer_provider._active_span_processor._lock

#1113 fixes the library by no longer deep-copying tools_dict. Generated projects, however, install a pinned veadk-python (_VEADK_VERSION), and the fix only reaches them once the pin is bumped. On main the pin is 1.1.11, which still has the old deepcopy(llm_request). #1114 bumps it to 1.1.13.

Fix

CreateAgentToolset in the generated quick_mode_compat.py now defines __deepcopy__, returning the same instance and recording it in memo. The toolset is shared runtime state (bootstrap agents, tracers, locks, caches) and should never be cloned. This makes generated agents safe regardless of which veadk-python the project pins. It complements #1113 and #1114 rather than replacing them:

It does not change the pin, so it does not conflict with #1114 and leaves the golden hashes unchanged (the golden drafts do not enable delegation).

Tests

  • New test_quick_mode_compat_toolset_survives_request_deepcopy: generates a delegation project, loads its quick_mode_compat.py, attaches an RLock to the toolset, and deep-copies a tools_dict-shaped mapping of its bound methods. Fails without the guard (TypeError: cannot pickle '_thread.RLock' object) and passes with it.
  • uv run pytest tests/cli/test_generated_agent_backend_codegen.py tests/cli/test_generated_agent_backend_codegen_extended.py tests/cli/test_generated_agent_component_matrix.py tests/models/test_retrying_lite_llm.py tests/tools -q: 385 passed, 1 skipped.
  • uv run --group dev pre-commit run --files ...: ruff check, ruff format, and secrets scan passed.
  • End-to-end (manual): regenerated a real failing agent's draft and ran it with ENABLE_APMPLUS=true against the pinned runtime veadk-python==1.1.11, google-adk==2.1.0:
    • Without the guard: TypeError: cannot pickle '_thread.lock' object.
    • With the guard: the agent replies normally.

Side note (not changed here): frontend/src/adk/runSseError.ts adds the shared-public-egress network hint to every run error, which makes in-agent failures like this one look like network problems.


中文

问题

在 Studio 中开启**动态子智能体(dynamic agent delegation)**的 Agent,部署到开启了 APMPlus 链路追踪ENABLE_APMPLUS=true,部署流程默认开启)的 AgentKit Runtime 后,发送第一条消息即失败,且发生在调用模型之前:

TypeError: cannot pickle '_thread.RLock' object

Studio 前端会附带“请检查共享公网出口等网络配置”的通用提示,但这具有误导性:错误实际发生在 Agent 进程内部,与网络无关。

根因

  1. RetryingLiteLlm.generate_content_async 通过 copy.deepcopy(llm_request) 保存请求快照(1.1.12 及之前版本)。
  2. LlmRequest.tools_dict 中的 collect_resources / create_agentsCreateAgentToolset 的绑定方法(bound method)。
  3. 该 toolset 引用了 bootstrap agents;开启 APMPlus 后,这些 agent 持有 OpentelemetryTracer,其 span processor 内部持有锁。
  4. 对请求做深拷贝时会递归到这个锁,从而抛出异常。

复现时定位到的对象路径:

llm_request.tools_dict['collect_resources'].func.__self__
  ._bootstrap_agents[0][0].tracers[0]
  ._global_tracer_provider._active_span_processor._lock

#1113 已在库层面修复:不再深拷贝 tools_dict。但 Studio 生成的项目会安装固定版本veadk-python_VEADK_VERSION),只有版本号更新后修复才会生效。目前 main 上固定的是 1.1.11,仍使用旧的 deepcopy(llm_request)#1114 会将其升级到 1.1.13

修复

生成的 quick_mode_compat.py 中的 CreateAgentToolset 现在定义了 __deepcopy__:返回同一个实例并写入 memo。toolset 属于共享运行时状态(bootstrap agents、tracer、锁、缓存),不应被复制。这样无论生成项目固定的是哪个 veadk-python 版本,Agent 都是安全的。本 PR 是对 #1113#1114 的补充,而非替代:

本 PR 不修改固定版本号,因此不会与 #1114 冲突,golden hash 也保持不变(golden 用例未开启动态子智能体)。

测试

  • 新增 test_quick_mode_compat_toolset_survives_request_deepcopy:生成开启动态子智能体的项目,加载其 quick_mode_compat.py,为 toolset 挂上 RLock,再对形如 tools_dict、包含其绑定方法的映射做深拷贝。未加保护时失败(TypeError: cannot pickle '_thread.RLock' object),加保护后通过。
  • uv run pytest tests/cli/test_generated_agent_backend_codegen.py tests/cli/test_generated_agent_backend_codegen_extended.py tests/cli/test_generated_agent_component_matrix.py tests/models/test_retrying_lite_llm.py tests/tools -q:385 通过,1 跳过。
  • uv run --group dev pre-commit run --files ...:ruff check、ruff format、密钥扫描均通过。
  • 端到端(手动):用一个真实出错 Agent 的草稿重新生成项目,在 ENABLE_APMPLUS=true 下使用固定的运行时版本 veadk-python==1.1.11google-adk==2.1.0 运行:
    • 未加保护:TypeError: cannot pickle '_thread.lock' object
    • 加保护后:Agent 正常回复。

附注(本 PR 未修改):frontend/src/adk/runSseError.ts 会给所有运行错误附加“共享公网出口”网络提示,导致此类 Agent 内部错误看起来像网络问题。

🤖 Generated with Claude Code

https://claude.ai/code/session_01VMFeYUADAnuzxhHCyFJA4N

Generated projects pin veadk-python, and releases before volcengine#1113 deep-copy
LlmRequest for 429 retries. tools_dict holds bound methods of the
generated CreateAgentToolset, which reaches bootstrap agents, their
OpenTelemetry tracers and span processor locks, so agents with dynamic
delegation and APMPlus fail with "cannot pickle '_thread.RLock' object".

Return the shared toolset instance from __deepcopy__ so generated agents
work regardless of the pinned runtime version.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VMFeYUADAnuzxhHCyFJA4N
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant