feat(ai): add agent hooks and OpenTelemetry tracing

[chaojixinren]

Summary

Introduces a config-driven, telemetry-agnostic agent hooks system in ai/, with
logging and OpenTelemetry (OTLP) tracing as the first observers. It instruments
the full ReAct lifecycle — interaction, iteration, stage, model call, and tool call —
without coupling the strategy code to any specific backend.

Motivation

Resolves #1525. There was previously no consistent way to observe agent execution or trace
a request end-to-end across the Agent ↔ MCP boundary, making agent behavior hard to debug
and spans impossible to correlate.

What changed

• Hook manager (ai/component/hooks/): immutable lifecycle events
  (interaction / iteration / stage / model_call / tool_call × start / end), read-only
  State snapshots, per-event registration, and panic isolation per hook.
• Context derivation: hooks may derive a context.Context for nested work; exactly one
  DerivesContext registration is accepted (a Go context carries one span lineage — fan out
  via a Collector instead). Contexts returned by plain observational hooks are ignored.
• Tracing hook: OTLP exporter (gRPC or HTTP/protobuf) with GenAI semantic attributes
  (gen_ai.operation.name, gen_ai.request.model, gen_ai.provider.name,
  gen_ai.conversation.id, gen_ai.tool.name, gen_ai.tool.call.id,
  gen_ai.input/output.messages, gen_ai.usage.*, agent.fallback.*, error.type, …).
  Content is serialized lazily only when the span IsRecording(), and content capture is
  opt-in (capture_content: none default; truncated ≤ 4096 bytes; full).
• Logging hook: structured lifecycle logging with the same opt-in content capture.
• Trace propagation: W3C traceparent / tracestate / baggage are honored inbound,
  propagated to MCP HTTP calls, and the active trace ID is returned on SSE responses via
  X-Trace-ID (CORS-exposed).
• Fallback metadata: timeout vs parse-error are distinguished (FallbackReason), written
  to both the model-call span and the stage span, with correct Evidence text; tool failures
  are recorded as error.type + agent.degraded without faking a gen_ai.tool.call.result.
• Cancellation semantics: context.Canceled propagates cleanly (only DeadlineExceeded
  is a timeout); SSE disconnect cancellation stays detached from the running interaction.
• Configuration: type: hooks component with logging / tracing blocks, JSON schema
  validation, and standard OTel env vars for endpoint and credentials.

Design constraints

• Hooks are observational: they read state and may derive context, but must never mutate
  Agent execution data.
• Content capture defaults to none for credential/PII safety; payloads are not serialized
  on the hot path unless a matching hook opts in.
• Tool-call hooks must explicitly select tool names ("*" for all).

Agent Hooks + OTel Tracing — Completed Test Checklist

1. Unit Tests

• go test -count=1 ./... — all passed (hooks, Agent, server engine, MCP tools, runtime, etc.)

2. Integration Tests

• go test -tags=integration -count=1 ./... — all passed

3. Race / Static Analysis

• go test -race ./component/hooks/... ./component/agent/... — passed
• go vet ./... — passed

4. E2E: Jaeger (OTLP)

• Local Jaeger OTLP end-to-end passed
• Trace ID: f6477cd5b8d5a5cce9ae07a0ad1d8470

5. E2E: Langfuse (Docker 4.11.0)

• Full stack via official Compose; OTLP/HTTP ingestion succeeded
• v2 Observations API: HTTP 200
• Correct hierarchy: AGENT invoke_agent → GENERATION chat qwen-max → TOOL lookup_service
• Confirmed in ClickHouse events_full: model input/output, 7/5/12 tokens, session ID, tool call ID
• 3 observations written in total
• Trace ID: fdc809bacdcdbf5b7cd7798ba8f7c1cb

6. Performance Benchmarks (0 allocs)

Benchmark	Result
BenchmarkDisabledHookFastPath	2.967 ns/op · 0 allocs
BenchmarkEmptyManagerFastPath	8.943 ns/op · 0 allocs
BenchmarkHookContentDisabled	34.23 ns/op · 0 allocs
BenchmarkHookContentLoggingOnly	34.68 ns/op · 0 allocs

7. Cleanup

• Temporary test files removed
• Jaeger / Langfuse containers and dedicated Docker network removed
• No code or commit changes; HEAD remains 7bbcab1

---

⚠️ Environment Gaps (out of scope for this issue — not completed)

• Hosted Langfuse cloud E2E — requires credentials
• External DashScope E2E — requires credentials; TestMultiTurnConversation therefore shows 0/14
• External Milvus E2E — requires credentials

[robocanic]

@ambiguous-pointer please help review this PR.

[Copilot]

Pull request overview

Adds configurable agent lifecycle hooks and OpenTelemetry tracing across the ReAct, SSE, MCP, and runtime layers.

Changes:

• Introduces hook management, logging, OTLP tracing, and content-capture policies.
• Instruments agent/model/tool lifecycles with propagation and fallback metadata.
• Adds shutdown handling, configuration, and extensive tests.

Reviewed changes

Copilot reviewed 39 out of 40 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ai/test/e2e/e2e_test.go	Wires hooks into E2E runtime.
ai/schema/json/hooks.schema.json	Defines hooks configuration schema.
ai/main.go	Registers hooks and adjusts lifecycle order.
ai/go.sum	Records HTTP OTLP dependencies.
ai/go.mod	Adds OpenTelemetry dependencies.
ai/config/test/loader_test.go	Tests hooks configuration defaults.
ai/config/loader.go	Maps hooks to its schema.
ai/config.yaml	Enables the hooks component.
ai/component/tools/engine/mcp_tools.go	Propagates trace headers to MCP.
ai/component/tools/engine/mcp_tools_test.go	Tests outbound trace propagation.
ai/component/server/engine/sse/sse.go	Centralizes CORS handling.
ai/component/server/engine/router.go	Allows and exposes tracing headers.
ai/component/server/engine/router_test.go	Tests tracing CORS headers.
ai/component/server/engine/handlers.go	Propagates context and trace IDs.
ai/component/server/engine/handlers_test.go	Tests context and detached output handling.
ai/component/hooks/tracing.go	Implements lifecycle span creation.
ai/component/hooks/tracing_test.go	Tests spans, attributes, and capture.
ai/component/hooks/README.md	Documents hooks and OTLP setup.
ai/component/hooks/manager.go	Implements hook registration and dispatch.
ai/component/hooks/manager_test.go	Tests manager behavior and concurrency.
ai/component/hooks/jaeger_e2e_test.go	Adds optional Jaeger verification.
ai/component/hooks/hooks.yaml	Provides default hooks configuration.
ai/component/hooks/factory.go	Adds the hooks factory.
ai/component/hooks/event.go	Defines lifecycle events and snapshots.
ai/component/hooks/component.go	Implements hooks component lifecycle.
ai/component/hooks/component_test.go	Tests configuration and shutdown.
ai/component/agent/react/steps.go	Instruments stages, models, and tools.
ai/component/agent/react/step_test.go	Tests fallback and hook emissions.
ai/component/agent/react/react.go	Adds interaction tracing and draining.
ai/component/agent/react/prompt.go	Retains stage names and models.
ai/component/agent/react/page_context_test.go	Updates context construction test.
ai/component/agent/react/orchestrator.go	Instruments iterations and stages.
ai/component/agent/react/orchestrator_test.go	Tests lifecycle sequencing.
ai/component/agent/react/lifecycle_test.go	Tests cancellation and concurrency.
ai/component/agent/react/hook_content.go	Converts messages for telemetry.
ai/component/agent/react/hook_content_test.go	Tests semantic message conversion.
ai/component/agent/react/component.go	Connects hooks and agent lifecycle.
ai/component/agent/react/component_wiring_test.go	Tests hooks manager wiring.
ai/component/agent/fallback/handler.go	Reports fallback parsing usage.
ai/component/agent/agent.go	Adds context-aware interaction channels.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ambiguous-pointer]

@chaojixinren 上述是我的一些个人拙见，可以按照您的设计进行实际的一些调整和修改 : )

[ambiguous-pointer]

Go 1.26 下 sonic v1.14.1 无法编译, 请先行合入远程更改

[ambiguous-pointer]

• 潜在问题：

  1. 没有错误/降级/取消类事件。工具失败、observe 超时/解析失败、显式取消，在 PR 里都只是 State 上的字段（Degraded/FallbackUsed/Error）。对 tracing/logging 两个消费方够用（span 上有 error.type、agent.degraded 属性），
     但对未来的 metrics / 审计类 hook（它们需要"按事件种类订阅"）就缺了第一类公民：
     • 例：生产上想统计"每月 observe 超时次数"或"工具失败率"，现在只能订阅 stage.end 然后过滤 State.FallbackUsed，
       而不是订阅一个语义明确的 agent.degraded 事件；若未来事件字段演进，这类消费方会静默错算。

• 个人建议: 事件种类扩展为 agent/stage/llm/tool × start/end/error + agent.degraded +
  agent.cancel + llm.chunk（预留），并给 State 增加 Seq uint64。事件字段只读约定保持。因为模型部署侧可能不一定都是稳定的模型，例如 VLLM 私有化部署的时候，工具调用参数模板没有绑定正确的时候，调用工具会出现偶发性的直接中断。所以会需要预设详细一些

• 生产场景：Dubbo 服务诊断场景（agent 通过 MCP 调 get_service_detail/诊断工具 feat: add PromQL and trace diagnosis tools #1499）——SRE 想要"按工具维度"的失败率报表，若没有独立 tool.error 事件种类，报表逻辑要散落在每个消费方里重复过滤，接入点越多越容易漏。

[ambiguous-pointer]

• 潜在问题：lazyContent 字段（manager.go:44,96）只有包内 NewTracingRegistration 能用（未导出），
  外部捕获内容的 hook 一律走 manager.go:193-195 的急切快照分支——也就是说"懒"只对内置 tracing hook 成立，对将来第三方内容型 hook（如审计）不成立。PR文档里"Content is serialized lazily"的表述容易误导。
• 个人建议：把 lazyContent 语义并入公开的 Registration（如 CaptureContent: CaptureLazy）或至少在
  Registration 上注释清楚两档行为。
• 生产场景：审计 hook 需要"模型输入/输出原文"留档——如果它被急切序列化，每次模型调用都会多一次完整 JSON marshal（大对话可能几百 KB），生产热点路径上不可忽略；同时内容进内存=更大的 PII 暴露面。应能声明"延迟到真正落盘前才序列化"。

[ambiguous-pointer]

// ← 这里没有 go discardAgentOutput(channels)！

对比另外两条退出路径（handlers.go:104-107 和 134-136）

1. handler 在流中途 panic（比如 MessageDelta 遇到未知 final 类型、或任何将来加的代码）；
2. defer recover 触发 → 发一条 internal_error SSE → handler 返回 → gin 关连接；
3. 交互 goroutine 不会死——interactionCtx := context.WithoutCancel(extractedCtx)（handlers.go:87）已经把请求取消剥掉了，客户端断开对它是透明的；
4. 交互 goroutine 继续生成，调用 chans.Send（agent.go:62-70）：

func (chans *Channels) Send(sf *schema.StreamFeedback) {
	sf.SetIndex(chans.nextIndex)
	chans.nextIndex++
	chans.UserRespChan <- sf   // ← 有界阻塞发送，缓冲满就永久卡住
}

5. 缓冲（bufferSize）塞满 ~16 条后，没有任何人排空 → 交互 goroutine 永久阻塞在 Send；
6. 连锁反应：阻塞在 Send 意味着 goroutine 的 defer 永远不执行——interaction.end 事件发不出去（trace 缺尾）、finishInteraction 不执行（ra.active 表里的条目永不删除）；
7. 进程关闭时 Stop()（react.go:207-222）对这条交互执行 activeWG.Wait() → Stop 也跟着挂起，直到外层 20s 超时兜底，关闭质量劣化。

一句话：handler 侧一个 panic，产生一个永久阻塞的 goroutine + 一条不完整的 trace + Stop 挂起——而这一切本来用一行 go discardAgentOutput(channels) 就能避免。

根子是 Channels.Send 的阻塞语义——discardAgentOutput 只是防呆补丁，而且只在 handler 这一侧有

[ambiguous-pointer]

个人理解 这个和整个 agent 调用的 : 交互持久化 + 事件日志 + 整段重放 有关
可能得 #1534 完成后全面的思考一下这个地方如何实现

[ambiguous-pointer]

这里可能对于开发者或者需要基于钩子实现私有化能力的时候,会存在 我不知道有哪些 hook、不知道接入点

type: hooks
spec:
  hooks:
    - name: "logging"
      enabled: true
      events: ["agent.start", "agent.end", "agent.error", "agent.degraded", "agent.cancel",
               "stage.start", "stage.end", "stage.error", "llm.start", "llm.end", "llm.error",
               "tool.start", "tool.end", "tool.error"]
      config: { level: "info" }

[ambiguous-pointer]

events 字段以 enum 形式列出全部事件种类

dubbo-admin/ai/schema/json/memory.schema.json

Lines 15 to 20 in 5d6df6e

	"properties": {
	"history_key": {
	"type": "string",
	"enum": ["chat_history", "system_memory", "core_memory"],
	"default": "chat_history"
	},

[chaojixinren]

@chaojixinren 上述是我的一些个人拙见，可以按照您的设计进行实际的一些调整和修改 : )

感谢详细的 review！

已修复：

• sonic 已升级到 v1.15.2，解决 Go 1.26 编译问题
• 新增 error/cancel/degrade 事件类型，错误事件在对应 .end 事件之前发射
• 已在 Registration.CaptureContent 注释中明确外部 hook 走急切快照，lazyContent 仅供内置 hook 使用。导出懒序列化 API 涉及设计权衡，建议后续单独 issue 跟踪
• panic drain 在原 commit 就修了，handlers.go 增加 discardAgentOutput + 回归测试
• hooks.yaml 和 README 都加了完整事件列表文档

说明：

• 认同 panic drain 修复与持久化层的关联
• 当前 hooks.yaml 只配置内置 hook 开关，不支持配置化 hook 注册（Registration 在代码里构造），所以 schema 暂无 events 字段需要 enum。如果未来支持配置化注册，确实需要加 enum 提供补全

[robocanic]

@larry-zy

[robocanic]

@chaojixinren please merge the develop branch and resolve the conflicts.

[chaojixinren]

@robocanic Merged the latest develop and ai branches into this PR branch and resolved the conflicts. Tests and builds passed; GitHub now shows the PR as mergeable. Ready for another review. Thanks!

[robocanic]

@chaojixinren 上述是我的一些个人拙见，可以按照您的设计进行实际的一些调整和修改 : )

感谢详细的 review！

已修复：

• sonic 已升级到 v1.15.2，解决 Go 1.26 编译问题
• 新增 error/cancel/degrade 事件类型，错误事件在对应 .end 事件之前发射
• 已在 Registration.CaptureContent 注释中明确外部 hook 走急切快照，lazyContent 仅供内置 hook 使用。导出懒序列化 API 涉及设计权衡，建议后续单独 issue 跟踪
• panic drain 在原 commit 就修了，handlers.go 增加 discardAgentOutput + 回归测试
• hooks.yaml 和 README 都加了完整事件列表文档

说明：

• 认同 panic drain 修复与持久化层的关联
• 当前 hooks.yaml 只配置内置 hook 开关，不支持配置化 hook 注册（Registration 在代码里构造），所以 schema 暂无 events 字段需要 enum。如果未来支持配置化注册，确实需要加 enum 提供补全

@chaojixinren 我觉得配置化hook注册长期来看是需要的，并且不管配置化还是非配置化，events字段都需要枚举列出来，你可以评估一下支持配置化改动大不大，如果不大的话，就在同一个PR上追加commit。如果改动大，就新开一个PR

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[chaojixinren]

@robocanic 已评估并在这个 PR 中补充了配置化 Hook 注册：

• 支持通过 YAML 注册内置 logging/tracing，logging 可按事件、工具名称筛选并设置日志级别。
• 明确枚举并校验全部 16 个生命周期事件。

改动集中在 hooks 组件及其配置 Schema，没有引入动态插件机制。单元测试、竞态检测和编译检查已通过，真实 Jaeger 的 gRPC/HTTP 导出也已验证。实际 HTTP 对话已验证到本地模型完成工具调用。

麻烦再帮忙看一下，谢谢！

[ambiguous-pointer]

最初的 Hook 实现（0114c79）：40-50 个文件
之后合并了 develop 分支（0d1d89a）：引入了流量规则版本历史、应用依赖图、事件流等大量功能
再次同步了 ai 分支（cc08248）：保留了 Hook 系统

这个地方应该不必合入develop相关内容 😂

---

⚠️ 潜在问题

Comment 1: 内存泄露风险

## ⚠️ 潜在内存泄露：活动交互映射无限增长

**位置**: `ai/component/agent/react/react.go:beginInteraction()`

**问题**:
```go
ra.active[interactionID] = cancel  // interactionID 不会自动清理

ra.active map 会无限增长，因为即使 finishInteraction() 删除了键，interactionID 是全局唯一的UUID，不会重复。随着时间推移，这会导致内存持续增长。

建议修复:

1. 定期清理过期条目（LRU 策略）
2. 添加最大容量限制
3. 或为 interactionID 使用可回收的池

测试建议:

func TestNoMemoryLeakOnManyInteractions(t *testing.T) {
  // 循环创建 10000 个交互后停止
  // 验证 ra.active 长度保持有界
}

---

#### **Comment 2: 并发安全问题**

⚠️ 并发修改风险：Hook 注册期间的遍历

位置: ai/component/hooks/manager.go:Emit()

问题:
当一个 goroutine 在 Emit() 遍历 registrations 时，另一个 goroutine 可能在 Register() 中修改列表：

// 线程 A：正在遍历
for _, reg := range registrations {  // 虽然copy了，但原列表可能改变
  // 使用 registrations...
}

// 线程 B：同时修改
m.registrations = append(m.registrations, newReg)  // 增长可能导致问题

虽然 copy 对当前迭代是安全的，但 NeedsContent() 方法直接访问 m.registrations，会导致 TOCTOU 竞态。

修复:

func (m *Manager) NeedsContent(event Event, toolName string) bool {
  m.mu.RLock()  // 这里已有保护，但可改进
  defer m.mu.RUnlock()
  // ... 检查逻辑
  // 但要保证不在锁内做耗时操作
}

验证:
使用 Go race detector 运行所有测试：

go test -race ./component/hooks/...

---

#### **Comment 4: 性能问题**

⚡ 性能问题：热路径中的不必要分配

位置: ai/component/hooks/manager.go:Emit()

问题:
每次 Emit() 都执行以下操作：

registrations := make([]compiledRegistration, len(m.registrations))
copy(registrations, m.registrations)  // 每次都分配 + 复制

在高频调用的 Agent 交互中（model call, tool call 每次交互都触发多次），这会产生大量垃圾：

基准测试:

BenchmarkEmptyManagerFastPath         // 0 alloc 是好的（当前代码）
BenchmarkWithoutHooksNilManager      // 可以进一步优化

// 但当有 registrations 时：
BenchmarkWithHooksEmit                // alloc 数量过多

改进:

1. 使用 sync.Pool 复用切片
2. 或改为 lock-free 结构（原子CAS）
3. 或缓存 registrations 快照（需要版本号控制）

回归测试:

func TestEmitAllocationBudget(t *testing.T) {
  m := NewManager(nil)
  m.Register(...)  // 注册几个 hook
  
  var allocs int64
  // 测量 1000 次 Emit() 的分配数
  // 期望: allocs < 1000 (ideally < 100)
}

---

#### **Comment 5: 测试覆盖缺陷**

📊 测试缺陷：缺少关键场景

缺失的测试场景:

1. 内存泄露:

   // 需要添加到 hook_loop_test.go
   func TestNoMemoryLeakOnConcurrentEmits(t *testing.T) {
     m := NewManager(...)
     m.Register(...)
     
     // 并发发送 10000 个事件，内存不应线性增长
     before := getHeapSize()
     
     for i := 0; i < 10000; i++ {
       go m.Emit(ctx, State{...})
     }
     
     time.Sleep(100*time.Millisecond)
     after := getHeapSize()
     
     // Verify: (after - before) < X MB
   }

2. 超大 Payload:

   func TestTracingHookWithLargeContent(t *testing.T) {
     // 10 MB JSON payload
     largeState := State{...WithInputContent(huge)}
     // 验证截断和避免OOM
   }

3. 竞态条件 (已有 race detector，但可加强):

   # 应该在 CI 中运行
   go test -race -count 10 ./...

---