Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,23 @@ VeADK includes a Python SDK, CLI/runtime integrations, and AgentKit Studio with
## Language and document ownership

- Keep this repository-wide development standard in English. Use English for new code identifiers, comments, and docstrings; user-facing text follows the owning area's localization rules.
- Keep durable change intent, scope, tasks, evidence, and acceptance in the PR/MR description, linked issue, external design document, or affected user documentation when the change needs long-lived context. Do not create repository-local design or component-spec files by default.
- Do not create standalone PRD or component-spec files, or recreate the root `prd-spec/` and `specs/` directories, as part of routine development. Create such documents only when the user explicitly requests them; generic skills or workflow templates do not make them mandatory.
- Discuss change-specific intent, scope, design decisions, and acceptance criteria in the conversation or PR description. Update existing maintained documentation when an affected interface or behavior needs clarification.
- When updating existing bilingual documentation, keep its language counterparts semantically equivalent in the same change, including identifiers, commands, examples, limits, and acceptance criteria.
- Before modifying frontend code, read `frontend/SPEC.md`; also apply it to its supporting backend interfaces as specified there. It remains the Studio development standard and is not replaced or translated by this document.
- Preserve existing user-documentation conventions: the documentation site uses Chinese `.mdx` and English `.en.mdx`; bilingual module/example READMEs use English `README.md` and Chinese `README.zh.md`.

## Required development workflow

1. **Inspect and agree on scope.** Read the affected implementation, tests, and contracts. When the user describes a requirement, propose an implementation plan and wait for the user to be satisfied before making edits. Distinguish verified facts from assumptions.
2. **Plan before implementation.** For features, refactors, and bugfixes, size the change against the affected contracts and risk. Record background/evidence, goals, non-goals, affected files, tasks, tests, risks, and acceptance criteria in the PR/MR description, linked issue, external design document, or user documentation when the change needs durable context.
3. **Assess contracts.** Identify changes to ownership, APIs, configuration, state/data, events, permissions, security, compatibility, runtime behavior, or observability. Record the contract impact or no-impact rationale in the PR/MR description, linked issue, external design document, or user documentation as appropriate. Do not backfill unrelated contracts.
4. **Clear scope review.** Review plan consistency, feasibility, boundaries, errors, security, compatibility, and testability. Resolve blockers and record user approval before modifying production code or tests.
5. **Implement against the approved design.** Map changes to requirements/tasks, write a failing regression or contract test first, then implement and refactor within scope. Contract or scope changes return to design review before implementation proceeds.
2. **Plan before implementation.** Explain the implementation approach, affected areas, risks, and verification plan in the conversation. Scale the detail to the change; no separate design document is required.
3. **Assess component contracts.** Check affected ownership, APIs, configuration, state/data, events, permissions, security, compatibility, runtime behavior, and observability. Update relevant existing documentation when necessary; do not generate separate component specs or no-impact reports by default.
4. **Review the approach.** Check feasibility, boundaries, errors, security, compatibility, and testability before editing. Resolve blockers and obtain user agreement for material scope changes. No standalone design review artifact or PRD/spec approval gate is required.
5. **Implement within the agreed scope.** Write a failing regression or contract test first for bugfixes, then implement and refactor within scope. Reassess the plan when contracts or scope change.
6. **Verify and review.** Run affected tests first, then required repository and component gates. Review both implementation correctness and edge cases, resource handling, security, test quality, and contract alignment. Fix blocking findings and rerun affected checks.
7. **Reconcile and deliver.** Synchronize affected user docs/examples and required generated artifacts. Record actual verification commands, results, unverified areas, and remaining risks. All agreed tasks and acceptance criteria must be satisfied before declaring completion; deferred scope requires user agreement.
7. **Reconcile and deliver.** Update affected existing user docs/examples, their applicable language counterparts, and required generated artifacts. Summarize actual verification commands, results, unverified areas, and remaining risks in the conversation or PR description. All agreed tasks and acceptance criteria must be satisfied before declaring completion; deferred scope requires user agreement.

Pure documentation, translation, or formatting changes that preserve runtime behavior and public contracts do not need a separate change design. They still require an agreed approach, applicable bilingual updates where the owning document convention requires them, and proportionate checks. This exception includes maintaining these development guidelines; it does not exempt code or behavioral changes.
Documentation, translation, formatting, and development-guideline changes require an agreed approach, applicable updates to existing language counterparts, and proportionate checks. They do not require a separate design document.

## Implementation boundaries

Expand Down Expand Up @@ -57,7 +59,7 @@ Use the repository-local Python environment and current `pyproject.toml`, `pytes
- Before committing, run `uv run --extra dev pre-commit run --all-files` and unit tests after synchronizing the branch as described below. Pre-commit includes Ruff and secret scanning; it does not replace browser, integration, or runtime verification.
- Prefer targeted checks during iteration. Expand to affected integration/E2E and regression gates for shared or high-risk changes and final delivery; explain any omitted required check.
- Record check outcomes as `pass`, `fail`, `blocked`, `not_run`, or `not_applicable`, with reasons for the last three. Include the tested revision/diff scope and execution date. Skipped tests do not prove the relevant behavior.
- Keep change-specific review and verification summaries in the PR/MR description, linked issue, external design document, or affected user documentation as appropriate. Separate evidence artifacts are optional when useful; OpenSpec and a multi-report hierarchy are not prerequisites.
- Keep change-specific review and verification summaries in the conversation or PR/MR description by default, with links to existing issues or external documentation when useful. Do not generate PRD/spec files, OpenSpec records, or a multi-report hierarchy unless the user requests them.

## Git and execution safety

Expand Down
27 changes: 15 additions & 12 deletions docs/content/docs/framework/tracing.en.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -189,20 +189,23 @@ In a multi-agent scenario, spans are linked through `parent_span_id`, showing th
</Callout>


## Request, model, and first-token timestamps
## Request spans and streaming TTFT

VeADK's default LiteLLM and Ark Responses adapters add the following extension attributes to model spans when OpenTelemetry tracing is enabled. UTC Unix nanoseconds are used throughout.
VeADK uses native OpenTelemetry span start/end times to describe execution boundaries. VeADK AgentKit/Harness HTTP applications create an `http_request` server span at ASGI ingress, before body parsing, and end it when the application finishes. Direct VeADK Runner execution creates a `runner_request` span; a Runner invoked inside the HTTP request reuses that request scope. These boundaries exclude client/network/gateway delays before the application receives the request.

| Attribute | Meaning |
Each LiteLLM Chat or Ark Responses SDK request creates a `model_request` client span under the owning model span. Its native start time is the same local SDK-request timestamp used to calculate TTFT; it is not provider inference start. VeADK-issued retries/fallbacks create separate spans. Hidden provider/LiteLLM retries remain within the observed SDK call.

| Attribute | Type / meaning |
| --- | --- |
| `veadk.request.start_time_unix_ns` | HTTP ASGI entry in VeADK AgentKit/Harness apps, or direct VeADK Runner execution entry. |
| `veadk.request.start_source` | `http` or `runner`; an inherited HTTP timestamp takes precedence. |
| `veadk.model.start_time_unix_ns` | Start of one adapter SDK request, not provider inference start. |
| `veadk.model.streaming` | Actual model SDK request mode, independent of HTTP SSE transport. |
| `veadk.model.first_token_time_unix_ns` | Receipt of the first nonempty generated content delta. |
| `veadk.model.first_token_type` | `text`, `reasoning`, or `tool_call`. |
| `veadk.model.first_token_status` | `observed`, `non_streaming`, `no_output`, or `unsupported`. |
| `gen_ai.span.kind` | `llm` on an SDK request span. |
| `gen_ai.request.model` | Actual requested model. |
| `gen_ai.request.stream` | Boolean actual SDK streaming mode, independent of HTTP SSE or Runner configuration. |
| `gen_ai.chat_completions.streaming_time_to_first_token` | Floating-point seconds from SDK request start to its first generated delta. |

The first nonempty text, reasoning, or tool-call delta records TTFT once. Metadata, empty deltas, usage, heartbeats, and completion-only aggregates do not count. Non-streaming, no-output, or uninstrumented paths omit TTFT; unknown SDK mode is not inferred from Runner configuration. One chunk can contain multiple tokens, so this is an SDK-observed duration, not a provider-side per-token timestamp.

The same value is recorded once in the `gen_ai.chat_completions.streaming_time_to_first_token` Histogram, unit `s`, using the existing global MeterProvider. Dimensions are `gen_ai.request.model` and `gen_ai.operation.name=chat`. Metrics do not depend on token usage or trace sampling; export requires a configured metrics pipeline. Negative wall-clock intervals are omitted with a diagnostic; genuine measured zero is valid.

First-token capture excludes role/id/signature metadata, empty deltas, usage, heartbeats, and completion-only aggregates. It does not require token usage or content tracing. One chunk may contain several tokens, so this is an SDK-observed first-output timestamp. Non-streaming calls have no first-token timestamp; completion time is never substituted.
Failed model requests set native Span Status to `ERROR` and `error.type` to `llm_error`, `timeout`, or `cancelled`. Request spans use `request_error`, `timeout`, or `cancelled`; HTTP 5xx responses use `http_error`. Explicit generator closure is not an error. Already observed TTFT survives interruption, and exception text is not added by this instrumentation.

Each SDK request creates a `model_request` child span; the owning model span also receives the latest attempt's fields. VeADK-issued retries and fallbacks create separate attempts. Internal LiteLLM/provider retries remain within the observed SDK request (`veadk.model.request_scope=sdk_request`). Unsupported external/custom engines are explicitly marked; invocation span start is not used as model start. Applications outside the VeADK HTTP factories must install the request timing middleware or use VeADK Runner to establish a request timestamp. Client-render timing and TTFT/TPOT histogram reporting are separate capabilities.
The instrumentation does not export custom `veadk.request.*` or `veadk.model.*` attributes or a custom first-token event. Start and first-output values remain internal calculation state. This replaces the previous custom timing fields without compatibility aliases. Client-render timing and TPOT histogram reporting remain outside this feature.
27 changes: 15 additions & 12 deletions docs/content/docs/framework/tracing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -187,20 +187,23 @@ Tracing 数据由一系列 **Span** 组成,每个 Span 记录一次操作(
</Callout>


## 请求、模型与首 Token 时间戳
## 请求 Span 与流式 TTFT

启用 OpenTelemetry Tracing 后,VeADK 默认 LiteLLM 和 Ark Responses 适配器会在模型 Span 上添加以下扩展属性。时间戳统一采用 UTC Unix 纳秒。
VeADK 使用 OpenTelemetry Span 原生起止时间表达执行边界。VeADK AgentKit/Harness HTTP 应用在 ASGI 入口、请求体解析前创建 `http_request` server Span,在应用执行结束时关闭。直接执行 VeADK Runner 时创建 `runner_request` Span;在 HTTP 请求中调用 Runner 时复用该请求范围。上述边界不包含应用接收请求之前的客户端、网络和网关耗时。

| 属性 | 含义 |
每次 LiteLLM Chat 或 Ark Responses SDK 请求,在所属模型 Span 下创建 `model_request` client Span。其原生开始时间与 TTFT 计算使用的本地 SDK 请求起点相同,不代表服务端开始推理。VeADK 发起的重试/降级分别创建 Span;供应商/LiteLLM 内部不可观测的重试仍属于同一次 SDK 调用。

| 属性 | 类型 / 含义 |
| --- | --- |
| `veadk.request.start_time_unix_ns` | VeADK AgentKit/Harness 应用的 HTTP ASGI 入口,或直接调用 VeADK Runner 的执行入口。 |
| `veadk.request.start_source` | `http` 或 `runner`;继承的 HTTP 时间戳优先。 |
| `veadk.model.start_time_unix_ns` | 一次适配器 SDK 请求的开始,不是服务端开始推理。 |
| `veadk.model.streaming` | 实际模型 SDK 请求模式,与 HTTP SSE 传输独立。 |
| `veadk.model.first_token_time_unix_ns` | 收到首个非空生成内容增量的时间。 |
| `veadk.model.first_token_type` | `text`、`reasoning` 或 `tool_call`。 |
| `veadk.model.first_token_status` | `observed`、`non_streaming`、`no_output` 或 `unsupported`。 |
| `gen_ai.span.kind` | SDK 请求 Span 上为 `llm`。 |
| `gen_ai.request.model` | 实际请求的模型。 |
| `gen_ai.request.stream` | 布尔值,表示实际 SDK 流式模式,不依赖 HTTP SSE 或 Runner 配置。 |
| `gen_ai.chat_completions.streaming_time_to_first_token` | 从 SDK 请求开始到首个生成增量到达的浮点秒数。 |

首个非空文本、推理或工具调用增量触发一次 TTFT 记录。元数据、空增量、usage、心跳和仅完成时返回的聚合内容不计入。非流式、无输出或未埋点路径省略 TTFT;SDK 模式未知时不从 Runner 配置推导。一个 chunk 可含多个 token,因此这是 SDK 侧观测耗时,不是供应商侧逐 token 时间戳。

同一个值通过现有全局 MeterProvider,在 `gen_ai.chat_completions.streaming_time_to_first_token` Histogram 记录一次,单位为 `s`。维度为 `gen_ai.request.model` 和 `gen_ai.operation.name=chat`。指标不依赖 token usage 或 Trace 采样;导出需要配置指标链路。负时钟间隔省略并产生诊断;真正测得的零值有效。

首 Token 采集排除角色/id/签名元数据、空增量、usage、心跳和仅完成时返回的聚合内容。不依赖 token usage 或内容采集。一个 chunk 可含多个 token,因此这里表示 SDK 侧观测到首输出的时间。非流式调用没有首 Token 时间戳,不能用响应完成时间替代。
失败模型请求设置原生 Span Status 为 `ERROR`,`error.type` 为 `llm_error`、`timeout` 或 `cancelled`。请求 Span 使用 `request_error`、`timeout` 或 `cancelled`;HTTP 5xx 响应使用 `http_error`。主动关闭生成器不算错误。已观测的 TTFT 在中断后保留,本埋点不添加异常正文。

每次 SDK 请求建立 `model_request` 子 Span,所属模型 Span 同时记录最近一次尝试的字段。VeADK 发起的重试和 fallback 分别记录尝试;LiteLLM/Provider 内部重试计入所观测的 SDK 请求(`veadk.model.request_scope=sdk_request`)。不支持的外部/自定义引擎明确标记,不用 invocation Span 起点替代模型开始。未使用 VeADK HTTP 工厂的应用需要安装请求计时中间件或使用 VeADK Runner,才能建立请求时间戳。客户端渲染计时和 TTFT/TPOT 直方图上报属于独立能力。
本埋点不再导出自定义 `veadk.request.*`、`veadk.model.*` 属性或自定义首 Token 事件。开始时间和首输出值仅作为内部计算状态。此变更替代此前自定义计时字段,不保留兼容别名。客户端渲染计时和 TPOT Histogram 上报不在本功能范围内。
Loading
Loading