Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/agents/skill-change.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ bailian-gen bailian-finetune bailian-managed-agent bailian-web-search

- [ ] **整包装齐**:安装 / 缺 skill 文案主推 `bl skill init`;升级同步用 `bl skill update`(只更新已安装,不装新 skill);业务 skill **不**声明 `companions`
- [ ] **协议读取**:CRITICAL / references 可链 `../bailian-protocol/…`;若读不到 → 停止执行 `bl`,提示 `bl skill init`
- [ ] **调用归因**:协议 `SKILL.md` 放 bash + PowerShell 双模板与协议样例;每个业务 skill **只留一条**填好的样例并链回协议;其余示例 / `assets/` / `reference/` 保持裸 `bl …`。Agent 执行时再套模板;人类终端不要设这些变量
- [ ] **调用归因**:协议 `SKILL.md` 放 bash + PowerShell 的 agent / script 模板与协议样例;每个业务 skill **只留一条**填好的 agent 样例并链回协议;其余示例 / `assets/` / `reference/` 保持裸 `bl …`。Agent 当面 inline 套 agent 四变量;**抄进可复用脚本时改用 protocol `script` 模板**,并写明「样例仅当面 inline」。禁止为冒充 agent 而设变量,也禁止写入 shell profile / direnv / CI 全局环境
- [ ] **高风险确认**:统一由 `bailian-protocol` 定义;reference / leaf help 以 `risk: high` 明示风险,业务 skill 不得引导 Agent 自动补 `--yes`。遇到 exit code 7 / `requires_confirmation` 时停止执行并请求确认;目标或范围变化后重新确认
- [ ] **正常控制流**:`requires_confirmation` 不是 CLI bug,`assets/issue-reporting.md` 必须将 exit code 7 保持在 EXCLUDE 范围
- [ ] **软 hand-off**:兄弟业务 skill **只写 skill 名**;已安装则 Read,未安装则 `bl … --help` 或提示整包安装;**不要**把 `../bailian-gen/…` 等写成执行前提
Expand Down
11 changes: 7 additions & 4 deletions docs/agents/telemetry-change.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,17 +97,20 @@ source-config 只用于百炼 / DashScope API 侧消费,不发送到通用网
- `cliVersion`、`nodeVersion`、`os`
- `commandPrefix`(调用入口,来自 `Identity`)
- `authMethod`
- `callerType`(必填:`agent` 或 `unknown`);`callerType=agent` 时可选 `callerName`、`skillName`、`skillVersion`
- `callerType`(必填:`agent` | `script` | `unknown`)
- `callerName`:`agent` 与 `script` 时发送(缺省/非法 → `"unknown"`);`skillName` / `skillVersion` 仅 `agent` 时可选发送
- 失败时的 `errorMessage`、`exitCode`、`httpStatus`、`requestId`
- 安全 allowlist 过滤后的 `params`

调用方归因只认本次进程的 `BAILIAN_CALLER_*`(由 `readCaller` 归一化后写入事件):

- `BAILIAN_CALLER_TYPE` 为 `agent` / `ai_agent`(大小写不敏感)→ `callerType=agent`
- `BAILIAN_CALLER_TYPE` 为 `script`(大小写不敏感,trim)→ `callerType=script`;读取 `callerName`,**忽略** skill env
- 未设置、空白或其他值 → `callerType=unknown`;不要把该字段当成人工
- `callerName` / `skillName` / `skillVersion` 仅在 `agent` 时发送;`kscli` 通常没有 skill 字段
- `callerName` 在 `agent` 与 `script` 时发送;缺省、空白或非法名称一律归一为 `"unknown"`(看板会出现 `callerType=script, callerName=unknown` 等组合,属预期兜底,不是漏字段)
- `skillName` / `skillVersion` 仅在 `agent` 时发送;`kscli` 通常没有 skill 字段

新 CLI 每条进入 middleware 的命令事件都有 `callerType`。旧 CLI 没有该字段。看板只统计带 `callerType` 的事件,不能把字段缺失混进分母。`unknown` 只表示未识别为 agent,不是人工。新 CLI + 旧 skill / Agent 漏设变量时,会落进 `unknown`。`BAILIAN_CALLER_*` 是调用方声明,不是 `Identity`,不得用来猜 `bl` / `kscli`。
新 CLI 每条进入 middleware 的命令事件都有 `callerType`。旧 CLI 没有该字段。看板只统计带 `callerType` 的事件,不能把字段缺失混进分母。已归因 := `agent + unknown + script`。Agent 调用率主口径:`agent / (agent + unknown + script)`;`script` 进分母、不进分子,也不得从分母剔除。`unknown` 只表示未识别为 agent/script,不是人工。新 CLI + 旧 skill / Agent 漏设变量时,会落进 `unknown`。`BAILIAN_CALLER_*` 是调用方声明,不是 `Identity`,不得用来猜 `bl` / `kscli`。看板禁止按 `callerName` 跨 `callerType` 聚合。

参数默认不上传,只有 `packages/core/src/telemetry/tracker.ts` 的 `PARAM_ALLOWLIST` 中字段会进入事件。不得加入 prompt、凭证、文件路径、URL、账号/租户/工作空间 ID 或其他用户内容。

Expand Down Expand Up @@ -173,7 +176,7 @@ AEM 映射:
- [ ] 更新 `TrackingEvent`、`createTrackingEvent()` 与 `buildRemoteAemOptions()` 的字段映射
- [ ] 本地 JSONL 与远端 AEM 必须基于同一结构化事件,不能维护两套字段口径
- [ ] 三种入口分别核对 `commandPrefix`;共享路径也必须可区分,旧数据缺字段保留为未知
- [ ] 调用方字段经 `readCaller` → `TrackingDeps.caller` → 事件;仅显式 `agent` / `ai_agent` 为 `agent`,其余为 `unknown`
- [ ] 调用方字段经 `readCaller` → `TrackingDeps.caller` → 事件;显式 `agent` / `ai_agent` → `agent`,显式 `script` → `script`,其余 → `unknown`
- [ ] 成功与失败均覆盖;遥测异常必须静默且不改变业务退出码
- [ ] runtime 本地语义错误应记录 `exitCode`;新增字段默认随 AEM `ext` 上报,无需占用新的 `c1`—`c5`
- [ ] 检查 `DO_NOT_TRACK=1` 与 `telemetry: false` 两个关闭入口
Expand Down
54 changes: 33 additions & 21 deletions packages/core/src/telemetry/caller.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/** Parse BAILIAN_CALLER_*. Only an explicit agent is `agent`; everything else is `unknown`. */
/** Parse BAILIAN_CALLER_*. Explicit agent/ai_agent → agent; explicit script → script; everything else → unknown. */

export type CallerAttribution =
| { callerType: "unknown" }
Expand All @@ -7,7 +7,8 @@ export type CallerAttribution =
callerName: string;
skillName?: string;
skillVersion?: string;
};
}
| { callerType: "script"; callerName: string };

const CALLER_TYPE_ENV = "BAILIAN_CALLER_TYPE";
const CALLER_NAME_ENV = "BAILIAN_CALLER_NAME";
Expand Down Expand Up @@ -35,25 +36,36 @@ function normalizeCallerName(raw: string | undefined): string {
/** Env-only attribution; no TTY or install-path heuristics. */
export function readCaller(env: NodeJS.ProcessEnv): CallerAttribution {
const callerType = readTrimmed(env, CALLER_TYPE_ENV)?.toLowerCase();
if (callerType !== "agent" && callerType !== "ai_agent") {
return { callerType: "unknown" };

if (callerType === "agent" || callerType === "ai_agent") {
const skillNameRaw = readTrimmed(env, SKILL_NAME_ENV)?.toLowerCase();
const skillName =
skillNameRaw !== undefined && SKILL_NAME_PATTERN.test(skillNameRaw)
? skillNameRaw
: undefined;
const skillVersionRaw = readTrimmed(env, SKILL_VERSION_ENV);
const skillVersion =
skillName !== undefined &&
skillVersionRaw !== undefined &&
SKILL_VERSION_PATTERN.test(skillVersionRaw)
? skillVersionRaw
: undefined;

return {
callerType: "agent",
callerName: normalizeCallerName(readTrimmed(env, CALLER_NAME_ENV)),
...(skillName !== undefined ? { skillName } : {}),
...(skillVersion !== undefined ? { skillVersion } : {}),
};
}

if (callerType === "script") {
return {
callerType: "script",
// script 只带执行/维护宿主名,不读取 skill*
callerName: normalizeCallerName(readTrimmed(env, CALLER_NAME_ENV)),
};
}

const skillNameRaw = readTrimmed(env, SKILL_NAME_ENV)?.toLowerCase();
const skillName =
skillNameRaw !== undefined && SKILL_NAME_PATTERN.test(skillNameRaw) ? skillNameRaw : undefined;
const skillVersionRaw = readTrimmed(env, SKILL_VERSION_ENV);
const skillVersion =
skillName !== undefined &&
skillVersionRaw !== undefined &&
SKILL_VERSION_PATTERN.test(skillVersionRaw)
? skillVersionRaw
: undefined;

return {
callerType: "agent",
callerName: normalizeCallerName(readTrimmed(env, CALLER_NAME_ENV)),
...(skillName !== undefined ? { skillName } : {}),
...(skillVersion !== undefined ? { skillVersion } : {}),
};
return { callerType: "unknown" };
}
4 changes: 3 additions & 1 deletion packages/core/src/telemetry/event.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ export interface TrackingEvent {
os: string;
authMethod?: AuthRequirement;
params?: Record<string, unknown>;
/** Always present on new CLI events: `agent` or `unknown`. */
/** Always present on new CLI events: `agent` | `script` | `unknown`. */
callerType: CallerAttribution["callerType"];
callerName?: string;
skillName?: string;
Expand Down Expand Up @@ -51,6 +51,8 @@ export function createTrackingEvent(opts: {
event.callerName = opts.caller.callerName;
if (opts.caller.skillName !== undefined) event.skillName = opts.caller.skillName;
if (opts.caller.skillVersion !== undefined) event.skillVersion = opts.caller.skillVersion;
} else if (opts.caller.callerType === "script") {
event.callerName = opts.caller.callerName;
}

if (opts.authMethod) {
Expand Down
26 changes: 26 additions & 0 deletions packages/core/tests/telemetry-caller.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,29 @@ test("ignores agent detail fields when callerType is not agent", () => {
}),
).toEqual({ callerType: "unknown" });
});

test.each([
{ label: "script", type: "script" },
{ label: "Script", type: "Script" },
{ label: "SCRIPT", type: "SCRIPT" },
{ label: "padded script", type: " script " },
])("accepts $label as script without skill fields", ({ type }) => {
expect(
readCaller({
BAILIAN_CALLER_TYPE: type,
BAILIAN_CALLER_NAME: "cursor",
BAILIAN_SKILL_NAME: "bailian-cli",
BAILIAN_SKILL_VERSION: "2.5.0",
}),
).toEqual({
callerType: "script",
callerName: "cursor",
});
});

test("script falls back to unknown callerName when NAME is missing", () => {
expect(readCaller({ BAILIAN_CALLER_TYPE: "script" })).toEqual({
callerType: "script",
callerName: "unknown",
});
});
38 changes: 38 additions & 0 deletions packages/core/tests/telemetry-tracker.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ const agentCaller: CallerAttribution = {
skillName: "bailian-cli",
skillVersion: "2.5.0",
};
const scriptCaller: CallerAttribution = {
callerType: "script",
callerName: "cursor",
};

test("derives event version and prefix from identity", () => {
const event = createTrackingEvent({
Expand Down Expand Up @@ -253,6 +257,40 @@ test("keeps agent attribution on success and in AEM ext", async () => {
});
});

test("keeps script attribution with callerName and without skill leak", async () => {
sinkMocks.localSink.mockClear();
sinkMocks.remoteSink.mockClear();

await trackCommandExecution(
{
identity,
settings: { telemetry: true } as Settings,
authMethod: "apiKey",
caller: scriptCaller,
},
["config", "show"],
{},
async () => {},
);

const event = sinkMocks.localSink.mock.calls[0]![0];
expect(event).toMatchObject({
success: true,
callerType: "script",
callerName: "cursor",
});
expect(event.skillName).toBeUndefined();
expect(event.skillVersion).toBeUndefined();
expect(event.params).toBeUndefined();
const remoteExt = buildRemoteAemOptions(event).ext as Record<string, unknown>;
expect(remoteExt).toMatchObject({
callerType: "script",
callerName: "cursor",
});
expect(remoteExt).not.toHaveProperty("skillName");
expect(remoteExt).not.toHaveProperty("skillVersion");
});

test("does not call sinks when telemetry is disabled", async () => {
sinkMocks.localSink.mockClear();
sinkMocks.remoteSink.mockClear();
Expand Down
19 changes: 19 additions & 0 deletions packages/runtime/tests/telemetry-stage.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,22 @@ test("defaults to unknown when caller env is unset", async () => {
};
expect(deps.caller).toEqual({ callerType: "unknown" });
});

test("wires script caller env without skill fields", async () => {
vi.stubEnv("BAILIAN_CALLER_TYPE", "script");
vi.stubEnv("BAILIAN_CALLER_NAME", "cursor");
vi.stubEnv("BAILIAN_SKILL_NAME", "bailian-cli");
vi.stubEnv("BAILIAN_SKILL_VERSION", "2.5.0");
const next = vi.fn(async () => {});

await telemetryStage(makeContext(), next);

const deps = trackerMocks.trackCommandExecution.mock.calls[0]![0] as {
caller: CallerAttribution;
};
expect(deps.caller).toEqual({
callerType: "script",
callerName: "cursor",
});
expect(next).toHaveBeenCalledOnce();
});
4 changes: 2 additions & 2 deletions skills/bailian-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ description: >-

**CRITICAL — Before executing, MUST read the shared protocol in [`../bailian-protocol/SKILL.md`](../bailian-protocol/SKILL.md): Provider selection and consent, Version & updates (pre-flight checklist), Call attribution (every Agent `bl` command), Setup & auth, and CLI errors: report an issue. If that protocol file is missing, stop and run `bl skill init`; do not guess auth/consent.**

**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-cli`; replace `codex` with this host):
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-cli`; replace `codex` with this host). **When copying commands into a reusable script / template / task definition, switch to the protocol `script` template (never keep `agent`); the sample below is for this-turn inline execution only. Strip attribution when handing a command to the user to paste.**

```bash
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-cli BAILIAN_SKILL_VERSION=2.5.0 bl text chat --message "Write a poem about spring"
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-cli BAILIAN_SKILL_VERSION=<metadata.version> bl text chat --message "Write a poem about spring"
```

> **Family hub** — This skill owns Bailian resource commands and the hub `reference/` (apps, usage, auth, config, …).
Expand Down
4 changes: 2 additions & 2 deletions skills/bailian-finetune/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ description: >-

**CRITICAL — Before executing, MUST read the shared protocol in [`../bailian-protocol/SKILL.md`](../bailian-protocol/SKILL.md): Version & updates (pre-flight checklist), Call attribution (every Agent `bl` command), Setup & auth, and CLI errors: report an issue. Command details are authoritative in [`reference/`](reference/index.md) (dataset / finetune / deploy) and `bl <command> --help` — do not guess flags. Examples in `reference/` stay bare; wrap with Call attribution when running. The whole pipeline requires an API key. If that protocol file is missing, stop and run `bl skill init`; do not guess auth/consent.**

**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-finetune`; replace `codex` with this host). After deploy, `bl text chat` is owned by `bailian-cli`.
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-finetune`; replace `codex` with this host). After deploy, `bl text chat` is owned by `bailian-cli`. **When copying commands into a reusable script / template / task definition, switch to the protocol `script` template (never keep `agent`); the sample below is for this-turn inline execution only. Strip attribution when handing a command to the user to paste.**

```bash
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-finetune BAILIAN_SKILL_VERSION=2.5.0 bl dataset validate --file train.jsonl
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-finetune BAILIAN_SKILL_VERSION=<metadata.version> bl dataset validate --file train.jsonl
```

## End-to-end workflow (follow in order)
Expand Down
4 changes: 2 additions & 2 deletions skills/bailian-gen/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ description: >-

**CRITICAL — Before executing, MUST read the shared protocol in [`../bailian-protocol/SKILL.md`](../bailian-protocol/SKILL.md): Provider selection and consent (one-time ask templates), Version & updates (pre-flight checklist), Call attribution (every Agent `bl` command), and CLI errors: report an issue. Command details are authoritative in [`reference/`](reference/index.md) and `bl <command> --help` — do not guess flags. Examples in `reference/` stay bare; wrap with Call attribution when running. If that protocol file is missing, stop and run `bl skill init`; do not guess auth/consent.**

**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-gen`; replace `codex` with this host):
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-gen`; replace `codex` with this host). **When copying commands into a reusable script / template / task definition, switch to the protocol `script` template (never keep `agent`); the sample below is for this-turn inline execution only. Strip attribution when handing a command to the user to paste.**

```bash
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-gen BAILIAN_SKILL_VERSION=2.5.0 bl image generate --prompt "A cat in space" --out-dir ./out/
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-gen BAILIAN_SKILL_VERSION=<metadata.version> bl image generate --prompt "A cat in space" --out-dir ./out/
```

## Consent (short version; full rules in bailian-protocol)
Expand Down
2 changes: 1 addition & 1 deletion skills/bailian-knowledge/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ description: >-

## 命令执行与参考

执行任何 `bl` 命令前读取 [共享协议](../bailian-protocol/SKILL.md),完成版本预检、鉴权和实际风险确认;协议缺失不妨碍本地发现与解释,但须提示 `bl skill init` 补齐后再执行。沿用当前产品,用 `bl knowledge --introspect` 或 `kscli --introspect` 发现能力,切换产品或更新版本后重新读取。不猜 flag,不替换前缀推导路径;当前 schema 不支持的能力如实说明,不承诺升级一定提供。
执行任何 `bl` 命令前读取 [共享协议](../bailian-protocol/SKILL.md),完成版本预检、鉴权、[调用归因](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command)和实际风险确认;协议缺失不妨碍本地发现与解释,但须提示 `bl skill init` 补齐后再执行。沿用当前产品,用 `bl knowledge --introspect` 或 `kscli --introspect` 发现能力,切换产品或更新版本后重新读取。不猜 flag,不替换前缀推导路径;当前 schema 不支持的能力如实说明,不承诺升级一定提供。 **把命令写入可复用脚本 / 模板 / 任务定义时,改用协议的 `script` 模板(不要保留 `agent`);若出现 agent 样例,仅适用于当面 inline。交给用户手动粘贴时去掉归因前缀。**

个人绑定的业务调用显式传已核实的 `--workspace-id`,优先 `--output json`。使用真实返回的 ID;知识库 index ID 不能当作检索 agent ID。凭证与签名上传 URL 不进入回复、绑定或版本库。用户文本作为参数传递,不作为 shell 代码执行。

Expand Down
Loading
Loading