Skip to content

Commit 3dae2aa

Browse files
committed
feat(telemetry): add callerType=script attribution
- Distinguish reusable script/template bl invocations from agent/unknown so dashboards can keep agent/(agent+unknown+script) without dropping scripts from the denominator. - reinforce script caller attribution in business skills
1 parent cadc301 commit 3dae2aa

16 files changed

Lines changed: 176 additions & 52 deletions

File tree

‎docs/agents/skill-change.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ bailian-gen bailian-finetune bailian-managed-agent bailian-web-search
4040

4141
- [ ] **整包装齐**:安装 / 缺 skill 文案主推 `bl skill init`;升级同步用 `bl skill update`(只更新已安装,不装新 skill);业务 skill **不**声明 `companions`
4242
- [ ] **协议读取**:CRITICAL / references 可链 `../bailian-protocol/…`;若读不到 → 停止执行 `bl`,提示 `bl skill init`
43-
- [ ] **调用归因**:协议 `SKILL.md` 放 bash + PowerShell 双模板与协议样例;每个业务 skill **只留一条**填好的样例并链回协议;其余示例 / `assets/` / `reference/` 保持裸 `bl …`。Agent 执行时再套模板;人类终端不要设这些变量
43+
- [ ] **调用归因**:协议 `SKILL.md` 放 bash + PowerShell 的 agent / script 模板与协议样例;每个业务 skill **只留一条**填好的 agent 样例并链回协议;其余示例 / `assets/` / `reference/` 保持裸 `bl …`。Agent 当面 inline 套 agent 四变量;**抄进可复用脚本时改用 protocol `script` 模板**,并写明「样例仅当面 inline」。禁止为冒充 agent 而设变量,也禁止写入 shell profile / direnv / CI 全局环境
4444
- [ ] **高风险确认**:统一由 `bailian-protocol` 定义;reference / leaf help 以 `risk: high` 明示风险,业务 skill 不得引导 Agent 自动补 `--yes`。遇到 exit code 7 / `requires_confirmation` 时停止执行并请求确认;目标或范围变化后重新确认
4545
- [ ] **正常控制流**:`requires_confirmation` 不是 CLI bug,`assets/issue-reporting.md` 必须将 exit code 7 保持在 EXCLUDE 范围
4646
- [ ] **软 hand-off**:兄弟业务 skill **只写 skill 名**;已安装则 Read,未安装则 `bl … --help` 或提示整包安装;**不要**把 `../bailian-gen/…` 等写成执行前提

‎docs/agents/telemetry-change.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -97,17 +97,20 @@ source-config 只用于百炼 / DashScope API 侧消费,不发送到通用网
9797
- `cliVersion`、`nodeVersion`、`os`
9898
- `commandPrefix`(调用入口,来自 `Identity`)
9999
- `authMethod`
100-
- `callerType`(必填:`agent` 或 `unknown`);`callerType=agent` 时可选 `callerName`、`skillName`、`skillVersion`
100+
- `callerType`(必填:`agent` | `script` | `unknown`)
101+
- `callerName`:`agent` 与 `script` 时发送(缺省/非法 → `"unknown"`);`skillName` / `skillVersion` 仅 `agent` 时可选发送
101102
- 失败时的 `errorMessage`、`exitCode`、`httpStatus`、`requestId`
102103
- 安全 allowlist 过滤后的 `params`
103104

104105
调用方归因只认本次进程的 `BAILIAN_CALLER_*`(由 `readCaller` 归一化后写入事件):
105106

106107
- `BAILIAN_CALLER_TYPE` 为 `agent` / `ai_agent`(大小写不敏感)→ `callerType=agent`
108+
- `BAILIAN_CALLER_TYPE` 为 `script`(大小写不敏感,trim)→ `callerType=script`;读取 `callerName`,**忽略** skill env
107109
- 未设置、空白或其他值 → `callerType=unknown`;不要把该字段当成人工
108-
- `callerName` / `skillName` / `skillVersion` 仅在 `agent` 时发送;`kscli` 通常没有 skill 字段
110+
- `callerName` 在 `agent` 与 `script` 时发送;缺省、空白或非法名称一律归一为 `"unknown"`(看板会出现 `callerType=script, callerName=unknown` 等组合,属预期兜底,不是漏字段)
111+
- `skillName` / `skillVersion` 仅在 `agent` 时发送;`kscli` 通常没有 skill 字段
109112

110-
新 CLI 每条进入 middleware 的命令事件都有 `callerType`。旧 CLI 没有该字段。看板只统计带 `callerType` 的事件,不能把字段缺失混进分母。`unknown` 只表示未识别为 agent,不是人工。新 CLI + 旧 skill / Agent 漏设变量时,会落进 `unknown`。`BAILIAN_CALLER_*` 是调用方声明,不是 `Identity`,不得用来猜 `bl` / `kscli`。
113+
新 CLI 每条进入 middleware 的命令事件都有 `callerType`。旧 CLI 没有该字段。看板只统计带 `callerType` 的事件,不能把字段缺失混进分母。已归因 := `agent + unknown + script`。Agent 调用率主口径:`agent / (agent + unknown + script)`;`script` 进分母、不进分子,也不得从分母剔除。`unknown` 只表示未识别为 agent/script,不是人工。新 CLI + 旧 skill / Agent 漏设变量时,会落进 `unknown`。`BAILIAN_CALLER_*` 是调用方声明,不是 `Identity`,不得用来猜 `bl` / `kscli`。看板禁止按 `callerName` 跨 `callerType` 聚合。
111114

112115
参数默认不上传,只有 `packages/core/src/telemetry/tracker.ts` 的 `PARAM_ALLOWLIST` 中字段会进入事件。不得加入 prompt、凭证、文件路径、URL、账号/租户/工作空间 ID 或其他用户内容。
113116

@@ -173,7 +176,7 @@ AEM 映射:
173176
- [ ] 更新 `TrackingEvent`、`createTrackingEvent()` 与 `buildRemoteAemOptions()` 的字段映射
174177
- [ ] 本地 JSONL 与远端 AEM 必须基于同一结构化事件,不能维护两套字段口径
175178
- [ ] 三种入口分别核对 `commandPrefix`;共享路径也必须可区分,旧数据缺字段保留为未知
176-
- [ ] 调用方字段经 `readCaller` → `TrackingDeps.caller` → 事件;仅显式 `agent` / `ai_agent` 为 `agent`,其余为 `unknown`
179+
- [ ] 调用方字段经 `readCaller` → `TrackingDeps.caller` → 事件;显式 `agent` / `ai_agent` → `agent`,显式 `script` → `script`,其余 → `unknown`
177180
- [ ] 成功与失败均覆盖;遥测异常必须静默且不改变业务退出码
178181
- [ ] runtime 本地语义错误应记录 `exitCode`;新增字段默认随 AEM `ext` 上报,无需占用新的 `c1`—`c5`
179182
- [ ] 检查 `DO_NOT_TRACK=1` 与 `telemetry: false` 两个关闭入口
Lines changed: 33 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
/** Parse BAILIAN_CALLER_*. Only an explicit agent is `agent`; everything else is `unknown`. */
1+
/** Parse BAILIAN_CALLER_*. Explicit agent/ai_agent → agent; explicit script → script; everything else → unknown. */
22

33
export type CallerAttribution =
44
| { callerType: "unknown" }
@@ -7,7 +7,8 @@ export type CallerAttribution =
77
callerName: string;
88
skillName?: string;
99
skillVersion?: string;
10-
};
10+
}
11+
| { callerType: "script"; callerName: string };
1112

1213
const CALLER_TYPE_ENV = "BAILIAN_CALLER_TYPE";
1314
const CALLER_NAME_ENV = "BAILIAN_CALLER_NAME";
@@ -35,25 +36,36 @@ function normalizeCallerName(raw: string | undefined): string {
3536
/** Env-only attribution; no TTY or install-path heuristics. */
3637
export function readCaller(env: NodeJS.ProcessEnv): CallerAttribution {
3738
const callerType = readTrimmed(env, CALLER_TYPE_ENV)?.toLowerCase();
38-
if (callerType !== "agent" && callerType !== "ai_agent") {
39-
return { callerType: "unknown" };
39+
40+
if (callerType === "agent" || callerType === "ai_agent") {
41+
const skillNameRaw = readTrimmed(env, SKILL_NAME_ENV)?.toLowerCase();
42+
const skillName =
43+
skillNameRaw !== undefined && SKILL_NAME_PATTERN.test(skillNameRaw)
44+
? skillNameRaw
45+
: undefined;
46+
const skillVersionRaw = readTrimmed(env, SKILL_VERSION_ENV);
47+
const skillVersion =
48+
skillName !== undefined &&
49+
skillVersionRaw !== undefined &&
50+
SKILL_VERSION_PATTERN.test(skillVersionRaw)
51+
? skillVersionRaw
52+
: undefined;
53+
54+
return {
55+
callerType: "agent",
56+
callerName: normalizeCallerName(readTrimmed(env, CALLER_NAME_ENV)),
57+
...(skillName !== undefined ? { skillName } : {}),
58+
...(skillVersion !== undefined ? { skillVersion } : {}),
59+
};
60+
}
61+
62+
if (callerType === "script") {
63+
return {
64+
callerType: "script",
65+
// script 只带执行/维护宿主名,不读取 skill*
66+
callerName: normalizeCallerName(readTrimmed(env, CALLER_NAME_ENV)),
67+
};
4068
}
4169

42-
const skillNameRaw = readTrimmed(env, SKILL_NAME_ENV)?.toLowerCase();
43-
const skillName =
44-
skillNameRaw !== undefined && SKILL_NAME_PATTERN.test(skillNameRaw) ? skillNameRaw : undefined;
45-
const skillVersionRaw = readTrimmed(env, SKILL_VERSION_ENV);
46-
const skillVersion =
47-
skillName !== undefined &&
48-
skillVersionRaw !== undefined &&
49-
SKILL_VERSION_PATTERN.test(skillVersionRaw)
50-
? skillVersionRaw
51-
: undefined;
52-
53-
return {
54-
callerType: "agent",
55-
callerName: normalizeCallerName(readTrimmed(env, CALLER_NAME_ENV)),
56-
...(skillName !== undefined ? { skillName } : {}),
57-
...(skillVersion !== undefined ? { skillVersion } : {}),
58-
};
70+
return { callerType: "unknown" };
5971
}

‎packages/core/src/telemetry/event.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ export interface TrackingEvent {
1818
os: string;
1919
authMethod?: AuthRequirement;
2020
params?: Record<string, unknown>;
21-
/** Always present on new CLI events: `agent` or `unknown`. */
21+
/** Always present on new CLI events: `agent` | `script` | `unknown`. */
2222
callerType: CallerAttribution["callerType"];
2323
callerName?: string;
2424
skillName?: string;
@@ -51,6 +51,8 @@ export function createTrackingEvent(opts: {
5151
event.callerName = opts.caller.callerName;
5252
if (opts.caller.skillName !== undefined) event.skillName = opts.caller.skillName;
5353
if (opts.caller.skillVersion !== undefined) event.skillVersion = opts.caller.skillVersion;
54+
} else if (opts.caller.callerType === "script") {
55+
event.callerName = opts.caller.callerName;
5456
}
5557

5658
if (opts.authMethod) {

‎packages/core/tests/telemetry-caller.test.ts‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,3 +140,29 @@ test("ignores agent detail fields when callerType is not agent", () => {
140140
}),
141141
).toEqual({ callerType: "unknown" });
142142
});
143+
144+
test.each([
145+
{ label: "script", type: "script" },
146+
{ label: "Script", type: "Script" },
147+
{ label: "SCRIPT", type: "SCRIPT" },
148+
{ label: "padded script", type: " script " },
149+
])("accepts $label as script without skill fields", ({ type }) => {
150+
expect(
151+
readCaller({
152+
BAILIAN_CALLER_TYPE: type,
153+
BAILIAN_CALLER_NAME: "cursor",
154+
BAILIAN_SKILL_NAME: "bailian-cli",
155+
BAILIAN_SKILL_VERSION: "2.5.0",
156+
}),
157+
).toEqual({
158+
callerType: "script",
159+
callerName: "cursor",
160+
});
161+
});
162+
163+
test("script falls back to unknown callerName when NAME is missing", () => {
164+
expect(readCaller({ BAILIAN_CALLER_TYPE: "script" })).toEqual({
165+
callerType: "script",
166+
callerName: "unknown",
167+
});
168+
});

‎packages/core/tests/telemetry-tracker.test.ts‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,10 @@ const agentCaller: CallerAttribution = {
3232
skillName: "bailian-cli",
3333
skillVersion: "2.5.0",
3434
};
35+
const scriptCaller: CallerAttribution = {
36+
callerType: "script",
37+
callerName: "cursor",
38+
};
3539

3640
test("derives event version and prefix from identity", () => {
3741
const event = createTrackingEvent({
@@ -253,6 +257,40 @@ test("keeps agent attribution on success and in AEM ext", async () => {
253257
});
254258
});
255259

260+
test("keeps script attribution with callerName and without skill leak", async () => {
261+
sinkMocks.localSink.mockClear();
262+
sinkMocks.remoteSink.mockClear();
263+
264+
await trackCommandExecution(
265+
{
266+
identity,
267+
settings: { telemetry: true } as Settings,
268+
authMethod: "apiKey",
269+
caller: scriptCaller,
270+
},
271+
["config", "show"],
272+
{},
273+
async () => {},
274+
);
275+
276+
const event = sinkMocks.localSink.mock.calls[0]![0];
277+
expect(event).toMatchObject({
278+
success: true,
279+
callerType: "script",
280+
callerName: "cursor",
281+
});
282+
expect(event.skillName).toBeUndefined();
283+
expect(event.skillVersion).toBeUndefined();
284+
expect(event.params).toBeUndefined();
285+
const remoteExt = buildRemoteAemOptions(event).ext as Record<string, unknown>;
286+
expect(remoteExt).toMatchObject({
287+
callerType: "script",
288+
callerName: "cursor",
289+
});
290+
expect(remoteExt).not.toHaveProperty("skillName");
291+
expect(remoteExt).not.toHaveProperty("skillVersion");
292+
});
293+
256294
test("does not call sinks when telemetry is disabled", async () => {
257295
sinkMocks.localSink.mockClear();
258296
sinkMocks.remoteSink.mockClear();

‎packages/runtime/tests/telemetry-stage.test.ts‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,3 +85,22 @@ test("defaults to unknown when caller env is unset", async () => {
8585
};
8686
expect(deps.caller).toEqual({ callerType: "unknown" });
8787
});
88+
89+
test("wires script caller env without skill fields", async () => {
90+
vi.stubEnv("BAILIAN_CALLER_TYPE", "script");
91+
vi.stubEnv("BAILIAN_CALLER_NAME", "cursor");
92+
vi.stubEnv("BAILIAN_SKILL_NAME", "bailian-cli");
93+
vi.stubEnv("BAILIAN_SKILL_VERSION", "2.5.0");
94+
const next = vi.fn(async () => {});
95+
96+
await telemetryStage(makeContext(), next);
97+
98+
const deps = trackerMocks.trackCommandExecution.mock.calls[0]![0] as {
99+
caller: CallerAttribution;
100+
};
101+
expect(deps.caller).toEqual({
102+
callerType: "script",
103+
callerName: "cursor",
104+
});
105+
expect(next).toHaveBeenCalledOnce();
106+
});

‎skills/bailian-cli/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,10 @@ description: >-
2323

2424
**CRITICAL — Before executing, MUST read the shared protocol in [`../bailian-protocol/SKILL.md`](../bailian-protocol/SKILL.md): Provider selection and consent, Version & updates (pre-flight checklist), Call attribution (every Agent `bl` command), Setup & auth, and CLI errors: report an issue. If that protocol file is missing, stop and run `bl skill init`; do not guess auth/consent.**
2525

26-
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-cli`; replace `codex` with this host):
26+
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-cli`; replace `codex` with this host). **When copying commands into a reusable script / template / task definition, switch to the protocol `script` template (never keep `agent`); the sample below is for this-turn inline execution only. Strip attribution when handing a command to the user to paste.**
2727

2828
```bash
29-
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-cli BAILIAN_SKILL_VERSION=2.5.0 bl text chat --message "Write a poem about spring"
29+
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-cli BAILIAN_SKILL_VERSION=<metadata.version> bl text chat --message "Write a poem about spring"
3030
```
3131

3232
> **Family hub** — This skill owns Bailian resource commands and the hub `reference/` (apps, usage, auth, config, …).

‎skills/bailian-finetune/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,10 @@ description: >-
1818

1919
**CRITICAL — Before executing, MUST read the shared protocol in [`../bailian-protocol/SKILL.md`](../bailian-protocol/SKILL.md): Version & updates (pre-flight checklist), Call attribution (every Agent `bl` command), Setup & auth, and CLI errors: report an issue. Command details are authoritative in [`reference/`](reference/index.md) (dataset / finetune / deploy) and `bl <command> --help` — do not guess flags. Examples in `reference/` stay bare; wrap with Call attribution when running. The whole pipeline requires an API key. If that protocol file is missing, stop and run `bl skill init`; do not guess auth/consent.**
2020

21-
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-finetune`; replace `codex` with this host). After deploy, `bl text chat` is owned by `bailian-cli`.
21+
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-finetune`; replace `codex` with this host). After deploy, `bl text chat` is owned by `bailian-cli`. **When copying commands into a reusable script / template / task definition, switch to the protocol `script` template (never keep `agent`); the sample below is for this-turn inline execution only. Strip attribution when handing a command to the user to paste.**
2222

2323
```bash
24-
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-finetune BAILIAN_SKILL_VERSION=2.5.0 bl dataset validate --file train.jsonl
24+
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-finetune BAILIAN_SKILL_VERSION=<metadata.version> bl dataset validate --file train.jsonl
2525
```
2626

2727
## End-to-end workflow (follow in order)

‎skills/bailian-gen/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@ description: >-
2222

2323
**CRITICAL — Before executing, MUST read the shared protocol in [`../bailian-protocol/SKILL.md`](../bailian-protocol/SKILL.md): Provider selection and consent (one-time ask templates), Version & updates (pre-flight checklist), Call attribution (every Agent `bl` command), and CLI errors: report an issue. Command details are authoritative in [`reference/`](reference/index.md) and `bl <command> --help` — do not guess flags. Examples in `reference/` stay bare; wrap with Call attribution when running. If that protocol file is missing, stop and run `bl skill init`; do not guess auth/consent.**
2424

25-
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-gen`; replace `codex` with this host):
25+
**Call attribution:** Examples below and in `reference/` stay bare `bl …`. When running a command, wrap it with [`bailian-protocol` → Call attribution](../bailian-protocol/SKILL.md#call-attribution-mandatory-for-every-agent-bl-command) (bash or PowerShell). One sample for this skill (`BAILIAN_SKILL_NAME=bailian-gen`; replace `codex` with this host). **When copying commands into a reusable script / template / task definition, switch to the protocol `script` template (never keep `agent`); the sample below is for this-turn inline execution only. Strip attribution when handing a command to the user to paste.**
2626

2727
```bash
28-
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-gen BAILIAN_SKILL_VERSION=2.5.0 bl image generate --prompt "A cat in space" --out-dir ./out/
28+
BAILIAN_CALLER_TYPE=agent BAILIAN_CALLER_NAME=codex BAILIAN_SKILL_NAME=bailian-gen BAILIAN_SKILL_VERSION=<metadata.version> bl image generate --prompt "A cat in space" --out-dir ./out/
2929
```
3030

3131
## Consent (short version; full rules in bailian-protocol)

0 commit comments

Comments
 (0)