Skip to content

Agent telemetry#36

Closed
purplecabbage wants to merge 7 commits into
masterfrom
agent-telemetry
Closed

Agent telemetry#36
purplecabbage wants to merge 7 commits into
masterfrom
agent-telemetry

Conversation

@purplecabbage

@purplecabbage purplecabbage commented Apr 7, 2026

Copy link
Copy Markdown
Member

Description

Flush architecture

  • Replace blocking await fetch() with a detached child process (flush-worker.js)
    so the parent CLI exits immediately without waiting on the HTTP POST
  • The worker owns the New Relic endpoint and ingest key; neither appears in
    spawn argv (no ps-aux exposure)
  • Worker is spawned with AIO_TELEMETRY_DISABLED=1 to prevent re-entrant telemetry

Persistent retry queue (queue-store.js)

  • Failed POSTs are persisted to ~/.config/aio/.telemetry-queue.json
    (honours XDG_CONFIG_HOME), separate from user-visible aio config
  • On the next CLI invocation the worker merges queued events with the current
    event and retries the full batch in a single POST
  • On success the queue file is removed; on failure the merged set is written back

Agent detection

  • Detect GitHub Copilot Chat via PATH heuristic (copilot-chat/debugCommand,
    copilot-chat/copilotCli shims)
  • Full AGENT_ENV_VARS table: AGENT, AI_AGENT, AIO_AGENT, AIO_INVOCATION_CONTEXT,
    CURSOR_AGENT, CLAUDECODE, CLAUDE_CODE, GEMINI_CLI, CODEX_SANDBOX, AUGMENT_AGENT,
    CLINE_ACTIVE, OPENCODE_CLIENT, REPL_ID, PATH (Copilot)
  • Events include invocation_context (agent|human) and agent_name attributes

AIO_TELEMETRY_DISABLED env var

  • Suppresses all telemetry at trackEvent, isEnabled, and isNull without
    modifying the persisted opt-in state — suitable for CI pipelines

Payload format

  • Migrated from legacy _adobeio envelope to New Relic Metric API format
    ([{ metrics: [{ name, type, value, timestamp, attributes }] }])

Tests

  • 100% statement/branch/function/line coverage across all 5 test suites (67 tests)
  • New suites: flush-worker.test.js, queue-store.test.js
  • hooks.test.js and telemetry-lib.test.js updated for new payload format and
    spawn-based assertions

README updated to reflect New Relic payload format, AIO_TELEMETRY_DISABLED,
flush architecture, and full agent detection table

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • I have signed the Adobe Open Source CLA.
  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

- Added `getInvocationContext` function to determine if the CLI is invoked by an AI agent or a human based on environment variables.
- Updated `trackEvent` to include `invocation_context` and `agent_name` in the payload sent to the telemetry service.
- Expanded test coverage for `trackEvent` and `getInvocationContext` to validate new functionality.
@codecov

codecov Bot commented Apr 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Comment thread src/telemetry-lib.js
}

// this is set by the init hook, ex. @adobe/aio-cli@8.2.0
// TODO: detect VSCODE run as an agent

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we tracking this TODO

Comment thread src/telemetry-lib.js
'x-adobe-flow-id': '18dce8db-f523-4ff1-8806-0719de3fd367',
'x-api-key': 'adobe_io',
'sandbox-name': 'developer-lifecycle-dev1'
'Api-Key': 'd6b73f9c1859dc462e6de8dee3de1eb2FFFFNRAL'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's a dupe in flush-worker, maybe centralize, not sure

Comment thread src/telemetry-lib.js
name: 'aio.cli.telemetry',
type: 'gauge',
value: 1,
// id: Math.floor(timestamp * Math.random()),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete this commented line?

@purplecabbage

Copy link
Copy Markdown
Member Author

Closing in favor of proxy version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants