Skip to content

feature: Report statistics to a central collector #898

Open

Description

@esnible

Feature Description

We'd like to collect AuthBridge usage statistics.

For example, how much are people using AuthBridge, which plugins, what is the cost savings from the cost-saving plugins.

This issue is to enable AuthBridge to report statistics to a remote endpoint, so that another tool can produce summaries of AuthBridge usage for a group or an entire organization.

cc @jimcadden

Proposed Solution

Either a plugin or core feature to ship statistics (e.g. token count by hour) to a remote always-up HTTP or message-based endpoint.

Want to contribute?

  • I would like to work on this issue.

Additional Context

No response

Activity

  1. added this to the Release V0.9.0 milestone on Sep 10, 2026
  2. github-actions commented on Nov 10, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. If this issue is still relevant, please comment or remove the stale label.

  3. mrsabath commented on Nov 11, 2026

    @mrsabath
    Member

    Still relevant — removing the stale consideration, and recording a scope decision that
    makes this smaller than when it was filed.

    #1810 proposes the per-event export layer this issue can consume, rather than #898
    building its own collection path. The split:

    #1810 A content-free per-event projection of SessionEvent (~30 scalars: host, model, durationMs, the four token counts, status, client, tool/message counts), an NDJSON sink, and the ability to replay it out of the 30-day archive for backfill
    #898 (this issue) Aggregation and shipping: roll those records up — "token count by hour", per-plugin usage, savings from the cost-saving plugins — and deliver them to a remote always-up endpoint

    So the work here becomes aggregate + transport + the organisational questions, and stops
    including "get the data out of the proxy in a usable shape." That part is #1810.

    Why this way round

    Two exporters reading the same events, with two schemas and two privacy reviews, is the
    outcome worth avoiding. The projection is also the harder half to get right, and it is
    shared: the fields this issue needs for "token count by hour" and "savings from the
    cost-saving plugins" are a strict subset of #1810's Tier 1, so nothing here needs a field
    that layer would not already carry.

    One specific trap worth inheriting rather than rediscovering, since it bites exactly the
    metric this issue asks for. Every inference request re-sends the whole conversation, so
    counting ToolResults rows overcounts by ~272x on a real archive (2,765,534 raw rows
    against 10,183 distinct, deduped by ToolUseID). A naive "tool error rate" comes out at
    ~8 per LLM call; the true figure is 3.6%. #1740 adds the per-event counts that make this a
    sum rather than a dedup pass. Any aggregate built here should read those, not the slices.

    What is still open on this issue, and is not in #1810

    • Aggregation window and resolution. No long-term view of inference server performance to compare against current performance #1466 notes the existing ledger serves 7d and
      month as a single bucket, so even the error rate cannot be plotted over days. Whatever
      this ships should not inherit that.
    • Transport and delivery semantics — endpoint config, retry, batching, what happens when
      the collector is unreachable for a week.
    • What leaves the machine, and consent. feature: export session events for post-processing — content-free per-event projection, NDJSON sink, archive-replayable #1810's Tier 1 is content-free by construction,
      which is most of the privacy argument, but two fields still need a decision before an
      organisational collector: httpPath can carry a secret in a path segment (its own doc
      says so — a bot token or webhook path lands there verbatim), and client is a spoofable
      request header, so it must not become an identity key in an org-wide report.
    • Whether per-developer attribution is in scope at all, which is a policy question rather
      than a technical one and is the part most likely to decide whether this gets adopted.

    Happy to be corrected on the boundary if you had something different in mind for this issue
    — the aggregate/per-event split is the version that avoids duplicate work, but you filed it
    and may have intended a narrower or broader scope.

    cc @esnible @jimcadden

  4. huang195 commented on Nov 17, 2026

    @huang195
    Member

    @boazdavid we had this issue opened for a long time. assigning it to you. thanks for picking it up!

  5. removed this from the Release V0.9.0 milestone on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

nice to haveIt would be nice to have this fixed

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions