Skip to content

Measure AI latency, token use, repairs, and edit isolation #34

Closed

Description

@triggeredcode

Goal

Create a trustworthy baseline before changing providers or prompt context.

Scope

  • Record time to first token and total time per Input, Command, and Advisor request.
  • Record input/output token counts when a provider supplies them; otherwise retain a clearly marked size estimate.
  • Record provider, model, lane, cancellation, repair attempts, final outcome, and affected block count.
  • Extend eval reporting with edit success and unrelated-block modification checks.
  • Never log prompt contents, API keys, or user document text.

Acceptance criteria

  • The same repeatable scenarios can be compared before and after later roadmap work.
  • Missing provider usage data is represented honestly, not as zero.
  • Instrumentation has negligible user-visible overhead.
  • Development/eval output explains the metrics without exposing note contents.

This is an enabling issue; it does not change provider selection or prompt construction.

Activity

  1. triggeredcode commented on Oct 19, 2026

    @triggeredcode
    OwnerAuthor

    Claimed. Implemented on feat/provider-context-roadmap in commit e873ae0 and included in draft PR #46. This issue will close automatically when the PR is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions