Report Grok usage with recorded and estimated spend - #3135
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09cf7edb0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
clawsweeper
Bot
added
merge-risk: 🚨 compatibility
|
Codex review: blocked before merge. Reviewed September 6, 2026, 5:57 PM ET / 21:57 UTC. ClawSweeper reviewWhat this changesThe PR counts completed Grok CLI turns, reports recorded spend with labeled price estimates, and imports qualifying OpenCodex OAuth usage into Usage & Spend. Merge readiness⛔ Blocked before merge - 3 items remain This remains useful work: current main and v0.56.7 still count context occupancy and publish no native Grok spend. Earlier pricing findings are resolved, and no new actionable defect was found; product acceptance remains unresolved. Priority: P2 Review scores
Verification
How this fits togetherCodexBar turns local usage logs into token and spending summaries for its menu and Usage & Spend dashboard. This change updates Grok accounting and the optional OpenCodex log importer while keeping remote subscription quotas separate. flowchart TD
A[Grok CLI session logs] --> B[Bounded completed-turn scanner]
C[Recorded spend and public prices] --> B
B --> D[Windowed usage summaries]
E[Optional OpenCodex logs] --> F[Reported OAuth attempt filter]
F --> G[Usage and Spend dashboard]
D --> G
D --> H[Grok menu]
Decision needed
Why: The corrected divisor and runtime evidence support the accounting, but they cannot supersede the earlier owner ruling or establish acceptance of the new display semantics. Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Keep bounded completed-turn accounting and source-specific disclosures, with dollar visibility governed by an explicit owner-approved upgrade policy. Do we have a high-confidence way to reproduce the issue? Yes: current main directly sums context-occupancy fields, and independent corpus evidence shows the resulting undercount. This read-only review did not execute a reproduction. Is this the best way to solve the issue? Yes for the accounting: completed-turn logs, recorded-first pricing, and request-time OpenCodex provenance provide a coherent correction. Default dollar visibility remains a separate product choice. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 31131374b2c5. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (50 earlier review cycles; latest 8 shown)
|
08360b5 to
e3cd3b9
Compare
olddonkey
changed the title
clawsweeper
Bot
added
merge-risk: 🚨 auth-provider
|
Both automated findings are addressed, plus the review's other checklist items. The inline comments were left against P1 — Preserve the Grok fallback on repeated probe failuresFixed in if provider == .grok {
if self.tokenSnapshotPublicationForCurrentProviderConfig(for: provider) == nil {
Task { @MainActor [weak self] in
await self?.scanAndPublishGrokLocalTokenSnapshot(...)
}
}
} else if Self.tokenCostRequiresProviderSnapshot(provider) {
self.clearTokenSnapshot(for: provider)
}Regression coverage is in P2 — Refresh pricing before scanning Grok sessionsCorrect, and thank you — this was a genuine gap and not one the local tests would have surfaced. Fixed in Note the inline comment still points at Coverage: Real-session evidence
The same corpus on Those figures were cross-checked against an independent reimplementation of the pricing formula over the same logs; the two agree to the cent. Merge risk / branch stateRebased onto current One thing deliberately left undone: no |
clawsweeper
Bot
added
rating: 🧂 unranked krab
|
Both new findings addressed at P1 — Do not map every xAI log record to the Grok subscriptionAgreed, and taken as specified rather than argued down. Routing on the prefix alone is right for the case that motivated this — traffic authenticated with the user's Grok account, which is what makes it consume SuperGrok quota — but it silently folds an API-key user's pay-as-you-go xAI spend into the subscription row. CodexBar already models the developer platform as its own The usage log carries no per-record credential evidence; I checked every field emitted for So attribution now requires positive OAuth evidence:
Coverage:
P2 — Republish the Grok snapshot after a missing catalog refreshesI looked at this closely and am deliberately not adding a republish path. Reasoning, so you can overrule it if you disagree: The refresh is fire-and-forget, so the scan that requests it returns whatever the cache currently holds — that part is accurate. But the parse cache stores parsed turns, not prices, so aggregation and pricing re-run on every The alternative — plumbing a completion signal back across the actor boundary into the Happy to build it if you would rather have it. EvidenceThe attribution itself only becomes visible in the app: The two halves stay distinguishable in the UI, which makes the attribution legible rather than something you have to take on trust: the CLI goes through the responses API so its SKU is
Independently recomputing the same corpus agrees to the cent on both halves: 54,121,501 tokens / $50.52 for the CLI logs, and $166.25 across 1,520 OpenCodex The negative direction — API-key traffic staying off the Grok row — is covered by tests rather than a screenshot, since demonstrating it live would mean rewriting the machine's OpenCodex config. StateFull suite on 
|
clawsweeper
Bot
added
proof: 📸 screenshot
olddonkey
changed the title
|
Addressed both current findings in
Validation on the exact pushed head:
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
merge-risk: 🚨 availability
Rebase onto current main and reconcile the surfaces it moved: - Route the Grok local summary through the injectable `localSummary`/`cliVersion` seams steipete#3237 introduced, keeping the 365-day lookback and the models.dev pricing refresh in the injected defaults rather than at the call sites. - Regenerate the Codex parser hash and record main's `21f10143afe00c55` as a compatible predecessor; the Grok-only parser additions leave persisted Codex rows unchanged. Drop the stale branch-internal predecessor entry. - Re-anchor the provider-architecture gatekeeper suppressions and allowlists to the line numbers main's Spend dashboard and usage store now sit at. - Anchor the unstubbed Grok publish test to the real clock instead of a fixed calendar day, which had drifted outside its own seven-day window. - Move the xAI pricing-fingerprint test next to the other Grok pricing tests so `CostUsagePricingTests.swift` stays byte-identical to main and inside the file- and type-length limits.
0.56.0 shipped without this change, so the entry belongs in the open `0.56.1 — Unreleased` section rather than the released one.
- Queue the refreshable Grok summary through `CostUsageScanExecutor`. It called the synchronous corpus scanner inline, so the probe and descriptor callers ran a potentially multi-minute scan on the cooperative pool the executor exists to protect. A cancelled scan now reports unestablished coverage rather than an authoritative zero. - Derive the Grok lookback cutoff from the local start of day minus `historyDays - 1` so the scan window matches the inclusive local-day window `narrowed(toHistoryDays:)` renders, instead of collecting a partial extra day consumers discard. Each fix carries a regression that was confirmed to fail without it.
- Regenerate `CodexParserHash.value` to `69eee1ccce7ed69c`. - Record main's `50a2507f4c10d080` in `CostUsageStore.compatiblePredecessorParserHashes` and in its exact-set assertion.
steipete#3345 measured `costUsdTicks` on an independent 934-turn corpus and established its divisor as 1e10. My own 2026-08-21 backtest had concluded 1e9, because that quotient came out to exactly 1.7x the public card on a 25-turn sample drawn entirely from an xAI promotional window. Re-running that arithmetic on this branch's corpus reproduces both readings to four decimals, so the divisor was wrong rather than the measurement. The consequence is not a rounding artifact. The six turns behind this branch's gated proof publish $2.181282 reconstructed from the public card; the spend the CLI recorded for the same turns is $0.370818. steipete#3345 reports 6.66x across a whole promotional period. Read `costUsdTicks` when a record has it, since it already carries the price tier and any promotional rate, and fall back to the public card only where it does not. Fallback entries are counted as estimated rather than priced, and the window publishes `.vendorMetered`, `.listPriceEstimate`, or `.mixed` accordingly, so the dashboard and menu name the source instead of asserting one. A record that omits the field or reports 0 has no recorded spend; steipete#3345 measured that at 2 of 934 turns. The fixtures no longer inject a sentinel tick value by default, so the existing suites keep exercising the fallback path. Note for the maintainer: the 2026-08-21 owner ruling chose the public card over `costUsdTicks` for Grok, and my incorrect divisor was part of the evidence behind it. That ruling should be revisited on these numbers rather than treated as overridden by this commit.
ClawSweeper found two window-projection defects that the 365-day Grok snapshot newly exposes, since a maximum-window snapshot is now narrowed per consumer. `grokLocalTokenSnapshot` recomputed tokens and requests from the retained days but copied the published cost total, so a 30-day menu view could render the 365-day dollar amount beside a 30-day token count. It now sums the retained rows. Both that projection and `CostUsageTokenSnapshot.narrowed(toHistoryDays:)` carried the snapshot-wide provenance into the derived window, so a window that excluded every recorded row still claimed recorded spend — newly reachable now that a Grok window can be mixed. The Grok projection derives the disclosure from the coverage counts of the days it kept, because its scanner counts a recorded turn as priced and a card fallback as estimated. The generic narrowing applies the existing `CostProvenance.forWindow` rule instead, which stops a costless window from claiming a provenance without guessing what the surviving rows of another provider mean. Each fix carries a regression confirmed to fail without it.
|
Rebased onto current main The existing opt-in OpenCodex import now accepts only reported physical Grok OAuth attempts with request-time provenance from OpenCodex #3642. It excludes API-key/historic/unknown records, avoids combo-parent and duplicate counting, preserves provenance through the derived cache, and keeps OpenCodex list-price estimates distinct from native CLI-recorded spend. The menu remains backed by native CLI logs. Validation on this head:
The PR description contains the exact commands and evidence. The default Grok dollar surface remains an owner/product decision before merge. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Addressed both custom-pricing findings on
The new regression suite reproduced both findings before the repair. Validation on the final tree: The PR body now contains the final-head validation and the explicit standalone-price contract. The Grok dollar-display decision remains with the maintainer. Please review this final description and head. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Synced #3135 with main The generated parser-hash conflict is resolved by regeneration from the merged source ( Main's Linux cache fixes, stacked-chart rendering, provider/widget changes, and release metadata are preserved. The two Grok changelog entries now live under 0.56.8 Unreleased; published release sections match main. The evidence report also records that the producer contract landed through the attributed OpenCodex #3762 carry; its original captured bytes remain pinned and are not represented as a new capture of that carry. Validation on this final head:
The PR description now contains the final source, commands, and evidence. The default Grok dollar display remains an owner decision. Please review the final head and description. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Updated to Current-head validation: The default Grok dollar display remains an owner decision, and the prior maintainer change-request review still needs reconsideration. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@steipete The latest ClawSweeper review of |
|
Updated to
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Final validation for
The PR body is updated with the current source and validation. The default Grok dollar-display policy and the earlier maintainer changes-requested review still require the owner's decision. |
2 participants
Problem and resulting behavior
Grok's local fallback reads ending context occupancy from
signals.json, which is not completed-turn usage, and publishes no local cost. This PR reads completed turns from bounded native CLI logs and carries recorded-versus-estimated cost through the menu and Usage & Spend.session/updateand_x.ai/session/updaterecords inupdates.jsonl, with daily, model, and request breakdowns.usage.costUsdTicks / 1e10as the authoritative turn total. Count it once. Show nested model dollars only when all nested ticks exist and reconcile to that total; otherwise retain model tokens with unknown model dollars. Missing recorded cost falls back to disclosed public xAI list prices.Grok CLI-recorded spend, list price where unrecorded · not a bill.OpenCodex integration
With the existing Include OpenCodex usage logs switch enabled (off by default), Usage & Spend also includes reported physical Grok OAuth attempts. The producer contract originally proposed in OpenCodex #3642 has now landed in
devthrough the attributed carry #3762, merged asf00f2bcaea251ebe7ad4de9e38337b4be0ccee47. It writesattempts[].credentialSourcefrom the resolved upstream transport. Compatibility is verified against the pinned producer commit in the evidence below; availability in a released OpenCodex version is not claimed.Only
provider: "xai",credentialSource: "grok-oauth"attempts with an upstream send and reported token usage qualify. API-key traffic, historic rows, locally answered requests, unknown sources, and estimated/unreported token counts stay excluded. Current configuration and top-level credential metadata cannot retroactively classify usage.Combo requests contribute each qualifying attempt's own token counts, never the parent aggregate. Duplicate request IDs are resolved before grouping; duplicate attempt ordinals are rejected. SQLite cache schema 3 retains attempt metadata and rebuilds older derived caches from the log. OpenCodex dollars use list prices and remain estimates, including when combined with native CLI-recorded spend. Missing token classes and unknown prices retain tokens without inventing a dollar value. The Grok menu continues to use native CLI logs.
Producer-to-dashboard evidence and review fixes
The committed evidence report and raw ledger fixture contain output from OpenCodex's unmodified production handlers and durable usage writer at
146ed679c9633e5d68726217fcadc8e0b107339b. Two localhost HTTP requests exercised OAuth 401 replay through Responses and native Chat API-key dispatch. Upstream responses and credentials used isolated fixtures; this is production-path capture evidence, not live vendor authentication or billing evidence. The capture helper rejects unexpected external fetches and is reproducible against that pinned checkout.CodexBar imports the exact captured bytes through the production disk loader, with no injected entries or loader closure, and reopens the persisted cache:
Standalone reports retain explicit xAI custom-price estimates from either the caller overlay or the application overlay, including known zero. Raw xAI records without an explicit price remain token-only, and the subscription fan-out still excludes API-key and historic records.
opencodeandopencode-freeretain their existing catalog and custom prices.Application overlays first match the original model name before the provider-qualified catalog name. Bare keys retain precedence when both keys exist; incomplete rates stay unknown, cached-input accounting is preserved, and caller-supplied custom pricing still takes precedence over the application overlay. New regressions cover these cases through the application-overlay parameter and the standalone disk/cache loader. Coverage includes known-zero overrides, incomplete rates, cache accounting, and both snapshot and application overlays.
Both async native Grok scan entry points propagate the executor's cancellation callback through discovery, JSONL reads, and aggregation. An in-flight regression cancels after parsing begins, observes
CancellationError, and confirms the next queued scan runs within one second. Cancelled partial parses are uncacheable and cannot establish complete history. The final serial proof stopped at 4,874/40,000 decoded records and released the queue after 0.001864583 seconds. The evidence report also retains the initial measurement.Bounds and compatibility
Native scans run on the dedicated executor with limits of 64 MiB / 20,000 turns per file, 1 MiB per record, 256 sessions / 256 MiB / 100,000 turns per scan, and 4,096 discovery entries. The process cache retains at most 64 files or 50,000 turns. Cancellation and truncated history cannot publish complete coverage.
Merged main
31131374b2c547914243f54303b2c5f5f09d77e9, retaining the new Claude account-warning preservation, MiMo malformed-row handling, and earlier Poe elapsed-week fix. The architecture-test line records match the merged source while retaining this PR's Grok bridge and reference fingerprints. All upstream and Grok release notes are retained under 0.56.8 Unreleased. The previously integrated Linux cache replacement fixes, stacked-chart rendering, provider/widget changes, and release metadata remain intact. Regenerated the native parser hash from the resulting source:1a4afd74939160fd. The released main hash9547dc9d7b7675f6and previous PR hash0bd6588c70196700are compatible predecessors because the catalog persistence change does not alter native parsed rows or checkpoints; existing predecessors are preserved. Parameterized SQLite adoption tests exercise both newly retained hashes without rebuilding. Release notes are under0.56.8 — Unreleased, with published sections unchanged.Validation
Head:
6f4a7d1058de2180f1f293824074cbbcd15ec5cc.make checkpassed: zero SwiftLint violations in 2,138 files.c3919a224: 136 tests in five suites passed (4.392 seconds).make testpassed: 1,032 selections, 86/86 groups successful on the first attempt, zero failures, retries, or timeouts (629.6 seconds execution; 633.9 seconds total).c3919a224passed: 206 tests in 14 suites (5.747 seconds), using--no-parallelas the repository test harness does.git diff --checkpassed. All nine GitHub CI checks passed on6f4a7d105, including both macOS shards, all three Linux builds, lint, and the aggregate gate.The supplemental proof and local-corpus measurements below were captured at
c3919a224, before the latest main sync. The Grok accounting and OpenCodex import source is unchanged by that sync. The supplemental proof uses the repository’s serial execution mode because these suites share scanner-cache counters; the required full suite passed again on the current head after the latest upstream integration.The September 6 local-corpus proof retained
vendorMeteredfor populated windows: 7 days = 95,891,889 tokens / $12.57875572; 30 days = 98,631,812 tokens / $12.94957366. The empty 1-day window retainedunknown. Controlled JSONL fixtures also proved recorded-only and estimate-only windows after narrowing mixed source history.The OpenCodex regressions cover mixed OAuth/API-key/provider attempts, historical and malformed records, duplicate suppression, cache reopen/incremental append/schema upgrade, missing-price behavior, recorded-plus-estimated date filtering, and the dashboard's opt-in switch. Existing native regressions cover bounded parsing, outer/nested cost reconciliation, local fallback freshness, account isolation, and menu/dashboard provenance.
All ordinary tests suppress Keychain access and isolate provider files. The optional native proof reads local Grok logs and the cached pricing catalog without authentication, browser-cookie import, or live provider requests. Presentation evidence uses production menu/dashboard models; no app-bundle screenshot is claimed.
Maintainer decision
Please revisit the 2026-08-21 Grok cost ruling before merge. It preferred public-card pricing based partly on my incorrect
1e9divisor; #3345 established1e10, and the corrected measurement explains the difference. This branch uses recorded spend with public-card fallback.Whether existing Grok users should receive this disclosed dollar surface by default remains an owner decision. The accounting corrections and source labeling do not override that decision. Maintainer approval is still required.