Skip to content

fix(antigravity): read sidecar-less WAL conversations with an immutable fallback - #3532

Open
urda wants to merge 2 commits into
steipete:mainfrom
urda:urda/agy-immutable-fallback
Open

fix(antigravity): read sidecar-less WAL conversations with an immutable fallback#3532
urda wants to merge 2 commits into
steipete:mainfrom
urda:urda/agy-immutable-fallback

Conversation

@urda

@urda urda commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

One cleanly closed Antigravity conversation makes the whole Antigravity spend history unavailable on macOS. The Usage & Spend pane then lists no Antigravity source and shows "Refresh failures: 1", and codexbar cost --provider antigravity returns an empty report with historyCoverageIsEstablished: false.

The cause is a SQLite platform behavior. When SQLite closes the last connection to a WAL database cleanly, it deletes the -wal and -shm sidecars. Apple's system SQLite (3.51.0 on macOS 26) then declines a read-only open of that file with SQLITE_CANTOPEN, because a read-only connection may not create the sidecars. The reader marks that database incomplete, the report becomes partial, and the fetcher withholds every row by design.

This PR adds a narrow fallback in AntigravityLocalReader.readDatabase:

  • When the ordinary read-only attempt fails with SQLITE_CANTOPEN before any row is read, and no -wal sidecar exists next to the database, the reader retries that one database with file:<path>?immutable=1 and SQLITE_OPEN_READONLY | SQLITE_OPEN_URI.
  • An absent -wal means no WAL connection holds the database, so the main file alone carries the checkpointed state. A database with a -wal sidecar present stays unavailable, as before.
  • The immutable open never creates or touches sidecars. The path is percent-encoded so %, ?, and # cannot change the URI.
  • An immutable connection neither locks nor detects changes. The reader therefore records the file's size, modification time, file system number, and 100-byte header before the retry, and accepts the result only when they and the sidecar state are unchanged afterwards. A writer that appears and checkpoints during the retry leaves that database incomplete, which keeps the report partial and withheld, as before.
  • Statistics.immutableFallbacks counts the retries.

The #3212 WAL test carried the comment "Some SQLite builds decline read-only WAL access without sidecars; that must stay unavailable." This PR argues against that ruling with field data: every new conversation lacks sidecars until some later bulk event recreates them, so the current policy blanks Antigravity spend for most macOS users most of the time.

Evidence

On the reporting machine, 214 conversation databases exist. 213 have sidecars and open read-only. One has a WAL header (writer version 2, read version 2) and no sidecars, and it fails:

$ sqlite3 -readonly 31125604-ffff-41db-b409-11d11a858825.db 'select 1'
Error: in prepare, unable to open database file (14)

A one-off Swift probe that links the system SQLite3 library reproduces rc=14 at prepare after a successful open and BEGIN DEFERRED. Python's bundled SQLite 3.53.4 opens the same file and creates the sidecars. Sidecar birth times on the 213 databases cluster on five moments (184 of them on one day), not on each conversation's creation, so new conversations regularly lack sidecars.

With this change, the same CLI command on the same machine returns 38 daily rows from 2026-06-20 through 2026-09-06 with historyCoverageIsEstablished: true, and the file still has no sidecars afterwards.

Trade-off

An immutable connection skips locking and change detection, so the read alone cannot prove one snapshot. The stability bracket above turns that into a fail-closed check: any change to the main file or the appearance of a -wal sidecar during the retry marks the database incomplete. The next scan reads the stable file. A temp copy would carry the same window and cost I/O on every refresh.

Tests

All three cases keep the existing raw-SQLite control pattern: a separate control fixture reports which outcome the platform's SQLite produced, and the expectations follow that outcome. On Apple's SQLite the control returns SQLITE_CANTOPEN and the fallback path runs; on a SQLite that creates the sidecars itself, the ordinary path runs and no fallback is counted.

  • AntigravityLocalWALTests: the existing sidecar-less case now expects .complete coverage and the fixture row on every platform, immutableFallbacks equal to 1 only when the control declined, and no sidecars afterwards in that case.
  • New: a sidecar-less database whose file name contains %, ?, and # reads completely through the escaped URI.
  • New: a database with an unreadable empty -wal sidecar never takes the fallback. Apple's SQLite declines that open, so the guard is exercised on macOS and the report stays partial. A root CI user can still read the file, and the control accepts that outcome.
  • New: a writer that reopens the database, commits a row, checkpoints, and closes while the reader is inside the immutable retry. On Apple's SQLite the fallback runs and the result is incomplete; the next scan is complete with both rows. On a SQLite that takes the ordinary path, the read-only snapshot excludes the later write as in the existing coordinated-writer test.
  • New: an unchanged two-row sidecar-less database reads completely through the bracket.

Commands run

swift test --filter AntigravityLocalWALTests
make check
make test
.build/debug/CodexBarCLI cost --provider antigravity --format json --days 365

Docs

docs/antigravity.md describes the fallback and its guard under "Local token history". No CHANGELOG entry, per repository convention.

@clawsweeper

clawsweeper Bot commented Sep 9, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@urda
urda marked this pull request as ready for review September 9, 2026 23:53
@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Sep 9, 2026
@clawsweeper

clawsweeper Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed September 10, 2026, 11:18 AM ET / 15:18 UTC (Revision 5).

ClawSweeper review

What this changes

Adds a guarded SQLite fallback for Antigravity conversation databases without WAL sidecars, with regression coverage, documentation, and a separate billing-test timing adjustment.

Merge readiness

Ready for maintainer review

This remains a useful fix: current main and v0.58.0 lack the fallback. The earlier review findings are addressed, the reported macOS run demonstrates restored history, and no blocking introduced defect was found.

Priority: P2
Reviewed head: a95e2b844520ed97c656b133e7aa50aa3abe7cfd

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused fix with relevant contributor runtime evidence, targeted regressions, and resolved prior findings.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The complete captured body reports the production cost CLI reading the same macOS conversation set after the fix: 38 daily rows and established coverage replace unavailable history, without creating sidecars. The new stability tests supplement that real-run evidence.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The complete captured body reports the production cost CLI reading the same macOS conversation set after the fix: 38 daily rows and established coverage replace unavailable history, without creating sidecars. The new stability tests supplement that real-run evidence.
Evidence reviewed 10 items Repository policy and scope: Read the full root AGENTS.md and applied its focused-test, provider-isolation, and safe-validation guidance. No nested AGENTS.md files or maintainer-note files were found under the inspected source, test, documentation, and agent directories. The repository profile also prohibits automatic closure.
Fallback and publication boundary: The fallback requires SQLITE_CANTOPEN before payload rows and absence of a WAL file. It compares file identity, size, modification time, header, and WAL absence after reading; detected changes mark the source incomplete. CostUsageFetcher continues to reject unavailable reports before publication.
Current main still needs the fix: The fetched main implementation uses ordinary SQLITE_OPEN_READONLY and returns incomplete on failed opening or schema inspection; it has no immutable retry.
Findings None None.
Security None None.

How this fits together

CodexBar reads Antigravity conversation databases to build local token history for its CLI and usage displays. Incomplete database scans prevent that history from being published as established usage.

flowchart TD
  A[Local conversation databases] --> B[Ordinary read-only SQLite scan]
  B --> C{Cannot open and no WAL file?}
  C -->|Yes| D[Immutable read with stability checks]
  C -->|No| E[Existing scan result]
  D --> E
  E --> F{Complete history?}
  F -->|Yes| G[Publish daily token history]
  F -->|No| H[Withhold incomplete history]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test delta Production +117/-13; tests +134/-2; docs +15/-6 Production growth implements the guarded retry and accounting, supported by focused WAL regressions.

Technical review

Best possible solution:

Restore stable, closed-conversation history through the bounded fallback while retaining normal WAL coordination and withholding scans whose stability cannot be established.

Do we have a high-confidence way to reproduce the issue?

Yes: the reported Apple SQLite failure maps directly to main's incomplete-scan path, and the sidecar-less WAL fixture exercises that platform condition. This review did not execute tests or a live account probe.

Is this the best way to solve the issue?

Yes: the retry stays within the existing reader, preserves its budgets and publication checks, and follows an established repository pattern with an additional stability guard.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 7fdc17636f16.

Labels

Label justifications:

  • P2: Repairs unavailable local Antigravity token history with a bounded provider-specific change.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The complete captured body reports the production cost CLI reading the same macOS conversation set after the fix: 38 daily rows and established coverage replace unavailable history, without creating sidecars. The new stability tests supplement that real-run evidence.
  • proof: sufficient: Contributor real behavior proof is sufficient. The complete captured body reports the production cost CLI reading the same macOS conversation set after the fix: 38 daily rows and established coverage replace unavailable history, without creating sidecars. The new stability tests supplement that real-run evidence.

Evidence

What I checked:

  • Repository policy and scope: Read the full root AGENTS.md and applied its focused-test, provider-isolation, and safe-validation guidance. No nested AGENTS.md files or maintainer-note files were found under the inspected source, test, documentation, and agent directories. The repository profile also prohibits automatic closure. (AGENTS.md, a95e2b844520)
  • Fallback and publication boundary: The fallback requires SQLITE_CANTOPEN before payload rows and absence of a WAL file. It compares file identity, size, modification time, header, and WAL absence after reading; detected changes mark the source incomplete. CostUsageFetcher continues to reject unavailable reports before publication. (Sources/CodexBarCore/Providers/Antigravity/AntigravityLocalSQLite.swift:72, a95e2b844520)
  • Current main still needs the fix: The fetched main implementation uses ordinary SQLITE_OPEN_READONLY and returns incomplete on failed opening or schema inspection; it has no immutable retry. (Sources/CodexBarCore/Providers/Antigravity/AntigravityLocalSQLite.swift:54, 7fdc17636f16)
  • Latest release check: The v0.58.0 source also retains the ordinary read-only implementation without this fallback. (Sources/CodexBarCore/Providers/Antigravity/AntigravityLocalSQLite.swift:54, 88fa2f45fa1e)
  • Review continuity: The prior snapshot concern is addressed by the before/after stability comparison and coordinated-writer regression; the contradictory documentation was corrected. Comparing the previous reviewed head with this head shows only the OpenCode Go test adjustment. (Tests/CodexBarTests/OpenCodeGoUsageFetcherErrorTests.swift:642, a95e2b844520)
  • Contributor real behavior evidence: The captured PR body reports a real macOS 26 installation with 214 conversation databases: one sidecar-less database caused SQLITE_CANTOPEN and withheld history. After the change, the production cost CLI returned 38 daily rows with historyCoverageIsEstablished true, while that file still had no sidecars. This is contributor-reported runtime evidence, separate from synthetic tests; the body was supplied completely.

Likely related people:

  • urda: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Yuxin-Qiao: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (4 earlier review cycles)
  • reviewed 2026-09-09T23:56:05.028Z sha 895a9ba :: blocked before merge. :: [P2] Preserve a stable snapshot throughout the fallback read
  • reviewed 2026-09-10T00:44:49.212Z sha 476c332 :: needs maintainer review before merge. :: none
  • reviewed 2026-09-10T00:48:36.666Z sha 476c332 :: needs maintainer review before merge. :: none
  • reviewed 2026-09-10T14:16:22.619Z sha 3ea3844 :: needs maintainer review before merge. :: none

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 895a9ba145

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread Sources/CodexBarCore/Providers/Antigravity/AntigravityLocalSQLite.swift Outdated
Comment thread docs/antigravity.md Outdated
@urda

urda commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@urda

urda commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 10, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. labels Sep 10, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 476c3322d4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@urda
urda force-pushed the urda/agy-immutable-fallback branch from 476c332 to 3ea3844 Compare September 10, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant