Compact File authority history with recognized, backed-up format upgrades

docs/architecture/rfcs/ledger/shared-goal-authority-state-provider-v0/2026-09-24-default-cutover-reconciliation.md

@@ -3,7 +3,7 @@
 - Baseline: `41ba6f4d9` on `main`, 2026-09-25; open PR states are a snapshot, not merge promises.
 - Owners: overall roadmap #4574 R5/G2; shared authority L2–L9/D1–D3; TS migration T1–T4.
 - Delivered #5040: current registration admission, complete saved migration intent and truthful fence recovery.
-- Current increment: long-history closeout reuse and TS-owned monitor evidence; the migration packages below remain open.
+- Current increment: File retained-state compaction stacked on #5063; the migration packages below remain open. #5063 has now merged; rebased onto main `eaa0c0fd0`.
 - This checkpoint supersedes numerical remaining-PR estimates in earlier delivery entries.
 
 ## Correct the accounting
@@ -32,6 +32,27 @@ transaction, complete-source and source-witness owners. The latest formal #4224
 (801.81 ms versus 250 ms). #4931 has not supplied a formal exact-head rerun.
 Reaching the planned ten-day soak end date is not a passing report.
 
+## File history cost: delivered slice, separate acceptance
+
+A detached long-history snapshot exposed File's repeated full-projection write
+cost. #5063 bounds RPC waits and retains verified read views; it does not remove
+that physical duplication. This stack reuses the SQLite-owned shared TS state-log
+codec for File checkpoints/deltas, preserving logical revisions, receipts and
+full scan results. See [format, upgrade and limits](../../../../reference/file-authority-state-log.md).
+Normal reads/writes accept only v1. Explicit upgrade automatically backs up and
+verifies File/SQLite before physical migration, and installation invokes it before
+activation. Cross-provider movement reuses logical archive recovery. Older binaries
+cannot read v1. Conversion, cold verification,
+steady writes and warm reads require separate evidence; cache limits are unchanged.
+
+Before this slice, the audited implementation plan therefore contains **four
+named packages**: this evidenced File cost repair plus the three below. After
+this slice it contains those **three planned packages**, not a new unchanged
+“5–8 PRs” estimate. #5063, #5054 and #4931 are existing PRs, not three new tasks.
+D1–D3 and an exact total PR count remain unqualified. This storage repair retires
+no Python business owner; bounded Python deletion belongs to actual caller
+migration in the packages below.
+
 ## Three concrete next code boundaries
 
 This delivery repairs integrated migration admission: stale registry snapshots
@@ -41,7 +62,7 @@ not implement another store or close the whole migration package or D2 gate.
 | Proposed PR | Observable result and owner | Exit |
 | --- | --- | --- |
 | 1. External-effect execution fencing | Lease/effect owners protect the actual execution interval, takeover, timeout, exit and uncertain completion. Reuse merged #4994/#4995. | Stale executors cannot continue or settle; real executor and receipt recovery matrix passes. A point-in-time proof check is insufficient. |
-| 2. Event-writer binding and whole-Goal migration/rollback | Bind event writer locks/atomic publication to existing outbox; integrate Markdown/event/lease capture, drain, saved cutover, consumers and fenced export/rollback; delete Python decisions replaced by TS. | Reuse #5003. Retain `event_log_writer_not_bound` until binding passes; close D1, command inventory and D3 cohort. One Goal without an event overlay does not prove this package. |
+| 2. Whole-Goal migration/rollback and retained source closure | Reconcile open #5054, which retires the legacy Todo event path and isolates supervisor logging; do not build another capture writer for a retired source. Integrate remaining supported sources, drain, saved cutover, consumers and fenced export/rollback; delete Python decisions replaced by TS. | Prove the supported command/source inventory after #5054, D1 and the D3 cohort; reject retired input explicitly. One Goal without a legacy event overlay does not prove every retained caller or rollback path. |
 | 3. Default entrypoints and bounded Python retirement | New Goals, settings, installation and packaged frontend/Lark/CLI select a qualified profile consistently; existing Goals have explicit migration/disable flows. | 1/2 and applicable D1–D3 pass; user entrypoints work; delete business writers only after their last callers migrate. Retain rendering, host IO and lawful import/export. |
 
 **Plan three named future implementation PRs, plus existing #4931 and outstanding

docs/architecture/rfcs/ledger/shared-goal-authority-state-provider-v0/2026-09-24-default-cutover-reconciliation.zh-CN.md

@@ -6,6 +6,24 @@
 - 当前增量：长历史 closeout 读取复用与 TS monitor 回执归一；没有完成下列迁移工作包。
 - 本检查点取代此前交付记录中的剩余 PR 数量估算。
 
+## File 历史编码与自动升级增量
+
+本次最初 stack 在 #5063 上；#5063 合入后已接到 main `eaa0c0fd0`。
+#5063 解决读取缓存和 RPC 预算，File 每次写入仍复制全历史完整投影。本次复用
+SQLite 已有 TS checkpoint/delta 编码，保留原始版本、回执和完整扫描结果。
+正常读写只接受新格式；安装／更新通过统一命令先自动备份、验证再迁移，旧解析
+只留在迁移工具。跨 provider 则复用逻辑归档，不新增两两转换器。
+[格式、备份迁移及成本边界](../../../../reference/file-authority-state-log.md)。
+
+当前增量前是四个具名开发包：本次有实际证据的 File 成本／升级修复，加下文三个
+业务边界；完成本次后仍剩三个规划包，不是继续复述“5–8 PR”。#5063、#5054、
+#4931 是已有 PR，不能重复计为新任务。D1–D3 的未通过证据另列，不能保证总 PR 数。
+本次没有删除 Python 业务 owner，只有升级命令的薄适配。
+
+整 Goal 来源闭环须按 #5054 当前方向核对：它退役旧 Todo event 路径并分离 supervisor
+日志，不应为已经退役的来源重建捕获 writer。剩余支持来源、consumer、回退与 cohort
+仍需完整验证。
+
 ## 先纠正统计口径
 
 此前“5–8”“6–8”“7–9”把宽泛工作包写成剩余 PR 数，部分实现合入、额外前置项

docs/architecture/rfcs/shared-goal-authority-state-provider-v0.md

@@ -34,6 +34,16 @@ bounded Python retirement. #4931 and outstanding D2 evidence are tracked
 separately. Three is a delivery plan, not a guaranteed total PR count.
 [Current inventory and exits](ledger/shared-goal-authority-state-provider-v0/2026-09-24-default-cutover-reconciliation.md).
 
+File retained-state storage now reuses the existing TS checkpoint/delta codec,
+stacked on #5063's verified read cache and RPC budgets. Original revisions,
+receipts and full historical projections survive the physical format upgrade.
+Normal reads/writes require v1. Installation runs explicit, verified backup and
+format migration; legacy decoding exists only in the migration owner. File and
+SQLite reuse logical archives for cross-provider isolated recovery.
+This adds no provider/default promotion and retires no Python business owner.
+[Automatic backup/migration, cold costs and qualification limits](../../reference/file-authority-state-log.md).
+
+
 ## Persistence route for steward scale (2026-09-16)
 
 [Roadmap](loopx-overall-roadmap-v0.md) R5 reuses D1 projection, D2 real-backend/capacity/applicable ten-day soak and D3 fenced cutover. R6 connects the selected shared profile to authenticated local/cloud execution. R1–R3 can advance on supported profiles without waiting for PostgreSQL or whole-Goal default promotion.

docs/architecture/rfcs/shared-goal-authority-state-provider-v0.zh-CN.md

@@ -28,6 +28,13 @@
 默认启用与最后一批有界 Python 退役。#4931 与 D2 的剩余资格证据单列；三个是
 可命名的开发批次，不是保证总 PR 数。[唯一当前清单与退出条件](ledger/shared-goal-authority-state-provider-v0/2026-09-24-default-cutover-reconciliation.zh-CN.md)。
 
+File 历史存储在 #5063 的读取缓存和 RPC 预算之上，复用现有 TS checkpoint/delta
+编码；物理格式升级保留原版本、回执和每条完整历史投影。正常读写只接受 v1，
+安装入口调用显式升级流程，先自动备份、验证再迁移；旧解析器仅用于迁移。File／
+SQLite 跨 provider 恢复复用逻辑归档。这不晋升 provider／默认值，也不算删除 Python
+业务 owner。[自动备份迁移、冷读成本与验收边界](../../reference/file-authority-state-log.md)。
+
+
 ## 旧观测退役检查点（2026-09-24）
 
 [当前交付清单](ledger/shared-goal-authority-state-provider-v0/2026-09-24-observation-retirement.zh-CN.md)

docs/reference/file-authority-state-log.md

@@ -0,0 +1,161 @@
+# Authority format upgrade and File retained state
+
+The File provider retains its `AuthorityStore` contract and `file_v0` routing
+identity. The physical document changes from `loopx_file_authority_store_v0` to
+`loopx_file_authority_store_v1`. This does not promote a Goal or select a provider.
+
+## Storage and semantics
+
+| | Old File document | Current File document |
+| --- | --- | --- |
+| Each committed row | Complete projection | Checkpoint at cursors 1, 65, 129, …; exact delta otherwise |
+| Events, operation ID, original receipts | Retained | Retained without rewriting |
+| Cursor and provider revision | Logical transaction identity | Identical after physical upgrade |
+| Historical reads | Full stored projection | Reconstruct the same projection from nearest checkpoint |
+| Runtime acceptance | Migration input only | Normal reads and writes |
+
+File reuses the TypeScript state-delta codec already used by SQLite. Object-key
+changes and array splices preserve all JSON data, including empty and `__proto__`
+keys. Writers prove reconstruction. Revisions still hash the logical full
+transaction, previous revision and store identity. The ledger remains logically
+append-only even though File atomically replaces its physical envelope.
+
+Cold reads verify every retained transaction and the final head; a valid head
+cannot hide a corrupt old delta or receipt. Verified pagination reconstructs at
+most 63 predecessor deltas plus the requested page. The exact-byte cache remains
+bounded. File still reads/hashes and rewrites one retained file: this reduces
+repeated data, not asymptotic growth. Cold verification can be slower. Measure
+upgrade, cold verification, warm reads and steady writes separately.
+
+## Format recognition
+
+```bash
+loopx --format json authority-archive inspect --source /absolute/store-or-backup
+```
+
+Recognition uses JSON schema tags or the SQLite file header plus database
+metadata and `user_version`, not filename extensions. It reports artifact kind,
+provider, physical format, Goal/store identity and migration route. Unknown
+versions and inconsistent SQLite version pairs are rejected by automatic
+upgrade. A recognized provider selector is only a routing record; PostgreSQL
+and NoKV still require their configured provider service for export.
+
+`metadata_only` recognition is not full history verification. Logical archives
+are verified through their complete digest/seal contract; backup packages check
+source bytes and lineage. Upgrade subsequently validates the complete store
+under its publication boundary. Multiple local stores are reported/upgraded
+independently, never silently chosen as a new live authority.
+
+Legacy Markdown/sidecar capture, project registry envelopes and shadow/outbox
+control records have different owners. They are not alternate File database
+encodings and are not rewritten by this command. Whole-Goal migration must use
+its source capture and writer-fence workflow.
+
+## Automatic upgrade and backups
+
+Normal readers and writers **do not accept the old File format**. Old parsing
+belongs only to migration. There is no migration-on-read or first-business-write
+conversion. SQLite's existing v1-to-v2 converter follows the same explicit gate.
+
+Default local installation and Windows installation run the candidate's upgrade
+command before launcher activation. `loopx update apply` for pip/pipx runs it
+after package installation and before host updates or service restart. Canary
+installation does not migrate stores. Source checkouts and externally managed
+package updates use the same explicit command:
+
+```bash
+# Preview selected runtime; global --runtime-root and --registry are supported.
+loopx --format json authority-archive upgrade
+# Back up and migrate known runtime roots from existing project registrations.
+loopx --format json authority-archive upgrade --all-known --execute
+# Read-only compatibility gate, also used before binary rollback.
+loopx --format json authority-archive upgrade --all-known --require-current
+```
+
+Discovery includes File and SQLite stores in each known runtime's authority
+folders, including unselected shadows and File rollback documents. It does not
+scan arbitrary home directories. Disconnected custom runtime roots must be
+supplied explicitly. Selectors, registries, writer fences and execution leases
+are not changed by format upgrade.
+
+Each store has its own durable publication boundary:
+
+- File holds the ordinary writer lock, verifies the entire old history, encodes
+  and decodes the target, and compares logical history digests. It saves exact
+  source bytes plus store identity under `format-backups/<source-sha256>/`,
+  verifies the backup and syncs directory entries before atomic replacement.
+- SQLite makes an online consistent backup, including committed WAL data. A
+  disposable copy proves the backup through the actual v1-to-v2 converter.
+  The source migration compares that history digest under `BEGIN IMMEDIATE`
+  before adopting new tables. Concurrent source advancement aborts the upgrade;
+  retry takes a fresh backup. It never silently discards intervening commits.
+- A manifest records source/target format, identity and integrity evidence.
+  File recovery uses manifest hashes and actual source/target bytes, not a
+  mutable completion flag. Interrupted conversion is retriable: before publish
+  the old store remains; after publish the new store is already current.
+
+Unknown formats, corrupt history, failed backups or failed validation stop the
+upgrade. A multi-store upgrade can have completed earlier stores when a later
+one fails; the report retains those results. Retry resumes per store. It does
+not pretend to roll back the whole runtime or overwrite later business writes.
+For package-manager updates, a failed data upgrade does not undo the package
+installation; service activation remains blocked until repair.
+
+## Provider migration and recovery
+
+Physical format upgrade preserves provider identity and old revisions. Changing
+providers uses the existing portable logical archive as the interchange format:
+
+```bash
+loopx --format json authority-archive export --goal-id example --archive /absolute/history.ndjson
+loopx --format json authority-archive verify --archive /absolute/history.ndjson
+loopx --format json authority-archive restore --goal-id example --archive /absolute/history.ndjson \
+  --destination /absolute/new-isolated-store --provider sqlite --archive-sha256 DIGEST --execute
+```
+
+File/SQLite exports restore into either File or SQLite. Restore creates a new
+provider identity/revisions while preserving logical history and receipts; it
+never selects the restored directory as live authority. This avoids one
+converter for every pair of storage formats. PostgreSQL's existing archive
+source contract remains unchanged; authenticated service activation, cutover
+and PostgreSQL destination administration are separate work.
+
+Old raw backups can be copied into an **isolated** provider directory, with their
+original identity and canonical filename, then upgraded and exported. Never
+rewrite a schema label or restore an old backup over newer acknowledged writes.
+Binary rollback requires the target runtime to pass `--require-current`; an old
+binary without that gate is not automatically activated. Data rollback and
+provider cutover require their own reviewed, fenced recovery operation.
+
+## Qualification and limits
+
+Tests cover legacy rejection, original historical identity/receipts, checkpoint
+pagination, malformed history, backup damage, interruptions around rename,
+competing migration processes, real SQLite backup/migration, and File/SQLite
+archive interchange. CLI validation uses the managed TS runtime. An authorized
+detached long-history snapshot additionally checks source immutability, exact
+backup bytes and every historical transaction/receipt against the parent.
+
+The CLI/install surfaces change; frontend and Lark business commands continue
+using the unchanged provider contract and need no new settings. This does not
+close default-provider promotion, D1–D3, PostgreSQL production qualification or
+Python business-owner retirement. Native Windows execution still requires its
+platform CI evidence; POSIX validation does not substitute for it.
+
+## 中文要点
+
+旧 File 每次提交都复制完整状态；新格式每 64 条保留完整检查点，其余保存差量。
+事件、原始回执、操作身份、版本号和历史内容不变。复用 SQLite 的 TS 编码规则，
+不删除 append-only ledger 抽象，也不切换 provider。
+
+正常读写只接受新格式。安装／更新调用统一升级入口：先验证、自动备份并核对，
+再迁移和读回；源码开发也可显式调用 `authority-archive upgrade --execute`。
+迁移解析旧数据属于升级工具，不是长期运行的旧格式兼容分支。
+
+同 provider 的格式升级保持身份和版本；跨 provider 则用逻辑归档导出、验证、隔离
+恢复，生成目标 provider 的新身份和版本，保留历史事实及原始回执。迁移数据不等于
+获得执行权，恢复目录不会自动成为线上 authority。
+
+升级失败时可能已有部分 store 完成，必须据实报告并重试，不能覆盖之后产生的写入。
+备份仍是旧格式，恢复时应先在隔离目录升级。只回退二进制并不等于安全回退数据。
+该方案减少重复存储和后续写入耗时，但冷校验仍验证全历史，可能更慢。