feat(service-storage)!: retire the sys_file scope option public and rewrite stored public rows to user (#22443)

.changeset/22443-sys-file-public-scope-rewrite.md

@@ -0,0 +1,55 @@
+---
+'@objectstack/service-storage': minor
+'@objectstack/spec': patch
+---
+
+feat(service-storage)!: the `sys_file` scope option `public` is retired, and rows already stored with it are rewritten to `user` by a one-time operator sweep (#22443)
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: not-required (already-registered storage-scope-public-retired) the retirement of the storage scope public is that step-18 entry; this changeset retires the same value from the stored sys_file vocabulary and rewrites its stored rows through an operator sweep -->
+
+**BREAKING** — an accept-set narrowing on the `sys_file` object's `scope` select, shipped as `minor` under the launch-window convention for accept-set narrowings. It completes the retirement registered as `storage-scope-public-retired`: `StorageScopeSchema` and both upload doors already refuse `public`, and now the stored vocabulary does too.
+
+### What changes
+
+- **`sys_file.scope` no longer lists `public`.** Its options are `user`, `tenant`, `private`, `temp` and `attachments`. The engine refuses a write of `public` to the column as it refuses any undeclared option: `VALIDATION_FAILED`, `invalid_option` on `scope`. No access behaviour changes: the only scope value any code reads is `attachments`, and whether a file can be read before sign-in is decided by `acl: 'public_read'` alone.
+- **Rows already stored with scope `public` are rewritten to `user`**, by a one-time sweep the operator runs (below), never at boot. `user` is the scope the upload doors and the field-reference copy path default to, and no reader tells it apart from `public`. One column moves: the storage key keeps its `public/` prefix, the acl and ownership columns are untouched, and no byte moves in the storage backend.
+
+### FROM → TO
+
+| before | write instead |
+| --- | --- |
+| a `sys_file` row written with `scope: 'public'` | `scope: 'user'`, or no scope; and `acl: 'public_read'` on the row if the file must be readable before sign-in |
+| rows already stored with `scope: 'public'` | nothing by hand: the sweep below rewrites them to `user` |
+
+**The one-line fix: stop writing scope `public` on `sys_file`, and run the sweep once on each deployment that stored it.**
+
+### The operator step: run the sweep once after upgrading
+
+Until the sweep has run on a deployment that stored `public` rows, a record write that names a `public` file another field already owns is refused. The field-reference copy path copies the file into a new row with the source row's scope, the engine refuses `public` there, and the write fails with `ERR_FILE_REFERENCE_COPY`. The data REST doors answer that `500 INTERNAL_ERROR` ("Internal server error"); the server log carries the full sentence, which ends `Scope must be one of: user, tenant, private, temp, attachments`. Reads, downloads and updates that do not write `scope` are unaffected.
+
+The sweep ships in `@objectstack/service-storage` itself: `planSysFilePublicScopeBackfill`, `applySysFilePublicScopeBackfill`, `runSysFilePublicScopeBackfill` and `formatSysFilePublicScopeBackfillReport` are exported from the package, with their report types. It has the shape of the `sys_file` organization backfill: an operator step, never a boot hook. Import it from the package you upgraded to and run it server-side, from a context that holds the engine. It is a dry run first, and by default:
+
+```ts
+import {
+  planSysFilePublicScopeBackfill,
+  applySysFilePublicScopeBackfill,
+  formatSysFilePublicScopeBackfillReport,
+} from '@objectstack/service-storage';
+
+const plan = await planSysFilePublicScopeBackfill(engine);            // counts, writes nothing
+console.log(formatSysFilePublicScopeBackfillReport(plan));
+const applied = await applySysFilePublicScopeBackfill(engine, plan);  // one scope write per row
+console.log(formatSysFilePublicScopeBackfillReport(applied));         // keep it: it is the rollback list
+```
+
+It counts before it writes (`scanned`), writes nothing where the count is zero, and is idempotent: every write moves its row out of `scope = 'public'`, so a second run scans and writes zero. A row whose write fails is reported, never retried, and is picked up by the next run.
+
+### Rollback
+
+The inverse is `user` back to `public` on exactly the ids the applied report lists. This release cannot write it through the engine, which refuses `public` like any undeclared option. So the inverse goes with a code rollback: on the previous release, which still declares the option, write `scope = 'public'` back to those ids, through the engine there or as one raw driver statement against `sys_file` filtered to them.
+
+### The step-18 entry says so too (`@objectstack/spec`)
+
+The D3 entry `storage-scope-public-retired`, which the protocol upgrade guide is built from, said that files already stored with scope `public` are not touched. That no longer holds for the `scope` column: after the sweep they read `user`. Its reason now names the `sys_file` option's retirement and the operator sweep above, and says that the rewrite changes no access: the storage key and the bytes stay as they are, and those files still download exactly as before. Its acceptance criteria add the sweep's end state: a dry run on each deployment scans zero `sys_file` records with scope `public`.

content/docs/permissions/tenant-audit-census.mdx

@@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way.
 
 The same holds twice over for the context. An options argument spelled as a
 literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
-forwarding shim cannot, and **53 of the 240 sites are spelled that way**. A
+forwarding shim cannot, and **53 of the 241 sites are spelled that way**. A
 context resolved from an inline literal or a local `const` can be tested for
 `isSystem`; one arriving from a helper call cannot.
 
@@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page:
 
 | carried figure | where it survives | this census |
 | :--- | :--- | ---: |
-| 175 write call sites | quoted in the merged changeset | **240** |
+| 175 write call sites | quoted in the merged changeset | **241** |
 | 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **32** more whose options argument is unreadable |
-| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **159 of 240** decidable, **81** undecidable |
-| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 128 decidably elevated, 0 decidably not, 104 undecidable |
+| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **160 of 241** decidable, **81** undecidable |
+| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 128 decidably elevated, 0 decidably not, 105 undecidable |
 | 141 and 132, two independent re-derivations | the card that filed this work | — |
 
 **The differences are not reconciled, and deliberately so.** The old census's
@@ -201,17 +201,17 @@ at any commit.
 
 Two structural facts do plausibly widen this reading against any hand or regex
 one, and both are counted in the generated tables below: the 53 sites reached
-through an erased (`any`) receiver, and the 52 that name their object through a
+through an erased (`any`) receiver, and the 53 that name their object through a
 `const` rather than inline. An instrument that read either the way a person does
 would report a smaller number and would not say so.
 
 The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
 figure has no surviving corroboration anywhere in the tree.** This census reads
-128 of 240 (53%) as decidably elevated, with 104 more whose elevation is a
+128 of 241 (53%) as decidably elevated, with 105 more whose elevation is a
 run-time fact — so the claim is neither confirmed nor refuted, and the honest
 answer is that a static reading cannot settle it.
 
-⇒ **Cite `2 / 240`, and say what it is**: the sites whose options argument was
+⇒ **Cite `2 / 241`, and say what it is**: the sites whose options argument was
 READ and holds no tenant context, against a decidably tenancy-enabled object.
 That is the control's provable yield surface. ⛔ Do not cite it as "the sites
 without tenant context" — **32 further sites** have an options argument this
@@ -223,29 +223,29 @@ cannot read, and they are neither in nor out.
 
 | what | count |
 | :--- | ---: |
-| write call sites on the application surface | **240** |
-| …whose object name is statically decidable | 159 |
+| write call sites on the application surface | **241** |
+| …whose object name is statically decidable | 160 |
 | …whose object name is chosen at run time | 81 |
-| …against an object with tenancy ENABLED | 158 |
+| …against an object with tenancy ENABLED | 159 |
 | …against an object that declares tenancy off | 1 |
-| threading a tenant context | 179 |
+| threading a tenant context | 180 |
 | PROVABLY carrying none (options read, no context key) | **8** |
 | …of those, against a decidably tenancy-enabled object | **2** |
 | options argument UNREADABLE — may or may not carry one | 53 |
 | …of those, against a decidably tenancy-enabled object | 32 |
 | threading a decidably ELEVATED (`isSystem`) context | 128 |
 | threading a context that is decidably NOT elevated | 0 |
-| threading a context whose elevation is a run-time fact | 104 |
+| threading a context whose elevation is a run-time fact | 105 |
 
 | how the instrument reached the site | count |
 | :--- | ---: |
-| receiver carried a readable engine type | 187 |
+| receiver carried a readable engine type | 188 |
 | receiver erased, placed by the object NAME | 33 |
 | receiver erased, placed by an `object: string` PARAMETER | 15 |
 | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 5 |
 
 | object name spelled inline | 107 |
-| object name spelled through a `const` | 52 |
+| object name spelled through a `const` | 53 |
 | object name is an `object: string` parameter | 19 |
 | object name is some other run-time expression | 62 |
 
@@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
 their values are not compared. The reasoning, and the measurement behind it,
 are in `scripts/check-tenant-audit-census.mjs`.
 
-Measured on 2026-10-08 at `715ba6f44`.
+Measured on 2026-10-09 at `5551f8d8d`.
 
 | corpus scale (not enforced) | count |
 | :--- | ---: |
-| tracked non-test sources scanned | 623 |
-| engine-shaped types recognised | 71 |
-| declared objects in the registry | 117 |
+| tracked non-test sources scanned | 625 |
+| engine-shaped types recognised | 72 |
+| declared objects in the registry | 116 |
 | same-named calls subtracted as non-engine | 162 |
 
 {/* END GENERATED: tenant-audit-census */}

docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

@@ -33,19 +33,19 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.
 
 | Measure | Value |
 |---|---:|
-| Write call sites | 240 |
-| Object name statically decidable | 159 |
+| Write call sites | 241 |
+| Object name statically decidable | 160 |
 | Object name chosen at run time | 81 |
-| Against a tenancy-enabled object | 158 |
+| Against a tenancy-enabled object | 159 |
 | Against an object declaring tenancy off | 1 |
-| Threading a tenant context | 179 |
+| Threading a tenant context | 180 |
 | Provably carrying none | 8 |
 | …and decidably tenancy-enabled | 2 |
 | Options argument unreadable | 53 |
 | …and decidably tenancy-enabled | 32 |
 | Threading a decidably elevated context | 128 |
 | Threading a decidably non-elevated context | 0 |
-| Threading a context of undecidable elevation | 104 |
+| Threading a context of undecidable elevation | 105 |
 
 ## Subtractions the census could NOT defend — enforced
 
@@ -90,13 +90,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
 their values are not compared. The reasoning, and the measurement behind it,
 are in `scripts/check-tenant-audit-census.mjs`.
 
-Measured on 2026-10-08 at `715ba6f44`.
+Measured on 2026-10-09 at `5551f8d8d`.
 
 | corpus scale (not enforced) | count |
 | :--- | ---: |
-| tracked non-test sources scanned | 623 |
-| engine-shaped types recognised | 71 |
-| declared objects in the registry | 117 |
+| tracked non-test sources scanned | 625 |
+| engine-shaped types recognised | 72 |
+| declared objects in the registry | 116 |
 | same-named calls subtracted as non-engine | 162 |
 
 ## Every site
@@ -251,6 +251,7 @@ Measured on 2026-10-08 at `715ba6f44`.
 | `packages/services/service-storage/src/backfill-file-references.ts` | `update` | `object` | undecidable | elevated | 1 |
 | `packages/services/service-storage/src/backfill-file-references.ts` | `insert` | `sys_file` | enabled | context, elevation undecidable | 1 |
 | `packages/services/service-storage/src/backfill-sys-file-organizations.ts` | `update` | `sys_file` | enabled | context, elevation undecidable | 1 |
+| `packages/services/service-storage/src/backfill-sys-file-public-scope.ts` | `update` | `sys_file` | enabled | context, elevation undecidable | 1 |
 | `packages/services/service-storage/src/file-reference-lifecycle.ts` | `insert` | `sys_file` | enabled | context, elevation undecidable | 1 |
 | `packages/services/service-storage/src/file-reference-lifecycle.ts` | `update` | `sys_file` | enabled | elevated | 2 |
 | `packages/services/service-storage/src/metadata-store.ts` | `delete` | `sys_file` | enabled | options unreadable | 1 |