diff --git a/.changeset/23306-storage-attach-verb.md b/.changeset/23306-storage-attach-verb.md new file mode 100644 index 00000000000..6d0571c762e --- /dev/null +++ b/.changeset/23306-storage-attach-verb.md @@ -0,0 +1,17 @@ +--- +'@objectstack/spec': minor +'@objectstack/service-storage': minor +--- + +A server-side producer can store a file and attach it to a record: `IFileAttachmentService.attachFile`, registered by `StorageServicePlugin` as the kernel service `file-attachments` (`FILE_ATTACHMENT_SERVICE`), writes the bytes, one `sys_file` row and one `sys_attachment` row linking it to the parent, under the checks every other attachment of that parent meets + +Clause-②: yes (widening: `@objectstack/spec/contracts` gains `IFileAttachmentService`, `AttachFileInput`, `AttachedFile` and `FILE_ATTACHMENT_SERVICE`, and `StorageServicePlugin` registers and declares one more service) + +- **The call.** `attachFile({ parentObject, parentId, organizationId, filename, mimeType, data })` answers `{ fileId, attachmentId, size }`. `organizationId` is required: the organization the caller acts in, or `null` for none. It acts as the system, for a producer with no browser upload in front of it (inbound mail first). Reach it with `ctx.getService(FILE_ATTACHMENT_SERVICE)` once the kernel has bootstrapped. +- **What it runs.** Each check is the one the rest of the platform already runs, not a copy: + - The `max_upload_mb` storage setting, through the same verdict the REST upload doors ask. Over it, the call is refused `413 PAYLOAD_TOO_LARGE` before anything is stored. + - The parent's organization. A parent walled into another organization than `organizationId` is refused `404 RECORD_NOT_FOUND`, as a missing parent is. Both rows are written in the parent's organization, or in the caller's for a global parent. + - Every gate on the `sys_attachment` insert, among them the `enable.files` opt-in: a parent object without it is refused `403 FILES_DISABLED`. + - Before the kernel has bootstrapped, or with no data engine, the call is refused `503 SERVICE_UNAVAILABLE`. A malformed input is refused `400 VALIDATION_ERROR`. +- **No refusal leaves a committed `sys_file`.** The row is written `pending` before the bytes, and turns `committed` only in the same transaction as the `sys_attachment` insert. A refused call removes the bytes and the row before it throws. If the process dies mid-call, the pending-upload retention reclaims both. +- `@objectstack/service-storage`: the upload doors' size refusal moved from the route module into `storage-limits.ts`, beside the limit it enforces, so the doors and the verb read one status, one code and one sentence. The doors answer exactly as before. diff --git a/content/docs/permissions/tenant-audit-census.mdx b/content/docs/permissions/tenant-audit-census.mdx index 22e67d02fac..49be48c29e5 100644 --- a/content/docs/permissions/tenant-audit-census.mdx +++ b/content/docs/permissions/tenant-audit-census.mdx @@ -83,7 +83,7 @@ what moved this page's population from 225 to 227; nothing about the two sites changed, only whether this instrument could see them. **The expensive failure direction is a keyword.** Sites whose receiver the author -typed `any` have no type to read, and there are 40 of them — just under a fifth +typed `any` have no type to read, and there are 41 of them — just under a fifth of the population, concentrated in exactly the seed and bootstrap paths this control exists for. Scoring an unreadable receiver as "not an engine" would have dropped every one of them silently, with a clean exit and a smaller number that @@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way. The same holds twice over for the context. An options argument spelled as a literal can be read; one spelled `options`, `{ ...opts }`, or handed through a -forwarding shim cannot, and **53 of the 226 sites are spelled that way**. A +forwarding shim cannot, and **53 of the 228 sites are spelled that way**. A context resolved from an inline literal or a local `const` can be tested for `isSystem`; one arriving from a helper call cannot. @@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page: | carried figure | where it survives | this census | | :--- | :--- | ---: | -| 175 write call sites | quoted in the merged changeset | **226** | +| 175 write call sites | quoted in the merged changeset | **228** | | 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **32** more whose options argument is unreadable | -| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **153 of 226** decidable, **73** undecidable | -| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 121 decidably elevated, 0 decidably not, 97 undecidable | +| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **155 of 228** decidable, **73** undecidable | +| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 121 decidably elevated, 0 decidably not, 99 undecidable | | 141 and 132, two independent re-derivations | the card that filed this work | — | **The differences are not reconciled, and deliberately so.** The old census's @@ -200,18 +200,18 @@ be stated is what this instrument counts, which is written above and re-runnable at any commit. Two structural facts do plausibly widen this reading against any hand or regex -one, and both are counted in the generated tables below: the 40 sites reached +one, and both are counted in the generated tables below: the 41 sites reached through an erased (`any`) receiver, and the 50 that name their object through a `const` rather than inline. An instrument that read either the way a person does would report a smaller number and would not say so. The fourth row is the one worth flagging to anyone citing it. **The 135 / 77% figure has no surviving corroboration anywhere in the tree.** This census reads -121 of 226 (54%) as decidably elevated, with 97 more whose elevation is a +121 of 228 (53%) as decidably elevated, with 99 more whose elevation is a run-time fact — so the claim is neither confirmed nor refuted, and the honest answer is that a static reading cannot settle it. -⇒ **Cite `2 / 226`, and say what it is**: the sites whose options argument was +⇒ **Cite `2 / 228`, and say what it is**: the sites whose options argument was READ and holds no tenant context, against a decidably tenancy-enabled object. That is the control's provable yield surface. ⛔ Do not cite it as "the sites without tenant context" — **32 further sites** have an options argument this @@ -223,28 +223,28 @@ cannot read, and they are neither in nor out. | what | count | | :--- | ---: | -| write call sites on the application surface | **226** | -| …whose object name is statically decidable | 153 | +| write call sites on the application surface | **228** | +| …whose object name is statically decidable | 155 | | …whose object name is chosen at run time | 73 | -| …against an object with tenancy ENABLED | 151 | +| …against an object with tenancy ENABLED | 153 | | …against an object that declares tenancy off | 1 | -| threading a tenant context | 165 | +| threading a tenant context | 167 | | PROVABLY carrying none (options read, no context key) | **8** | | …of those, against a decidably tenancy-enabled object | **2** | | options argument UNREADABLE — may or may not carry one | 53 | | …of those, against a decidably tenancy-enabled object | 32 | | threading a decidably ELEVATED (`isSystem`) context | 121 | | threading a context that is decidably NOT elevated | 0 | -| threading a context whose elevation is a run-time fact | 97 | +| threading a context whose elevation is a run-time fact | 99 | | how the instrument reached the site | count | | :--- | ---: | -| receiver carried a readable engine type | 186 | -| receiver erased, placed by the object NAME | 27 | +| receiver carried a readable engine type | 187 | +| receiver erased, placed by the object NAME | 28 | | receiver erased, placed by an `object: string` PARAMETER | 7 | | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 6 | -| object name spelled inline | 103 | +| object name spelled inline | 105 | | object name spelled through a `const` | 50 | | object name is an `object: string` parameter | 9 | | object name is some other run-time expression | 64 | @@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-13 at `c33c6dc67`. +Measured on 2026-10-14 at `5d4205c6b`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 650 | -| engine-shaped types recognised | 73 | +| tracked non-test sources scanned | 652 | +| engine-shaped types recognised | 75 | | declared objects in the registry | 116 | -| same-named calls subtracted as non-engine | 162 | +| same-named calls subtracted as non-engine | 163 | {/* END GENERATED: tenant-audit-census */} diff --git a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md index a01258d23a8..c7be6d41a4d 100644 --- a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md +++ b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md @@ -33,19 +33,19 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution. | Measure | Value | |---|---:| -| Write call sites | 226 | -| Object name statically decidable | 153 | +| Write call sites | 228 | +| Object name statically decidable | 155 | | Object name chosen at run time | 73 | -| Against a tenancy-enabled object | 151 | +| Against a tenancy-enabled object | 153 | | Against an object declaring tenancy off | 1 | -| Threading a tenant context | 165 | +| Threading a tenant context | 167 | | Provably carrying none | 8 | | …and decidably tenancy-enabled | 2 | | Options argument unreadable | 53 | | …and decidably tenancy-enabled | 32 | | Threading a decidably elevated context | 121 | | Threading a decidably non-elevated context | 0 | -| Threading a context of undecidable elevation | 97 | +| Threading a context of undecidable elevation | 99 | ## Subtractions the census could NOT defend — enforced @@ -90,14 +90,14 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-13 at `c33c6dc67`. +Measured on 2026-10-14 at `5d4205c6b`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 650 | -| engine-shaped types recognised | 73 | +| tracked non-test sources scanned | 652 | +| engine-shaped types recognised | 75 | | declared objects in the registry | 116 | -| same-named calls subtracted as non-engine | 162 | +| same-named calls subtracted as non-engine | 163 | ## Every site @@ -238,6 +238,8 @@ Measured on 2026-10-13 at `c33c6dc67`. | `packages/services/service-storage/src/backfill-file-references.ts` | `insert` | `sys_file` | enabled | context, elevation undecidable | 1 | | `packages/services/service-storage/src/backfill-sys-file-organizations.ts` | `update` | `sys_file` | enabled | context, elevation undecidable | 1 | | `packages/services/service-storage/src/backfill-sys-file-public-scope.ts` | `update` | `sys_file` | enabled | context, elevation undecidable | 1 | +| `packages/services/service-storage/src/file-attachment-service.ts` | `delete` | `sys_attachment` | enabled | context, elevation undecidable | 1 | +| `packages/services/service-storage/src/file-attachment-service.ts` | `insert` | `sys_attachment` | enabled | context, elevation undecidable | 1 | | `packages/services/service-storage/src/file-reference-lifecycle.ts` | `insert` | `sys_file` | enabled | context, elevation undecidable | 1 | | `packages/services/service-storage/src/file-reference-lifecycle.ts` | `update` | `sys_file` | enabled | elevated | 2 | | `packages/services/service-storage/src/metadata-store.ts` | `delete` | `sys_file` | enabled | options unreadable | 1 | diff --git a/packages/services/service-storage/package.json b/packages/services/service-storage/package.json index 5659769384e..da71c51f4ca 100644 --- a/packages/services/service-storage/package.json +++ b/packages/services/service-storage/package.json @@ -46,6 +46,7 @@ "devDependencies": { "@objectstack/objectql": "workspace:*", "@objectstack/driver-sql": "workspace:*", + "@objectstack/plugin-audit": "workspace:*", "@types/node": "^26.6.3", "tsx": "^4.23.15", "typescript": "^6.0.3", diff --git a/packages/services/service-storage/src/file-attachment-service.test.ts b/packages/services/service-storage/src/file-attachment-service.test.ts new file mode 100644 index 00000000000..ab68704b2c4 --- /dev/null +++ b/packages/services/service-storage/src/file-attachment-service.test.ts @@ -0,0 +1,420 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#23306] The server-side attach verb — `FILE_ATTACHMENT_SERVICE`, registered + * by `StorageServicePlugin` — stores bytes and attaches them to a record as one + * `sys_file` and one `sys_attachment`, and no refusal leaves a committed + * `sys_file` (the #22547 class). + * + * What these pin, through the plugin's OWN wiring (a real `ObjectQL` over + * sqlite `:memory:`, the package's ruled test backend; the plugin's own + * `kernel:ready` gates; `plugin-audit`'s REAL `enable.files` gate on the + * `sys_attachment` insert; the plugin's local adapter in a temp directory; and + * a settings namespace carrying `max_upload_mb`), reached by its registry name: + * + * - an admitted call writes exactly one committed `sys_file` (attachments + * scope, the bytes behind its key) and one `sys_attachment` linking it to + * the parent, both in the parent's organization; + * - the three refusals the card names — over the size limit, `enable.files` + * off on the parent object, a parent outside the caller's organization — + * each answer their ADR-0112 envelope and leave NO `sys_file` row at all, + * committed or pending, no `sys_attachment` row and no stored bytes; each is + * paired with the same call minus the refused condition, which succeeds; + * - a failure after the attachment insert (the commit refused) rolls the + * attachment back with it, and on an engine with no transaction the call + * removes the row it inserted itself; + * - the verb serves only once the kernel has bootstrapped, and refuses a + * malformed input before anything is read or written. + * + * The ablations that turn each pin red are one-off proofs, recorded in the PR. + */ + +import { describe, it, expect, afterEach, vi } from 'vitest'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { installAuditWriters } from '@objectstack/plugin-audit'; +import { SysAttachment } from '@objectstack/platform-objects/audit'; +import { FILE_ATTACHMENT_SERVICE } from '@objectstack/spec/contracts'; +import type { AttachFileInput, IFileAttachmentService, IStorageService } from '@objectstack/spec/contracts'; +import { SystemFile } from './objects/system-file.object.js'; +import { StorageMetadataStoreError } from './metadata-store.js'; +import { StorageServicePlugin } from './storage-service-plugin.js'; +import type { SwappableStorageService } from './swappable-storage-service.js'; +import { FileAttachmentService, type FileAttachmentEngine } from './file-attachment-service.js'; + +const SYS = { context: { isSystem: true } } as const; +/** `max_upload_mb` as saved: 0.001 MB ⇒ floor(0.001 × 1 MiB) = 1048 bytes. */ +const MAX_UPLOAD_MB = 0.001; +const MAX_UPLOAD_BYTES = 1048; + +function quietLogger() { + const logger: any = { info: vi.fn(), warn: vi.fn(), debug: vi.fn(), error: vi.fn(), trace: vi.fn(), fatal: vi.fn() }; + logger.child = () => logger; + return logger; +} + +/** A plugin context with a service map, and the two boot phases the plugin hooks, run on demand. */ +function makeCtx() { + const services = new Map(); + const phases = new Map unknown>>(); + const ctx: any = { + logger: quietLogger(), + registerService: (name: string, svc: unknown) => { + services.set(name, svc); + }, + getService: (name: string) => { + if (!services.has(name)) throw new Error(`service '${name}' not registered`); + return services.get(name); + }, + hook: (event: string, fn: () => unknown) => { + phases.set(event, [...(phases.get(event) ?? []), fn]); + }, + }; + const run = async (event: string) => { + for (const fn of phases.get(event) ?? []) await fn(); + }; + return { ctx, run }; +} + +/** The `storage` settings namespace, carrying only the saved upload size limit. */ +function storageSettings() { + return { + createClient: () => ({}), + getNamespace: async () => ({ values: { max_upload_mb: { value: MAX_UPLOAD_MB, source: 'global' } } }), + subscribe: () => undefined, + registerAction: () => undefined, + }; +} + +const engines: ObjectQL[] = []; +afterEach(async () => { + for (const ql of engines.splice(0)) { + try { + await ql.destroy(); + } catch { + /* already torn down */ + } + } +}); + +/** + * A real engine, the platform's objects, and two parent objects: `att_case` + * opts into attachments (`enable.files: true`), `att_memo` does not. Parents + * in `org_a`, in `org_b`, and one global (no organization). + */ +async function engineWithParents(): Promise { + const ql = new ObjectQL({ logger: quietLogger() } as any); + ql.registerDriver( + new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }) as any, + true, + ); + await ql.init(); + engines.push(ql); + ql.registry.registerObject(SystemFile as any, 'test-23306'); + ql.registry.registerObject(SysAttachment as any, 'test-23306'); + ql.registry.registerObject( + { name: 'att_case', enable: { files: true }, fields: { name: { type: 'text' } } } as any, + 'test-23306', + ); + ql.registry.registerObject({ name: 'att_memo', fields: { name: { type: 'text' } } } as any, 'test-23306'); + await ql.syncSchemas(); + // plugin-audit's REAL `enable.files` gate (and the rest of its writers), as + // AuditPlugin installs them on the composed engine. + installAuditWriters(ql as any); + await ql.insert('att_case', { id: 'case_a', name: 'in org_a', organization_id: 'org_a' }, SYS); + await ql.insert('att_case', { id: 'case_b', name: 'in org_b', organization_id: 'org_b' }, SYS); + await ql.insert('att_case', { id: 'case_global', name: 'in no organization' }, SYS); + await ql.insert('att_memo', { id: 'memo_a', name: 'files off', organization_id: 'org_a' }, SYS); + return ql; +} + +/** Boot the plugin over `ql` through its real phases; `bootstrapped: false` stops before the verb is armed. */ +async function boot(opts: { bootstrapped?: boolean; engine?: boolean } = {}) { + const ql = await engineWithParents(); + const rootDir = await fs.mkdtemp(join(tmpdir(), 'oss-23306-')); + const plugin = new StorageServicePlugin({ adapter: 'local', local: { rootDir }, registerRoutes: false }); + const { ctx, run } = makeCtx(); + if (opts.engine !== false) ctx.registerService('objectql', ql); + ctx.registerService('settings', storageSettings()); + await plugin.init(ctx); + await plugin.start(ctx); + await run('kernel:ready'); + if (opts.bootstrapped !== false) await run('kernel:bootstrapped'); + const attachments = ctx.getService(FILE_ATTACHMENT_SERVICE) as IFileAttachmentService; + const storage = ctx.getService('storage') as SwappableStorageService; + return { ql, run, attachments, storage, rootDir }; +} + +const input = (over: Partial = {}): AttachFileInput => ({ + parentObject: 'att_case', + parentId: 'case_a', + organizationId: 'org_a', + filename: 'forwarded.eml', + mimeType: 'message/rfc822', + data: Buffer.from('From: someone@example.com\r\n\r\nhello'), + ...over, +}); + +const rows = (ql: ObjectQL, object: string) => ql.find(object, { ...SYS }) as Promise>>; + +/** Every key under the attachments prefix — the bytes this verb could have left behind. */ +async function storedKeys(storage: IStorageService): Promise { + const page = await storage.list!('attachments/'); + return page.items.map((i) => i.key); +} + +async function refusalOf(call: Promise): Promise<{ code?: string; status?: number }> { + try { + await call; + } catch (err) { + return err as { code?: string; status?: number }; + } + throw new Error('expected the call to be refused'); +} + +/** The #22547 assertion: nothing of the refused call is left — no sys_file (committed or pending), no link, no bytes. */ +async function expectNothingLeft(ql: ObjectQL, storage: IStorageService) { + expect(await rows(ql, 'sys_file')).toEqual([]); + expect(await rows(ql, 'sys_attachment')).toEqual([]); + expect(await storedKeys(storage)).toEqual([]); +} + +describe('FILE_ATTACHMENT_SERVICE: the server-side attach verb (#23306)', () => { + it('is registered under its contract slot by StorageServicePlugin, and declared in providesServices', async () => { + expect(FILE_ATTACHMENT_SERVICE).toBe('file-attachments'); + expect(new StorageServicePlugin().providesServices).toContain(FILE_ATTACHMENT_SERVICE); + const { attachments } = await boot(); + expect(attachments).toBeInstanceOf(FileAttachmentService); + }); + + it('stores the bytes and writes one committed sys_file and one sys_attachment, in the parent organization', async () => { + const { ql, attachments, storage } = await boot(); + const data = Buffer.from('a forwarded message'); + const out = await attachments.attachFile(input({ data })); + + const files = await rows(ql, 'sys_file'); + expect(files).toHaveLength(1); + expect(files[0]).toMatchObject({ + id: out.fileId, + status: 'committed', + scope: 'attachments', + acl: 'private', + name: 'forwarded.eml', + mime_type: 'message/rfc822', + size: data.byteLength, + organization_id: 'org_a', + }); + expect(files[0].key).toBe(`attachments/${out.fileId}.eml`); + expect((await storage.download(files[0].key)).equals(data)).toBe(true); + + const links = await rows(ql, 'sys_attachment'); + expect(links).toHaveLength(1); + expect(links[0]).toMatchObject({ + id: out.attachmentId, + parent_object: 'att_case', + parent_id: 'case_a', + file_id: out.fileId, + file_name: 'forwarded.eml', + mime_type: 'message/rfc822', + size: data.byteLength, + organization_id: 'org_a', + }); + expect(out.size).toBe(data.byteLength); + }); + + describe('refusal: over the upload size limit (max_upload_mb)', () => { + it('refuses 413 PAYLOAD_TOO_LARGE before anything is stored or written', async () => { + const { ql, attachments, storage } = await boot(); + const err = await refusalOf(attachments.attachFile(input({ data: Buffer.alloc(MAX_UPLOAD_BYTES + 1) }))); + expect(err.code).toBe('PAYLOAD_TOO_LARGE'); + expect(err.status).toBe(413); + await expectNothingLeft(ql, storage); + }); + + it('control: a file exactly at the limit is attached', async () => { + const { ql, attachments } = await boot(); + const out = await attachments.attachFile(input({ data: Buffer.alloc(MAX_UPLOAD_BYTES) })); + expect(out.size).toBe(MAX_UPLOAD_BYTES); + expect((await rows(ql, 'sys_file')).map((f) => f.status)).toEqual(['committed']); + }); + }); + + describe('refusal: files disabled on the parent object', () => { + it('refuses 403 FILES_DISABLED (the enable.files gate on the insert) and leaves no sys_file, link or bytes', async () => { + const { ql, attachments, storage } = await boot(); + const err = await refusalOf(attachments.attachFile(input({ parentObject: 'att_memo', parentId: 'memo_a' }))); + expect(err.code).toBe('FILES_DISABLED'); + expect(err.status).toBe(403); + await expectNothingLeft(ql, storage); + }); + + it('control: the same call on an object that opts into files is attached', async () => { + const { ql, attachments } = await boot(); + await attachments.attachFile(input()); + expect(await rows(ql, 'sys_attachment')).toHaveLength(1); + }); + }); + + describe('refusal: a parent outside the organization', () => { + it('refuses 404 RECORD_NOT_FOUND for a parent walled into another organization, before anything is written', async () => { + const { ql, attachments, storage } = await boot(); + const err = await refusalOf(attachments.attachFile(input({ parentId: 'case_b', organizationId: 'org_a' }))); + expect(err.code).toBe('RECORD_NOT_FOUND'); + expect(err.status).toBe(404); + await expectNothingLeft(ql, storage); + }); + + it('control: the same parent, acted on from its own organization, is attached there', async () => { + const { ql, attachments } = await boot(); + await attachments.attachFile(input({ parentId: 'case_b', organizationId: 'org_b' })); + expect((await rows(ql, 'sys_file')).map((f) => f.organization_id)).toEqual(['org_b']); + expect((await rows(ql, 'sys_attachment')).map((a) => a.organization_id)).toEqual(['org_b']); + }); + + it('answers a parent that does not exist the same way', async () => { + const { ql, attachments, storage } = await boot(); + const err = await refusalOf(attachments.attachFile(input({ parentId: 'case_missing' }))); + expect(err.code).toBe('RECORD_NOT_FOUND'); + expect(err.status).toBe(404); + await expectNothingLeft(ql, storage); + }); + }); + + describe('the organization both rows are written in', () => { + it('is the parent organization when the caller acts in none', async () => { + const { ql, attachments } = await boot(); + await attachments.attachFile(input({ parentId: 'case_b', organizationId: null })); + expect((await rows(ql, 'sys_file')).map((f) => f.organization_id)).toEqual(['org_b']); + expect((await rows(ql, 'sys_attachment')).map((a) => a.organization_id)).toEqual(['org_b']); + }); + + it("is the caller's for a global parent", async () => { + const { ql, attachments } = await boot(); + await attachments.attachFile(input({ parentId: 'case_global', organizationId: 'org_a' })); + expect((await rows(ql, 'sys_file')).map((f) => f.organization_id)).toEqual(['org_a']); + expect((await rows(ql, 'sys_attachment')).map((a) => a.organization_id)).toEqual(['org_a']); + }); + }); + + describe('a failure after the attachment insert', () => { + /** Refuse the verb's commit (`status: 'committed'`) on sys_file, after the attachment was inserted. */ + const refuseCommit = (ql: ObjectQL) => + ql.registerHook( + 'beforeUpdate', + async (hook: any) => { + if (hook?.input?.data?.status === 'committed') { + const err: any = new Error('commit refused by a test gate'); + err.code = 'VALIDATION_ERROR'; + err.status = 400; + throw err; + } + }, + { object: 'sys_file', packageId: 'test-23306' }, + ); + + /** + * The commit is a `sys_file` write, so it fails the way every write of the + * store does: as the store's own error, carrying the engine's refusal. + */ + const expectCommitRefusal = (err: unknown) => { + expect(err).toBeInstanceOf(StorageMetadataStoreError); + expect((err as StorageMetadataStoreError).operation).toBe('update'); + expect(((err as StorageMetadataStoreError).cause as { code?: string }).code).toBe('VALIDATION_ERROR'); + }; + + it('rolls the attachment back with the refused commit, and leaves nothing', async () => { + const { ql, attachments, storage } = await boot(); + refuseCommit(ql); + expectCommitRefusal(await refusalOf(attachments.attachFile(input()))); + await expectNothingLeft(ql, storage); + }); + + it('on an engine with no transaction, removes the attachment row it inserted', async () => { + const { ql, storage } = await boot(); + refuseCommit(ql); + const noTransaction: FileAttachmentEngine = { + find: ql.find.bind(ql), + findOne: ql.findOne.bind(ql), + insert: ql.insert.bind(ql), + update: ql.update.bind(ql), + delete: ql.delete.bind(ql), + count: ql.count.bind(ql), + aggregate: ql.aggregate.bind(ql), + getSchema: ql.getSchema.bind(ql), + } as FileAttachmentEngine; + const verb = new FileAttachmentService({ + getEngine: () => noTransaction, + getStorage: () => storage, + getLimitsSnapshot: () => storage.getLimitsSnapshot(), + logger: quietLogger(), + }); + expectCommitRefusal(await refusalOf(verb.attachFile(input()))); + await expectNothingLeft(ql, storage); + }); + }); + + describe('a failure storing the bytes', () => { + it('propagates the backend error unchanged and removes the pending sys_file row', async () => { + const { ql, storage } = await boot(); + const backendDown = new Error('the storage backend refused the write'); + const failingUpload: IStorageService = { + upload: async () => { + throw backendDown; + }, + download: (key) => storage.download(key), + delete: (key) => storage.delete(key), + exists: (key) => storage.exists(key), + getInfo: (key) => storage.getInfo(key), + }; + const verb = new FileAttachmentService({ + getEngine: () => ql as unknown as FileAttachmentEngine, + getStorage: () => failingUpload, + getLimitsSnapshot: () => storage.getLimitsSnapshot(), + logger: quietLogger(), + }); + expect(await refusalOf(verb.attachFile(input()))).toBe(backendDown); + await expectNothingLeft(ql, storage); + }); + }); + + describe('before it can serve', () => { + it('refuses 503 SERVICE_UNAVAILABLE until kernel:bootstrapped, then serves', async () => { + const { ql, run, attachments, storage } = await boot({ bootstrapped: false }); + const err = await refusalOf(attachments.attachFile(input())); + expect(err.code).toBe('SERVICE_UNAVAILABLE'); + expect(err.status).toBe(503); + await expectNothingLeft(ql, storage); + await run('kernel:bootstrapped'); + await attachments.attachFile(input()); + expect(await rows(ql, 'sys_attachment')).toHaveLength(1); + }); + + it('refuses 503 SERVICE_UNAVAILABLE on a kernel with no data engine', async () => { + const { attachments } = await boot({ engine: false }); + const err = await refusalOf(attachments.attachFile(input())); + expect(err.code).toBe('SERVICE_UNAVAILABLE'); + expect(err.status).toBe(503); + }); + + it('refuses a malformed input 400 VALIDATION_ERROR before anything is read or written', async () => { + const { ql, attachments, storage } = await boot(); + const malformed: Array>> = [ + { parentObject: '' }, + { parentId: 42 }, + { organizationId: undefined }, + { organizationId: '' }, + { filename: '' }, + { mimeType: undefined }, + { data: 'not a buffer' }, + ]; + for (const over of malformed) { + const err = await refusalOf(attachments.attachFile({ ...input(), ...over } as AttachFileInput)); + expect({ over, code: err.code, status: err.status }).toEqual({ over, code: 'VALIDATION_ERROR', status: 400 }); + } + await expectNothingLeft(ql, storage); + }); + }); +}); diff --git a/packages/services/service-storage/src/file-attachment-service.ts b/packages/services/service-storage/src/file-attachment-service.ts new file mode 100644 index 00000000000..2024b60cb81 --- /dev/null +++ b/packages/services/service-storage/src/file-attachment-service.ts @@ -0,0 +1,334 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { randomUUID } from 'node:crypto'; +import type { RegisteredErrorCode, StandardErrorCode } from '@objectstack/spec/api'; +import type { + AttachFileInput, + AttachedFile, + IDataEngine, + IFileAttachmentService, + IObjectQLEngine, + IStorageService, +} from '@objectstack/spec/contracts'; +import { StorageMetadataStore, type StorageWriteContext } from './metadata-store.js'; +// The package's one answer to "which organization is this record walled +// into?", read off the registered schema — never a hard-coded column. +import { createWallOrganizationResolver } from './backfill-sys-file-organizations.js'; +// The upload doors' own size verdict and refusal, asked here rather than +// restated. +import { + resolveStorageLimits, + uploadSizeRefusal, + UPLOAD_TOO_LARGE_CODE, + UPLOAD_TOO_LARGE_STATUS, + type StorageLimitsSnapshot, +} from './storage-limits.js'; +// The upload doors' own storage-key layout. +import { buildKey } from './storage-routes.js'; + +/** + * [#23306] The server-side attach verb — `IFileAttachmentService`, registered + * by `StorageServicePlugin` under `FILE_ATTACHMENT_SERVICE` + * (`@objectstack/spec/contracts`): store bytes, and attach them to a record as + * one `sys_file` row and one `sys_attachment` row. + * + * ## One writer — what it runs, and where each check comes from + * + * A server-side producer (inbound mail first, then imports and generated + * output) has no browser upload in front of it, so this verb acts as the + * SYSTEM. It restates no check; each one is the code every other attachment of + * the parent already meets: + * + * - **Size.** `uploadSizeRefusal` (`storage-limits.ts`), the verdict the REST + * upload doors ask before they store a byte, against the same resolved + * `max_upload_mb` (`resolveStorageLimits` over the `storage` service's + * Limits snapshot). Refused `413 PAYLOAD_TOO_LARGE`, with the doors' own + * sentence, before anything is stored. + * - **The `sys_file` row.** `StorageMetadataStore` — the upload doors' writer, + * carrying the organization through the same `StorageWriteContext` channel, + * so the driver stamps it exactly as it stamps an upload. + * - **The `sys_attachment` row.** A data-engine insert, so every gate the + * engine runs on a `sys_attachment` insert judges it — the gate's decision, + * never a copy of it here. Measured under this verb's system context: + * `plugin-audit`'s `enable.files` gate runs and refuses `403 + * FILES_DISABLED` for a parent object without the opt-in; the attachment + * lifecycle hooks run; the parent-EDIT gate + * (`installAttachmentAccessHooks`) returns at its system-context line by its + * own declared design — there is no caller whose edit access to the parent + * could be asked. + * - **The organization.** The parent is read by id as the system, and its + * organization comes from `createWallOrganizationResolver` — the column the + * parent is WALLED by. A parent walled into another organization than the + * caller's is refused `404 RECORD_NOT_FOUND`, exactly like a parent that + * does not exist, so the answer says nothing about the other organization. + * Both rows are then written in the parent's organization (or the caller's, + * for a global parent), so the file sits behind the same wall as the record + * it is attached to. + * + * ## No refusal leaves a committed `sys_file` + * + * The write order is the presigned upload door's: the `sys_file` row first, + * `pending`, then the bytes — so the row is always the pointer the platform's + * pending-upload retention (`sys_file`'s `retention` on `status: 'pending'`, + * and the reap guard that deletes a pending row's bytes) can reclaim both by, + * even if the process dies mid-call. The row turns `committed` only in the SAME + * engine transaction as the `sys_attachment` insert, so a refused insert rolls + * the commit back with it; on an engine that offers no transaction the commit + * simply never runs after a refused insert. Every refusal then removes the + * bytes and the pending row before it rethrows, unchanged. + */ + +/** + * The data-engine surface this verb uses: the reads and writes every engine + * has, plus — where the engine offers one — its transaction and its schema + * read (the wall resolver's input). + */ +export type FileAttachmentEngine = IDataEngine & + Partial> & { + getSchema?(objectName: string): unknown; + }; + +/** What {@link FileAttachmentService} is composed over — each read at call time. */ +export interface FileAttachmentServiceDeps { + /** + * The data engine, once every plugin has installed its `sys_attachment` + * gates on it (armed at `kernel:bootstrapped`). `null` before that, or when + * no engine is wired: the verb then refuses rather than write rows a gate has + * not judged. + */ + getEngine(): FileAttachmentEngine | null; + /** The `storage` service the bytes go to. */ + getStorage(): IStorageService | null; + /** The `storage` namespace's Limits group as last read — the upload doors' own source. */ + getLimitsSnapshot(): StorageLimitsSnapshot | undefined; + logger: { warn(message: string): void }; +} + +/** The upload scope of every file this verb writes: the attachments surface. */ +const ATTACHMENTS_SCOPE = 'attachments'; + +const NOT_READY_MESSAGE = + 'The file-attachments service cannot store a file: no data engine is wired into this kernel, or the kernel ' + + 'has not finished booting. It serves once kernel:bootstrapped has fired, in a kernel that composes the data engine.'; + +/** An ADR-0112-enveloped refusal: `code` and `status` on the thrown error. */ +function refuse(code: StandardErrorCode | RegisteredErrorCode, status: number, message: string): never { + const err = new Error(message) as Error & { code: string; status: number }; + err.code = code; + err.status = status; + throw err; +} + +function nonEmptyString(value: unknown): value is string { + return typeof value === 'string' && value.length > 0; +} + +/** + * The input, checked field by field: a JavaScript caller can hand anything, + * and a malformed field is the caller's to fix (`400 VALIDATION_ERROR`) before + * anything is read or written. Length limits are not judged here — the engine + * judges the rows against their declared fields. + */ +function validateAttachFileInput(input: AttachFileInput): AttachFileInput { + const fail = (message: string): never => refuse('VALIDATION_ERROR', 400, `attachFile: ${message}`); + if (!input || typeof input !== 'object') fail('the input must be an object'); + for (const field of ['parentObject', 'parentId', 'filename', 'mimeType'] as const) { + if (!nonEmptyString(input[field])) fail(`'${field}' must be a non-empty string`); + } + if (input.organizationId !== null && !nonEmptyString(input.organizationId)) { + fail( + "'organizationId' must be the organization the call acts in, or null for none — it is required, " + + 'so acting in no organization is stated rather than left out', + ); + } + if (!Buffer.isBuffer(input.data)) fail("'data' must be a Buffer holding the file's bytes"); + return input; +} + +export class FileAttachmentService implements IFileAttachmentService { + constructor(private readonly deps: FileAttachmentServiceDeps) {} + + async attachFile(input: AttachFileInput): Promise { + const request = validateAttachFileInput(input); + const engine = this.deps.getEngine(); + const storage = this.deps.getStorage(); + if (!engine || !storage) refuse('SERVICE_UNAVAILABLE', 503, NOT_READY_MESSAGE); + + // ① Size — before a byte is stored or a row is written. + const size = request.data.byteLength; + const tooLarge = uploadSizeRefusal( + size, + resolveStorageLimits(this.deps.getLimitsSnapshot()).maxUploadBytes, + (n) => `The file is ${n} bytes`, + ); + if (tooLarge !== null) refuse(UPLOAD_TOO_LARGE_CODE, UPLOAD_TOO_LARGE_STATUS, tooLarge); + + // ② The parent, and the organization both rows are written in. + const organizationId = await this.organizationOfParent(engine, request); + const writeContext: StorageWriteContext = { organizationId }; + const store = new StorageMetadataStore(engine); + + // ③ The `sys_file` row, `pending` — the pointer the bytes are reclaimed by. + const fileId = randomUUID(); + const key = buildKey(ATTACHMENTS_SCOPE, fileId, request.filename); + await store.createFile( + { + id: fileId, + key, + name: request.filename, + mime_type: request.mimeType, + size, + scope: ATTACHMENTS_SCOPE, + acl: 'private', + status: 'pending', + }, + writeContext, + ); + + // ④ The bytes. + try { + await storage.upload(key, request.data, { contentType: request.mimeType }); + } catch (err) { + await this.discard(engine, store, storage, { fileId, key, writeContext }); + throw err; + } + + // ⑤ The attachment and the commit — one unit. + let attachmentId: string | undefined; + let ownedUnit = false; + try { + await this.inOneUnit(engine, organizationId, async (context, owned) => { + ownedUnit = owned; + const row = (await engine.insert( + 'sys_attachment', + { + parent_object: request.parentObject, + parent_id: request.parentId, + file_id: fileId, + file_name: request.filename, + mime_type: request.mimeType, + size, + }, + { context }, + )) as { id?: unknown } | null | undefined; + if (row?.id === undefined || row.id === null || row.id === '') { + throw new Error('attachFile: the sys_attachment insert answered no row id, so the attachment cannot be named'); + } + attachmentId = String(row.id); + const committed = await store.updateFile(fileId, { status: 'committed' }, writeContext); + if (!committed) { + throw new Error(`attachFile: the sys_file row ${fileId} was gone before it could be committed`); + } + }); + } catch (err) { + // An owned transaction rolled the attachment back with the commit. A + // unit this call does not own — no transaction, or a caller's it joined — + // keeps an inserted row, so it is removed here. + await this.discard(engine, store, storage, { + fileId, + key, + writeContext, + attachmentId: ownedUnit ? undefined : attachmentId, + }); + throw err; + } + return { fileId, attachmentId: attachmentId!, size }; + } + + /** + * Read the parent as the system and answer the organization both rows are + * written in: the parent's own when it is walled into one, otherwise the + * caller's (`null` for none). A parent that does not exist, or that is walled + * into another organization than the caller's, is refused with one answer. + */ + private async organizationOfParent(engine: FileAttachmentEngine, request: AttachFileInput): Promise { + const parent = await engine.findOne( + request.parentObject, + { where: { id: request.parentId } }, + { context: { isSystem: true } }, + ); + const parentOrganization = parent + ? createWallOrganizationResolver(engine).organizationOf(request.parentObject, parent) + : null; + const outside = + request.organizationId !== null && parentOrganization !== null && parentOrganization !== request.organizationId; + if (!parent || outside) { + refuse( + 'RECORD_NOT_FOUND', + 404, + request.organizationId === null + ? `Cannot attach a file to ${request.parentObject}/${request.parentId}: the record does not exist.` + : `Cannot attach a file to ${request.parentObject}/${request.parentId}: the record does not exist in ` + + 'the organization this call acts in.', + ); + } + return parentOrganization ?? request.organizationId; + } + + /** + * Run `work` in one engine transaction when the engine offers one, telling it + * whether this call owns that transaction (`false` when it joined a caller's, + * and on an engine with none). + */ + private async inOneUnit( + engine: FileAttachmentEngine, + organizationId: string | null, + work: (context: Record, owned: boolean) => Promise, + ): Promise { + const base: Record = + organizationId === null ? { isSystem: true } : { isSystem: true, tenantId: organizationId }; + if (typeof engine.transaction !== 'function') { + await work(base, false); + return; + } + await engine.transaction((trxCtx, info) => work(trxCtx as Record, info?.owned === true), base); + } + + /** + * Undo a refused call: the attachment row it still holds (if any), then the + * bytes, then the pending row. Best-effort, and the caller's refusal is what + * is thrown either way. A failure only delays the reclaim: while the pending + * row stands, the pending-upload retention removes it and its bytes — so a + * failed byte delete keeps the row on purpose. + */ + private async discard( + engine: FileAttachmentEngine, + store: StorageMetadataStore, + storage: IStorageService, + call: { fileId: string; key: string; writeContext: StorageWriteContext; attachmentId?: string }, + ): Promise { + const { logger } = this.deps; + if (call.attachmentId !== undefined) { + try { + const tenant = call.writeContext.organizationId; + await engine.delete('sys_attachment', { + where: { id: call.attachmentId }, + context: tenant ? { isSystem: true, tenantId: tenant } : { isSystem: true }, + }); + } catch (err) { + logger.warn( + `[storage] attachFile: a refused call could not remove its sys_attachment row ${call.attachmentId} ` + + `(${(err as Error)?.message ?? err}). It points at the pending sys_file ${call.fileId}; delete the ` + + 'attachment row by hand.', + ); + } + } + try { + await storage.delete(call.key); + } catch (err) { + logger.warn( + `[storage] attachFile: a refused call could not delete its stored bytes (${(err as Error)?.message ?? err}). ` + + `The sys_file row ${call.fileId} stays pending, so the pending-upload retention reclaims the bytes and the row.`, + ); + return; + } + try { + await store.deleteFile(call.fileId, call.writeContext); + } catch (err) { + logger.warn( + `[storage] attachFile: a refused call could not delete its pending sys_file row ${call.fileId} ` + + `(${(err as Error)?.message ?? err}). The pending-upload retention removes it.`, + ); + } + } +} diff --git a/packages/services/service-storage/src/storage-limits.ts b/packages/services/service-storage/src/storage-limits.ts index 595d3fdde76..8c3629eca62 100644 --- a/packages/services/service-storage/src/storage-limits.ts +++ b/packages/services/service-storage/src/storage-limits.ts @@ -45,6 +45,8 @@ * signature), never re-derived. */ +import type { RegisteredErrorCode } from '@objectstack/spec/api'; + /** One limit key as the `storage` settings cascade answered it. */ export interface StorageLimitReading { /** The effective value, in the key's own unit: seconds for the TTLs, MB (MiB) for the size. */ @@ -173,6 +175,60 @@ export function resolveStorageLimits( }; } +/** + * [#22283] The answer every door that accepts bytes gives a file over the + * resolved `max_upload_mb` — before the backend stores a byte and before any + * row is written. The REST upload doors (`storage-routes.ts`) send it as their + * response; [#23306] the server-side attach verb (`file-attachment-service.ts`) + * throws it. One status, one code and one sentence, read from here by both. + * + * `413` because that is the condition (RFC 9110, content too large). + * `PAYLOAD_TOO_LARGE` because it is the one registered code for that condition + * (ADR-0112 ledger, listed under this package's owner key beside + * `@objectstack/rest`, which answers it for its import row ceilings): a client + * branches on "too large" by the code alone, at every door that refuses it. + * Not `VALIDATION_ERROR`, the bucket `standardErrorCodeForHttpStatus` derives + * for a `413`: the standard catalog names no `413` member, so that code would + * say only "malformed request" and leave the client to read the status. + * + * Typed `RegisteredErrorCode` so a misspelling, or the code leaving the + * ledger, fails to compile; that this package's own owner key lists it is + * `check:error-code-provenance`'s to hold. The message names the limit and + * where it is set. + */ +export const UPLOAD_TOO_LARGE_STATUS = 413; +export const UPLOAD_TOO_LARGE_CODE: RegisteredErrorCode = 'PAYLOAD_TOO_LARGE'; + +/** The refusal's message: what was measured, the limit, and the one setting that moves it. */ +function uploadTooLargeMessage(measured: string, maxUploadBytes: number): string { + const mb = Math.round((maxUploadBytes / BYTES_PER_UPLOAD_MB) * 100) / 100; + return ( + `${measured}, over the maximum upload size of ${mb} MB (${maxUploadBytes} bytes). ` + + 'The limit is the "Max upload size (MB)" storage setting (max_upload_mb in the storage settings), ' + + 'which an administrator can change.' + ); +} + +/** + * [#22283, #23306] The ONE size verdict: `null` when `bytes` may be stored, + * otherwise the refusal's message (sent or thrown with + * {@link UPLOAD_TOO_LARGE_STATUS} / {@link UPLOAD_TOO_LARGE_CODE}). + * + * No limit (`maxUploadBytes` undefined: no settings namespace bound) admits + * any size. `bytes` that is not a finite number cannot be judged and is + * admitted — the byte-carrying doors judge what they actually receive. + * `measured` words the size for the message ("The declared file size is N + * bytes"), because each door knows what it measured. + */ +export function uploadSizeRefusal( + bytes: number, + maxUploadBytes: number | undefined, + measured: (bytes: number) => string, +): string | null { + if (maxUploadBytes === undefined || !Number.isFinite(bytes) || bytes <= maxUploadBytes) return null; + return uploadTooLargeMessage(measured(bytes), maxUploadBytes); +} + /** Whether two snapshots say the same thing — so a re-read that changed nothing logs nothing. */ export function sameStorageLimits(a: StorageLimitsSnapshot | undefined, b: StorageLimitsSnapshot | undefined): boolean { for (const { field } of LIMIT_KEYS) { diff --git a/packages/services/service-storage/src/storage-routes.ts b/packages/services/service-storage/src/storage-routes.ts index 287e774517f..eb4df02d268 100644 --- a/packages/services/service-storage/src/storage-routes.ts +++ b/packages/services/service-storage/src/storage-routes.ts @@ -50,9 +50,13 @@ import type { FileHolder } from './attachment-lifecycle.js'; import { contentDispositionValue } from './content-disposition.js'; // [#22283] The `storage` settings namespace's Limits group — read and resolved // in ONE module, called per request by the doors below. +// [#23306] The size refusal itself lives there too, beside the limit it +// enforces, because a second door asks it: the server-side attach verb. import { resolveStorageLimits, - BYTES_PER_UPLOAD_MB, + uploadSizeRefusal, + UPLOAD_TOO_LARGE_CODE, + UPLOAD_TOO_LARGE_STATUS, type ResolvedStorageLimits, type StorageLimitsSnapshot, } from './storage-limits.js'; @@ -83,38 +87,6 @@ const ORGANIZATION_CHANGED_MESSAGE = 'This upload was started in a different organization than your active one. ' + 'Switch your active organization back to the one the upload was started in, then finish the upload.'; -/** - * [#22283] The answer every upload door gives a file over the resolved - * `max_upload_mb` — before the backend stores a byte and before any row is - * written. - * - * `413` because that is the condition (RFC 9110, content too large). - * `PAYLOAD_TOO_LARGE` because it is the one registered code for that condition - * (ADR-0112 ledger, listed under this package's owner key beside - * `@objectstack/rest`, which answers it for its import row ceilings): a client - * branches on "too large" by the code alone, at every door that refuses it. - * Not `VALIDATION_ERROR`, the bucket `standardErrorCodeForHttpStatus` derives - * for a `413`: the standard catalog names no `413` member, so that code would - * say only "malformed request" and leave the client to read the status. - * - * Typed `RegisteredErrorCode` so a misspelling, or the code leaving the - * ledger, fails to compile; that this package's own owner key lists it is - * `check:error-code-provenance`'s to hold. The message names the limit and - * where it is set. - */ -const UPLOAD_TOO_LARGE_STATUS = 413; -const UPLOAD_TOO_LARGE_CODE: RegisteredErrorCode = 'PAYLOAD_TOO_LARGE'; - -/** The refusal's message: what was measured, the limit, and the one setting that moves it. */ -function uploadTooLargeMessage(measured: string, maxUploadBytes: number): string { - const mb = Math.round((maxUploadBytes / BYTES_PER_UPLOAD_MB) * 100) / 100; - return ( - `${measured}, over the maximum upload size of ${mb} MB (${maxUploadBytes} bytes). ` + - 'The limit is the "Max upload size (MB)" storage setting (max_upload_mb in the storage settings), ' + - 'which an administrator can change.' - ); -} - /** * [#22313] The answer the chunked-completion door gives an upload that does * not hold the file it declared — a declared chunk it never received, a chunk @@ -472,8 +444,9 @@ export function registerStorageFamily( measured: (bytes: number) => string, res: IHttpResponse, ): boolean => { - if (maxUploadBytes === undefined || !Number.isFinite(bytes) || bytes <= maxUploadBytes) return true; - sendError(res, UPLOAD_TOO_LARGE_STATUS, UPLOAD_TOO_LARGE_CODE, uploadTooLargeMessage(measured(bytes), maxUploadBytes)); + const refusal = uploadSizeRefusal(bytes, maxUploadBytes, measured); + if (refusal === null) return true; + sendError(res, UPLOAD_TOO_LARGE_STATUS, UPLOAD_TOO_LARGE_CODE, refusal); return false; }; @@ -1839,7 +1812,12 @@ function judgeCompletion(session: UploadSessionRecord, listed: ReadonlyArray/<.ext>`. Package-internal + * (not re-exported from `index.ts`); [#23306] exported so the server-side + * attach verb lays its keys out exactly as the upload doors do. + */ +export function buildKey(scope: string, fileId: string, filename: string): string { const ext = filename.includes('.') ? '.' + filename.split('.').pop() : ''; return `${scope}/${fileId}${ext}`; } diff --git a/packages/services/service-storage/src/storage-service-plugin.ts b/packages/services/service-storage/src/storage-service-plugin.ts index d5b02fdd149..f32cfd8389b 100644 --- a/packages/services/service-storage/src/storage-service-plugin.ts +++ b/packages/services/service-storage/src/storage-service-plugin.ts @@ -42,6 +42,7 @@ import { installAttachmentLifecycleHooks, createSysFileReapGuard, createUploadSe import { installFileReferenceHooks } from './file-reference-lifecycle.js'; import { installAttachmentAccessHooks, installAttachmentReadVisibility } from './attachment-access-hooks.js'; import { contributeAttachmentDeleteFloorAlternate } from './attachment-delete-floor-alternate.js'; +import { FileAttachmentService, type FileAttachmentEngine } from './file-attachment-service.js'; import { SystemFile, SystemUploadSession } from './objects/index.js'; // ADR-0052 §3 ownership: `sys_attachment` (a file↔record link) belongs with the // storage domain, not the audit/compliance ledger. Definition stays in @@ -168,8 +169,12 @@ export class StorageServicePlugin implements Plugin { * instance so existing callers of either spelling resolve identically — * the `http.server` / `http-server` pattern. The alias registration is * dropped through the standard retirement flow at the next major. + * + * [#23306] `file-attachments` (`FILE_ATTACHMENT_SERVICE`) is the server-side + * attach verb (`file-attachment-service.ts`): a slot of its own, because the + * `storage` slot's contract is the adapters' blob store. */ - providesServices = ['storage', 'file-storage']; + providesServices = ['storage', 'file-storage', 'file-attachments']; /** * `com.objectstack.engine.objectql` — init() registers sys_file / * sys_upload_session / sys_attachment through the `manifest` service @@ -212,6 +217,15 @@ export class StorageServicePlugin implements Plugin { private readonly options: StorageServicePluginOptions; private storage: SwappableStorageService | null = null; + /** + * [#23306] The data engine the attach verb writes through — set at + * `kernel:bootstrapped`, after every plugin's `kernel:ready` handler has + * installed its gates on the `sys_attachment` insert, and never before: until + * then the verb refuses (`503`) rather than write a row a gate has not yet + * judged. Stays `null` on a kernel with no hook-capable engine, where those + * gates cannot exist. + */ + private attachmentEngine: FileAttachmentEngine | null = null; private metrics: MetricsRegistry = new NoopMetricsRegistry(); /** * What the CURRENTLY installed adapter points at (#4096). Set beside every @@ -328,6 +342,20 @@ export class StorageServicePlugin implements Plugin { ctx.registerService('storage', this.storage); // Deprecated v17 alias — same instance under the old spelling (#9683). ctx.registerService('file-storage', this.storage); + // [#23306] The server-side attach verb, under `FILE_ATTACHMENT_SERVICE`. + // Registered on every path (ADR-0116); everything it writes through is read + // per call, so it serves once `kernel:bootstrapped` arms the engine + // (`start()`). + const storage = this.storage; + ctx.registerService( + 'file-attachments', + new FileAttachmentService({ + getEngine: () => this.attachmentEngine, + getStorage: () => storage, + getLimitsSnapshot: () => storage.getLimitsSnapshot(), + logger: ctx.logger, + }), + ); ctx.logger.info( `StorageServicePlugin: registered ${adapter} storage adapter (swappable, metrics=${this.metrics.constructor?.name ?? 'unknown'})`, ); @@ -365,6 +393,16 @@ export class StorageServicePlugin implements Plugin { } async start(ctx: PluginContext): Promise { + // [#23306] The engine the `kernel:ready` hook below installs this plugin's + // `sys_attachment` gates on. The attach verb is armed with it only at + // `kernel:bootstrapped`, once EVERY plugin's `kernel:ready` handler has run — + // other plugins install their gates on the same insert from theirs (the + // `enable.files` gate among them), so a verb armed any earlier could write + // a row one of them never judged. + let gatedEngine: FileAttachmentEngine | null = null; + ctx.hook('kernel:bootstrapped', async () => { + this.attachmentEngine = gatedEngine; + }); ctx.hook('kernel:ready', async () => { let engine: IDataEngine | null = null; try { @@ -422,6 +460,9 @@ export class StorageServicePlugin implements Plugin { // stays the last word there (maintainer ruling; the reasoning is in // `attachment-delete-floor-alternate.ts`). Delete limb only. contributeAttachmentDeleteFloorAlternate(() => ctx.getService('security'), ctx.logger); + // [#23306] The engine the attach verb writes through, once armed at + // `kernel:bootstrapped` (see the top of `start()`). + gatedEngine = engine as FileAttachmentEngine; // Parent-derived READ visibility (#2970 item 1) — list/find/count of // sys_attachment only returns rows whose parent record the caller can // read. Middleware (not a hook) so list `total` is filtered too. diff --git a/packages/services/service-storage/vitest.config.ts b/packages/services/service-storage/vitest.config.ts index 7bbb33530ae..7f30a511a76 100644 --- a/packages/services/service-storage/vitest.config.ts +++ b/packages/services/service-storage/vitest.config.ts @@ -65,9 +65,15 @@ export default defineConfig({ // // Anchored patterns in array form, per the note above: a bare key whose // replacement is a FILE swallows the package's subpaths. + // + // `@objectstack/plugin-audit` joins them for the same reason (#23306): the + // attach verb's pins run its REAL `enable.files` gate on the + // `sys_attachment` insert, and a gate read from `dist/` would make those + // pins a verdict about build state. alias: [ { find: /^@objectstack\/core$/, replacement: path.resolve(__dirname, '../../core/src/index.ts') }, { find: /^@objectstack\/driver-sql$/, replacement: path.resolve(__dirname, '../../drivers/driver-sql/src/index.ts') }, + { find: /^@objectstack\/plugin-audit$/, replacement: path.resolve(__dirname, '../../plugins/plugin-audit/src/index.ts') }, ], }, }); diff --git a/packages/spec/api-surface/contracts.json b/packages/spec/api-surface/contracts.json index 41ebcdd13a4..9b513551625 100644 --- a/packages/spec/api-surface/contracts.json +++ b/packages/spec/api-surface/contracts.json @@ -44,6 +44,8 @@ "ApprovalSendBackResult (interface)", "ApprovalStatus (type)", "AssistantModelMessage (type)", + "AttachFileInput (interface)", + "AttachedFile (interface)", "AudienceBindingSuggestion (type)", "AudienceBindingSuggestionFilter (interface)", "AudienceBindingSuggestionSync (interface)", @@ -96,6 +98,7 @@ "EngineTransactionOptions (interface)", "ExecuteUpgradeInput (interface)", "ExplainAccessRequest (interface)", + "FILE_ATTACHMENT_SERVICE (const)", "FinishReason (type)", "FlowRuntimeState (interface)", "GenerateDraftOpts (interface)", @@ -127,6 +130,7 @@ "IEmbedder (interface)", "IExternalDatasourceService (interface)", "IFileAccessDelegate (interface)", + "IFileAttachmentService (interface)", "IHierarchyScopeResolver (interface)", "IHttpRequest (interface)", "IHttpResponse (interface)", diff --git a/packages/spec/export-origins/contracts.json b/packages/spec/export-origins/contracts.json index b70dc9a63f0..95141781216 100644 --- a/packages/spec/export-origins/contracts.json +++ b/packages/spec/export-origins/contracts.json @@ -44,6 +44,8 @@ "ApprovalSendBackResult": "src/contracts/approval-service.ts#ApprovalSendBackResult (interface)", "ApprovalStatus": "src/contracts/approval-service.ts#ApprovalStatus (type)", "AssistantModelMessage": "node_modules/@ai-sdk/provider-utils/dist/index.d.ts#AssistantModelMessage (type)", + "AttachFileInput": "src/contracts/storage-service.ts#AttachFileInput (interface)", + "AttachedFile": "src/contracts/storage-service.ts#AttachedFile (interface)", "AudienceBindingSuggestion": "src/contracts/security-service.ts#AudienceBindingSuggestion (type)", "AudienceBindingSuggestionFilter": "src/contracts/security-service.ts#AudienceBindingSuggestionFilter (interface)", "AudienceBindingSuggestionSync": "src/contracts/security-service.ts#AudienceBindingSuggestionSync (interface)", @@ -96,6 +98,7 @@ "EngineTransactionOptions": "src/contracts/objectql-engine.ts#EngineTransactionOptions (interface)", "ExecuteUpgradeInput": "src/contracts/package-service.ts#ExecuteUpgradeInput (interface)", "ExplainAccessRequest": "src/contracts/security-service.ts#ExplainAccessRequest (interface)", + "FILE_ATTACHMENT_SERVICE": "src/contracts/storage-service.ts#FILE_ATTACHMENT_SERVICE (const)", "FinishReason": "node_modules/ai/dist/index.d.ts#FinishReason (type)", "FlowRuntimeState": "src/contracts/automation-service.ts#FlowRuntimeState (interface)", "GenerateDraftOpts": "src/contracts/external-datasource-service.ts#GenerateDraftOpts (interface)", @@ -127,6 +130,7 @@ "IEmbedder": "src/contracts/embedder.ts#IEmbedder (interface)", "IExternalDatasourceService": "src/contracts/external-datasource-service.ts#IExternalDatasourceService (interface)", "IFileAccessDelegate": "src/contracts/storage-service.ts#IFileAccessDelegate (interface)", + "IFileAttachmentService": "src/contracts/storage-service.ts#IFileAttachmentService (interface)", "IHierarchyScopeResolver": "src/contracts/sharing-service.ts#IHierarchyScopeResolver (interface)", "IHttpRequest": "src/contracts/http-server.ts#IHttpRequest (interface)", "IHttpResponse": "src/contracts/http-server.ts#IHttpResponse (interface)", diff --git a/packages/spec/src/contracts/storage-service.test.ts b/packages/spec/src/contracts/storage-service.test.ts index 3392da86a75..fb1be6e4883 100644 --- a/packages/spec/src/contracts/storage-service.test.ts +++ b/packages/spec/src/contracts/storage-service.test.ts @@ -1,11 +1,14 @@ import { describe, it, expect } from 'vitest'; import type { + AttachFileInput, + IFileAttachmentService, IStorageService, StorageFileInfo, StorageListPage, } from './storage-service'; import { DEFAULT_STORAGE_LIST_LIMIT, + FILE_ATTACHMENT_SERVICE, decodeStorageListCursor, encodeStorageListCursor, resolveStorageListLimit, @@ -370,3 +373,29 @@ describe('Storage Service Contract', () => { expect(aborted).toBe(true); }); }); + +describe('File Attachment Service Contract (#23306)', () => { + it('names its own registry slot, beside the blob store rather than on it', () => { + expect(FILE_ATTACHMENT_SERVICE).toBe('file-attachments'); + }); + + it('is one verb from an input naming the parent, the organization and the bytes to the ids it wrote', async () => { + const seen: AttachFileInput[] = []; + const service: IFileAttachmentService = { + attachFile: async (input) => { + seen.push(input); + return { fileId: 'f1', attachmentId: 'a1', size: input.data.byteLength }; + }, + }; + const out = await service.attachFile({ + parentObject: 'sys_mail_message', + parentId: 'm1', + organizationId: null, + filename: 'forwarded.eml', + mimeType: 'message/rfc822', + data: Buffer.from('hello'), + }); + expect(out).toEqual({ fileId: 'f1', attachmentId: 'a1', size: 5 }); + expect(seen[0].organizationId).toBeNull(); + }); +}); diff --git a/packages/spec/src/contracts/storage-service.ts b/packages/spec/src/contracts/storage-service.ts index dcb446f9beb..59091e64009 100644 --- a/packages/spec/src/contracts/storage-service.ts +++ b/packages/spec/src/contracts/storage-service.ts @@ -434,3 +434,95 @@ export interface IFileAccessDelegate { */ authorizeFileRead(recordId: string, context: unknown): Promise; } + +// ========================================== +// Server-side attach: store bytes, attach them to a record (#23306) +// ========================================== + +/** + * Service-registry slot for {@link IFileAttachmentService}. + * + * `@objectstack/service-storage` registers it: that package owns `sys_file` + * and `sys_attachment` and runs every gate on them. A slot of its own, not a + * member of {@link IStorageService}: that contract is the blob store a + * storage ADAPTER implements (the local and S3 adapters, a third party's), and + * none of them holds the data engine this verb writes rows through. + */ +export const FILE_ATTACHMENT_SERVICE = 'file-attachments' as const; + +/** What {@link IFileAttachmentService.attachFile} stores, and where it attaches it. */ +export interface AttachFileInput { + /** + * The parent record's object, by its short name. The object must declare + * `enable: { files: true }`, the opt-in every attachment of it needs. + */ + parentObject: string; + /** The parent record's id. */ + parentId: string; + /** + * The organization the caller acts in, or `null` for a caller that acts in + * none (a single-tenant deployment, a platform-wide job). Required, so that + * acting in no organization is stated, never a field left out. + * + * With an organization, the parent must belong to it or to none (a global + * record); a parent walled into another organization is refused as not + * found. Both rows are written in the parent's own organization when it has + * one, and otherwise in this one. + */ + organizationId: string | null; + /** The file name an attachment list shows. Its extension is kept on the storage key. */ + filename: string; + /** The file's media type. */ + mimeType: string; + /** + * The bytes. Their length is judged against the deployment's upload size + * limit (the `max_upload_mb` storage setting) before anything is stored. + */ + data: Buffer; +} + +/** What {@link IFileAttachmentService.attachFile} wrote. */ +export interface AttachedFile { + /** The new `sys_file` row's id: the opaque file id the download doors take. */ + fileId: string; + /** The new `sys_attachment` row's id. */ + attachmentId: string; + /** The number of bytes stored. */ + size: number; +} + +/** + * The server-side door that stores bytes and attaches them to a record: one + * `sys_file` row and one `sys_attachment` row linking it to the parent. + * + * For a producer with no browser upload in front of it (inbound mail, an + * import, generated output), so it acts as the SYSTEM, not as a user: there is + * no caller whose edit access to the parent could be asked. What it does run is + * what every attachment of the parent runs, through the same code: the upload + * size limit the REST upload doors apply, the parent's organization, and every + * gate that judges a `sys_attachment` insert, the `enable.files` opt-in among + * them. + * + * Refusals carry `code` and `status` (ADR-0112): + * - `VALIDATION_ERROR` / 400: a malformed input; + * - `PAYLOAD_TOO_LARGE` / 413: the bytes are over the upload size limit, + * refused before anything is stored, with the REST upload doors' own answer; + * - `RECORD_NOT_FOUND` / 404: no such parent record, or one walled into + * another organization than `organizationId`; + * - `FILES_DISABLED` / 403: the parent object does not declare + * `enable: { files: true }`; + * - any other refusal a gate on the `sys_attachment` insert raises, unchanged; + * - `SERVICE_UNAVAILABLE` / 503: no data engine is wired, or the kernel has + * not finished booting. It serves once `kernel:bootstrapped` has fired — + * after every plugin's `kernel:ready` handler has installed its gates. + * + * ⛔ No refusal leaves a committed `sys_file`. The row is written `pending` + * before the bytes, so the platform's pending-upload retention can reclaim + * both if the process dies mid-call, and turns `committed` only in the same + * transaction as the `sys_attachment` insert. A refused call removes the + * bytes and the row before it throws. + */ +export interface IFileAttachmentService { + /** Store `input.data` and attach it to the parent record, as one file and one attachment. */ + attachFile(input: AttachFileInput): Promise; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 47a2cc02146..d290d546a0a 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -2958,6 +2958,9 @@ importers: '@objectstack/objectql': specifier: workspace:* version: link:../../objectql + '@objectstack/plugin-audit': + specifier: workspace:* + version: link:../../plugins/plugin-audit '@types/node': specifier: ^26.6.3 version: 26.6.3