feat(connector-imap, service-mailbox): the IMAP mail source — a read-only IMAP connector and the mailbox sync that feeds the ingest seam

.changeset/22871-imap-mailbox-sync.md

@@ -0,0 +1,27 @@
+---
+'@objectstack/connector-imap': minor
+'@objectstack/service-mailbox': minor
+---
+
+Inbound mail has a mail source: the new `@objectstack/connector-imap` reads a mailbox over IMAP, and `@objectstack/service-mailbox` polls each mailbox's folder into its ingest seam, keeping a UID cursor on `sys_mailbox`
+
+Clause-②: yes (a new published package, `@objectstack/connector-imap`, and its exports; new exports and plugin options in `@objectstack/service-mailbox`, and four new fields on its published `sys_mailbox` object)
+
+- **`@objectstack/connector-imap` (new) — protocol and authentication only.**
+  - `openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes?, maxCommandBytes? })` opens a read-only IMAP4rev1 session: `examine(folder)` (`EXAMINE`, never `SELECT`; answers `uidValidity`, `uidNext`, `exists`), `listMessages(fromUid)`, `fetchMessages(uids)` (`BODY.PEEK[]`, so no message is marked `\Seen`), `logout()` and `close()`. A folder opened without a UIDVALIDITY is refused.
+  - The account and credential abstraction is shaped for the per-user mailbox: a `MailAccount` (`{ id, username, imap: { host, port?, security? } }`) and a `MailCredentialProvider` that resolves the account's credential each time a session opens. `staticCredentials(credential)` answers one fixed credential.
+  - A `password` credential is sent with `AUTHENTICATE PLAIN` when the server offers it, otherwise `LOGIN`; an `oauth2` access token with `AUTHENTICATE XOAUTH2`. A server that refuses `LOGIN` and offers no `PLAIN` is refused before anything is sent.
+  - `security: 'tls'` (the default) is implicit TLS with the certificate verified. A plaintext connection (`security: 'none'`) is refused to any host but loopback, before a socket opens.
+  - Failures are `ImapError` with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused`, `protocol` or `oversized`.
+  - An IMAP server is treated as an untrusted peer. A line above 4 MiB, a literal above `maxLiteralBytes` (64 MiB by default), a response of more than `IMAP_MAX_RESPONSE_LITERALS` (32) literals, and more than `maxCommandBytes` of responses to one command (128 MiB by default) are refused as `oversized` before the excess is kept, and the connection is closed.
+  - `@objectstack/connector-imap/testing` exports `ImapServerDouble`, an in-process IMAP server on loopback for tests.
+  - No runtime dependencies.
+- **`@objectstack/service-mailbox` — the IMAP sync.**
+  - `new MailboxServicePlugin({ imap: { accountFor, pollIntervalMs?, batchSize?, maxMessagesPerRun?, maxMessageBytes?, maxBatchBytes? } })` schedules the interval job `mailbox:imap-sync` on the composed job service. For each mailbox `accountFor` answers (asked per mailbox per run; `null` for a mailbox the host does not sync), it fetches the messages above the stored cursor, in UID order, and hands each one to `ingest` under the system context. Dedupe, association, the pool and auto-reply suppression are unchanged.
+  - Without `imap` — the capability boot constructs the plugin with no options — no mailbox is synced, and the boot says so once. With `imap` but no job service, the boot warns, and `plugin.syncService.syncAll({ isSystem: true })` runs a pass on the host's own schedule. A run in which a mailbox could not be synced reports `degraded` to the job service.
+  - `sys_mailbox` gains the cursor, written by the sync only: `imap_folder`, `imap_uid_validity`, `imap_last_uid` and `imap_synced_at`. A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start; the messages already stored answer `duplicate`. No credential is stored on the object: the account and its credential come from `accountFor`.
+  - The cursor moves after each batch past every message stored, already stored, suppressed or skipped. A store failure stops the run below the failed message, so the next run retries it. A message above `maxMessageBytes` (32 MiB by default) or one the seam refuses as unreadable is skipped and logged at `error` with its UID.
+  - A batch is bounded by count (`batchSize`, 25) and by listed bytes (`maxBatchBytes`, 64 MiB). The session's literal bound is `maxMessageBytes` plus a 64 KiB framing envelope, so a server that omits or understates a message's size cannot hand over more. That message is skipped as `too_large`, after the batch is read again one message at a time on a fresh session, and the rest of the mailbox keeps arriving.
+  - A forwarded `message/global` part (RFC 6532, the internationalized `message/rfc822`) is now read as an `attachment` source, the same as `message/rfc822`.
+  - New exports: `MailboxSyncService`, `MAILBOX_SYNC_JOB`, `DEFAULT_MAILBOX_POLL_INTERVAL_MS`, `DEFAULT_SYNC_FOLDER`, `DEFAULT_SYNC_BATCH_SIZE`, `DEFAULT_SYNC_MAX_MESSAGES_PER_RUN`, `DEFAULT_SYNC_MAX_MESSAGE_BYTES`, `DEFAULT_SYNC_MAX_BATCH_BYTES`, `SYNC_LITERAL_ENVELOPE_BYTES`, and the types `MailboxServicePluginOptions`, `MailboxImapSyncOptions`, `MailboxAccountResolver`, `MailboxImapAccount`, `MailboxSyncTarget`, `MailboxSyncResult`, `MailboxSyncAllResult`, `MailboxSyncSkipReason`, `MailboxSyncServiceOptions`, `MailSyncEngine` and `MailSyncLogger`.
+  - This supersedes the ingest seam entry's "no mail source feeds the seam": the IMAP sync does. Attachments are still not stored.

.changeset/config.json

@@ -52,6 +52,7 @@
       "@objectstack/trigger-record-change",
       "@objectstack/trigger-api",
       "@objectstack/trigger-schedule",
+      "@objectstack/connector-imap",
       "@objectstack/connector-mcp",
       "@objectstack/connector-rest",
       "@objectstack/connector-slack",

README.md

@@ -387,6 +387,7 @@ control plane.
 | [`@objectstack/connector-openapi`](packages/connectors/connector-openapi) | Connector actions generated from an OpenAPI document |
 | [`@objectstack/connector-mcp`](packages/connectors/connector-mcp) | Any MCP server's tools as connector actions |
 | [`@objectstack/connector-slack`](packages/connectors/connector-slack) | Slack Web API connector |
+| [`@objectstack/connector-imap`](packages/connectors/connector-imap) | Read-only IMAP session and mail account abstraction for the mailbox sync |
 | [`@objectstack/trigger-record-change`](packages/triggers/trigger-record-change) | Launch flows on insert / update / delete |
 | [`@objectstack/trigger-schedule`](packages/triggers/trigger-schedule) | Launch flows on a cron, interval, or one-off schedule |
 | [`@objectstack/trigger-api`](packages/triggers/trigger-api) | Inbound HTTP / webhook flow trigger with HMAC verification |

content/docs/permissions/system-context.mdx

@@ -170,6 +170,7 @@ One of the largest consumers of the flag;
 | 46 | Knowledge search returns hits unfiltered | service-knowledge | Lose: the permission filter over search results | `packages/services/service-knowledge/src/knowledge-service.ts#applyPermissionFilter` |
 | 47 | The inbound-mail read gates return early (the read AST) | service-mailbox | Get: every `sys_mail_message` and `sys_mail_association` row on `find` / `findOne` — the ingest writer and each gate's own pre-scan, which read the association and mailbox rows a decision needs. Lose: the narrowing of a message to callers who can read a record it is associated with or who own its mailbox (so the unassociated pool is restricted), and of an association to callers who can read the record it names (on `count` and `aggregate` too, so a total agrees with the rows) | `packages/services/service-mailbox/src/mail-read-visibility.ts#gateMiddleware` |
 | 47b | The inbound-mail ingest seam: `ingest` admits the system context ALONE, and `rematch` admits it without the organization, owner or `manage_mailboxes` check | service-mailbox | Get: a mail source hands a mailbox its messages, and server code re-matches any mailbox. Lose: nothing a user held — every other caller of `ingest` is refused (`PERMISSION_DENIED`), because an ingested message is shown on every record its correspondents' addresses match and a user who could hand the seam a message could write one in a correspondent's name; every other caller of `rematch` must be in the mailbox's organization and own the mailbox or hold `manage_mailboxes` | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts#ingest`, `#authorizeRematch` |
+| 47c | The inbound-mail IMAP sync: `syncMailbox` and `syncAll` admit the system context ALONE | service-mailbox | Get: the sync job reads every mailbox row (its organization, address and IMAP cursor), writes the cursor back, and hands each fetched message to `ingest` under the system context. Lose: nothing a user held — every other caller is refused (`PERMISSION_DENIED`), because a sync stores a mailbox's mail and moves its cursor | `packages/services/service-mailbox/src/sync/mailbox-sync-service.ts#syncMailbox`, `#syncAll` |
 
 ### 5. Actions, metadata plane, provenance, the organization wall
 

content/docs/permissions/tenant-audit-census.mdx

@@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way.
 
 The same holds twice over for the context. An options argument spelled as a
 literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
-forwarding shim cannot, and **53 of the 225 sites are spelled that way**. A
+forwarding shim cannot, and **53 of the 226 sites are spelled that way**. A
 context resolved from an inline literal or a local `const` can be tested for
 `isSystem`; one arriving from a helper call cannot.
 
@@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page:
 
 | carried figure | where it survives | this census |
 | :--- | :--- | ---: |
-| 175 write call sites | quoted in the merged changeset | **225** |
+| 175 write call sites | quoted in the merged changeset | **226** |
 | 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **32** more whose options argument is unreadable |
-| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **153 of 225** decidable, **72** undecidable |
-| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 120 decidably elevated, 0 decidably not, 97 undecidable |
+| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **153 of 226** decidable, **73** undecidable |
+| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 121 decidably elevated, 0 decidably not, 97 undecidable |
 | 141 and 132, two independent re-derivations | the card that filed this work | — |
 
 **The differences are not reconciled, and deliberately so.** The old census's
@@ -207,11 +207,11 @@ would report a smaller number and would not say so.
 
 The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
 figure has no surviving corroboration anywhere in the tree.** This census reads
-120 of 225 (53%) as decidably elevated, with 97 more whose elevation is a
+121 of 226 (54%) as decidably elevated, with 97 more whose elevation is a
 run-time fact — so the claim is neither confirmed nor refuted, and the honest
 answer is that a static reading cannot settle it.
 
-⇒ **Cite `2 / 225`, and say what it is**: the sites whose options argument was
+⇒ **Cite `2 / 226`, and say what it is**: the sites whose options argument was
 READ and holds no tenant context, against a decidably tenancy-enabled object.
 That is the control's provable yield surface. ⛔ Do not cite it as "the sites
 without tenant context" — **32 further sites** have an options argument this
@@ -223,31 +223,31 @@ cannot read, and they are neither in nor out.
 
 | what | count |
 | :--- | ---: |
-| write call sites on the application surface | **225** |
+| write call sites on the application surface | **226** |
 | …whose object name is statically decidable | 153 |
-| …whose object name is chosen at run time | 72 |
+| …whose object name is chosen at run time | 73 |
 | …against an object with tenancy ENABLED | 151 |
 | …against an object that declares tenancy off | 1 |
-| threading a tenant context | 164 |
+| threading a tenant context | 165 |
 | PROVABLY carrying none (options read, no context key) | **8** |
 | …of those, against a decidably tenancy-enabled object | **2** |
 | options argument UNREADABLE — may or may not carry one | 53 |
 | …of those, against a decidably tenancy-enabled object | 32 |
-| threading a decidably ELEVATED (`isSystem`) context | 120 |
+| threading a decidably ELEVATED (`isSystem`) context | 121 |
 | threading a context that is decidably NOT elevated | 0 |
 | threading a context whose elevation is a run-time fact | 97 |
 
 | how the instrument reached the site | count |
 | :--- | ---: |
-| receiver carried a readable engine type | 185 |
+| receiver carried a readable engine type | 186 |
 | receiver erased, placed by the object NAME | 27 |
 | receiver erased, placed by an `object: string` PARAMETER | 7 |
 | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 6 |
 
 | object name spelled inline | 103 |
 | object name spelled through a `const` | 50 |
 | object name is an `object: string` parameter | 9 |
-| object name is some other run-time expression | 63 |
+| object name is some other run-time expression | 64 |
 
 ### Subtractions the census could NOT defend — enforced
 
@@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
 their values are not compared. The reasoning, and the measurement behind it,
 are in `scripts/check-tenant-audit-census.mjs`.
 
-Measured on 2026-10-13 at `66949b772`.
+Measured on 2026-10-13 at `c33c6dc67`.
 
 | corpus scale (not enforced) | count |
 | :--- | ---: |
-| tracked non-test sources scanned | 648 |
-| engine-shaped types recognised | 72 |
+| tracked non-test sources scanned | 650 |
+| engine-shaped types recognised | 73 |
 | declared objects in the registry | 116 |
-| same-named calls subtracted as non-engine | 160 |
+| same-named calls subtracted as non-engine | 162 |
 
 {/* END GENERATED: tenant-audit-census */}

content/docs/protocol/backward-compatibility.mdx

@@ -114,7 +114,7 @@ Read the **Breaking?** column, not the version number: in the 17.x minors every
 1. **RFC (Request for Comments)** — Breaking changes are proposed as GitHub issues with the `protocol:breaking` label.
 2. **Deprecation** — The old behavior is deprecated in a MINOR release (see timeline above).
 3. **Migration Guide** — A detailed migration guide is published before the removal lands, in the release notes for the version that carries it.
-4. **Release** — In the 17.x minors the breaking change shipped in the next **MINOR** version, carrying a changeset entry marked `**BREAKING**`, and `scripts/check-changeset-no-major.mjs` failed a pull request that declared a `major` bump, because under lockstep one `major` would promote all 71 published packages. On the v18 line it ships in the next `18.0.0-next.N`: the guard stands aside in pre mode, so a pull request may declare a major bump. See [how a breaking stage lands on the v18 line](/docs/releases/v18#how-a-breaking-stage-lands-on-this-line).
+4. **Release** — In the 17.x minors the breaking change shipped in the next **MINOR** version, carrying a changeset entry marked `**BREAKING**`, and `scripts/check-changeset-no-major.mjs` failed a pull request that declared a `major` bump, because under lockstep one `major` would promote all 72 published packages. On the v18 line it ships in the next `18.0.0-next.N`: the guard stands aside in pre mode, so a pull request may declare a major bump. See [how a breaking stage lands on the v18 line](/docs/releases/v18#how-a-breaking-stage-lands-on-this-line).
 
 ---
 
@@ -192,7 +192,7 @@ The `@objectstack/spec` package provides additional stability guarantees:
 
 ### Which surfaces this covers
 
-**All of them.** It was not scoped to an experimental corner or a pre-release channel: all **71** packages published from this repository belong to a single Changesets `fixed` group, so they share one version number and one policy. No published surface was exempt.
+**All of them.** It was not scoped to an experimental corner or a pre-release channel: all **72** packages published from this repository belong to a single Changesets `fixed` group, so they share one version number and one policy. No published surface was exempt.
 
 The convention was enforced rather than informal — `scripts/check-changeset-no-major.mjs` failed a pull request that introduced a `major` bump, because under lockstep a single `major` on one package would promote the entire stack. Changesets is now in pre mode with the tag `next`, and the guard stands aside in pre mode, so a pull request on the v18 line may declare a major bump.
 

docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

@@ -33,17 +33,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.
 
 | Measure | Value |
 |---|---:|
-| Write call sites | 225 |
+| Write call sites | 226 |
 | Object name statically decidable | 153 |
-| Object name chosen at run time | 72 |
+| Object name chosen at run time | 73 |
 | Against a tenancy-enabled object | 151 |
 | Against an object declaring tenancy off | 1 |
-| Threading a tenant context | 164 |
+| Threading a tenant context | 165 |
 | Provably carrying none | 8 |
 | …and decidably tenancy-enabled | 2 |
 | Options argument unreadable | 53 |
 | …and decidably tenancy-enabled | 32 |
-| Threading a decidably elevated context | 120 |
+| Threading a decidably elevated context | 121 |
 | Threading a decidably non-elevated context | 0 |
 | Threading a context of undecidable elevation | 97 |
 
@@ -90,14 +90,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
 their values are not compared. The reasoning, and the measurement behind it,
 are in `scripts/check-tenant-audit-census.mjs`.
 
-Measured on 2026-10-13 at `66949b772`.
+Measured on 2026-10-13 at `c33c6dc67`.
 
 | corpus scale (not enforced) | count |
 | :--- | ---: |
-| tracked non-test sources scanned | 648 |
-| engine-shaped types recognised | 72 |
+| tracked non-test sources scanned | 650 |
+| engine-shaped types recognised | 73 |
 | declared objects in the registry | 116 |
-| same-named calls subtracted as non-engine | 160 |
+| same-named calls subtracted as non-engine | 162 |
 
 ## Every site
 
@@ -208,6 +208,7 @@ Measured on 2026-10-13 at `66949b772`.
 | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_ASSOCIATION_OBJECT` | undecidable | elevated | 1 |
 | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 |
 | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `update` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 |
+| `packages/services/service-mailbox/src/sync/mailbox-sync-service.ts` | `update` | `MAILBOX_OBJECT` | undecidable | elevated | 1 |
 | `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
 | `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `receiptObject` | undecidable | context, elevation undecidable | 1 |
 | `packages/services/service-messaging/src/messaging-service.ts` | `insert` | `RECEIPT_OBJECT` | undecidable | context, elevation undecidable | 1 |