Document per-query and per-principal memory limits

[puzpuzpuz]

Summary

Documents the new memory-limit feature across configuration, monitoring, and access control.

OSS — per-workload memory limits

• configuration/cairo-engine.md: new "Memory limits" section covering the reloadable workload limits (cairo.query.memory.limit.bytes, cairo.mat.view.refresh.memory.limit.bytes, cairo.wal.apply.memory.limit.bytes, plus a pointer to cairo.live.view.refresh.memory.limit.bytes, which the live-views config page already documents), per-workload breach behavior, and a note that memory-mapped memory does not count toward a limit. Cross-links added from the WAL, materialized-views, and capacity-planning pages.
• query/functions/meta.md: query_activity gains the memory_used and memory_limit columns.
• configuration/materialized-views.md: documents cairo.mat.view.refresh.busy.retry.limit and cairo.mat.view.refresh.busy.retry.timeout, which govern what happens after a refresh breaches its memory limit and were previously undocumented. The breach description in cairo-engine.md links to them.

Enterprise — per-principal memory limits

• query/sql/acl/alter-user.md, alter-service-account.md, and a new alter-group.md (+ sidebar): SET MEMORY LIMIT { <size> | UNLIMITED }.
• security/rbac.md: new "Memory limits" section — how limits resolve, what a per-principal limit does and does not cover, how to inspect limits, the SET MEMORY LIMIT permission, and ALTER GROUP in the command list.
• query/sql/show.md: the memory_limit column on SHOW USERS, SHOW GROUPS and SHOW GROUPS <user>, and SHOW SERVICE ACCOUNTS and SHOW SERVICE ACCOUNTS <user | group>.

Limit resolution — override, not min. Limits resolve by strict precedence: a principal's own limit → (users only) the most-restrictive limit among its groups → the global cairo.query.memory.limit.bytes workload limit. A 0/UNLIMITED value at a level means "not set" and falls through to the next. A more specific limit, when set, fully overrides the broader one and binds even when larger, so a per-user or per-group override can raise a principal's ceiling above the workload limit, not only lower it. Service accounts never inherit group limits. (Earlier drafts of these docs described a most-restrictive "min" behavior; revised per questdb/questdb-enterprise@fb4e050 "Make per-entity memory limits override, not min".)

Corrections made after checking against the implementation

Two statements in earlier revisions of this PR were wrong, and both were verified against the Enterprise code rather than re-reasoned:

• UPDATE was described as never picking up a per-principal limit. False for a non-WAL table. UpdateOperation.apply() passes the caller's own SqlExecutionContext to UpdateOperatorImpl.executeUpdate, which calls queryRegistry.register(...), which acquires a QUERY-workload tracker from that context's security context — so a non-WAL UPDATE is capped by SET MEMORY LIMIT. An operator sizing one from the old text would have met an unexpected OOM abort. Only the WAL path (the default table type) runs under the WAL apply job's tracker. Replaced with a "What a per-principal limit covers" section that splits the two.
• SHOW GROUPS <user> and SHOW SERVICE ACCOUNTS <user | group> were shown without a memory_limit column, with prose saying so explicitly. They carry it — AbstractShowLinkedEntitiesCursorFactory adds it to both metadata shapes, pinned by ShowAclTest. Since these are among the results that break a positional client, the omission hid half the breaking change.

Example tables that disagreed with each other on the same entity's limit were also made consistent.

Behaviour the docs previously did not mention at all

• COPY ... TO exports are capped by the issuing principal's limit. The export runs under that principal's own context on the query workload, so a limit must be sized for the largest single thing the principal runs, exports included — not only for its interactive queries.
• SET MEMORY LIMIT is self-escalating. It takes no entity name, so its holder can set any principal's limit including its own; combined with override-not-min, a non-admin holding it can raise its own ceiling above cairo.query.memory.limit.bytes. Documented as a warning, together with the GRANT ALL expansion — a principal granted ALL before the upgrade does not acquire the new permission.
• The breaking change. Five results gain a column on upgrade (SHOW USERS, SHOW GROUPS and SHOW GROUPS <user>, SHOW SERVICE ACCOUNTS and SHOW SERVICE ACCOUNTS <user | group>), plus SELECT * on sys.acl_entities. Positional clients must be updated; name-based clients are unaffected.
• sys.acl_entities stores the value, and is readable only by the built-in admin — a principal holding DATABASE ADMIN is still denied.
• memory_limit means different things per statement: the effective limit in SHOW USERS, the entity's own limit in SHOW GROUPS / SHOW SERVICE ACCOUNTS, since neither inherits one.
• An external (SSO/OIDC) user's inherited group limit refreshes at next login, not on its current session — the same refresh-on-login model that already governs group-granted permissions for external users.

Re-validation against the merged code

Both source PRs merged after the last revision of this doc PR (OSS on 2026-06-20, Enterprise on 2026-09-08), and OSS master moved on afterwards. Every claim was re-checked against merged Enterprise main and OSS master (questdb submodule pin fef82615), and these were stale:

• There are four workload limits, not three. Live views (feat(sql): add live views questdb#7461) added LIVE_VIEW_REFRESH as a workload with its own reloadable cairo.live.view.refresh.memory.limit.bytes. The live-views config page on main already documents the key, so the memory-limits section now points to it rather than duplicating it, and the intro, capacity-planning blurb, and the RBAC "not covered by a principal limit" list name live view refresh.
• A mat view refresh breach no longer invalidates the view on first breach. fix(core): retry mat view refresh on transient errors instead of invalidating questdb#7275 reclassified a refresh OOM as a transient, retried failure: the refresh is deferred by cairo.mat.view.refresh.busy.retry.timeout and the view is invalidated only after cairo.mat.view.refresh.busy.retry.limit consecutive failures (MatViewRefreshJob.isRetriableRefreshError, pinned by WorkloadMemoryTrackerTest). A live view refresh breach, by contrast, invalidates immediately (LiveViewRefreshJob). The breach paragraph is now a per-workload list.
• ALTER USER ... SET MEMORY LIMIT on an external (SSO/OIDC) user is rejected (AccessListModel.setMemoryLimit: Cannot set memory limit for external user). The docs previously only said such users inherit from groups; they now say the direct form is refused, on both rbac.md and alter-user.md.
• Coverage note trimmed. The earlier "coverage is best-effort" paragraph, with its list of tracked allocation sites, is replaced by the one fact an operator needs: memory-mapped memory, such as the column files a query reads, does not count toward a limit.
• WAL apply limit rarely fires. WAL apply runs only simple UPDATEs, metadata ALTERs, and commits, so the entry now says the limit mainly isolates WAL apply from the query budget.
• Upgrade note. The memory_limit column is added by an automatic migration (SysMig6) on the first primary boot or on promote; an ACL statement refused with a message naming the memory_limit column migration means the node is not yet migrated. The window normally closes once WAL apply catches up, so retry first; a persistent refusal calls for a restart, which re-runs the migration.
• query_activity's two memory columns are null for nested registrations, which share the outer query's budget.

Confirmed unchanged and still accurate: the SET MEMORY LIMIT grammar and permission name, override-not-min precedence and 0/UNLIMITED semantics (EntMemoryTrackerProvider.effectiveLimit), the memory_limit column on all five SHOW results including the filtered forms and the enabled/external_alias/grant_option neighbours (AbstractShowEntitiesCursorFactory, AbstractShowLinkedEntitiesCursorFactory, ShowAclTest), the COPY ... TO and non-WAL UPDATE coverage, the breach message format (Unsafe.checkPerQueryAllocLimit), reloadability of every limit (DynamicPropServerConfiguration), and the built-in admin being unlimitable.

Dependencies

Documents:

• feat(core): add per-query memory limits questdb#7184 — per-query / per-workload memory limits (OSS), merged
• questdb/questdb-enterprise#1039 — per-principal memory limits (Enterprise), merged

Web-console syntax highlighting for the new clause: questdb/sql-parser#27

Verification

yarn build, which this repo configures to throw on broken links, broken markdown links, and broken anchors.

Review follow-up

A level-2 /review-pr pass (six agents, findings verified against OSS fef82615 and Enterprise main) found no wrong technical claims. Its major findings are fixed in 4be3ffa: stale SHOW SERVICE ACCOUNTS tables on the grant and revoke assume-service-account pages, the OUT OF MEMORY error tag literal and the wal_tables() errorTag/errorMessage columns, the queryId mapping for materialized view refreshes, CREATE GROUP ... WITH EXTERNAL ALIAS, the two migration-window refusal messages, the RBAC page description, and the query_activity column order with the missing is_wal column.

[github-actions]

🚀 Build success!

Latest successful preview: https://preview-462--questdb-documentation.netlify.app/docs/

Commit SHA: 339231a

📦 Build generates a preview & updates the link on each commit.