From 8c906ac51cbbc6884546c3af167c26deeb0b0d1c Mon Sep 17 00:00:00 2001 From: victor Date: Wed, 9 Sep 2026 16:34:03 +0800 Subject: [PATCH 1/4] docs: document QuestDB Enterprise resource groups Add three pages and wire them into the sidebar. The concept page covers how a query is assigned to a group, what is managed, and how strong each of the four controls actually is: admission is an exact gate, CPU weight is a share that only bites under contention, the CPU cap is a rate over a short window, and memory limits are batched per worker. It also explains why CPU control is cooperative, what happens on a replica and after an internal fault, and what the feature costs when nothing competes. The operations page covers day-to-day use: quick start, requirements, the instance settings, group and mapping statements, policy parameters, six worked scenarios, the inspection functions, the per-group metrics, the errors clients see, and troubleshooting. The scenarios start with an instance that stops answering while CPU looks idle, which is the case resource groups address most directly, since a query that reaches a cooperative checkpoint releases its worker instead of holding it to completion. The configuration page documents the seven instance settings, including that the feature turns itself off rather than refusing to start when it was left at its default and an SQL pool is in legacy mode. Complete the query_activity() column list, document current_resource_group(), resource_groups() and resource_group_mappings() in the function reference, and add the resource group series to the metrics reference. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Nua5uVhTxq3uBnUD31b1aD --- documentation/concepts/resource-groups.md | 223 +++++++ .../configuration/resource-groups.md | 127 ++++ documentation/operations/logging-metrics.md | 105 +++- documentation/operations/resource-groups.md | 563 ++++++++++++++++++ documentation/query/functions/meta.md | 271 +++++++-- documentation/sidebars.js | 15 + 6 files changed, 1210 insertions(+), 94 deletions(-) create mode 100644 documentation/concepts/resource-groups.md create mode 100644 documentation/configuration/resource-groups.md create mode 100644 documentation/operations/resource-groups.md diff --git a/documentation/concepts/resource-groups.md b/documentation/concepts/resource-groups.md new file mode 100644 index 000000000..7a8418133 --- /dev/null +++ b/documentation/concepts/resource-groups.md @@ -0,0 +1,223 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Resource groups isolate query workloads inside one QuestDB instance. Learn how + a query is assigned to a group, and what admission, CPU weight, CPU caps and + memory limits actually guarantee. +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + Resource groups isolate competing query workloads inside a single QuestDB + instance. + + +A resource group is a named policy that limits what a set of principals may +consume while their queries run. One instance typically serves several workloads +at once: dashboards that must answer in milliseconds, an ad-hoc analyst, and a +nightly report that scans a year of data. When these workloads compete without +resource controls, the report can increase dashboard latency. + +Resource groups control four things at the query execution boundary: + +- **Admission** — how many queries a group may run at once, how many may wait, + and how long they may wait. +- **Weighted CPU** — the share of query CPU a group receives while groups + compete. +- **A CPU rate limit** — an absolute ceiling expressed as a percentage of + instance capacity. +- **Memory** — process and group budgets for tracked native query memory. + +The design is cooperative. QuestDB executes query work on shared worker pools, +and resource groups do not create one operating-system thread pool per group. A +query and all of its parallel tasks use the same resource group, while every +worker stays available to every group. + +## How a query is assigned to a group + +Assignment follows the authenticated principal, not the statement: + +1. A **direct mapping** on the user or service account wins. +2. Otherwise, for users only, QuestDB looks at the mappings of the ACL groups + the user belongs to and takes the highest `mapping_priority`. Ties go to the + most recently created mapping. +3. Otherwise the query runs in **DEFAULT**. + +Service accounts inherit nothing from ACL groups; they are either mapped +directly or they run in DEFAULT. + +The group is resolved once, when the query registers, and stays fixed for the +statement's lifetime. Changing a mapping affects statements that start after the +change, never one already running. + +`DEFAULT` always exists. By default it carries no limits of its own, so unmapped +principals run with a CPU weight of 100, no CPU cap, unlimited admission, and +the instance-wide memory limits. You can change its policy, but you cannot drop +or rename it. + +The default behaviour is: + +| Setting | Behaviour | +| ----------------------------------- | ------------------------------------------------------ | +| Feature enabled | Yes, when the SQL worker pools support Fiber execution | +| Group admission | Unlimited active and queued queries | +| Group CPU | Weight 100; no percentage cap | +| Group and process memory budgets | Unlimited unless configured | +| Existing single-query memory limits | Still apply, including principal-specific limits | +| Memory accounting without limits | Remains enabled for tracked native query memory | + +## What is managed + +Resource groups govern the statements that read data: + +- `SELECT` and `EXPLAIN` +- the source query of `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` +- query exports + +Everything else runs outside the feature and consumes no admission slot, CPU +grant or group memory budget: value `INSERT`, `UPDATE`, ordinary DDL, `COPY`, +transaction and session control, ILP ingestion, WAL apply, materialized and live +view refresh, and QuestDB's own internal SQL. + +For `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` the owner covers cursor +open, the source scan, transforms, parallel query work and the row pump. Source +evaluation and writer append are fused in that pump, so inseparable foreground +CPU may be charged conservatively to the group. Durability, the commit and any +work handed to writer or WAL queues are outside the guarantee. + +Resource groups account **tracked native query memory**. They do not represent +JVM heap, resident set size, memory-mapped table pages or long-lived engine +caches. Existing process memory protection remains the outer boundary. + +## What each control guarantees + +The four controls differ in how strong their guarantee is, which matters when +you decide what to configure. + +### Admission is a hard gate + +`max_active_queries` is an exact count. A group at its limit queues the next +query until a slot frees, up to `max_queued_queries`; beyond that the query is +rejected immediately. A queued query that waits longer than `queue_timeout` +fails. + +A slot is held only while the query is actually executing a segment. A protocol +cursor that is suspended between pages releases its slot and passes through the +gate again when the client asks for more rows, so a paging client does not hold +capacity while the application thinks. The consequence is that admission can be +refused on a later page: a client that received its first rows may still see the +queue-full or timeout error when it asks for more, and the connection stays +usable. + +### CPU weight is a share, not a reservation + +Weights only matter when groups compete. A group that is alone on the instance +uses everything it can, regardless of its weight. When two groups both have +work, the scheduler hands out CPU so that measured CPU divided by `cpu_weight` +stays balanced: weights 100 and 50 converge to a 2:1 split of query CPU. + +Weights are relative. 100 and 50 are the same as 2 and 1. A group that becomes +active starts level with the groups already running, so it neither banks the CPU +it did not use while idle nor is punished for having been busy. + +Within a group, pending execution requests are served in arrival order. A +parallel query can submit more than one request, so this does not promise equal +CPU shares between individual queries. + +### The CPU cap is a rate, not an instantaneous ceiling + +`cpu_max_percent` is enforced with a token bucket measured in CPU nanoseconds +against the instance's CPU capacity. It is an average over a short window, not a +per-instant limit: a capped group that has been idle may burst for about 100 ms +of accumulated allowance before it is pushed back to its configured rate. Usage +beyond a grant becomes debt that must be repaid before the group runs again, so +the average holds even when an individual query overruns. + +Capacity comes from `resource.groups.cpu.capacity.cores`, which detects +container CPU quota by default. On a fractional quota such as 500m, detection +preserves the fraction, so a 50% cap really means half of half a core. + +### Memory limits use batched accounting + +Accounting has three levels: query, group and process. Allocation and release +deltas accumulate locally on the executing worker and are published to the +shared counters at an adaptive threshold or an execution boundary. Exceeding a +checked limit fails the query with `query memory limit exceeded`; it does not +queue the allocation until memory becomes available. + +The single-query ceiling starts with the principal's effective query memory +limit, when set, or the instance default `cairo.query.memory.limit.bytes`. Any +group `memory_limit` and process memory budget further cap that ceiling. The +group budget also bounds the total tracked memory held by its queries; the +process budget covers tracked native query memory across groups. + +An unset group `memory_limit` adds no group ceiling. A process budget of `0` +adds no process ceiling. Existing single-query limits still apply, and memory +accounting remains enabled even when all limits are unlimited. + +The counters can temporarily omit worker-local deltas. A group can therefore +briefly overshoot its limit by a bounded amount related to the number of workers +running its queries. These budgets are not byte-exact, instantaneous ceilings. + +## Why CPU control is cooperative + +QuestDB does not preempt a running query. The scheduler grants a query a short +slice of CPU on a worker and expects it to reach a cooperative checkpoint, which +is the same circuit breaker check that makes queries cancellable. At that point +the query either renews its grant or yields the worker to another group. + +Yielding returns the worker to its dispatch loop rather than to the end of the +query. The loop interleaves other queries on that worker and, within a bounded +window, hands it back to network I/O so new connections are accepted. A long +single-threaded query that reaches these checkpoints can therefore share its +worker before finishing. This improves responsiveness while heavy queries run, +even before any custom group policy is written. Resource groups add this CPU +time slicing to the existing cancellation and I/O suspension mechanisms. + +Two consequences follow. + +The guarantee is statistical over a short window. Between checkpoints a query +holds its worker, so instantaneous CPU can deviate from the configured share. +Settlement charges the measured CPU either way, so a query that overran repays +it and the average is preserved. + +A query that cannot reach a checkpoint keeps its worker. When managed CPU +accounting is engaged, its CPU is charged when the grant settles, but no +cooperative limit can shorten that stretch. + +## Behaviour under failure and on replicas + +Resource groups are stored in a replicated system catalog, so a read-only +replica receives group definitions and mappings through normal replication. + +- A **fresh replica** that has not yet received the catalog runs queries + unmanaged, exactly as if the feature were disabled, and reports how many + queries took that path. It does not reject queries or serve them under a + policy it cannot see yet. +- If **CPU scheduling** hits an internal fault, it degrades: queries continue to + run without CPU grants, and the condition is visible in metrics until the + instance restarts. Admission and memory limits do not depend on CPU scheduling + and stay enforced. +- A **fault on one query** fences only that query's owner. Other queries and + other groups are unaffected, and the faulted segment is charged conservatively + rather than being dropped from the accounting. + +## Cost when nothing competes + +While a single uncapped group owns all running queries, dispatch takes a +lock-free path that avoids CPU sampling and weighted scheduling accounting. +Query registration, admission, cooperative checks and memory accounting still +run, so this does not imply the same cost as disabling the feature. Actual +overhead depends on the workload. Managed scheduling engages as soon as a second +group has work or a capped group is active, and disengages again when it does +not. The transition happens at the next dispatch boundary, not at a query +boundary, so a newly arriving group does not wait for a long query to finish +before its policy applies. + +## See also + +- [Configure and use resource groups](/docs/operations/resource-groups/) +- [Resource groups configuration](/docs/configuration/resource-groups/) +- [Role-based access control](/docs/security/rbac/) diff --git a/documentation/configuration/resource-groups.md b/documentation/configuration/resource-groups.md new file mode 100644 index 000000000..f87252777 --- /dev/null +++ b/documentation/configuration/resource-groups.md @@ -0,0 +1,127 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Configuration settings for QuestDB Enterprise resource groups, covering the + master switch, CPU capacity, memory ceiling, admission defaults and catalog + limits. +--- + +:::note + +Resource groups are [Enterprise](/enterprise/) only. + +::: + +[Resource groups](/docs/concepts/resource-groups/) isolate competing query +workloads inside one instance. These settings are instance-wide. The per-group +policy that decides admission, CPU share and memory budgets is set in SQL, not +here. See [Configure and use resource groups](/docs/operations/resource-groups/) +for those statements. + +None of these settings are reloadable: changing any of them requires a restart. + +Resource groups also require access control to be enabled (`acl.enabled=true`) +before principals can be mapped to a group, and every pool that executes SQL +must run in Fiber mode, which is the default. What happens when a pool is in +legacy mode depends on how the feature was turned on. Left at its default, it +turns itself off and logs an error naming the pool and the setting to change. +Asked for explicitly, it fails startup with the same error, because an explicit +request and a legacy pool cannot both be honoured. + +## General + +### resource.groups.enabled + +- **Default**: `true` +- **Reloadable**: no + +Master switch. When `false`, resource group admission, CPU scheduling and group +memory accounting are disabled. Group definitions and principal mappings remain +in the catalog, so turning the feature back on restores the policies that were +already there. + +Existing principal-specific and instance-default single-query memory limits +continue to apply when resource groups are disabled. + +Left unset, this resolves to `false` on an instance whose SQL pools are in +legacy mode, so upgrading such an instance does not turn the feature on and does +not stop the instance from starting. `SHOW PARAMETERS` then reports `false`, +which is the value that took effect. Set it to `true` explicitly and a legacy +pool becomes a startup error instead. + +### resource.groups.cpu.capacity.cores + +- **Default**: `auto` +- **Reloadable**: no + +The CPU capacity that `cpu_max_percent` is a percentage of, as `auto` or a +positive decimal number of cores. `auto` detects the process affinity mask and +the most restrictive cgroup quota, preserving fractional quotas such as `500m`, +so a 50% cap on half a core really means a quarter of a core. An explicit value +is capped by successful detection; if detection fails, the explicit value stands +and the failure is logged. + +When detection fails and no explicit value is set, capacity falls back to the +processor count and `questdb_resource_groups_cpu_capacity_fallback` reports `1`. + +### resource.groups.process.memory.limit.bytes + +- **Default**: `0` +- **Reloadable**: no + +Ceiling for tracked native query memory across all groups. `0` leaves the +instance without a process ceiling, which is the default. When set, it bounds +every group and every query, so no group policy can grant more than this. + +An unlimited process budget does not disable memory accounting or remove an +existing single-query limit. The group-level SQL parameter `memory_limit` uses +`RESET (memory_limit)` to clear its ceiling; it does not accept `0`. + +This is not a process RSS limit. It covers tracked query memory only, not JVM +heap, memory-mapped table pages or long-lived engine caches. + +### resource.groups.queue.timeout.millis + +- **Default**: `30000` +- **Reloadable**: no + +How long a queued query waits for an admission slot in a group that does not set +its own `queue_timeout`. A query that waits longer fails with +`Resource Group admission queue timeout`. + +## Catalog limits + +Group definitions and principal mappings live in a replicated system catalog. +These bounds limit the catalog size. Increase them if the deployment requires +more definitions or mappings. + +### resource.groups.catalog.max.snapshot.bytes + +- **Default**: `16777216` +- **Reloadable**: no + +Size ceiling for one serialized catalog snapshot. + +### resource.groups.max.user.groups + +- **Default**: `4096` +- **Reloadable**: no + +Maximum number of user-created resource groups, not counting `DEFAULT`. Dropped +groups stop counting towards this limit immediately, even while their existing +queries finish. + +### resource.groups.max.principal.links + +- **Default**: `65536` +- **Reloadable**: no + +Maximum number of principal mappings, counting users, service accounts and ACL +groups together. + +## See also + +- [Resource groups concept](/docs/concepts/resource-groups/) +- [Configure and use resource groups](/docs/operations/resource-groups/) +- [Identity and Access Management configuration](/docs/configuration/iam/) diff --git a/documentation/operations/logging-metrics.md b/documentation/operations/logging-metrics.md index 240647f3c..ce3e3bf94 100644 --- a/documentation/operations/logging-metrics.md +++ b/documentation/operations/logging-metrics.md @@ -1,10 +1,12 @@ --- title: Logging and metrics -description: Configure and understand QuestDB logging and metrics, including log levels, configuration options, and Prometheus integration. +description: + Configure and understand QuestDB logging and metrics, including log levels, + configuration options, and Prometheus integration. --- - -This page outlines logging in QuestDB. It covers how to configure logs via `log.conf` and expose metrics via Prometheus. +This page outlines logging in QuestDB. It covers how to configure logs via +`log.conf` and expose metrics via Prometheus. - [Logging](/docs/operations/logging-metrics/#logging) - [Metrics](/docs/operations/logging-metrics/#metrics) @@ -206,19 +208,20 @@ For configuration options, see the :::warning On systems with -[8 Cores and less](/docs/getting-started/capacity-planning/#cpu-cores), contention -for threads might increase the latency of health check service responses. If you -use a load balancer, and it thinks the QuestDB service is dead with nothing -apparent in the QuestDB logs, you may need to configure a dedicated thread pool -for the health check service. To do so, increase `http.min.worker.count` to `1`. +[8 Cores and less](/docs/getting-started/capacity-planning/#cpu-cores), +contention for threads might increase the latency of health check service +responses. If you use a load balancer, and it thinks the QuestDB service is dead +with nothing apparent in the QuestDB logs, you may need to configure a dedicated +thread pool for the health check service. To do so, increase +`http.min.worker.count` to `1`. ::: #### Lifecycle endpoint `GET /lifecycle` on the same port returns the startup and shutdown state of -every server component as JSON, for probes and coordinators that need more -than the `200` of the health check: +every server component as JSON, for probes and coordinators that need more than +the `200` of the health check: ```shell curl http://127.0.0.1:9003/lifecycle @@ -338,23 +341,23 @@ When [cold storage](/docs/concepts/cold-storage/) is enabled, the endpoint exposes fifteen additional metrics under the `questdb_cold_chunk_` prefix, covering the chunk cache and the range reads that serve remote partitions: -| Metric | Type | Description | -| ------ | ---- | ----------- | -| `questdb_cold_chunk_acquire_full_hit_total` | counter | Reads that found every chunk already resident | -| `questdb_cold_chunk_acquire_partial_hit_total` | counter | Reads that found some chunks and fetched the rest | -| `questdb_cold_chunk_acquire_full_miss_total` | counter | Reads where every chunk had to be fetched | -| `questdb_cold_chunk_acquire_hit_chunks_total` | counter | Chunk lookups served from the cache | -| `questdb_cold_chunk_acquire_miss_chunks_total` | counter | Chunk lookups that had to be fetched | -| `questdb_cold_chunk_download_started_total` | counter | Range requests dispatched, one per coalesced group | -| `questdb_cold_chunk_download_finished_total` | counter | Range requests that returned data | -| `questdb_cold_chunk_download_failed_total` | counter | Range requests that failed after retries | -| `questdb_cold_chunk_download_coalesced_total` | counter | Readers that attached to an in-flight download instead of starting a new one | -| `questdb_cold_chunk_release_evictions_total` | counter | Chunks evicted when their last lease was released | -| `questdb_cold_chunk_in_flight_downloads` | gauge | Range requests dispatched but not yet complete | -| `questdb_cold_chunk_pending_batches` | gauge | Batches the read coordinator is tracking | -| `questdb_cold_chunk_busy_leases` | gauge | Currently allocated leases | -| `questdb_cold_chunk_ready_chunks` | gauge | Chunks resident in the ready cache | -| `questdb_cold_chunk_pinned_bytes` | gauge | Compressed bytes resident in the ready cache | +| Metric | Type | Description | +| ---------------------------------------------- | ------- | ---------------------------------------------------------------------------- | +| `questdb_cold_chunk_acquire_full_hit_total` | counter | Reads that found every chunk already resident | +| `questdb_cold_chunk_acquire_partial_hit_total` | counter | Reads that found some chunks and fetched the rest | +| `questdb_cold_chunk_acquire_full_miss_total` | counter | Reads where every chunk had to be fetched | +| `questdb_cold_chunk_acquire_hit_chunks_total` | counter | Chunk lookups served from the cache | +| `questdb_cold_chunk_acquire_miss_chunks_total` | counter | Chunk lookups that had to be fetched | +| `questdb_cold_chunk_download_started_total` | counter | Range requests dispatched, one per coalesced group | +| `questdb_cold_chunk_download_finished_total` | counter | Range requests that returned data | +| `questdb_cold_chunk_download_failed_total` | counter | Range requests that failed after retries | +| `questdb_cold_chunk_download_coalesced_total` | counter | Readers that attached to an in-flight download instead of starting a new one | +| `questdb_cold_chunk_release_evictions_total` | counter | Chunks evicted when their last lease was released | +| `questdb_cold_chunk_in_flight_downloads` | gauge | Range requests dispatched but not yet complete | +| `questdb_cold_chunk_pending_batches` | gauge | Batches the read coordinator is tracking | +| `questdb_cold_chunk_busy_leases` | gauge | Currently allocated leases | +| `questdb_cold_chunk_ready_chunks` | gauge | Chunks resident in the ready cache | +| `questdb_cold_chunk_pinned_bytes` | gauge | Compressed bytes resident in the ready cache | Watch rates and ratios rather than raw totals. Sustained `download_failed_total`, `pending_batches` sitting at its configured cap, or @@ -369,10 +372,52 @@ _Enterprise only._ Two gauges describe the state of an in-place [role switch](/docs/high-availability/failover/): -| Metric | Type | Description | -| ------ | ---- | ----------- | +| Metric | Type | Description | +| ---------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `questdb_replication_pending_upload_txn` | gauge | Committed transactions not yet uploaded to the object store, summed over the replicated tables. Poll it before demoting a primary: a demote that cannot bring it to zero within its timeout is not completed | -| `questdb_backup_active_at_last_demote` | gauge | `1` if a backup was still running when the node was last demoted, `0` otherwise. Cleared by the next promotion | +| `questdb_backup_active_at_last_demote` | gauge | `1` if a backup was still running when the node was last demoted, `0` otherwise. Cleared by the next promotion | + +### Resource group metrics + +_Enterprise only._ + +When [resource groups](/docs/concepts/resource-groups/) are enabled, the +endpoint exposes one series per group, labelled with `resource_group`: + +| Metric | Type | Description | +| --------------------------------------------------- | ------- | ------------------------------------------------------------ | +| `questdb_resource_group_active_queries` | gauge | Queries holding an admission slot | +| `questdb_resource_group_queued_queries` | gauge | Queries waiting for a slot | +| `questdb_resource_group_oldest_queue_wait_millis` | gauge | How long the longest waiting query has waited | +| `questdb_resource_group_memory_bytes` | gauge | Tracked query memory in use | +| `questdb_resource_group_memory_limit_bytes` | gauge | Effective group memory ceiling, `0` when the group sets none | +| `questdb_resource_group_cpu_nanos_total` | counter | CPU charged to the group | +| `questdb_resource_group_cpu_wait_nanos_total` | counter | Time the group spent waiting for CPU | +| `questdb_resource_group_cpu_max_percent` | gauge | Effective CPU cap, `-1` when uncapped | +| `questdb_resource_group_admission_rejections_total` | counter | Queries rejected because the queue was full | +| `questdb_resource_group_admission_timeouts_total` | counter | Queries that timed out while queued | + +The single uncapped group dispatch path does not sample CPU. Consequently, +`cpu_nanos_total` counts CPU measured by managed scheduling, not every query's +CPU consumption. A flat counter does not imply that the group is idle; also +check `questdb_resource_groups_cpu_managed_dispatch` and query activity. Memory +gauges show published accounting and can lag worker-local deltas. + +Instance-wide series describe the feature itself: + +| Metric | Type | Description | +| ------------------------------------------------------- | ------- | --------------------------------------------------------------------- | +| `questdb_resource_groups_enabled` | gauge | `1` when the feature is on | +| `questdb_resource_groups_catalog_current` | gauge | `1` when the group catalog is current; `0` while a replica catches up | +| `questdb_resource_groups_catalog_lag_unmanaged_queries` | counter | Queries that ran unmanaged because the catalog was not current yet | +| `questdb_resource_groups_cpu_capacity_microcores` | gauge | Capacity that `cpu_max_percent` applies to | +| `questdb_resource_groups_cpu_capacity_fallback` | gauge | `1` when capacity detection failed and the processor count was used | +| `questdb_resource_groups_cpu_managed_dispatch` | gauge | `1` while managed CPU scheduling is engaged | +| `questdb_resource_groups_cpu_scheduler_degraded` | gauge | `1` when CPU scheduling has degraded to unmanaged | + +A non-zero `questdb_resource_groups_cpu_scheduler_degraded` means CPU shares are +no longer enforced until the instance restarts. Admission and memory limits stay +enforced. ### Prometheus Alertmanager diff --git a/documentation/operations/resource-groups.md b/documentation/operations/resource-groups.md new file mode 100644 index 000000000..3e8a07211 --- /dev/null +++ b/documentation/operations/resource-groups.md @@ -0,0 +1,563 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Create resource groups, map users and ACL groups to them, and tune admission, + CPU and memory limits so one workload cannot starve another. +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + Resource groups isolate competing query workloads inside a single QuestDB + instance. + + +This page covers day-to-day use: creating groups, mapping principals, choosing +limits, and watching the result. For what the limits actually guarantee, read +[the concept page](/docs/concepts/resource-groups/) first. + +## Quick start + +This example separates reporting from the default workload and limits its +concurrency and memory. Run it as an administrator on an instance that meets the +[requirements](#requirements). Use unused example names and replace the password +placeholders. Later examples on this page can be adapted independently. + +First create the ACL principals and allow SQL connections: + +```questdb-sql +CREATE GROUP analysts; +GRANT HTTP, PGWIRE TO analysts; +CREATE USER reporting_user WITH PASSWORD ''; +ADD USER reporting_user TO analysts; + +CREATE USER nightly_batch WITH PASSWORD ''; +GRANT HTTP, PGWIRE TO nightly_batch; +``` + +Then create the resource group and mappings: + +```questdb-sql +-- 1. Create a group. Unset parameters fall back to the instance defaults. +CREATE RESOURCE GROUP reporting WITH ( + cpu_weight = 50, + max_active_queries = 4, + max_queued_queries = 32, + queue_timeout = '15s', + memory_limit = '2G' +); + +-- 2. reporting_user inherits this mapping unless a higher-precedence one applies. +ALTER GROUP analysts SET RESOURCE GROUP reporting MAPPING PRIORITY 10; + +-- 3. Map one user directly. A direct mapping beats any ACL group mapping. +ALTER USER nightly_batch SET RESOURCE GROUP reporting; +``` + +Verify: + +```questdb-sql +SELECT name, cpu_weight, max_active_queries, active_queries, queued_queries +FROM resource_groups(); + +SELECT * FROM resource_group_mappings(); +``` + +Reconnect as `reporting_user` or `nightly_batch` and run: + +```questdb-sql +SELECT current_resource_group(); +``` + +| current_resource_group | +| ---------------------- | +| reporting | + +These grants allow connections. Grant access to the application's tables +separately, as described in [RBAC](/docs/security/rbac/). + +Everything not mapped keeps running in `DEFAULT`, which has a CPU weight of 100. +Against `reporting`'s weight of 50, that is a 2:1 split of query CPU while both +have work. + +## Requirements + +- QuestDB Enterprise. +- Access control enabled (`acl.enabled=true`). Groups can be created without it, + but mapping statements require it, since mappings attach to ACL principals. +- The pools that execute SQL must run in Fiber mode, which is the default, + because cooperative admission and CPU control cannot be made complete on the + legacy path. On an instance whose pools are in legacy mode, resource groups + left at their default turn themselves off and log an error naming the pool and + the setting to change. Setting `resource.groups.enabled=true` on such an + instance fails startup with that same error. +- Administrator rights for group management, mappings and instance-wide + inspection. Ordinary users can call `current_resource_group()` to check their + own query's group. + +## Configuration + +Resource groups are enabled by default. These are instance-wide settings; the +per-group policy is set in SQL. Each setting is described in full in the +[resource groups configuration reference](/docs/configuration/resource-groups/). + +| Property | Default | Meaning | +| -------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------- | +| `resource.groups.enabled` | `true` | Set to `false` to disable resource group enforcement. Existing single-query memory limits still apply. | +| `resource.groups.cpu.capacity.cores` | `auto` | CPU capacity that `cpu_max_percent` is a percentage of. `auto` detects container quota, including fractional quotas. | +| `resource.groups.process.memory.limit.bytes` | `0` | Ceiling for tracked query memory across all groups, `0` for none. Every group limit is capped by it. | +| `resource.groups.queue.timeout.millis` | `30000` | Default admission queue timeout for groups that do not set `queue_timeout`. | +| `resource.groups.catalog.max.snapshot.bytes` | `16777216` | Size ceiling for the serialized catalog. | +| `resource.groups.max.user.groups` | `4096` | Maximum number of user-created resource groups. | +| `resource.groups.max.principal.links` | `65536` | Maximum number of principal mappings. | + +Turning the feature off is a restart with `resource.groups.enabled=false`. +Definitions and mappings stay in the catalog, so nothing is lost and the +policies apply again when it is re-enabled. + +## Managing groups + +```questdb-sql +CREATE RESOURCE GROUP analytics; + +CREATE RESOURCE GROUP IF NOT EXISTS analytics WITH (cpu_weight = 300); + +ALTER RESOURCE GROUP analytics SET (cpu_weight = 300, cpu_max_percent = 25.5); + +-- Clear parameters so they fall back to the instance defaults again. +ALTER RESOURCE GROUP analytics RESET (memory_limit, cpu_max_percent); + +ALTER RESOURCE GROUP analytics RENAME TO reporting; + +DROP RESOURCE GROUP reporting; +DROP RESOURCE GROUP IF EXISTS reporting; +``` + +A group policy change applies online to the shared group budget. It does not +cancel existing queries at the moment `ALTER` runs: + +| Change | Effect on existing work | +| ---------------------- | ------------------------------------------------------------------------------------------------- | +| CPU weight or cap | Subsequent scheduling uses the new policy; issued CPU grants are settled normally | +| Active-query limit | Existing slots are retained; subsequent admission, including a resumed cursor, uses the new limit | +| Queue limit or timeout | New admission requests use the new settings; an already queued request keeps its deadline | +| Group memory limit | Subsequent allocations check the new budget; existing memory is released normally | + +Lowering a memory budget below current usage can make subsequent allocations +fail. The principal-specific or instance-default single-query limit is captured +when the query starts; updating the group budget does not replace that limit. +Changing a principal mapping affects new queries only. + +`DROP` is refused while any live principal is still mapped to the group; unmap +them first. Once unmapped, a group can be dropped while queries still use it. It +disappears from `resource_groups()` immediately. Running and queued queries, +including suspended cursors, continue using the deleted group's existing +settings. Their memory still counts towards the process budget. + +Recreating a group with the same name starts fresh usage counters. Queries that +still use the deleted group do not move to the new group or use its settings. +Map principals to the new group to assign their subsequent queries to it. + +`DEFAULT` cannot be dropped or renamed, but it can be altered: + +```questdb-sql +ALTER RESOURCE GROUP DEFAULT SET (max_active_queries = 16); +``` + +## Mapping principals + +```questdb-sql +ALTER USER alice SET RESOURCE GROUP analytics; +ALTER SERVICE ACCOUNT ingest_bot SET RESOURCE GROUP analytics; +ALTER GROUP analysts SET RESOURCE GROUP analytics MAPPING PRIORITY 10; + +ALTER USER alice UNSET RESOURCE GROUP; +ALTER GROUP analysts UNSET RESOURCE GROUP; +``` + +`MAPPING PRIORITY` is a non-negative integer and applies only to ACL group +mappings, because a user can belong to several ACL groups. The highest priority +wins; ties go to the most recent mapping. It defaults to 0 and is rejected on +user and service account mappings, which are one-to-one. + +Resolution order for a query is: direct mapping on the principal, then the +highest-priority mapping among the user's ACL groups, then `DEFAULT`. Service +accounts do not inherit ACL group mappings. + +## Policy parameters + +All parameters are optional. An unset parameter is not "unlimited" in every +case: it falls back to the instance default shown here. + +| Parameter | Accepted values | Unset behaviour | +| -------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | +| `cpu_weight` | integer, 1 to 10000 | 100 | +| `cpu_max_percent` | 0.01 to 100, at most two decimals | no cap | +| `max_active_queries` | integer, 1 or more | unlimited | +| `max_queued_queries` | integer, 0 or more | unlimited | +| `queue_timeout` | a positive whole number of milliseconds, or a duration such as `'15s'`, `'2m'` | `resource.groups.queue.timeout.millis` | +| `memory_limit` | a positive byte size, plain or suffixed such as `'8G'` | no group ceiling; other memory limits still apply | + +`memory_limit` is the budget for everything the group runs at once. Where the +instance sets `resource.groups.process.memory.limit.bytes`, the group budget is +capped by it, so a group cannot be granted more than the instance allows. A +group ceiling only lowers what its queries may use; it never raises a limit set +elsewhere. + +A group that does not set `memory_limit` carries no ceiling of its own, and +`resource_groups().memory_limit_bytes` reports `0` for it. Its queries are then +bounded by any existing single-query limit and the process limit. A principal's +effective query memory limit takes precedence over the instance default +`cairo.query.memory.limit.bytes`; group and process budgets can only lower the +resulting ceiling. Resource groups do not have a separate `query_memory_limit` +policy parameter. + +To remove a group memory ceiling, use +`ALTER RESOURCE GROUP reporting RESET (memory_limit)`. Setting the SQL parameter +to `0` is invalid; `0` means unlimited for the instance process-memory property. +Accounting continues when limits are unlimited. + +Two examples of what the values mean in practice: + +```questdb-sql +-- A share: reporting gets a third of query CPU when DEFAULT also has work, +-- and all of it when DEFAULT is idle. +CREATE RESOURCE GROUP reporting WITH (cpu_weight = 50); + +-- A ceiling: exports never average more than a quarter of instance CPU, +-- even when the instance is otherwise idle. +CREATE RESOURCE GROUP exports WITH (cpu_max_percent = 25); +``` + +Use `cpu_weight` to decide who wins under contention, and `cpu_max_percent` to +leave headroom for work that resource groups do not manage, such as ingestion +and WAL apply. Setting a cap on a group also switches the whole instance to +managed scheduling while that group has queries. + +## Common scenarios + +### The instance stops answering while CPU looks idle + +Every HTTP or PGWire worker is occupied by a long query, new requests are not +picked up, and instance CPU is low because those queries run on one core each. +Clients time out and retry, which produces more of the same queries. A plan such +as a `LATEST ON` over a non-indexed filter is a typical cause: it scans frames +on one thread, so it is slow without ever being CPU-hungry. + +Four steps. The first is a prerequisite to confirm, the second is what enabling +the feature already gives you, and the last two are policy you choose. + +**1. Confirm the SQL pools are Fiber pools.** This is the prerequisite for +everything below. A protocol runs either on its own pool, when its worker count +is above zero, or on the shared network pool. The setting that matters is the +one for the pool it actually uses: + +| Where the protocol runs | Setting to check | +| ---------------------------------------------------- | ------------------------------------- | +| Its own HTTP pool (`http.worker.count` above zero) | `http.worker.fiber.enabled` | +| Its own PGWire pool (`pg.worker.count` above zero) | `pg.worker.fiber.enabled` | +| The shared network pool (worker count zero, default) | `shared.network.worker.fiber.enabled` | + +Parallel query work is separate and follows `shared.query.worker.fiber.enabled` +whenever the shared query pool has workers. A shared query pool set to zero +workers turns parallel SQL off by default and needs no check of its own. + +The first two settings default to `true`, so a dedicated pool is a Fiber pool +unless someone turned it off. `shared.network.worker.fiber.enabled` defaults to +`true` exactly when HTTP or PGWire actually runs there, which is the case out of +the box because both worker counts default to zero. You normally have nothing to +change here; check these only when the instance was tuned by hand. + +After the restart, confirm the feature came up. `SHOW PARAMETERS` must report +`resource.groups.enabled` as `true`, and `questdb_resource_groups_enabled` must +be `1`. If a pool that executes SQL is in legacy mode and resource groups were +left at their default, the feature disables itself and logs the reason. An +explicit `resource.groups.enabled=true` fails startup in that configuration. + +**2. Enabling the feature already frees the workers.** A query yields its worker +at the checkpoints that already make it cancellable, so a long single-threaded +scan releases the worker while it is still running and the instance keeps +accepting connections. This needs no group and no policy, and it holds even when +every query resolves to `DEFAULT`. + +There is no separate switch to verify. Cooperative yielding is on exactly when +resource groups are on, which step 1 already confirmed. Fiber pools on their own +do not produce it: the checkpoints are compiled into every build, but they only +yield while resource groups are enabled. A query that never reaches a checkpoint +still holds its worker, so this does not remove every cause of an unresponsive +instance. + +**3. Separate the workloads so shares apply.** While a single uncapped group +owns every running query, dispatch stays on the unmanaged path and weights have +nothing to arbitrate. Two groups with queries in flight at the same time, or any +group with a `cpu_max_percent`, is what engages weighted scheduling: + +```questdb-sql +CREATE RESOURCE GROUP dashboards WITH (cpu_weight = 400); +CREATE RESOURCE GROUP adhoc WITH (cpu_weight = 100); + +ALTER USER app SET RESOURCE GROUP dashboards; +ALTER USER analyst SET RESOURCE GROUP adhoc; +``` + +**4. Bound concurrent requests with admission.** + +```questdb-sql +ALTER RESOURCE GROUP adhoc SET ( + max_active_queries = 4, + max_queued_queries = 8, + queue_timeout = '5s' +); +``` + +The fifth concurrent query waits instead of running, and it does not hold a +worker while it waits. The thirteenth fails immediately with +`Resource Group admission queue is full`, so a client that keeps resending gets +a clear answer in seconds instead of adding to the pile. + +Admission directly bounds the number of concurrent queries. A CPU cap bounds +their combined CPU rate and can also slow a single-threaded query: on an 8-core +instance, a 10% cap permits 0.8 cores of CPU. Choose a cap when that rate limit +is useful; it does not replace the admission limits in this scenario. Clients +should use bounded retries with backoff after admission failures. + +Afterwards the symptom is also diagnosable rather than mysterious. Low instance +CPU together with a high `queued_queries` and a rising +`oldest_queue_wait_millis` on one group says the work is being held at the +admission gate, not that the machine is busy. `query_activity()` shows which +group each running query was admitted to. + +What resource groups do not do here: they do not make the slow plan faster, and +they do not bound how long one query may run. Wall-clock limits still come from +the instance-wide +[`query.timeout`](/docs/configuration/cairo-engine/#querytimeout). + +### Dashboards must stay responsive while analysts run heavy queries + +Use weights. Shares are per group, not per query, so a group running fifty +queries does not outvote a group running one: + +```questdb-sql +CREATE RESOURCE GROUP dashboards WITH (cpu_weight = 400); +CREATE RESOURCE GROUP analysts WITH (cpu_weight = 100); +``` + +When these are the only competing groups and both can use their shares, weights +target a 4:1 split of managed query CPU. Actual use also depends on runnable +work, available parallelism and any CPU caps. When analysts are idle, dashboards +can use the available query CPU. Weights are integers from 1 to 10000 and every +group starts at 100, so a group left alone keeps an equal share against any +group you do not change. + +### A background job must never take the whole instance + +Use a cap, which applies whether or not anything else is running: + +```questdb-sql +CREATE RESOURCE GROUP exports WITH (cpu_max_percent = 20); +``` + +This also leaves headroom for work resource groups do not manage, such as +ingestion and WAL apply. The cap accepts two decimals, down to `0.01`, and is a +percentage of the +[detected CPU capacity](/docs/configuration/resource-groups/#resourcegroupscpucapacitycores), +not of the host's core count, so it stays correct under a container quota. + +### One workload must not exhaust query memory + +Bound the group rather than each query, so the limit holds however many queries +the workload starts: + +```questdb-sql +CREATE RESOURCE GROUP reporting WITH (memory_limit = '8G'); +``` + +A query that would push the group over its budget fails with +`query memory limit exceeded` and releases what it held. + +### An ingestion or automation account runs queries too + +Service accounts resolve differently from users: they honour a direct mapping, +but they never inherit a mapping from an ACL group. A service account with no +direct mapping runs in `DEFAULT` however its ACL groups are mapped, so map it +explicitly: + +```questdb-sql +CREATE RESOURCE GROUP automation WITH (cpu_weight = 50, max_active_queries = 2); + +ALTER SERVICE ACCOUNT ingest_bot SET RESOURCE GROUP automation; +``` + +This governs the queries the account runs. It does not throttle ingestion +itself, which resource groups do not manage. + +### Many teams share one instance + +Map ACL groups rather than individual users, and use `MAPPING PRIORITY` to +decide what happens to someone who belongs to more than one: + +```questdb-sql +ALTER GROUP analysts SET RESOURCE GROUP adhoc MAPPING PRIORITY 10; +ALTER GROUP oncall SET RESOURCE GROUP dashboards MAPPING PRIORITY 20; +``` + +Someone in both groups resolves to `dashboards`, because the higher priority +wins. If two mappings tie on priority, the more recently created one wins. A +direct mapping on the user beats every group mapping regardless of priority, +which is the way to make one person an exception without touching the groups: + +```questdb-sql +ALTER USER lead_analyst SET RESOURCE GROUP dashboards; +``` + +`MAPPING PRIORITY` is rejected on user and service account mappings, because +those are one-to-one and have nothing to break a tie between. Confirm any of +this from the client's own session with `SELECT current_resource_group();`. + +## Inspecting + +`resource_groups()` returns one row per group, combining the configured policy +with live counters: + +| Column | Meaning | +| ------------------------------------------------------------------ | --------------------------------------------------- | +| `name` | Group name | +| `memory_limit_bytes` | Effective group memory budget | +| `max_active_queries`, `max_queued_queries`, `queue_timeout_millis` | Effective admission policy | +| `cpu_weight`, `cpu_max_percent` | Effective CPU policy | +| `active_queries`, `queued_queries` | Live admission state | +| `oldest_queue_wait_millis` | How long the longest waiting query has waited | +| `memory_used_bytes` | Tracked query memory in use | +| `cpu_nanos_total`, `cpu_wait_nanos_total` | Cumulative CPU consumed and spent waiting for CPU | +| `admission_rejections`, `admission_timeouts` | Cumulative queue-full rejections and queue timeouts | + +`resource_group_mappings()` returns one row per mapping with `principal_type`, +`principal_name`, `principal_generation`, `resource_group_id`, `resource_group`, +`mapping_priority` and `mapping_revision`. + +`current_resource_group()` returns the calling query's group, which is the +quickest way to confirm a mapping from the client's own connection: + +```questdb-sql +SELECT current_resource_group(); +``` + +It returns `NULL` when that execution is unmanaged, including when the feature +is disabled or a replica's group catalog is not ready. See the +[function reference](/docs/query/functions/meta/#current_resource_group) for +permissions and return values, and the references for +[`resource_groups()`](/docs/query/functions/meta/#resource_groups) and +[`resource_group_mappings()`](/docs/query/functions/meta/#resource_group_mappings) +for complete schemas. + +`query_activity()` carries a `resource_group` column, so you can see which group +each running query was admitted to. It is `NULL` for executions that resource +groups do not manage: + +```questdb-sql +SELECT resource_group, username, query_start, query +FROM query_activity() +WHERE resource_group IS NOT NULL +ORDER BY query_start; +``` + +## Monitoring + +The Prometheus endpoint exposes one series per group, labelled with +`resource_group`. The full list lives in the +[metrics reference](/docs/operations/logging-metrics/#resource-group-metrics): + +``` +questdb_resource_group_active_queries{resource_group="reporting"} +questdb_resource_group_queued_queries{resource_group="reporting"} +questdb_resource_group_oldest_queue_wait_millis{resource_group="reporting"} +questdb_resource_group_memory_bytes{resource_group="reporting"} +questdb_resource_group_memory_limit_bytes{resource_group="reporting"} +questdb_resource_group_cpu_nanos_total{resource_group="reporting"} +questdb_resource_group_cpu_wait_nanos_total{resource_group="reporting"} +questdb_resource_group_cpu_max_percent{resource_group="reporting"} +questdb_resource_group_admission_rejections_total{resource_group="reporting"} +questdb_resource_group_admission_timeouts_total{resource_group="reporting"} +``` + +Instance-wide series: + +| Metric | Meaning | +| ------------------------------------------------------- | --------------------------------------------------------------------- | +| `questdb_resource_groups_enabled` | 1 when the feature is on | +| `questdb_resource_groups_catalog_current` | 1 when the catalog is current; 0 while a replica is still catching up | +| `questdb_resource_groups_catalog_lag_unmanaged_queries` | Queries that ran unmanaged because the catalog was not current yet | +| `questdb_resource_groups_cpu_capacity_microcores` | Capacity that `cpu_max_percent` applies to | +| `questdb_resource_groups_cpu_capacity_fallback` | 1 when capacity detection failed and the processor count was used | +| `questdb_resource_groups_cpu_managed_dispatch` | 1 while managed CPU scheduling is engaged | +| `questdb_resource_groups_cpu_scheduler_degraded` | 1 when CPU scheduling has degraded to unmanaged | + +Two signals are worth alerting on: a non-zero +`questdb_resource_groups_cpu_scheduler_degraded`, which means CPU shares are no +longer enforced until the next restart, and a steadily growing +`questdb_resource_group_admission_timeouts_total`, which means a group's queue +settings are rejecting work the application expects to succeed. + +## Errors clients see + +| Message | Cause | Usual fix | +| ----------------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `Resource Group admission queue is full` | The group is at `max_active_queries` and its queue is at `max_queued_queries` | Raise the limits, or let the client retry | +| `Resource Group admission queue timeout` | The query waited longer than `queue_timeout` | Raise `queue_timeout` or `max_active_queries`, or reduce concurrency | +| Either admission error while fetching a later page | A suspended cursor re-enters admission when the client asks for more rows | Adjust admission limits or retry the query with backoff; the failed cursor cannot continue | +| `query memory limit exceeded` | A single-query, group or process memory limit rejected an allocation | Inspect `query_activity().memory_limit` and the group/process budgets; reduce memory use or adjust the relevant limit | +| `Resource Group is referenced by an active principal link` | `DROP RESOURCE GROUP` while principals are still mapped | `UNSET RESOURCE GROUP` on those principals first | +| `built-in Resource Group cannot be dropped` / `cannot be renamed` | `DROP` or `RENAME` on `DEFAULT` | Alter it instead | + +## Troubleshooting + +**A group's CPU share is not what I configured.** Weights only apply while +groups compete. Check `active_queries` on both groups at the same moment: if one +is idle, the other is expected to use everything. Also confirm the work you are +watching is managed at all, since ingestion, WAL apply and view refresh are +outside the feature. `query_activity()` shows the group each running query +belongs to, which is the quickest way to tell whether the load you are watching +is attributed where you expect. + +**A capped group is slower than the cap suggests.** Very small caps release CPU +in pulses. The cap is a rate over roughly a 100 ms window, so a group whose +share works out to less than one 2 ms slice per window waits between slices. For +example, 0.1% of an 8-core instance allows about 8 ms of CPU per second. Small +caps still allow progress, but the waits between slices can substantially +increase latency; there is no special 0.25% cutoff. + +**Queries on a fresh replica are not limited.** Until the catalog has +replicated, a replica runs queries unmanaged and counts them in +`questdb_resource_groups_catalog_lag_unmanaged_queries`. The counter stops +growing once `questdb_resource_groups_catalog_current` reaches 1. + +**Startup fails naming a worker pool.** A pool that executes SQL is in legacy +mode while `resource.groups.enabled=true` was set explicitly. Either restore the +default Fiber mode for that pool or stop setting the property, which lets the +instance start with resource groups off. + +**The feature is off although the default is on.** Check the log at startup for +an error naming a worker pool, and check `SHOW PARAMETERS` for the value that +took effect. A legacy SQL pool turns the feature off when the property is left +unset. + +## Limitations + +- Only query statements are managed. See + [what is managed](/docs/concepts/resource-groups/#what-is-managed). +- Memory accounting covers tracked native query memory, not JVM heap, resident + set size or memory-mapped table pages. +- CPU control is cooperative, so shares hold over a short window rather than + instantaneously, and a query that cannot reach a cooperative checkpoint holds + its worker until it does. +- Principal mapping changes affect new queries. Group budgets change online; + dropping a group retains its runtime state for existing queries until they + finish. + +## See also + +- [Resource groups concept](/docs/concepts/resource-groups/) +- [Resource groups configuration](/docs/configuration/resource-groups/) +- [Role-based access control](/docs/security/rbac/) +- [Logging and metrics](/docs/operations/logging-metrics/) diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index d2156a88f..22c7a4003 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -61,9 +61,9 @@ SELECT current_data_id(); ## current database, schema, or user -`current_database()`, `current_schema()`, `current_user()`, and -`session_user()` are standard SQL functions that return information about the -current database, schema, and user. +`current_database()`, `current_schema()`, `current_user()`, and `session_user()` +are standard SQL functions that return information about the current database, +schema, and user. ```questdb-sql -- Get the current database @@ -86,6 +86,28 @@ statement without any arguments. and are interchangeable in QuestDB. Both report the user that authenticated on the current connection, whichever protocol it arrived on. +## current_resource_group + +_QuestDB Enterprise only._ + +Returns the resource group assigned to the calling query. Ordinary users can use +this function to check their own assignment; administrator rights are not +required. See [resource groups](/docs/concepts/resource-groups/) for mapping +precedence and the scope of managed execution. + +**Arguments:** none. + +**Return value:** `STRING`. Returns `NULL` when the execution is unmanaged, +including when resource groups are disabled or a replica's catalog is not ready. +A managed query without a principal mapping returns `DEFAULT`. + +```questdb-sql +SELECT current_resource_group(); +``` + +The result follows the query's acquired group, including across suspended cursor +pages. A subsequent mapping change affects the next query. + ## flush_query_cache() `flush_query_cache' invalidates cached query execution plans. @@ -314,7 +336,6 @@ materialized_views(); | trades_OHLC_15m | immediate | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | | trades_latest_1d | immediate | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | - ## memory_metrics **Arguments:** @@ -382,10 +403,10 @@ SELECT node_role(); :::warning `node_role()` cannot be used in a materialized view or a live view. Avoid it in -`UPDATE` on a WAL table as well: the statement is re-executed on every node of -a replicated cluster and each node evaluates its own role, so the primary and -its replicas would write different values. Tagging rows on `INSERT` is safe, -because inserted rows replicate as data. +`UPDATE` on a WAL table as well: the statement is re-executed on every node of a +replicated cluster and each node evaluates its own role, so the primary and its +replicas would write different values. Tagging rows on `INSERT` is safe, because +inserted rows replicate as data. ::: @@ -410,18 +431,33 @@ Returns metadata on running SQL queries, including columns such as: - state_change - timestamp of latest query state change, such as a cancellation - state - state of running query, can be `active` or `cancelled` - query - text of sql query +- is_wal - whether the query runs as part of WAL apply +- memory_used, memory_limit - tracked native memory the query holds and its + ceiling, `NULL` when no tracker is bound; `memory_limit` is also `NULL` when + the query has no ceiling +- resource_group - the [resource group](/docs/concepts/resource-groups/) the + query was admitted to in QuestDB Enterprise, `NULL` when resource groups do + not manage the execution **Examples:** ```questdb-sql -SELECT * FROM query_activity(); +SELECT query_id, worker_id, worker_pool, username, query_start, state_change, state, query +FROM query_activity(); ``` | query_id | worker_id | worker_pool | username | query_start | state_change | state | query | | -------- | --------- | ----------- | -------- | --------------------------- | --------------------------- | ------ | --------------------------------------------------------- | -| 62179 | 5 | shared | bob | 2024-01-09T10:03:05.557397Z | 2024-01-09T10:03:05.557397 | active | select \* from query_activity() | +| 62179 | 5 | shared | bob | 2024-01-09T10:03:05.557397Z | 2024-01-09T10:03:05.557397Z | active | SELECT count() FROM trades | | 57777 | 6 | shared | bob | 2024-01-09T08:58:55.988017Z | 2024-01-09T08:58:55.988017Z | active | SELECT symbol,approx_percentile(price, 50, 2) from trades | +To inspect query memory and resource group assignment in QuestDB Enterprise: + +```questdb-sql +SELECT query_id, username, resource_group, memory_used, memory_limit +FROM query_activity(); +``` + ## reader_pool **Arguments:** @@ -470,15 +506,94 @@ Edit `server.conf` and run `reload_config`: SELECT reload_config(); ``` +## resource_group_mappings + +_QuestDB Enterprise only. Requires administrator rights._ + +Returns the principal mappings in the resource group catalog. Definitions remain +available when resource group enforcement is disabled. + +**Arguments:** none. + +**Return value:** a table with these columns: + +| Column | Type | Description | +| ---------------------- | --------- | -------------------------------------------------------------------- | +| `principal_type` | `VARCHAR` | `USER`, `GROUP` or `SERVICE_ACCOUNT` | +| `principal_name` | `VARCHAR` | ACL principal name | +| `principal_generation` | `LONG` | Distinguishes a principal from a later recreation of the same name | +| `resource_group_id` | `LONG` | System-assigned identifier of the mapped resource group | +| `resource_group` | `VARCHAR` | Group name | +| `mapping_priority` | `INT` | Priority for ACL group mappings; defaults to `0` | +| `mapping_revision` | `LONG` | Revision used to break equal-priority ties; the higher revision wins | + +```questdb-sql +SELECT principal_type, principal_name, resource_group, mapping_priority +FROM resource_group_mappings() +ORDER BY principal_type, principal_name; +``` + +This lists mappings rather than expanding inherited assignments into one row per +user. Use `current_resource_group()` from a user's own session to confirm the +resolved assignment. + +## resource_groups + +_QuestDB Enterprise only. Requires administrator rights._ + +Returns one row per current catalog group, including `DEFAULT`, with resolved +policies and live counters. Group definitions remain visible when enforcement is +disabled; their runtime counters are zero. + +**Arguments:** none. + +**Return value:** a table with these columns: + +| Column | Type | Description | +| -------------------------- | --------- | ------------------------------------------------------------------------------------------------------- | +| `name` | `VARCHAR` | Group name | +| `memory_limit_bytes` | `LONG` | Effective group ceiling in bytes, capped by the process budget when enabled; `0` means no group ceiling | +| `max_active_queries` | `INT` | Concurrent admission limit; `2147483647` represents unlimited | +| `max_queued_queries` | `INT` | Queue capacity; `2147483647` represents unlimited, and `0` disables queueing | +| `queue_timeout_millis` | `LONG` | Effective admission timeout in milliseconds | +| `cpu_weight` | `INT` | Relative scheduling weight | +| `cpu_max_percent` | `DOUBLE` | CPU percentage cap; `NULL` when uncapped | +| `active_queries` | `LONG` | Queries currently holding admission slots | +| `queued_queries` | `LONG` | Queries waiting for admission | +| `oldest_queue_wait_millis` | `LONG` | Age of the oldest admission waiter in milliseconds; `0` when none | +| `memory_used_bytes` | `LONG` | Published tracked native query memory in bytes | +| `cpu_nanos_total` | `LONG` | CPU nanoseconds measured by managed scheduling | +| `cpu_wait_nanos_total` | `LONG` | Cumulative query waiting time for CPU, in nanoseconds | +| `admission_rejections` | `LONG` | Cumulative queue-full rejections | +| `admission_timeouts` | `LONG` | Cumulative admission timeouts | + +```questdb-sql +SELECT name, memory_limit_bytes, memory_used_bytes, active_queries, queued_queries +FROM resource_groups() +ORDER BY name; +``` + +Counters describe the current runtime and reset on restart or group recreation. +Worker-local memory deltas can be temporarily unpublished. The single uncapped +group dispatch path does not sample CPU, so `cpu_nanos_total` does not cover all +query CPU use. Dropped groups disappear from this table while their existing +queries finish using retained state. + +For the corresponding +[Prometheus metrics](/docs/operations/logging-metrics/#resource-group-metrics), +an uncapped CPU limit is represented by `-1`, whereas SQL returns `NULL`. An +unlimited group memory ceiling is `0` in both interfaces; it does not remove +principal-specific, instance-default single-query or process memory limits. + ## sleep() -Pauses the query for the given number of seconds, then returns the timestamp -at which it resumed. Intended for testing and demonstration, for example to -hold a query open while inspecting -[`query_activity()`](#query_activity) from another session. +Pauses the query for the given number of seconds, then returns the timestamp at +which it resumed. Intended for testing and demonstration, for example to hold a +query open while inspecting [`query_activity()`](#query_activity) from another +session. -`sleep()` does not hold a worker thread while it waits, so many concurrent -calls can be parked at once without exhausting the shared worker pool. +`sleep()` does not hold a worker thread while it waits, so many concurrent calls +can be parked at once without exhausting the shared worker pool. **Arguments:** @@ -502,8 +617,8 @@ SELECT * FROM sleep(1); :::note -Storage policies — and the `storage_policies` view — are available in -**QuestDB Enterprise** only. +Storage policies — and the `storage_policies` view — are available in **QuestDB +Enterprise** only. ::: @@ -532,9 +647,9 @@ SELECT * FROM storage_policies; - TTL values are rendered in two units: `h` for hours and `m` for **months**. Hour-, day-, and week-based durations are stored as hours (e.g. `3 DAYS` → `72h`, `1 WEEK` → `168h`). Month- and year-based durations are stored as - months (e.g. `1 MONTH` → `1m`, `1 YEAR` → `12m`). Despite the visual - collision with "minute", `m` in this view is **months**; QuestDB's duration - shorthand has no unit for minutes. + months (e.g. `1 MONTH` → `1m`, `1 YEAR` → `12m`). Despite the visual collision + with "minute", `m` in this view is **months**; QuestDB's duration shorthand + has no unit for minutes. - An unset stage renders as `0h`, not blank. **Example:** @@ -557,11 +672,15 @@ stage set and has been temporarily disabled. Every unset stage renders as `0h`. :::note -[Cold storage](/docs/concepts/cold-storage/) and the `table_cold_partitions()` function are available in **QuestDB Enterprise** only. +[Cold storage](/docs/concepts/cold-storage/) and the `table_cold_partitions()` +function are available in **QuestDB Enterprise** only. ::: -`table_cold_partitions('tableName')` returns one row per partition in the table's remote manifest, with the state of its object in the store. Use it to follow a partition through upload and sealing, and to find partitions that are not progressing. +`table_cold_partitions('tableName')` returns one row per partition in the +table's remote manifest, with the state of its object in the store. Use it to +follow a partition through upload and sealing, and to find partitions that are +not progressing. **Arguments:** @@ -609,9 +728,16 @@ WHERE state = 'pending'; **Notes:** -- The cold storage manager answers from its own in-memory view. A refresher answers from its mirrored copy, which it updates when the catalog generation changes, so the two can differ briefly. -- While an instance is transitioning between the manager and refresher roles, the function returns zero rows rather than blocking. Check the live role with [`SWITCH COLD STORAGE STATUS`](/docs/query/sql/switch-cold-storage-role/). -- The function reflects the remote manifest, not local partition state. Use [`SHOW PARTITIONS`](/docs/query/sql/show/#show-partitions) or [`table_partitions()`](#table_partitions) to see whether a partition is actually being served remotely. +- The cold storage manager answers from its own in-memory view. A refresher + answers from its mirrored copy, which it updates when the catalog generation + changes, so the two can differ briefly. +- While an instance is transitioning between the manager and refresher roles, + the function returns zero rows rather than blocking. Check the live role with + [`SWITCH COLD STORAGE STATUS`](/docs/query/sql/switch-cold-storage-role/). +- The function reflects the remote manifest, not local partition state. Use + [`SHOW PARTITIONS`](/docs/query/sql/show/#show-partitions) or + [`table_partitions()`](#table_partitions) to see whether a partition is + actually being served remotely. ## table_columns @@ -634,14 +760,14 @@ Returns a `table` with the following columns: - `symbolCached` - whether this `symbol` column is cached - `symbolCapacity` - how many distinct values this column of `symbol` type is expected to have -- `symbolTableSize` - current number of distinct values stored in this - `symbol` column's table +- `symbolTableSize` - current number of distinct values stored in this `symbol` + column's table - `designated` - if this is set as the designated timestamp column for this table - `upsertKey` - if this column is a part of UPSERT KEYS list for table [deduplication](/docs/concepts/deduplication) -- `indexType` - the [index type](/docs/concepts/deep-dive/indexes/) - (`POSTING`, `POSTING DELTA`, `POSTING EF`, `BITMAP`, or empty) +- `indexType` - the [index type](/docs/concepts/deep-dive/indexes/) (`POSTING`, + `POSTING DELTA`, `POSTING EF`, `BITMAP`, or empty) - `indexInclude` - comma-separated names of columns included in a [posting index's](/docs/concepts/deep-dive/posting-index/) covering sidecar @@ -719,16 +845,16 @@ Returns a table with the following columns: partition will contain the `.detached` extension) - `attachable` - _BOOLEAN_, true if the partition is detached and can be attached (`name` of the partition will contain the `.attachable` extension) -- `hasParquetGenerated` - _BOOLEAN_, true if a Parquet copy of the partition - has been generated. Set by either +- `hasParquetGenerated` - _BOOLEAN_, true if a Parquet copy of the partition has + been generated. Set by either [manual Parquet conversion](/docs/concepts/parquet/#in-place-conversion) (`ALTER TABLE ... CONVERT PARTITION TO PARQUET`) or by a [storage policy](/docs/concepts/storage-policy/)'s `TO PARQUET` stage (Enterprise) - `isParquet` - _BOOLEAN_, true if the partition is stored in Parquet format: - the native files have been removed and reads are served from the Parquet - file. Set the same way as `hasParquetGenerated`: either manually or by a - storage policy's `TO PARQUET` stage + the native files have been removed and reads are served from the Parquet file. + Set the same way as `hasParquetGenerated`: either manually or by a storage + policy's `TO PARQUET` stage - `parquetFileSize` - _LONG_, size in bytes of the partition's `data.parquet` file when `hasParquetGenerated` or `isParquet` is true; `-1` otherwise - `seqTxn` - _LONG_, WAL transaction version the partition was last written at @@ -889,7 +1015,7 @@ Returns a `table` with the following columns: ::: -### Table metrics (table_* prefix) +### Table metrics (table\_\* prefix) | Column | Type | Description | | ----------------------------- | --------- | ----------------------------------------------------------------------------------------- | @@ -912,16 +1038,18 @@ Returns a `table` with the following columns: | `table_merge_rate_p99` | LONG | Throughput that 99% of jobs **exceeded** (slowest 1%) | | `table_merge_rate_max` | LONG | Maximum throughput in rows/second | -Write amplification measures O3 (out-of-order) merge overhead as `physicalRowsWritten / logicalRows`. -A ratio of `1.0` means no amplification. Higher values indicate O3 merge overhead. +Write amplification measures O3 (out-of-order) merge overhead as +`physicalRowsWritten / logicalRows`. A ratio of `1.0` means no amplification. +Higher values indicate O3 merge overhead. :::note -Merge rate P99 shows the *lowest* throughput (worst performance), not the highest. +Merge rate P99 shows the _lowest_ throughput (worst performance), not the +highest. ::: -### WAL metrics (wal_* prefix) +### WAL metrics (wal\_\* prefix) | Column | Type | Description | | --------------------------------- | --------- | ------------------------------------------------------------- | @@ -935,9 +1063,10 @@ Merge rate P99 shows the *lowest* throughput (worst performance), not the highes | `wal_tx_size_p99` | LONG | 99th percentile transaction size in rows | | `wal_tx_size_max` | LONG | Maximum transaction size in rows | -### Replica metrics (replica_* prefix) +### Replica metrics (replica\_\* prefix) -These columns are populated on **replicas only** via replication download tracking: +These columns are populated on **replicas only** via replication download +tracking: | Column | Type | Description | | ------------------------ | ------- | ------------------------------------------------------------------------ | @@ -954,17 +1083,31 @@ On primary instances, these columns will be `0` or `false`. These values are approximations, not precise real-time metrics: -- **Null when not tracked**: Values are `null` for tables not written to since server start, or evicted from the tracker -- **Writer stats updated on pool return**: `table_row_count`, `table_last_write_timestamp`, `table_txn` are captured when TableWriter returns to the pool, not on every commit. A writer held for a long time won't update these columns until released. +- **Null when not tracked**: Values are `null` for tables not written to since + server start, or evicted from the tracker +- **Writer stats updated on pool return**: `table_row_count`, + `table_last_write_timestamp`, `table_txn` are captured when TableWriter + returns to the pool, not on every commit. A writer held for a long time won't + update these columns until released. - **WAL stats updated in real-time**: - - On WAL commit: `wal_pending_row_count` (incremented), `wal_txn`, `wal_max_timestamp`, `wal_tx_size_*` histogram - - On WAL apply: `wal_pending_row_count` (decremented), `wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, `table_write_amp_*`, `table_merge_rate_*` -- **LRU eviction**: Tracker maintains bounded memory (default 1000 tables). Least recently written tables are evicted when capacity is exceeded -- **Startup hydration**: Values are hydrated from table metadata (`TxReader`) on startup, but diverge as writes occur - -**Non-WAL tables**: `wal_txn`, `wal_max_timestamp`, `wal_pending_row_count`, `wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, `table_memory_pressure_level`, and histogram columns are `null` or `0`. - -**WAL tables**: All columns populated when tracked. `wal_max_timestamp` reflects the max data timestamp from the WAL transaction, not wall-clock time. `table_min_timestamp` and `table_max_timestamp` reflect the actual data range in the table after WAL merge. + - On WAL commit: `wal_pending_row_count` (incremented), `wal_txn`, + `wal_max_timestamp`, `wal_tx_size_*` histogram + - On WAL apply: `wal_pending_row_count` (decremented), + `wal_dedup_row_count_since_start`, `table_min_timestamp`, + `table_max_timestamp`, `table_write_amp_*`, `table_merge_rate_*` +- **LRU eviction**: Tracker maintains bounded memory (default 1000 tables). + Least recently written tables are evicted when capacity is exceeded +- **Startup hydration**: Values are hydrated from table metadata (`TxReader`) on + startup, but diverge as writes occur + +**Non-WAL tables**: `wal_txn`, `wal_max_timestamp`, `wal_pending_row_count`, +`wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, +`table_memory_pressure_level`, and histogram columns are `null` or `0`. + +**WAL tables**: All columns populated when tracked. `wal_max_timestamp` reflects +the max data timestamp from the WAL transaction, not wall-clock time. +`table_min_timestamp` and `table_max_timestamp` reflect the actual data range in +the table after WAL merge. ### Configuration @@ -1206,9 +1349,8 @@ concurrent waiters is not bounded by the shared worker pool. **Arguments:** -- `tableName` (`string`): name of the table to wait for. Must be a constant, - not a column reference. On a non-WAL table the call returns `true` - immediately. +- `tableName` (`string`): name of the table to wait for. Must be a constant, not + a column reference. On a non-WAL table the call returns `true` immediately. - `seqTxn` (optional, `long`): the sequencer transaction to wait for. When omitted, the call captures the table's current `seqTxn` when it starts and waits for that, which is what you want after your own write. @@ -1218,8 +1360,8 @@ concurrent waiters is not bounded by the shared worker pool. Returns `boolean`. `true` once the writer has caught up. Throws if the table is dropped while the call is waiting, and if the table -becomes [suspended](/docs/query/sql/alter-table-resume-wal/), since a -suspended table would otherwise never catch up. +becomes [suspended](/docs/query/sql/alter-table-resume-wal/), since a suspended +table would otherwise never catch up. **Examples:** @@ -1243,11 +1385,10 @@ SELECT wait_wal_table('trades', 42); :::note -For monitoring and observability, use [`tables()`](#tables) instead. -`tables()` provides all the same information plus additional metrics -(pending rows, memory pressure, deduplication stats, throughput histograms), -and is fully in-memory. `wal_tables()` reads from disk and is less suitable -for frequent polling. +For monitoring and observability, use [`tables()`](#tables) instead. `tables()` +provides all the same information plus additional metrics (pending rows, memory +pressure, deduplication stats, throughput histograms), and is fully in-memory. +`wal_tables()` reads from disk and is less suitable for frequent polling. ::: @@ -1265,11 +1406,13 @@ Returns a `table` including the following information: - `name` - table or materialized view name - `suspended` - suspended status flag -- `writerTxn` - the last committed transaction in TableWriter (equivalent to `table_txn` in `tables()`) +- `writerTxn` - the last committed transaction in TableWriter (equivalent to + `table_txn` in `tables()`) - `writerLagTxnCount` - the number of transactions that are kept invisible when writing to the table; these transactions will be eventually moved to the table data and become visible for readers (equivalent to `wal_txn - table_txn`) -- `sequencerTxn` - the last committed transaction in the sequencer (equivalent to `wal_txn` in `tables()`) +- `sequencerTxn` - the last committed transaction in the sequencer (equivalent + to `wal_txn` in `tables()`) **Examples:** diff --git a/documentation/sidebars.js b/documentation/sidebars.js index f0b4f587f..a324b8b78 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -635,6 +635,11 @@ module.exports = { type: "doc", label: "Cold Storage", }, + { + id: "concepts/resource-groups", + type: "doc", + label: "Resource Groups", + }, "concepts/write-ahead-log", ], }, @@ -698,6 +703,11 @@ module.exports = { "configuration/postgres-wire-protocol", "configuration/qwp", "configuration/database-replication", + { + id: "configuration/resource-groups", + type: "doc", + label: "Resource groups", + }, "configuration/shared-workers", "configuration/storage-policy", "configuration/telemetry", @@ -787,6 +797,11 @@ module.exports = { type: "doc", label: "Cold storage", }, + { + id: "operations/resource-groups", + type: "doc", + label: "Resource groups", + }, "operations/logging-metrics", "operations/monitoring-alerting", "operations/data-retention", From b5f4f4df5a348907379adf4ab3f30f52a0cf6d91 Mon Sep 17 00:00:00 2001 From: victor Date: Thu, 10 Sep 2026 11:25:15 +0800 Subject: [PATCH 2/4] update docs --- documentation/concepts/resource-groups.md | 12 ++++--- .../configuration/resource-groups.md | 33 +------------------ documentation/operations/resource-groups.md | 15 ++++----- 3 files changed, 15 insertions(+), 45 deletions(-) diff --git a/documentation/concepts/resource-groups.md b/documentation/concepts/resource-groups.md index 7a8418133..fa90633d8 100644 --- a/documentation/concepts/resource-groups.md +++ b/documentation/concepts/resource-groups.md @@ -72,14 +72,18 @@ The default behaviour is: Resource groups govern the statements that read data: -- `SELECT` and `EXPLAIN` +- `SELECT` - the source query of `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` - query exports Everything else runs outside the feature and consumes no admission slot, CPU -grant or group memory budget: value `INSERT`, `UPDATE`, ordinary DDL, `COPY`, -transaction and session control, ILP ingestion, WAL apply, materialized and live -view refresh, and QuestDB's own internal SQL. +grant or group memory budget: `EXPLAIN`, value `INSERT`, `UPDATE`, ordinary DDL, +`COPY`, transaction and session control, ILP ingestion, WAL apply, materialized +and live view refresh, and QuestDB's own internal SQL. + +`EXPLAIN` is deliberately outside the feature. It walks a plan tree and opens no +base cursor, so it reads no data, and holding an admission slot for it would +block the one statement an operator reaches for while a group is saturated. For `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` the owner covers cursor open, the source scan, transforms, parallel query work and the row pump. Source diff --git a/documentation/configuration/resource-groups.md b/documentation/configuration/resource-groups.md index f87252777..002050f7a 100644 --- a/documentation/configuration/resource-groups.md +++ b/documentation/configuration/resource-groups.md @@ -3,8 +3,7 @@ title: Resource groups sidebar_label: Resource groups description: Configuration settings for QuestDB Enterprise resource groups, covering the - master switch, CPU capacity, memory ceiling, admission defaults and catalog - limits. + master switch, CPU capacity, memory ceiling and admission defaults. --- :::note @@ -90,36 +89,6 @@ How long a queued query waits for an admission slot in a group that does not set its own `queue_timeout`. A query that waits longer fails with `Resource Group admission queue timeout`. -## Catalog limits - -Group definitions and principal mappings live in a replicated system catalog. -These bounds limit the catalog size. Increase them if the deployment requires -more definitions or mappings. - -### resource.groups.catalog.max.snapshot.bytes - -- **Default**: `16777216` -- **Reloadable**: no - -Size ceiling for one serialized catalog snapshot. - -### resource.groups.max.user.groups - -- **Default**: `4096` -- **Reloadable**: no - -Maximum number of user-created resource groups, not counting `DEFAULT`. Dropped -groups stop counting towards this limit immediately, even while their existing -queries finish. - -### resource.groups.max.principal.links - -- **Default**: `65536` -- **Reloadable**: no - -Maximum number of principal mappings, counting users, service accounts and ACL -groups together. - ## See also - [Resource groups concept](/docs/concepts/resource-groups/) diff --git a/documentation/operations/resource-groups.md b/documentation/operations/resource-groups.md index 3e8a07211..f1604f9bb 100644 --- a/documentation/operations/resource-groups.md +++ b/documentation/operations/resource-groups.md @@ -102,15 +102,12 @@ Resource groups are enabled by default. These are instance-wide settings; the per-group policy is set in SQL. Each setting is described in full in the [resource groups configuration reference](/docs/configuration/resource-groups/). -| Property | Default | Meaning | -| -------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------- | -| `resource.groups.enabled` | `true` | Set to `false` to disable resource group enforcement. Existing single-query memory limits still apply. | -| `resource.groups.cpu.capacity.cores` | `auto` | CPU capacity that `cpu_max_percent` is a percentage of. `auto` detects container quota, including fractional quotas. | -| `resource.groups.process.memory.limit.bytes` | `0` | Ceiling for tracked query memory across all groups, `0` for none. Every group limit is capped by it. | -| `resource.groups.queue.timeout.millis` | `30000` | Default admission queue timeout for groups that do not set `queue_timeout`. | -| `resource.groups.catalog.max.snapshot.bytes` | `16777216` | Size ceiling for the serialized catalog. | -| `resource.groups.max.user.groups` | `4096` | Maximum number of user-created resource groups. | -| `resource.groups.max.principal.links` | `65536` | Maximum number of principal mappings. | +| Property | Default | Meaning | +| -------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------- | +| `resource.groups.enabled` | `true` | Set to `false` to disable resource group enforcement. Existing single-query memory limits still apply. | +| `resource.groups.cpu.capacity.cores` | `auto` | CPU capacity that `cpu_max_percent` is a percentage of. `auto` detects container quota, including fractional quotas. | +| `resource.groups.process.memory.limit.bytes` | `0` | Ceiling for tracked query memory across all groups, `0` for none. Every group limit is capped by it. | +| `resource.groups.queue.timeout.millis` | `30000` | Default admission queue timeout for groups that do not set `queue_timeout`. | Turning the feature off is a restart with `resource.groups.enabled=false`. Definitions and mappings stay in the catalog, so nothing is lost and the From 4cc0d2df42196fbe3c9765f1af8130220d581ce6 Mon Sep 17 00:00:00 2001 From: victor Date: Thu, 10 Sep 2026 14:16:23 +0800 Subject: [PATCH 3/4] update docs --- documentation/concepts/resource-groups.md | 12 ++++++++++++ documentation/configuration/resource-groups.md | 5 +++++ documentation/operations/resource-groups.md | 18 ++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/documentation/concepts/resource-groups.md b/documentation/concepts/resource-groups.md index fa90633d8..c8fed80aa 100644 --- a/documentation/concepts/resource-groups.md +++ b/documentation/concepts/resource-groups.md @@ -200,6 +200,18 @@ replica receives group definitions and mappings through normal replication. unmanaged, exactly as if the feature were disabled, and reports how many queries took that path. It does not reject queries or serve them under a policy it cannot see yet. +- A **replica being promoted** validates the catalog after replication has + switched and before writes are admitted. If the old primary predated resource + groups and never created the catalog table, the promoted node creates it and + continues. With the feature enabled, a catalog that is unreadable or still + lagging refuses the promotion: the switch fails part-way, the node lands in + the `UNKNOWN` role and keeps serving reads as before, and the failure reason + starts with `RESOURCE_GROUP_CATALOG_UNAVAILABLE` or + `RESOURCE_GROUP_CATALOG_LAGGING`. Retrying the switch repeats the check. With + the feature disabled the condition is logged and the promotion proceeds. +- At **startup** an unreadable catalog stops an instance with the feature + enabled from starting, in either role. A lagging catalog does not: the + instance starts and the refresh job catches up. - If **CPU scheduling** hits an internal fault, it degrades: queries continue to run without CPU grants, and the condition is visible in metrics until the instance restarts. Admission and memory limits do not depend on CPU scheduling diff --git a/documentation/configuration/resource-groups.md b/documentation/configuration/resource-groups.md index 002050f7a..d2cbca800 100644 --- a/documentation/configuration/resource-groups.md +++ b/documentation/configuration/resource-groups.md @@ -40,6 +40,11 @@ memory accounting are disabled. Group definitions and principal mappings remain in the catalog, so turning the feature back on restores the policies that were already there. +`true` also makes the catalog a hard dependency: an instance whose catalog +cannot be read does not start, and a replica whose catalog is not current is not +promoted. With `false`, both conditions are logged and ignored. See +[Behaviour under failure and on replicas](/docs/concepts/resource-groups/#behaviour-under-failure-and-on-replicas). + Existing principal-specific and instance-default single-query memory limits continue to apply when resource groups are disabled. diff --git a/documentation/operations/resource-groups.md b/documentation/operations/resource-groups.md index f1604f9bb..759f38ea8 100644 --- a/documentation/operations/resource-groups.md +++ b/documentation/operations/resource-groups.md @@ -529,6 +529,24 @@ replicated, a replica runs queries unmanaged and counts them in `questdb_resource_groups_catalog_lag_unmanaged_queries`. The counter stops growing once `questdb_resource_groups_catalog_current` reaches 1. +**Promotion fails naming the resource group catalog.** With the feature enabled, +`SWITCH ROLE TO PRIMARY` does not admit writes over a catalog that is unreadable +or lagging. The node lands in the `UNKNOWN` role and still serves reads; +`GET /lifecycle` and the server log carry a reason that starts with +`RESOURCE_GROUP_CATALOG_LAGGING` or `RESOURCE_GROUP_CATALOG_UNAVAILABLE`. +Lagging means the replica has not finished applying the transactions the catalog +depends on: wait for WAL apply to catch up and run `SWITCH ROLE TO PRIMARY` +again. Unavailable means the catalog table is missing or its contents cannot be +read, and retrying does not help: promote another replica, or restart this node +as primary with `resource.groups.enabled=false`, which turns the check into a +logged error. See +[Refusals and the torn state](/docs/high-availability/failover/#refusals-and-the-torn-state). + +**Startup fails naming the resource group catalog.** The catalog table cannot be +read while the feature is enabled; the log says +`Resource Group catalog startup validation failed`. Starting with +`resource.groups.enabled=false` logs the condition instead of failing. + **Startup fails naming a worker pool.** A pool that executes SQL is in legacy mode while `resource.groups.enabled=true` was set explicitly. Either restore the default Fiber mode for that pool or stop setting the property, which lets the From 8d2f7345f010ee66651f3201840ea75c99eedc97 Mon Sep 17 00:00:00 2001 From: victor Date: Thu, 10 Sep 2026 17:32:08 +0800 Subject: [PATCH 4/4] update docs --- documentation/concepts/resource-groups.md | 8 ++++---- documentation/operations/resource-groups.md | 17 ++++++++--------- 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/documentation/concepts/resource-groups.md b/documentation/concepts/resource-groups.md index c8fed80aa..9466ca879 100644 --- a/documentation/concepts/resource-groups.md +++ b/documentation/concepts/resource-groups.md @@ -205,10 +205,10 @@ replica receives group definitions and mappings through normal replication. groups and never created the catalog table, the promoted node creates it and continues. With the feature enabled, a catalog that is unreadable or still lagging refuses the promotion: the switch fails part-way, the node lands in - the `UNKNOWN` role and keeps serving reads as before, and the failure reason - starts with `RESOURCE_GROUP_CATALOG_UNAVAILABLE` or - `RESOURCE_GROUP_CATALOG_LAGGING`. Retrying the switch repeats the check. With - the feature disabled the condition is logged and the promotion proceeds. + the `UNKNOWN` role and keeps serving reads as before, and the log names + `RESOURCE_GROUP_CATALOG_UNAVAILABLE` or `RESOURCE_GROUP_CATALOG_LAGGING`. + Retrying the switch repeats the check. With the feature disabled the condition + is logged and the promotion proceeds. - At **startup** an unreadable catalog stops an instance with the feature enabled from starting, in either role. A lagging catalog does not: the instance starts and the refresh job catches up. diff --git a/documentation/operations/resource-groups.md b/documentation/operations/resource-groups.md index 759f38ea8..e44999e16 100644 --- a/documentation/operations/resource-groups.md +++ b/documentation/operations/resource-groups.md @@ -531,15 +531,14 @@ growing once `questdb_resource_groups_catalog_current` reaches 1. **Promotion fails naming the resource group catalog.** With the feature enabled, `SWITCH ROLE TO PRIMARY` does not admit writes over a catalog that is unreadable -or lagging. The node lands in the `UNKNOWN` role and still serves reads; -`GET /lifecycle` and the server log carry a reason that starts with -`RESOURCE_GROUP_CATALOG_LAGGING` or `RESOURCE_GROUP_CATALOG_UNAVAILABLE`. -Lagging means the replica has not finished applying the transactions the catalog -depends on: wait for WAL apply to catch up and run `SWITCH ROLE TO PRIMARY` -again. Unavailable means the catalog table is missing or its contents cannot be -read, and retrying does not help: promote another replica, or restart this node -as primary with `resource.groups.enabled=false`, which turns the check into a -logged error. See +or lagging. The node lands in the `UNKNOWN` role and still serves reads; the +server log names `RESOURCE_GROUP_CATALOG_LAGGING` or +`RESOURCE_GROUP_CATALOG_UNAVAILABLE`. Lagging means the replica has not finished +applying the access control transactions the catalog depends on: wait for WAL +apply to catch up and run `SWITCH ROLE TO PRIMARY` again. Unavailable means the +catalog table is missing or its contents cannot be read, and retrying does not +help: promote another replica, or restart this node as primary with +`resource.groups.enabled=false`, which turns the check into a logged error. See [Refusals and the torn state](/docs/high-availability/failover/#refusals-and-the-torn-state). **Startup fails naming the resource group catalog.** The catalog table cannot be