diff --git a/documentation/concepts/resource-groups.md b/documentation/concepts/resource-groups.md new file mode 100644 index 000000000..fcf25ed3e --- /dev/null +++ b/documentation/concepts/resource-groups.md @@ -0,0 +1,225 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Resource groups isolate query workloads inside one QuestDB instance. Learn how + a query is assigned to a group, and what admission, CPU weight and memory + limits actually guarantee. +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + Resource groups isolate competing query workloads inside a single QuestDB + instance. + + +A resource group is a named policy that limits what a set of principals may +consume while their queries run. One instance typically serves several workloads +at once: dashboards that must answer in milliseconds, an ad-hoc analyst, and a +nightly report that scans a year of data. When these workloads compete without +resource controls, the report can increase dashboard latency. + +Resource groups control three things at the query execution boundary: + +- **Admission** — how many queries a group may run at once, how many may wait, + and how long they may wait. +- **Weighted CPU** — the share of query CPU a group receives while groups + compete. +- **Memory** — process and group budgets for tracked native query memory. + +The design is cooperative. QuestDB executes query work on shared worker pools, +and resource groups do not create one operating-system thread pool per group. A +query and all of its parallel tasks use the same resource group, while every +worker stays available to every group. + +With no configuration the feature is on and no group policy is in force: + +| Setting | Behaviour | +| ----------------------------------- | ------------------------------------------------------- | +| Feature enabled | Yes; turns itself off when a SQL pool is in legacy mode | +| Group admission | Unlimited active and queued queries | +| Group CPU | Weight 100 | +| Group and process memory budgets | Unlimited unless configured | +| Existing single-query memory limits | Still apply, including principal-specific limits | +| Memory accounting without limits | Remains enabled for tracked native query memory | + +## How a query is assigned to a group + +Assignment follows the authenticated principal, not the statement: + +1. A **direct mapping** on the user or service account wins. +2. Otherwise, for users only, QuestDB looks at the mappings of the ACL groups + the user belongs to and takes the highest `mapping_priority`. If two tie, the + mapping to the resource group that was created first wins, so give the groups + distinct priorities when the order matters. +3. Otherwise the query runs in **DEFAULT**. + +Service accounts inherit nothing from ACL groups; they are either mapped +directly or they run in DEFAULT. A session that assumes a service account keeps +the group of the principal that logged in; the service account's own mapping +applies to sessions that authenticate as that account. + +The group is resolved once, when the query starts, and stays fixed for the +statement's lifetime. Changing a mapping affects statements that start after the +change, never one already running. + +`DEFAULT` always exists. By default it carries no limits of its own, so unmapped +principals run with a CPU weight of 100, unlimited admission, and the +instance-wide memory limits. You can change its policy, but you cannot drop or +rename it. + +## What is managed + +Resource groups govern the statements that read data: + +- `SELECT` +- the source query of `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` +- query exports + +Everything else runs outside the feature and consumes no admission slot, CPU +grant or group memory budget: `EXPLAIN`, value `INSERT`, `UPDATE`, ordinary DDL, +`COPY`, transaction and session control, ILP and QWP ingestion, WAL apply, +materialized and live view refresh, and QuestDB's own internal SQL. + +For `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` the group is charged +for reading the source and producing rows, including parallel work. Where +writing a row cannot be separated from producing it, that CPU is charged to the +group as well. The commit, durability and any work handed to writer or WAL +queues are outside the guarantee. + +Resource groups account **tracked native query memory**. They do not represent +JVM heap, resident set size, memory-mapped table pages or long-lived engine +caches. Existing process memory protection remains the outer boundary. + +## What each control guarantees + +The three controls differ in how strong their guarantee is, which matters when +you decide what to configure. + +### Admission is a hard gate + +`max_active_queries` is an exact count. A group at its limit queues the next +query until a slot frees, up to `max_queued_queries`; beyond that the query is +rejected immediately. A queued query that waits longer than `queue_timeout` +fails, and the instance-wide `query.timeout` keeps running while it waits, so +whichever of the two expires first ends the wait. + +A slot is held only while the query is running on a worker. A protocol cursor +that is suspended between pages releases its slot and passes through the gate +again when the client asks for more rows, so a paging client does not hold +capacity while the application thinks. The consequence is that admission can be +refused on a later page: a client that received its first rows may still see the +queue-full or timeout error when it asks for more, and the connection stays +usable. + +### CPU weight is a share, not a reservation + +Weights only matter when groups compete. A group that is alone on the instance +uses everything it can, regardless of its weight, and a query that started while +its group was alone keeps running that way until it next suspends or finishes. +When two groups both have work, the scheduler hands out CPU so that measured CPU +divided by `cpu_weight` stays balanced: weights 100 and 50 converge to a 2:1 +split of query CPU. + +Weights are relative. 100 and 50 are the same as 2 and 1. A group that becomes +active starts level with the groups already running, so it neither banks the CPU +it did not use while idle nor is punished for having been busy. + +Shares are between groups, not between queries. Within a group, work is served +in arrival order, and a parallel query can hold several places in that order, so +there is no promise of equal CPU between individual queries. + +### Memory limits use batched accounting + +Accounting has three levels: query, group and process. Allocation and release +deltas accumulate on the executing worker and are published to the shared +counters in batches. Exceeding a checked limit fails the query with +`query memory limit exceeded`; it does not queue the allocation until memory +becomes available. + +The single-query ceiling starts with the principal's effective query memory +limit, when set, or the instance default `cairo.query.memory.limit.bytes`. Any +group `memory_limit` and process memory budget further cap that ceiling. The +group budget also bounds the total tracked memory held by its queries; the +process budget covers tracked native query memory across groups. + +An unset group `memory_limit`, or one set to `0` or `UNLIMITED`, adds no group +ceiling. A process budget of `0` adds no process ceiling. Existing single-query +limits still apply, and memory accounting remains enabled even when all limits +are unlimited. + +The counters can temporarily omit worker-local deltas, so a group can briefly +overshoot its limit by less than 64 KiB per worker running its queries. These +budgets are not byte-exact, instantaneous ceilings. + +## Why CPU control is cooperative + +QuestDB does not preempt a running query. The scheduler grants a query a short +slice of CPU on a worker and expects it to reach a cooperative checkpoint, which +is the same circuit breaker check that makes queries cancellable. At that point +the query either renews its grant or yields the worker to another group. + +A yielded worker runs other queries and, within a bounded window, returns to +accepting connections; the query that yielded resumes later. A long +single-threaded query that reaches these checkpoints therefore shares its worker +before finishing, which keeps the instance responsive while heavy queries run. +Slicing happens only under managed scheduling: while no policy is in force, a +query holds its worker exactly as it does with the feature disabled. + +Two consequences follow. + +The guarantee is statistical over a short window. Between checkpoints a query +holds its worker, so instantaneous CPU can deviate from the configured share. +The CPU actually used is charged either way, so a query that overran repays it +and the average is preserved. + +A query that cannot reach a checkpoint keeps its worker. While managed +scheduling is engaged its CPU is still charged when the slice ends, but no +cooperative limit can shorten that stretch. + +## Behaviour under failure and on replicas + +Resource groups are stored in a replicated system catalog, so a read-only +replica receives group definitions and mappings through normal replication. + +- A **fresh replica** that has not yet received the catalog runs queries + unmanaged, exactly as if the feature were disabled, and counts them in + `questdb_resource_groups_catalog_lag_unmanaged_queries_total`. It does not + reject queries or serve them under a policy it cannot see yet. +- A **replica being promoted** validates the catalog after replication has + switched and before writes are admitted. If the old primary predated resource + groups and never created the catalog table, the promoted node creates it and + continues. With the feature enabled, a catalog that is unreadable or that the + replica has not received yet refuses the promotion: the switch fails part-way, + the node lands in the `UNKNOWN` role and keeps serving reads as before, and + the log names `RESOURCE_GROUP_CATALOG_UNAVAILABLE` with the reason. Retrying + the switch repeats the check. With the feature disabled the condition is + logged and the promotion proceeds. +- At **startup** an unreadable catalog stops an instance with the feature + enabled from starting, in either role. A lagging catalog does not: the + instance starts and the refresh job catches up. +- If **CPU scheduling** hits an internal fault, it degrades: queries continue to + run without CPU grants, and the condition is visible in metrics until the + instance restarts. Admission and memory limits do not depend on CPU scheduling + and stay enforced. +- An **internal fault in one query** affects only that query. Other queries and + other groups are unaffected, and the CPU it used is still charged to its + group. + +## Cost when nothing competes + +While a single group owns all running queries, dispatch is unmanaged: no CPU is +sampled, no query yields, and every query holds its worker exactly as it does +with the feature disabled. Registration, admission and memory accounting still +run, so this is not free, but the cost is a fixed few microseconds per query. +Managed scheduling engages as soon as a second group has work, and disengages +again when it does not. A query that is already running stays unmanaged until it +next suspends or finishes; the new policy applies to queries that start or +resume after the change. + +## See also + +- [Configure and use resource groups](/docs/operations/resource-groups/) +- [Resource groups configuration](/docs/configuration/resource-groups/) +- [Role-based access control](/docs/security/rbac/) diff --git a/documentation/configuration/cairo-engine.md b/documentation/configuration/cairo-engine.md index e809a8adc..b52d45143 100644 --- a/documentation/configuration/cairo-engine.md +++ b/documentation/configuration/cairo-engine.md @@ -62,6 +62,11 @@ When `false`, disables the `reload_config()` SQL function. A global timeout for long-running queries, given as a duration: `500ms`, `120s`, `2m` and `1h` are all valid, and a plain number is read as milliseconds. +The timer starts when the server receives the statement and includes any time +the query spends queued for Resource Group admission. Over PGWire each `Execute` +message restarts it, so a client that fetches a cursor in batches is timed per +batch rather than across the whole result. + This key replaces `query.timeout.sec`. When both are set, `query.timeout` takes precedence; when neither is set, queries time out after 60 seconds. diff --git a/documentation/configuration/resource-groups.md b/documentation/configuration/resource-groups.md new file mode 100644 index 000000000..6440bf562 --- /dev/null +++ b/documentation/configuration/resource-groups.md @@ -0,0 +1,77 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Configuration settings for QuestDB Enterprise resource groups, covering the + master switch and the process memory ceiling. +--- + +:::note + +Resource groups are [Enterprise](/enterprise/) only. + +::: + +[Resource groups](/docs/concepts/resource-groups/) isolate competing query +workloads inside one instance. These settings are instance-wide. The per-group +policy that decides admission, CPU share and memory budgets is set in SQL, not +here. See [Configure and use resource groups](/docs/operations/resource-groups/) +for those statements. + +None of these settings are reloadable: changing any of them requires a restart. + +Resource groups also require access control to be enabled (`acl.enabled=true`) +before principals can be mapped to a group, and every pool that executes SQL +must run in Fiber mode, which is the default. What happens when a pool is in +legacy mode depends on how the feature was turned on. Left at its default, it +turns itself off and logs an error naming the pool and the setting to change. +Asked for explicitly, it fails startup with the same error, because an explicit +request and a legacy pool cannot both be honoured. + +## General + +### resource.groups.enabled + +- **Default**: `true` +- **Reloadable**: no + +Master switch. When `false`, resource group admission, CPU scheduling and group +memory accounting are disabled. Group definitions and principal mappings remain +in the catalog, so turning the feature back on restores the policies that were +already there. + +`true` also makes the catalog a hard dependency: an instance whose catalog +cannot be read does not start, and a replica whose catalog is not current is not +promoted. With `false`, both conditions are logged and ignored. See +[Behaviour under failure and on replicas](/docs/concepts/resource-groups/#behaviour-under-failure-and-on-replicas). + +Existing principal-specific and instance-default single-query memory limits +continue to apply when resource groups are disabled. + +Left unset, this resolves to `false` on an instance whose SQL pools are in +legacy mode, so upgrading such an instance does not turn the feature on and does +not stop the instance from starting. `SHOW PARAMETERS` then reports `false`, +which is the value that took effect. Set it to `true` explicitly and a legacy +pool becomes a startup error instead. + +### resource.groups.process.memory.limit.bytes + +- **Default**: `0` +- **Reloadable**: no + +Ceiling for tracked native query memory across all groups. `0` leaves the +instance without a process ceiling, which is the default. When set, it bounds +every group and every query, so no group policy can grant more than this. + +An unlimited process budget does not disable memory accounting or remove an +existing single-query limit. The group-level SQL parameter `memory_limit` treats +`0` and `UNLIMITED` as no ceiling, as does `RESET (memory_limit)`. + +This is not a process RSS limit. It covers tracked query memory only, not JVM +heap, memory-mapped table pages or long-lived engine caches. + +## See also + +- [Resource groups concept](/docs/concepts/resource-groups/) +- [Configure and use resource groups](/docs/operations/resource-groups/) +- [Identity and Access Management configuration](/docs/configuration/iam/) diff --git a/documentation/operations/logging-metrics.md b/documentation/operations/logging-metrics.md index 240647f3c..46e5a6620 100644 --- a/documentation/operations/logging-metrics.md +++ b/documentation/operations/logging-metrics.md @@ -1,10 +1,12 @@ --- title: Logging and metrics -description: Configure and understand QuestDB logging and metrics, including log levels, configuration options, and Prometheus integration. +description: + Configure and understand QuestDB logging and metrics, including log levels, + configuration options, and Prometheus integration. --- - -This page outlines logging in QuestDB. It covers how to configure logs via `log.conf` and expose metrics via Prometheus. +This page outlines logging in QuestDB. It covers how to configure logs via +`log.conf` and expose metrics via Prometheus. - [Logging](/docs/operations/logging-metrics/#logging) - [Metrics](/docs/operations/logging-metrics/#metrics) @@ -206,19 +208,20 @@ For configuration options, see the :::warning On systems with -[8 Cores and less](/docs/getting-started/capacity-planning/#cpu-cores), contention -for threads might increase the latency of health check service responses. If you -use a load balancer, and it thinks the QuestDB service is dead with nothing -apparent in the QuestDB logs, you may need to configure a dedicated thread pool -for the health check service. To do so, increase `http.min.worker.count` to `1`. +[8 Cores and less](/docs/getting-started/capacity-planning/#cpu-cores), +contention for threads might increase the latency of health check service +responses. If you use a load balancer, and it thinks the QuestDB service is dead +with nothing apparent in the QuestDB logs, you may need to configure a dedicated +thread pool for the health check service. To do so, increase +`http.min.worker.count` to `1`. ::: #### Lifecycle endpoint `GET /lifecycle` on the same port returns the startup and shutdown state of -every server component as JSON, for probes and coordinators that need more -than the `200` of the health check: +every server component as JSON, for probes and coordinators that need more than +the `200` of the health check: ```shell curl http://127.0.0.1:9003/lifecycle @@ -338,23 +341,23 @@ When [cold storage](/docs/concepts/cold-storage/) is enabled, the endpoint exposes fifteen additional metrics under the `questdb_cold_chunk_` prefix, covering the chunk cache and the range reads that serve remote partitions: -| Metric | Type | Description | -| ------ | ---- | ----------- | -| `questdb_cold_chunk_acquire_full_hit_total` | counter | Reads that found every chunk already resident | -| `questdb_cold_chunk_acquire_partial_hit_total` | counter | Reads that found some chunks and fetched the rest | -| `questdb_cold_chunk_acquire_full_miss_total` | counter | Reads where every chunk had to be fetched | -| `questdb_cold_chunk_acquire_hit_chunks_total` | counter | Chunk lookups served from the cache | -| `questdb_cold_chunk_acquire_miss_chunks_total` | counter | Chunk lookups that had to be fetched | -| `questdb_cold_chunk_download_started_total` | counter | Range requests dispatched, one per coalesced group | -| `questdb_cold_chunk_download_finished_total` | counter | Range requests that returned data | -| `questdb_cold_chunk_download_failed_total` | counter | Range requests that failed after retries | -| `questdb_cold_chunk_download_coalesced_total` | counter | Readers that attached to an in-flight download instead of starting a new one | -| `questdb_cold_chunk_release_evictions_total` | counter | Chunks evicted when their last lease was released | -| `questdb_cold_chunk_in_flight_downloads` | gauge | Range requests dispatched but not yet complete | -| `questdb_cold_chunk_pending_batches` | gauge | Batches the read coordinator is tracking | -| `questdb_cold_chunk_busy_leases` | gauge | Currently allocated leases | -| `questdb_cold_chunk_ready_chunks` | gauge | Chunks resident in the ready cache | -| `questdb_cold_chunk_pinned_bytes` | gauge | Compressed bytes resident in the ready cache | +| Metric | Type | Description | +| ---------------------------------------------- | ------- | ---------------------------------------------------------------------------- | +| `questdb_cold_chunk_acquire_full_hit_total` | counter | Reads that found every chunk already resident | +| `questdb_cold_chunk_acquire_partial_hit_total` | counter | Reads that found some chunks and fetched the rest | +| `questdb_cold_chunk_acquire_full_miss_total` | counter | Reads where every chunk had to be fetched | +| `questdb_cold_chunk_acquire_hit_chunks_total` | counter | Chunk lookups served from the cache | +| `questdb_cold_chunk_acquire_miss_chunks_total` | counter | Chunk lookups that had to be fetched | +| `questdb_cold_chunk_download_started_total` | counter | Range requests dispatched, one per coalesced group | +| `questdb_cold_chunk_download_finished_total` | counter | Range requests that returned data | +| `questdb_cold_chunk_download_failed_total` | counter | Range requests that failed after retries | +| `questdb_cold_chunk_download_coalesced_total` | counter | Readers that attached to an in-flight download instead of starting a new one | +| `questdb_cold_chunk_release_evictions_total` | counter | Chunks evicted when their last lease was released | +| `questdb_cold_chunk_in_flight_downloads` | gauge | Range requests dispatched but not yet complete | +| `questdb_cold_chunk_pending_batches` | gauge | Batches the read coordinator is tracking | +| `questdb_cold_chunk_busy_leases` | gauge | Currently allocated leases | +| `questdb_cold_chunk_ready_chunks` | gauge | Chunks resident in the ready cache | +| `questdb_cold_chunk_pinned_bytes` | gauge | Compressed bytes resident in the ready cache | Watch rates and ratios rather than raw totals. Sustained `download_failed_total`, `pending_batches` sitting at its configured cap, or @@ -369,10 +372,49 @@ _Enterprise only._ Two gauges describe the state of an in-place [role switch](/docs/high-availability/failover/): -| Metric | Type | Description | -| ------ | ---- | ----------- | +| Metric | Type | Description | +| ---------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `questdb_replication_pending_upload_txn` | gauge | Committed transactions not yet uploaded to the object store, summed over the replicated tables. Poll it before demoting a primary: a demote that cannot bring it to zero within its timeout is not completed | -| `questdb_backup_active_at_last_demote` | gauge | `1` if a backup was still running when the node was last demoted, `0` otherwise. Cleared by the next promotion | +| `questdb_backup_active_at_last_demote` | gauge | `1` if a backup was still running when the node was last demoted, `0` otherwise. Cleared by the next promotion | + +### Resource group metrics + +_Enterprise only._ + +When [resource groups](/docs/concepts/resource-groups/) are enabled, the +endpoint exposes one series per group, labelled with `resource_group`: + +| Metric | Type | Description | +| --------------------------------------------------- | ------- | ------------------------------------------------------------ | +| `questdb_resource_group_active_queries` | gauge | Queries holding an admission slot | +| `questdb_resource_group_queued_queries` | gauge | Queries waiting for a slot | +| `questdb_resource_group_oldest_queue_wait_millis` | gauge | How long the longest waiting query has waited | +| `questdb_resource_group_memory_bytes` | gauge | Tracked query memory in use | +| `questdb_resource_group_memory_limit_bytes` | gauge | Effective group memory ceiling, `0` when the group sets none | +| `questdb_resource_group_cpu_nanos_total` | counter | CPU measured under managed scheduling | +| `questdb_resource_group_cpu_wait_nanos_total` | counter | Time the group spent waiting for CPU | +| `questdb_resource_group_admission_rejections_total` | counter | Queries rejected because the queue was full | +| `questdb_resource_group_admission_timeouts_total` | counter | Queries that timed out while queued | + +The single-group dispatch path does not sample CPU. Consequently, +`cpu_nanos_total` counts CPU measured by managed scheduling, not every query's +CPU consumption. A flat counter does not imply that the group is idle; also +check `questdb_resource_groups_cpu_managed_dispatch` and query activity. Memory +gauges show published accounting and can lag worker-local deltas. + +Instance-wide series describe the feature itself: + +| Metric | Type | Description | +| ------------------------------------------------------------- | ------- | --------------------------------------------------------------------- | +| `questdb_resource_groups_enabled` | gauge | `1` when the feature is on | +| `questdb_resource_groups_catalog_current` | gauge | `1` when the group catalog is current; `0` while a replica catches up | +| `questdb_resource_groups_catalog_lag_unmanaged_queries_total` | counter | Queries that ran unmanaged because the catalog was not current yet | +| `questdb_resource_groups_cpu_managed_dispatch` | gauge | `1` while managed CPU scheduling is engaged | +| `questdb_resource_groups_cpu_scheduler_degraded` | gauge | `1` when CPU scheduling has degraded to unmanaged | + +A non-zero `questdb_resource_groups_cpu_scheduler_degraded` means CPU shares are +no longer enforced until the instance restarts. Admission and memory limits stay +enforced. ### Prometheus Alertmanager diff --git a/documentation/operations/resource-groups.md b/documentation/operations/resource-groups.md new file mode 100644 index 000000000..19096ede4 --- /dev/null +++ b/documentation/operations/resource-groups.md @@ -0,0 +1,555 @@ +--- +title: Resource groups +sidebar_label: Resource groups +description: + Create resource groups, map users and ACL groups to them, and tune admission, + CPU and memory limits so one workload cannot starve another. +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + Resource groups isolate competing query workloads inside a single QuestDB + instance. + + +This page covers day-to-day use: creating groups, mapping principals, choosing +limits, and watching the result. For what the limits actually guarantee, read +[the concept page](/docs/concepts/resource-groups/) first. + +## Quick start + +This example separates reporting from the default workload and limits its +concurrency and memory. Run it as an administrator on an instance that meets the +[requirements](#requirements). Use unused example names and replace the password +placeholders. Later examples on this page can be adapted independently. + +First create the ACL principals and allow SQL connections: + +```questdb-sql +CREATE GROUP analysts; +GRANT HTTP, PGWIRE TO analysts; +CREATE USER reporting_user WITH PASSWORD ''; +ADD USER reporting_user TO analysts; + +CREATE USER nightly_batch WITH PASSWORD ''; +GRANT HTTP, PGWIRE TO nightly_batch; +``` + +Then create the resource group and mappings: + +```questdb-sql +-- 1. Create a group. Unset parameters fall back to the instance defaults. +CREATE RESOURCE GROUP reporting WITH ( + cpu_weight = 50, + max_active_queries = 4, + max_queued_queries = 32, + queue_timeout = '15s', + memory_limit = '2G' +); + +-- 2. reporting_user inherits this mapping unless a higher-precedence one applies. +ALTER GROUP analysts SET RESOURCE GROUP reporting MAPPING PRIORITY 10; + +-- 3. Map one user directly. A direct mapping beats any ACL group mapping. +ALTER USER nightly_batch SET RESOURCE GROUP reporting; +``` + +Verify: + +```questdb-sql +SELECT name, cpu_weight, max_active_queries, active_queries, queued_queries +FROM resource_groups(); + +SELECT * FROM resource_group_mappings(); +``` + +Reconnect as `reporting_user` or `nightly_batch` and run: + +```questdb-sql +SELECT current_resource_group(); +``` + +| current_resource_group | +| ---------------------- | +| reporting | + +These grants allow connections. Grant access to the application's tables +separately, as described in [RBAC](/docs/security/rbac/). + +Everything not mapped keeps running in `DEFAULT`, which has a CPU weight of 100. +Against `reporting`'s weight of 50, that is a 2:1 split of query CPU while both +have work. + +## Requirements + +- QuestDB Enterprise. +- Access control enabled (`acl.enabled=true`). Groups can be created without it, + but mapping statements require it, since mappings attach to ACL principals. +- The pools that execute SQL must run in Fiber mode, which is the default and + which the feature depends on. On an instance whose pools are in legacy mode, + resource groups left at their default turn themselves off and log an error + naming the pool and the setting to change. Setting + `resource.groups.enabled=true` on such an instance fails startup with that + same error. +- Administrator rights for group management, mappings and instance-wide + inspection. Ordinary users can call `current_resource_group()` to check their + own query's group. + +A protocol runs on its own pool when its worker count is above zero, otherwise +on the shared network pool. The setting that matters is the one for the pool it +actually uses: + +| Where the protocol runs | Setting to check | +| ---------------------------------------------------- | ------------------------------------- | +| Its own HTTP pool (`http.worker.count` above zero) | `http.worker.fiber.enabled` | +| Its own PGWire pool (`pg.worker.count` above zero) | `pg.worker.fiber.enabled` | +| The shared network pool (worker count zero, default) | `shared.network.worker.fiber.enabled` | + +Parallel query work is separate and follows `shared.query.worker.fiber.enabled` +whenever the shared query pool has workers. A shared query pool set to zero +workers turns parallel SQL off by default and needs no check of its own. + +The first two settings default to `true`, so a dedicated pool is a Fiber pool +unless someone turned it off. `shared.network.worker.fiber.enabled` defaults to +`true` exactly when HTTP or PGWire actually runs there, which is the case out of +the box because both worker counts default to zero. Check these only when the +instance was tuned by hand. + +## Configuration + +Resource groups are enabled by default. These are instance-wide settings; the +per-group policy is set in SQL. Each setting is described in full in the +[resource groups configuration reference](/docs/configuration/resource-groups/). + +| Property | Default | Meaning | +| -------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------ | +| `resource.groups.enabled` | `true` | Set to `false` to disable resource group enforcement. Existing single-query memory limits still apply. | +| `resource.groups.process.memory.limit.bytes` | `0` | Ceiling for tracked query memory across all groups, `0` for none. Every group limit is capped by it. | + +Turning the feature off is a restart with `resource.groups.enabled=false`. +Definitions and mappings stay in the catalog, so nothing is lost and the +policies apply again when it is re-enabled. + +## Managing groups + +```questdb-sql +CREATE RESOURCE GROUP analytics; + +CREATE RESOURCE GROUP IF NOT EXISTS analytics WITH (cpu_weight = 300); + +ALTER RESOURCE GROUP analytics SET (cpu_weight = 300, max_active_queries = 8); + +-- Clear parameters so they fall back to the instance defaults again. +ALTER RESOURCE GROUP analytics RESET (memory_limit, max_active_queries); + +ALTER RESOURCE GROUP analytics RENAME TO reporting; + +DROP RESOURCE GROUP reporting; +DROP RESOURCE GROUP IF EXISTS reporting; +``` + +A group policy change applies online to the shared group budget. It does not +cancel existing queries at the moment `ALTER` runs: + +| Change | Effect on existing work | +| ---------------------- | ------------------------------------------------------------------------------------------------- | +| CPU weight | Subsequent scheduling uses the new policy | +| Active-query limit | Existing slots are retained; subsequent admission, including a resumed cursor, uses the new limit | +| Queue limit or timeout | New admission requests use the new settings; an already queued request keeps its deadline | +| Group memory limit | Subsequent allocations check the new budget; existing memory is released normally | + +Lowering a memory budget below current usage can make subsequent allocations +fail. The principal-specific or instance-default single-query limit is captured +when the query starts; updating the group budget does not replace that limit. +Changing a principal mapping affects new queries only. + +`DROP` is refused while any live principal is still mapped to the group; unmap +them first. Once unmapped, a group can be dropped while queries still use it. It +disappears from `resource_groups()` immediately. Running and queued queries, +including suspended cursors, continue using the deleted group's existing +settings. Their memory still counts towards the process budget. + +Recreating a group with the same name starts fresh usage counters. Queries that +still use the deleted group do not move to the new group or use its settings. +Map principals to the new group to assign their subsequent queries to it. + +`DEFAULT` cannot be dropped or renamed, but it can be altered: + +```questdb-sql +ALTER RESOURCE GROUP DEFAULT SET (max_active_queries = 16); +``` + +## Mapping principals + +```questdb-sql +ALTER USER alice SET RESOURCE GROUP analytics; +ALTER SERVICE ACCOUNT ingest_bot SET RESOURCE GROUP analytics; +ALTER GROUP analysts SET RESOURCE GROUP analytics MAPPING PRIORITY 10; + +ALTER USER alice UNSET RESOURCE GROUP; +ALTER GROUP analysts UNSET RESOURCE GROUP; +``` + +`MAPPING PRIORITY` is a non-negative integer and applies only to ACL group +mappings, because a user can belong to several ACL groups. The highest priority +wins; if two ACL groups tie, the mapping to the resource group that was created +first wins, so give them distinct priorities when the order matters. It defaults +to 0 and is rejected on user and service account mappings, which are one-to-one. + +Resolution order for a query is: direct mapping on the principal, then the +highest-priority mapping among the user's ACL groups, then `DEFAULT`. Service +accounts do not inherit ACL group mappings. `ASSUME SERVICE ACCOUNT` does not +change the group: the session keeps the group of the principal that logged in. + +## Policy parameters + +All parameters are optional. An unset parameter is not "unlimited" in every +case: it falls back to the instance default shown here. + +| Parameter | Accepted values | Unset behaviour | +| -------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | +| `cpu_weight` | integer, 1 to 10000 | 100 | +| `max_active_queries` | integer, 1 or more | unlimited | +| `max_queued_queries` | integer, 0 or more | unlimited | +| `queue_timeout` | a positive whole number of milliseconds, or a duration such as `'15s'`, `'2m'` | 30 seconds | +| `memory_limit` | a byte size, plain or suffixed such as `'8G'`, or `0` or `UNLIMITED` for no group ceiling | no group ceiling; other memory limits still apply | + +`memory_limit` is the budget for everything the group runs at once. Where the +instance sets `resource.groups.process.memory.limit.bytes`, the group budget is +capped by it, so a group cannot be granted more than the instance allows. A +group ceiling only lowers what its queries may use; it never raises a limit set +elsewhere. + +A group that does not set `memory_limit` carries no ceiling of its own, and +`resource_groups().memory_limit_bytes` reports `0` for it. Its queries are then +bounded by any existing single-query limit and the process limit. A principal's +effective query memory limit takes precedence over the instance default +`cairo.query.memory.limit.bytes`; group and process budgets can only lower the +resulting ceiling. + +To remove a group memory ceiling, use +`ALTER RESOURCE GROUP reporting RESET (memory_limit)` or set `memory_limit` to +`0` or `UNLIMITED`; `0` matches the instance process-memory property. Accounting +continues when limits are unlimited. + +An example of what a weight means in practice: + +```questdb-sql +-- A share: reporting gets a third of query CPU when DEFAULT also has work, +-- and all of it when DEFAULT is idle. +CREATE RESOURCE GROUP reporting WITH (cpu_weight = 50); +``` + +Weights arbitrate only between groups that have work at the same time; they +never hold CPU back from a group that is alone. + +## Common scenarios + +### The instance stops answering while CPU looks idle + +Every HTTP or PGWire worker is occupied by a large query, new requests are not +picked up, and instance CPU is low because those queries run on one core each. +Clients time out and retry, which produces more of the same queries. + +Three steps. The first is a prerequisite to confirm; the other two are policy +you choose. + +**1. Confirm the SQL pools are Fiber pools.** This is the prerequisite for +everything below; the [requirements](#requirements) list which setting governs +each pool. With the instance running, `SHOW PARAMETERS` must report +`resource.groups.enabled` as `true` and `questdb_resource_groups_enabled` must +be `1`. Anything else means a SQL pool is in legacy mode and the feature turned +itself off; the startup log names the pool. + +**2. Separate the workloads into groups.** With no policy written, queries hold +their workers exactly as they do with the feature disabled. Managed scheduling +engages while two groups have queries in flight at the same time. Under it a +query yields its worker at the checkpoints that already make it cancellable, so +a long single-threaded scan releases the worker while it is still running and +the instance keeps accepting connections, and CPU is split by weight: + +```questdb-sql +CREATE RESOURCE GROUP dashboards WITH (cpu_weight = 400); +CREATE RESOURCE GROUP adhoc WITH (cpu_weight = 100); + +ALTER USER app SET RESOURCE GROUP dashboards; +ALTER USER analyst SET RESOURCE GROUP adhoc; +``` + +`questdb_resource_groups_cpu_managed_dispatch` reports `1` while managed +scheduling is engaged. A query that started while its group was alone keeps its +worker until it finishes, and a query that never reaches a checkpoint holds its +worker either way, so this does not remove every cause of an unresponsive +instance. + +**3. Bound concurrent requests with admission.** + +```questdb-sql +ALTER RESOURCE GROUP adhoc SET ( + max_active_queries = 4, + max_queued_queries = 8, + queue_timeout = '5s' +); +``` + +The fifth concurrent query waits instead of running, and it does not hold a +worker while it waits. The thirteenth fails immediately with +`Resource Group admission queue is full`, so a client that keeps resending gets +a clear answer in seconds instead of adding to the pile. + +Admission directly bounds the number of concurrent queries; weights do not, so +the two are complementary. Clients should use bounded retries with backoff after +admission failures. + +Afterwards the symptom is also diagnosable rather than mysterious. Low instance +CPU together with a high `queued_queries` and a rising +`oldest_queue_wait_millis` on one group says the work is being held at the +admission gate, not that the machine is busy. `query_activity()` shows which +group each running query was admitted to. + +What resource groups do not do here: they do not make the slow plan faster, and +they do not bound how long one query may run. Wall-clock limits still come from +the instance-wide +[`query.timeout`](/docs/configuration/cairo-engine/#querytimeout), and that +clock includes time spent waiting in the admission queue: a query can time out +before it starts, and the error then says +`while queued for Resource Group admission`. + +### Dashboards must stay responsive while analysts run heavy queries + +Use weights. Shares are per group, not per query, so a group running fifty +queries does not outvote a group running one: + +```questdb-sql +CREATE RESOURCE GROUP dashboards WITH (cpu_weight = 400); +CREATE RESOURCE GROUP analysts WITH (cpu_weight = 100); +``` + +When these are the only competing groups and both can use their shares, weights +target a 4:1 split of managed query CPU. Actual use also depends on runnable +work and available parallelism. When analysts are idle, dashboards can use the +available query CPU. Weights are integers from 1 to 10000 and every group starts +at 100, so a group left alone keeps an equal share against any group you do not +change. + +### A background job must yield to everything else + +Give it a small weight and a small concurrency limit: + +```questdb-sql +CREATE RESOURCE GROUP exports WITH (cpu_weight = 10, max_active_queries = 1); +``` + +Under contention the job receives a tenth of the CPU that a group at the default +weight of 100 receives, and it runs one query at a time. When nothing else has +work it uses the CPU that would otherwise be idle; resource groups do not hold +CPU back from a group that is alone. + +### One workload must not exhaust query memory + +Bound the group rather than each query, so the limit holds however many queries +the workload starts: + +```questdb-sql +CREATE RESOURCE GROUP reporting WITH (memory_limit = '8G'); +``` + +A query that would push the group over its budget fails with +`query memory limit exceeded` reporting `scope=group`, and releases what it +held. + +### An ingestion or automation account runs queries too + +Service accounts resolve differently from users: they honour a direct mapping, +but they never inherit a mapping from an ACL group. A service account with no +direct mapping runs in `DEFAULT` however its ACL groups are mapped, so map it +explicitly: + +```questdb-sql +CREATE RESOURCE GROUP automation WITH (cpu_weight = 50, max_active_queries = 2); + +ALTER SERVICE ACCOUNT ingest_bot SET RESOURCE GROUP automation; +``` + +This governs the queries the account runs. It does not throttle ingestion +itself, which resource groups do not manage. + +### Many teams share one instance + +Map ACL groups rather than individual users, and use `MAPPING PRIORITY` to +decide what happens to someone who belongs to more than one: + +```questdb-sql +ALTER GROUP analysts SET RESOURCE GROUP adhoc MAPPING PRIORITY 10; +ALTER GROUP oncall SET RESOURCE GROUP dashboards MAPPING PRIORITY 20; +``` + +Someone in both groups resolves to `dashboards`, because the higher priority +wins. If two mappings tie on priority, the one whose resource group was created +first wins, so keep priorities distinct. A direct mapping on the user beats +every group mapping regardless of priority, which is the way to make one person +an exception without touching the groups: + +```questdb-sql +ALTER USER lead_analyst SET RESOURCE GROUP dashboards; +``` + +`MAPPING PRIORITY` is rejected on user and service account mappings, because +those are one-to-one and have nothing to break a tie between. Confirm any of +this from the client's own session with `SELECT current_resource_group();`. + +## Inspecting + +`resource_groups()` returns one row per group, combining the configured policy +with live counters: + +| Column | Meaning | +| ------------------------------------------------------------------ | --------------------------------------------------- | +| `name` | Group name | +| `memory_limit_bytes` | Effective group memory budget | +| `max_active_queries`, `max_queued_queries`, `queue_timeout_millis` | Effective admission policy | +| `cpu_weight` | Effective CPU weight | +| `active_queries`, `queued_queries` | Live admission state | +| `oldest_queue_wait_millis` | How long the longest waiting query has waited | +| `memory_used_bytes` | Tracked query memory in use | +| `cpu_nanos_total`, `cpu_wait_nanos_total` | Cumulative CPU consumed and spent waiting for CPU | +| `admission_rejections`, `admission_timeouts` | Cumulative queue-full rejections and queue timeouts | + +`resource_group_mappings()` returns one row per mapping with `principal_type`, +`principal_name`, `resource_group_id`, `resource_group` and `mapping_priority`. + +`current_resource_group()` returns the calling query's group, which is the +quickest way to confirm a mapping from the client's own connection: + +```questdb-sql +SELECT current_resource_group(); +``` + +It returns `NULL` when that execution is unmanaged, including when the feature +is disabled or a replica's group catalog is not ready. See the +[function reference](/docs/query/functions/meta/#current_resource_group) for +permissions and return values, and the references for +[`resource_groups()`](/docs/query/functions/meta/#resource_groups) and +[`resource_group_mappings()`](/docs/query/functions/meta/#resource_group_mappings) +for complete schemas. + +`query_activity()` carries a `resource_group` column, so you can see which group +each running query was admitted to. It is `NULL` for executions that resource +groups do not manage: + +```questdb-sql +SELECT resource_group, username, query_start, query +FROM query_activity() +WHERE resource_group IS NOT NULL +ORDER BY query_start; +``` + +## Monitoring + +The Prometheus endpoint exposes one series per group, labelled with +`resource_group`. The full list lives in the +[metrics reference](/docs/operations/logging-metrics/#resource-group-metrics): + +``` +questdb_resource_group_active_queries{resource_group="reporting"} +questdb_resource_group_queued_queries{resource_group="reporting"} +questdb_resource_group_oldest_queue_wait_millis{resource_group="reporting"} +questdb_resource_group_memory_bytes{resource_group="reporting"} +questdb_resource_group_memory_limit_bytes{resource_group="reporting"} +questdb_resource_group_cpu_nanos_total{resource_group="reporting"} +questdb_resource_group_cpu_wait_nanos_total{resource_group="reporting"} +questdb_resource_group_admission_rejections_total{resource_group="reporting"} +questdb_resource_group_admission_timeouts_total{resource_group="reporting"} +``` + +Instance-wide series: + +| Metric | Meaning | +| ------------------------------------------------------------- | --------------------------------------------------------------------- | +| `questdb_resource_groups_enabled` | 1 when the feature is on | +| `questdb_resource_groups_catalog_current` | 1 when the catalog is current; 0 while a replica is still catching up | +| `questdb_resource_groups_catalog_lag_unmanaged_queries_total` | Queries that ran unmanaged because the catalog was not current yet | +| `questdb_resource_groups_cpu_managed_dispatch` | 1 while managed CPU scheduling is engaged | +| `questdb_resource_groups_cpu_scheduler_degraded` | 1 when CPU scheduling has degraded to unmanaged | + +Two signals are worth alerting on: a non-zero +`questdb_resource_groups_cpu_scheduler_degraded`, which means CPU shares are no +longer enforced until the next restart, and a steadily growing +`questdb_resource_group_admission_timeouts_total`, which means a group's queue +settings are rejecting work the application expects to succeed. + +## Errors clients see + +| Message | Cause | Usual fix | +| ----------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| `Resource Group admission queue is full` | The group is at `max_active_queries` and its queue is at `max_queued_queries` | Raise the limits, or let the client retry | +| `Resource Group admission queue timeout` | The query waited longer than `queue_timeout` | Raise `queue_timeout` or `max_active_queries`, or reduce concurrency | +| Either admission error while fetching a later page | A suspended cursor re-enters admission when the client asks for more rows | Adjust admission limits or retry the query with backoff; the failed cursor cannot continue | +| `query memory limit exceeded` | A single-query, group or process memory limit rejected an allocation | `scope` in the message names the level: `query`, `group` or `process`. Reduce memory use or raise that limit | +| `Resource Group is assigned to an ACL entity` | `DROP RESOURCE GROUP` while principals are still mapped | `UNSET RESOURCE GROUP` on those principals first | +| `built-in Resource Group cannot be dropped` / `cannot be renamed` | `DROP` or `RENAME` on `DEFAULT` | Alter it instead | + +## Troubleshooting + +**A group's CPU share is not what I configured.** Weights only apply while +groups compete. Check `active_queries` on both groups at the same moment: if one +is idle, the other is expected to use everything. A query that started while its +group was alone stays outside CPU scheduling until it next suspends or finishes, +so only work that starts or resumes under contention is weighted. Also confirm +the work you are watching is managed at all, since ingestion, WAL apply and view +refresh are outside the feature. `query_activity()` shows the group each running +query belongs to, which is the quickest way to tell whether the load you are +watching is attributed where you expect. + +**Queries on a fresh replica are not limited.** Until the catalog has +replicated, a replica runs queries unmanaged and counts them in +`questdb_resource_groups_catalog_lag_unmanaged_queries_total`. The counter stops +growing once `questdb_resource_groups_catalog_current` reaches 1. + +**Promotion fails naming the resource group catalog.** With the feature enabled, +`SWITCH ROLE TO PRIMARY` does not admit writes over a catalog that is unreadable +or that the replica has not received yet. The node lands in the `UNKNOWN` role +and still serves reads; the server log names +`RESOURCE_GROUP_CATALOG_UNAVAILABLE` with the reason. When the reason is +`Resource Group catalog table is not locally available`, replication has not +delivered the catalog table yet: wait for it and run `SWITCH ROLE TO PRIMARY` +again. Any other reason means the table cannot be read, and retrying does not +help: promote another replica, or restart this node as primary with +`resource.groups.enabled=false`, which turns the check into a logged error. See +[Refusals and the torn state](/docs/high-availability/failover/#refusals-and-the-torn-state). + +**Startup fails naming the resource group catalog.** The catalog table cannot be +created or read while the feature is enabled; the startup error names the +Resource Group catalog and the reason. Starting with +`resource.groups.enabled=false` logs the condition instead of failing. + +**Startup fails naming a worker pool.** A pool that executes SQL is in legacy +mode while `resource.groups.enabled=true` was set explicitly. Either restore the +default Fiber mode for that pool or stop setting the property, which lets the +instance start with resource groups off. + +**The feature is off although the default is on.** Check the log at startup for +an error naming a worker pool, and check `SHOW PARAMETERS` for the value that +took effect. A legacy SQL pool turns the feature off when the property is left +unset. + +## Limitations + +- Only query statements are managed. See + [what is managed](/docs/concepts/resource-groups/#what-is-managed). +- Memory accounting covers tracked native query memory, not JVM heap, resident + set size or memory-mapped table pages. +- CPU control is cooperative, so shares hold over a short window rather than + instantaneously, and a query that cannot reach a cooperative checkpoint holds + its worker until it does. +- Principal mapping changes affect new queries. Group budgets change online; + dropping a group retains its runtime state for existing queries until they + finish. + +## See also + +- [Resource groups concept](/docs/concepts/resource-groups/) +- [Resource groups configuration](/docs/configuration/resource-groups/) +- [Role-based access control](/docs/security/rbac/) +- [Logging and metrics](/docs/operations/logging-metrics/) diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index d2156a88f..d1fbf4003 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -61,9 +61,9 @@ SELECT current_data_id(); ## current database, schema, or user -`current_database()`, `current_schema()`, `current_user()`, and -`session_user()` are standard SQL functions that return information about the -current database, schema, and user. +`current_database()`, `current_schema()`, `current_user()`, and `session_user()` +are standard SQL functions that return information about the current database, +schema, and user. ```questdb-sql -- Get the current database @@ -86,6 +86,28 @@ statement without any arguments. and are interchangeable in QuestDB. Both report the user that authenticated on the current connection, whichever protocol it arrived on. +## current_resource_group + +_QuestDB Enterprise only._ + +Returns the resource group assigned to the calling query. Ordinary users can use +this function to check their own assignment; administrator rights are not +required. See [resource groups](/docs/concepts/resource-groups/) for mapping +precedence and the scope of managed execution. + +**Arguments:** none. + +**Return value:** `STRING`. Returns `NULL` when the execution is unmanaged, +including when resource groups are disabled or a replica's catalog is not ready. +A managed query without a principal mapping returns `DEFAULT`. + +```questdb-sql +SELECT current_resource_group(); +``` + +The result follows the query's acquired group, including across suspended cursor +pages. A subsequent mapping change affects the next query. + ## flush_query_cache() `flush_query_cache' invalidates cached query execution plans. @@ -314,7 +336,6 @@ materialized_views(); | trades_OHLC_15m | immediate | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | | trades_latest_1d | immediate | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null | - ## memory_metrics **Arguments:** @@ -382,10 +403,10 @@ SELECT node_role(); :::warning `node_role()` cannot be used in a materialized view or a live view. Avoid it in -`UPDATE` on a WAL table as well: the statement is re-executed on every node of -a replicated cluster and each node evaluates its own role, so the primary and -its replicas would write different values. Tagging rows on `INSERT` is safe, -because inserted rows replicate as data. +`UPDATE` on a WAL table as well: the statement is re-executed on every node of a +replicated cluster and each node evaluates its own role, so the primary and its +replicas would write different values. Tagging rows on `INSERT` is safe, because +inserted rows replicate as data. ::: @@ -410,18 +431,33 @@ Returns metadata on running SQL queries, including columns such as: - state_change - timestamp of latest query state change, such as a cancellation - state - state of running query, can be `active` or `cancelled` - query - text of sql query +- is_wal - whether the query runs as part of WAL apply +- memory_used, memory_limit - tracked native memory the query holds and its + ceiling, `NULL` when no tracker is bound; `memory_limit` is also `NULL` when + the query has no ceiling +- resource_group - the [resource group](/docs/concepts/resource-groups/) the + query was admitted to in QuestDB Enterprise, `NULL` when resource groups do + not manage the execution **Examples:** ```questdb-sql -SELECT * FROM query_activity(); +SELECT query_id, worker_id, worker_pool, username, query_start, state_change, state, query +FROM query_activity(); ``` | query_id | worker_id | worker_pool | username | query_start | state_change | state | query | | -------- | --------- | ----------- | -------- | --------------------------- | --------------------------- | ------ | --------------------------------------------------------- | -| 62179 | 5 | shared | bob | 2024-01-09T10:03:05.557397Z | 2024-01-09T10:03:05.557397 | active | select \* from query_activity() | +| 62179 | 5 | shared | bob | 2024-01-09T10:03:05.557397Z | 2024-01-09T10:03:05.557397Z | active | SELECT count() FROM trades | | 57777 | 6 | shared | bob | 2024-01-09T08:58:55.988017Z | 2024-01-09T08:58:55.988017Z | active | SELECT symbol,approx_percentile(price, 50, 2) from trades | +To inspect query memory and resource group assignment in QuestDB Enterprise: + +```questdb-sql +SELECT query_id, username, resource_group, memory_used, memory_limit +FROM query_activity(); +``` + ## reader_pool **Arguments:** @@ -470,15 +506,90 @@ Edit `server.conf` and run `reload_config`: SELECT reload_config(); ``` +## resource_group_mappings + +_QuestDB Enterprise only. Requires administrator rights._ + +Returns the principal mappings in the resource group catalog. Definitions remain +available when resource group enforcement is disabled. + +**Arguments:** none. + +**Return value:** a table with these columns: + +| Column | Type | Description | +| ------------------- | --------- | -------------------------------------------------------------------------------------------------------------- | +| `principal_type` | `VARCHAR` | `USER`, `GROUP` or `SERVICE_ACCOUNT` | +| `principal_name` | `VARCHAR` | ACL principal name | +| `resource_group_id` | `LONG` | System-assigned identifier of the mapped resource group | +| `resource_group` | `VARCHAR` | Group name | +| `mapping_priority` | `INT` | Priority for ACL group mappings; defaults to `0`. Equal priorities resolve to the resource group created first | + +```questdb-sql +SELECT principal_type, principal_name, resource_group, mapping_priority +FROM resource_group_mappings() +ORDER BY principal_type, principal_name; +``` + +This lists mappings rather than expanding inherited assignments into one row per +user. Use `current_resource_group()` from a user's own session to confirm the +resolved assignment. + +## resource_groups + +_QuestDB Enterprise only. Requires administrator rights._ + +Returns one row per current catalog group, including `DEFAULT`, with resolved +policies and live counters. Group definitions remain visible when enforcement is +disabled; their runtime counters are zero. + +**Arguments:** none. + +**Return value:** a table with these columns: + +| Column | Type | Description | +| -------------------------- | --------- | ------------------------------------------------------------------------------------------------------- | +| `name` | `VARCHAR` | Group name | +| `memory_limit_bytes` | `LONG` | Effective group ceiling in bytes, capped by the process budget when enabled; `0` means no group ceiling | +| `max_active_queries` | `INT` | Concurrent admission limit; `2147483647` represents unlimited | +| `max_queued_queries` | `INT` | Queue capacity; `2147483647` represents unlimited, and `0` disables queueing | +| `queue_timeout_millis` | `LONG` | Effective admission timeout in milliseconds | +| `cpu_weight` | `INT` | Relative scheduling weight | +| `active_queries` | `LONG` | Queries currently holding admission slots | +| `queued_queries` | `LONG` | Queries waiting for admission | +| `oldest_queue_wait_millis` | `LONG` | Age of the oldest admission waiter in milliseconds; `0` when none | +| `memory_used_bytes` | `LONG` | Published tracked native query memory in bytes | +| `cpu_nanos_total` | `LONG` | CPU nanoseconds measured by managed scheduling | +| `cpu_wait_nanos_total` | `LONG` | Cumulative query waiting time for CPU, in nanoseconds | +| `admission_rejections` | `LONG` | Cumulative queue-full rejections | +| `admission_timeouts` | `LONG` | Cumulative admission timeouts | + +```questdb-sql +SELECT name, memory_limit_bytes, memory_used_bytes, active_queries, queued_queries +FROM resource_groups() +ORDER BY name; +``` + +Counters describe the current runtime and reset on restart or group recreation. +Worker-local memory deltas can be temporarily unpublished. The single-group +dispatch path does not sample CPU, so `cpu_nanos_total` does not cover all query +CPU use. Dropped groups disappear from this table while their existing queries +finish using retained state. + +For the corresponding +[Prometheus metrics](/docs/operations/logging-metrics/#resource-group-metrics), +an unlimited group memory ceiling is `0` in both interfaces; it does not remove +principal-specific, instance-default single-query or process memory limits. + ## sleep() -Pauses the query for the given number of seconds, then returns the timestamp -at which it resumed. Intended for testing and demonstration, for example to -hold a query open while inspecting -[`query_activity()`](#query_activity) from another session. +Pauses the query for the given number of seconds, then returns the timestamp at +which it resumed. Intended for testing and demonstration, for example to hold a +query open while inspecting [`query_activity()`](#query_activity) from another +session. -`sleep()` does not hold a worker thread while it waits, so many concurrent -calls can be parked at once without exhausting the shared worker pool. +`sleep()` does not hold a worker thread while it waits, so many concurrent calls +can be parked at once without exhausting the shared worker pool. **Arguments:** @@ -502,8 +613,8 @@ SELECT * FROM sleep(1); :::note -Storage policies — and the `storage_policies` view — are available in -**QuestDB Enterprise** only. +Storage policies — and the `storage_policies` view — are available in **QuestDB +Enterprise** only. ::: @@ -532,9 +643,9 @@ SELECT * FROM storage_policies; - TTL values are rendered in two units: `h` for hours and `m` for **months**. Hour-, day-, and week-based durations are stored as hours (e.g. `3 DAYS` → `72h`, `1 WEEK` → `168h`). Month- and year-based durations are stored as - months (e.g. `1 MONTH` → `1m`, `1 YEAR` → `12m`). Despite the visual - collision with "minute", `m` in this view is **months**; QuestDB's duration - shorthand has no unit for minutes. + months (e.g. `1 MONTH` → `1m`, `1 YEAR` → `12m`). Despite the visual collision + with "minute", `m` in this view is **months**; QuestDB's duration shorthand + has no unit for minutes. - An unset stage renders as `0h`, not blank. **Example:** @@ -557,11 +668,15 @@ stage set and has been temporarily disabled. Every unset stage renders as `0h`. :::note -[Cold storage](/docs/concepts/cold-storage/) and the `table_cold_partitions()` function are available in **QuestDB Enterprise** only. +[Cold storage](/docs/concepts/cold-storage/) and the `table_cold_partitions()` +function are available in **QuestDB Enterprise** only. ::: -`table_cold_partitions('tableName')` returns one row per partition in the table's remote manifest, with the state of its object in the store. Use it to follow a partition through upload and sealing, and to find partitions that are not progressing. +`table_cold_partitions('tableName')` returns one row per partition in the +table's remote manifest, with the state of its object in the store. Use it to +follow a partition through upload and sealing, and to find partitions that are +not progressing. **Arguments:** @@ -609,9 +724,16 @@ WHERE state = 'pending'; **Notes:** -- The cold storage manager answers from its own in-memory view. A refresher answers from its mirrored copy, which it updates when the catalog generation changes, so the two can differ briefly. -- While an instance is transitioning between the manager and refresher roles, the function returns zero rows rather than blocking. Check the live role with [`SWITCH COLD STORAGE STATUS`](/docs/query/sql/switch-cold-storage-role/). -- The function reflects the remote manifest, not local partition state. Use [`SHOW PARTITIONS`](/docs/query/sql/show/#show-partitions) or [`table_partitions()`](#table_partitions) to see whether a partition is actually being served remotely. +- The cold storage manager answers from its own in-memory view. A refresher + answers from its mirrored copy, which it updates when the catalog generation + changes, so the two can differ briefly. +- While an instance is transitioning between the manager and refresher roles, + the function returns zero rows rather than blocking. Check the live role with + [`SWITCH COLD STORAGE STATUS`](/docs/query/sql/switch-cold-storage-role/). +- The function reflects the remote manifest, not local partition state. Use + [`SHOW PARTITIONS`](/docs/query/sql/show/#show-partitions) or + [`table_partitions()`](#table_partitions) to see whether a partition is + actually being served remotely. ## table_columns @@ -634,14 +756,14 @@ Returns a `table` with the following columns: - `symbolCached` - whether this `symbol` column is cached - `symbolCapacity` - how many distinct values this column of `symbol` type is expected to have -- `symbolTableSize` - current number of distinct values stored in this - `symbol` column's table +- `symbolTableSize` - current number of distinct values stored in this `symbol` + column's table - `designated` - if this is set as the designated timestamp column for this table - `upsertKey` - if this column is a part of UPSERT KEYS list for table [deduplication](/docs/concepts/deduplication) -- `indexType` - the [index type](/docs/concepts/deep-dive/indexes/) - (`POSTING`, `POSTING DELTA`, `POSTING EF`, `BITMAP`, or empty) +- `indexType` - the [index type](/docs/concepts/deep-dive/indexes/) (`POSTING`, + `POSTING DELTA`, `POSTING EF`, `BITMAP`, or empty) - `indexInclude` - comma-separated names of columns included in a [posting index's](/docs/concepts/deep-dive/posting-index/) covering sidecar @@ -719,16 +841,16 @@ Returns a table with the following columns: partition will contain the `.detached` extension) - `attachable` - _BOOLEAN_, true if the partition is detached and can be attached (`name` of the partition will contain the `.attachable` extension) -- `hasParquetGenerated` - _BOOLEAN_, true if a Parquet copy of the partition - has been generated. Set by either +- `hasParquetGenerated` - _BOOLEAN_, true if a Parquet copy of the partition has + been generated. Set by either [manual Parquet conversion](/docs/concepts/parquet/#in-place-conversion) (`ALTER TABLE ... CONVERT PARTITION TO PARQUET`) or by a [storage policy](/docs/concepts/storage-policy/)'s `TO PARQUET` stage (Enterprise) - `isParquet` - _BOOLEAN_, true if the partition is stored in Parquet format: - the native files have been removed and reads are served from the Parquet - file. Set the same way as `hasParquetGenerated`: either manually or by a - storage policy's `TO PARQUET` stage + the native files have been removed and reads are served from the Parquet file. + Set the same way as `hasParquetGenerated`: either manually or by a storage + policy's `TO PARQUET` stage - `parquetFileSize` - _LONG_, size in bytes of the partition's `data.parquet` file when `hasParquetGenerated` or `isParquet` is true; `-1` otherwise - `seqTxn` - _LONG_, WAL transaction version the partition was last written at @@ -889,7 +1011,7 @@ Returns a `table` with the following columns: ::: -### Table metrics (table_* prefix) +### Table metrics (table\_\* prefix) | Column | Type | Description | | ----------------------------- | --------- | ----------------------------------------------------------------------------------------- | @@ -912,16 +1034,18 @@ Returns a `table` with the following columns: | `table_merge_rate_p99` | LONG | Throughput that 99% of jobs **exceeded** (slowest 1%) | | `table_merge_rate_max` | LONG | Maximum throughput in rows/second | -Write amplification measures O3 (out-of-order) merge overhead as `physicalRowsWritten / logicalRows`. -A ratio of `1.0` means no amplification. Higher values indicate O3 merge overhead. +Write amplification measures O3 (out-of-order) merge overhead as +`physicalRowsWritten / logicalRows`. A ratio of `1.0` means no amplification. +Higher values indicate O3 merge overhead. :::note -Merge rate P99 shows the *lowest* throughput (worst performance), not the highest. +Merge rate P99 shows the _lowest_ throughput (worst performance), not the +highest. ::: -### WAL metrics (wal_* prefix) +### WAL metrics (wal\_\* prefix) | Column | Type | Description | | --------------------------------- | --------- | ------------------------------------------------------------- | @@ -935,9 +1059,10 @@ Merge rate P99 shows the *lowest* throughput (worst performance), not the highes | `wal_tx_size_p99` | LONG | 99th percentile transaction size in rows | | `wal_tx_size_max` | LONG | Maximum transaction size in rows | -### Replica metrics (replica_* prefix) +### Replica metrics (replica\_\* prefix) -These columns are populated on **replicas only** via replication download tracking: +These columns are populated on **replicas only** via replication download +tracking: | Column | Type | Description | | ------------------------ | ------- | ------------------------------------------------------------------------ | @@ -954,17 +1079,31 @@ On primary instances, these columns will be `0` or `false`. These values are approximations, not precise real-time metrics: -- **Null when not tracked**: Values are `null` for tables not written to since server start, or evicted from the tracker -- **Writer stats updated on pool return**: `table_row_count`, `table_last_write_timestamp`, `table_txn` are captured when TableWriter returns to the pool, not on every commit. A writer held for a long time won't update these columns until released. +- **Null when not tracked**: Values are `null` for tables not written to since + server start, or evicted from the tracker +- **Writer stats updated on pool return**: `table_row_count`, + `table_last_write_timestamp`, `table_txn` are captured when TableWriter + returns to the pool, not on every commit. A writer held for a long time won't + update these columns until released. - **WAL stats updated in real-time**: - - On WAL commit: `wal_pending_row_count` (incremented), `wal_txn`, `wal_max_timestamp`, `wal_tx_size_*` histogram - - On WAL apply: `wal_pending_row_count` (decremented), `wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, `table_write_amp_*`, `table_merge_rate_*` -- **LRU eviction**: Tracker maintains bounded memory (default 1000 tables). Least recently written tables are evicted when capacity is exceeded -- **Startup hydration**: Values are hydrated from table metadata (`TxReader`) on startup, but diverge as writes occur - -**Non-WAL tables**: `wal_txn`, `wal_max_timestamp`, `wal_pending_row_count`, `wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, `table_memory_pressure_level`, and histogram columns are `null` or `0`. - -**WAL tables**: All columns populated when tracked. `wal_max_timestamp` reflects the max data timestamp from the WAL transaction, not wall-clock time. `table_min_timestamp` and `table_max_timestamp` reflect the actual data range in the table after WAL merge. + - On WAL commit: `wal_pending_row_count` (incremented), `wal_txn`, + `wal_max_timestamp`, `wal_tx_size_*` histogram + - On WAL apply: `wal_pending_row_count` (decremented), + `wal_dedup_row_count_since_start`, `table_min_timestamp`, + `table_max_timestamp`, `table_write_amp_*`, `table_merge_rate_*` +- **LRU eviction**: Tracker maintains bounded memory (default 1000 tables). + Least recently written tables are evicted when capacity is exceeded +- **Startup hydration**: Values are hydrated from table metadata (`TxReader`) on + startup, but diverge as writes occur + +**Non-WAL tables**: `wal_txn`, `wal_max_timestamp`, `wal_pending_row_count`, +`wal_dedup_row_count_since_start`, `table_min_timestamp`, `table_max_timestamp`, +`table_memory_pressure_level`, and histogram columns are `null` or `0`. + +**WAL tables**: All columns populated when tracked. `wal_max_timestamp` reflects +the max data timestamp from the WAL transaction, not wall-clock time. +`table_min_timestamp` and `table_max_timestamp` reflect the actual data range in +the table after WAL merge. ### Configuration @@ -1206,9 +1345,8 @@ concurrent waiters is not bounded by the shared worker pool. **Arguments:** -- `tableName` (`string`): name of the table to wait for. Must be a constant, - not a column reference. On a non-WAL table the call returns `true` - immediately. +- `tableName` (`string`): name of the table to wait for. Must be a constant, not + a column reference. On a non-WAL table the call returns `true` immediately. - `seqTxn` (optional, `long`): the sequencer transaction to wait for. When omitted, the call captures the table's current `seqTxn` when it starts and waits for that, which is what you want after your own write. @@ -1218,8 +1356,8 @@ concurrent waiters is not bounded by the shared worker pool. Returns `boolean`. `true` once the writer has caught up. Throws if the table is dropped while the call is waiting, and if the table -becomes [suspended](/docs/query/sql/alter-table-resume-wal/), since a -suspended table would otherwise never catch up. +becomes [suspended](/docs/query/sql/alter-table-resume-wal/), since a suspended +table would otherwise never catch up. **Examples:** @@ -1243,11 +1381,10 @@ SELECT wait_wal_table('trades', 42); :::note -For monitoring and observability, use [`tables()`](#tables) instead. -`tables()` provides all the same information plus additional metrics -(pending rows, memory pressure, deduplication stats, throughput histograms), -and is fully in-memory. `wal_tables()` reads from disk and is less suitable -for frequent polling. +For monitoring and observability, use [`tables()`](#tables) instead. `tables()` +provides all the same information plus additional metrics (pending rows, memory +pressure, deduplication stats, throughput histograms), and is fully in-memory. +`wal_tables()` reads from disk and is less suitable for frequent polling. ::: @@ -1265,11 +1402,13 @@ Returns a `table` including the following information: - `name` - table or materialized view name - `suspended` - suspended status flag -- `writerTxn` - the last committed transaction in TableWriter (equivalent to `table_txn` in `tables()`) +- `writerTxn` - the last committed transaction in TableWriter (equivalent to + `table_txn` in `tables()`) - `writerLagTxnCount` - the number of transactions that are kept invisible when writing to the table; these transactions will be eventually moved to the table data and become visible for readers (equivalent to `wal_txn - table_txn`) -- `sequencerTxn` - the last committed transaction in the sequencer (equivalent to `wal_txn` in `tables()`) +- `sequencerTxn` - the last committed transaction in the sequencer (equivalent + to `wal_txn` in `tables()`) **Examples:** diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 2c4c83c2a..6ac7b5e11 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -635,6 +635,11 @@ module.exports = { type: "doc", label: "Cold Storage", }, + { + id: "concepts/resource-groups", + type: "doc", + label: "Resource Groups", + }, "concepts/write-ahead-log", ], }, @@ -698,6 +703,11 @@ module.exports = { "configuration/postgres-wire-protocol", "configuration/qwp", "configuration/database-replication", + { + id: "configuration/resource-groups", + type: "doc", + label: "Resource groups", + }, "configuration/shared-workers", "configuration/storage-policy", "configuration/telemetry", @@ -787,6 +797,11 @@ module.exports = { type: "doc", label: "Cold storage", }, + { + id: "operations/resource-groups", + type: "doc", + label: "Resource groups", + }, "operations/logging-metrics", "operations/monitoring-alerting", "operations/data-retention",