diff --git a/documentation/architecture/query-engine.md b/documentation/architecture/query-engine.md index 260d35b61..c4071a693 100644 --- a/documentation/architecture/query-engine.md +++ b/documentation/architecture/query-engine.md @@ -59,8 +59,13 @@ to process data in table page frames for better CPU use. multi-core fashion. Some queries, for example those involving an index, are executed on a single thread. Other queries, like those involving `GROUP BY` and `SAMPLE BY`, execute a pipeline with some single-threaded stages and some multi-threaded stages to avoid slow downs when groups are unbalanced. -- **Worker pools:** QuestDB allows to configure different pools for specialized functions, like -parsing incoming data, applying WAL file changes, handling PostgreSQL-Wire protocol, or responding to HTTP connections. By default, most tasks are handled by a shared worker pool. +- **Worker pools:** QuestDB splits work across three thread pools that are sized +independently: a network pool for HTTP, PostgreSQL and ILP server I/O, a query pool +for parallel query execution, and a write pool for WAL apply jobs, table writes and +materialized view refresh. Because query execution has its own pool, a burst of +ingestion does not starve readers of CPU. Individual subsystems can still be given +dedicated threads on top of this. See +[shared workers](/docs/configuration/shared-workers/) for sizing. - **Query plan caching:** The system caches query plans for reuse within the same connection. (Query results are not diff --git a/documentation/deployment/hetzner.md b/documentation/deployment/hetzner.md index 6a5b4d35f..92ad6cc6e 100644 --- a/documentation/deployment/hetzner.md +++ b/documentation/deployment/hetzner.md @@ -337,7 +337,8 @@ questdb01$ docker restart questdb For comprehensive configuration options, see the [Configuration reference](/docs/configuration/overview/) documentation. Common production settings include: - **Connection limits**: `pg.connection.pool.size` -- **Memory settings**: `shared.worker.count` +- **Worker threads**: `shared.network.worker.count`, `shared.query.worker.count`, + `shared.write.worker.count` - **Security**: [TLS configuration](/docs/security/tls/) - **Authentication**: [RBAC setup](/docs/security/rbac/) diff --git a/documentation/getting-started/capacity-planning.md b/documentation/getting-started/capacity-planning.md index 3548de46b..af7a6b3c4 100644 --- a/documentation/getting-started/capacity-planning.md +++ b/documentation/getting-started/capacity-planning.md @@ -188,12 +188,36 @@ for other database processes to use. ### CPU cores -By default, QuestDB tries to use all available CPU cores. +By default, QuestDB tries to use all available CPU cores. Assuming that the disk +is not bottlenecked on IOPS, the throughput of read-only queries scales +proportionally with the number of available cores, so a machine with more cores +will provide better query performance. + +Work is divided across three independently sized thread pools, which is what you +tune when one class of work needs more CPU than another: + +| Pool | Handles | Setting | +| ---- | ------- | ------- | +| Network | HTTP, PostgreSQL and ILP server I/O | `shared.network.worker.count` | +| Query | Parallel query execution — filters, `GROUP BY` | `shared.query.worker.count` | +| Write | WAL apply, table writes, materialized view refresh, housekeeping | `shared.write.worker.count` | + +Each pool defaults to roughly the CPU count, so the pools oversubscribe the +machine by design and the OS scheduler arbitrates between them. That is a +reasonable starting point. Move capacity deliberately when a workload is +lopsided: + +- **Read-heavy** — raise `shared.query.worker.count` and lower + `shared.network.worker.count`, provided connection counts are modest. +- **Ingestion-heavy** — raise `shared.write.worker.count` so WAL apply keeps up + with incoming data. A growing WAL apply backlog is the signal to watch. +- **Many concurrent clients** — raise `shared.network.worker.count`, since I/O + rather than query execution is the constraint. + +On small machines, cutting a pool too far can leave a subsystem unable to make +progress; keep at least two threads in each. [The guide on shared worker configuration](/docs/configuration/shared-workers/) -explains how to change the default settings. Assuming that the disk is not -bottlenecked on IOPS, the throughput of read-only queries scales proportionally -with the number of available cores. As a result, a machine with more cores will -provide better query performance. +covers the defaults and the per-pool affinity settings. ### Writer page size