From dae877b516a695c2b3e6a9ba18bd1b7a7352a70f Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 3 Jul 2026 01:28:49 +0100 Subject: [PATCH 1/5] Correct storage policy docs against source behavior - SET STORAGE POLICY fully replaces the policy; omitted stages are cleared, not merged (docs claimed partial updates) - TO REMOTE is accepted and stored but not enforced, not rejected at parse time; only DROP REMOTE is rejected - storage_policies renders unset stages as 0h, not blank - Fix Enterprise TTL rejection message; CREATE TABLE ... TTL is translated to DROP LOCAL, not rejected - Config: check.interval 5m (was 15m), worker.count 4 (was 2); drop nonexistent writer.wait.timeout; document worker.* tuning props - SHOW CREATE TABLE does not render a disabled policy - meta.md: year TTLs are stored as months (1 YEAR -> 12m) Co-Authored-By: Claude Opus 4.8 (1M context) --- documentation/concepts/storage-policy.md | 90 +++++++++++-------- documentation/concepts/ttl.md | 2 +- documentation/configuration/storage-policy.md | 31 +++++-- .../getting-started/enterprise-quick-start.md | 12 +-- documentation/query/functions/meta.md | 29 +++--- .../sql/alter-table-set-storage-policy.md | 40 +++++---- .../query/sql/alter-table-set-ttl.md | 2 +- documentation/query/sql/create-table.md | 17 ++-- documentation/query/sql/show.md | 6 +- 9 files changed, 135 insertions(+), 94 deletions(-) diff --git a/documentation/concepts/storage-policy.md b/documentation/concepts/storage-policy.md index a0ecc70cba..1ae4d774a0 100644 --- a/documentation/concepts/storage-policy.md +++ b/documentation/concepts/storage-policy.md @@ -16,13 +16,14 @@ Converting a partition to Parquet removes its native files and serves reads dire :::info -Storage policies currently operate **locally only**. Parquet files are not -automatically uploaded to object storage, so the `TO REMOTE` and `DROP REMOTE` -clauses are reserved syntax — they are rejected at SQL parse time with -`'TO REMOTE' is not supported yet` and `'DROP REMOTE' is not supported yet`. -Accordingly, the `to_remote` and `drop_remote` columns in the -[`storage_policies`](/docs/query/functions/meta/#storage_policies) view are -always blank in the current release; they are kept for forward compatibility. +Storage policies currently operate **locally only**. `TO PARQUET` and +`DROP LOCAL` are the enforced stages. `TO REMOTE` is accepted and stored but +not yet enforced, so setting it has no effect for now: no upload to object +storage happens yet. `DROP REMOTE` is not yet supported and is rejected at SQL +parse time with `'DROP REMOTE' is not supported yet`. In the +[`storage_policies`](/docs/query/functions/meta/#storage_policies) view the +`drop_remote` column is therefore always `0h`, and `to_remote` reads `0h` +unless you set a `TO REMOTE` value (which is stored but has no effect yet). Object storage integration will be added in a future release. ::: @@ -43,9 +44,9 @@ stage in the partition lifecycle: | Setting | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `TO PARQUET` | Convert the partition from native binary format to Parquet. The native files are removed and reads are served from the Parquet file | -| `TO REMOTE` | _Reserved._ Will upload the Parquet file to object storage when remote upload is supported | +| `TO REMOTE` | Accepted and stored but **not yet enforced**; no upload happens yet. Reserved for future object storage upload | | `DROP LOCAL` | Remove all local data (native or Parquet) | -| `DROP REMOTE` | _Reserved._ Will remove the Parquet file from object storage when remote upload is supported | +| `DROP REMOTE` | _Not yet supported._ Rejected at parse time with `'DROP REMOTE' is not supported yet`. Reserved for future object storage removal | All settings are optional. Use only the ones relevant to your use case. All TTL values must be **positive**; `0` is rejected. @@ -87,7 +88,7 @@ the wall-clock cap for both TTL and storage policy evaluation. See [TTL § Reference time](/docs/concepts/ttl/#reference-time) for the rationale and the data-loss hazard of disabling the cap. -QuestDB checks storage policies periodically (every 15 minutes by default) and +QuestDB checks storage policies periodically (every 5 minutes by default) and processes eligible partitions automatically. ## Storage policy vs TTL @@ -102,8 +103,10 @@ you are already familiar with TTL, this comparison is the fastest way in: | **Parquet conversion** | No | Yes (automatic local conversion) | | **Granularity** | Single retention window | Up to four independent TTL stages | -In QuestDB Enterprise, `CREATE TABLE ... TTL` and `ALTER TABLE SET TTL` are -deprecated. Use storage policies instead: +In QuestDB Enterprise, use storage policies instead of TTL. On a regular table, +`ALTER TABLE SET TTL` with a non-zero value is rejected, while +`CREATE TABLE ... TTL` is accepted only for backward compatibility and is +translated into a `STORAGE POLICY(DROP LOCAL ...)`: ```questdb-sql -- Instead of: @@ -115,10 +118,11 @@ ALTER TABLE trades SET STORAGE POLICY(DROP LOCAL 30d); :::note -If a table already has a TTL set, you must clear it with -`ALTER TABLE SET TTL 0` before setting a storage policy. `SET TTL 0` is the -only `SET TTL` value Enterprise accepts; any non-zero value is rejected with -`TTL settings are deprecated, please, create a storage policy instead`. +If a table already has a TTL set, clear it with `ALTER TABLE SET TTL 0` before +setting a storage policy; otherwise `SET STORAGE POLICY` is rejected with +`Cannot set storage policy, please, remove TTL settings`. On Enterprise tables +`SET TTL 0` is the only accepted `SET TTL` value; any non-zero value is rejected +with `TTL is not supported on Enterprise tables; use a storage policy instead`. ::: @@ -145,7 +149,8 @@ ALTER TABLE trades SET STORAGE POLICY( ); ``` -Only the specified settings are changed. Omitted settings remain unchanged. +`SET STORAGE POLICY` replaces the policy in full: every stage you omit is +cleared, not preserved. To keep a stage, restate it in the same statement. For full syntax details, see [ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/). @@ -173,11 +178,11 @@ TO PARQUET <= DROP LOCAL TO REMOTE <= DROP LOCAL <= DROP REMOTE ``` -`TO PARQUET` and `TO REMOTE` are **independent** — neither has to precede the -other. If `TO REMOTE` fires before `TO PARQUET`, both the native and Parquet -copies are written locally and reads continue to be served from the native -format until `TO PARQUET` removes the native files. All TTL values must be -positive — `0` is rejected. +`TO PARQUET` and `TO REMOTE` are **independent**: neither has to precede the +other. Once remote upload is enforced, a `TO REMOTE` that runs before +`TO PARQUET` would keep both the native and Parquet copies on local disk, with +reads served from the native format until `TO PARQUET` removes the native +files. All TTL values must be positive; `0` is rejected. ## Disabling and enabling @@ -214,21 +219,19 @@ SELECT * FROM storage_policies; | table_dir_name | to_parquet | to_remote | drop_local | drop_remote | status | last_updated | | -------------- | ---------- | --------- | ---------- | ----------- | ------ | --------------------------- | -| trades~12 | 72h | | 1m | | A | 2025-01-15T10:30:00.000000Z | - -- TTL values are rendered in just two units: `h` for hours and `m` for - **months**. Hour-, day-, and week-based durations are normalized to hours - when stored, so a `3 DAYS` TTL appears as `72h` and `1 WEEK` appears as - `168h`. Month-based durations keep the lowercase `m` suffix — **`1m` in - this view means one month, not one minute**; QuestDB's duration shorthand - has no unit for minutes +| trades~12 | 72h | 0h | 1m | 0h | A | 2025-01-15T10:30:00.000000Z | + +- TTL values are rendered in two units: `h` for hours and `m` for **months**. + Hour-, day-, and week-based durations are stored as hours, so a `3 DAYS` TTL + appears as `72h` and `1 WEEK` as `168h`. Month- and year-based durations are + stored as months, so `1 MONTH` appears as `1m` and `1 YEAR` as `12m`. In this + view **`m` means months, not minutes**; QuestDB's duration shorthand has no + unit for minutes - Status `A` means active; `D` means disabled (see [Disabling and enabling](#disabling-and-enabling)) -- Unset stages appear blank. `to_remote` and `drop_remote` are **always blank - in the current release** because `TO REMOTE` and `DROP REMOTE` are rejected - at SQL parse time with `'TO REMOTE' is not supported yet` and - `'DROP REMOTE' is not supported yet`; the columns are kept for forward - compatibility +- An unset stage renders as `0h`, not blank. Because `DROP REMOTE` is rejected + at parse time, `drop_remote` is always `0h`; `to_remote` reads `0h` unless a + `TO REMOTE` value is set (stored, but not yet enforced) For the full column reference and types, see [`storage_policies`](/docs/query/functions/meta/#storage_policies). @@ -255,14 +258,19 @@ values: | Property | Default | Description | | ------------------------------------- | ------------------ | ---------------------------------------------------------------- | -| `storage.policy.check.interval` | `15m` (15 min) | How often QuestDB scans for partitions to process | +| `storage.policy.check.interval` | `5m` (5 min) | How often QuestDB scans for partitions to process | | `storage.policy.retry.interval` | `1m` (1 min) | Retry interval for failed tasks | | `storage.policy.max.reschedule.count` | `20` | Maximum retries before abandoning a task | -| `storage.policy.writer.wait.timeout` | `30s` (30 sec) | Timeout for acquiring the table writer | -| `storage.policy.worker.count` | `2` | Number of storage policy worker threads (0 disables the feature) | +| `storage.policy.worker.count` | `4` | Number of storage policy worker threads (0 disables the feature) | | `storage.policy.worker.affinity` | `-1` (no affinity) | CPU affinity for each worker thread (comma-separated list) | | `storage.policy.worker.sleep.timeout` | `100ms` | Sleep duration when worker has no tasks | +See +[Storage policy configuration](/docs/configuration/storage-policy/) for the +complete list, including additional worker-pool tuning properties +(`worker.haltOnError`, `worker.nap.threshold`, `worker.sleep.threshold`, and +`worker.yield.threshold`). + ## Permissions Storage policy operations require specific permissions in QuestDB Enterprise: @@ -307,7 +315,7 @@ WHERE table_dir_name LIKE 'trades%'; | -------------- | ---------- | ---------- | ------ | | trades~12 | 72h | 1m | A | -```questdb-sql title="3. Modify one stage (others remain unchanged)" +```questdb-sql title="3. Replace the policy (omitted stages are cleared)" ALTER TABLE trades SET STORAGE POLICY(TO PARQUET 1d); ``` @@ -324,6 +332,10 @@ CREATE TABLE 'trades' ( STORAGE POLICY(TO PARQUET 1 DAY) WAL; ``` +The `DROP LOCAL 1 MONTH` stage from step 1 is gone: step 3 restated only +`TO PARQUET`, and `SET STORAGE POLICY` replaces the whole policy rather than +merging into it. + ```questdb-sql title="5. Temporarily suspend the policy (e.g. during a backfill)" ALTER TABLE trades DISABLE STORAGE POLICY; -- status in storage_policies changes to 'D' diff --git a/documentation/concepts/ttl.md b/documentation/concepts/ttl.md index d7b9e34aba..f544a70266 100644 --- a/documentation/concepts/ttl.md +++ b/documentation/concepts/ttl.md @@ -13,7 +13,7 @@ window - no cron jobs or manual cleanup required. **QuestDB Enterprise: TTL is superseded by [Storage Policy](/docs/concepts/storage-policy/).** Enterprise rejects any non-zero `SET TTL` with -`TTL settings are deprecated, please, create a storage policy instead`. +`TTL is not supported on Enterprise tables; use a storage policy instead`. Storage policies extend TTL with graduated lifecycle management (convert to Parquet, then drop) and are the recommended retention primitive for Enterprise users. The rest of this page describes TTL behavior on QuestDB Open Source diff --git a/documentation/configuration/storage-policy.md b/documentation/configuration/storage-policy.md index e97d19bd67..8b0db13201 100644 --- a/documentation/configuration/storage-policy.md +++ b/documentation/configuration/storage-policy.md @@ -18,7 +18,7 @@ For details, see the ## storage.policy.check.interval -- **Default**: `15m` +- **Default**: `5m` - **Reloadable**: no How often QuestDB scans for partitions to process. @@ -47,11 +47,32 @@ CPU affinity for each storage policy worker thread (comma-separated list). ## storage.policy.worker.count -- **Default**: `2` +- **Default**: `4` - **Reloadable**: no Number of storage policy worker threads. Setting to `0` disables the feature. +## storage.policy.worker.haltOnError + +- **Default**: `false` +- **Reloadable**: no + +Whether a storage policy worker thread halts when it hits an unhandled error. + +## storage.policy.worker.nap.threshold + +- **Default**: `100` +- **Reloadable**: no + +Number of idle worker-loop iterations before a storage policy worker naps. + +## storage.policy.worker.sleep.threshold + +- **Default**: `500` +- **Reloadable**: no + +Number of idle worker-loop iterations before a storage policy worker sleeps. + ## storage.policy.worker.sleep.timeout - **Default**: `100ms` @@ -59,9 +80,9 @@ Number of storage policy worker threads. Setting to `0` disables the feature. Sleep duration when a storage policy worker has no tasks to process. -## storage.policy.writer.wait.timeout +## storage.policy.worker.yield.threshold -- **Default**: `30s` +- **Default**: `10` - **Reloadable**: no -Timeout for acquiring the table writer during storage policy operations. +Number of idle worker-loop iterations before a storage policy worker yields. diff --git a/documentation/getting-started/enterprise-quick-start.md b/documentation/getting-started/enterprise-quick-start.md index 59c3a5b4f3..85a3b3e985 100644 --- a/documentation/getting-started/enterprise-quick-start.md +++ b/documentation/getting-started/enterprise-quick-start.md @@ -479,16 +479,16 @@ too — on a schedule you define. This supersedes plain TTL in Enterprise, where ### Migrating from TTL when upgrading from OSS Tables that were created in OSS keep their existing `TTL` setting after you -upgrade to Enterprise — no data is lost at upgrade time. However, Enterprise -rejects any **new** `TTL` changes on tables (both `CREATE TABLE ... TTL` and -`ALTER TABLE SET TTL `) with: +upgrade to Enterprise — no data is lost at upgrade time. On an existing table, +`ALTER TABLE SET TTL` with a non-zero value is then rejected with: ``` -TTL settings are deprecated, please, create a storage policy instead +TTL is not supported on Enterprise tables; use a storage policy instead ``` -Materialized views are not affected: they continue to use `TTL` for retention -in Enterprise. +(`CREATE TABLE ... TTL` is still accepted for backward compatibility: Enterprise +translates it into a `STORAGE POLICY(DROP LOCAL ...)`.) Materialized views are +not affected: they continue to use `TTL` for retention in Enterprise. To move a legacy table from `TTL` to a storage policy: diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 77ebe6ac84..b345a38d66 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -307,21 +307,22 @@ SELECT * FROM storage_policies; | Column | Type | Description | |--------|------|-------------| | `table_dir_name` | _STRING_ | Directory name of the table the policy is attached to. Matches the `table_dir_name` column in [`tables()`](#tables). | -| `to_parquet` | _STRING_ | TTL for the `TO PARQUET` stage (e.g. `72h`, `1m`). Blank when the stage is not configured. | -| `to_remote` | _STRING_ | Reserved — always blank in the current release. The `TO REMOTE` clause is rejected at SQL parse time with `'TO REMOTE' is not supported yet`. The column is kept for forward compatibility. | -| `drop_local` | _STRING_ | TTL for the `DROP LOCAL` stage. Blank when the stage is not configured. | -| `drop_remote` | _STRING_ | Reserved — always blank in the current release. The `DROP REMOTE` clause is rejected at SQL parse time with `'DROP REMOTE' is not supported yet`. The column is kept for forward compatibility. | +| `to_parquet` | _STRING_ | TTL for the `TO PARQUET` stage (e.g. `72h`, `1m`). `0h` when the stage is not configured. | +| `to_remote` | _STRING_ | TTL for the `TO REMOTE` stage. Accepted and stored but not yet enforced, so setting it has no effect for now. `0h` when not configured. | +| `drop_local` | _STRING_ | TTL for the `DROP LOCAL` stage. `0h` when the stage is not configured. | +| `drop_remote` | _STRING_ | Reserved for future object storage removal. `DROP REMOTE` is rejected at parse time with `'DROP REMOTE' is not supported yet`, so this column is always `0h`. | | `status` | _CHAR_ | Policy status. `A` = active (the policy is being enforced), `D` = disabled (via [`ALTER TABLE DISABLE STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/)). | | `last_updated` | _TIMESTAMP_ | Timestamp of the most recent change to the policy definition (not the last time partitions were processed). | **Notes on TTL formatting:** -- TTL values are rendered in just two units: `h` for hours and `m` for - **months**. Durations written in the DDL as days, weeks, or years are - normalized to hours when stored (e.g., `3 DAYS` → `72h`, `1 WEEK` → - `168h`). Month-based durations are stored and rendered with the lowercase - `m` suffix — despite the visual collision with "minute", `m` in this view - is **months**, and QuestDB's duration shorthand has no unit for minutes. +- TTL values are rendered in two units: `h` for hours and `m` for **months**. + Hour-, day-, and week-based durations are stored as hours (e.g. `3 DAYS` → + `72h`, `1 WEEK` → `168h`). Month- and year-based durations are stored as + months (e.g. `1 MONTH` → `1m`, `1 YEAR` → `12m`). Despite the visual + collision with "minute", `m` in this view is **months**; QuestDB's duration + shorthand has no unit for minutes. +- An unset stage renders as `0h`, not blank. **Example:** @@ -331,13 +332,13 @@ SELECT * FROM storage_policies; | table_dir_name | to_parquet | to_remote | drop_local | drop_remote | status | last_updated | |----------------|------------|-----------|------------|-------------|--------|--------------| -| trades~12 | 72h | | 1m | | A | 2025-01-15T10:30:00.000000Z | -| metrics~18 | 168h | | | | D | 2025-01-14T09:15:42.000000Z | +| trades~12 | 72h | 0h | 1m | 0h | A | 2025-01-15T10:30:00.000000Z | +| metrics~18 | 168h | 0h | 0h | 0h | D | 2025-01-14T09:15:42.000000Z | The first row is a policy with two active stages (3-day Parquet conversion and 1-month local drop) and is currently enforced. The second row has only the -`TO PARQUET` stage set and has been temporarily disabled. The `to_remote` and -`drop_remote` columns are reserved and always blank in the current release. +`TO PARQUET` stage set and has been temporarily disabled. Unset stages, and the +not-yet-enforced `to_remote` and `drop_remote` columns, all render as `0h`. ## table_columns diff --git a/documentation/query/sql/alter-table-set-storage-policy.md b/documentation/query/sql/alter-table-set-storage-policy.md index 5346af177c..56f081671f 100644 --- a/documentation/query/sql/alter-table-set-storage-policy.md +++ b/documentation/query/sql/alter-table-set-storage-policy.md @@ -28,8 +28,8 @@ ALTER TABLE table_name SET STORAGE POLICY( ); ``` -Only the specified settings are changed. Omitted settings retain their current -values. +`SET STORAGE POLICY` replaces the policy as a whole. Any stage you do not list +is cleared, not preserved, so restate every stage you want to keep. ### Enable or disable a storage policy @@ -56,20 +56,19 @@ transition from native format to Parquet and eventually get removed: | Setting | Effect | |---------|--------| | `TO PARQUET ` | Convert partition from native format to Parquet locally. The native files are removed and reads are served from the Parquet file | -| `TO REMOTE ` | _Reserved._ Will upload the partition to object storage when remote upload is supported | +| `TO REMOTE ` | Accepted and stored but not yet enforced; no upload happens yet. Reserved for future object storage upload | | `DROP LOCAL ` | Remove all local copies of the partition | -| `DROP REMOTE ` | _Reserved._ Will remove the partition from object storage when remote upload is supported | +| `DROP REMOTE ` | _Not yet supported._ Rejected at parse time with `'DROP REMOTE' is not supported yet`. Reserved for future object storage removal | :::info -`TO REMOTE` and `DROP REMOTE` are reserved syntax. They are rejected at SQL -parse time with `'TO REMOTE' is not supported yet` and -`'DROP REMOTE' is not supported yet`. Automatic upload of Parquet files to -object storage is not currently supported — storage policies operate locally -only. Because these clauses cannot take effect, the `to_remote` and -`drop_remote` columns in the -[`storage_policies`](/docs/query/functions/meta/#storage_policies) view are -always blank in the current release. +Storage policies operate locally only for now. `TO REMOTE` is accepted and +stored but not yet enforced: no upload to object storage happens yet. +`DROP REMOTE` is not yet supported and is rejected at parse time with +`'DROP REMOTE' is not supported yet`. In the +[`storage_policies`](/docs/query/functions/meta/#storage_policies) view, +`drop_remote` is therefore always `0h`, and `to_remote` reads `0h` unless a +`TO REMOTE` value is set (stored, but not yet enforced). ::: @@ -89,13 +88,17 @@ Both singular and plural forms are accepted. `TO PARQUET <= DROP LOCAL`, `TO REMOTE <= DROP LOCAL`, and `DROP LOCAL <= DROP REMOTE`. `TO PARQUET` and `TO REMOTE` are independent of each other -- All TTL values must be positive — `0` is rejected +- All TTL values must be positive; `0` is rejected +- Each TTL must be an integer multiple of the partition size. For example, a + `MONTH`-partitioned table accepts only month- or year-based values, not + `HOUR`, `DAY`, or `WEEK` - Each setting can only appear once per statement - The table must have a designated timestamp and partitioning enabled -- If the table has a TTL set, clear it with `ALTER TABLE SET TTL 0` before - setting a storage policy. Any non-zero `SET TTL` value is rejected in - Enterprise with `TTL settings are deprecated, please, create a storage policy - instead` +- If the table has a TTL set, clear it with `ALTER TABLE SET TTL 0` first; + otherwise `SET STORAGE POLICY` is rejected with `Cannot set storage policy, + please, remove TTL settings`. On Enterprise tables, any non-zero `SET TTL` + value is itself rejected with `TTL is not supported on Enterprise tables; use + a storage policy instead` - `ENABLE` and `DISABLE` require a policy to exist on the table; both return an error otherwise @@ -121,7 +124,8 @@ ALTER TABLE sensor_data SET STORAGE POLICY( ); ``` -Update only the Parquet conversion threshold: +Replace the policy with a single Parquet-conversion stage (any previously set +stages are cleared): ```questdb-sql ALTER TABLE sensor_data SET STORAGE POLICY(TO PARQUET 7d); diff --git a/documentation/query/sql/alter-table-set-ttl.md b/documentation/query/sql/alter-table-set-ttl.md index 06b8d43095..c71318a913 100644 --- a/documentation/query/sql/alter-table-set-ttl.md +++ b/documentation/query/sql/alter-table-set-ttl.md @@ -10,7 +10,7 @@ Sets the time-to-live (TTL) period on a table. **QuestDB Enterprise: TTL is deprecated.** Enterprise rejects any non-zero `SET TTL` with -`TTL settings are deprecated, please, create a storage policy instead`. Use +`TTL is not supported on Enterprise tables; use a storage policy instead`. Use [`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/) instead. `SET TTL 0` is still accepted, for clearing a pre-existing TTL before attaching a storage policy. See [Storage Policy](/docs/concepts/storage-policy/) diff --git a/documentation/query/sql/create-table.md b/documentation/query/sql/create-table.md index f866f25d28..3455526f86 100644 --- a/documentation/query/sql/create-table.md +++ b/documentation/query/sql/create-table.md @@ -262,10 +262,13 @@ information on the behavior of this feature. :::note -In QuestDB Enterprise, `TTL` is deprecated — `CREATE TABLE ... TTL` is -rejected with `TTL settings are deprecated, please, create a storage policy -instead`. Use `STORAGE POLICY` instead. If a legacy table has a TTL set, clear -it with `ALTER TABLE SET TTL 0` before setting a storage policy. +In QuestDB Enterprise, use `STORAGE POLICY` instead of `TTL`. +`CREATE TABLE ... TTL` is still accepted for backward compatibility and is +translated into a `STORAGE POLICY(DROP LOCAL ...)`. On an existing table, +`ALTER TABLE SET TTL` with a non-zero value is rejected with `TTL is not +supported on Enterprise tables; use a storage policy instead`. If a legacy +table has a TTL set, clear it with `ALTER TABLE SET TTL 0` before setting a +storage policy. ::: @@ -299,9 +302,9 @@ A storage policy supports up to four settings: `TO PARQUET`, `TO REMOTE`, positive. A drop stage may not precede the write it depends on (`TO PARQUET` and `TO REMOTE` before `DROP LOCAL`; `DROP LOCAL` before `DROP REMOTE`), while `TO PARQUET` and `TO REMOTE` are independent. Converting a partition to Parquet -removes its native files and serves reads from the Parquet file. `TO REMOTE` -and `DROP REMOTE` are reserved syntax and are currently rejected at SQL parse -time with `'TO REMOTE' is not supported yet` and +removes its native files and serves reads from the Parquet file. Storage +policies currently operate locally only: `TO REMOTE` is accepted and stored but +not yet enforced, and `DROP REMOTE` is rejected at SQL parse time with `'DROP REMOTE' is not supported yet`. To modify a storage policy after table creation, see diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 241caecc55..64ade637e7 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -160,9 +160,9 @@ CREATE TABLE 'sensor_data' ( STORAGE POLICY(TO PARQUET 3 DAYS, DROP LOCAL 1 MONTH) WAL; ``` -Stages that are not configured on the policy are omitted from the clause. A -disabled policy (`ALTER TABLE ... DISABLE STORAGE POLICY`) still renders — the -disabled state is not part of the DDL. See +Stages that are not configured on the policy are omitted from the clause. Only +an active policy renders: after `ALTER TABLE ... DISABLE STORAGE POLICY`, the +policy is not shown in `SHOW CREATE TABLE`. See [ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/). #### Enterprise variant From a3872e396ccbf187bd328917a4646c917ecbdebd Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 3 Jul 2026 13:11:18 +0100 Subject: [PATCH 2/5] Refine storage policy docs details Co-Authored-By: Claude Opus 4.8 (1M context) --- documentation/concepts/deep-dive/sql-extensions.md | 5 +++-- documentation/concepts/storage-policy.md | 11 +++++------ documentation/query/functions/meta.md | 6 ++++-- .../query/sql/alter-table-set-storage-policy.md | 5 +++-- documentation/query/sql/create-table.md | 5 +++-- documentation/query/sql/show.md | 6 +++--- 6 files changed, 21 insertions(+), 17 deletions(-) diff --git a/documentation/concepts/deep-dive/sql-extensions.md b/documentation/concepts/deep-dive/sql-extensions.md index a294564eec..00ccda8a77 100644 --- a/documentation/concepts/deep-dive/sql-extensions.md +++ b/documentation/concepts/deep-dive/sql-extensions.md @@ -340,8 +340,9 @@ QuestDB exposes a few SQL-level features for operating tables at scale: - **[TTL](/docs/concepts/ttl/)**. Per-table retention that drops whole partitions older than the configured horizon. -- **[Storage policy](/docs/concepts/storage-policy/)** (Enterprise). Moves - old partitions to Parquet on object storage while keeping them queryable. +- **[Storage policy](/docs/concepts/storage-policy/)** (Enterprise). Converts + old partitions to Parquet locally, keeping them queryable, and can drop local + copies on a schedule. - **[RBAC](/docs/security/rbac/)** (Enterprise). Users, groups, and service accounts with granular permissions over tables and operations. - **[Backup / CHECKPOINT](/docs/query/sql/checkpoint/)**. Filesystem-level diff --git a/documentation/concepts/storage-policy.md b/documentation/concepts/storage-policy.md index 1ae4d774a0..bc714efe3e 100644 --- a/documentation/concepts/storage-policy.md +++ b/documentation/concepts/storage-policy.md @@ -118,8 +118,9 @@ ALTER TABLE trades SET STORAGE POLICY(DROP LOCAL 30d); :::note -If a table already has a TTL set, clear it with `ALTER TABLE SET TTL 0` before -setting a storage policy; otherwise `SET STORAGE POLICY` is rejected with +A table can carry a TTL only if it was created in QuestDB Open Source and later +upgraded to Enterprise. Clear that legacy TTL with `ALTER TABLE SET TTL 0` +before setting a storage policy; otherwise `SET STORAGE POLICY` is rejected with `Cannot set storage policy, please, remove TTL settings`. On Enterprise tables `SET TTL 0` is the only accepted `SET TTL` value; any non-zero value is rejected with `TTL is not supported on Enterprise tables; use a storage policy instead`. @@ -253,7 +254,7 @@ for details. ## Configuration Storage policy behavior can be tuned in `server.conf`. Time-based properties -accept values with unit suffixes (e.g., `15m`, `30s`, `1h`) or raw microsecond +accept values with unit suffixes (e.g., `5m`, `1h`, `100ms`) or raw microsecond values: | Property | Default | Description | @@ -267,9 +268,7 @@ values: See [Storage policy configuration](/docs/configuration/storage-policy/) for the -complete list, including additional worker-pool tuning properties -(`worker.haltOnError`, `worker.nap.threshold`, `worker.sleep.threshold`, and -`worker.yield.threshold`). +complete list, including the remaining worker-pool tuning properties. ## Permissions diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index b345a38d66..56dbf42470 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -337,8 +337,10 @@ SELECT * FROM storage_policies; The first row is a policy with two active stages (3-day Parquet conversion and 1-month local drop) and is currently enforced. The second row has only the -`TO PARQUET` stage set and has been temporarily disabled. Unset stages, and the -not-yet-enforced `to_remote` and `drop_remote` columns, all render as `0h`. +`TO PARQUET` stage set and has been temporarily disabled. Every unset stage +renders as `0h`: here neither policy sets `TO REMOTE`, so `to_remote` is `0h`, +and `drop_remote` is always `0h` because `DROP REMOTE` is rejected at parse +time. ## table_columns diff --git a/documentation/query/sql/alter-table-set-storage-policy.md b/documentation/query/sql/alter-table-set-storage-policy.md index 56f081671f..37de6da969 100644 --- a/documentation/query/sql/alter-table-set-storage-policy.md +++ b/documentation/query/sql/alter-table-set-storage-policy.md @@ -89,9 +89,10 @@ Both singular and plural forms are accepted. `DROP LOCAL <= DROP REMOTE`. `TO PARQUET` and `TO REMOTE` are independent of each other - All TTL values must be positive; `0` is rejected -- Each TTL must be an integer multiple of the partition size. For example, a +- The TTL unit cannot be finer than the table's partition size. For example, a `MONTH`-partitioned table accepts only month- or year-based values, not - `HOUR`, `DAY`, or `WEEK` + `HOUR`, `DAY`, or `WEEK`; a `DAY`-partitioned table also accepts coarser + units such as `DROP LOCAL 1 MONTH` - Each setting can only appear once per statement - The table must have a designated timestamp and partitioning enabled - If the table has a TTL set, clear it with `ALTER TABLE SET TTL 0` first; diff --git a/documentation/query/sql/create-table.md b/documentation/query/sql/create-table.md index 3455526f86..343aa87f20 100644 --- a/documentation/query/sql/create-table.md +++ b/documentation/query/sql/create-table.md @@ -266,8 +266,9 @@ In QuestDB Enterprise, use `STORAGE POLICY` instead of `TTL`. `CREATE TABLE ... TTL` is still accepted for backward compatibility and is translated into a `STORAGE POLICY(DROP LOCAL ...)`. On an existing table, `ALTER TABLE SET TTL` with a non-zero value is rejected with `TTL is not -supported on Enterprise tables; use a storage policy instead`. If a legacy -table has a TTL set, clear it with `ALTER TABLE SET TTL 0` before setting a +supported on Enterprise tables; use a storage policy instead`. A table can still +carry a TTL if it was created in QuestDB Open Source and later upgraded to +Enterprise; clear that legacy TTL with `ALTER TABLE SET TTL 0` before setting a storage policy. ::: diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 64ade637e7..0d19bae2bb 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -144,9 +144,9 @@ CREATE TABLE sensors ( #### Storage policy clause -When a [storage policy](/docs/concepts/storage-policy/) is attached to a table -(Enterprise only), the policy renders as a `STORAGE POLICY(...)` clause in the -`SHOW CREATE TABLE` output: +When an active [storage policy](/docs/concepts/storage-policy/) is attached to a +table (Enterprise only), the policy renders as a `STORAGE POLICY(...)` clause in +the `SHOW CREATE TABLE` output: ```questdb-sql SHOW CREATE TABLE sensor_data; From c4722940c42ff4e2ed379f08f4176b54b80b36ad Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 3 Jul 2026 13:35:40 +0100 Subject: [PATCH 3/5] Address review findings: cross-ref, clause order, TTL framing - meta.md: storage_policies.table_dir_name cross-reference now names the real tables() column (directoryName, not table_dir_name) - create-table.md: reorder syntax diagram so TTL/STORAGE POLICY precedes WAL, matching every example and SHOW CREATE TABLE output - alter-table-set-ttl.md: align framing with ttl.md ("superseded by Storage Policy" instead of "deprecated") - ttl.md: note materialized views still use TTL on Enterprise Co-Authored-By: Claude Opus 4.8 (1M context) --- documentation/concepts/ttl.md | 9 ++++++--- documentation/query/functions/meta.md | 2 +- documentation/query/sql/alter-table-set-ttl.md | 5 +++-- documentation/query/sql/create-table.md | 8 ++++---- 4 files changed, 14 insertions(+), 10 deletions(-) diff --git a/documentation/concepts/ttl.md b/documentation/concepts/ttl.md index f544a70266..2013061d99 100644 --- a/documentation/concepts/ttl.md +++ b/documentation/concepts/ttl.md @@ -16,9 +16,12 @@ non-zero `SET TTL` with `TTL is not supported on Enterprise tables; use a storage policy instead`. Storage policies extend TTL with graduated lifecycle management (convert to Parquet, then drop) and are the recommended retention primitive for Enterprise -users. The rest of this page describes TTL behavior on QuestDB Open Source -(and the `SET TTL 0` case on Enterprise, used to clear an older TTL before -attaching a storage policy). +users. Materialized views are the exception: they continue to use `TTL` for +retention on Enterprise (set via +[`ALTER MATERIALIZED VIEW SET TTL`](/docs/query/sql/alter-mat-view-set-ttl/)). +The rest of this page describes TTL behavior on QuestDB Open Source (and the +`SET TTL 0` case on Enterprise, used to clear an older TTL before attaching a +storage policy). ::: diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 56dbf42470..13bbfeb8a3 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -306,7 +306,7 @@ SELECT * FROM storage_policies; | Column | Type | Description | |--------|------|-------------| -| `table_dir_name` | _STRING_ | Directory name of the table the policy is attached to. Matches the `table_dir_name` column in [`tables()`](#tables). | +| `table_dir_name` | _STRING_ | Directory name of the table the policy is attached to. Matches the `directoryName` column in [`tables()`](#tables). | | `to_parquet` | _STRING_ | TTL for the `TO PARQUET` stage (e.g. `72h`, `1m`). `0h` when the stage is not configured. | | `to_remote` | _STRING_ | TTL for the `TO REMOTE` stage. Accepted and stored but not yet enforced, so setting it has no effect for now. `0h` when not configured. | | `drop_local` | _STRING_ | TTL for the `DROP LOCAL` stage. `0h` when the stage is not configured. | diff --git a/documentation/query/sql/alter-table-set-ttl.md b/documentation/query/sql/alter-table-set-ttl.md index c71318a913..e5ece7effb 100644 --- a/documentation/query/sql/alter-table-set-ttl.md +++ b/documentation/query/sql/alter-table-set-ttl.md @@ -8,8 +8,9 @@ Sets the time-to-live (TTL) period on a table. :::caution -**QuestDB Enterprise: TTL is deprecated.** Enterprise rejects any non-zero -`SET TTL` with +**QuestDB Enterprise: TTL is superseded by +[Storage Policy](/docs/concepts/storage-policy/).** Enterprise rejects any +non-zero `SET TTL` with `TTL is not supported on Enterprise tables; use a storage policy instead`. Use [`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/) instead. `SET TTL 0` is still accepted, for clearing a pre-existing TTL before diff --git a/documentation/query/sql/create-table.md b/documentation/query/sql/create-table.md index 343aa87f20..23b84c6628 100644 --- a/documentation/query/sql/create-table.md +++ b/documentation/query/sql/create-table.md @@ -37,9 +37,9 @@ TABLE [IF NOT EXISTS] tableName [, INDEX (columnRef [CAPACITY n | TYPE POSTING [DELTA | EF]]) ...] -- see Column indexes [TIMESTAMP (columnName) [PARTITION BY { NONE | YEAR | MONTH | DAY | HOUR } - [BYPASS WAL | WAL] [ TTL n { HOUR[S] | DAY[S] | WEEK[S] | MONTH[S] | YEAR[S] } - | STORAGE POLICY ( policyStage [, policyStage ...] ) ]]] + | STORAGE POLICY ( policyStage [, policyStage ...] ) ] + [BYPASS WAL | WAL]]] [DEDUP UPSERT KEYS (columnName [, columnName ...])] [WITH tableParameter] [IN VOLUME 'alias'] @@ -58,9 +58,9 @@ TABLE [IF NOT EXISTS] tableName [, INDEX (columnRef [CAPACITY n | TYPE POSTING [DELTA | EF]]) ...] -- see Column indexes [TIMESTAMP (columnName) [PARTITION BY { NONE | YEAR | MONTH | DAY | HOUR } - [BYPASS WAL | WAL] [ TTL n { HOUR[S] | DAY[S] | WEEK[S] | MONTH[S] | YEAR[S] } - | STORAGE POLICY ( policyStage [, policyStage ...] ) ]]] + | STORAGE POLICY ( policyStage [, policyStage ...] ) ] + [BYPASS WAL | WAL]]] [DEDUP UPSERT KEYS (columnName [, columnName ...])] [WITH tableParameter] [IN VOLUME 'alias'] From 11bfa1bccbe8a13dc66d6c4619a8546548150cfd Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 3 Jul 2026 14:23:38 +0100 Subject: [PATCH 4/5] Fix leftover object-storage overclaims in storage policy docs Address review findings on PR #475: three wording corrections consistent with the local-only behavior documented elsewhere in the PR. - configuration/storage-policy.md: intro no longer claims present-tense "cold storage offloading"; describes local Parquet conversion and drop. - concepts/storage-policy.md: the storage_policies view lists all policies and their status, not just active ones (it also returns disabled rows). - concepts/ttl.md: storage-policy cross-reference drops "offload to object storage" from the current lifecycle description. Co-Authored-By: Claude Opus 4.8 (1M context) --- documentation/concepts/storage-policy.md | 2 +- documentation/concepts/ttl.md | 4 ++-- documentation/configuration/storage-policy.md | 7 ++++--- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/documentation/concepts/storage-policy.md b/documentation/concepts/storage-policy.md index bc714efe3e..7b1b46558b 100644 --- a/documentation/concepts/storage-policy.md +++ b/documentation/concepts/storage-policy.md @@ -212,7 +212,7 @@ ALTER TABLE trades DROP STORAGE POLICY; ## Checking storage policies -Query the `storage_policies` system view to see all active policies: +Query the `storage_policies` system view to see all policies and their status: ```questdb-sql SELECT * FROM storage_policies; diff --git a/documentation/concepts/ttl.md b/documentation/concepts/ttl.md index 2013061d99..8620df2387 100644 --- a/documentation/concepts/ttl.md +++ b/documentation/concepts/ttl.md @@ -169,6 +169,6 @@ ALTER TABLE trades SET TTL 0h; - TTL should be significantly larger than your partition interval - For manual control instead of automatic TTL, see [Data Retention](/docs/operations/data-retention/) -- For graduated lifecycle management (convert to Parquet, offload to object - storage, then drop), see [Storage Policy](/docs/concepts/storage-policy/) +- For graduated lifecycle management (convert to Parquet locally, then drop on + a schedule), see [Storage Policy](/docs/concepts/storage-policy/) (Enterprise) diff --git a/documentation/configuration/storage-policy.md b/documentation/configuration/storage-policy.md index 8b0db13201..69fa445e03 100644 --- a/documentation/configuration/storage-policy.md +++ b/documentation/configuration/storage-policy.md @@ -9,9 +9,10 @@ Storage policy is [Enterprise](/enterprise/) only. ::: -Storage policies automate partition lifecycle management, including local -deletion and cold storage offloading. These settings control the scan interval, -retry behavior, and worker threads for the storage policy engine. +Storage policies automate partition lifecycle management: they convert older +partitions to Parquet locally and drop local copies on a schedule. These +settings control the scan interval, retry behavior, and worker threads for the +storage policy engine. For details, see the [storage policy concept](/docs/concepts/storage-policy/) page. From 46852e8f1649a157a476770be84677ab950731cf Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 3 Jul 2026 15:01:35 +0100 Subject: [PATCH 5/5] Fix table_partitions Parquet flag docs, TO REMOTE wording, WEEK unit - meta.md: drop the non-existent DROP NATIVE stage and the old two-step "served from native until switched" framing; TO PARQUET removes native files and serves reads from the Parquet file - storage-policy.md: stop calling the accepted-and-stored TO REMOTE stage "reserved" - create-table.md: add the missing WEEK partition unit to both syntax blocks Co-Authored-By: Claude Opus 4.8 (1M context) --- documentation/concepts/storage-policy.md | 6 +++--- documentation/query/functions/meta.md | 13 ++++++------- documentation/query/sql/create-table.md | 4 ++-- 3 files changed, 11 insertions(+), 12 deletions(-) diff --git a/documentation/concepts/storage-policy.md b/documentation/concepts/storage-policy.md index 7b1b46558b..df9665c599 100644 --- a/documentation/concepts/storage-policy.md +++ b/documentation/concepts/storage-policy.md @@ -76,9 +76,9 @@ eligible when: partition_end_time < reference_time - TTL **This rule is applied independently for each stage's TTL.** A partition can be eligible for `TO PARQUET` long before it is eligible for `DROP LOCAL` (or, -one day, the reserved `TO REMOTE` and `DROP REMOTE` stages). Each stage uses -its own `TTL` in the formula above; the stages share only the reference time -and the [ordering constraints](#ordering-constraint). +once remote upload is enforced, the `TO REMOTE` and `DROP REMOTE` stages). +Each stage uses its own `TTL` in the formula above; the stages share only the +reference time and the [ordering constraints](#ordering-constraint). The reference time is `min(wall_clock_time, latest_timestamp)` by default — the same formula used by TTL. The diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 13bbfeb8a3..ada4004dd0 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -449,16 +449,15 @@ Returns a table with the following columns: - `attachable` - _BOOLEAN_, true if the partition is detached and can be attached (`name` of the partition will contain the `.attachable` extension) - `hasParquetGenerated` - _BOOLEAN_, true if a Parquet copy of the partition - has been produced alongside the native files. Set by either + has been generated. Set by either [manual Parquet conversion](/docs/query/export-parquet/#in-place-conversion) (`ALTER TABLE ... CONVERT PARTITION TO PARQUET`) or by a [storage policy](/docs/concepts/storage-policy/)'s `TO PARQUET` stage - (Enterprise). The partition is still served from native storage until it is - switched to Parquet-only format -- `isParquet` - _BOOLEAN_, true if the partition is stored in Parquet format - (native files have been replaced). Set the same way as - `hasParquetGenerated` — either manually or by a storage policy's `DROP - NATIVE` stage + (Enterprise) +- `isParquet` - _BOOLEAN_, true if the partition is stored in Parquet format: + the native files have been removed and reads are served from the Parquet + file. Set the same way as `hasParquetGenerated` — either manually or by a + storage policy's `TO PARQUET` stage - `parquetFileSize` - _LONG_, size in bytes of the partition's `data.parquet` file when `hasParquetGenerated` or `isParquet` is true; `-1` otherwise diff --git a/documentation/query/sql/create-table.md b/documentation/query/sql/create-table.md index 23b84c6628..a1f6fdd95d 100644 --- a/documentation/query/sql/create-table.md +++ b/documentation/query/sql/create-table.md @@ -36,7 +36,7 @@ TABLE [IF NOT EXISTS] tableName (columnName columnTypeDef [, columnName columnTypeDef ...]) -- see Type definition [, INDEX (columnRef [CAPACITY n | TYPE POSTING [DELTA | EF]]) ...] -- see Column indexes [TIMESTAMP (columnName) - [PARTITION BY { NONE | YEAR | MONTH | DAY | HOUR } + [PARTITION BY { NONE | YEAR | MONTH | WEEK | DAY | HOUR } [ TTL n { HOUR[S] | DAY[S] | WEEK[S] | MONTH[S] | YEAR[S] } | STORAGE POLICY ( policyStage [, policyStage ...] ) ] [BYPASS WAL | WAL]]] @@ -57,7 +57,7 @@ TABLE [IF NOT EXISTS] tableName [, cast(columnRef AS columnTypeDef) ...] -- see Type definition [, INDEX (columnRef [CAPACITY n | TYPE POSTING [DELTA | EF]]) ...] -- see Column indexes [TIMESTAMP (columnName) - [PARTITION BY { NONE | YEAR | MONTH | DAY | HOUR } + [PARTITION BY { NONE | YEAR | MONTH | WEEK | DAY | HOUR } [ TTL n { HOUR[S] | DAY[S] | WEEK[S] | MONTH[S] | YEAR[S] } | STORAGE POLICY ( policyStage [, policyStage ...] ) ] [BYPASS WAL | WAL]]]