diff --git a/documentation/concepts/deep-dive/sql-extensions.md b/documentation/concepts/deep-dive/sql-extensions.md index a294564ee..00ccda8a7 100644 --- a/documentation/concepts/deep-dive/sql-extensions.md +++ b/documentation/concepts/deep-dive/sql-extensions.md @@ -340,8 +340,9 @@ QuestDB exposes a few SQL-level features for operating tables at scale: - **[TTL](/docs/concepts/ttl/)**. Per-table retention that drops whole partitions older than the configured horizon. -- **[Storage policy](/docs/concepts/storage-policy/)** (Enterprise). Moves - old partitions to Parquet on object storage while keeping them queryable. +- **[Storage policy](/docs/concepts/storage-policy/)** (Enterprise). Converts + old partitions to Parquet locally, keeping them queryable, and can drop local + copies on a schedule. - **[RBAC](/docs/security/rbac/)** (Enterprise). Users, groups, and service accounts with granular permissions over tables and operations. - **[Backup / CHECKPOINT](/docs/query/sql/checkpoint/)**. Filesystem-level diff --git a/documentation/concepts/storage-policy.md b/documentation/concepts/storage-policy.md index a0ecc70cb..df9665c59 100644 --- a/documentation/concepts/storage-policy.md +++ b/documentation/concepts/storage-policy.md @@ -16,13 +16,14 @@ Converting a partition to Parquet removes its native files and serves reads dire :::info -Storage policies currently operate **locally only**. Parquet files are not -automatically uploaded to object storage, so the `TO REMOTE` and `DROP REMOTE` -clauses are reserved syntax — they are rejected at SQL parse time with -`'TO REMOTE' is not supported yet` and `'DROP REMOTE' is not supported yet`. -Accordingly, the `to_remote` and `drop_remote` columns in the -[`storage_policies`](/docs/query/functions/meta/#storage_policies) view are -always blank in the current release; they are kept for forward compatibility. +Storage policies currently operate **locally only**. `TO PARQUET` and +`DROP LOCAL` are the enforced stages. `TO REMOTE` is accepted and stored but +not yet enforced, so setting it has no effect for now: no upload to object +storage happens yet. `DROP REMOTE` is not yet supported and is rejected at SQL +parse time with `'DROP REMOTE' is not supported yet`. In the +[`storage_policies`](/docs/query/functions/meta/#storage_policies) view the +`drop_remote` column is therefore always `0h`, and `to_remote` reads `0h` +unless you set a `TO REMOTE` value (which is stored but has no effect yet). Object storage integration will be added in a future release. ::: @@ -43,9 +44,9 @@ stage in the partition lifecycle: | Setting | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `TO PARQUET` | Convert the partition from native binary format to Parquet. The native files are removed and reads are served from the Parquet file | -| `TO REMOTE` | _Reserved._ Will upload the Parquet file to object storage when remote upload is supported | +| `TO REMOTE` | Accepted and stored but **not yet enforced**; no upload happens yet. Reserved for future object storage upload | | `DROP LOCAL` | Remove all local data (native or Parquet) | -| `DROP REMOTE` | _Reserved._ Will remove the Parquet file from object storage when remote upload is supported | +| `DROP REMOTE` | _Not yet supported._ Rejected at parse time with `'DROP REMOTE' is not supported yet`. Reserved for future object storage removal | All settings are optional. Use only the ones relevant to your use case. All TTL values must be **positive**; `0` is rejected. @@ -75,9 +76,9 @@ eligible when: partition_end_time < reference_time - TTL **This rule is applied independently for each stage's TTL.** A partition can be eligible for `TO PARQUET` long before it is eligible for `DROP LOCAL` (or, -one day, the reserved `TO REMOTE` and `DROP REMOTE` stages). Each stage uses -its own `TTL` in the formula above; the stages share only the reference time -and the [ordering constraints](#ordering-constraint). +once remote upload is enforced, the `TO REMOTE` and `DROP REMOTE` stages). +Each stage uses its own `TTL` in the formula above; the stages share only the +reference time and the [ordering constraints](#ordering-constraint). The reference time is `min(wall_clock_time, latest_timestamp)` by default — the same formula used by TTL. The @@ -87,7 +88,7 @@ the wall-clock cap for both TTL and storage policy evaluation. See [TTL § Reference time](/docs/concepts/ttl/#reference-time) for the rationale and the data-loss hazard of disabling the cap. -QuestDB checks storage policies periodically (every 15 minutes by default) and +QuestDB checks storage policies periodically (every 5 minutes by default) and processes eligible partitions automatically. ## Storage policy vs TTL @@ -102,8 +103,10 @@ you are already familiar with TTL, this comparison is the fastest way in: | **Parquet conversion** | No | Yes (automatic local conversion) | | **Granularity** | Single retention window | Up to four independent TTL stages | -In QuestDB Enterprise, `CREATE TABLE ... TTL` and `ALTER TABLE SET TTL` are -deprecated. Use storage policies instead: +In QuestDB Enterprise, use storage policies instead of TTL. On a regular table, +`ALTER TABLE SET TTL` with a non-zero value is rejected, while +`CREATE TABLE ... TTL` is accepted only for backward compatibility and is +translated into a `STORAGE POLICY(DROP LOCAL ...)`: ```questdb-sql -- Instead of: @@ -115,10 +118,12 @@ ALTER TABLE trades SET STORAGE POLICY(DROP LOCAL 30d); :::note -If a table already has a TTL set, you must clear it with -`ALTER TABLE SET TTL 0` before setting a storage policy. `SET TTL 0` is the -only `SET TTL` value Enterprise accepts; any non-zero value is rejected with -`TTL settings are deprecated, please, create a storage policy instead`. +A table can carry a TTL only if it was created in QuestDB Open Source and later +upgraded to Enterprise. Clear that legacy TTL with `ALTER TABLE SET TTL 0` +before setting a storage policy; otherwise `SET STORAGE POLICY` is rejected with +`Cannot set storage policy, please, remove TTL settings`. On Enterprise tables +`SET TTL 0` is the only accepted `SET TTL` value; any non-zero value is rejected +with `TTL is not supported on Enterprise tables; use a storage policy instead`. ::: @@ -145,7 +150,8 @@ ALTER TABLE trades SET STORAGE POLICY( ); ``` -Only the specified settings are changed. Omitted settings remain unchanged. +`SET STORAGE POLICY` replaces the policy in full: every stage you omit is +cleared, not preserved. To keep a stage, restate it in the same statement. For full syntax details, see [ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/). @@ -173,11 +179,11 @@ TO PARQUET <= DROP LOCAL TO REMOTE <= DROP LOCAL <= DROP REMOTE ``` -`TO PARQUET` and `TO REMOTE` are **independent** — neither has to precede the -other. If `TO REMOTE` fires before `TO PARQUET`, both the native and Parquet -copies are written locally and reads continue to be served from the native -format until `TO PARQUET` removes the native files. All TTL values must be -positive — `0` is rejected. +`TO PARQUET` and `TO REMOTE` are **independent**: neither has to precede the +other. Once remote upload is enforced, a `TO REMOTE` that runs before +`TO PARQUET` would keep both the native and Parquet copies on local disk, with +reads served from the native format until `TO PARQUET` removes the native +files. All TTL values must be positive; `0` is rejected. ## Disabling and enabling @@ -206,7 +212,7 @@ ALTER TABLE trades DROP STORAGE POLICY; ## Checking storage policies -Query the `storage_policies` system view to see all active policies: +Query the `storage_policies` system view to see all policies and their status: ```questdb-sql SELECT * FROM storage_policies; @@ -214,21 +220,19 @@ SELECT * FROM storage_policies; | table_dir_name | to_parquet | to_remote | drop_local | drop_remote | status | last_updated | | -------------- | ---------- | --------- | ---------- | ----------- | ------ | --------------------------- | -| trades~12 | 72h | | 1m | | A | 2025-01-15T10:30:00.000000Z | - -- TTL values are rendered in just two units: `h` for hours and `m` for - **months**. Hour-, day-, and week-based durations are normalized to hours - when stored, so a `3 DAYS` TTL appears as `72h` and `1 WEEK` appears as - `168h`. Month-based durations keep the lowercase `m` suffix — **`1m` in - this view means one month, not one minute**; QuestDB's duration shorthand - has no unit for minutes +| trades~12 | 72h | 0h | 1m | 0h | A | 2025-01-15T10:30:00.000000Z | + +- TTL values are rendered in two units: `h` for hours and `m` for **months**. + Hour-, day-, and week-based durations are stored as hours, so a `3 DAYS` TTL + appears as `72h` and `1 WEEK` as `168h`. Month- and year-based durations are + stored as months, so `1 MONTH` appears as `1m` and `1 YEAR` as `12m`. In this + view **`m` means months, not minutes**; QuestDB's duration shorthand has no + unit for minutes - Status `A` means active; `D` means disabled (see [Disabling and enabling](#disabling-and-enabling)) -- Unset stages appear blank. `to_remote` and `drop_remote` are **always blank - in the current release** because `TO REMOTE` and `DROP REMOTE` are rejected - at SQL parse time with `'TO REMOTE' is not supported yet` and - `'DROP REMOTE' is not supported yet`; the columns are kept for forward - compatibility +- An unset stage renders as `0h`, not blank. Because `DROP REMOTE` is rejected + at parse time, `drop_remote` is always `0h`; `to_remote` reads `0h` unless a + `TO REMOTE` value is set (stored, but not yet enforced) For the full column reference and types, see [`storage_policies`](/docs/query/functions/meta/#storage_policies). @@ -250,19 +254,22 @@ for details. ## Configuration Storage policy behavior can be tuned in `server.conf`. Time-based properties -accept values with unit suffixes (e.g., `15m`, `30s`, `1h`) or raw microsecond +accept values with unit suffixes (e.g., `5m`, `1h`, `100ms`) or raw microsecond values: | Property | Default | Description | | ------------------------------------- | ------------------ | ---------------------------------------------------------------- | -| `storage.policy.check.interval` | `15m` (15 min) | How often QuestDB scans for partitions to process | +| `storage.policy.check.interval` | `5m` (5 min) | How often QuestDB scans for partitions to process | | `storage.policy.retry.interval` | `1m` (1 min) | Retry interval for failed tasks | | `storage.policy.max.reschedule.count` | `20` | Maximum retries before abandoning a task | -| `storage.policy.writer.wait.timeout` | `30s` (30 sec) | Timeout for acquiring the table writer | -| `storage.policy.worker.count` | `2` | Number of storage policy worker threads (0 disables the feature) | +| `storage.policy.worker.count` | `4` | Number of storage policy worker threads (0 disables the feature) | | `storage.policy.worker.affinity` | `-1` (no affinity) | CPU affinity for each worker thread (comma-separated list) | | `storage.policy.worker.sleep.timeout` | `100ms` | Sleep duration when worker has no tasks | +See +[Storage policy configuration](/docs/configuration/storage-policy/) for the +complete list, including the remaining worker-pool tuning properties. + ## Permissions Storage policy operations require specific permissions in QuestDB Enterprise: @@ -307,7 +314,7 @@ WHERE table_dir_name LIKE 'trades%'; | -------------- | ---------- | ---------- | ------ | | trades~12 | 72h | 1m | A | -```questdb-sql title="3. Modify one stage (others remain unchanged)" +```questdb-sql title="3. Replace the policy (omitted stages are cleared)" ALTER TABLE trades SET STORAGE POLICY(TO PARQUET 1d); ``` @@ -324,6 +331,10 @@ CREATE TABLE 'trades' ( STORAGE POLICY(TO PARQUET 1 DAY) WAL; ``` +The `DROP LOCAL 1 MONTH` stage from step 1 is gone: step 3 restated only +`TO PARQUET`, and `SET STORAGE POLICY` replaces the whole policy rather than +merging into it. + ```questdb-sql title="5. Temporarily suspend the policy (e.g. during a backfill)" ALTER TABLE trades DISABLE STORAGE POLICY; -- status in storage_policies changes to 'D' diff --git a/documentation/concepts/ttl.md b/documentation/concepts/ttl.md index d7b9e34ab..8620df238 100644 --- a/documentation/concepts/ttl.md +++ b/documentation/concepts/ttl.md @@ -13,12 +13,15 @@ window - no cron jobs or manual cleanup required. **QuestDB Enterprise: TTL is superseded by [Storage Policy](/docs/concepts/storage-policy/).** Enterprise rejects any non-zero `SET TTL` with -`TTL settings are deprecated, please, create a storage policy instead`. +`TTL is not supported on Enterprise tables; use a storage policy instead`. Storage policies extend TTL with graduated lifecycle management (convert to Parquet, then drop) and are the recommended retention primitive for Enterprise -users. The rest of this page describes TTL behavior on QuestDB Open Source -(and the `SET TTL 0` case on Enterprise, used to clear an older TTL before -attaching a storage policy). +users. Materialized views are the exception: they continue to use `TTL` for +retention on Enterprise (set via +[`ALTER MATERIALIZED VIEW SET TTL`](/docs/query/sql/alter-mat-view-set-ttl/)). +The rest of this page describes TTL behavior on QuestDB Open Source (and the +`SET TTL 0` case on Enterprise, used to clear an older TTL before attaching a +storage policy). ::: @@ -166,6 +169,6 @@ ALTER TABLE trades SET TTL 0h; - TTL should be significantly larger than your partition interval - For manual control instead of automatic TTL, see [Data Retention](/docs/operations/data-retention/) -- For graduated lifecycle management (convert to Parquet, offload to object - storage, then drop), see [Storage Policy](/docs/concepts/storage-policy/) +- For graduated lifecycle management (convert to Parquet locally, then drop on + a schedule), see [Storage Policy](/docs/concepts/storage-policy/) (Enterprise) diff --git a/documentation/configuration/storage-policy.md b/documentation/configuration/storage-policy.md index e97d19bd6..69fa445e0 100644 --- a/documentation/configuration/storage-policy.md +++ b/documentation/configuration/storage-policy.md @@ -9,16 +9,17 @@ Storage policy is [Enterprise](/enterprise/) only. ::: -Storage policies automate partition lifecycle management, including local -deletion and cold storage offloading. These settings control the scan interval, -retry behavior, and worker threads for the storage policy engine. +Storage policies automate partition lifecycle management: they convert older +partitions to Parquet locally and drop local copies on a schedule. These +settings control the scan interval, retry behavior, and worker threads for the +storage policy engine. For details, see the [storage policy concept](/docs/concepts/storage-policy/) page. ## storage.policy.check.interval -- **Default**: `15m` +- **Default**: `5m` - **Reloadable**: no How often QuestDB scans for partitions to process. @@ -47,11 +48,32 @@ CPU affinity for each storage policy worker thread (comma-separated list). ## storage.policy.worker.count -- **Default**: `2` +- **Default**: `4` - **Reloadable**: no Number of storage policy worker threads. Setting to `0` disables the feature. +## storage.policy.worker.haltOnError + +- **Default**: `false` +- **Reloadable**: no + +Whether a storage policy worker thread halts when it hits an unhandled error. + +## storage.policy.worker.nap.threshold + +- **Default**: `100` +- **Reloadable**: no + +Number of idle worker-loop iterations before a storage policy worker naps. + +## storage.policy.worker.sleep.threshold + +- **Default**: `500` +- **Reloadable**: no + +Number of idle worker-loop iterations before a storage policy worker sleeps. + ## storage.policy.worker.sleep.timeout - **Default**: `100ms` @@ -59,9 +81,9 @@ Number of storage policy worker threads. Setting to `0` disables the feature. Sleep duration when a storage policy worker has no tasks to process. -## storage.policy.writer.wait.timeout +## storage.policy.worker.yield.threshold -- **Default**: `30s` +- **Default**: `10` - **Reloadable**: no -Timeout for acquiring the table writer during storage policy operations. +Number of idle worker-loop iterations before a storage policy worker yields. diff --git a/documentation/getting-started/enterprise-quick-start.md b/documentation/getting-started/enterprise-quick-start.md index 59c3a5b4f..85a3b3e98 100644 --- a/documentation/getting-started/enterprise-quick-start.md +++ b/documentation/getting-started/enterprise-quick-start.md @@ -479,16 +479,16 @@ too — on a schedule you define. This supersedes plain TTL in Enterprise, where ### Migrating from TTL when upgrading from OSS Tables that were created in OSS keep their existing `TTL` setting after you -upgrade to Enterprise — no data is lost at upgrade time. However, Enterprise -rejects any **new** `TTL` changes on tables (both `CREATE TABLE ... TTL` and -`ALTER TABLE SET TTL `) with: +upgrade to Enterprise — no data is lost at upgrade time. On an existing table, +`ALTER TABLE SET TTL` with a non-zero value is then rejected with: ``` -TTL settings are deprecated, please, create a storage policy instead +TTL is not supported on Enterprise tables; use a storage policy instead ``` -Materialized views are not affected: they continue to use `TTL` for retention -in Enterprise. +(`CREATE TABLE ... TTL` is still accepted for backward compatibility: Enterprise +translates it into a `STORAGE POLICY(DROP LOCAL ...)`.) Materialized views are +not affected: they continue to use `TTL` for retention in Enterprise. To move a legacy table from `TTL` to a storage policy: diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 77ebe6ac8..ada4004dd 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -306,22 +306,23 @@ SELECT * FROM storage_policies; | Column | Type | Description | |--------|------|-------------| -| `table_dir_name` | _STRING_ | Directory name of the table the policy is attached to. Matches the `table_dir_name` column in [`tables()`](#tables). | -| `to_parquet` | _STRING_ | TTL for the `TO PARQUET` stage (e.g. `72h`, `1m`). Blank when the stage is not configured. | -| `to_remote` | _STRING_ | Reserved — always blank in the current release. The `TO REMOTE` clause is rejected at SQL parse time with `'TO REMOTE' is not supported yet`. The column is kept for forward compatibility. | -| `drop_local` | _STRING_ | TTL for the `DROP LOCAL` stage. Blank when the stage is not configured. | -| `drop_remote` | _STRING_ | Reserved — always blank in the current release. The `DROP REMOTE` clause is rejected at SQL parse time with `'DROP REMOTE' is not supported yet`. The column is kept for forward compatibility. | +| `table_dir_name` | _STRING_ | Directory name of the table the policy is attached to. Matches the `directoryName` column in [`tables()`](#tables). | +| `to_parquet` | _STRING_ | TTL for the `TO PARQUET` stage (e.g. `72h`, `1m`). `0h` when the stage is not configured. | +| `to_remote` | _STRING_ | TTL for the `TO REMOTE` stage. Accepted and stored but not yet enforced, so setting it has no effect for now. `0h` when not configured. | +| `drop_local` | _STRING_ | TTL for the `DROP LOCAL` stage. `0h` when the stage is not configured. | +| `drop_remote` | _STRING_ | Reserved for future object storage removal. `DROP REMOTE` is rejected at parse time with `'DROP REMOTE' is not supported yet`, so this column is always `0h`. | | `status` | _CHAR_ | Policy status. `A` = active (the policy is being enforced), `D` = disabled (via [`ALTER TABLE DISABLE STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/)). | | `last_updated` | _TIMESTAMP_ | Timestamp of the most recent change to the policy definition (not the last time partitions were processed). | **Notes on TTL formatting:** -- TTL values are rendered in just two units: `h` for hours and `m` for - **months**. Durations written in the DDL as days, weeks, or years are - normalized to hours when stored (e.g., `3 DAYS` → `72h`, `1 WEEK` → - `168h`). Month-based durations are stored and rendered with the lowercase - `m` suffix — despite the visual collision with "minute", `m` in this view - is **months**, and QuestDB's duration shorthand has no unit for minutes. +- TTL values are rendered in two units: `h` for hours and `m` for **months**. + Hour-, day-, and week-based durations are stored as hours (e.g. `3 DAYS` → + `72h`, `1 WEEK` → `168h`). Month- and year-based durations are stored as + months (e.g. `1 MONTH` → `1m`, `1 YEAR` → `12m`). Despite the visual + collision with "minute", `m` in this view is **months**; QuestDB's duration + shorthand has no unit for minutes. +- An unset stage renders as `0h`, not blank. **Example:** @@ -331,13 +332,15 @@ SELECT * FROM storage_policies; | table_dir_name | to_parquet | to_remote | drop_local | drop_remote | status | last_updated | |----------------|------------|-----------|------------|-------------|--------|--------------| -| trades~12 | 72h | | 1m | | A | 2025-01-15T10:30:00.000000Z | -| metrics~18 | 168h | | | | D | 2025-01-14T09:15:42.000000Z | +| trades~12 | 72h | 0h | 1m | 0h | A | 2025-01-15T10:30:00.000000Z | +| metrics~18 | 168h | 0h | 0h | 0h | D | 2025-01-14T09:15:42.000000Z | The first row is a policy with two active stages (3-day Parquet conversion and 1-month local drop) and is currently enforced. The second row has only the -`TO PARQUET` stage set and has been temporarily disabled. The `to_remote` and -`drop_remote` columns are reserved and always blank in the current release. +`TO PARQUET` stage set and has been temporarily disabled. Every unset stage +renders as `0h`: here neither policy sets `TO REMOTE`, so `to_remote` is `0h`, +and `drop_remote` is always `0h` because `DROP REMOTE` is rejected at parse +time. ## table_columns @@ -446,16 +449,15 @@ Returns a table with the following columns: - `attachable` - _BOOLEAN_, true if the partition is detached and can be attached (`name` of the partition will contain the `.attachable` extension) - `hasParquetGenerated` - _BOOLEAN_, true if a Parquet copy of the partition - has been produced alongside the native files. Set by either + has been generated. Set by either [manual Parquet conversion](/docs/query/export-parquet/#in-place-conversion) (`ALTER TABLE ... CONVERT PARTITION TO PARQUET`) or by a [storage policy](/docs/concepts/storage-policy/)'s `TO PARQUET` stage - (Enterprise). The partition is still served from native storage until it is - switched to Parquet-only format -- `isParquet` - _BOOLEAN_, true if the partition is stored in Parquet format - (native files have been replaced). Set the same way as - `hasParquetGenerated` — either manually or by a storage policy's `DROP - NATIVE` stage + (Enterprise) +- `isParquet` - _BOOLEAN_, true if the partition is stored in Parquet format: + the native files have been removed and reads are served from the Parquet + file. Set the same way as `hasParquetGenerated` — either manually or by a + storage policy's `TO PARQUET` stage - `parquetFileSize` - _LONG_, size in bytes of the partition's `data.parquet` file when `hasParquetGenerated` or `isParquet` is true; `-1` otherwise diff --git a/documentation/query/sql/alter-table-set-storage-policy.md b/documentation/query/sql/alter-table-set-storage-policy.md index 5346af177..37de6da96 100644 --- a/documentation/query/sql/alter-table-set-storage-policy.md +++ b/documentation/query/sql/alter-table-set-storage-policy.md @@ -28,8 +28,8 @@ ALTER TABLE table_name SET STORAGE POLICY( ); ``` -Only the specified settings are changed. Omitted settings retain their current -values. +`SET STORAGE POLICY` replaces the policy as a whole. Any stage you do not list +is cleared, not preserved, so restate every stage you want to keep. ### Enable or disable a storage policy @@ -56,20 +56,19 @@ transition from native format to Parquet and eventually get removed: | Setting | Effect | |---------|--------| | `TO PARQUET ` | Convert partition from native format to Parquet locally. The native files are removed and reads are served from the Parquet file | -| `TO REMOTE ` | _Reserved._ Will upload the partition to object storage when remote upload is supported | +| `TO REMOTE ` | Accepted and stored but not yet enforced; no upload happens yet. Reserved for future object storage upload | | `DROP LOCAL ` | Remove all local copies of the partition | -| `DROP REMOTE ` | _Reserved._ Will remove the partition from object storage when remote upload is supported | +| `DROP REMOTE ` | _Not yet supported._ Rejected at parse time with `'DROP REMOTE' is not supported yet`. Reserved for future object storage removal | :::info -`TO REMOTE` and `DROP REMOTE` are reserved syntax. They are rejected at SQL -parse time with `'TO REMOTE' is not supported yet` and -`'DROP REMOTE' is not supported yet`. Automatic upload of Parquet files to -object storage is not currently supported — storage policies operate locally -only. Because these clauses cannot take effect, the `to_remote` and -`drop_remote` columns in the -[`storage_policies`](/docs/query/functions/meta/#storage_policies) view are -always blank in the current release. +Storage policies operate locally only for now. `TO REMOTE` is accepted and +stored but not yet enforced: no upload to object storage happens yet. +`DROP REMOTE` is not yet supported and is rejected at parse time with +`'DROP REMOTE' is not supported yet`. In the +[`storage_policies`](/docs/query/functions/meta/#storage_policies) view, +`drop_remote` is therefore always `0h`, and `to_remote` reads `0h` unless a +`TO REMOTE` value is set (stored, but not yet enforced). ::: @@ -89,13 +88,18 @@ Both singular and plural forms are accepted. `TO PARQUET <= DROP LOCAL`, `TO REMOTE <= DROP LOCAL`, and `DROP LOCAL <= DROP REMOTE`. `TO PARQUET` and `TO REMOTE` are independent of each other -- All TTL values must be positive — `0` is rejected +- All TTL values must be positive; `0` is rejected +- The TTL unit cannot be finer than the table's partition size. For example, a + `MONTH`-partitioned table accepts only month- or year-based values, not + `HOUR`, `DAY`, or `WEEK`; a `DAY`-partitioned table also accepts coarser + units such as `DROP LOCAL 1 MONTH` - Each setting can only appear once per statement - The table must have a designated timestamp and partitioning enabled -- If the table has a TTL set, clear it with `ALTER TABLE SET TTL 0` before - setting a storage policy. Any non-zero `SET TTL` value is rejected in - Enterprise with `TTL settings are deprecated, please, create a storage policy - instead` +- If the table has a TTL set, clear it with `ALTER TABLE SET TTL 0` first; + otherwise `SET STORAGE POLICY` is rejected with `Cannot set storage policy, + please, remove TTL settings`. On Enterprise tables, any non-zero `SET TTL` + value is itself rejected with `TTL is not supported on Enterprise tables; use + a storage policy instead` - `ENABLE` and `DISABLE` require a policy to exist on the table; both return an error otherwise @@ -121,7 +125,8 @@ ALTER TABLE sensor_data SET STORAGE POLICY( ); ``` -Update only the Parquet conversion threshold: +Replace the policy with a single Parquet-conversion stage (any previously set +stages are cleared): ```questdb-sql ALTER TABLE sensor_data SET STORAGE POLICY(TO PARQUET 7d); diff --git a/documentation/query/sql/alter-table-set-ttl.md b/documentation/query/sql/alter-table-set-ttl.md index 06b8d4309..e5ece7eff 100644 --- a/documentation/query/sql/alter-table-set-ttl.md +++ b/documentation/query/sql/alter-table-set-ttl.md @@ -8,9 +8,10 @@ Sets the time-to-live (TTL) period on a table. :::caution -**QuestDB Enterprise: TTL is deprecated.** Enterprise rejects any non-zero -`SET TTL` with -`TTL settings are deprecated, please, create a storage policy instead`. Use +**QuestDB Enterprise: TTL is superseded by +[Storage Policy](/docs/concepts/storage-policy/).** Enterprise rejects any +non-zero `SET TTL` with +`TTL is not supported on Enterprise tables; use a storage policy instead`. Use [`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/) instead. `SET TTL 0` is still accepted, for clearing a pre-existing TTL before attaching a storage policy. See [Storage Policy](/docs/concepts/storage-policy/) diff --git a/documentation/query/sql/create-table.md b/documentation/query/sql/create-table.md index f866f25d2..a1f6fdd95 100644 --- a/documentation/query/sql/create-table.md +++ b/documentation/query/sql/create-table.md @@ -36,10 +36,10 @@ TABLE [IF NOT EXISTS] tableName (columnName columnTypeDef [, columnName columnTypeDef ...]) -- see Type definition [, INDEX (columnRef [CAPACITY n | TYPE POSTING [DELTA | EF]]) ...] -- see Column indexes [TIMESTAMP (columnName) - [PARTITION BY { NONE | YEAR | MONTH | DAY | HOUR } - [BYPASS WAL | WAL] + [PARTITION BY { NONE | YEAR | MONTH | WEEK | DAY | HOUR } [ TTL n { HOUR[S] | DAY[S] | WEEK[S] | MONTH[S] | YEAR[S] } - | STORAGE POLICY ( policyStage [, policyStage ...] ) ]]] + | STORAGE POLICY ( policyStage [, policyStage ...] ) ] + [BYPASS WAL | WAL]]] [DEDUP UPSERT KEYS (columnName [, columnName ...])] [WITH tableParameter] [IN VOLUME 'alias'] @@ -57,10 +57,10 @@ TABLE [IF NOT EXISTS] tableName [, cast(columnRef AS columnTypeDef) ...] -- see Type definition [, INDEX (columnRef [CAPACITY n | TYPE POSTING [DELTA | EF]]) ...] -- see Column indexes [TIMESTAMP (columnName) - [PARTITION BY { NONE | YEAR | MONTH | DAY | HOUR } - [BYPASS WAL | WAL] + [PARTITION BY { NONE | YEAR | MONTH | WEEK | DAY | HOUR } [ TTL n { HOUR[S] | DAY[S] | WEEK[S] | MONTH[S] | YEAR[S] } - | STORAGE POLICY ( policyStage [, policyStage ...] ) ]]] + | STORAGE POLICY ( policyStage [, policyStage ...] ) ] + [BYPASS WAL | WAL]]] [DEDUP UPSERT KEYS (columnName [, columnName ...])] [WITH tableParameter] [IN VOLUME 'alias'] @@ -262,10 +262,14 @@ information on the behavior of this feature. :::note -In QuestDB Enterprise, `TTL` is deprecated — `CREATE TABLE ... TTL` is -rejected with `TTL settings are deprecated, please, create a storage policy -instead`. Use `STORAGE POLICY` instead. If a legacy table has a TTL set, clear -it with `ALTER TABLE SET TTL 0` before setting a storage policy. +In QuestDB Enterprise, use `STORAGE POLICY` instead of `TTL`. +`CREATE TABLE ... TTL` is still accepted for backward compatibility and is +translated into a `STORAGE POLICY(DROP LOCAL ...)`. On an existing table, +`ALTER TABLE SET TTL` with a non-zero value is rejected with `TTL is not +supported on Enterprise tables; use a storage policy instead`. A table can still +carry a TTL if it was created in QuestDB Open Source and later upgraded to +Enterprise; clear that legacy TTL with `ALTER TABLE SET TTL 0` before setting a +storage policy. ::: @@ -299,9 +303,9 @@ A storage policy supports up to four settings: `TO PARQUET`, `TO REMOTE`, positive. A drop stage may not precede the write it depends on (`TO PARQUET` and `TO REMOTE` before `DROP LOCAL`; `DROP LOCAL` before `DROP REMOTE`), while `TO PARQUET` and `TO REMOTE` are independent. Converting a partition to Parquet -removes its native files and serves reads from the Parquet file. `TO REMOTE` -and `DROP REMOTE` are reserved syntax and are currently rejected at SQL parse -time with `'TO REMOTE' is not supported yet` and +removes its native files and serves reads from the Parquet file. Storage +policies currently operate locally only: `TO REMOTE` is accepted and stored but +not yet enforced, and `DROP REMOTE` is rejected at SQL parse time with `'DROP REMOTE' is not supported yet`. To modify a storage policy after table creation, see diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 241caecc5..0d19bae2b 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -144,9 +144,9 @@ CREATE TABLE sensors ( #### Storage policy clause -When a [storage policy](/docs/concepts/storage-policy/) is attached to a table -(Enterprise only), the policy renders as a `STORAGE POLICY(...)` clause in the -`SHOW CREATE TABLE` output: +When an active [storage policy](/docs/concepts/storage-policy/) is attached to a +table (Enterprise only), the policy renders as a `STORAGE POLICY(...)` clause in +the `SHOW CREATE TABLE` output: ```questdb-sql SHOW CREATE TABLE sensor_data; @@ -160,9 +160,9 @@ CREATE TABLE 'sensor_data' ( STORAGE POLICY(TO PARQUET 3 DAYS, DROP LOCAL 1 MONTH) WAL; ``` -Stages that are not configured on the policy are omitted from the clause. A -disabled policy (`ALTER TABLE ... DISABLE STORAGE POLICY`) still renders — the -disabled state is not part of the DDL. See +Stages that are not configured on the policy are omitted from the clause. Only +an active policy renders: after `ALTER TABLE ... DISABLE STORAGE POLICY`, the +policy is not shown in `SHOW CREATE TABLE`. See [ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/). #### Enterprise variant