diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index fd08cbb65..001de1b61 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -17,12 +17,17 @@ This page tracks significant updates to the QuestDB documentation. ### New - [Migrate QuestDB onto the Kubernetes Operator](/docs/enterprise-kubernetes-operator/getting-started/migrate/) - Move an existing Enterprise deployment onto the Operator with a replica-first cutover: restore the source backup, consume replication WAL, then promote after a controlled source drain +- [ALTER LIVE VIEW](/docs/query/sql/alter-live-view/) - New reference page for managing a live view's disk tier: `SET TTL`, `DROP PARTITION`, `CONVERT PARTITION` between native and Parquet, `RESUME WAL` and `SUSPEND WAL`, with the asynchronous apply, why a manual `DROP PARTITION` can be undone by recovery, the rejected clauses, the per-clause permissions, and how replicas apply the statements locally ### Updated - [Kafka connector](/docs/connect/message-brokers/kafka/) - Updated for QWP with a quick start, guidance on preventing duplicates and recovering from outages, and migration steps for existing HTTP pipelines - [Kubernetes Operator](/docs/enterprise-kubernetes-operator/) - Refreshed for Operator 0.2.1 across installation, configuration, high availability, backup and restore, known limitations, troubleshooting, and the generated [API reference](/docs/enterprise-kubernetes-operator/reference/api/) - [Rust](/docs/connect/clients/rust/) and [C and C++](/docs/connect/clients/c-and-cpp/) clients - Documented the UUID byte order, which was previously unstated. `Buffer::column_uuid` takes `(lo, hi)` where `hi` is the most significant half, matching `java.util.UUID`, and the chunk setter takes the 16 bytes in canonical RFC-4122 order +- [CREATE LIVE VIEW](/docs/query/sql/create-live-view/#ttl) - Documented the `TTL` clause, its partition-size granularity rule, and how the window is enforced during the initial seed +- [Live views](/docs/concepts/live-views/#retention-and-storage-format) - New retention and storage format section, replacing the "no TTL on the view" limitation, plus the replication behavior of the durable-tier statements +- [TTL](/docs/concepts/ttl/#on-live-views) - Live views take TTL at creation and afterwards, evaluated when the view's own table commits +- [Parquet](/docs/concepts/parquet/#bloom-filters-for-in-place-conversion) - Corrected the in-place conversion example: `ALTER TABLE ... CONVERT PARTITION` names the option `fpp` and requires a quoted value, while `bloom_filter_fpp` is the `COPY ... TO` spelling ## August 2026 diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 3271bd516..174d46a26 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -262,6 +262,37 @@ supported. A keep-last `UPSERT` replacement at an earlier timestamp is reflected in the view. A view over a deduplicated base is one `FLUSH EVERY` cycle behind rather than sub-cycle fresh, because its refresh is coupled to base apply. +## Retention and storage format + +A live view's disk tier is a WAL-backed table, and +[`ALTER LIVE VIEW`](/docs/query/sql/alter-live-view/) manages it with the same +verbs a table has: + +- [`SET TTL`](/docs/query/sql/alter-live-view/#set-ttl), or the `TTL` clause of + [`CREATE LIVE VIEW`](/docs/query/sql/create-live-view/#ttl), drops partitions + older than a retention window. The view enforces it on its own commits, so the + window holds without operator action. +- [`DROP PARTITION`](/docs/query/sql/alter-live-view/#drop-partition) removes + partitions on demand. +- [`CONVERT PARTITION`](/docs/query/sql/alter-live-view/#convert-partition) + moves partitions between the native and Parquet storage formats. + +A view's retention is independent of its base table's. The view stores computed +rows, so a wide projection can outgrow the base table even when both keep the +same window. + +A live view is derived rather than ingested, which gives its retention two +properties a table's does not have. The +[`ALTER LIVE VIEW`](/docs/query/sql/alter-live-view/) page covers both in detail: + +- **A manual `DROP PARTITION` is not permanent.** An out-of-order base commit, or + a rebuild from the view's `START FROM` boundary, recomputes output over the + dropped period and writes those rows back. TTL survives recovery, because the + rule is re-applied to the recomputed rows. +- **The two tiers evict independently.** Rows removed from disk can still be + served from the in-memory tier until it is rebuilt, so neither statement makes + data unreadable at a known point in time. + ## Monitoring The [`live_views()`](/docs/query/functions/meta/#live_views) function exposes @@ -343,14 +374,16 @@ in [Base table lifecycle](#base-table-lifecycle). - **Deterministic queries only.** Non-deterministic functions such as `now()`, `sysdate()`, `systimestamp()`, and `rnd_*()` are rejected in the projection, the `WHERE` filter, and window-function arguments. -- **No TTL on the view.** Live-view disk growth is unbounded in this version. - Size retention on the base table instead. +- **Retention is partition-granular.** A live view drops whole partitions, as a + table does. Choose the view's `PARTITION BY` with the retention window in + mind, and see [Retention and storage format](#retention-and-storage-format). ## Tradeoffs - **Storage grows with output.** The computed rows are stored on the live view's disk tier in addition to the base table's rows. For wide projections or long - retention the view's footprint can exceed the base table. + retention the view's footprint can exceed the base table. Bound it with a + [TTL](#retention-and-storage-format) on the view. - **No admission control.** A view that cannot keep up with ingestion stays correct but stale, with no automatic throttle or drop. - **Per-partition state for partitioned windows grows with distinct partition @@ -391,6 +424,14 @@ A role switch continues the local refresh state; it does not reconstruct or transfer the former primary's live-view rows. Replica freshness therefore also depends on base-table replication and apply lag. +[`ALTER LIVE VIEW`](/docs/query/sql/alter-live-view/) statements that manage the +disk tier do replicate. They travel over a replicated control table and each node +applies them to its own copy of the view, holding a partition change until its own +refresh has reached the base-table progress the primary had when the change was +taken. A +node with live views or refresh disabled applies them through the ordinary WAL +apply job. + ### Backup and restore A live view is captured by the object-store backup like a materialized view: its @@ -402,6 +443,8 @@ base table. - **SQL commands** - [`CREATE LIVE VIEW`](/docs/query/sql/create-live-view/): Create a live view + - [`ALTER LIVE VIEW`](/docs/query/sql/alter-live-view/): Manage a live view's + retention, storage format and WAL - [`DROP LIVE VIEW`](/docs/query/sql/drop-live-view/): Remove a live view - **Related concepts** diff --git a/documentation/concepts/parquet.md b/documentation/concepts/parquet.md index fd48b3a0d..ace53a461 100644 --- a/documentation/concepts/parquet.md +++ b/documentation/concepts/parquet.md @@ -131,14 +131,15 @@ optionally set the false positive probability (FPP) using `WITH`: ```questdb-sql title="Convert with explicit bloom filter columns" ALTER TABLE trades CONVERT PARTITION TO PARQUET WHERE timestamp < '2025-08-31' -WITH (bloom_filter_columns = 'symbol,side', bloom_filter_fpp = 0.01); +WITH (bloom_filter_columns = 'symbol,side', fpp = '0.01'); ``` :::note When an explicit `bloom_filter_columns` list is provided, it overrides any per-column `PARQUET(BLOOM_FILTER)` metadata on the table. If the option is -omitted, per-column metadata is used. +omitted, per-column metadata is used. The option is named `fpp` here, and its +value must be quoted. `COPY ... TO` spells the same setting `bloom_filter_fpp`. ::: diff --git a/documentation/concepts/ttl.md b/documentation/concepts/ttl.md index 9c1ee9f40..81cd713f6 100644 --- a/documentation/concepts/ttl.md +++ b/documentation/concepts/ttl.md @@ -90,6 +90,33 @@ A view's TTL is independent of its base table's TTL. For full syntax, see and [ALTER MATERIALIZED VIEW SET TTL](/docs/query/sql/alter-mat-view-set-ttl/). +### On live views + +[Live views](/docs/concepts/live-views/) take TTL the same way, bounding the +disk tier that holds their computed rows: + +```questdb-sql +-- At view creation +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +PARTITION BY DAY +TTL 4 WEEKS +START FROM NOW +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; + +-- On an existing view +ALTER LIVE VIEW trades_ma SET TTL 4 WEEKS; +``` + +A live view evaluates its TTL when its own table commits, which happens on the +`FLUSH EVERY` cadence while the view is producing rows. For full syntax, see +[CREATE LIVE VIEW](/docs/query/sql/create-live-view/#ttl) and +[ALTER LIVE VIEW SET TTL](/docs/query/sql/alter-live-view/#set-ttl). + ## How TTL works TTL drops partitions based on the **partition's time range**, not individual row diff --git a/documentation/query/sql/alter-live-view.md b/documentation/query/sql/alter-live-view.md new file mode 100644 index 000000000..5ae5b2ee6 --- /dev/null +++ b/documentation/query/sql/alter-live-view.md @@ -0,0 +1,347 @@ +--- +title: ALTER LIVE VIEW +sidebar_label: ALTER LIVE VIEW +description: + ALTER LIVE VIEW SQL keyword reference documentation, covering TTL, partition + retention, Parquet conversion and WAL recovery on a live view. +--- + +Manages the disk tier of a [live view](/docs/concepts/live-views/): how long its +computed rows are retained, the storage format of its partitions, and the +recovery of its WAL writer. Retention keeps a view's footprint bounded, and +Parquet conversion compresses the older partitions a view no longer appends to. + +A live view's query, output schema and refresh cadence are fixed at creation and +cannot be altered. `ALTER LIVE VIEW` accepts five clauses: + +- [`CONVERT PARTITION`](#convert-partition) switches partitions between the + native and Parquet storage formats. +- [`DROP PARTITION`](#drop-partition) removes partitions from disk immediately. +- [`RESUME WAL`](#resume-wal) restarts a suspended view. +- [`SET TTL`](#set-ttl) sets a retention period the view enforces itself. +- [`SUSPEND WAL`](#suspend-wal) stops the view's apply on purpose. + +## Syntax + +```questdb-sql title="CONVERT PARTITION" +ALTER LIVE VIEW viewName CONVERT PARTITION TO PARQUET + { LIST partitionName [, partitionName ...] | WHERE booleanExpression } + [ WITH ( parquetOption [, parquetOption ...] ) ]; + +ALTER LIVE VIEW viewName CONVERT PARTITION TO NATIVE + { LIST partitionName [, partitionName ...] | WHERE booleanExpression }; +``` + +```questdb-sql title="DROP PARTITION" +ALTER LIVE VIEW viewName DROP PARTITION LIST partitionName [, partitionName ...]; + +ALTER LIVE VIEW viewName DROP PARTITION WHERE booleanExpression; +``` + +```questdb-sql title="RESUME WAL" +ALTER LIVE VIEW viewName RESUME WAL [ FROM { TRANSACTION | TXN } sequencerTxn ]; +``` + +```questdb-sql title="SET TTL" +ALTER LIVE VIEW viewName SET TTL + n { HOUR[S] | DAY[S] | WEEK[S] | MONTH[S] | YEAR[S] }; +``` + +```questdb-sql title="SUSPEND WAL" +ALTER LIVE VIEW viewName SUSPEND WAL; +``` + +Where [`parquetOption`](#parquet-options) is a bloom filter setting for the +conversion, and `booleanExpression` filters on the view's designated timestamp. + +## Parameters + +| Parameter | Description | +| --------- | ----------- | +| `viewName` | Name of the live view to modify | +| `partitionName` | Partition directory name, following the [partition naming convention](/docs/concepts/partitions/) | +| `booleanExpression` | Predicate on the view's designated timestamp, selecting the partitions to act on | +| `n` | Number of time units to retain | +| `sequencerTxn` | Transaction to resume from. Defaults to the failed transaction | + +## How the change is applied + +A live view holds its computed rows in two tiers: an in-memory tier that serves +fresh reads, and a WAL-backed disk tier written on the `FLUSH EVERY` cadence. +`ALTER LIVE VIEW` changes the disk tier, and it does so asynchronously, the way +a WAL table does. + +The statement returns once the change is committed to the view's sequencer. The +refresh worker applies it alongside the next flush, or on its next scan if the +view is idle. `FLUSH EVERY` sets the cadence, not a deadline: writer contention, +apply backoff and an in-flight out-of-order repair can all delay the change. + +Rows the change removes from disk can still be served from the in-memory tier +until that tier is rebuilt, so neither `DROP PARTITION` nor `SET TTL` makes data +unreadable at a known point in time. + +A suspended view applies nothing until [`RESUME WAL`](#resume-wal), including +these statements. Suspension is visible in +[`wal_tables()`](/docs/query/functions/meta/#wal_tables). + +## CONVERT PARTITION + +Converts partitions of the view's disk tier between QuestDB's native format and +[Parquet](/docs/concepts/parquet/). This changes the storage format only: the +rows, the view's output and every query over it are unaffected. + +```questdb-sql title="Convert older partitions to Parquet" +ALTER LIVE VIEW trades_ma CONVERT PARTITION TO PARQUET +WHERE timestamp < '2026-08-01'; +``` + +```questdb-sql title="Convert one partition back to native" +ALTER LIVE VIEW trades_ma CONVERT PARTITION TO NATIVE LIST '2026-07-15'; +``` + +Convert partitions the view has moved past. The newest partition takes every +flush, and a write into a Parquet partition is a merge that rewrites the whole +file, so leaving it native keeps the flush cheap. + +### Parquet options + +`WITH` accepts the same bloom filter options as +[`ALTER TABLE CONVERT PARTITION`](/docs/concepts/parquet/#bloom-filters-for-in-place-conversion), +and only when converting to Parquet: + +- `bloom_filter_columns = 'col[,col ...]'` builds bloom filters for the listed + columns, enabling row group pruning for equality and `IN` queries. +- `fpp = 'probability'` sets the false positive probability, exclusive between 0 + and 1. Quote the value. + +```questdb-sql title="Convert with bloom filters" +ALTER LIVE VIEW trades_ma CONVERT PARTITION TO PARQUET +LIST '2026-07-15' +WITH (bloom_filter_columns = 'symbol', fpp = '0.01'); +``` + +:::caution + +`WITH` options are not stored on the view. Anything that rewrites a converted +partition later re-encodes it from the server's Parquet configuration, without +the bloom filters. A live view has no per-column `PARQUET()` metadata to fall +back on, so bloom filters set here survive only until the partition is next +rewritten. + +::: + +### Out-of-order base commits over a Parquet partition + +Rows cannot be removed from a Parquet file in place. When a late base-table +commit forces the view to correct output it has already written, the writer +converts every Parquet partition the correction covers back to native, applies +the correction, and converts them back. Reads of those partitions are unaffected +and the partitions are Parquet again when the apply finishes. + +The cost is a full decode and re-encode of each covered partition, per +correction. A correction that cannot anchor on a checkpoint covers the view's +whole range, and then pays it for every Parquet partition the view holds. A view +over a base table that takes frequent out-of-order writes is a poor candidate +for Parquet conversion. + +If the process dies during that rewrite, the partition is left native and the +next writer to open the view finishes the re-encode. Nothing is lost but the +compaction, and `CONVERT PARTITION TO PARQUET` restores it. + +## DROP PARTITION + +Removes whole partitions from the view's disk tier. Both selectors of +[`ALTER TABLE DROP PARTITION`](/docs/query/sql/alter-table-drop-partition/) are +accepted, and partition names follow the same convention. Inspect the view's +partitions with +[`table_partitions()`](/docs/query/functions/meta/#table_partitions). + +```questdb-sql title="Drop a partition by name" +ALTER LIVE VIEW trades_ma DROP PARTITION LIST '2026-07-15'; +``` + +```questdb-sql title="Drop everything before a date" +ALTER LIVE VIEW trades_ma DROP PARTITION WHERE timestamp < '2026-08-01'; +``` + +The newest partition cannot be dropped. The refresh pipeline appends to it and +an out-of-order correction rewrites it, so it is rejected with +`cannot drop the active partition of a live view [partition=...]`, both when the +statement is compiled and again when it is applied. + +:::caution + +`DROP PARTITION` removes durable rows now, and only now. A live view is derived +from its base table, so any later recovery that recomputes output over the +dropped period brings those rows back: + +- an out-of-order base commit whose correction range covers the dropped period + re-emits the overlapping rows; +- a restart that cannot resume from the view's checkpoint rebuilds the view from + its `START FROM` boundary, re-materializing every dropped partition the base + table still holds. + +For retention that survives recovery, use [`SET TTL`](#set-ttl), which the view +re-enforces on every commit. + +::: + +## RESUME WAL + +Restarts WAL transactions on a live view after the error that suspended it has +been resolved. It behaves as +[`ALTER TABLE RESUME WAL`](/docs/query/sql/alter-table-resume-wal/) does, and +recovers only a suspended WAL writer. It does not revalidate a view that was +invalidated by a base-table schema change. + +```questdb-sql title="Resume from the failed transaction" +ALTER LIVE VIEW trades_ma RESUME WAL; +``` + +```questdb-sql title="Skip past a transaction" +ALTER LIVE VIEW trades_ma RESUME WAL FROM TRANSACTION 5; +``` + +## SET TTL + +Sets a [time-to-live](/docs/concepts/ttl/) period on the view's disk tier, +dropping partitions whose entire time range falls outside the window. A view's +TTL is independent of its base table's TTL. + +```questdb-sql title="Keep four weeks of computed rows" +ALTER LIVE VIEW trades_ma SET TTL 4 WEEKS; +``` + +```questdb-sql title="Shorthand form" +ALTER LIVE VIEW trades_ma SET TTL 12h; +``` + +Accepted units are `HOUR[S]`, `DAY[S]`, `WEEK[S]`, `MONTH[S]` and `YEAR[S]`, with +the `h`, `d`, `w`, `M` and `y` shorthands. The period must be a whole number +multiple of the view's partition size, which is the view's `PARTITION BY` if it +declared one, and the base table's scheme otherwise. Reference-time and +partition-boundary rules are the table rules, described in +[TTL](/docs/concepts/ttl/). + +Clearing a TTL needs a unit as well: + +```questdb-sql title="Clear the retention period" +ALTER LIVE VIEW trades_ma SET TTL 0h; +``` + +The view evaluates its TTL whenever its own table commits, which is the flush +cadence while the view is producing rows, plus the commit that applies this +statement. A view that has stopped producing output stops evicting. + +Unlike [`DROP PARTITION`](#drop-partition), a TTL survives recovery: a rebuilt +view re-applies the same rule to the recomputed rows, so the retention window +converges again without operator action. + +Read the current setting from [`tables()`](/docs/query/functions/meta/#tables), +where a `ttlValue` of `0` means no TTL: + +```questdb-sql title="Check the retention period" +SELECT table_name, ttlValue, ttlUnit FROM tables() +WHERE table_name = 'trades_ma'; +``` + +`SHOW CREATE LIVE VIEW` re-emits a non-zero TTL, so a view altered here +round-trips through its own DDL. + +:::caution + +On QuestDB Enterprise, TTL is superseded by +[storage policy](/docs/concepts/storage-policy/) for tables, and a non-zero +`SET TTL` on a live view is currently rejected with +`TTL is not supported on Enterprise tables; use a storage policy instead`. +`SET TTL 0` is accepted, and the `TTL` clause of +[`CREATE LIVE VIEW`](/docs/query/sql/create-live-view/#ttl) is accepted. + +::: + +## SUSPEND WAL + +Stops the apply of the view's WAL, leaving the view quiescent. It behaves as +[`ALTER TABLE SUSPEND WAL`](/docs/query/sql/alter-table-suspend-wal/) does: +refresh keeps computing and committing to the sequencer, nothing is applied, and +queries stop seeing new rows until `RESUME WAL` drains the queued transactions in +order. Suspending and resuming share a single authorization. + +```questdb-sql title="Suspend a live view" +ALTER LIVE VIEW trades_ma SUSPEND WAL; +``` + +## Unsupported clauses + +`ALTER TABLE` never reaches a live view. It fails with `cannot modify live view`, +whichever clause follows. + +Every `ALTER LIVE VIEW` clause outside the five above is rejected, because a live +view's schema is a function of its `SELECT`. That covers `ADD COLUMN`, +`ALTER COLUMN`, `RENAME`, `ATTACH PARTITION`, `DETACH PARTITION`, +`SQUASH PARTITIONS`, `DEDUP`, `SET PARAM`, `SET TYPE` and `SET FORMAT`. + +`FORCE DROP PARTITION` is rejected separately, with +`FORCE DROP PARTITION is not supported on live views`. On a table it bypasses the +WAL and writes through a directly acquired writer, which on a live view is owned +by the refresh worker. The recovery a live view has is `SUSPEND WAL`, +`RESUME WAL` and the ordinary sequenced `DROP PARTITION`. + +## Permissions (Enterprise) + +Each clause is authorized with the same permission its `ALTER TABLE` counterpart +uses, checked against the live view: + +| Clause | Permission | +| ------ | ---------- | +| `CONVERT PARTITION TO NATIVE` | `CONVERT PARTITION TO NATIVE` | +| `CONVERT PARTITION TO PARQUET` | `CONVERT PARTITION TO PARQUET` | +| `DROP PARTITION` | `DROP PARTITION` | +| `RESUME WAL`, `SUSPEND WAL` | `RESUME WAL` | +| `SET TTL` | `SET TABLE PARAM` | + +```questdb-sql title="Grant retention management on one view" +GRANT DROP PARTITION, SET TABLE PARAM ON trades_ma TO user1; +``` + +See [Role-based access control](/docs/security/rbac/) for the full model. + +## Replication (Enterprise) + +A live view's rows are node-local: every node with live views enabled computes +and flushes its own copy, and live-view WAL is never transferred between nodes. +These statements are relayed to replicas over a replicated control table and +applied by each node to its own copy of the view, so retention and storage format +converge without shipping rows. + +A replica holds a relayed partition change until its own refresh has reached the +base-table progress the primary had when it took the change, so both nodes remove +or convert the same rows. `SET TTL` carries no such fence, since the view applies +the rule to whatever it holds. A `WHERE` selector is resolved to a concrete partition list on the +primary and travels as that list, which keeps the two nodes from resolving the +same predicate against different data. A node with live views or refresh disabled +applies the change through the ordinary WAL apply job. + +## Errors + +| Error | Cause | +| ----- | ----- | +| `cannot modify live view` | `ALTER TABLE` was used on a live view. Use `ALTER LIVE VIEW` | +| `'set', 'drop', 'convert', 'resume' or 'suspend' expected` | The clause is not part of the `ALTER LIVE VIEW` grammar | +| `'ttl' expected` | `SET` was followed by something other than `TTL` | +| `FORCE DROP PARTITION is not supported on live views` | `FORCE DROP PARTITION` was used on a live view | +| `cannot drop the active partition of a live view [partition=...]` | The dropped partition is the one the view is appending to | +| `TTL value must be an integer multiple of the partition size` | The TTL period is not a whole multiple of the view's partition size | +| `missing unit, 'HOUR(S)', 'DAY(S)', 'WEEK(S)', 'MONTH(S)' or 'YEAR(S)' expected` | `SET TTL 0` was written without a unit | +| `no partitions matched WHERE clause` | The `WHERE` selector matched no partition when the statement was compiled | +| `bloom_filter_columns or fpp expected` | An unknown option was passed to `WITH` | +| `permission denied` | Missing permission (Enterprise) | + +## See also + +- [Live views concept](/docs/concepts/live-views/) +- [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) +- [DROP LIVE VIEW](/docs/query/sql/drop-live-view/) +- [TTL concept](/docs/concepts/ttl/) +- [Parquet](/docs/concepts/parquet/) +- [table_partitions()](/docs/query/functions/meta/#table_partitions) diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index 6c6fd579d..d5b7266ec 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -25,6 +25,7 @@ CREATE LIVE VIEW [ IF NOT EXISTS ] viewName FLUSH EVERY duration [ IN MEMORY duration ] [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) ] +[ TTL n { HOUR[S] | DAY[S] | WEEK[S] | MONTH[S] | YEAR[S] } ] START FROM ( NOW | BEGINNING | 'timestamp' ) AS [ ( ] query [ ) ] [ OWNED BY ownerName ] @@ -38,9 +39,9 @@ Where: [window functions](/docs/query/functions/window-functions/overview/). `FLUSH EVERY` is required and must come first. `START FROM` is also required and -may appear in any order with the optional `IN MEMORY` and `PARTITION BY` -clauses. These clauses all precede `AS`; the optional `OWNED BY` clause follows -the query. +may appear in any order with the optional `IN MEMORY`, `PARTITION BY` and `TTL` +clauses. Each may appear at most once. These clauses all precede `AS`; the +optional `OWNED BY` clause follows the query. ## Parameters @@ -51,6 +52,7 @@ the query. | `FLUSH EVERY` | How often computed rows are persisted to disk. Required | | `IN MEMORY` | Window of recent rows kept in RAM for fresh reads. Defaults to `FLUSH EVERY` | | `PARTITION BY` | Partitioning unit for the view's disk tier. Defaults to the base table's scheme | +| `TTL` | Retention period for the view's disk tier. No retention by default | | `START FROM` | Inclusive event-time boundary: `NOW`, `BEGINNING`, or a timestamp literal. Required | | `query` | A window-function `SELECT` over a single WAL-backed base table | | `OWNED BY` | Assign ownership (Enterprise) | @@ -122,6 +124,37 @@ SELECT timestamp, symbol, FROM trades; ``` +### TTL + +`TTL` sets a [time-to-live](/docs/concepts/ttl/) period on the view's disk tier. +QuestDB drops partitions of the view whose entire time range falls outside the +window, which bounds the view's footprint without touching the base table. A +view's TTL is independent of its base table's TTL. + +The period must be a whole number multiple of the view's partition size, whether +that comes from `PARTITION BY` or from the base table. Accepted units are +`HOUR[S]`, `DAY[S]`, `WEEK[S]`, `MONTH[S]` and `YEAR[S]`, with the `h`, `d`, `w`, +`M` and `y` shorthands. + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +PARTITION BY DAY +TTL 4 WEEKS +START FROM NOW +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +The window is enforced against the view's own rows as the refresh worker writes +them, so a view created with `START FROM BEGINNING` and a short `TTL` still +recomputes the whole history during its initial seed, evicting as it goes. Change +or clear the period afterwards with +[`ALTER LIVE VIEW SET TTL`](/docs/query/sql/alter-live-view/#set-ttl). + ### START FROM `START FROM` defines the inclusive event-time boundary for rows in the live @@ -336,11 +369,14 @@ OWNED BY analysts; | `wildcard column select is not allowed in live view queries` | The top-level projection contains `*` | | `live view unbounded window must have an ANCHOR clause` | A stateful partitioned window uses the default unbounded frame without an anchor | | `non-deterministic function cannot be used in live view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | +| `TTL value must be an integer multiple of the partition size` | The `TTL` period is not a whole multiple of the view's partition size | +| `live view TTL clause specified more than once` | The `TTL` clause appears twice | | `permission denied` | Missing required permission (Enterprise) | ## See also - [Live views concept](/docs/concepts/live-views/) +- [ALTER LIVE VIEW](/docs/query/sql/alter-live-view/) - [DROP LIVE VIEW](/docs/query/sql/drop-live-view/) - [Window functions](/docs/query/functions/window-functions/overview/) - [live_views()](/docs/query/functions/meta/#live_views) diff --git a/documentation/sidebars.js b/documentation/sidebars.js index f0b4f587f..9dc4d761c 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -329,6 +329,7 @@ module.exports = { "query/sql/alter-table-change-symbol-capacity", // SYMBOL CAPACITY ], }, + "query/sql/alter-live-view", { type: "category", label: "ALTER MATERIALIZED VIEW",