diff --git a/TOC-tidb-cloud-lake.md b/TOC-tidb-cloud-lake.md index 90d850110c7f0..6d0d219539976 100644 --- a/TOC-tidb-cloud-lake.md +++ b/TOC-tidb-cloud-lake.md @@ -248,7 +248,7 @@ - [OPTIMIZE TABLE](/tidb-cloud-lake/sql/optimize-table.md) - [FLASHBACK TABLE](/tidb-cloud-lake/sql/flashback-table.md) - [ALTER TABLE](/tidb-cloud-lake/sql/alter-table.md) - - [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md) + - [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md) - [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md) - [ATTACH TABLE](/tidb-cloud-lake/sql/attach-table.md) - [SHOW CREATE TABLE](/tidb-cloud-lake/sql/show-create-table.md) @@ -525,8 +525,10 @@ - [SHOW TABLE FUNCTIONS](/tidb-cloud-lake/sql/show-table-functions.md) - [SHOW PROCESSLIST](/tidb-cloud-lake/sql/show-processlist.md) - [SHOW METRICS](/tidb-cloud-lake/sql/show-metrics.md) - - [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table-sql.md) - - [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table-sql.md) + - [VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md) + - [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md) + - [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md) + - [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md) - [VACUUM TEMPORARY FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md) - [VACUUM VIRTUAL COLUMN](/tidb-cloud-lake/sql/vacuum-virtual-column.md) - [EXECUTE IMMEDIATE](/tidb-cloud-lake/sql/execute-immediate.md) diff --git a/tidb-cloud-lake/guides/data-management.md b/tidb-cloud-lake/guides/data-management.md deleted file mode 100644 index c0bb769c62bf6..0000000000000 --- a/tidb-cloud-lake/guides/data-management.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: Data Management -summary: An overview of data management in TiDB Cloud Lake, covering data lifecycle, recovery, protection, and recycling. ---- - -# Data Management - -| Category | Description | Key Features | Common Operations | -|----------|-------------|--------------|------------------| -| **[Data Lifecycle](/tidb-cloud-lake/guides/data-lifecycle.md)** | Create and manage objects | • Database & Table
• External Tables
• Streams & Views
• Indexes & Stages | • CREATE/DROP/ALTER
• SHOW TABLES
• DESCRIBE TABLE | -| **[Data Protection](/tidb-cloud-lake/guides/data-protection.md)** | Secure access and prevent loss | • Network Policies
• Access Control
• Time Travel & Fail-safe
• Data Encryption | • NETWORK POLICY
• GRANT/REVOKE
• USER/ROLE | -| **[Data Recycle](/tidb-cloud-lake/guides/data-purge-and-recycle.md)** | Free up storage space | • VACUUM Commands
• Retention Policies
• Orphan File Cleanup
• Temporary File Management | • VACUUM TABLE
• VACUUM DROP TABLE
• DATA_RETENTION_TIME | diff --git a/tidb-cloud-lake/guides/data-purge-and-recycle.md b/tidb-cloud-lake/guides/data-purge-and-recycle.md index 2bc8f9547c5ee..e2fdce0083595 100644 --- a/tidb-cloud-lake/guides/data-purge-and-recycle.md +++ b/tidb-cloud-lake/guides/data-purge-and-recycle.md @@ -5,135 +5,88 @@ summary: In {{{ .lake }}}, data is not immediately deleted when you run DROP, TR # Data Purge and Recycle -## Overview - -In {{{ .lake }}}, data is not immediately deleted when you run `DROP`, `TRUNCATE`, or `DELETE` commands. This enables {{{ .lake }}}'s time travel feature, allowing you to access previous states of your data. However, this approach means that storage space is not automatically freed up after these operations. - -``` -Before DELETE: After DELETE: After VACUUM: -+----------------+ +----------------+ +----------------+ -| Current Data | | New Version | | Current Data | -| | | (After DELETE) | | (After DELETE) | -+----------------+ +----------------+ +----------------+ -| Historical Data| | Historical Data| | | -| (Time Travel) | | (Original Data)| | | -+----------------+ +----------------+ +----------------+ - Storage not freed Storage freed -``` - -## VACUUM Commands and Cleanup Scope - -{{{ .lake }}} provides three VACUUM commands with **different cleanup scopes**. Understanding what each command cleans is crucial for data management. +This document describes how to reclaim storage space in {{{ .lake }}} after deleting rows or dropping tables. It also explains how to manage temporary spill files and inactive temporary-table sessions. -``` -VACUUM DROP TABLE -├── Target: Dropped tables (after DROP TABLE command) -├── S3 Storage: ✅ Removes ALL data (files, segments, blocks, indexes, statistics) -├── Meta Service: ✅ Removes ALL metadata (schema, permissions, records) -└── Result: Complete table removal - CANNOT be recovered - -VACUUM TABLE -├── Target: Historical data and orphan files for active tables -├── S3 Storage: ✅ Removes old snapshots, orphan segments/blocks, indexes/stats -├── Meta Service: ❌ Preserves table structure and current metadata -└── Result: Table stays active, only history cleaned - -VACUUM TEMPORARY FILES -├── Target: Temporary spill files from queries (joins, sorts, aggregates) -├── S3 Storage: ✅ Removes temp files from crashed/interrupted queries -├── Meta Service: ❌ No metadata (temp files don't have any) -└── Result: Storage cleanup only, rarely needed -``` +## Overview ---- +Deleting rows or dropping a table does not necessarily release its storage immediately. {{{ .lake }}} retains historical data and dropped objects for recovery. `VACUUM` reclaims storage once data becomes eligible for cleanup. Cleaned history and dropped objects cannot be recovered. -> **🚨 Critical**: Only `VACUUM DROP TABLE` affects the meta service. Other commands only clean storage files. +## Choose a Cleanup Scope -## Using VACUUM Commands +| Command | Cleanup scope | Effect | +|---------|---------------|--------| +| [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md) | One writable FUSE table | Removes eligible history while preserving the table and current data. | +| [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md) | Writable FUSE tables in a specified database, or all non-system databases in the current catalog | Performs the same historical cleanup in bulk. | +| [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md) | Dropped objects in a specified database, or all databases in the current catalog, including dropped databases | Removes eligible dropped objects, their storage, and their metadata. | +| [VACUUM TEMPORARY FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md) | Tenant temporary spill files and inactive temporary-table sessions | Cleans temporary storage. | +| [VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md) | Table history, dropped objects, then temporary files | Runs the three cleanup steps in order using their respective retention rules. | -The VACUUM command family is the primary method for cleaning data in {{{ .lake }}}. +Single-table cleanup requires `SUPER` access to the table. Database-scoped batch or dropped-object cleanup requires `SUPER` access to that database. Batch table and dropped-object cleanup without FROM, VACUUM ALL, and temporary-file cleanup require global `SUPER` privilege. -### VACUUM DROP TABLE +Batch table cleanup skips non-FUSE and read-only tables. Ordinary per-table failures are logged and other tables are processed; cancellation and errors listing databases or tables can stop the operation. These commands do not return result sets. -Permanently removes dropped tables from both storage and metadata. +## Clean Table History ```sql -VACUUM DROP TABLE [FROM ] [DRY RUN [SUMMARY]] [LIMIT ]; +VACUUM TABLE default.my_table; ``` -**Options:** - -- `FROM `: Restrict to a specific database -- `DRY RUN [SUMMARY]`: Preview files to be removed without actually deleting them -- `LIMIT `: Limit the number of files to be vacuumed - -**Examples:** +Compaction combines small blocks and segments. To compact first and then reclaim eligible historical storage: ```sql --- Preview files that would be removed -VACUUM DROP TABLE DRY RUN; +OPTIMIZE TABLE default.my_table COMPACT; +VACUUM TABLE default.my_table; +``` --- Preview summary of files that would be removed -VACUUM DROP TABLE DRY RUN SUMMARY; +For batch cleanup: --- Remove dropped tables from the "default" database -VACUUM DROP TABLE FROM default; +```sql +-- One database +VACUUM TABLES FROM default; --- Remove up to 1000 files from dropped tables -VACUUM DROP TABLE LIMIT 1000; +-- All non-system databases in the current catalog +VACUUM TABLES; ``` -### VACUUM TABLE - -Removes historical data and orphan files for active tables (storage-only cleanup). +## Clean Dropped Objects ```sql -VACUUM TABLE [DRY RUN [SUMMARY]]; -``` +-- One database +VACUUM DROPPED OBJECTS FROM default; -**Options:** +-- All databases in the current catalog, including dropped databases +VACUUM DROPPED OBJECTS; +``` -- `DRY RUN [SUMMARY]`: Preview files to be removed without actually deleting them +This removes eligible dropped objects and their metadata as well as storage. They can no longer be recovered with `UNDROP`. -**Examples:** +## Clean Temporary Files or Run All Steps ```sql --- Preview files that would be removed -VACUUM TABLE my_table DRY RUN; +VACUUM TEMPORARY FILES; +``` --- Preview summary of files that would be removed -VACUUM TABLE my_table DRY RUN SUMMARY; +To run table-history, dropped-object, and temporary-file cleanup in sequence: --- Remove historical data from my_table -VACUUM TABLE my_table; +```sql +VACUUM ALL; ``` -### VACUUM TEMPORARY FILES +A failure propagated by a step prevents later steps from running. Cleanup already completed is not rolled back. + +## Retention and Protection -Removes temporary spill files created during query execution. +For table history and dropped objects, use `data_retention_time_in_days` (1 day by default). For example, set a 2-day retention period for the current session: ```sql -VACUUM TEMPORARY FILES; +SET data_retention_time_in_days = 2; +SHOW SETTINGS LIKE 'data_retention_time_in_days'; ``` -> **Note:** -> -> Rarely needed during normal operation since {{{ .lake }}} automatically handles cleanup. Manual cleanup is typically only required when {{{ .lake }}} crashes during query execution. +Active-table cleanup preserves snapshots and data referenced by unexpired snapshot tags, including tags with no expiration. Expired tags no longer protect history; VACUUM attempts to remove them without aborting cleanup if tag deletion fails. -## Adjusting Data Retention Time - -The VACUUM commands remove data files older than the `DATA_RETENTION_TIME_IN_DAYS` setting. By default, {{{ .lake }}} retains historical data for 1 day (24 hours). You can adjust this setting: +Temporary spill files have a separate retention period of 3 days by default. Override it with RETAIN; this option does not set the lifetime of temporary-table sessions: ```sql --- Change retention period to 2 days -SET GLOBAL DATA_RETENTION_TIME_IN_DAYS = 2; - --- Check current retention setting -SHOW SETTINGS LIKE 'DATA_RETENTION_TIME_IN_DAYS'; +VACUUM TEMPORARY FILES RETAIN 2 DAYS; ``` - -| Edition | Default Retention | Maximum Retention | -| ---------------------------------------- | ----------------- | ---------------- | -| {{{ .lake }}} Community & Enterprise Editions | 1 day (24 hours) | 90 days | -| {{{ .lake }}} (Personal) | 1 day (24 hours) | 1 day (24 hours) | -| {{{ .lake }}} (Business) | 1 day (24 hours) | 90 days | diff --git a/tidb-cloud-lake/sql/administration-commands.md b/tidb-cloud-lake/sql/administration-commands.md index 4cdb1cd787dce..5e068ee0751c1 100644 --- a/tidb-cloud-lake/sql/administration-commands.md +++ b/tidb-cloud-lake/sql/administration-commands.md @@ -44,8 +44,10 @@ This page provides reference information for the system administration commands | Command | Description | |---------|-------------| | **[VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md)** | Reclaim storage space from tables | -| **[VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md)** | Clean up dropped table data | -| **[VACUUM TEMP FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md)** | Remove temporary files | +| **[VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md)** | Clean up eligible dropped objects and their storage and metadata | +| **[VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md)** | Clean table history across a database or the current catalog | +| **[VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md)** | Clean table history, dropped objects, and temporary files | +| **[VACUUM TEMPORARY FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md)** | Remove temporary files | | **[VACUUM VIRTUAL COLUMN](/tidb-cloud-lake/sql/vacuum-virtual-column.md)** | Remove obsolete virtual column files | | **[SHOW INDEXES](/tidb-cloud-lake/sql/show-indexes.md)** | Display table indexes | diff --git a/tidb-cloud-lake/sql/ddl-table-overview.md b/tidb-cloud-lake/sql/ddl-table-overview.md index 8b8fe6d7f3a5b..6cc2ce55bd5c9 100644 --- a/tidb-cloud-lake/sql/ddl-table-overview.md +++ b/tidb-cloud-lake/sql/ddl-table-overview.md @@ -43,13 +43,15 @@ This page provides a comprehensive overview of table operations in {{{ .lake }}} | [TRUNCATE TABLE](/tidb-cloud-lake/sql/truncate-table.md) | Removes all data from a table while preserving the table's schema | [FLASHBACK TABLE](/tidb-cloud-lake/sql/flashback-table.md) | | [DROP TABLE](/tidb-cloud-lake/sql/drop-table.md) | Deletes a table | [UNDROP TABLE](/tidb-cloud-lake/sql/undrop-table.md) | | [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md) | Permanently removes historical data files of a table (Enterprise Edition) | Not recoverable | -| [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md) | Permanently removes data files of dropped tables (Enterprise Edition) | Not recoverable | +| [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md) | Permanently removes eligible dropped objects and their storage and metadata (Enterprise Edition) | Not recoverable | +| [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md) | Cleans eligible history from tables across a database or the current catalog (Enterprise Edition) | Not recoverable | +| [VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md) | Cleans table history, dropped objects, and temporary files (Enterprise Edition) | Not recoverable | ## Table Optimization | Command | Description | |---------|-------------| -| [OPTIMIZE TABLE](/tidb-cloud-lake/sql/optimize-table.md) | Compacts or purges historical data to save storage space and enhance query performance | +| [OPTIMIZE TABLE](/tidb-cloud-lake/sql/optimize-table.md) | Compacts segments and blocks to enhance query performance | | [SET CLUSTER KEY](/tidb-cloud-lake/sql/set-cluster-key.md) | Configures a cluster key to enhance query performance for large tables | > **Note:** diff --git a/tidb-cloud-lake/sql/drop-table.md b/tidb-cloud-lake/sql/drop-table.md index 997353a175292..a53f79b9ee9c9 100644 --- a/tidb-cloud-lake/sql/drop-table.md +++ b/tidb-cloud-lake/sql/drop-table.md @@ -21,13 +21,13 @@ DROP TABLE [ IF EXISTS ] [ . ] This command only marks the table schema as deleted in the metadata service, ensuring that the actual data remains intact. If you need to recover the deleted table schema, you can use the [UNDROP TABLE](/tidb-cloud-lake/sql/undrop-table.md) command. -For completely removing a table along with its data files, consider using the [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md) command. +For completely removing a table along with its data files, consider using the [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md) command. ## Examples ### Deleting a Table -This example highlights the use of the DROP TABLE command to delete the "test" table. After dropping the table, any attempt to SELECT from it results in an "Unknown table" error. It also demonstrates how to recover the dropped "test" table using the UNDROP TABLE command, allowing you to SELECT data from it again. +This example highlights the use of the DROP TABLE command to delete the "test" table. After dropping the table, any attempt to SELECT from it results in an "Unknown table" error. It also demonstrates how to recover the dropped "test" table using the `UNDROP TABLE` command, allowing you to SELECT data from it again. ```sql CREATE TABLE test(a INT, b VARCHAR); diff --git a/tidb-cloud-lake/sql/optimize-table.md b/tidb-cloud-lake/sql/optimize-table.md index 9a831b944d829..a95a2708dda97 100644 --- a/tidb-cloud-lake/sql/optimize-table.md +++ b/tidb-cloud-lake/sql/optimize-table.md @@ -5,13 +5,13 @@ summary: Optimizing a table in {{{ .lake }}} involves compacting or purging hist # OPTIMIZE TABLE -Optimizing a table in {{{ .lake }}} involves compacting or purging historical data to save storage space and enhance query performance. +`OPTIMIZE TABLE` compacts small segments and blocks to improve query performance. To reclaim storage occupied by eligible historical data after compaction, use [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md).
Why Optimize?
{{{ .lake }}} stores data in tables using the Parquet format, which is organized into blocks. Additionally, {{{ .lake }}} supports time travel functionality, where each operation that modifies a table generates a Parquet file that captures and reflects the changes made to the table.

-
As a table accumulates more Parquet files over time, it can lead to performance issues and increased storage requirements. To optimize the table's performance, historical Parquet files can be deleted when they are no longer needed. This optimization can help to improve query performance and reduce the amount of storage space used by the table.
+
As a table accumulates small blocks and segments, queries may need to read more files and metadata. Compaction merges them into larger units to reduce this overhead. Historical files are retained until they become eligible for cleanup by VACUUM TABLE.
## {{{ .lake }}} Data Storage: Snapshot, Segment, and Block @@ -156,7 +156,7 @@ OPTIMIZE TABLE [database.]table_name COMPACT [LIMIT ] Compacts the table data by merging small blocks and segments into larger ones. -- This command creates a new snapshot (along with compacted segments and blocks) of the most recent table data without affecting the existing storage files, so the storage space won't be released until you purge the historical data. +- This command creates a new snapshot (along with compacted segments and blocks) of the most recent table data without affecting the existing storage files. To reclaim storage from eligible historical data, run [`VACUUM TABLE`](/tidb-cloud-lake/sql/vacuum-table.md) after compaction. - Depending on the size of the given table, it may take quite a while to complete the execution. @@ -169,3 +169,10 @@ Compacts the table data by merging small blocks and segments into larger ones. ```sql OPTIMIZE TABLE my_database.my_table COMPACT LIMIT 50; ``` + +To compact and then clean up eligible historical files: + +```sql +OPTIMIZE TABLE my_database.my_table COMPACT; +VACUUM TABLE my_database.my_table; +``` diff --git a/tidb-cloud-lake/sql/vacuum-all.md b/tidb-cloud-lake/sql/vacuum-all.md new file mode 100644 index 0000000000000..92e14d49101e3 --- /dev/null +++ b/tidb-cloud-lake/sql/vacuum-all.md @@ -0,0 +1,30 @@ +--- +title: VACUUM ALL +summary: Runs VACUUM TABLES, VACUUM DROPPED OBJECTS, and VACUUM TEMPORARY FILES in order to reclaim storage. +--- + +# VACUUM ALL + +`VACUUM ALL` cleans table history, dropped objects, and temporary files. `VACUUM ALL` runs the following cleanup operations in order: + +1. [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md): Cleans eligible history from writable FUSE tables across non-system databases in the current catalog. +2. [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md): Cleans eligible dropped objects across databases in the current catalog, including dropped databases. +3. [VACUUM TEMPORARY FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md): Cleans the tenant's temporary spill files and inactive temporary-table sessions using the default retention and no explicit limit. + +Each step follows its own retention and protection rules. Current data in active tables is preserved; cleaned history and dropped objects cannot be recovered. Temporary spill-file retention is separate from `data_retention_time_in_days`. + +## Syntax + +```sql +VACUUM ALL +``` + +Requires global `SUPER` privilege. There is no database filter or command option. The command does not return a result set. + +A failure that propagates from one step stops execution before the following steps. Per-table errors handled by batch cleanup retain the behavior described in [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md). Cleanup already completed is not rolled back. + +## Example + +```sql +VACUUM ALL; +``` diff --git a/tidb-cloud-lake/sql/vacuum-drop-table-sql.md b/tidb-cloud-lake/sql/vacuum-drop-table-sql.md deleted file mode 100644 index 1259fedbfa9ec..0000000000000 --- a/tidb-cloud-lake/sql/vacuum-drop-table-sql.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: VACUUM DROP TABLE -summary: This page describes VACUUM DROP TABLE in TiDB Cloud Lake. ---- - -# VACUUM DROP TABLE - -The VACUUM DROP TABLE command helps save storage space by permanently removing data files of dropped tables, freeing up storage space, and enabling you to manage the process efficiently. It offers optional parameters to target specific databases, preview, and limit the number of data files to be vacuumed. To list the dropped tables of a database, use [SHOW DROP TABLES](/tidb-cloud-lake/sql/show-drop-databases.md). - -See also: [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table-sql.md) - -## Syntax - -```sql -VACUUM DROP TABLE - [ FROM ] - [ DRY RUN [SUMMARY] ] - [ LIMIT ] -``` - -- `FROM `: This parameter restricts the search for dropped tables to a specific database. If not specified, the command will scan all databases, including those that have been dropped. - - ```sql title="Example:" - -- Remove dropped tables from the "default" database - // highlight-next-line - VACUUM DROP TABLE FROM default; - - -- Remove dropped tables from all databases - // highlight-next-line - VACUUM DROP TABLE; - ``` - -- `DRY RUN [SUMMARY]`: When this parameter is specified, data files will not be removed; instead, it returns a result showing which data files would have been removed if this parameter were not specified. See examples in the [Output](#output) section. - -- `LIMIT `: This parameter can be used with or without the DRY RUN parameter. When used with DRY RUN, it limits the number of data files to be shown in the `DRY RUN` result. When used without `DRY RUN`, it limits the number of data files to be vacuumed. - -### Output - -The VACUUM DROP TABLE command returns a result when the `DRY RUN` or `DRY RUN SUMMARY` parameter is specified: - -- `DRY RUN`: Returns a list of up to 1,000 candidate files and their sizes in bytes for each dropped table. -- `DRY RUN SUMMARY`: Returns the total number of files to be removed and their combined size for each dropped table. - -```sql title='Example:' -// highlight-next-line -VACUUM DROP TABLE DRY RUN; - -┌──────────────────────────────────────────────────────────────────┐ -│ table │ file │ file_size │ -├────────┼─────────────────────────────────────────────┼───────────┤ -│ b │ 313ebd4da5cc493f9a7d491da8253ce2_v2.parquet │ 210 │ -│ b │ 737f2215b8ac4a268d5b7f2218273358_v2.parquet │ 210 │ -│ b │ 737f2215b8ac4a268d5b7f2218273358_v4.parquet │ 340 │ -│ b │ 313ebd4da5cc493f9a7d491da8253ce2_v4.parquet │ 340 │ -│ b │ last_snapshot_location_hint │ 72 │ -│ b │ 7e01fa5c2e0a495298942671447dc8cb_v4.mpk │ 515 │ -│ b │ 2bc90e5be55c44258a736d27e5f7ac9e_v4.mpk │ 459 │ -│ b │ 85e73803aabc4eb48774db3d932312dd_v4.mpk │ 534 │ -│ b │ f0e507d0b825428dbfe57c8d8b620a15_v4.mpk │ 533 │ -│ c │ cee790e76f6e4e92bc9dab3b9e873dcf_v2.parquet │ 210 │ -│ c │ 4bcb2cef3b6344cb951908ebee5ceb36_v2.parquet │ 210 │ -│ c │ cee790e76f6e4e92bc9dab3b9e873dcf_v4.parquet │ 340 │ -│ c │ 4bcb2cef3b6344cb951908ebee5ceb36_v4.parquet │ 340 │ -│ c │ last_snapshot_location_hint │ 71 │ -│ c │ 414fc6a8dc6746afbc576cf8fddfcdf3_v4.mpk │ 516 │ -│ c │ 8d0d115c438244c295e3bfd50d556e39_v4.mpk │ 458 │ -│ c │ 28e4f551cc634d3d8d7e648c3baa5f5c_v4.mpk │ 534 │ -│ c │ 007b57e08eda419fbb451a3a3ed71de8_v4.mpk │ 533 │ -└──────────────────────────────────────────────────────────────────┘ -// highlight-next-line -VACUUM DROP TABLE DRY RUN SUMMARY; - -┌───────────────────────────────────┐ -│ table │ total_files │ total_size │ -├────────┼─────────────┼────────────┤ -│ b │ 9 │ 3213 │ -│ c │ 9 │ 3212 │ -└───────────────────────────────────┘ -``` - -### Adjusting Data Retention Time - -The VACUUM DROP TABLE command removes data files older than the `DATA_RETENTION_TIME_IN_DAYS` setting. This retention period can be adjusted as needed, for example, to 2 days: - -```sql -SET GLOBAL DATA_RETENTION_TIME_IN_DAYS = 2; -``` - -`DATA_RETENTION_TIME_IN_DAYS` defaults to 1 day (24 hours), and the maximum value varies across {{{ .lake }}} editions: - -| Edition | Default Retention | Max. Retention | -| ---------------------------------------- | ----------------- | ---------------- | -| {{{ .lake }}} Community & Enterprise Editions | 1 day (24 hours) | 90 days | -| {{{ .lake }}} (Personal) | 1 day (24 hours) | 1 day (24 hours) | -| {{{ .lake }}} (Business) | 1 day (24 hours) | 90 days | - -To check the current value of `DATA_RETENTION_TIME_IN_DAYS`: - -```sql -SHOW SETTINGS LIKE 'DATA_RETENTION_TIME_IN_DAYS'; -``` diff --git a/tidb-cloud-lake/sql/vacuum-drop-table.md b/tidb-cloud-lake/sql/vacuum-drop-table.md deleted file mode 100644 index 0db4b6b96b9ef..0000000000000 --- a/tidb-cloud-lake/sql/vacuum-drop-table.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: VACUUM DROP TABLE -summary: The VACUUM DROP TABLE command helps save storage space by permanently removing data files of dropped tables, freeing up storage space, and enabling you to manage the process efficiently. It offers optional parameters to target specific databases, preview, and limit the number of data files to be vacuumed. To list the dropped tables of a database, use SHOW DROP TABLES. ---- - -# VACUUM DROP TABLE - -The VACUUM DROP TABLE command helps save storage space by permanently removing data files of dropped tables, freeing up storage space, and enabling you to manage the process efficiently. It offers optional parameters to target specific databases, preview, and limit the number of data files to be vacuumed. To list the dropped tables of a database, use [SHOW DROP TABLES](/tidb-cloud-lake/sql/show-drop-tables.md). - -See also: [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md) - -## Syntax - -```sql -VACUUM DROP TABLE - [ FROM ] - [ DRY RUN [SUMMARY] ] - [ LIMIT ] -``` - -- `FROM `: This parameter restricts the search for dropped tables to a specific database. If not specified, the command will scan all databases, including those that have been dropped. - - ```sql title="Example:" - -- Remove dropped tables from the "default" database - // highlight-next-line - VACUUM DROP TABLE FROM default; - - -- Remove dropped tables from all databases - // highlight-next-line - VACUUM DROP TABLE; - ``` - -- `DRY RUN [SUMMARY]`: When this parameter is specified, data files will not be removed; instead, it returns a result showing which data files would have been removed if this parameter were not specified. See examples in the [Output](#output) section. - -- `LIMIT `: This parameter can be used with or without the DRY RUN parameter. When used with DRY RUN, it limits the number of data files to be shown in the `DRY RUN` result. When used without `DRY RUN`, it limits the number of data files to be vacuumed. - -### Output - -The VACUUM DROP TABLE command returns a result when the `DRY RUN` or `DRY RUN SUMMARY` parameter is specified: - -- `DRY RUN`: Returns a list of up to 1,000 candidate files and their sizes in bytes for each dropped table. -- `DRY RUN SUMMARY`: Returns the total number of files to be removed and their combined size for each dropped table. - -```sql title='Example:' -// highlight-next-line -VACUUM DROP TABLE DRY RUN; - -┌──────────────────────────────────────────────────────────────────┐ -│ table │ file │ file_size │ -├────────┼─────────────────────────────────────────────┼───────────┤ -│ b │ 313ebd4da5cc493f9a7d491da8253ce2_v2.parquet │ 210 │ -│ b │ 737f2215b8ac4a268d5b7f2218273358_v2.parquet │ 210 │ -│ b │ 737f2215b8ac4a268d5b7f2218273358_v4.parquet │ 340 │ -│ b │ 313ebd4da5cc493f9a7d491da8253ce2_v4.parquet │ 340 │ -│ b │ last_snapshot_location_hint │ 72 │ -│ b │ 7e01fa5c2e0a495298942671447dc8cb_v4.mpk │ 515 │ -│ b │ 2bc90e5be55c44258a736d27e5f7ac9e_v4.mpk │ 459 │ -│ b │ 85e73803aabc4eb48774db3d932312dd_v4.mpk │ 534 │ -│ b │ f0e507d0b825428dbfe57c8d8b620a15_v4.mpk │ 533 │ -│ c │ cee790e76f6e4e92bc9dab3b9e873dcf_v2.parquet │ 210 │ -│ c │ 4bcb2cef3b6344cb951908ebee5ceb36_v2.parquet │ 210 │ -│ c │ cee790e76f6e4e92bc9dab3b9e873dcf_v4.parquet │ 340 │ -│ c │ 4bcb2cef3b6344cb951908ebee5ceb36_v4.parquet │ 340 │ -│ c │ last_snapshot_location_hint │ 71 │ -│ c │ 414fc6a8dc6746afbc576cf8fddfcdf3_v4.mpk │ 516 │ -│ c │ 8d0d115c438244c295e3bfd50d556e39_v4.mpk │ 458 │ -│ c │ 28e4f551cc634d3d8d7e648c3baa5f5c_v4.mpk │ 534 │ -│ c │ 007b57e08eda419fbb451a3a3ed71de8_v4.mpk │ 533 │ -└──────────────────────────────────────────────────────────────────┘ -// highlight-next-line -VACUUM DROP TABLE DRY RUN SUMMARY; - -┌───────────────────────────────────┐ -│ table │ total_files │ total_size │ -├────────┼─────────────┼────────────┤ -│ b │ 9 │ 3213 │ -│ c │ 9 │ 3212 │ -└───────────────────────────────────┘ -``` - -### Adjusting Data Retention Time - -The VACUUM DROP TABLE command removes data files older than the `DATA_RETENTION_TIME_IN_DAYS` setting. This retention period can be adjusted as needed, for example, to 2 days: - -```sql -SET GLOBAL DATA_RETENTION_TIME_IN_DAYS = 2; -``` - -`DATA_RETENTION_TIME_IN_DAYS` defaults to 1 day (24 hours), and the maximum value varies across {{{ .lake }}} editions: - -| Edition | Default Retention | Max. Retention | -| ---------------------------------------- | ----------------- | ---------------- | -| {{{ .lake }}} Community & Enterprise Editions | 1 day (24 hours) | 90 days | -| {{{ .lake }}} (Personal) | 1 day (24 hours) | 1 day (24 hours) | -| {{{ .lake }}} (Business) | 1 day (24 hours) | 90 days | - -To check the current value of `DATA_RETENTION_TIME_IN_DAYS`: - -```sql -SHOW SETTINGS LIKE 'DATA_RETENTION_TIME_IN_DAYS'; -``` diff --git a/tidb-cloud-lake/sql/vacuum-dropped-objects.md b/tidb-cloud-lake/sql/vacuum-dropped-objects.md new file mode 100644 index 0000000000000..96bb89f0adfc5 --- /dev/null +++ b/tidb-cloud-lake/sql/vacuum-dropped-objects.md @@ -0,0 +1,48 @@ +--- +title: VACUUM DROPPED OBJECTS +summary: Permanently removes eligible dropped objects and reclaims their storage and metadata. +aliases: ['/tidbcloudlake/vacuum-drop-table/','/tidbcloudlake/vacuum-drop-table-sql/'] +--- + +# VACUUM DROPPED OBJECTS + +Permanently removes eligible dropped objects and reclaims their storage and metadata. This includes dropped tables and, when scanning all databases, dropped databases. Dropped view metadata is also cleaned up. Objects that have been vacuumed cannot be recovered with `UNDROP`. + +To list dropped tables in a database, use [SHOW DROP TABLES](/tidb-cloud-lake/sql/show-drop-tables.md). To clean history from an active table, use [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md). + +## Syntax + +```sql +VACUUM DROPPED OBJECTS [FROM ] +``` + +| Parameter | Description | +|-----------|-------------| +| `FROM ` | Restricts cleanup to the specified database in the current catalog. If omitted, scans all databases in the current catalog, including dropped databases. | + +With `FROM`, the command requires `SUPER` access to the database. Without `FROM`, it requires global `SUPER` privilege. + +The command does not return a result set. + +## Retention + +Dropped objects become eligible for cleanup after the retention period configured by `data_retention_time_in_days` (1 day by default). Removing storage and metadata is permanent. + +```sql +SET data_retention_time_in_days = 2; +SHOW SETTINGS LIKE 'data_retention_time_in_days'; +``` + +## Examples + +Clean up eligible dropped objects from a specific database: + +```sql +VACUUM DROPPED OBJECTS FROM default; +``` + +Clean up eligible dropped objects across all databases in the current catalog: + +```sql +VACUUM DROPPED OBJECTS; +``` diff --git a/tidb-cloud-lake/sql/vacuum-table-sql.md b/tidb-cloud-lake/sql/vacuum-table-sql.md deleted file mode 100644 index aa26b11fab6ba..0000000000000 --- a/tidb-cloud-lake/sql/vacuum-table-sql.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -title: VACUUM TABLE -summary: This page describes VACUUM TABLE in TiDB Cloud Lake. ---- - -# VACUUM TABLE - -The VACUUM TABLE command helps optimize system performance by freeing up storage space through the permanent removal of historical data files from a table. This includes: - -- Snapshots associated with the table, as well as their relevant segments and blocks. - -- Orphan files. Orphan files in {{{ .lake }}} refer to snapshots, segments, and blocks that are no longer associated with the table. Orphan files might be generated from various operations and errors, such as during data backups and restores, and can take up valuable disk space and degrade the system performance over time. - -See also: [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table-sql.md) - -## Syntax and Examples - -```sql -VACUUM TABLE [ DRY RUN [SUMMARY] ] -``` - -- `DRY RUN [SUMMARY]`: When this parameter is specified, candidate orphan files will not be removed. Instead, a list of up to 1,000 candidate files and their sizes (in bytes) will be returned, showing what would have been removed if the option was not used. When the optional parameter `SUMMARY` is included, the command returns the total number of files to be removed and their combined size in bytes. - -### Output - -The VACUUM TABLE command (without `DRY RUN`) returns a table summarizing vital statistics of the vacuumed files, containing the following columns: - -| Column | Description | -| -------------- | ----------------------------------------- | -| snapshot_files | Number of snapshot files | -| snapshot_size | Total size of snapshot files in bytes | -| segments_files | Number of segment files | -| segments_size | Total size of segment files in bytes | -| block_files | Number of block files | -| block_size | Total size of block files in bytes | -| index_files | Number of index files | -| index_size | Total size of index files in bytes | -| total_files | Total number of all types of files | -| total_size | Total size of all types of files in bytes | - -```sql title='Example:' -// highlight-next-line -VACUUM TABLE c; - -┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ -│ snapshot_files │ snapshot_size │ segments_files │ segments_size │ block_files │ block_size │ index_files │ index_size │ total_files │ total_size │ -├────────────────┼───────────────┼────────────────┼───────────────┼─────────────┼────────────┼─────────────┼────────────┼─────────────┼────────────┤ -│ 3 │ 1954 │ 9 │ 4802 │ 9 │ 1890 │ 9 │ 3060 │ 30 │ 11706 │ -└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ -``` - -When the `DRY RUN` parameter is specified with the VACUUM TABLE command, it returns a list of up to 1,000 candidate files and their sizes in bytes. If `DRY RUN SUMMARY` is specified, the command provides the total number of files to be removed and their combined size. - -```sql title='Example:' -// highlight-next-line -VACUUM TABLE c DRY RUN; - -┌──────────────────────────────────────────────────────────────┐ -│ file │ file_size │ -├──────────────────────────────────────────────────┼───────────┤ -│ 1/67/_ss/61aaf678b9af41568b539099b4b09908_v4.mpk │ 543 │ -│ 1/67/_ss/dd149d21151c459d8c87076f9412c12c_v4.mpk │ 516 │ -│ 1/67/_ss/7ba0b2e2f63c4d42897a48830027dcf3_v4.mpk │ 462 │ -│ 1/67/_ss/db55dac72b29452db976cf0af0f8d962_v4.mpk │ 588 │ -│ 1/67/_ss/d8055967298f478d97cddaa66cf67e11_v4.mpk │ 563 │ -│ 1/67/_ss/00c4288dac014760808006f821f1ecbe_v4.mpk │ 609 │ -└──────────────────────────────────────────────────────────────┘ -// highlight-next-line -VACUUM TABLE c DRY RUN SUMMARY; - -┌──────────────────────────┐ -│ total_files │ total_size │ -├─────────────┼────────────┤ -│ 6 │ 3281 │ -└──────────────────────────┘ -``` - -### Adjusting Data Retention Time - -The VACUUM TABLE command removes data files older than the `data_retention_time_in_days` setting. This retention period can be adjusted as needed, for example, to 2 days: - -```sql -SET GLOBAL data_retention_time_in_days = 2; -``` - -`data_retention_time_in_days` defaults to 1 day (24 hours), and the maximum value varies across {{{ .lake }}} editions: - -| Edition | Default Retention | Max. Retention | -| ---------------------------------------- | ----------------- | ---------------- | -| {{{ .lake }}} Community & Enterprise Editions | 1 day (24 hours) | 90 days | -| {{{ .lake }}} (Personal) | 1 day (24 hours) | 1 day (24 hours) | -| {{{ .lake }}} (Business) | 1 day (24 hours) | 90 days | - -To check the current value of `data_retention_time_in_days`: - -```sql -SHOW SETTINGS LIKE 'data_retention_time_in_days'; -``` - -### VACUUM TABLE vs. OPTIMIZE TABLE - -{{{ .lake }}} provides two commands for removing historical data files from a table: VACUUM TABLE and [OPTIMIZE TABLE](/tidb-cloud-lake/sql/optimize-table.md) (with the PURGE option). Although both commands are capable of permanently deleting data files, they differ in how they handle orphan files: OPTIMIZE TABLE is able to remove orphan snapshots, as well as the corresponding segments and blocks. However, there is a possibility of orphan segments and blocks existing without any associated snapshots. In such a scenario, only VACUUM TABLE can help clean them up. - -Both VACUUM TABLE and OPTIMIZE TABLE allow you to specify a period to determine which historical data files to remove. However, OPTIMIZE TABLE requires you to obtain the snapshot ID or timestamp from a query beforehand, whereas VACUUM TABLE allows you to specify the number of hours to retain the data files directly. VACUUM TABLE provides enhanced control over your historical data files both before their removal with the DRY RUN option, which allows you to preview the data files to be removed before applying the command. This provides a safe removal experience and helps you avoid unintended data loss. - -| | VACUUM TABLE | OPTIMIZE TABLE | -| ------------------------------------------------ | ------------ | -------------- | -| Associated snapshots (incl. segments and blocks) | Yes | Yes | -| Orphan snapshots (incl. segments and blocks) | Yes | Yes | -| Orphan segments and blocks only | Yes | No | -| DRY RUN | Yes | No | diff --git a/tidb-cloud-lake/sql/vacuum-table.md b/tidb-cloud-lake/sql/vacuum-table.md index f213d5e35d31c..da1a6fd1751df 100644 --- a/tidb-cloud-lake/sql/vacuum-table.md +++ b/tidb-cloud-lake/sql/vacuum-table.md @@ -1,111 +1,55 @@ --- title: VACUUM TABLE -summary: The VACUUM TABLE command helps optimize system performance by freeing up storage space through the permanent removal of historical data files from a table. This includes. +summary: Permanently removes eligible historical snapshots and their unneeded segments, blocks, and related files from a FUSE table to reclaim storage. +aliases: ['/tidbcloudlake/vacuum-table-sql/'] --- # VACUUM TABLE -The VACUUM TABLE command helps optimize system performance by freeing up storage space through the permanent removal of historical data files from a table. This includes: +The `VACUUM TABLE` command permanently removes eligible historical snapshots and their unneeded segments, blocks, and related files from a FUSE table to reclaim storage. The table and its current data remain available. -- Snapshots associated with the table, as well as their relevant segments and blocks. +See also: [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md), [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md), [VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md). -- Orphan files. Orphan files in {{{ .lake }}} refer to snapshots, segments, and blocks that are no longer associated with the table. Orphan files might be generated from various operations and errors, such as during data backups and restores, and can take up valuable disk space and degrade the system performance over time. - -See also: [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md) - -## Syntax and Examples +## Syntax ```sql -VACUUM TABLE [ DRY RUN [SUMMARY] ] +VACUUM TABLE [.] ``` -- `DRY RUN [SUMMARY]`: When this parameter is specified, candidate orphan files will not be removed. Instead, a list of up to 1,000 candidate files and their sizes (in bytes) will be returned, showing what would have been removed if the option was not used. When the optional parameter `SUMMARY` is included, the command returns the total number of files to be removed and their combined size in bytes. - -### Output - -The VACUUM TABLE command (without `DRY RUN`) returns a table summarizing vital statistics of the vacuumed files, containing the following columns: - -| Column | Description | -| -------------- | ----------------------------------------- | -| snapshot_files | Number of snapshot files | -| snapshot_size | Total size of snapshot files in bytes | -| segments_files | Number of segment files | -| segments_size | Total size of segment files in bytes | -| block_files | Number of block files | -| block_size | Total size of block files in bytes | -| index_files | Number of index files | -| index_size | Total size of index files in bytes | -| total_files | Total number of all types of files | -| total_size | Total size of all types of files in bytes | - -```sql title='Example:' -// highlight-next-line -VACUUM TABLE c; - -┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ -│ snapshot_files │ snapshot_size │ segments_files │ segments_size │ block_files │ block_size │ index_files │ index_size │ total_files │ total_size │ -├────────────────┼───────────────┼────────────────┼───────────────┼─────────────┼────────────┼─────────────┼────────────┼─────────────┼────────────┤ -│ 3 │ 1954 │ 9 │ 4802 │ 9 │ 1890 │ 9 │ 3060 │ 30 │ 11706 │ -└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ -``` +If the database is omitted, the current database is used. The command operates in the current catalog and requires `SUPER` access to the table. Only writable FUSE tables are supported. -When the `DRY RUN` parameter is specified with the VACUUM TABLE command, it returns a list of up to 1,000 candidate files and their sizes in bytes. If `DRY RUN SUMMARY` is specified, the command provides the total number of files to be removed and their combined size. - -```sql title='Example:' -// highlight-next-line -VACUUM TABLE c DRY RUN; - -┌──────────────────────────────────────────────────────────────┐ -│ file │ file_size │ -├──────────────────────────────────────────────────┼───────────┤ -│ 1/67/_ss/61aaf678b9af41568b539099b4b09908_v4.mpk │ 543 │ -│ 1/67/_ss/dd149d21151c459d8c87076f9412c12c_v4.mpk │ 516 │ -│ 1/67/_ss/7ba0b2e2f63c4d42897a48830027dcf3_v4.mpk │ 462 │ -│ 1/67/_ss/db55dac72b29452db976cf0af0f8d962_v4.mpk │ 588 │ -│ 1/67/_ss/d8055967298f478d97cddaa66cf67e11_v4.mpk │ 563 │ -│ 1/67/_ss/00c4288dac014760808006f821f1ecbe_v4.mpk │ 609 │ -└──────────────────────────────────────────────────────────────┘ -// highlight-next-line -VACUUM TABLE c DRY RUN SUMMARY; - -┌──────────────────────────┐ -│ total_files │ total_size │ -├─────────────┼────────────┤ -│ 6 │ 3281 │ -└──────────────────────────┘ -``` +The command does not return a result set. + +## Retention and Snapshot Tags + +Cleanup respects `data_retention_time_in_days` (1 day by default). Historical data that has been removed cannot be queried through Time Travel or restored with Flashback. -### Adjusting Data Retention Time +Snapshots referenced by unexpired [snapshot tags](/tidb-cloud-lake/sql/create-snapshot-tag.md), including tags without an expiration time, and the data they reference are protected from cleanup. Expired tags no longer protect their snapshots. VACUUM attempts to remove expired tags; a failure to delete an expired tag does not stop cleanup. -The VACUUM TABLE command removes data files older than the `data_retention_time_in_days` setting. This retention period can be adjusted as needed, for example, to 2 days: +Use a session setting to adjust the retention period for subsequent cleanup operations: ```sql -SET GLOBAL data_retention_time_in_days = 2; +SET data_retention_time_in_days = 2; +SHOW SETTINGS LIKE 'data_retention_time_in_days'; ``` -`data_retention_time_in_days` defaults to 1 day (24 hours), and the maximum value varies across {{{ .lake }}} editions: - -| Edition | Default Retention | Max. Retention | -| ---------------------------------------- | ----------------- | ---------------- | -| {{{ .lake }}} Community & Enterprise Editions | 1 day (24 hours) | 90 days | -| {{{ .lake }}} (Personal) | 1 day (24 hours) | 1 day (24 hours) | -| {{{ .lake }}} (Business) | 1 day (24 hours) | 90 days | +## Examples -To check the current value of `data_retention_time_in_days`: +Create a table and change its data, then clean up eligible history: ```sql -SHOW SETTINGS LIKE 'data_retention_time_in_days'; +CREATE OR REPLACE TABLE vacuum_example (id INT); +INSERT INTO vacuum_example VALUES (1), (2); +DELETE FROM vacuum_example WHERE id = 1; +VACUUM TABLE vacuum_example; +SELECT * FROM vacuum_example; ``` -### VACUUM TABLE vs. OPTIMIZE TABLE +The current row remains available. Recently created history is retained until it becomes eligible for cleanup. -{{{ .lake }}} provides two commands for removing historical data files from a table: VACUUM TABLE and [OPTIMIZE TABLE](/tidb-cloud-lake/sql/optimize-table.md) (with the PURGE option). Although both commands are capable of permanently deleting data files, they differ in how they handle orphan files: OPTIMIZE TABLE is able to remove orphan snapshots, as well as the corresponding segments and blocks. However, there is a possibility of orphan segments and blocks existing without any associated snapshots. In such a scenario, only VACUUM TABLE can help clean them up. +Compaction and storage reclamation are separate operations. To compact a table and then clean up eligible historical files: -Both VACUUM TABLE and OPTIMIZE TABLE allow you to specify a period to determine which historical data files to remove. However, OPTIMIZE TABLE requires you to obtain the snapshot ID or timestamp from a query beforehand, whereas VACUUM TABLE allows you to specify the number of hours to retain the data files directly. VACUUM TABLE provides enhanced control over your historical data files both before their removal with the DRY RUN option, which allows you to preview the data files to be removed before applying the command. This provides a safe removal experience and helps you avoid unintended data loss. - -| | VACUUM TABLE | OPTIMIZE TABLE | -| ------------------------------------------------ | ------------ | -------------- | -| Associated snapshots (incl. segments and blocks) | Yes | Yes | -| Orphan snapshots (incl. segments and blocks) | Yes | Yes | -| Orphan segments and blocks only | Yes | No | -| DRY RUN | Yes | No | +```sql +OPTIMIZE TABLE vacuum_example COMPACT; +VACUUM TABLE vacuum_example; +``` diff --git a/tidb-cloud-lake/sql/vacuum-tables.md b/tidb-cloud-lake/sql/vacuum-tables.md new file mode 100644 index 0000000000000..96d54aeb142df --- /dev/null +++ b/tidb-cloud-lake/sql/vacuum-tables.md @@ -0,0 +1,35 @@ +--- +title: VACUUM TABLES +summary: Reclaims storage by running VACUUM TABLE cleanup on writable FUSE tables in one database or across databases in the current catalog. +--- + +# VACUUM TABLES + +Reclaims storage by running [`VACUUM TABLE`](/tidb-cloud-lake/sql/vacuum-table.md) cleanup on writable FUSE tables in one database or across databases in the current catalog. It preserves current table data and follows the same retention and snapshot-tag protection rules as single-table cleanup. + +## Syntax + +```sql +VACUUM TABLES [FROM ] +``` + +| Scope | Behavior | Required privilege | +|-------|----------|--------------------| +| `FROM ` | Cleans eligible tables in the specified database. | `SUPER` access to that database | +| No `FROM` clause | Cleans eligible tables in all non-system databases in the current catalog, not just the current database. | Global `SUPER` | + +Non-FUSE tables and read-only tables are skipped. Ordinary per-table cleanup errors are logged and processing continues with other tables. Cancellation stops the operation; errors listing databases or tables can also stop it. Completion does not guarantee that every table was cleaned successfully. + +The command does not return a result set. It does not clean dropped objects or temporary files; use [`VACUUM ALL`](/tidb-cloud-lake/sql/vacuum-all.md) to include those operations. + +## Examples + +```sql +VACUUM TABLES FROM default; +``` + +To clean eligible tables across all non-system databases in the current catalog: + +```sql +VACUUM TABLES; +``` diff --git a/tidb-cloud-lake/sql/vacuum-temporary-files.md b/tidb-cloud-lake/sql/vacuum-temporary-files.md index cd2409ed4e05c..1151b19db8ece 100644 --- a/tidb-cloud-lake/sql/vacuum-temporary-files.md +++ b/tidb-cloud-lake/sql/vacuum-temporary-files.md @@ -5,87 +5,39 @@ summary: Removes temporary files created by {{{ .lake }}}, such as spill files. # VACUUM TEMPORARY FILES -Removes temporary files created by {{{ .lake }}}, such as spill files. +Removes temporary spill files created during query execution and cleans up inactive temporary-table sessions for the tenant. -See also: [system.temp_files](/tidb-cloud-lake/sql/system-temp-files.md) +See also: [system.temp_files](/tidb-cloud-lake/sql/system-temp-files.md), [VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md). ## Syntax ```sql -VACUUM TEMPORARY FILES [ LIMIT ] +VACUUM TEMPORARY FILES [RETAIN {SECONDS | DAYS}] [LIMIT ] ``` -| Parameter | Description | -|-----------|------------------------------------------------------| -| LIMIT | The maximum number of temporary files to be removed. | +| Parameter | Description | +|-----------|-------------| +| `RETAIN` | Retention period for temporary spill files. Defaults to 3 days. This is independent of `data_retention_time_in_days` and does not set the lifetime of temporary-table sessions. | +| `LIMIT` | Limits spill-file deletion first. The remaining allowance limits the number of inactive temporary-table sessions to clean, rather than individual files within those sessions. If omitted, no explicit limit is applied. | -## Output +Requires global `SUPER` privilege. The command does not return a result set. -Returns the number of deleted temporary files. +## Examples -## Example +Inspect temporary files: ```sql SELECT * FROM system.temp_files; +``` -┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ -│ file_type │ file_name │ file_content_length │ file_last_modified_time │ -├───────────┼─────────────────────────────────────────────────────────────┼─────────────────────┼─────────────────────────┤ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/0tUE8EqsxxjO4ftZA8Zni6 │ 591239232 │ 2024-11-19 03:06:03 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/440NKJwbRrW8HCFfQuNmb4 │ 607193920 │ 2024-11-19 03:05:18 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/5oQxtCB58oRhTA7EgO3027 │ 787631104 │ 2024-11-19 03:04:34 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/7nBLkWobl4jaDQtROAIow1 │ 596923264 │ 2024-11-19 03:06:04 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/Dx1xSJ5kv5vZyoWdmSpe32 │ 780189824 │ 2024-11-19 03:04:34 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/KSHXnVch2KUbHCqE0rgpx7 │ 741196608 │ 2024-11-19 03:05:21 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/O3TvRQja41NrpME8qXjJE3 │ 792868608 │ 2024-11-19 03:06:06 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/OMXTEzXmvR5Zw3jk2BVlR5 │ 661675392 │ 2024-11-19 03:05:19 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/PkSYoCjNxwDqCwP3k0axs1 │ 797124864 │ 2024-11-19 03:04:33 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/Sdr4ew2l60k90e7zZs3mF │ 797046144 │ 2024-11-19 03:05:21 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/TPMpuE1ypRSwRiSx2bRhh6 │ 531469504 │ 2024-11-19 03:06:03 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/VdzW61PcSugFIGyCR4B6P6 │ 736063616 │ 2024-11-19 03:04:34 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/WcTI2vVUfyzy8XUyjQAhc2 │ 791146496 │ 2024-11-19 03:04:34 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/Y6cCfDUkIkeD7Mnm0Zut67 │ 738694976 │ 2024-11-19 03:06:05 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/eNACGmJy00y8Pr1xSPCT25 │ 790728256 │ 2024-11-19 03:05:21 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/fojFryduQMoru0kAwnzys5 │ 795929344 │ 2024-11-19 03:05:20 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/iWcusSG1zW0pnbo76j0vr7 │ 797382080 │ 2024-11-19 03:06:04 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/klWl0CxOZQ08IHymUdeHr │ 796787712 │ 2024-11-19 03:04:34 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/kyX7EzdFBOVEBDNwKtexC6 │ 743725184 │ 2024-11-19 03:05:20 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/qDOtHrPpdpPPxqqb2Ybht7 │ 794764672 │ 2024-11-19 03:04:34 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/qp7GnofqSBZXnJrFuuxqa6 │ 797497664 │ 2024-11-19 03:06:02 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/w2qbixYObBOaMlgk7IQms1 │ 716091520 │ 2024-11-19 03:06:05 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/xO0ozrAKnq0naJ85BO53I4 │ 779609664 │ 2024-11-19 03:04:34 │ -│ Spill │ 1e9411e1-3c2f-48ee-9712-9d3ce396d1b3/xpejvCn9HNGJOc0szcC5b4 │ 793506112 │ 2024-11-19 03:05:21 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/2Gk62gm2GgFSLjTIGJbWv6 │ 796510336 │ 2024-11-19 03:01:55 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/2iq9gvwTVgpyp4CSQhimY3 │ 613255680 │ 2024-11-19 03:01:52 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/6gwj6vG0FDPLbBPIfcBDK │ 750276224 │ 2024-11-19 03:02:36 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/FXMxh0kA9W6QM5gmJizr92 │ 640907328 │ 2024-11-19 03:02:35 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/JQgHSgRphfQNtty8iZYGV5 │ 592456704 │ 2024-11-19 03:02:35 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/KZh2d7Av3UgfFu63dLKyh7 │ 639652608 │ 2024-11-19 03:01:53 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/L2NiIGBOUUsWHwqhqYVni5 │ 767536768 │ 2024-11-19 03:01:07 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/MG1fGMXMqlX9x1iQhUHQr4 │ 750610560 │ 2024-11-19 03:01:54 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/ObfLpkM6boMbeZHvckEGy3 │ 796252032 │ 2024-11-19 03:02:36 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/QuP4XULWwFtw1eWN9wYPf4 │ 664751936 │ 2024-11-19 03:02:35 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/Qy1i0PXzXMmJQ5DYWqfhN5 │ 789342016 │ 2024-11-19 03:01:08 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/UblG5Do43sJ4eRrr7Jh2O5 │ 796920768 │ 2024-11-19 03:01:07 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/UvmDHe3hIAO1uGubX0O8K3 │ 789434112 │ 2024-11-19 03:01:55 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/UxEvvrlyQdZws1Ou8Qhy62 │ 595680768 │ 2024-11-19 03:02:34 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/ZrbdJT9xOcsHjExj6wCum3 │ 796707456 │ 2024-11-19 03:01:53 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/e2sY1RKXFsHtNFx4PkFxT1 │ 753388160 │ 2024-11-19 03:01:54 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/fnQanOb1s6OmwJtooi35K5 │ 796730688 │ 2024-11-19 03:01:08 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/hHSrrfmMjtnPP4gbKBuTc2 │ 764452672 │ 2024-11-19 03:01:07 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/pZ06OYSniDzOLp8vTtruQ6 │ 796523712 │ 2024-11-19 03:02:37 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/rPdRFZz6g3NlhIlbnk5b16 │ 790081408 │ 2024-11-19 03:01:07 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/sxOeDcjyKRrahkCHtAbeG2 │ 605099776 │ 2024-11-19 03:01:52 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/v1PU6oEHSSEieXL7mceqz5 │ 682199616 │ 2024-11-19 03:02:36 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/xRRUU3sKkjyFAKv4LDuUt2 │ 786471232 │ 2024-11-19 03:01:08 │ -│ Spill │ 71c1bdcd-8ebb-45c1-98af-2700df2f9e10/ys9yCVJNSRBY73ce46bCA6 │ 747938176 │ 2024-11-19 03:01:07 │ -└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ +Clean up using the default retention: +```sql VACUUM TEMPORARY FILES; +``` -┌────────┐ -│ Files │ -├────────┤ -│ 48 │ -└────────┘ +Retain spill files for 2 days and limit cleanup: + +```sql +VACUUM TEMPORARY FILES RETAIN 2 DAYS LIMIT 1000; ```