diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index fd08cbb65..5d2ed424d 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -17,6 +17,7 @@ This page tracks significant updates to the QuestDB documentation. ### New - [Migrate QuestDB onto the Kubernetes Operator](/docs/enterprise-kubernetes-operator/getting-started/migrate/) - Move an existing Enterprise deployment onto the Operator with a replica-first cutover: restore the source backup, consume replication WAL, then promote after a controlled source drain +- [Copy a schema to another instance](/docs/cookbook/operations/copy-schema-between-instances/) - Recreate one instance's tables, views, and materialized views on another from `SHOW CREATE DATABASE`, either by replaying the statements over the REST API or by dumping them to a `.sql` file, with `INCLUDE (SCHEMA)` and `INCLUDE (ACL)` for separating structure from permissions on Enterprise ### Updated diff --git a/documentation/cookbook/operations/copy-data-between-instances.md b/documentation/cookbook/operations/copy-data-between-instances.md index 54244449c..a75ce5fc0 100644 --- a/documentation/cookbook/operations/copy-data-between-instances.md +++ b/documentation/cookbook/operations/copy-data-between-instances.md @@ -8,7 +8,7 @@ Copy a subset of data from one QuestDB instance to another for testing or develo ## Problem -You want to copy data between QuestDB instances. This method allows you to copy any arbitrary query result, but if you want a full database copy please check the [backup and restore documentation](/docs/operations/backup/). +You want to copy data between QuestDB instances. This method allows you to copy any arbitrary query result, but if you want a full database copy please check the [backup and restore documentation](/docs/operations/backup/). To copy only the structure, see [Copy a schema to another instance](/docs/cookbook/operations/copy-schema-between-instances/). ## Solution: Table2Ilp utility diff --git a/documentation/cookbook/operations/copy-schema-between-instances.md b/documentation/cookbook/operations/copy-schema-between-instances.md new file mode 100644 index 000000000..975a374db --- /dev/null +++ b/documentation/cookbook/operations/copy-schema-between-instances.md @@ -0,0 +1,141 @@ +--- +title: Copy a schema to another instance +sidebar_label: Copy schema between instances +description: + Use SHOW CREATE DATABASE and the REST API to dump a QuestDB schema to a .sql + file or recreate it directly on another instance. +--- + +Recreate the structure of one QuestDB instance on another: tables, views, +materialized views, and, on QuestDB Enterprise, the access control layer. + +## Problem + +You want to copy the schema from one QuestDB instance to another. + +## Solution + +[`SHOW CREATE DATABASE`](/docs/query/sql/show/#show-create-database) returns a +data-free dump of the whole database, with one round-trippable DDL statement per +row: + +```questdb-sql title="Dump the database schema" demo +SHOW CREATE DATABASE; +``` + +Statements come back in dependency order, so replaying them top to bottom on an +empty instance recreates the database. + +### Get the dump as JSON + +Running the statement through the +[REST API](/docs/connect/compatibility/rest-api/#execute) gives you the same +result set as JSON, with the statements under `.dataset[][0]`: + +```shell +curl -s -G "https://demo.questdb.io/api/v1/sql/execute" \ + --data-urlencode "query=SHOW CREATE DATABASE" +``` + +From there you can either save the statements to a file or apply them straight +to the target instance. + +### Write the statements to a .sql file + +```shell +curl -s -G "https://demo.questdb.io/api/v1/sql/execute" \ + --data-urlencode "query=SHOW CREATE DATABASE" \ +| jq -r '.dataset[][0] | sub(";?$";";") + "\n"' > schema.sql +``` + +This keeps QuestDB's original multi-line formatting. `sub(";?$";";")` anchors to +the end of the whole string, not to each line, so it guarantees exactly one +trailing semicolon per statement without touching the intermediate lines. +Appending `"\n"` separates consecutive statements with a blank line. + +### Apply directly to the target instance + +Send each statement to the target's `/execute` endpoint as you read it: + +```shell +curl -s -G "https://demo.questdb.io/api/v1/sql/execute" \ + --data-urlencode "query=SHOW CREATE DATABASE" \ +| jq --raw-output0 '.dataset[][0]' \ +| xargs -0 -n1 sh -c \ + 'curl -sG "http://localhost:9000/api/v1/sql/execute" \ + --data-urlencode "query=$1"; echo' _ +``` + +Statements are sent one at a time on purpose. `/execute` is a `GET` endpoint, so +a whole schema sent as a single request would very likely exceed the maximum +request size the target accepts. + +`--raw-output0` emits NUL-separated values rather than newline-separated ones, +which matters because each statement is itself multi-line. `xargs -0 -n1` then +hands one whole statement to each `sh -c` invocation, and the trailing `_` fills +in `$0` so the statement lands in `$1`. + +Each response is printed on its own line. A statement that succeeds returns +`{"ddl":"OK"}`; one that fails returns HTTP 400 and an `error` field, for +example `{"query":"...","error":"table already exists","position":13}`. + +:::note + +`--raw-output0` requires jq 1.7 or later. + +::: + +If either instance requires authentication, pass the credentials on the +corresponding `curl` call. QuestDB open source supports +[HTTP basic authentication](/docs/connect/compatibility/rest-api/#http-basic-authentication) +only, so use `-u user:password`. On +[QuestDB Enterprise](/enterprise/), prefer a +[REST API token](/docs/connect/compatibility/rest-api/#authentication-via-token-in-questdb-enterprise) +over basic authentication, since it can be scoped, given an expiry, and revoked +on its own. Read it from an environment variable rather than pasting it into the +command, so it stays out of your shell history: + +```shell +export QDB_TOKEN="your-rest-api-token" + +curl -s -G "https://source:9000/api/v1/sql/execute" \ + -H "Authorization: Bearer $QDB_TOKEN" \ + --data-urlencode "query=SHOW CREATE DATABASE" +``` + +## Enterprise: schema without permissions + +In [QuestDB Enterprise](/enterprise/), the default dump also carries the access +control layer: users, groups, service accounts, memberships, and grants. That is +usually not what you want when seeding a development instance from production. + +Select what to copy with `INCLUDE`: + +```shell +# Structure only, no users or permissions +curl -s -G "https://source:9000/api/v1/sql/execute" \ + -H "Authorization: Bearer $QDB_TOKEN" \ + --data-urlencode "query=SHOW CREATE DATABASE INCLUDE (SCHEMA)" + +# Permissions only, no tables or views +curl -s -G "https://source:9000/api/v1/sql/execute" \ + -H "Authorization: Bearer $QDB_TOKEN" \ + --data-urlencode "query=SHOW CREATE DATABASE INCLUDE (ACL)" +``` + +The ACL categories each require the matching `LIST` or `USER DETAILS` +permission, while the schema categories need none, so a user with only `SELECT` +can still dump the structure. + +Passwords and tokens are never dumped, so `CREATE USER` and +`CREATE SERVICE ACCOUNT` statements replay without credentials. Set those on the +target after the replay. + +:::info Related documentation + +- [`SHOW CREATE DATABASE`](/docs/query/sql/show/#show-create-database) +- [REST API `/execute`](/docs/connect/compatibility/rest-api/#execute) +- [Copy data between instances](/docs/cookbook/operations/copy-data-between-instances/) +- [Backup and restore](/docs/operations/backup/) + +::: diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index ae64c09cd..533ba5a33 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -129,6 +129,10 @@ Each `ddl` value is stored with formatting characters, so pasting a row into a text editor expands it to the indented form shown by [`SHOW CREATE TABLE`](#show-create-table). +For turning the result set into a `.sql` file or replaying it on another server, +see +[Copy a schema to another instance](/docs/cookbook/operations/copy-schema-between-instances/). + #### Output order Objects are emitted in dependency order: a materialized view or view is never diff --git a/documentation/sidebars.js b/documentation/sidebars.js index f0b4f587f..2c4c83c2a 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -1175,6 +1175,7 @@ module.exports = { "cookbook/operations/csv-import-milliseconds", "cookbook/operations/tls-pgbouncer", "cookbook/operations/copy-data-between-instances", + "cookbook/operations/copy-schema-between-instances", "cookbook/operations/query-times-histogram", "cookbook/operations/optimize-many-tables", "cookbook/operations/check-transaction-applied",