diff --git a/getting-started/migrate-from-heroku.mdx b/getting-started/migrate-from-heroku.mdx index d33258e534..26972fbf07 100644 --- a/getting-started/migrate-from-heroku.mdx +++ b/getting-started/migrate-from-heroku.mdx @@ -37,9 +37,8 @@ fly launch --no-deploy fly mpg create --name your-db --region iad # ⚠️ Replace CLUSTER_ID below with the cluster ID from the output above. # If you missed it, run: fly mpg list -fly mpg proxy CLUSTER_ID --local-port 15432 & -pg_dump --no-owner --no-acl "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" | \ - PGPASSWORD=YOUR_FLY_DB_PASSWORD psql -h localhost -p 15432 -U fly-user -d fly-db +pg_dump --no-owner --no-acl -f heroku-dump.sql "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" && \ + fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 fly mpg attach CLUSTER_ID -a your-fly-app # 4. Import secrets @@ -151,23 +150,28 @@ Connection string: postgresql://fly-user:PASSWORD@pgbouncer.CLUSTER_ID.flympg.ne Note the cluster ID (the `CLUSTER_ID` part) - you'll need it for the next few commands. If you lose it, you can always find it with `fly mpg list`. -To import your data, use `fly mpg proxy` to create a local tunnel, then `pg_dump` and `psql`: +To import your data, dump it from Heroku with `pg_dump`, then restore it with `fly mpg connect`, which opens a tunnel to your database and runs `psql` for you: ```bash -# Terminal 1: Start a proxy to your Fly database (replace CLUSTER_ID with your cluster ID) -fly mpg proxy CLUSTER_ID --local-port 15432 - -# Terminal 2: Dump from Heroku and restore to Fly +# Dump the Heroku database (plain SQL format) export HEROKU_DB=$(heroku config:get DATABASE_URL -a $HEROKU_APP) +pg_dump --no-owner --no-acl -f heroku-dump.sql "$HEROKU_DB" + +# Restore to Managed Postgres (replace CLUSTER_ID with your cluster ID) +fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 +``` -# Dump the database (plain SQL format) -pg_dump --no-owner --no-acl "$HEROKU_DB" > heroku-dump.sql +Make sure `pg_dump` finishes without an error before you restore: a dump that failed partway can restore without complaint. The restore prints a few query results, such as `setval`, and carries on past errors. You'll see errors about the `pg_stat_statements` extension and about Heroku's own event triggers, like these: -# Restore to Fly (via the proxy) -PGPASSWORD=YOUR_PASSWORD psql -h localhost -p 15432 -U fly-user -d fly-db -f heroku-dump.sql +```out +psql:heroku-dump.sql:68: ERROR: permission denied to create extension "pg_stat_statements" +psql:heroku-dump.sql:75: ERROR: extension "pg_stat_statements" does not exist +psql:heroku-dump.sql:25952: ERROR: permission denied to create event trigger "00_validate_before_start" ``` -You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. Your actual data will import successfully. +These are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. + +Read any other error before you go on: it means a statement didn't run. If the output ends with `connection to server was lost` and `Error: exit status 2`, the connection dropped partway and the import isn't complete. Don't run the restore again on the same database, because tables that already loaded would get their rows twice. Create an empty database with `fly mpg databases create CLUSTER_ID --name heroku_import`, then add `--database heroku_import` before the `--` in the restore command, and to `fly mpg attach` below. See [Restore errors](/postgres/import#restore-errors). Attach the database to your app: diff --git a/getting-started/troubleshooting.mdx b/getting-started/troubleshooting.mdx index 6999ee08d5..c6b6e4d8fa 100644 --- a/getting-started/troubleshooting.mdx +++ b/getting-started/troubleshooting.mdx @@ -488,6 +488,7 @@ MPG clusters run on Fly's private network and aren't accessible over the public **To connect from your local machine::** - Interactive psql: `fly mpg connect` +- SQL without a session: `fly mpg connect -c 'select 1'` or `fly mpg connect -f script.sql` - Proxy to localhost: `fly mpg proxy` — forwards a local port to your database - WireGuard: connect to your org's private network, then use the `.flympg.net` connection string directly. Read more in this [reference guide](/blueprints/connect-private-network-wireguard). diff --git a/postgres/create-and-connect.mdx b/postgres/create-and-connect.mdx index 115e195afd..4dd19937e4 100644 --- a/postgres/create-and-connect.mdx +++ b/postgres/create-and-connect.mdx @@ -1,6 +1,6 @@ --- title: Create and Connect to a Managed Postgres Cluster -description: "Create a Managed Postgres cluster with fly mpg create or the dashboard, attach an app with fly mpg attach, and connect locally via mpg proxy or WireGuard." +description: "Create a Managed Postgres cluster, attach an app with fly mpg attach, and connect or run SQL locally with fly mpg connect, mpg proxy, or WireGuard." source_url: https://fly.io/docs/mpg/create-and-connect/ --- @@ -72,7 +72,7 @@ From the "Connect" tab of your MPG Cluster page you can attach a specific app in To attach an application to your database using flyctl run: ```bash -fly mpg attach -a +fly mpg attach -a ``` This will also trigger a restart of the selected app in order to add the secret. You can find the cluster ID in your MPG dashboard or by using the `fly mpg list` command. @@ -100,6 +100,57 @@ fly mpg connect [flags] This command will create a direct connection to your database and open a psql session. You'll need a compatible psql version installed locally. +### Running SQL without a session + +To run SQL from a script, a CI job, or an agent, pass it to [`fly mpg connect`](/flyctl/cmd/fly_mpg_connect) with `-c` or `-f`. The command runs the SQL, prints the result, and exits. Pass the cluster ID: with `-c` or `-f`, flyctl doesn't prompt for a cluster, user, or database. + +The `-c` and `-f` flags are available in flyctl v0.5.9 and later. If your `fly` does not have them, run `fly version upgrade` or [install flyctl](/flyctl/install). + +Run one statement: + +```bash +fly mpg connect -c 'select count(*) from orders' +``` + +Run a SQL file: + +```bash +fly mpg connect -f migrate.sql +``` + +Both flags stop at the first error and skip your psqlrc file. You can repeat either flag, but you can't combine `-c` and `-f` in one command. Use `--database` and `--username` to connect to something other than the defaults. + +Everything after `--` goes to psql unchanged. For example, to get a bare value into a shell variable, add psql's `-At` flags: + +```bash +count=$(fly mpg connect -c 'select count(*) from orders' -- -At) +``` + +psql writes query results to stdout. The `Proxying...` line that flyctl prints goes to stderr. + +To read SQL from stdin, pass `-f -`. It behaves like any other file: + +```bash +fly mpg connect -f - < queries.sql +``` + +SQL sent on stdin without `-c` or `-f` keeps psql's default behavior: it reads your psqlrc file and carries on after an error. + +`fly mpg connect` exits with status `0` when psql succeeds and `1` when psql fails, so a script can check it. On failure, psql's own error is followed by a line with psql's exit status: + +```out +$ fly mpg connect -f migrate.sql +Proxying localhost:16380 to remote [direct..flympg.net]:5432 +psql:migrate.sql:2: ERROR: column "nope" does not exist +LINE 1: select nope; + ^ +Error: exit status 3 +``` + +psql uses `1` for a failed `-c` statement or an error of its own, such as a file it can't read. It uses `2` for a connection that couldn't be made or was lost, and `3` for a script that stopped at an error. flyctl also exits with status `1` for its own errors, such as a cluster it can't find. + +`fly mpg connect` always listens on local port 16380, which is also the default port for `fly mpg proxy`. Run one `fly mpg connect` at a time on your local machine, and give a proxy you run alongside it a different `--local-port`. + ### Setting up a Proxy Connection You can create a proxy connection to your Managed Postgres database using the '[mpg proxy](/flyctl/cmd/fly_mpg_proxy)' command: diff --git a/postgres/import.mdx b/postgres/import.mdx index 537faa5d5d..f2b50ab65a 100644 --- a/postgres/import.mdx +++ b/postgres/import.mdx @@ -1,6 +1,6 @@ --- title: Import data from another postgres cluster -description: "Import data into Fly.io Managed Postgres from Fly Postgres or another provider using pg_dump and psql over fly mpg proxy, plus size limits and pgloader." +description: "Import data into Fly.io Managed Postgres from Fly Postgres or another provider using pg_dump and fly mpg connect, plus size limits and pgloader." source_url: https://fly.io/docs/mpg/import/ --- @@ -12,13 +12,7 @@ If you're migrating to Fly Managed Postgres (MPG) from an unmanaged Fly Postgres There isn't currently an automated way to migrate an existing DB's data, but you can accomplish this with a standard Postgres dump and restore process after creating your MPG cluster. -For simple databases, this can be as easy as running : - -```bash - pg_dump "$LEGACY_PG_CONNECTION_STRING" | psql "$MPG_URI". -``` - -Because your MPG cluster runs within your Fly Private Network, you'll need to be connected to your organizations Wireguard network in order to restore into your MPG database. To run the import process from your local machine, you would take the following steps: +Your MPG cluster runs within your Fly.io private network, so it isn't reachable over the public internet. `fly mpg connect` opens the tunnel, looks up the credentials, and runs psql for you, so you can restore from your local machine without setting up a proxy or copying a password. You'll need `pg_dump` and psql installed locally, and flyctl v0.5.9 or later. 1. Create your Managed Postgres cluster from the Fly.io dashboard, or using `fly mpg create` @@ -26,25 +20,57 @@ Because your MPG cluster runs within your Fly Private Network, you'll need to be **Warning:** Your new cluster must be created with a volume at least as large as the database you're importing, otherwise the import process will fail. -2. Open a proxy connection to your MPG database with - ```out - $ fly mpg proxy - ? Select Organization: My Organization (personal) - ? Select a Postgres cluster my-test-cluster (iad) - Proxying localhost:16380 to remote [fdaa:1:2345:0:0::67]:5432 +2. Dump your existing database to a file: + ```bash + pg_dump --no-owner --no-acl -f dump.sql "$LEGACY_PG_CONNECTION_STRING" + ``` + + Make sure `pg_dump` finishes without an error before you go on. A dump that failed partway leaves an incomplete file, and an incomplete file can restore without complaint. + +3. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16 (the `fly mpg create` default), remove one setting from the top of the dump. `pg_dump` 17 and later writes `SET transaction_timeout = 0;` there, and Postgres 16 doesn't know that setting. Check your version with `pg_dump --version`, then remove the line: + ```bash + sed '1,20{/^SET transaction_timeout = 0;$/d;}' dump.sql > dump-pg16.sql ``` -3. Find your MPG cluster's connection string in the dashboard, and modify it to use the local proxy: - ```out - Connection String: postgres://fly-user:@localhost:16380/fly-db - ``` + The `1,20` keeps the edit inside the dump's header, so it can't touch your data. -4. Run the dump and restore command +4. Find the cluster ID with `fly mpg list`, then restore the file into your MPG cluster. If you ran step 3, restore `dump-pg16.sql` instead of `dump.sql`: ```bash - pg_dump "$LEGACY_PG_CONNECTION_STRING" | psql postgres://fly-user:@localhost:16380/fly-db + fly mpg connect -f dump.sql -- -q --single-transaction ``` -5. If everything succeeds, all data from your legacy cluster will be imported into your new MPG cluster. + `-q` keeps psql quiet, so errors stand out. `--single-transaction` wraps the restore in one transaction: if an error stops it, nothing is left behind and you can run it again. The exception is a dump that contains large objects: it carries its own transactions, so a restore that fails after them leaves what it already restored. The restore goes into the default database as the default user. Add `--database` or `--username` before the `--` to pick others. + +5. Check the result. A restore that worked prints no errors, only a few query results such as `set_config` and `setval`, and exits with status `0`. One that failed prints psql's error and a line like `Error: exit status 3`, and exits with status `1`. + +### Restore errors + +- **`unrecognized configuration parameter "transaction_timeout"`:** The dump came from `pg_dump` 17 or later and the cluster runs Postgres 16. Go back to step 3, then restore `dump-pg16.sql`. +- **`must be owner of extension`:** The dump has a `COMMENT ON EXTENSION` statement for an extension that comes preinstalled on MPG, such as `pgaudit` or `pg_stat_monitor`, and the default user doesn't own those. You'll see this when the source is another MPG cluster. If you reach that cluster through `fly mpg proxy`, give the proxy a different `--local-port`, because `fly mpg connect` always uses 16380. Dump again with `pg_dump` 17 or later and leave those extensions out, then repeat steps 3 and 4: + ```bash + pg_dump --no-owner --no-acl --exclude-extension=pgaudit --exclude-extension=pg_stat_monitor -f dump.sql "$LEGACY_PG_CONNECTION_STRING" + ``` +- **`server closed the connection unexpectedly` or `connection to server was lost`:** The connection to your cluster dropped. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, a restore that was still running is rolled back, unless the dump contains large objects. Check whether your tables are there before you run it again, in case the connection dropped just as the restore finished. +- **`already exists`:** The database isn't empty. An earlier restore finished, ran without `--single-transaction`, or was piped from a `pg_dump` that failed partway. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` before the `--` in the restore command. Pass the same flag to `fly mpg attach` when you attach your app. +- **`out of shared memory`, with a hint about `max_locks_per_transaction`:** The dump has more tables than one transaction can lock, so the restore was rolled back. Restore without `--single-transaction`, as in `fly mpg connect -f dump.sql -- -q`. psql still stops at the first error, but what it restored before the error stays. If that happens, restore into a new, empty database before you try again. +- **Another error you'd rather skip:** Replace `--single-transaction` with `-v ON_ERROR_STOP=0`, as in `fly mpg connect -f dump.sql -- -q -v ON_ERROR_STOP=0`. psql then carries on past errors, and the exit status is `0` even when statements fail, so read every error it prints. Only do this into an empty database: over a partial import, tables that already loaded get their rows twice. + +### Piping a small database + +For a small database you can skip the file and pipe `pg_dump` straight in. Pass `-f -` to read the dump from stdin. Set `pipefail` as well: without it, a `pg_dump` that fails looks like a restore that worked. + +```bash +set -o pipefail +pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect -f - -- -q --single-transaction +``` + +If step 3 applies to you, add its `sed` expression to the pipe, without the file names: `pg_dump ... | sed '1,20{/^SET transaction_timeout = 0;$/d;}' | fly mpg connect ...`. + +If `pg_dump` fails partway, psql can commit what it received, which leaves part of your data imported even though the command fails. Restore into a new, empty database before you try again, as described under `already exists`. Prefer the file for anything large. + +### Using another tool + +Tools that make their own connection, like `pg_restore` with a custom-format dump, need an address to connect to. Run `fly mpg proxy ` and point the tool at the local port it prints, using the username and password from your cluster's connection string in the dashboard. See [Setting up a Proxy Connection](/postgres/create-and-connect#setting-up-a-proxy-connection). For larger or more complex databases, you may need to break the dump and restore process into multiple steps. You may also consider using dedicated PG migration tools like [pgloader](https://pgloader.io/) or [pgcopydb](https://github.com/dimitri/pgcopydb). diff --git a/postgres/index.mdx b/postgres/index.mdx index 8ca1d70ee0..c1f2e68df6 100644 --- a/postgres/index.mdx +++ b/postgres/index.mdx @@ -31,11 +31,11 @@ You'll be able to access: ### Current limitations {#whats-not-there-yet} -- **Major version upgrades:** You choose Postgres 16 or 17 when you create a cluster, and you can't upgrade a cluster in place. To move to a newer major version, create a new cluster and copy your data with `pg_dump` and `pg_restore`. See [Import data](/postgres/import). +- **Major version upgrades:** You choose Postgres 16 or 17 when you create a cluster, and you can't upgrade a cluster in place. To move to a newer major version, create a new cluster and copy your data with `pg_dump` and psql. See [Import data](/postgres/import). - **Minor versions and security patches:** New Postgres images include the latest minor release. We roll out new images during your cluster's [maintenance window](/postgres/maintenance-windows). v1 clusters, which have no **v2** badge next to the cluster name in the dashboard, don't get new images. PgBouncer, PostGIS and pg_repack versions are pinned and change only when we update them. - **Third-party extensions:** `pgvector` and `PostGIS` are supported, and `pg_repack` on v2 clusters, along with the bundled Postgres extensions marked as available on the [Extensions page](/postgres/extensions). To request another extension, [contact support](/about/support). - **Alerting:** We email your organization when maintenance starts and finishes, but there are no alerts for metrics like CPU or storage. Use [metrics](/postgres/metrics) to watch your cluster. -- **Migration tools:** There's no automated migration from another database. Use `pg_dump` and `pg_restore`, as described in [Import data](/postgres/import). +- **Migration tools:** There's no automated migration from another database. Use `pg_dump` and psql, as described in [Import data](/postgres/import). ## Regions