From 8701f697919a51c72f8e72c4efd7e4ac2ee01530 Mon Sep 17 00:00:00 2001 From: vincent Date: Thu, 12 Nov 2026 12:44:58 -0500 Subject: [PATCH 1/8] Document scripted SQL in fly mpg connect and use it for imports Covers -c, -f, the -- pass-through and exit status from flyctl#5436, and replaces the fly mpg proxy + psql import recipes with fly mpg connect. Co-Authored-By: Claude Fable 5.2 --- getting-started/migrate-from-heroku.mdx | 18 +++----- getting-started/troubleshooting.mdx | 1 + postgres/create-and-connect.mdx | 44 +++++++++++++++++++ postgres/import.mdx | 57 ++++++++++++++++--------- 4 files changed, 87 insertions(+), 33 deletions(-) diff --git a/getting-started/migrate-from-heroku.mdx b/getting-started/migrate-from-heroku.mdx index d33258e534..509e7edce5 100644 --- a/getting-started/migrate-from-heroku.mdx +++ b/getting-started/migrate-from-heroku.mdx @@ -37,9 +37,8 @@ fly launch --no-deploy fly mpg create --name your-db --region iad # ⚠️ Replace CLUSTER_ID below with the cluster ID from the output above. # If you missed it, run: fly mpg list -fly mpg proxy CLUSTER_ID --local-port 15432 & pg_dump --no-owner --no-acl "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" | \ - PGPASSWORD=YOUR_FLY_DB_PASSWORD psql -h localhost -p 15432 -U fly-user -d fly-db + fly mpg connect CLUSTER_ID fly mpg attach CLUSTER_ID -a your-fly-app # 4. Import secrets @@ -151,23 +150,18 @@ Connection string: postgresql://fly-user:PASSWORD@pgbouncer.CLUSTER_ID.flympg.ne Note the cluster ID (the `CLUSTER_ID` part) - you'll need it for the next few commands. If you lose it, you can always find it with `fly mpg list`. -To import your data, use `fly mpg proxy` to create a local tunnel, then `pg_dump` and `psql`: +To import your data, dump it from Heroku with `pg_dump`, then restore it with `fly mpg connect`, which opens a tunnel to your database and runs `psql` for you: ```bash -# Terminal 1: Start a proxy to your Fly database (replace CLUSTER_ID with your cluster ID) -fly mpg proxy CLUSTER_ID --local-port 15432 - -# Terminal 2: Dump from Heroku and restore to Fly +# Dump the Heroku database (plain SQL format) export HEROKU_DB=$(heroku config:get DATABASE_URL -a $HEROKU_APP) - -# Dump the database (plain SQL format) pg_dump --no-owner --no-acl "$HEROKU_DB" > heroku-dump.sql -# Restore to Fly (via the proxy) -PGPASSWORD=YOUR_PASSWORD psql -h localhost -p 15432 -U fly-user -d fly-db -f heroku-dump.sql +# Restore to Fly (replace CLUSTER_ID with your cluster ID) +fly mpg connect CLUSTER_ID < heroku-dump.sql ``` -You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. Your actual data will import successfully. +You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. Your actual data will import successfully. If your `pg_dump` is version 17 or later, you'll also see `unrecognized configuration parameter "transaction_timeout"` on a Postgres 16 cluster, which is safe to ignore too. Attach the database to your app: diff --git a/getting-started/troubleshooting.mdx b/getting-started/troubleshooting.mdx index 6999ee08d5..c16969703b 100644 --- a/getting-started/troubleshooting.mdx +++ b/getting-started/troubleshooting.mdx @@ -488,6 +488,7 @@ MPG clusters run on Fly's private network and aren't accessible over the public **To connect from your local machine::** - Interactive psql: `fly mpg connect` +- One statement or a SQL file, no session: `fly mpg connect -c 'select 1'` or `fly mpg connect -f script.sql` - Proxy to localhost: `fly mpg proxy` — forwards a local port to your database - WireGuard: connect to your org's private network, then use the `.flympg.net` connection string directly. Read more in this [reference guide](/blueprints/connect-private-network-wireguard). diff --git a/postgres/create-and-connect.mdx b/postgres/create-and-connect.mdx index 115e195afd..bae6e8f653 100644 --- a/postgres/create-and-connect.mdx +++ b/postgres/create-and-connect.mdx @@ -100,6 +100,50 @@ fly mpg connect [flags] This command will create a direct connection to your database and open a psql session. You'll need a compatible psql version installed locally. +### Run SQL without a session + +To run SQL from a script, a CI job, or an agent, pass it to `fly mpg connect` with `-c` or `-f`. The command runs the SQL, prints the result, and exits. It doesn't prompt for a user or database, so pass the cluster ID. + +Run one statement: + +```bash +fly mpg connect -c 'select count(*) from orders' +``` + +Run a SQL file: + +```bash +fly mpg connect -f migrate.sql +``` + +Both flags stop at the first error and skip your `~/.psqlrc`. You can repeat either flag, but you can't combine `-c` and `-f` in one command. Use `--database` and `--username` to connect to something other than the defaults. + +Everything after `--` goes to psql unchanged. For example, to get a bare value into a shell variable, add psql's `-At` flags: + +```bash +count=$(fly mpg connect -c 'select count(*) from orders' -- -At) +``` + +Only psql's output goes to stdout. The `Proxying...` line that flyctl prints goes to stderr. + +You can also pipe SQL in. Piped SQL keeps psql's default behavior and carries on after an error, so add `ON_ERROR_STOP` if you want it to stop: + +```bash +cat queries.sql | fly mpg connect -- -v ON_ERROR_STOP=1 +``` + +`fly mpg connect` exits with status `0` when the SQL runs cleanly and `1` when psql fails, so a script can check it. On failure, psql's own error is followed by a line with psql's exit status: + +```out +$ fly mpg connect -f migrate.sql +psql:migrate.sql:2: ERROR: column "nope" does not exist +Error: exit status 3 +``` + +psql uses `1` for a failed `-c` statement, `2` for a connection that couldn't be made or was lost, and `3` for a script that stopped at an error. + +`fly mpg connect` uses a fixed local port, so run one at a time on a given machine. + ### Setting up a Proxy Connection You can create a proxy connection to your Managed Postgres database using the '[mpg proxy](/flyctl/cmd/fly_mpg_proxy)' command: diff --git a/postgres/import.mdx b/postgres/import.mdx index 537faa5d5d..017d7e23fd 100644 --- a/postgres/import.mdx +++ b/postgres/import.mdx @@ -1,6 +1,6 @@ --- title: Import data from another postgres cluster -description: "Import data into Fly.io Managed Postgres from Fly Postgres or another provider using pg_dump and psql over fly mpg proxy, plus size limits and pgloader." +description: "Import data into Fly.io Managed Postgres from Fly Postgres or another provider using pg_dump and fly mpg connect, plus size limits and pgloader." source_url: https://fly.io/docs/mpg/import/ --- @@ -12,13 +12,7 @@ If you're migrating to Fly Managed Postgres (MPG) from an unmanaged Fly Postgres There isn't currently an automated way to migrate an existing DB's data, but you can accomplish this with a standard Postgres dump and restore process after creating your MPG cluster. -For simple databases, this can be as easy as running : - -```bash - pg_dump "$LEGACY_PG_CONNECTION_STRING" | psql "$MPG_URI". -``` - -Because your MPG cluster runs within your Fly Private Network, you'll need to be connected to your organizations Wireguard network in order to restore into your MPG database. To run the import process from your local machine, you would take the following steps: +Your MPG cluster runs within your Fly.io private network, so it isn't reachable over the public internet. `fly mpg connect` opens the tunnel, looks up the credentials, and runs psql for you, so you can restore from your local machine without setting up a proxy or copying a password. You'll need `pg_dump` and `psql` installed locally. 1. Create your Managed Postgres cluster from the Fly.io dashboard, or using `fly mpg create` @@ -26,25 +20,46 @@ Because your MPG cluster runs within your Fly Private Network, you'll need to be **Warning:** Your new cluster must be created with a volume at least as large as the database you're importing, otherwise the import process will fail. -2. Open a proxy connection to your MPG database with - ```out - $ fly mpg proxy - ? Select Organization: My Organization (personal) - ? Select a Postgres cluster my-test-cluster (iad) - Proxying localhost:16380 to remote [fdaa:1:2345:0:0::67]:5432 +2. Dump your existing database to a file + ```bash + pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" > dump.sql ``` -3. Find your MPG cluster's connection string in the dashboard, and modify it to use the local proxy: - ```out - Connection String: postgres://fly-user:@localhost:16380/fly-db - ``` +3. Restore the file into your MPG cluster. Find the cluster ID with `fly mpg list` + ```bash + fly mpg connect -f dump.sql + ``` + + The restore goes into the default database as the default user. Add `--database` or `--username` to pick others. + +4. Check the exit status. `0` means every statement ran. With `-f`, the restore stops at the first error and `fly mpg connect` exits with status `1`, so a failed import can't pass for a finished one. -4. Run the dump and restore command +To skip the intermediate file, pipe `pg_dump` straight in. Piped SQL doesn't stop on errors by default, so turn that on: + +```bash +pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect -- -v ON_ERROR_STOP=1 +``` + +### If the restore stops + +- **`unrecognized configuration parameter "transaction_timeout"`**: `pg_dump` version 17 and later writes `SET transaction_timeout = 0;` at the top of every dump, and Postgres 16 doesn't know that setting. +- **`must be owner of extension`**: the dump includes a `COMMENT ON EXTENSION` line for each extension, and the default user doesn't own the extensions that come preinstalled on MPG. + + Neither statement affects your data. Remove those lines and run the restore again: + ```bash + grep -vE '^(SET transaction_timeout|COMMENT ON EXTENSION)' dump.sql > dump-clean.sql + fly mpg connect -f dump-clean.sql + ``` + In the piped form, put the same `grep -vE` between `pg_dump` and `fly mpg connect`. +- **The connection drops partway**: the restore needs your network connection for the whole run, and a large dump can take a while. Statements that already ran stay in the database, so drop the partial tables before you retry. To make the restore all-or-nothing instead, add `-- --single-transaction`: ```bash - pg_dump "$LEGACY_PG_CONNECTION_STRING" | psql postgres://fly-user:@localhost:16380/fly-db + fly mpg connect -f dump.sql -- --single-transaction ``` +- **Another error you expect and want to skip**: tell psql to carry on with `fly mpg connect -f dump.sql -- -v ON_ERROR_STOP=0`, then read the output for anything else that failed. In this mode the exit status is `0` even when statements fail. + +### Use another tool -5. If everything succeeds, all data from your legacy cluster will be imported into your new MPG cluster. +Tools that make their own connection, like `pg_restore` with a custom-format dump, need an address to connect to. Run `fly mpg proxy ` and point the tool at the local port it prints, using the username and password from your cluster's connection string in the dashboard. See [Setting up a Proxy Connection](/postgres/create-and-connect#setting-up-a-proxy-connection). For larger or more complex databases, you may need to break the dump and restore process into multiple steps. You may also consider using dedicated PG migration tools like [pgloader](https://pgloader.io/) or [pgcopydb](https://github.com/dimitri/pgcopydb). From 201dee6e5a5c2816db9fbe15b645c74a1c7a871d Mon Sep 17 00:00:00 2001 From: vincent Date: Thu, 12 Nov 2026 15:35:14 -0500 Subject: [PATCH 2/8] Address review: safe header edit, single-transaction restore, v0.5.9 Replace the whole-dump grep filter with a header-only edit, make the file restore single-transaction so a stopped run can be retried, limit the piped form to small databases, and match page conventions. Co-Authored-By: Claude Fable 5.2 --- getting-started/migrate-from-heroku.mdx | 14 ++++--- getting-started/troubleshooting.mdx | 2 +- postgres/create-and-connect.mdx | 21 ++++++---- postgres/import.mdx | 52 ++++++++++++++----------- postgres/index.mdx | 4 +- 5 files changed, 53 insertions(+), 40 deletions(-) diff --git a/getting-started/migrate-from-heroku.mdx b/getting-started/migrate-from-heroku.mdx index 509e7edce5..df2f893230 100644 --- a/getting-started/migrate-from-heroku.mdx +++ b/getting-started/migrate-from-heroku.mdx @@ -37,8 +37,8 @@ fly launch --no-deploy fly mpg create --name your-db --region iad # ⚠️ Replace CLUSTER_ID below with the cluster ID from the output above. # If you missed it, run: fly mpg list -pg_dump --no-owner --no-acl "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" | \ - fly mpg connect CLUSTER_ID +pg_dump --no-owner --no-acl -f heroku-dump.sql "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" +fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 fly mpg attach CLUSTER_ID -a your-fly-app # 4. Import secrets @@ -155,13 +155,15 @@ To import your data, dump it from Heroku with `pg_dump`, then restore it with `f ```bash # Dump the Heroku database (plain SQL format) export HEROKU_DB=$(heroku config:get DATABASE_URL -a $HEROKU_APP) -pg_dump --no-owner --no-acl "$HEROKU_DB" > heroku-dump.sql +pg_dump --no-owner --no-acl -f heroku-dump.sql "$HEROKU_DB" -# Restore to Fly (replace CLUSTER_ID with your cluster ID) -fly mpg connect CLUSTER_ID < heroku-dump.sql +# Restore to Managed Postgres (replace CLUSTER_ID with your cluster ID) +fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 ``` -You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. Your actual data will import successfully. If your `pg_dump` is version 17 or later, you'll also see `unrecognized configuration parameter "transaction_timeout"` on a Postgres 16 cluster, which is safe to ignore too. +The restore prints little besides errors, and carries on past them. You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. + +Read any other error before you go on: it means a statement didn't run. If the output ends with `Error: exit status 2`, the connection dropped and the import isn't complete. See [Restore errors](/postgres/import#restore-errors). Attach the database to your app: diff --git a/getting-started/troubleshooting.mdx b/getting-started/troubleshooting.mdx index c16969703b..3057c62993 100644 --- a/getting-started/troubleshooting.mdx +++ b/getting-started/troubleshooting.mdx @@ -488,7 +488,7 @@ MPG clusters run on Fly's private network and aren't accessible over the public **To connect from your local machine::** - Interactive psql: `fly mpg connect` -- One statement or a SQL file, no session: `fly mpg connect -c 'select 1'` or `fly mpg connect -f script.sql` +- One statement or a SQL file, no session: `fly mpg connect -c 'select 1'` or `fly mpg connect -f script.sql` - Proxy to localhost: `fly mpg proxy` — forwards a local port to your database - WireGuard: connect to your org's private network, then use the `.flympg.net` connection string directly. Read more in this [reference guide](/blueprints/connect-private-network-wireguard). diff --git a/postgres/create-and-connect.mdx b/postgres/create-and-connect.mdx index bae6e8f653..0b1a314d36 100644 --- a/postgres/create-and-connect.mdx +++ b/postgres/create-and-connect.mdx @@ -72,7 +72,7 @@ From the "Connect" tab of your MPG Cluster page you can attach a specific app in To attach an application to your database using flyctl run: ```bash -fly mpg attach -a +fly mpg attach -a ``` This will also trigger a restart of the selected app in order to add the secret. You can find the cluster ID in your MPG dashboard or by using the `fly mpg list` command. @@ -100,9 +100,11 @@ fly mpg connect [flags] This command will create a direct connection to your database and open a psql session. You'll need a compatible psql version installed locally. -### Run SQL without a session +### Running SQL without a session -To run SQL from a script, a CI job, or an agent, pass it to `fly mpg connect` with `-c` or `-f`. The command runs the SQL, prints the result, and exits. It doesn't prompt for a user or database, so pass the cluster ID. +To run SQL from a script, a CI job, or an agent, pass it to [`fly mpg connect`](/flyctl/cmd/fly_mpg_connect) with `-c` or `-f`. The command runs the SQL, prints the result, and exits. Pass the cluster ID: with `-c` or `-f`, flyctl doesn't prompt for a cluster, user, or database. + +The `-c` and `-f` flags are available in flyctl v0.5.9 and later. If your `fly` does not have them, run `fly version upgrade` or [install flyctl](/flyctl/install). Run one statement: @@ -116,7 +118,7 @@ Run a SQL file: fly mpg connect -f migrate.sql ``` -Both flags stop at the first error and skip your `~/.psqlrc`. You can repeat either flag, but you can't combine `-c` and `-f` in one command. Use `--database` and `--username` to connect to something other than the defaults. +Both flags stop at the first error and skip your psqlrc file. You can repeat either flag, but you can't combine `-c` and `-f` in one command. Use `--database` and `--username` to connect to something other than the defaults. Everything after `--` goes to psql unchanged. For example, to get a bare value into a shell variable, add psql's `-At` flags: @@ -126,23 +128,26 @@ count=$(fly mpg connect -c 'select count(*) from orders' -- -At) Only psql's output goes to stdout. The `Proxying...` line that flyctl prints goes to stderr. -You can also pipe SQL in. Piped SQL keeps psql's default behavior and carries on after an error, so add `ON_ERROR_STOP` if you want it to stop: +You can also pipe SQL in. Piped SQL keeps psql's default behavior: it reads your psqlrc file and carries on after an error. To stop at the first error, add `-X` and `ON_ERROR_STOP`: ```bash -cat queries.sql | fly mpg connect -- -v ON_ERROR_STOP=1 +cat queries.sql | fly mpg connect -- -X -v ON_ERROR_STOP=1 ``` `fly mpg connect` exits with status `0` when the SQL runs cleanly and `1` when psql fails, so a script can check it. On failure, psql's own error is followed by a line with psql's exit status: ```out $ fly mpg connect -f migrate.sql +Proxying localhost:16380 to remote [direct..flympg.net]:5432 psql:migrate.sql:2: ERROR: column "nope" does not exist +LINE 1: select nope; + ^ Error: exit status 3 ``` -psql uses `1` for a failed `-c` statement, `2` for a connection that couldn't be made or was lost, and `3` for a script that stopped at an error. +psql uses `1` for a failed `-c` statement or an error of its own, such as a file it can't read, `2` for a connection that couldn't be made or was lost, and `3` for a script that stopped at an error. flyctl also exits with status `1` for its own errors, such as a cluster it can't find. -`fly mpg connect` uses a fixed local port, so run one at a time on a given machine. +`fly mpg connect` listens on local port 16380, which is also the default port for `fly mpg proxy`. Run one at a time on your local machine, or give the proxy a different `--local-port`. ### Setting up a Proxy Connection diff --git a/postgres/import.mdx b/postgres/import.mdx index 017d7e23fd..b09552ae48 100644 --- a/postgres/import.mdx +++ b/postgres/import.mdx @@ -12,7 +12,7 @@ If you're migrating to Fly Managed Postgres (MPG) from an unmanaged Fly Postgres There isn't currently an automated way to migrate an existing DB's data, but you can accomplish this with a standard Postgres dump and restore process after creating your MPG cluster. -Your MPG cluster runs within your Fly.io private network, so it isn't reachable over the public internet. `fly mpg connect` opens the tunnel, looks up the credentials, and runs psql for you, so you can restore from your local machine without setting up a proxy or copying a password. You'll need `pg_dump` and `psql` installed locally. +Your MPG cluster runs within your Fly.io private network, so it isn't reachable over the public internet. `fly mpg connect` opens the tunnel, looks up the credentials, and runs psql for you, so you can restore from your local machine without setting up a proxy or copying a password. You'll need `pg_dump` and psql installed locally, and flyctl v0.5.9 or later. 1. Create your Managed Postgres cluster from the Fly.io dashboard, or using `fly mpg create` @@ -22,40 +22,46 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable 2. Dump your existing database to a file ```bash - pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" > dump.sql + pg_dump --no-owner --no-acl -f dump.sql "$LEGACY_PG_CONNECTION_STRING" ``` -3. Restore the file into your MPG cluster. Find the cluster ID with `fly mpg list` +3. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, remove one setting from the top of the dump. `pg_dump` 17 and later writes `SET transaction_timeout = 0;` there, and Postgres 16 doesn't know that setting. Check your version with `pg_dump --version` ```bash - fly mpg connect -f dump.sql + sed '1,20{/^SET transaction_timeout = 0;$/d;}' dump.sql > dump-pg16.sql ``` - The restore goes into the default database as the default user. Add `--database` or `--username` to pick others. + The `1,20` keeps the edit inside the dump's header, so it can't touch your data. Use `dump-pg16.sql` in the next step. -4. Check the exit status. `0` means every statement ran. With `-f`, the restore stops at the first error and `fly mpg connect` exits with status `1`, so a failed import can't pass for a finished one. +4. Restore the file into your MPG cluster. Find the cluster ID with `fly mpg list` + ```bash + fly mpg connect -f dump.sql -- -q --single-transaction + ``` -To skip the intermediate file, pipe `pg_dump` straight in. Piped SQL doesn't stop on errors by default, so turn that on: + `-q` keeps psql quiet, so errors stand out. `--single-transaction` makes the restore all-or-nothing: if it stops for any reason, nothing is left behind and you can run it again. The restore goes into the default database as the default user. Add `--database` or `--username` to pick others. -```bash -pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect -- -v ON_ERROR_STOP=1 -``` +5. Check the result. A restore that worked prints no errors and exits with status `0`. One that failed prints psql's error and a line like `Error: exit status 3`, and exits with status `1`. -### If the restore stops +### Restore errors -- **`unrecognized configuration parameter "transaction_timeout"`**: `pg_dump` version 17 and later writes `SET transaction_timeout = 0;` at the top of every dump, and Postgres 16 doesn't know that setting. -- **`must be owner of extension`**: the dump includes a `COMMENT ON EXTENSION` line for each extension, and the default user doesn't own the extensions that come preinstalled on MPG. - - Neither statement affects your data. Remove those lines and run the restore again: - ```bash - grep -vE '^(SET transaction_timeout|COMMENT ON EXTENSION)' dump.sql > dump-clean.sql - fly mpg connect -f dump-clean.sql - ``` - In the piped form, put the same `grep -vE` between `pg_dump` and `fly mpg connect`. -- **The connection drops partway**: the restore needs your network connection for the whole run, and a large dump can take a while. Statements that already ran stay in the database, so drop the partial tables before you retry. To make the restore all-or-nothing instead, add `-- --single-transaction`: +- **`unrecognized configuration parameter "transaction_timeout"`:** The dump came from `pg_dump` 17 or later and the cluster runs Postgres 16. Go back to step 3. +- **`must be owner of extension`:** The dump has a `COMMENT ON EXTENSION` statement for an extension that comes preinstalled on MPG, such as `pgaudit` or `pg_stat_monitor`, and the default user doesn't own those. You'll see this when the source is another MPG cluster. Dump again and leave those extensions out, which needs `pg_dump` 17 or later: ```bash - fly mpg connect -f dump.sql -- --single-transaction + pg_dump --no-owner --no-acl --exclude-extension=pgaudit --exclude-extension=pg_stat_monitor -f dump.sql "$LEGACY_PG_CONNECTION_STRING" ``` -- **Another error you expect and want to skip**: tell psql to carry on with `fly mpg connect -f dump.sql -- -v ON_ERROR_STOP=0`, then read the output for anything else that failed. In this mode the exit status is `0` even when statements fail. +- **`server closed the connection unexpectedly`:** The connection to your cluster dropped before the restore finished. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, nothing was imported, so run the restore again. +- **`already exists`:** An earlier restore without `--single-transaction` left tables, schemas, or types behind. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` to the restore. +- **Another error you'd rather skip:** Replace `--single-transaction` with `-v ON_ERROR_STOP=0`, as in `fly mpg connect -f dump.sql -- -q -v ON_ERROR_STOP=0`. psql then carries on past errors, and the exit status is `0` even when statements fail, so read every error it prints. + +### Piping a small database + +For a small database you can skip the file and pipe `pg_dump` straight in. Piped SQL keeps psql's defaults, so add the flags that `-f` sets for you. Set `pipefail` as well: without it, a `pg_dump` that fails looks like a restore that worked. + +```bash +set -o pipefail +pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect -- -X -q -v ON_ERROR_STOP=1 --single-transaction +``` + +If step 3 applies to you, put the same `sed` command between `pg_dump` and `fly mpg connect`. If the pipe fails, you have to dump again, so prefer the file for anything large. ### Use another tool diff --git a/postgres/index.mdx b/postgres/index.mdx index e18e324e3f..031d30cf28 100644 --- a/postgres/index.mdx +++ b/postgres/index.mdx @@ -31,11 +31,11 @@ You'll be able to access: ### Current limitations {#whats-not-there-yet} -- **Major version upgrades:** You choose Postgres 16 or 17 when you create a cluster, and you can't upgrade a cluster in place. To move to a newer major version, create a new cluster and copy your data with `pg_dump` and `pg_restore`. See [Import data](/postgres/import). +- **Major version upgrades:** You choose Postgres 16 or 17 when you create a cluster, and you can't upgrade a cluster in place. To move to a newer major version, create a new cluster and copy your data with `pg_dump` and psql. See [Import data](/postgres/import). - **Minor versions and security patches:** New Postgres images include the latest minor release. We roll out new images during your cluster's [maintenance window](/postgres/maintenance-windows), but not on a fixed schedule. PgBouncer and PostGIS versions are pinned and change only when we update them. - **Third-party extensions:** `pgvector` and `PostGIS` are supported, along with the bundled Postgres extensions marked as available on the [Extensions page](/postgres/extensions). To request another extension, [contact support](/about/support). - **Alerting:** We email your organization when maintenance starts and finishes, but there are no alerts for metrics like CPU or storage. Use [metrics](/postgres/metrics) to watch your cluster. -- **Migration tools:** There's no automated migration from another database. Use `pg_dump` and `pg_restore`, as described in [Import data](/postgres/import). +- **Migration tools:** There's no automated migration from another database. Use `pg_dump` and psql, as described in [Import data](/postgres/import). ## Regions From cefa7550b4bdfd85eb941ff26554c9c1e7ed448e Mon Sep 17 00:00:00 2001 From: vincent Date: Thu, 12 Nov 2026 16:39:33 -0500 Subject: [PATCH 3/8] Address second review round: recovery paths and wording Say when a failed dump or dropped connection leaves partial data and how to recover, spell out the piped sed, read SQL from stdin without cat, and tidy step punctuation and headings. Co-Authored-By: Claude Fable 5.2 --- getting-started/migrate-from-heroku.mdx | 8 +++---- getting-started/troubleshooting.mdx | 2 +- postgres/create-and-connect.mdx | 14 ++++++------ postgres/import.mdx | 30 ++++++++++++++----------- 4 files changed, 29 insertions(+), 25 deletions(-) diff --git a/getting-started/migrate-from-heroku.mdx b/getting-started/migrate-from-heroku.mdx index df2f893230..1e17a86215 100644 --- a/getting-started/migrate-from-heroku.mdx +++ b/getting-started/migrate-from-heroku.mdx @@ -37,8 +37,8 @@ fly launch --no-deploy fly mpg create --name your-db --region iad # ⚠️ Replace CLUSTER_ID below with the cluster ID from the output above. # If you missed it, run: fly mpg list -pg_dump --no-owner --no-acl -f heroku-dump.sql "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" -fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 +pg_dump --no-owner --no-acl -f heroku-dump.sql "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" && \ + fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 fly mpg attach CLUSTER_ID -a your-fly-app # 4. Import secrets @@ -161,9 +161,9 @@ pg_dump --no-owner --no-acl -f heroku-dump.sql "$HEROKU_DB" fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 ``` -The restore prints little besides errors, and carries on past them. You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. +Make sure `pg_dump` finishes without an error before you restore: a dump that failed partway restores without complaint. The restore prints a few query results, such as `setval`, and carries on past errors. You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. -Read any other error before you go on: it means a statement didn't run. If the output ends with `Error: exit status 2`, the connection dropped and the import isn't complete. See [Restore errors](/postgres/import#restore-errors). +Read any other error before you go on: it means a statement didn't run. If the output ends with `Error: exit status 2`, the connection dropped and the import isn't complete. Don't run the restore again on the same database, because tables that already loaded would get their rows twice. Create a new cluster with `fly mpg create` and restore into that one instead. See [Restore errors](/postgres/import#restore-errors). Attach the database to your app: diff --git a/getting-started/troubleshooting.mdx b/getting-started/troubleshooting.mdx index 3057c62993..27c3cb5838 100644 --- a/getting-started/troubleshooting.mdx +++ b/getting-started/troubleshooting.mdx @@ -488,7 +488,7 @@ MPG clusters run on Fly's private network and aren't accessible over the public **To connect from your local machine::** - Interactive psql: `fly mpg connect` -- One statement or a SQL file, no session: `fly mpg connect -c 'select 1'` or `fly mpg connect -f script.sql` +- SQL without a session: `fly mpg connect -c 'select 1'` or `fly mpg connect -f script.sql` - Proxy to localhost: `fly mpg proxy` — forwards a local port to your database - WireGuard: connect to your org's private network, then use the `.flympg.net` connection string directly. Read more in this [reference guide](/blueprints/connect-private-network-wireguard). diff --git a/postgres/create-and-connect.mdx b/postgres/create-and-connect.mdx index 0b1a314d36..dafda3fb49 100644 --- a/postgres/create-and-connect.mdx +++ b/postgres/create-and-connect.mdx @@ -1,6 +1,6 @@ --- title: Create and Connect to a Managed Postgres Cluster -description: "Create a Managed Postgres cluster with fly mpg create or the dashboard, attach an app with fly mpg attach, and connect locally via mpg proxy or WireGuard." +description: "Create a Managed Postgres cluster, attach an app with fly mpg attach, and connect or run SQL locally with fly mpg connect, mpg proxy, or WireGuard." source_url: https://fly.io/docs/mpg/create-and-connect/ --- @@ -126,15 +126,15 @@ Everything after `--` goes to psql unchanged. For example, to get a bare value i count=$(fly mpg connect -c 'select count(*) from orders' -- -At) ``` -Only psql's output goes to stdout. The `Proxying...` line that flyctl prints goes to stderr. +psql writes query results to stdout. The `Proxying...` line that flyctl prints goes to stderr. -You can also pipe SQL in. Piped SQL keeps psql's default behavior: it reads your psqlrc file and carries on after an error. To stop at the first error, add `-X` and `ON_ERROR_STOP`: +You can also send SQL on stdin. SQL from stdin keeps psql's default behavior: it reads your psqlrc file and carries on after an error. To stop at the first error, add `-X` and `ON_ERROR_STOP`: ```bash -cat queries.sql | fly mpg connect -- -X -v ON_ERROR_STOP=1 +fly mpg connect -- -X -v ON_ERROR_STOP=1 < queries.sql ``` -`fly mpg connect` exits with status `0` when the SQL runs cleanly and `1` when psql fails, so a script can check it. On failure, psql's own error is followed by a line with psql's exit status: +`fly mpg connect` exits with status `0` when psql succeeds and `1` when psql fails, so a script can check it. On failure, psql's own error is followed by a line with psql's exit status: ```out $ fly mpg connect -f migrate.sql @@ -145,9 +145,9 @@ LINE 1: select nope; Error: exit status 3 ``` -psql uses `1` for a failed `-c` statement or an error of its own, such as a file it can't read, `2` for a connection that couldn't be made or was lost, and `3` for a script that stopped at an error. flyctl also exits with status `1` for its own errors, such as a cluster it can't find. +psql uses `1` for a failed `-c` statement or an error of its own, such as a file it can't read. It uses `2` for a connection that couldn't be made or was lost, and `3` for a script that stopped at an error. flyctl also exits with status `1` for its own errors, such as a cluster it can't find. -`fly mpg connect` listens on local port 16380, which is also the default port for `fly mpg proxy`. Run one at a time on your local machine, or give the proxy a different `--local-port`. +`fly mpg connect` always listens on local port 16380, which is also the default port for `fly mpg proxy`. Run one `fly mpg connect` at a time on your local machine, and give a proxy you run alongside it a different `--local-port`. ### Setting up a Proxy Connection diff --git a/postgres/import.mdx b/postgres/import.mdx index b09552ae48..a90fd6d37a 100644 --- a/postgres/import.mdx +++ b/postgres/import.mdx @@ -20,37 +20,39 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable **Warning:** Your new cluster must be created with a volume at least as large as the database you're importing, otherwise the import process will fail. -2. Dump your existing database to a file +2. Dump your existing database to a file: ```bash pg_dump --no-owner --no-acl -f dump.sql "$LEGACY_PG_CONNECTION_STRING" ``` -3. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, remove one setting from the top of the dump. `pg_dump` 17 and later writes `SET transaction_timeout = 0;` there, and Postgres 16 doesn't know that setting. Check your version with `pg_dump --version` + Make sure `pg_dump` finishes without an error before you go on. A dump that failed partway leaves an incomplete file, and an incomplete file restores without complaint. + +3. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16 (the `fly mpg create` default), remove one setting from the top of the dump. `pg_dump` 17 and later writes `SET transaction_timeout = 0;` there, and Postgres 16 doesn't know that setting. Check your version with `pg_dump --version`, then remove the line: ```bash sed '1,20{/^SET transaction_timeout = 0;$/d;}' dump.sql > dump-pg16.sql ``` - The `1,20` keeps the edit inside the dump's header, so it can't touch your data. Use `dump-pg16.sql` in the next step. + The `1,20` keeps the edit inside the dump's header, so it can't touch your data. -4. Restore the file into your MPG cluster. Find the cluster ID with `fly mpg list` +4. Find the cluster ID with `fly mpg list`, then restore the file into your MPG cluster. If you ran step 3, restore `dump-pg16.sql` instead of `dump.sql`: ```bash fly mpg connect -f dump.sql -- -q --single-transaction ``` - `-q` keeps psql quiet, so errors stand out. `--single-transaction` makes the restore all-or-nothing: if it stops for any reason, nothing is left behind and you can run it again. The restore goes into the default database as the default user. Add `--database` or `--username` to pick others. + `-q` keeps psql quiet, so errors stand out. `--single-transaction` wraps the restore in one transaction: if an error stops it, nothing is left behind and you can run it again. The restore goes into the default database as the default user. Add `--database` or `--username` to pick others. -5. Check the result. A restore that worked prints no errors and exits with status `0`. One that failed prints psql's error and a line like `Error: exit status 3`, and exits with status `1`. +5. Check the result. A restore that worked prints no errors, only a few `set_config` and `setval` results, and exits with status `0`. One that failed prints psql's error and a line like `Error: exit status 3`, and exits with status `1`. ### Restore errors -- **`unrecognized configuration parameter "transaction_timeout"`:** The dump came from `pg_dump` 17 or later and the cluster runs Postgres 16. Go back to step 3. -- **`must be owner of extension`:** The dump has a `COMMENT ON EXTENSION` statement for an extension that comes preinstalled on MPG, such as `pgaudit` or `pg_stat_monitor`, and the default user doesn't own those. You'll see this when the source is another MPG cluster. Dump again and leave those extensions out, which needs `pg_dump` 17 or later: +- **`unrecognized configuration parameter "transaction_timeout"`:** The dump came from `pg_dump` 17 or later and the cluster runs Postgres 16. Go back to step 3, then restore `dump-pg16.sql`. +- **`must be owner of extension`:** The dump has a `COMMENT ON EXTENSION` statement for an extension that comes preinstalled on MPG, such as `pgaudit` or `pg_stat_monitor`, and the default user doesn't own those. You'll see this when the source is another MPG cluster. If you reach that cluster through `fly mpg proxy`, give the proxy a different `--local-port`, because `fly mpg connect` always uses 16380. Dump again with `pg_dump` 17 or later and leave those extensions out, then repeat steps 3 and 4: ```bash pg_dump --no-owner --no-acl --exclude-extension=pgaudit --exclude-extension=pg_stat_monitor -f dump.sql "$LEGACY_PG_CONNECTION_STRING" ``` -- **`server closed the connection unexpectedly`:** The connection to your cluster dropped before the restore finished. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, nothing was imported, so run the restore again. -- **`already exists`:** An earlier restore without `--single-transaction` left tables, schemas, or types behind. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` to the restore. -- **Another error you'd rather skip:** Replace `--single-transaction` with `-v ON_ERROR_STOP=0`, as in `fly mpg connect -f dump.sql -- -q -v ON_ERROR_STOP=0`. psql then carries on past errors, and the exit status is `0` even when statements fail, so read every error it prints. +- **`server closed the connection unexpectedly` or `connection to server was lost`:** The connection to your cluster dropped. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, a restore that was still running is rolled back. Check whether your tables are there before you run it again, in case the connection dropped just as the restore finished. +- **`already exists`:** The database isn't empty. An earlier restore finished, ran without `--single-transaction`, or was piped from a `pg_dump` that failed partway. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` to the restore. +- **Another error you'd rather skip:** Replace `--single-transaction` with `-v ON_ERROR_STOP=0`, as in `fly mpg connect -f dump.sql -- -q -v ON_ERROR_STOP=0`. psql then carries on past errors, and the exit status is `0` even when statements fail, so read every error it prints. Only do this into an empty database: over a partial import, tables that already loaded get their rows twice. ### Piping a small database @@ -61,9 +63,11 @@ set -o pipefail pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect -- -X -q -v ON_ERROR_STOP=1 --single-transaction ``` -If step 3 applies to you, put the same `sed` command between `pg_dump` and `fly mpg connect`. If the pipe fails, you have to dump again, so prefer the file for anything large. +If step 3 applies to you, add its `sed` expression to the pipe, without the file names: `pg_dump ... | sed '1,20{/^SET transaction_timeout = 0;$/d;}' | fly mpg connect ...`. + +If `pg_dump` fails partway, psql has already committed what it received, so part of your data is imported even though the command fails. Restore into a new, empty database before you try again, as described under `already exists`. Prefer the file for anything large. -### Use another tool +### Using another tool Tools that make their own connection, like `pg_restore` with a custom-format dump, need an address to connect to. Run `fly mpg proxy ` and point the tool at the local port it prints, using the username and password from your cluster's connection string in the dashboard. See [Setting up a Proxy Connection](/postgres/create-and-connect#setting-up-a-proxy-connection). From 628f503ea61944c431f09560d765831b258c89e5 Mon Sep 17 00:00:00 2001 From: vincent Date: Thu, 12 Nov 2026 17:38:50 -0500 Subject: [PATCH 4/8] Address final review gate: qualify recovery claims Co-Authored-By: Claude Fable 5.2 --- getting-started/migrate-from-heroku.mdx | 4 ++-- postgres/import.mdx | 11 ++++++----- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/getting-started/migrate-from-heroku.mdx b/getting-started/migrate-from-heroku.mdx index 1e17a86215..100c9840b6 100644 --- a/getting-started/migrate-from-heroku.mdx +++ b/getting-started/migrate-from-heroku.mdx @@ -161,9 +161,9 @@ pg_dump --no-owner --no-acl -f heroku-dump.sql "$HEROKU_DB" fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 ``` -Make sure `pg_dump` finishes without an error before you restore: a dump that failed partway restores without complaint. The restore prints a few query results, such as `setval`, and carries on past errors. You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. +Make sure `pg_dump` finishes without an error before you restore: a dump that failed partway can restore without complaint. The restore prints a few query results, such as `setval`, and carries on past errors. You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. -Read any other error before you go on: it means a statement didn't run. If the output ends with `Error: exit status 2`, the connection dropped and the import isn't complete. Don't run the restore again on the same database, because tables that already loaded would get their rows twice. Create a new cluster with `fly mpg create` and restore into that one instead. See [Restore errors](/postgres/import#restore-errors). +Read any other error before you go on: it means a statement didn't run. If the output ends with `connection to server was lost` and `Error: exit status 2`, the connection dropped partway and the import isn't complete. Don't run the restore again on the same database, because tables that already loaded would get their rows twice. Create a new cluster with `fly mpg create` and restore into that one instead. See [Restore errors](/postgres/import#restore-errors). Attach the database to your app: diff --git a/postgres/import.mdx b/postgres/import.mdx index a90fd6d37a..b00d409992 100644 --- a/postgres/import.mdx +++ b/postgres/import.mdx @@ -25,7 +25,7 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable pg_dump --no-owner --no-acl -f dump.sql "$LEGACY_PG_CONNECTION_STRING" ``` - Make sure `pg_dump` finishes without an error before you go on. A dump that failed partway leaves an incomplete file, and an incomplete file restores without complaint. + Make sure `pg_dump` finishes without an error before you go on. A dump that failed partway leaves an incomplete file, and an incomplete file can restore without complaint. 3. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16 (the `fly mpg create` default), remove one setting from the top of the dump. `pg_dump` 17 and later writes `SET transaction_timeout = 0;` there, and Postgres 16 doesn't know that setting. Check your version with `pg_dump --version`, then remove the line: ```bash @@ -39,9 +39,9 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable fly mpg connect -f dump.sql -- -q --single-transaction ``` - `-q` keeps psql quiet, so errors stand out. `--single-transaction` wraps the restore in one transaction: if an error stops it, nothing is left behind and you can run it again. The restore goes into the default database as the default user. Add `--database` or `--username` to pick others. + `-q` keeps psql quiet, so errors stand out. `--single-transaction` wraps the restore in one transaction: if an error stops it, nothing is left behind and you can run it again. The exception is a dump that contains large objects, which carries its own transactions. The restore goes into the default database as the default user. Add `--database` or `--username` before the `--` to pick others. -5. Check the result. A restore that worked prints no errors, only a few `set_config` and `setval` results, and exits with status `0`. One that failed prints psql's error and a line like `Error: exit status 3`, and exits with status `1`. +5. Check the result. A restore that worked prints no errors, only a few query results such as `set_config` and `setval`, and exits with status `0`. One that failed prints psql's error and a line like `Error: exit status 3`, and exits with status `1`. ### Restore errors @@ -51,7 +51,8 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable pg_dump --no-owner --no-acl --exclude-extension=pgaudit --exclude-extension=pg_stat_monitor -f dump.sql "$LEGACY_PG_CONNECTION_STRING" ``` - **`server closed the connection unexpectedly` or `connection to server was lost`:** The connection to your cluster dropped. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, a restore that was still running is rolled back. Check whether your tables are there before you run it again, in case the connection dropped just as the restore finished. -- **`already exists`:** The database isn't empty. An earlier restore finished, ran without `--single-transaction`, or was piped from a `pg_dump` that failed partway. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` to the restore. +- **`already exists`:** The database isn't empty. An earlier restore finished, ran without `--single-transaction`, or was piped from a `pg_dump` that failed partway. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` before the `--` in the restore command. +- **`out of shared memory`, with a hint about `max_locks_per_transaction`:** The dump has more tables than one transaction can lock, so the restore was rolled back. Restore without `--single-transaction`, as in `fly mpg connect -f dump.sql -- -q`. psql still stops at the first error, but what it restored before the error stays, so restore into an empty database. - **Another error you'd rather skip:** Replace `--single-transaction` with `-v ON_ERROR_STOP=0`, as in `fly mpg connect -f dump.sql -- -q -v ON_ERROR_STOP=0`. psql then carries on past errors, and the exit status is `0` even when statements fail, so read every error it prints. Only do this into an empty database: over a partial import, tables that already loaded get their rows twice. ### Piping a small database @@ -65,7 +66,7 @@ pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect Date: Thu, 12 Nov 2026 18:10:12 -0500 Subject: [PATCH 5/8] Use -f - for stdin and the database option on the Heroku page From review: read piped SQL with -f - so it gets the file defaults, recover a dropped Heroku restore into a new database instead of a new cluster, and match the cluster placeholder on the troubleshooting page. Co-Authored-By: Claude Fable 5.2 --- getting-started/migrate-from-heroku.mdx | 2 +- getting-started/troubleshooting.mdx | 2 +- postgres/create-and-connect.mdx | 6 ++++-- postgres/import.mdx | 4 ++-- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/getting-started/migrate-from-heroku.mdx b/getting-started/migrate-from-heroku.mdx index 100c9840b6..cc9d77257c 100644 --- a/getting-started/migrate-from-heroku.mdx +++ b/getting-started/migrate-from-heroku.mdx @@ -163,7 +163,7 @@ fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 Make sure `pg_dump` finishes without an error before you restore: a dump that failed partway can restore without complaint. The restore prints a few query results, such as `setval`, and carries on past errors. You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. -Read any other error before you go on: it means a statement didn't run. If the output ends with `connection to server was lost` and `Error: exit status 2`, the connection dropped partway and the import isn't complete. Don't run the restore again on the same database, because tables that already loaded would get their rows twice. Create a new cluster with `fly mpg create` and restore into that one instead. See [Restore errors](/postgres/import#restore-errors). +Read any other error before you go on: it means a statement didn't run. If the output ends with `connection to server was lost` and `Error: exit status 2`, the connection dropped partway and the import isn't complete. Don't run the restore again on the same database, because tables that already loaded would get their rows twice. Create an empty database with `fly mpg databases create CLUSTER_ID --name heroku_import`, then add `--database heroku_import` before the `--` in the restore command, and to `fly mpg attach` below. See [Restore errors](/postgres/import#restore-errors). Attach the database to your app: diff --git a/getting-started/troubleshooting.mdx b/getting-started/troubleshooting.mdx index 27c3cb5838..c6b6e4d8fa 100644 --- a/getting-started/troubleshooting.mdx +++ b/getting-started/troubleshooting.mdx @@ -488,7 +488,7 @@ MPG clusters run on Fly's private network and aren't accessible over the public **To connect from your local machine::** - Interactive psql: `fly mpg connect` -- SQL without a session: `fly mpg connect -c 'select 1'` or `fly mpg connect -f script.sql` +- SQL without a session: `fly mpg connect -c 'select 1'` or `fly mpg connect -f script.sql` - Proxy to localhost: `fly mpg proxy` — forwards a local port to your database - WireGuard: connect to your org's private network, then use the `.flympg.net` connection string directly. Read more in this [reference guide](/blueprints/connect-private-network-wireguard). diff --git a/postgres/create-and-connect.mdx b/postgres/create-and-connect.mdx index dafda3fb49..4dd19937e4 100644 --- a/postgres/create-and-connect.mdx +++ b/postgres/create-and-connect.mdx @@ -128,12 +128,14 @@ count=$(fly mpg connect -c 'select count(*) from orders' -- -At) psql writes query results to stdout. The `Proxying...` line that flyctl prints goes to stderr. -You can also send SQL on stdin. SQL from stdin keeps psql's default behavior: it reads your psqlrc file and carries on after an error. To stop at the first error, add `-X` and `ON_ERROR_STOP`: +To read SQL from stdin, pass `-f -`. It behaves like any other file: ```bash -fly mpg connect -- -X -v ON_ERROR_STOP=1 < queries.sql +fly mpg connect -f - < queries.sql ``` +SQL sent on stdin without `-c` or `-f` keeps psql's default behavior: it reads your psqlrc file and carries on after an error. + `fly mpg connect` exits with status `0` when psql succeeds and `1` when psql fails, so a script can check it. On failure, psql's own error is followed by a line with psql's exit status: ```out diff --git a/postgres/import.mdx b/postgres/import.mdx index b00d409992..ffcd280861 100644 --- a/postgres/import.mdx +++ b/postgres/import.mdx @@ -57,11 +57,11 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable ### Piping a small database -For a small database you can skip the file and pipe `pg_dump` straight in. Piped SQL keeps psql's defaults, so add the flags that `-f` sets for you. Set `pipefail` as well: without it, a `pg_dump` that fails looks like a restore that worked. +For a small database you can skip the file and pipe `pg_dump` straight in. Pass `-f -` to read the dump from stdin. Set `pipefail` as well: without it, a `pg_dump` that fails looks like a restore that worked. ```bash set -o pipefail -pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect -- -X -q -v ON_ERROR_STOP=1 --single-transaction +pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect -f - -- -q --single-transaction ``` If step 3 applies to you, add its `sed` expression to the pipe, without the file names: `pg_dump ... | sed '1,20{/^SET transaction_timeout = 0;$/d;}' | fly mpg connect ...`. From 83ef64a21d26cc6531e0e37e2c05df551751d1a6 Mon Sep 17 00:00:00 2001 From: vincent Date: Thu, 12 Nov 2026 18:16:10 -0500 Subject: [PATCH 6/8] Mention --database on attach after restoring into a new database Co-Authored-By: Claude Fable 5.2 --- postgres/import.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/postgres/import.mdx b/postgres/import.mdx index ffcd280861..1634e3f299 100644 --- a/postgres/import.mdx +++ b/postgres/import.mdx @@ -51,7 +51,7 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable pg_dump --no-owner --no-acl --exclude-extension=pgaudit --exclude-extension=pg_stat_monitor -f dump.sql "$LEGACY_PG_CONNECTION_STRING" ``` - **`server closed the connection unexpectedly` or `connection to server was lost`:** The connection to your cluster dropped. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, a restore that was still running is rolled back. Check whether your tables are there before you run it again, in case the connection dropped just as the restore finished. -- **`already exists`:** The database isn't empty. An earlier restore finished, ran without `--single-transaction`, or was piped from a `pg_dump` that failed partway. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` before the `--` in the restore command. +- **`already exists`:** The database isn't empty. An earlier restore finished, ran without `--single-transaction`, or was piped from a `pg_dump` that failed partway. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` before the `--` in the restore command. Pass the same flag to `fly mpg attach` when you attach your app. - **`out of shared memory`, with a hint about `max_locks_per_transaction`:** The dump has more tables than one transaction can lock, so the restore was rolled back. Restore without `--single-transaction`, as in `fly mpg connect -f dump.sql -- -q`. psql still stops at the first error, but what it restored before the error stays, so restore into an empty database. - **Another error you'd rather skip:** Replace `--single-transaction` with `-v ON_ERROR_STOP=0`, as in `fly mpg connect -f dump.sql -- -q -v ON_ERROR_STOP=0`. psql then carries on past errors, and the exit status is `0` even when statements fail, so read every error it prints. Only do this into an empty database: over a partial import, tables that already loaded get their rows twice. From 3e6d81566a41834e04dc60b0e9d20f96792224a1 Mon Sep 17 00:00:00 2001 From: vincent Date: Thu, 12 Nov 2026 18:35:43 -0500 Subject: [PATCH 7/8] Qualify the large-objects exception in restore errors Co-Authored-By: Claude Fable 5.2 --- postgres/import.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/postgres/import.mdx b/postgres/import.mdx index 1634e3f299..f2b50ab65a 100644 --- a/postgres/import.mdx +++ b/postgres/import.mdx @@ -39,7 +39,7 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable fly mpg connect -f dump.sql -- -q --single-transaction ``` - `-q` keeps psql quiet, so errors stand out. `--single-transaction` wraps the restore in one transaction: if an error stops it, nothing is left behind and you can run it again. The exception is a dump that contains large objects, which carries its own transactions. The restore goes into the default database as the default user. Add `--database` or `--username` before the `--` to pick others. + `-q` keeps psql quiet, so errors stand out. `--single-transaction` wraps the restore in one transaction: if an error stops it, nothing is left behind and you can run it again. The exception is a dump that contains large objects: it carries its own transactions, so a restore that fails after them leaves what it already restored. The restore goes into the default database as the default user. Add `--database` or `--username` before the `--` to pick others. 5. Check the result. A restore that worked prints no errors, only a few query results such as `set_config` and `setval`, and exits with status `0`. One that failed prints psql's error and a line like `Error: exit status 3`, and exits with status `1`. @@ -50,9 +50,9 @@ Your MPG cluster runs within your Fly.io private network, so it isn't reachable ```bash pg_dump --no-owner --no-acl --exclude-extension=pgaudit --exclude-extension=pg_stat_monitor -f dump.sql "$LEGACY_PG_CONNECTION_STRING" ``` -- **`server closed the connection unexpectedly` or `connection to server was lost`:** The connection to your cluster dropped. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, a restore that was still running is rolled back. Check whether your tables are there before you run it again, in case the connection dropped just as the restore finished. +- **`server closed the connection unexpectedly` or `connection to server was lost`:** The connection to your cluster dropped. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, a restore that was still running is rolled back, unless the dump contains large objects. Check whether your tables are there before you run it again, in case the connection dropped just as the restore finished. - **`already exists`:** The database isn't empty. An earlier restore finished, ran without `--single-transaction`, or was piped from a `pg_dump` that failed partway. Restore into an empty database instead: create one with `fly mpg databases create --name `, then add `--database ` before the `--` in the restore command. Pass the same flag to `fly mpg attach` when you attach your app. -- **`out of shared memory`, with a hint about `max_locks_per_transaction`:** The dump has more tables than one transaction can lock, so the restore was rolled back. Restore without `--single-transaction`, as in `fly mpg connect -f dump.sql -- -q`. psql still stops at the first error, but what it restored before the error stays, so restore into an empty database. +- **`out of shared memory`, with a hint about `max_locks_per_transaction`:** The dump has more tables than one transaction can lock, so the restore was rolled back. Restore without `--single-transaction`, as in `fly mpg connect -f dump.sql -- -q`. psql still stops at the first error, but what it restored before the error stays. If that happens, restore into a new, empty database before you try again. - **Another error you'd rather skip:** Replace `--single-transaction` with `-v ON_ERROR_STOP=0`, as in `fly mpg connect -f dump.sql -- -q -v ON_ERROR_STOP=0`. psql then carries on past errors, and the exit status is `0` even when statements fail, so read every error it prints. Only do this into an empty database: over a partial import, tables that already loaded get their rows twice. ### Piping a small database From d36b0d2dc20a35fc5de305dcc6f27714cb2ea9ac Mon Sep 17 00:00:00 2001 From: vincent Date: Thu, 12 Nov 2026 18:54:27 -0500 Subject: [PATCH 8/8] Show the errors a real Heroku dump produces on restore Co-Authored-By: Claude Fable 5.2 --- getting-started/migrate-from-heroku.mdx | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/getting-started/migrate-from-heroku.mdx b/getting-started/migrate-from-heroku.mdx index cc9d77257c..26972fbf07 100644 --- a/getting-started/migrate-from-heroku.mdx +++ b/getting-started/migrate-from-heroku.mdx @@ -161,7 +161,15 @@ pg_dump --no-owner --no-acl -f heroku-dump.sql "$HEROKU_DB" fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0 ``` -Make sure `pg_dump` finishes without an error before you restore: a dump that failed partway can restore without complaint. The restore prints a few query results, such as `setval`, and carries on past errors. You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. +Make sure `pg_dump` finishes without an error before you restore: a dump that failed partway can restore without complaint. The restore prints a few query results, such as `setval`, and carries on past errors. You'll see errors about the `pg_stat_statements` extension and about Heroku's own event triggers, like these: + +```out +psql:heroku-dump.sql:68: ERROR: permission denied to create extension "pg_stat_statements" +psql:heroku-dump.sql:75: ERROR: extension "pg_stat_statements" does not exist +psql:heroku-dump.sql:25952: ERROR: permission denied to create event trigger "00_validate_before_start" +``` + +These are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too. Read any other error before you go on: it means a statement didn't run. If the output ends with `connection to server was lost` and `Error: exit status 2`, the connection dropped partway and the import isn't complete. Don't run the restore again on the same database, because tables that already loaded would get their rows twice. Create an empty database with `fly mpg databases create CLUSTER_ID --name heroku_import`, then add `--database heroku_import` before the `--` in the restore command, and to `fly mpg attach` below. See [Restore errors](/postgres/import#restore-errors).