Skip to content
Merged
30 changes: 17 additions & 13 deletions getting-started/migrate-from-heroku.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,8 @@ fly launch --no-deploy
fly mpg create --name your-db --region iad
# ⚠️ Replace CLUSTER_ID below with the cluster ID from the output above.
# If you missed it, run: fly mpg list
fly mpg proxy CLUSTER_ID --local-port 15432 &
pg_dump --no-owner --no-acl "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" | \
PGPASSWORD=YOUR_FLY_DB_PASSWORD psql -h localhost -p 15432 -U fly-user -d fly-db
pg_dump --no-owner --no-acl -f heroku-dump.sql "$(heroku config:get DATABASE_URL -a YOUR_HEROKU_APP)" && \
fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0
fly mpg attach CLUSTER_ID -a your-fly-app

# 4. Import secrets
Expand Down Expand Up @@ -151,23 +150,28 @@ Connection string: postgresql://fly-user:PASSWORD@pgbouncer.CLUSTER_ID.flympg.ne

Note the cluster ID (the `CLUSTER_ID` part) - you'll need it for the next few commands. If you lose it, you can always find it with `fly mpg list`.

To import your data, use `fly mpg proxy` to create a local tunnel, then `pg_dump` and `psql`:
To import your data, dump it from Heroku with `pg_dump`, then restore it with `fly mpg connect`, which opens a tunnel to your database and runs `psql` for you:

```bash
# Terminal 1: Start a proxy to your Fly database (replace CLUSTER_ID with your cluster ID)
fly mpg proxy CLUSTER_ID --local-port 15432

# Terminal 2: Dump from Heroku and restore to Fly
# Dump the Heroku database (plain SQL format)
export HEROKU_DB=$(heroku config:get DATABASE_URL -a $HEROKU_APP)
pg_dump --no-owner --no-acl -f heroku-dump.sql "$HEROKU_DB"

# Restore to Managed Postgres (replace CLUSTER_ID with your cluster ID)
fly mpg connect CLUSTER_ID -f heroku-dump.sql -- -q -v ON_ERROR_STOP=0
```

# Dump the database (plain SQL format)
pg_dump --no-owner --no-acl "$HEROKU_DB" > heroku-dump.sql
Make sure `pg_dump` finishes without an error before you restore: a dump that failed partway can restore without complaint. The restore prints a few query results, such as `setval`, and carries on past errors. You'll see errors about the `pg_stat_statements` extension and about Heroku's own event triggers, like these:

# Restore to Fly (via the proxy)
PGPASSWORD=YOUR_PASSWORD psql -h localhost -p 15432 -U fly-user -d fly-db -f heroku-dump.sql
```out
psql:heroku-dump.sql:68: ERROR: permission denied to create extension "pg_stat_statements"
psql:heroku-dump.sql:75: ERROR: extension "pg_stat_statements" does not exist
psql:heroku-dump.sql:25952: ERROR: permission denied to create event trigger "00_validate_before_start"
```

You'll see some errors about Heroku-specific extensions (pg_stat_statements, event triggers) - these are safe to ignore. Your actual data will import successfully.
These are safe to ignore. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16, you'll also see `unrecognized configuration parameter "transaction_timeout"`, which is safe to ignore too.

Read any other error before you go on: it means a statement didn't run. If the output ends with `connection to server was lost` and `Error: exit status 2`, the connection dropped partway and the import isn't complete. Don't run the restore again on the same database, because tables that already loaded would get their rows twice. Create an empty database with `fly mpg databases create CLUSTER_ID --name heroku_import`, then add `--database heroku_import` before the `--` in the restore command, and to `fly mpg attach` below. See [Restore errors](/postgres/import#restore-errors).

Attach the database to your app:

Expand Down
1 change: 1 addition & 0 deletions getting-started/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -488,6 +488,7 @@ MPG clusters run on Fly's private network and aren't accessible over the public
**To connect from your local machine::**

- Interactive psql: `fly mpg connect`
- SQL without a session: `fly mpg connect <CLUSTER_ID> -c 'select 1'` or `fly mpg connect <CLUSTER_ID> -f script.sql`
- Proxy to localhost: `fly mpg proxy` — forwards a local port to your database
- WireGuard: connect to your org's private network, then use the `.flympg.net` connection string directly. Read more in this [reference guide](/blueprints/connect-private-network-wireguard).

Expand Down
55 changes: 53 additions & 2 deletions postgres/create-and-connect.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Create and Connect to a Managed Postgres Cluster
description: "Create a Managed Postgres cluster with fly mpg create or the dashboard, attach an app with fly mpg attach, and connect locally via mpg proxy or WireGuard."
description: "Create a Managed Postgres cluster, attach an app with fly mpg attach, and connect or run SQL locally with fly mpg connect, mpg proxy, or WireGuard."
source_url: https://fly.io/docs/mpg/create-and-connect/
---

Expand Down Expand Up @@ -72,9 +72,9 @@

To attach an application to your database using flyctl run:
```bash
fly mpg attach <clusterID> -a <app-to-attach>
fly mpg attach <CLUSTER_ID> -a <app-to-attach>
```
This will also trigger a restart of the selected app in order to add the secret. You can find the cluster ID in your MPG dashboard or by using the `fly mpg list` command.

Check warning on line 77 in postgres/create-and-connect.mdx

View workflow job for this annotation

GitHub Actions / Vale linter

[vale] reported by reviewdog 🐶 Consider using 'to' instead of 'in order to'. Raw Output: {"message":"Consider using 'to' instead of 'in order to'.","location":{"path":"postgres/create-and-connect.mdx","range":{"start":{"line":77,"column":54},"end":{"line":77,"column":65}}},"severity":"INFO","code":{"value":"Fly.WordList"}}

Both of the attachment options will use the pooled connection URL, which uses PGBouncer to help manage database connections efficiently. This is recommended for most apps.

Expand All @@ -100,6 +100,57 @@

This command will create a direct connection to your database and open a psql session. You'll need a compatible psql version installed locally.

### Running SQL without a session

To run SQL from a script, a CI job, or an agent, pass it to [`fly mpg connect`](/flyctl/cmd/fly_mpg_connect) with `-c` or `-f`. The command runs the SQL, prints the result, and exits. Pass the cluster ID: with `-c` or `-f`, flyctl doesn't prompt for a cluster, user, or database.

The `-c` and `-f` flags are available in flyctl v0.5.9 and later. If your `fly` does not have them, run `fly version upgrade` or [install flyctl](/flyctl/install).

Run one statement:

```bash
fly mpg connect <CLUSTER_ID> -c 'select count(*) from orders'
```

Run a SQL file:

```bash
fly mpg connect <CLUSTER_ID> -f migrate.sql
```

Both flags stop at the first error and skip your psqlrc file. You can repeat either flag, but you can't combine `-c` and `-f` in one command. Use `--database` and `--username` to connect to something other than the defaults.

Everything after `--` goes to psql unchanged. For example, to get a bare value into a shell variable, add psql's `-At` flags:

```bash
count=$(fly mpg connect <CLUSTER_ID> -c 'select count(*) from orders' -- -At)
```

psql writes query results to stdout. The `Proxying...` line that flyctl prints goes to stderr.

To read SQL from stdin, pass `-f -`. It behaves like any other file:

```bash
fly mpg connect <CLUSTER_ID> -f - < queries.sql
```

SQL sent on stdin without `-c` or `-f` keeps psql's default behavior: it reads your psqlrc file and carries on after an error.

`fly mpg connect` exits with status `0` when psql succeeds and `1` when psql fails, so a script can check it. On failure, psql's own error is followed by a line with psql's exit status:

```out
$ fly mpg connect <CLUSTER_ID> -f migrate.sql
Proxying localhost:16380 to remote [direct.<CLUSTER_ID>.flympg.net]:5432
psql:migrate.sql:2: ERROR: column "nope" does not exist
LINE 1: select nope;
^
Error: exit status 3
```

psql uses `1` for a failed `-c` statement or an error of its own, such as a file it can't read. It uses `2` for a connection that couldn't be made or was lost, and `3` for a script that stopped at an error. flyctl also exits with status `1` for its own errors, such as a cluster it can't find.

`fly mpg connect` always listens on local port 16380, which is also the default port for `fly mpg proxy`. Run one `fly mpg connect` at a time on your local machine, and give a proxy you run alongside it a different `--local-port`.

### Setting up a Proxy Connection

You can create a proxy connection to your Managed Postgres database using the '[mpg proxy](/flyctl/cmd/fly_mpg_proxy)' command:
Expand Down
68 changes: 47 additions & 21 deletions postgres/import.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Import data from another postgres cluster
description: "Import data into Fly.io Managed Postgres from Fly Postgres or another provider using pg_dump and psql over fly mpg proxy, plus size limits and pgloader."
description: "Import data into Fly.io Managed Postgres from Fly Postgres or another provider using pg_dump and fly mpg connect, plus size limits and pgloader."
source_url: https://fly.io/docs/mpg/import/
---

Expand All @@ -12,39 +12,65 @@ If you're migrating to Fly Managed Postgres (MPG) from an unmanaged Fly Postgres

There isn't currently an automated way to migrate an existing DB's data, but you can accomplish this with a standard Postgres dump and restore process after creating your MPG cluster.

For simple databases, this can be as easy as running :

```bash
pg_dump "$LEGACY_PG_CONNECTION_STRING" | psql "$MPG_URI".
```

Because your MPG cluster runs within your Fly Private Network, you'll need to be connected to your organizations Wireguard network in order to restore into your MPG database. To run the import process from your local machine, you would take the following steps:
Your MPG cluster runs within your Fly.io private network, so it isn't reachable over the public internet. `fly mpg connect` opens the tunnel, looks up the credentials, and runs psql for you, so you can restore from your local machine without setting up a proxy or copying a password. You'll need `pg_dump` and psql installed locally, and flyctl v0.5.9 or later.

1. Create your Managed Postgres cluster from the Fly.io dashboard, or using `fly mpg create`

<Warning>
**Warning:** Your new cluster must be created with a volume at least as large as the database you're importing, otherwise the import process will fail.
</Warning>

2. Open a proxy connection to your MPG database with
```out
$ fly mpg proxy
? Select Organization: My Organization (personal)
? Select a Postgres cluster my-test-cluster (iad)
Proxying localhost:16380 to remote [fdaa:1:2345:0:0::67]:5432
2. Dump your existing database to a file:
```bash
pg_dump --no-owner --no-acl -f dump.sql "$LEGACY_PG_CONNECTION_STRING"
```

Make sure `pg_dump` finishes without an error before you go on. A dump that failed partway leaves an incomplete file, and an incomplete file can restore without complaint.

3. If your `pg_dump` is version 17 or later and your cluster runs Postgres 16 (the `fly mpg create` default), remove one setting from the top of the dump. `pg_dump` 17 and later writes `SET transaction_timeout = 0;` there, and Postgres 16 doesn't know that setting. Check your version with `pg_dump --version`, then remove the line:
```bash
sed '1,20{/^SET transaction_timeout = 0;$/d;}' dump.sql > dump-pg16.sql
```

3. Find your MPG cluster's connection string in the dashboard, and modify it to use the local proxy:
```out
Connection String: postgres://fly-user:<password>@localhost:16380/fly-db
```
The `1,20` keeps the edit inside the dump's header, so it can't touch your data.

4. Run the dump and restore command
4. Find the cluster ID with `fly mpg list`, then restore the file into your MPG cluster. If you ran step 3, restore `dump-pg16.sql` instead of `dump.sql`:
```bash
pg_dump "$LEGACY_PG_CONNECTION_STRING" | psql postgres://fly-user:<password>@localhost:16380/fly-db
fly mpg connect <CLUSTER_ID> -f dump.sql -- -q --single-transaction
```

5. If everything succeeds, all data from your legacy cluster will be imported into your new MPG cluster.
`-q` keeps psql quiet, so errors stand out. `--single-transaction` wraps the restore in one transaction: if an error stops it, nothing is left behind and you can run it again. The exception is a dump that contains large objects: it carries its own transactions, so a restore that fails after them leaves what it already restored. The restore goes into the default database as the default user. Add `--database` or `--username` before the `--` to pick others.

5. Check the result. A restore that worked prints no errors, only a few query results such as `set_config` and `setval`, and exits with status `0`. One that failed prints psql's error and a line like `Error: exit status 3`, and exits with status `1`.

### Restore errors

- **`unrecognized configuration parameter "transaction_timeout"`:** The dump came from `pg_dump` 17 or later and the cluster runs Postgres 16. Go back to step 3, then restore `dump-pg16.sql`.
- **`must be owner of extension`:** The dump has a `COMMENT ON EXTENSION` statement for an extension that comes preinstalled on MPG, such as `pgaudit` or `pg_stat_monitor`, and the default user doesn't own those. You'll see this when the source is another MPG cluster. If you reach that cluster through `fly mpg proxy`, give the proxy a different `--local-port`, because `fly mpg connect` always uses 16380. Dump again with `pg_dump` 17 or later and leave those extensions out, then repeat steps 3 and 4:
```bash
pg_dump --no-owner --no-acl --exclude-extension=pgaudit --exclude-extension=pg_stat_monitor -f dump.sql "$LEGACY_PG_CONNECTION_STRING"
```
- **`server closed the connection unexpectedly` or `connection to server was lost`:** The connection to your cluster dropped. A restore needs the connection for the whole run, and a large one can take a while. With `--single-transaction`, a restore that was still running is rolled back, unless the dump contains large objects. Check whether your tables are there before you run it again, in case the connection dropped just as the restore finished.
- **`already exists`:** The database isn't empty. An earlier restore finished, ran without `--single-transaction`, or was piped from a `pg_dump` that failed partway. Restore into an empty database instead: create one with `fly mpg databases create <CLUSTER_ID> --name <database-name>`, then add `--database <database-name>` before the `--` in the restore command. Pass the same flag to `fly mpg attach` when you attach your app.
- **`out of shared memory`, with a hint about `max_locks_per_transaction`:** The dump has more tables than one transaction can lock, so the restore was rolled back. Restore without `--single-transaction`, as in `fly mpg connect <CLUSTER_ID> -f dump.sql -- -q`. psql still stops at the first error, but what it restored before the error stays. If that happens, restore into a new, empty database before you try again.
- **Another error you'd rather skip:** Replace `--single-transaction` with `-v ON_ERROR_STOP=0`, as in `fly mpg connect <CLUSTER_ID> -f dump.sql -- -q -v ON_ERROR_STOP=0`. psql then carries on past errors, and the exit status is `0` even when statements fail, so read every error it prints. Only do this into an empty database: over a partial import, tables that already loaded get their rows twice.

### Piping a small database

For a small database you can skip the file and pipe `pg_dump` straight in. Pass `-f -` to read the dump from stdin. Set `pipefail` as well: without it, a `pg_dump` that fails looks like a restore that worked.

```bash
set -o pipefail
pg_dump --no-owner --no-acl "$LEGACY_PG_CONNECTION_STRING" | fly mpg connect <CLUSTER_ID> -f - -- -q --single-transaction
```

If step 3 applies to you, add its `sed` expression to the pipe, without the file names: `pg_dump ... | sed '1,20{/^SET transaction_timeout = 0;$/d;}' | fly mpg connect ...`.

If `pg_dump` fails partway, psql can commit what it received, which leaves part of your data imported even though the command fails. Restore into a new, empty database before you try again, as described under `already exists`. Prefer the file for anything large.

### Using another tool

Tools that make their own connection, like `pg_restore` with a custom-format dump, need an address to connect to. Run `fly mpg proxy <CLUSTER_ID>` and point the tool at the local port it prints, using the username and password from your cluster's connection string in the dashboard. See [Setting up a Proxy Connection](/postgres/create-and-connect#setting-up-a-proxy-connection).

For larger or more complex databases, you may need to break the dump and restore process into multiple steps. You may also consider using dedicated PG migration tools like [pgloader](https://pgloader.io/) or [pgcopydb](https://github.com/dimitri/pgcopydb).

Expand Down
4 changes: 2 additions & 2 deletions postgres/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,11 @@ You'll be able to access:

### Current limitations {#whats-not-there-yet}

- **Major version upgrades:** You choose Postgres 16 or 17 when you create a cluster, and you can't upgrade a cluster in place. To move to a newer major version, create a new cluster and copy your data with `pg_dump` and `pg_restore`. See [Import data](/postgres/import).
- **Major version upgrades:** You choose Postgres 16 or 17 when you create a cluster, and you can't upgrade a cluster in place. To move to a newer major version, create a new cluster and copy your data with `pg_dump` and psql. See [Import data](/postgres/import).
- **Minor versions and security patches:** New Postgres images include the latest minor release. We roll out new images during your cluster's [maintenance window](/postgres/maintenance-windows). v1 clusters, which have no **v2** badge next to the cluster name in the dashboard, don't get new images. PgBouncer, PostGIS and pg_repack versions are pinned and change only when we update them.
- **Third-party extensions:** `pgvector` and `PostGIS` are supported, and `pg_repack` on v2 clusters, along with the bundled Postgres extensions marked as available on the [Extensions page](/postgres/extensions). To request another extension, [contact support](/about/support).
- **Alerting:** We email your organization when maintenance starts and finishes, but there are no alerts for metrics like CPU or storage. Use [metrics](/postgres/metrics) to watch your cluster.
- **Migration tools:** There's no automated migration from another database. Use `pg_dump` and `pg_restore`, as described in [Import data](/postgres/import).
- **Migration tools:** There's no automated migration from another database. Use `pg_dump` and psql, as described in [Import data](/postgres/import).

## Regions

Expand Down
Loading