diff --git a/docs/docs/plugins/database.md b/docs/docs/plugins/database.md index a7efdeede..b514aec9c 100644 --- a/docs/docs/plugins/database.md +++ b/docs/docs/plugins/database.md @@ -21,10 +21,9 @@ not support OBO and does not apply per-user or per-row authorization. Every call who can reach the generated API can perform every enabled operation on every exposed row, including delete. -Restrict access to the app and grant its service principal only the database -permissions it needs. If users need different permissions or row ownership checks, -disable the relevant generated routes and implement authorization in custom -server routes. App admission alone does not provide row-level isolation. +Restrict access to the app. If users need different permissions or row ownership +checks, disable the relevant generated routes and implement authorization in +custom server routes. App admission alone does not provide row-level isolation. ::: ## Basic usage @@ -39,9 +38,11 @@ identifier-only messages for known missing-column or missing-table errors details, and hints are omitted because they may contain row values. Apps scaffolded with the Database plugin selected include an empty -`config/database/schema.ts`, so `database()` can start without requiring sample -tables. Replace the empty declaration with your models when their PostgreSQL -tables are ready. +`config/database/schema.ts` in the Postgres schema `app`, and set +`APPKIT_DB_MIGRATE_ON_START=true` in `app.yaml` so the deployed app applies its +migrations on start. Run `npx appkit db init` once to point `.env` at a branch of +your own, declare your tables, then run `npx appkit db migrate` to create them there +(see [Development workflow](#development-workflow)). For local development, the PostgreSQL username is resolved from your Databricks credentials when `PGUSER` and `DATABRICKS_CLIENT_ID` are absent. An explicitly @@ -112,7 +113,7 @@ Migrations applied on start into `public` also need `GRANT CREATE ON SCHEMA public` to the app's service principal, because it does not own `public`. -## Generate and apply migrations locally +## Development workflow :::note `appkit db` is hidden from `appkit --help` while the database development flow is @@ -120,16 +121,78 @@ in beta. Its logic ships in `@databricks/appkit/beta/database-tooling`, which only the AppKit CLI of the same release imports. It has no semver guarantee. ::: -`npx appkit db migrate` compares `config/database/schema.ts` with the latest -committed snapshot. When they differ, it writes -`config/database/migrations/NNNN_migration.sql`. It regenerates -`shared/appkit-types/database.d.ts`, then applies pending migrations to the -branch `.env` names. Commit the migrations with `schema.ts`. +You develop against a Lakebase branch of your own, never the deployed one. Tables +are created and changed through migrations: generated locally from `schema.ts`, +committed with your code, and applied by the deployed app when it starts. + +### Set up your branch + +Run `npx appkit db init` once after scaffolding, and again after cloning. Until +`databricks apps init` runs it for you, the scaffolded `.env` points at the branch +you picked in its prompts, usually the branch the app deploys to, and +`npx appkit db verify` warns about that. + +`db init` asks nothing: + +1. It keeps this checkout's session branch when `.env` points at one, and + otherwise uses your dev branch, `dev--`. It creates the branch when + it does not exist yet. `--session` uses a branch for this checkout that expires + after 7 days (`--ttl` changes that), and `--dev` goes back to your dev branch. +2. It makes sure the branch has an endpoint and your Postgres role, points `.env` + at the branch, and hands you the app schema the fork copied (see + [One owner per branch](#one-owner-per-branch)). +3. It sets up the schema. With `schema.ts` it runs `migrate`: it writes a + migration when `schema.ts` changed since the last one, then applies the + committed migrations. Without `schema.ts`, it writes `schema.ts` from the + branch's tables (`introspect`). + `--schema clean` drops the app schema first. Without a terminal, + `--schema clean` needs `--yes`. `--schema none` only sets up the branch and + `.env`: it reads no `schema.ts`, hands over no schema, and leaves the schema + and its owner as the fork copied them. +4. When that needs drizzle-kit and the app does not have it, it installs the + version AppKit pins as a dev dependency. `migrate` and `introspect` generate + migrations with it. + +`db init` never changes the project's default branch, a protected branch or a +branch `databricks.yml` deploys to, and never edits `databricks.yml`. + +:::caution[A new branch copies the default branch] +New branches fork the project's default branch, so they start with a copy of its +rows, production data included. Your role there is a member of +`DATABRICKS_SUPERUSER`, which reads and writes every table on the branch and +bypasses row-level security. Creating a branch needs CAN MANAGE on the Lakebase +project, which also allows deleting any branch, the deploy branch included. Use a +project whose default branch holds only data every developer may see. +::: -`migrate` applies migrations only to your dev branch or to this checkout's -session branch. A session branch name includes a hash of the user who created -it, so a `.env` copied from another developer does not make their session branch -yours. +### Commands + +| Command | What it does | +| --- | --- | +| `npx appkit db init` | Points `.env` at your dev branch, or with `--session` at a branch for this checkout that expires, then sets up the schema. | +| `npx appkit db migrate` | Writes `config/database/migrations/NNNN_migration.sql` when `schema.ts` changed, regenerates `shared/appkit-types/database.d.ts` from `schema.ts`, and applies pending migrations to your branch. | +| `npx appkit db introspect` | Writes `schema.ts` from tables that already exist, and a `0000_baseline` migration (or `NNNN_introspect` when migrations already exist). | +| `npx appkit db clean` | Drops the app schema on your branch; `migrate` then rebuilds it from the migrations. `--delete-branch` deletes your session branch instead. | +| `npx appkit db seed` | Runs `config/database/seed.sql` on your branch in one transaction. | +| `npx appkit db verify` | Read-only check of your branch: missing tables or columns, pending migrations, migrations that will never run there, and migrations it ran that this checkout does not commit. `--deployed` checks a target's deployed app instead. | + +`migrate`, `clean` and `seed` change only branches `db init` created for you (your +dev branch, or this checkout's session branch). A session branch name includes a +hash of the user who created it, so a `.env` copied from another developer does +not make their session branch yours. On any other branch, including the +project's default branch, a protected branch, or a branch `databricks.yml` +deploys to, `clean` and `seed` refuse, and `migrate` only writes the migration +files and `shared/appkit-types/database.d.ts`. + +A typical change: + +```bash +# edit config/database/schema.ts +npx appkit db migrate # writes and applies NNNN_migration.sql on your branch +git add config/database # commit schema.ts with its migrations +databricks apps deploy # the deployed app applies it on start +npx appkit db verify --deployed # the app runs, with every committed migration +``` ### Branches `databricks.yml` deploys to @@ -174,6 +237,132 @@ checkout, delete `config/database/.appkit-db.lock` and run the command again. AppKit never deletes a lock it does not hold. Scaffolded apps list the file in `.gitignore`. +### Changing tables in use + +While a new release migrates on start, instances of the previous release keep +serving, so every migration must work with the code that is still running. A +change that drops or renames a table or column, changes a column's type, or makes +a column required breaks them, and `db migrate` warns when it writes one. Split such a +change over two deploys. First deploy code that works with both the old and the +new schema, with a migration that only adds. Then, in a later deploy, drop what +the old code used. + +### Migrations on deploy + +When the deployed app starts with `APPKIT_DB_MIGRATE_ON_START=true` in `app.yaml`, +`database()` applies the committed migrations the deploy branch has not recorded +yet, before it checks the tables and serves requests. Apps scaffolded with the +plugin set it; add it to the `env` of an app that predates it: + +```yaml +env: + - name: APPKIT_DB_MIGRATE_ON_START + value: "true" +``` + +Migrations run as the app's service principal, one instance at a time, each in +its own transaction. If one fails, it is rolled back, the ones before it stay +applied, and the app does not start. The server log names the migration and the +SQLSTATE, or the missing migration file. An instance waits up to 5 minutes for +another one to finish migrating. A migration that waits more than 5 seconds for a +table lock, for example behind a long transaction of the running app, is rolled +back and retried twice, then fails instead of blocking that table. + +`.env` never sets the flag, so a local start (`npm run dev`, `npm start`) never +migrates: use `npx appkit db migrate`. An app pointed at a dev or session branch +does not migrate it on start either. + +A `0000_baseline` from `introspect` is recorded without running when its tables +already exist on the deploy branch and no migration was ever applied there. If +only some of them exist, the app does not start, and nothing is changed. + +### Existing apps + +- **Tables created by hand or by earlier code.** Without `schema.ts`, + `npx appkit db init` creates your dev branch from the project's default branch and + introspects it: it writes `schema.ts` from the tables there and a `0000_baseline` + migration. With a `schema.ts` already, run `npx appkit db introspect` on your + branch. Commit both files: the deployed app records the baseline without running + it, because the tables already exist on the deploy branch. +- **Tables in `public`.** Earlier releases used `public` by default. Keep them there + with `schemaName: "public"`, as [Apps with tables in `public`](#apps-with-tables-in-public) + shows. +- **No migrate-on-start flag.** Add `APPKIT_DB_MIGRATE_ON_START=true` to `app.yaml`, + as [Migrations on deploy](#migrations-on-deploy) shows. Without it, the app checks + its tables on start but creates none. + +### Rolling back + +Migrations only go forward. There are no down migrations, and the app never undoes +one. Redeploying an older commit leaves the newer tables and their recorded +migrations in place: the app logs that the branch recorded migrations the older +code does not have, applies nothing, and starts. The older code must work with the +newer schema, which [two-step changes](#changing-tables-in-use) guarantee. To undo +a schema change, commit a new migration that reverses it, and deploy it. + +### One owner per branch + +Each branch has one owner of the app schema, and only the owner can change its +tables: + +- On the deploy branch, the app's service principal creates schema `app` on its + first start and owns everything in it. The `postgres` resource's + `CAN_CONNECT_AND_CREATE` permission is all it needs; no grants are required. +- On your dev or session branch, you own it. A fork copies its parent's objects + with their owners, so `db init` hands the app schema to you: it deletes the + branch's copy of the role that owns it, with its objects reassigned to you. + `db init --schema none` hands over nothing. Schema `public` is never handed + over, because it belongs to the database rather than to an app. + +Do not apply migrations to the deploy branch yourself: you would own those objects, +and the app could no longer change them. If schema `app` on the deploy branch +already belongs to another role, the app cannot migrate it, and the server log +names both roles. Use another `schemaName` in `defineSchema()`, or have the owner +transfer the schema. + +An app that was deleted and created again has a new service principal, and the old +one still owns the schema. Someone with CAN MANAGE on the Lakebase project moves +the objects to the new one, which deletes the old role: + +```bash +# Both service principals' roles, by their client IDs +databricks postgres list-roles projects//branches/ +databricks postgres delete-role projects//branches//roles/ \ + --reassign-owned-to projects//branches//roles/ +``` + +### Verify the deployed app + +`npx appkit db verify --deployed` checks a `databricks.yml` target's app through +the Databricks API: the app exists, its `postgres` resource names the target's +branch and database with `CAN_CONNECT_AND_CREATE`, its service principal has a role +on the branch, and the app is running. When the checkout has +`.databricks/bundle//variable-overrides.json`, the branch and database +come from it, as a deploy from this checkout uses them. It also reads the source +the app's active deployment runs: every committed migration must be there with +the same journal entry and SQL, and the deployment must have no migration this +checkout does not commit. When verify cannot read that source, it fails and +says why. +`-t, --target` picks the target; without it, verify uses the bundle's default +target. When the target's `workspace.host` is not the workspace your shell is +authenticated against, verify refuses and names the profile to set. It needs no +access to the deploy branch's database and changes nothing. A failed check names +`databricks apps logs`, `start` or `deploy`. + +### CI + +`appkit db init` sets up branches for developers, and CI does not need one. After +a deploy, check the deployed app with workspace credentials: + +```bash +databricks apps deploy -t prod +npx appkit db verify --deployed -t prod +``` + +It exits 1 when the app is not running or is bound to another branch, when +verify cannot confirm that its deployment has every committed migration with the +same SQL, and when the deployment has a migration this checkout does not commit. + ## Schema discovery and overrides `database()` and `database({})` use the same defaults. During setup, the plugin diff --git a/packages/appkit/src/database/migrations/ownership.ts b/packages/appkit/src/database/migrations/ownership.ts index 59a7be8d4..6f0951da1 100644 --- a/packages/appkit/src/database/migrations/ownership.ts +++ b/packages/appkit/src/database/migrations/ownership.ts @@ -63,8 +63,9 @@ export function ownershipHint( return ( `Schema "${ownership.schemaName}" is owned by "${ownership.owner}", but the app connects as "${ownership.currentUser}". ` + "This happens when the app's database resource was removed and added again, or the app was recreated with a new service principal: " + - `Lakebase hands the objects to another role. Have "${ownership.owner}" transfer the schema and its tables to "${ownership.currentUser}" ` + - "(ALTER … OWNER TO), or set another schemaName in schema.ts. " + + `Lakebase hands the objects to another role. If "${ownership.owner}" belonged to a deleted app, someone with CAN MANAGE on the Lakebase project moves its objects with ` + + `\`databricks postgres delete-role --reassign-owned-to \` (\`databricks postgres list-roles \` names both), which also deletes its role. ` + + `Otherwise have "${ownership.owner}" transfer the schema and its tables to "${ownership.currentUser}" (ALTER … OWNER TO), or set another schemaName in schema.ts. ` + "On your dev or session branch, rerun `appkit db init`, which hands you the schema." ); } diff --git a/packages/appkit/src/database/migrations/tests/ownership.test.ts b/packages/appkit/src/database/migrations/tests/ownership.test.ts index d1dda3838..c5977aa39 100644 --- a/packages/appkit/src/database/migrations/tests/ownership.test.ts +++ b/packages/appkit/src/database/migrations/tests/ownership.test.ts @@ -25,6 +25,9 @@ describe("ownership: one owner per branch", () => { expect(hint).toContain( "the app's database resource was removed and added again, or the app was recreated", ); + expect(hint).toContain( + `If "old-app-sp" belonged to a deleted app, someone with CAN MANAGE on the Lakebase project moves its objects with \`databricks postgres delete-role --reassign-owned-to \``, + ); expect(hint).toContain("set another schemaName in schema.ts"); expect(hint).toContain( "On your dev or session branch, rerun `appkit db init`, which hands you the schema.", diff --git a/packages/appkit/src/database/tooling/tests/verify.test.ts b/packages/appkit/src/database/tooling/tests/verify.test.ts index f79a12b9a..d0532c944 100644 --- a/packages/appkit/src/database/tooling/tests/verify.test.ts +++ b/packages/appkit/src/database/tooling/tests/verify.test.ts @@ -5,12 +5,19 @@ import path from "node:path"; import { afterEach, beforeEach, describe, expect, test, vi } from "vitest"; -const { withBranchPool } = vi.hoisted(() => ({ withBranchPool: vi.fn() })); +const { withBranchPool, resolveEnvBranch } = vi.hoisted(() => ({ + withBranchPool: vi.fn(), + resolveEnvBranch: vi.fn(), +})); vi.mock("../context", async (importOriginal) => ({ ...(await importOriginal()), withBranchPool, })); +vi.mock("../guard", async (importOriginal) => ({ + ...(await importOriginal()), + resolveEnvBranch, +})); import type { WorkspaceRequest } from "../../../workspace-client"; import type { DbContext } from "../context"; @@ -153,8 +160,15 @@ function branch(state: { const WRITES = /\b(insert|update|delete|create|drop|alter|grant|truncate)\b/i; +const RESOLVED = { branches: new Set(), unresolved: [] }; + beforeEach(() => { withBranchPool.mockReset(); + resolveEnvBranch.mockReset().mockResolvedValue({ + branch: { name: BRANCH }, + boundary: RESOLVED, + kind: "dev", + }); }); afterEach(() => { @@ -169,6 +183,8 @@ describe("runVerify", () => { { tag: "0001_migration", when: 2000 }, ]; const clean = { + notYours: null, + refusesEveryBranch: null, schemaProblems: [], pendingMigrations: [], strandedMigrations: [], @@ -270,6 +286,74 @@ describe("runVerify", () => { }); }); + test("a branch that is not your dev or session branch is named, and still verified", async () => { + const ctx = app(two); + branch({ + columns: { notes: ["id", "body"] }, + recorded: recordedAs(ctx, two), + }); + resolveEnvBranch.mockResolvedValue({ + branch: { name: "projects/p/branches/production" }, + boundary: RESOLVED, + kind: "default", + reason: "it is the project's default branch", + }); + + expect(await runVerify(ctx)).toMatchObject({ + branch: "projects/p/branches/production", + notYours: "it is the project's default branch", + refusesEveryBranch: null, + ok: true, + }); + }); + + test.each([ + [ + "your dev branch", + { + kind: "deploy-target", + reason: + "databricks.yml may deploy to it through a reference this checkout's files do not resolve", + }, + ], + [ + "the default branch", + { kind: "default", reason: "it is the project's default branch" }, + ], + ] as const)( + "a reference databricks.yml leaves unresolved refuses every branch, and the report names it, on %s", + async (_label, classification) => { + const ctx = app(two); + branch({ + columns: { notes: ["id", "body"] }, + recorded: recordedAs(ctx, two), + }); + resolveEnvBranch.mockResolvedValue({ + branch: { name: BRANCH }, + boundary: { + branches: new Set(), + unresolved: [ + { + target: "default", + at: "variables.postgres_branch", + written: "", + file: ".databricks/bundle/default/variable-overrides.json", + }, + ], + }, + ...classification, + }); + + expect(await runVerify(ctx)).toMatchObject({ + branch: BRANCH, + notYours: classification.reason, + refusesEveryBranch: + "The files do not resolve variables.postgres_branch in target default with .databricks/bundle/default/variable-overrides.json (no value), so `appkit db migrate` applies migrations to no branch, and `init`, `introspect`, `clean` and `seed` refuse every branch. Give it a value in databricks.yml or the variable-overrides.json named", + ok: true, + }); + }, + ); + test("no committed migrations: only the schema is checked", async () => { const ctx = app(); branch({ columns: { notes: ["id", "body"] }, recorded: "absent" }); @@ -307,6 +391,7 @@ describe("runVerify", () => { }); test("without a configured branch it points to init before connecting", async () => { + resolveEnvBranch.mockResolvedValue(null); await expect(runVerify(app([], {}))).rejects.toThrow( "Did not verify: no branch is configured in .env (PGHOST, PGDATABASE and LAKEBASE_ENDPOINT); run `appkit db init`.", ); @@ -509,6 +594,7 @@ describe("runVerify --deployed", () => { `workspace.export ${SNAPSHOT}/${MIGRATIONS}/0000_migration.sql`, ]); expect(withBranchPool).not.toHaveBeenCalled(); + expect(resolveEnvBranch).not.toHaveBeenCalled(); expect(lakebase.calls.sort()).toEqual([ `getBranch ${PRODUCTION}`, `getDatabase ${DATABASE}`, diff --git a/packages/appkit/src/database/tooling/verify.ts b/packages/appkit/src/database/tooling/verify.ts index 4fb817927..8ed4c840a 100644 --- a/packages/appkit/src/database/tooling/verify.ts +++ b/packages/appkit/src/database/tooling/verify.ts @@ -16,17 +16,12 @@ import { readJournal, } from "../migrations/paths"; import type { DataPath } from "../runtime"; -import { - type DbContext, - hasEnvBranch, - NO_ENV_BRANCH, - withBranchPool, -} from "./context"; +import { type DbContext, NO_ENV_BRANCH, withBranchPool } from "./context"; +import { everyBranchRefusal, resolveEnvBranch, takesWrites } from "./guard"; import { requireSchemaFile } from "./migrate"; import { type BundleTarget, defaultBundleTarget, - parseResourceName, readBundle, resourceId, } from "./targets"; @@ -42,6 +37,16 @@ export interface VerifyOptions { interface LocalVerifyReport { readonly kind: "local"; readonly branch: string; + /** Why the branch is not your dev or session branch, e.g. "it is the project's default branch"; null when it is. */ + readonly notYours: string | null; + /** + * While databricks.yml has a reference this checkout's files do not + * resolve, `migrate` applies migrations to no branch, and `init`, + * `introspect`, `clean` and `seed` refuse every branch: this says so and + * names the references and where to give them values. null when every + * reference resolves. + */ + readonly refusesEveryBranch: string | null; /** Declared tables or columns the branch lacks (the plugin's startup check). */ readonly schemaProblems: readonly string[]; /** Committed migrations the branch has not run yet, newer than every one it ran. */ @@ -111,10 +116,9 @@ function rawOn(client: Pick): Pick { /** Read-only, as the developer, on the `.env` branch: one READ ONLY transaction, rolled back. */ async function verifyLocal(ctx: DbContext): Promise { requireSchemaFile(ctx.root, "verify"); - const branch = parseResourceName(ctx.env.LAKEBASE_ENDPOINT)?.branch; - if (!branch || !hasEnvBranch(ctx.env)) { - throw new Error(`Did not verify: ${NO_ENV_BRANCH}.`); - } + const env = await resolveEnvBranch(ctx); + if (!env) throw new Error(`Did not verify: ${NO_ENV_BRANCH}.`); + const branch = env.branch.name; const schema = await loadDefaultDatabaseSchema(ctx.root, { showImportError: true, }); @@ -130,6 +134,8 @@ async function verifyLocal(ctx: DbContext): Promise { return { kind: "local", branch, + notYours: takesWrites(env.kind) ? null : (env.reason ?? null), + refusesEveryBranch: everyBranchRefusal(env.boundary), schemaProblems, pendingMigrations: drift.pending, strandedMigrations: drift.stranded, diff --git a/packages/appkit/src/plugins/database/manifest.json b/packages/appkit/src/plugins/database/manifest.json index 111a7b6be..cc17d6a1b 100644 --- a/packages/appkit/src/plugins/database/manifest.json +++ b/packages/appkit/src/plugins/database/manifest.json @@ -17,6 +17,7 @@ "project": { "description": "Lakebase project resource name", "examples": ["projects/{project-id}"], + "bundleIgnore": true, "discovery": { "type": "kind", "resourceKind": "postgres_project", diff --git a/packages/shared/src/cli/commands/db/output.ts b/packages/shared/src/cli/commands/db/output.ts index ca814d8ad..5537627a3 100644 --- a/packages/shared/src/cli/commands/db/output.ts +++ b/packages/shared/src/cli/commands/db/output.ts @@ -119,6 +119,13 @@ function localVerifyHint(report: LocalVerifyReport): string { function printLocalVerifyReport(report: LocalVerifyReport): void { const branch = shortName(report.branch); + if (report.notYours) { + warn( + report.refusesEveryBranch + ? `${branch} takes no writes (${report.notYours}). ${report.refusesEveryBranch}.` + : `.env points at ${branch}, not a branch \`appkit db init\` set up for you: ${report.notYours}. \`npx appkit db init\` points it at a branch of your own.`, + ); + } if (report.ok) { ok( `${branch} has every table and column config/database/schema.ts declares, and every committed migration`, diff --git a/packages/shared/src/cli/commands/db/tooling-contract.ts b/packages/shared/src/cli/commands/db/tooling-contract.ts index 3c120d125..4973b3e78 100644 --- a/packages/shared/src/cli/commands/db/tooling-contract.ts +++ b/packages/shared/src/cli/commands/db/tooling-contract.ts @@ -75,6 +75,8 @@ export interface VerifyOptions { export interface LocalVerifyReport { readonly kind: "local"; readonly branch: string; + readonly notYours: string | null; + readonly refusesEveryBranch: string | null; readonly schemaProblems: readonly string[]; readonly pendingMigrations: readonly string[]; readonly strandedMigrations: readonly string[]; diff --git a/packages/shared/src/cli/commands/db/verify.test.ts b/packages/shared/src/cli/commands/db/verify.test.ts index e5436e0a0..6c0fa057c 100644 --- a/packages/shared/src/cli/commands/db/verify.test.ts +++ b/packages/shared/src/cli/commands/db/verify.test.ts @@ -32,6 +32,8 @@ function output(spy: MockInstance): string { const local = { kind: "local" as const, branch: "projects/p/branches/dev-alice-3f9a2c", + notYours: null, + refusesEveryBranch: null, schemaProblems: [], pendingMigrations: [], strandedMigrations: [], @@ -87,6 +89,54 @@ describe("appkit db verify", () => { expect(process.exitCode).toBeUndefined(); }); + test("a .env branch that is not yours is named, with init as the way to your own", async () => { + runVerify.mockResolvedValue({ + ...local, + branch: "projects/p/branches/production", + notYours: "it is the project's default branch", + }); + + await dbCommand.parseAsync(["verify"], { from: "user" }); + + expect(output(log)).toContain( + ".env points at production, not a branch `appkit db init` set up for you: it is the project's default branch. `npx appkit db init` points it at a branch of your own.", + ); + expect(process.exitCode).toBeUndefined(); + }); + + test.each([ + [ + "dev-alice-3f9a2c", + "projects/p/branches/dev-alice-3f9a2c", + "databricks.yml may deploy to it through a reference this checkout's files do not resolve", + ], + [ + "production", + "projects/p/branches/production", + "it is the project's default branch", + ], + ])( + "while databricks.yml leaves a reference unresolved, the warning on %s names it and gives no init advice", + async (shown, branch, notYours) => { + const refusesEveryBranch = + "The files do not resolve variables.postgres_branch in target default with .databricks/bundle/default/variable-overrides.json (no value), so `appkit db migrate` applies migrations to no branch, and `init`, `introspect`, `clean` and `seed` refuse every branch. Give it a value in databricks.yml or the variable-overrides.json named"; + runVerify.mockResolvedValue({ + ...local, + branch, + notYours, + refusesEveryBranch, + }); + + await dbCommand.parseAsync(["verify"], { from: "user" }); + + expect(output(log)).toContain( + `${shown} takes no writes (${notYours}). ${refusesEveryBranch}.`, + ); + expect(output(log)).not.toContain("npx appkit db init"); + expect(process.exitCode).toBeUndefined(); + }, + ); + test("prints every schema problem and pending migration, and exits 1", async () => { runVerify.mockResolvedValue({ ...local, diff --git a/template/README.md.tmpl b/template/README.md.tmpl index 5d3f9ce20..32b763e8b 100644 --- a/template/README.md.tmpl +++ b/template/README.md.tmpl @@ -49,13 +49,38 @@ The Lakebase plugin requires additional environment variables for PostgreSQL con {{- if .plugins.database}} -#### Database schema +#### Database -`database()` loads `config/database/schema.ts`. The generated schema starts empty, -so initialization does not require sample tables or expose existing data. -Add your table declarations after creating the corresponding PostgreSQL tables; -AppKit does not create or migrate them. CRUD routes are enabled for declared tables -by default, and can be restricted with `api` in the plugin configuration. +`database()` loads `config/database/schema.ts`, whose tables live in the Postgres +schema `app`. CRUD routes are enabled for declared tables by default, and can be +restricted with `api` in the plugin configuration. + +Local development runs against a Lakebase branch of your own, never the deployed one. +Run `npx appkit db init` once after scaffolding, and again after cloning. It asks +nothing: it points `.env` at your branch and sets up its schema. + +```bash +npx appkit db init # your dev branch, forked from the project's default branch +npx appkit db init --session # a branch for this checkout that expires after 7 days +npx appkit db init --dev # back to your dev branch +``` + +A new branch starts with a copy of the default branch's rows, production data included. + +To change the schema, edit `config/database/schema.ts`, then: + +```bash +npx appkit db migrate # writes config/database/migrations/NNNN_migration.sql and applies it to your branch +npx appkit db verify # read-only check of your branch against schema.ts and the migrations +``` + +Commit the migrations. The deployed app applies them on start, because `app.yaml` +sets `APPKIT_DB_MIGRATE_ON_START=true`. To adopt tables that already exist, run +`npx appkit db introspect`: it writes `schema.ts` from the branch's tables, and a +`0000_baseline` migration (or `NNNN_introspect` when migrations already exist). +`npx appkit db clean` drops the app schema on your branch, and `npx appkit db seed` +runs `config/database/seed.sql` on it; both refuse any branch `db init` did not +create for you, such as the deployed one. {{- end}} ### CLI Authentication @@ -183,6 +208,14 @@ databricks apps deploy ```bash databricks apps deploy -t prod ``` +{{- if .plugins.database}} + +Then check that the deployed app runs with every committed migration: + +```bash +npx appkit db verify --deployed -t prod +``` +{{- end}} > **Restarting a stopped app:** apps stop after a period of inactivity. To start one again without redeploying, run `databricks apps start `. @@ -196,10 +229,15 @@ databricks apps deploy -t prod * server.ts # Server entry point * routes/ # Routes * shared/ # Shared types -{{- if .plugins.analytics}} +{{- if or .plugins.analytics .plugins.database}} * config/ # Configuration +{{- end}} +{{- if .plugins.analytics}} * queries/ # SQL query files {{- end}} +{{- if .plugins.database}} + * database/ # schema.ts and its migrations +{{- end}} * databricks.yml # Bundle configuration * app.yaml # App configuration * .env.example # Environment variables example diff --git a/template/_gitignore b/template/_gitignore index f0faf4e16..d5169910f 100644 --- a/template/_gitignore +++ b/template/_gitignore @@ -9,6 +9,9 @@ build/ test-results/ playwright-report/ +# `appkit db introspect` keeps the previous schema.ts here +config/database/*.bak + # `appkit db` holds it while a command changes the migration history config/database/.appkit-db.lock diff --git a/template/app.yaml.tmpl b/template/app.yaml.tmpl index 077ea14c1..e85391b63 100644 --- a/template/app.yaml.tmpl +++ b/template/app.yaml.tmpl @@ -5,3 +5,8 @@ health_check: env: {{.appEnv}} {{- end}} +{{- if .plugins.database}} +{{- /* The plugin's postgres resource always renders LAKEBASE_ENDPOINT into .appEnv, so env: is open. */}} + - name: APPKIT_DB_MIGRATE_ON_START + value: "true" +{{- end}} diff --git a/template/config/database/schema.ts b/template/config/database/schema.ts index c0f6dc1e3..19163eae6 100644 --- a/template/config/database/schema.ts +++ b/template/config/database/schema.ts @@ -1,7 +1,8 @@ {{if .plugins.database -}} import { defineSchema } from '@databricks/appkit/beta'; -// database() discovers this file automatically. Start with no exposed tables. -// Add declarations here after creating the corresponding PostgreSQL tables. +// database() discovers this file automatically. Edit tables here, then run +// `npx appkit db migrate` on your dev branch. The deployed app's service principal +// owns schema "app" and applies committed migrations on start. export const schema = defineSchema(() => ({})); {{- end}} diff --git a/template/server/server.ts b/template/server/server.ts index b33bb94d8..14af3fcca 100644 --- a/template/server/server.ts +++ b/template/server/server.ts @@ -27,4 +27,7 @@ createApp({ await setupSampleLakebaseRoutes(appkit); }, {{- end}} -}).catch(console.error); +}).catch((error) => { + console.error(error); + process.exit(1); +});