Skip to content

feat(appkit): apply committed database migrations on deploy start - #654

Draft
ditadi wants to merge 5 commits into
mainfrom
stack/database-dev-flow/01-migrations-on-deploy
Draft

ditadi wants to merge 5 commits into
mainfrom
stack/database-dev-flow/01-migrations-on-deploy

Conversation

@ditadi

@ditadi ditadi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR (generated by Stack)

Use this link to review incremental changes.


First implementation PR of the DatabasePlugin development flow stack. The design is #677. The plugin stays hidden from databricks apps init through the whole stack.

What did you change, and why?

A deployed app has to apply its committed migrations by itself, because App Builder apps have no human or CI to run them (design 001).

database() applies the drizzle migrations under config/database/migrations after the pool opens, before select 1 and the declared-schema check. It runs only when APPKIT_DB_MIGRATE_ON_START=true and the service principal's credentials are set, so a local run never migrates on start. The service principal runs the migrations and owns the app schema.

  • Each migration runs and is recorded in its own transaction. A failure keeps the migrations before it.
  • The journal is reconciled with the recorded rows. A migration older than the last recorded one that never ran fails startup and changes nothing. An edited migration, or a recorded row with no committed file, logs a warning.
  • A start with nothing pending takes no lock. Otherwise a per-schema advisory lock makes instances apply in turn. The wait is capped at 5 minutes and logs the holder's pid and application_name. Each table lock waits 5 s, with three tries per migration.
  • A dev-… or session-… branch is skipped on start. A -pooler host is refused, because the session lock does not hold through a transaction-mode pooler.
  • An unrecorded 0000_baseline whose tables and columns all exist is recorded as applied. A partial match fails startup.
  • Every failure names the step and the SQLSTATE. On 42501 the error also names the schema owner and the connecting role. Clients only see Database setup failed. One summary line per start logs the counts, the lock wait and the duration.

This revision changes two things:

  • Two journal entries that share a when each keep their own recorded row. Before, the entry that never ran read as edited and was never applied. Now it is pending or stranded.
  • knip.json declares three tools/ scripts and packages/appkit/bin/appkit.js as entry points. The pre-commit knip run had reported the bin, and the esbuild, ora and zod imports of those scripts, as unused.

Nothing migrates until an app sets APPKIT_DB_MIGRATE_ON_START=true, so this PR is safe to deploy alone. It adds no span and no metric for migrations.

How do you know it works?

  • apply.test.ts, history.test.ts and migrate-on-start.test.ts cover the transactions, each reconcile case including two entries with one when, the locks, the message for each failed step, the flag, the branch skip and the pooler refusal.
  • apply.pg.test.ts runs against Postgres when APPKIT_TEST_PG_URL is set, and CI sets it with a postgres:17 service. It covers the enum case that fails with 55P04 in one transaction, a stranded migration, three instances starting together, a schema owned by another role, and baseline adoption.
  • At this PR's head, the history tests passed, and a start against a local PostgreSQL 14 applied both migrations while a second start applied none. The full vitest suite, a clean build, the docs build and typecheck passed at the top of the stack (feat(appkit): add appkit db check #678), which contains this PR, and its packed tarballs export the same root and ./beta names as main. CI runs the full suite on this PR.
  • An earlier revision ran on a Lakebase project with a throwaway deploy branch and app. This revision was not run against a real Lakebase workspace.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.3 MB (+25 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.3 MB (+28 KB) 468 KB (+10 KB)
Type declarations 464 KB 168 KB (+2 B)
Source maps 2.6 MB (+51 KB) 873 KB (+17 KB)
Other 11 KB 3.7 KB
Total 4.3 MB (+80 KB) 1.5 MB (+28 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 106 KB (-1 B) 2.5 KB (+1 B) 108 KB external 343 KB
./beta 107 KB (+4.8 KB) 485 B 108 KB (+4.8 KB) external 322 KB (+13 KB)
./testing 45 KB (+1 B) 32 KB (-3 B) 77 KB (-2 B) external 223 KB
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 26 KB 0 B 26 KB external 74 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 101 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 87 KB
./beta modular.js initial 6.0 KB
./beta stream-manager.js initial 5.9 KB
./beta service-context.js initial 4.2 KB
./beta databricks.js initial 3.3 KB
./beta client.js initial 647 B
./beta index.js initial 20 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 178 B
./beta index.js lazy 115 B
./testing manifest.js initial 32 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 28 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB (+4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 429 KB 147 KB
Type declarations 267 KB 98 KB (+1 B)
Source maps 853 KB 286 KB
CSS 16 KB 3.2 KB
Total 1.5 MB 534 KB (+1 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 2.1 KB 0 B 2.1 KB 0 B 5.0 KB
./react 433 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 4.9 KB 0 B 4.9 KB 0 B 12 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 2.1 KB
./react index.js initial 431 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 4.9 KB

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 41109403947 -R databricks/appkit -n appkit-template-0.89.0-pr.16ec3ee-stack-database-dev-flow-01-migrations-on-deploy-654 -D appkit-pr-654 \
  && unzip -o "appkit-pr-654/appkit-template-0.89.0-pr.16ec3ee-stack-database-dev-flow-01-migrations-on-deploy-654.zip" -d "appkit-pr-654" \
  && databricks apps init --template "appkit-pr-654"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

A deployed DatabasePlugin app now applies the migrations committed under
config/database/migrations before its startup checks, as its own service
principal and under a per-schema advisory lock. Pending migrations run in
one transaction, and a table lock held for over 30s fails the migration
instead of stalling the table. A 0000_baseline whose tables already exist
is recorded, in its own transaction, instead of run. Any failure stops
startup, and the server log names it: the tags and SQLSTATE, or the
missing migration file. Local runs never migrate on start.

Signed-off-by: ditadi <victordperd@gmail.com>
- Run and record each pending migration in its own transaction. An enum value
  added by one migration can now be used by the next (55P04), and a failure
  keeps the migrations before it.
- Compare the committed journal with the recorded rows by hash and created_at.
  Fail closed on a migration older than the last recorded one; warn on edited
  migrations and on recorded rows with no committed file.
- Read the history without the lock first. Take the lock only when something
  is pending, then read again.
- Bound the wait for the migration lock to 5 minutes and name its holder. Set
  application_name and client_connection_check_interval. Lower lock_timeout to
  5s with three tries per migration.
- Migrate on start only with APPKIT_DB_MIGRATE_ON_START=true and service
  principal credentials. Skip dev- and session- branches; refuse a -pooler
  host.
- Name the failed step and the SQLSTATE or system code. On 42501, name the
  schema owner and the connecting role.
- Add real-Postgres tests gated on APPKIT_TEST_PG_URL; CI runs them against a
  postgres:17 service.

Signed-off-by: ditadi <victordperd@gmail.com>
@ditadi
ditadi force-pushed the stack/database-dev-flow/01-migrations-on-deploy branch from 5505de7 to 19adf06 Compare October 20, 2026 12:23
Signed-off-by: ditadi <victordperd@gmail.com>
knip.json ignores tools/**, so knip reported esbuild, ora and zod as unused even though
tools/bundle-size.ts, tools/generate-json-schema.ts and tools/playground/deploy-playground.ts
import them. It also reported packages/appkit/bin/appkit.js as an unused file. Declare these
as entry points so the pre-commit knip check passes without suppressing any finding.

Signed-off-by: ditadi <victordperd@gmail.com>
reconcileMigrations kept one recorded hash per created_at, so a committed migration whose
journal entry shares a when with an applied sibling was reported as edited and never ran.
Keep every recorded hash per when, and classify an entry whose sibling owns the recorded
row as pending or stranded.

Signed-off-by: ditadi <victordperd@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant