Skip to content

Migrate pop RawQuery call sites to sqlc (proof of concept) - #2799

Open
hf wants to merge 1 commit into
masterfrom
claude/pop-to-sqlc-migration-83a585
Open

Migrate pop RawQuery call sites to sqlc (proof of concept)#2799
hf wants to merge 1 commit into
masterfrom
claude/pop-to-sqlc-migration-83a585

Conversation

@hf

@hf hf commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Why

We're evaluating moving off gobuffalo/pop and onto sqlc for query generation. RawQuery call sites are the natural starting point: they're already hand-written SQL, so porting them doesn't require redesigning anything (unlike pop's Eager-loaded associations, which have no sqlc equivalent and need real query redesign). This PR is that proof of concept — every pop.RawQuery call in internal/models for sessions, refresh tokens, recovery codes, and the identity provider_id update now runs through sqlc-generated queries instead.

LOC honesty check: this diff is ~1550 lines, but only ~400 of them are hand-written and worth reviewing carefully (the model files, storage/dial.go, the new internal/models/sqlc.go bridge, the .sql query files, sqlc.yaml, and the Makefile/.gitignore additions). The remaining ~1150 lines are 100% sqlc-generated (internal/storage/dbsql/sqlcgen/, marked DO NOT EDIT) — that's the bulk of the diff and isn't something anyone needs to read line by line, only trust the tool for.

The schema problem

sqlc needs to see table schemas to type its generated queries. Feeding it this repo's raw migration files doesn't work: most of them wrap their DDL in do $$ ... end $$ blocks for idempotency (create table if not exists, exception handlers, etc.), and sqlc's static parser can't see into a DO block — it just reports the tables as never having been created. sqlc also doesn't support pointing schema: at a live database connection string despite that config shape looking like it should work.

The fix: make sqlc-generate brings up the dev postgres, runs the real migrations against it, then pg_dump --schema-onlys the result into internal/storage/dbsql/schema/schema.sql, which sqlc reads. That dump is a derived artifact of the migrations, not a second source of truth, so it's gitignored and regenerated on demand rather than committed.

The harder problem: sharing a connection between pop and sqlc

sqlc's generated code wants a database/sql-shaped executor (ExecContext/QueryContext/QueryRowContext/PrepareContext). A freshly-dialed pop.Connection satisfies that directly — its Store is a concrete *pop.dB wrapping *sqlx.DB, and Go promotes the full method set through concrete struct embedding.

But that's not the connection real code actually uses. Every connection that has been through .WithContext(ctx) or .Transaction(...) — which is to say, every request handler and every transactional model function — has pop rewrap Store in its private contextStore type. contextStore embeds the store type as an interface, not a concrete struct, and Go only promotes an embedded interface's own declared method set. That interface doesn't declare QueryContext/QueryRowContext/PrepareContext (pop talks to sqlx in its own vocabulary internally), so a naive type assertion silently fails on any wrapped connection.

This didn't show up as a compile error or an obvious runtime error — it showed up as a 10-minute test hang. TestReplaceRecoveryCodesSerializesConcurrentReplacements spawns two goroutines that both call ReplaceRecoveryCodes inside db.Transaction(...); the first one's call failed the (silent) executor lookup, returned an error before signaling a channel, and the main goroutine sat parked on that channel forever. There's actually a comment in this codebase's existing popConnToStd acknowledging this exact wall ("I couldn't find any way to obtain the store when wrapped with context due to the private store field") — it just never mattered before, because that helper only ever runs once, on a freshly-dialed connection, at startup.

storage.Connection.SQLExecutor() (in internal/storage/dial.go) solves it with a bounded, recover()-guarded reflection unwrap that reaches past contextStore's unexported field to get back to the concrete *pop.dB/*pop.Tx underneath — which, being concrete, does promote its full method set. This is the piece that makes it safe for sqlc queries to share a live pop transaction, which is what makes atomic multi-statement operations like ReplaceRecoveryCodes (lock + delete + insert + reset, all-or-nothing) actually work correctly.

What's next

If we like this direction, the remaining RawQuery sites (webauthn challenges/credentials, factors, AMR claims, OAuth consents/authorizations, OAuth client state, a couple of test-only helpers with genuinely dynamic SQL) are mechanical follow-ups using the same pattern established here. The bigger open question for a fuller pop migration is .Eager()-loaded associations, which have no sqlc equivalent and need real query redesign rather than a straight port.

Test plan

  • go build ./...
  • go test ./internal/models/... — full suite passes (was hanging indefinitely before the SQLExecutor fix)
  • go test ./internal/storage/... — full suite passes
  • make sqlc-generate verified end-to-end against a freshly migrated dev postgres

🤖 Generated with Claude Code

Ports the hand-written pop.RawQuery call sites in internal/models (sessions,
refresh tokens, recovery codes, the identity provider_id update) to
sqlc-generated queries, as a proof of concept for a broader pop -> sqlc
migration.

sqlc's schema source can't be the raw migration files as-is: most of this
repo's migrations wrap their DDL in `do $$ ... end $$` blocks for
idempotency, which sqlc's static parser can't see into. Schema is instead
pg_dump'd from a live, migrated dev database (`make sqlc-generate`); the
dump is gitignored and regenerated on demand, never committed.

Bridging pop and sqlc onto the same connection/transaction turned out to be
the hard part. sqlc's generated code wants a database/sql-shaped executor
(ExecContext/QueryContext/QueryRowContext/PrepareContext). A plain, just-dialed
pop.Connection satisfies that directly, since its Store is a concrete
*pop.dB wrapping *sqlx.DB, and Go promotes the full method set through
concrete embedding. But every connection that has passed through
WithContext or Transaction -- i.e. virtually every real request path --
has pop rewrap Store in its unexported contextStore type, which embeds
the narrow `store` *interface* instead. Go only promotes an embedded
interface's own declared method set, and that interface doesn't include
Query/QueryRow/Prepare. The assertion then fails silently, and a test
exercising concurrent transactional recovery-code replacement hung for
10 minutes before that showed up: the failing goroutine returned an error
before signaling a channel the main goroutine was waiting on forever.

storage.Connection.SQLExecutor() fixes this by reaching past contextStore's
unexported field with a bounded, recover()-guarded reflection unwrap to get
back to the concrete *pop.dB/*pop.Tx underneath, which does promote fully.
This is the same wall a prior comment on popConnToStd already flagged as
unsolved ("I couldn't find any way to obtain the store when wrapped with
context") -- it just never mattered before because that helper only ever
ran once, on a fresh connection, at dial time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@hf
hf requested a review from a team as a code owner September 11, 2026 17:55
SELECT id FROM auth.refresh_tokens WHERE token = $1 LIMIT 1 FOR UPDATE SKIP LOCKED;

-- name: RevokeTokenFamilyBySessionID :exec
UPDATE auth.refresh_tokens SET revoked = true, updated_at = now() WHERE session_id = $1 AND revoked = false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: LOW

While the change appears deliberate, this query hard-codes auth even though the supported DB_NAMESPACE setting relocates these tables and Pop uses the configured search path. In a non-auth deployment, a replayed stolen refresh token reaches the wrong revocation table, leaving its configured token family unrevoked and weakening reuse containment.
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: Remove the hardcoded auth. schema prefix from all SQL query files and instead rely on the PostgreSQL search_path setting, which should be configured at connection time to use the value of DB_NAMESPACE (defaulting to auth). Concretely:

  1. In the connection setup (e.g., internal/storage/dial.go), after opening the pop connection, execute SET search_path = '<namespace>' using the configured config.DB.Namespace value (e.g., via db.RawQuery("SET search_path = " + config.DB.Namespace).Exec() or by appending search_path=<namespace> to the DSN's connection options).

  2. Remove the auth. prefix from all queries across all four .sql files (refresh_tokens.sql, sessions.sql, recovery_codes.sql, identities.sql) — there are 20 occurrences in total. For example, line 5 of refresh_tokens.sql should become:
    UPDATE refresh_tokens SET revoked = true, updated_at = now() WHERE session_id = $1 AND revoked = false;

  3. After modifying the query files, re-run make sqlc-generate to regenerate the sqlc output in internal/storage/dbsql/sqlcgen/.

This ensures that every sqlc query resolves tables against the namespace configured via DB_NAMESPACE, so token-family revocation works correctly in non-default deployments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant