Migrate pop RawQuery call sites to sqlc (proof of concept) - #2799
Conversation
Ports the hand-written pop.RawQuery call sites in internal/models (sessions,
refresh tokens, recovery codes, the identity provider_id update) to
sqlc-generated queries, as a proof of concept for a broader pop -> sqlc
migration.
sqlc's schema source can't be the raw migration files as-is: most of this
repo's migrations wrap their DDL in `do $$ ... end $$` blocks for
idempotency, which sqlc's static parser can't see into. Schema is instead
pg_dump'd from a live, migrated dev database (`make sqlc-generate`); the
dump is gitignored and regenerated on demand, never committed.
Bridging pop and sqlc onto the same connection/transaction turned out to be
the hard part. sqlc's generated code wants a database/sql-shaped executor
(ExecContext/QueryContext/QueryRowContext/PrepareContext). A plain, just-dialed
pop.Connection satisfies that directly, since its Store is a concrete
*pop.dB wrapping *sqlx.DB, and Go promotes the full method set through
concrete embedding. But every connection that has passed through
WithContext or Transaction -- i.e. virtually every real request path --
has pop rewrap Store in its unexported contextStore type, which embeds
the narrow `store` *interface* instead. Go only promotes an embedded
interface's own declared method set, and that interface doesn't include
Query/QueryRow/Prepare. The assertion then fails silently, and a test
exercising concurrent transactional recovery-code replacement hung for
10 minutes before that showed up: the failing goroutine returned an error
before signaling a channel the main goroutine was waiting on forever.
storage.Connection.SQLExecutor() fixes this by reaching past contextStore's
unexported field with a bounded, recover()-guarded reflection unwrap to get
back to the concrete *pop.dB/*pop.Tx underneath, which does promote fully.
This is the same wall a prior comment on popConnToStd already flagged as
unsolved ("I couldn't find any way to obtain the store when wrapped with
context") -- it just never mattered before because that helper only ever
ran once, on a fresh connection, at dial time.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
| SELECT id FROM auth.refresh_tokens WHERE token = $1 LIMIT 1 FOR UPDATE SKIP LOCKED; | ||
|
|
||
| -- name: RevokeTokenFamilyBySessionID :exec | ||
| UPDATE auth.refresh_tokens SET revoked = true, updated_at = now() WHERE session_id = $1 AND revoked = false; |
There was a problem hiding this comment.
⚪ Severity: LOW
While the change appears deliberate, this query hard-codes auth even though the supported DB_NAMESPACE setting relocates these tables and Pop uses the configured search path. In a non-auth deployment, a replayed stolen refresh token reaches the wrong revocation table, leaving its configured token family unrevoked and weakening reuse containment.
Helpful? Add 👍 / 👎
💡 Fix Suggestion
Suggestion: Remove the hardcoded auth. schema prefix from all SQL query files and instead rely on the PostgreSQL search_path setting, which should be configured at connection time to use the value of DB_NAMESPACE (defaulting to auth). Concretely:
-
In the connection setup (e.g.,
internal/storage/dial.go), after opening the pop connection, executeSET search_path = '<namespace>'using the configuredconfig.DB.Namespacevalue (e.g., viadb.RawQuery("SET search_path = " + config.DB.Namespace).Exec()or by appendingsearch_path=<namespace>to the DSN's connection options). -
Remove the
auth.prefix from all queries across all four.sqlfiles (refresh_tokens.sql,sessions.sql,recovery_codes.sql,identities.sql) — there are 20 occurrences in total. For example, line 5 ofrefresh_tokens.sqlshould become:
UPDATE refresh_tokens SET revoked = true, updated_at = now() WHERE session_id = $1 AND revoked = false; -
After modifying the query files, re-run
make sqlc-generateto regenerate the sqlc output ininternal/storage/dbsql/sqlcgen/.
This ensures that every sqlc query resolves tables against the namespace configured via DB_NAMESPACE, so token-family revocation works correctly in non-default deployments.
Why
We're evaluating moving off gobuffalo/pop and onto sqlc for query generation.
RawQuerycall sites are the natural starting point: they're already hand-written SQL, so porting them doesn't require redesigning anything (unlike pop'sEager-loaded associations, which have no sqlc equivalent and need real query redesign). This PR is that proof of concept — everypop.RawQuerycall ininternal/modelsfor sessions, refresh tokens, recovery codes, and the identityprovider_idupdate now runs through sqlc-generated queries instead.LOC honesty check: this diff is ~1550 lines, but only ~400 of them are hand-written and worth reviewing carefully (the model files,
storage/dial.go, the newinternal/models/sqlc.gobridge, the.sqlquery files,sqlc.yaml, and theMakefile/.gitignoreadditions). The remaining ~1150 lines are 100% sqlc-generated (internal/storage/dbsql/sqlcgen/, markedDO NOT EDIT) — that's the bulk of the diff and isn't something anyone needs to read line by line, only trust the tool for.The schema problem
sqlc needs to see table schemas to type its generated queries. Feeding it this repo's raw migration files doesn't work: most of them wrap their DDL in
do $$ ... end $$blocks for idempotency (create table if not exists, exception handlers, etc.), and sqlc's static parser can't see into aDOblock — it just reports the tables as never having been created. sqlc also doesn't support pointingschema:at a live database connection string despite that config shape looking like it should work.The fix:
make sqlc-generatebrings up the dev postgres, runs the real migrations against it, thenpg_dump --schema-onlys the result intointernal/storage/dbsql/schema/schema.sql, which sqlc reads. That dump is a derived artifact of the migrations, not a second source of truth, so it's gitignored and regenerated on demand rather than committed.The harder problem: sharing a connection between pop and sqlc
sqlc's generated code wants a
database/sql-shaped executor (ExecContext/QueryContext/QueryRowContext/PrepareContext). A freshly-dialedpop.Connectionsatisfies that directly — itsStoreis a concrete*pop.dBwrapping*sqlx.DB, and Go promotes the full method set through concrete struct embedding.But that's not the connection real code actually uses. Every connection that has been through
.WithContext(ctx)or.Transaction(...)— which is to say, every request handler and every transactional model function — has pop rewrapStorein its privatecontextStoretype.contextStoreembeds thestoretype as an interface, not a concrete struct, and Go only promotes an embedded interface's own declared method set. That interface doesn't declareQueryContext/QueryRowContext/PrepareContext(pop talks to sqlx in its own vocabulary internally), so a naive type assertion silently fails on any wrapped connection.This didn't show up as a compile error or an obvious runtime error — it showed up as a 10-minute test hang.
TestReplaceRecoveryCodesSerializesConcurrentReplacementsspawns two goroutines that both callReplaceRecoveryCodesinsidedb.Transaction(...); the first one's call failed the (silent) executor lookup, returned an error before signaling a channel, and the main goroutine sat parked on that channel forever. There's actually a comment in this codebase's existingpopConnToStdacknowledging this exact wall ("I couldn't find any way to obtain the store when wrapped with context due to the private store field") — it just never mattered before, because that helper only ever runs once, on a freshly-dialed connection, at startup.storage.Connection.SQLExecutor()(ininternal/storage/dial.go) solves it with a bounded,recover()-guarded reflection unwrap that reaches pastcontextStore's unexported field to get back to the concrete*pop.dB/*pop.Txunderneath — which, being concrete, does promote its full method set. This is the piece that makes it safe for sqlc queries to share a live pop transaction, which is what makes atomic multi-statement operations likeReplaceRecoveryCodes(lock + delete + insert + reset, all-or-nothing) actually work correctly.What's next
If we like this direction, the remaining
RawQuerysites (webauthn challenges/credentials, factors, AMR claims, OAuth consents/authorizations, OAuth client state, a couple of test-only helpers with genuinely dynamic SQL) are mechanical follow-ups using the same pattern established here. The bigger open question for a fuller pop migration is.Eager()-loaded associations, which have no sqlc equivalent and need real query redesign rather than a straight port.Test plan
go build ./...go test ./internal/models/...— full suite passes (was hanging indefinitely before theSQLExecutorfix)go test ./internal/storage/...— full suite passesmake sqlc-generateverified end-to-end against a freshly migrated dev postgres🤖 Generated with Claude Code