Share PostgreSQL with Substrate

[iplay88keys]

Important

This PR depends on agent-substrate/substrate#1752 and kagent-dev/substrate#32. After both merge and release, this branch still needs its Substrate image/chart dependency bumped and locked.

Summary

• Bootstrap Kagent's fixed kagent_user login, kagent_owner group role, and schema with embedded identity SQL before table migrations. The bundled install uses a separate administrator Secret and a Kagent application connection Secret. The binary checks the fixed development credentials in that connection Secret and never resets an existing login's password.
• Let embedded Substrate share Kagent's bundled kagent database while using separate owner and read/write connections in the substrate schema. The chart renders those connection Secrets with Substrate-owned Helm helpers and validates the shared database and administrator Secret references.
• Support existing connection Secrets for an external database or operator-managed users on the bundled database. Disabling identity bootstrap leaves table migrations enabled. Kagent's default schema remains public.
• Read @file: connection strings when opening new physical PostgreSQL connections and assume a stable group role. DB_MAX_CONN_LIFETIME (Helm database.postgres.pool.maxConnLifetime) bounds Kagent connection lifetime; Helm substrate.postgres.pool.maxConnLifetime sets the corresponding Substrate flag. These settings bound how long old credentials remain in use. The migration CLI also accepts a role.
• Configure the one pgvector extension schema separately from Kagent's table schema (default public). Bootstrap installs the extension there; external databases must preinstall it there. Migrations check the installed location before writing, and both migrations and runtime qualify pgvector references by schema while keeping only Kagent's table schema on the normal search path. Multiple installs in one database can share the extension schema, even if it contains other application objects.
• When the extension schema is separate from the table schema, the application role needs USAGE there. The runtime checks that its table schema is accessible before querying.
• BYO installations can set Kagent and Substrate group role names through chart values. The standalone identity SQL accepts an optional owner role name for manual provisioning, defaulting to kagent_owner. Bundled bootstrap keeps its fixed role names, which the binary passes explicitly.

Scope and dependencies

This identity layout is for a fresh database. Database upgrades into this layout are not supported by this PR. Operators managing their own users must create the users, roles, schema, and grants, supply application connection Secrets, and disable bootstrap.

The bundled chart's default administrator and application passwords are fixed, published values. Fixed-identity bootstrap is for development and evaluation, not production. Production deployments should provision unique credentials and database permissions outside the application, use existing connection Secrets, and disable bootstrap.

Embedded Substrate requires the runtime in agent-substrate/substrate#1752 and the chart helpers in kagent-dev/substrate#32. This branch still pins an older Substrate module, CI version, and Helm dependency; those pins need a released companion version before integration or release. Local Helm validation used a packaged copy of the companion chart.

Multiple installs sharing one database should use distinct table schemas, group roles, and logins, with each login granted only its own install's roles.

Validation

• From go/, go test ./core/internal/database ./core/pkg/migrations -count=1 passed; make -C go lint also passed.
• helm unittest helm/kagent passed (336 tests). A local umbrella-chart render verified custom Kagent and Substrate role names.

[smeegoan]

Three gaps after this PR.

--postgres-schema defaults to public, and atepg.go assigns RuntimeParams["search_path"] unconditionally after parsing the DSN. In the shared-Secret setup this PR documents, a connection string carrying search_path=agents is therefore overridden to public on the ate-api-server side, silently, unless the flag is set to match.

Migrations cannot use a separate identity. RunUp runs in-process on the same connection string before the pool opens, so the role serving traffic must also hold DDL rights on its own schema, and every migrated object ends up owned by it. A schema owner distinct from its runtime consumer is not expressible.

Rotation has no input left. POSTGRES_DATABASE_URL is an env var, frozen for the life of the process. urlFile was the only input a rotation system could write to, and passfile is no substitute because ParseConfig resolves that once too. Is rotation meant to work another way?

[iplay88keys]

@smeegoan, thanks for taking a look! Just updating this PR and reviewing your comment. Just a heads up that Codex helped put together this response, though I reviewed it.

--postgres-schema defaults to public, and atepg.go assigns RuntimeParams["search_path"] unconditionally after parsing the DSN. In the shared-Secret setup this PR documents, a connection string carrying search_path=agents is therefore overridden to public on the ate-api-server side, silently, unless the flag is set to match.

This PR defaults substrate.postgres.schema to substrate, which becomes the explicit --postgres-schema=substrate argument. We intentionally do not rely on the DSN’s search_path; users can override the Substrate schema value.

Migrations cannot use a separate identity. RunUp runs in-process on the same connection string before the pool opens, so the role serving traffic must also hold DDL rights on its own schema, and every migrated object ends up owned by it. A schema owner distinct from its runtime consumer is not expressible.

Kagent already supports this operationally. An operator can create the schema/grants and run kagent db migrate up --db-url <migrator-url> with privileged credentials, then configure the controller with restricted runtime credentials and database.postgres.skipMigrations=true. The chart does not provision those roles or grants. RunUp is skipped in that mode.

On the Substrate side, even if we added a way to pre-apply and skip startup migrations, that would not provide complete DDL/DML separation. ateapi also performs runtime DDL to create and remove outbox partitions, so its runtime role must retain those privileges.

Operators can configure Substrate with a separate connection Secret and role, even when it points to the same PostgreSQL server and database. This keeps Substrate’s DDL privileges off Kagent’s runtime role, but Substrate currently has no supported mechanism for separating its own migration/DDL and runtime/DML identities. Additional Substrate work would be required for that stricter separation.

Rotation has no input left. POSTGRES_DATABASE_URL is an env var, frozen for the life of the process. urlFile was the only input a rotation system could write to, and passfile is no substitute because ParseConfig resolves that once too. Is rotation meant to work another way?

urlFile was read only once during startup, so changing it did not rotate an active pool. A Secret-backed environment variable has the same restart requirement. Live rotation would require explicit reload and pool reconnection support.

So the PR does not remove an existing Kagent credential-separation or live-rotation capability. The real follow-up is adding equivalent migration-role separation to Substrate if that is required for shared production databases.

[iplay88keys]

@smeegoan, I see now how urlFile can be used to rotate credentials. In order to maintain that functionality, while still getting rid of the urlFile convention, I have updated this PR (as well as the required upstream Substrate PR) to now mount the secretRef as a volume and use the db url env to instead pass a path to that file.

As such, pgx's beforeConnect will re-read file-backed credentials and TLS material before each new physical connection. I also added the maxConnLifetime to substrate so the same mechanism for rotation can exist there.

So the updated design preserves a live Kubernetes Secret input and supports credential rotation without a pod restart, while replacing the arbitrary file-path urlFile value with explicit Secret references.

[smeegoan]

Thank you @iplay88keys, the @file: plus BeforeConnect design closes the rotation gap.

One thing stops us adopting it as is. sameConnectionIdentity treats user as part of connection identity, so a rotation that changes the username fails every new acquire with database connection identity changed; restart required, while existing connections keep serving until maxConnLifetime expires. Our rotation changes the username by design: each cycle creates a new user with the new password and leaves the previous user able to log in until the following cycle, so that two credentials authenticate at once and no consumer fails while the secret store promotes the new version. That is a property of the database's rotation, not something a single consumer opts into. New dials should authenticate as the incoming user while older connections finish on the outgoing one, which is what the pool already does.

Proposal: move User out of the identity comparison and into the set the refresh adopts, next to Password. Host, port, database and fallback host and port stay fenced, since those are the pool's endpoint and changing them mid-life does need a restart. That removes the user comparison rather than adding anything.

[iplay88keys]

@smeegoan, thanks for taking another look. I'm glad this approach overall will work for you. I have updated both sides of this (Substrate/Kagent) to allow for rotating the user to allow for the flow you laid out.