Repository navigation
Server fails behind PgBouncer transaction pooling while reporting ready #524
Description
Problem
The self-hosting guide says a connection proxy such as PgBouncer between the Server and its database "is permitted; it does not change any guarantee". Transaction pooling is PgBouncer's common production mode, and the Server does not work under it. The Server's pgsql connection uses PDO server-side prepared statements and has no setting to turn them off. The Server still reports itself ready while every workflow fails.
Reproduction
Published durableworkflow/server:2.9.7 from docker-compose.published.yml on Postgres 16, with PgBouncer 1.26.1 in front of every Server process. Two PyPI durable-workflow==2.6.1 workers ran concurrent one-activity workflows.
| Configuration | Result |
|---|---|
| Direct to Postgres | 10/10 and 20/20 completed, no SQL errors |
| Session pooling | 10/10 and 20/20 completed, no SQL errors |
Transaction pooling, max_prepared_statements=0 |
Migrations work. 0/10 completed. Hundreds of prepared statement "pdo_stmt_…" does not exist / already exists errors, cascading 25P02 current transaction is aborted, and twice 08P01 bind message supplies 8 parameters, but prepared statement … requires 2, meaning one client's execute reached another client's statement. Starts returned 503 workflow_v2_blocked or 500. |
Transaction pooling, max_prepared_statements=200 (the 1.26 default) |
Bootstrap fails at the first migration. pdo_pgsql's SQL DEALLOCATE fails on the server about 2,000 times in 30 s, which aborts transactions. 0/10 workflows started. |
In every failing configuration the container health check (/api/ready) stayed healthy.
Acceptance criteria
- The Server runs correctly behind PgBouncer in transaction mode, under both prepared-statement settings. It never sends one client's statement to another client's connection.
/api/readyfails, with a reason, when the database path cannot serve workflow queries.- The self-hosting guide states which pooler modes and settings are supported, verified against a published image.
- A published-image test covers direct, session-pooled and transaction-pooled Postgres with concurrent workers.
Activity
rmcdaniel commentedon Oct 20, 2026
Accepted as P1 product correctness work, including readiness, rather than a documentation-only caveat. The active #518 correction is now in draft #525; this is the next PHP database-path priority while its normal gates finish.
I will first preserve the reported direct/session/transaction-pooling counterfactual on the published image, including both prepared-statement settings, then qualify the smallest PDO/database-path correction with concurrent workers, bootstrap/migrations, typed stored values and transaction rollback. Readiness must exercise the relevant configured database path and report its actual scope. The existing connection-only check does not prove that path can execute workflow queries.
PgBouncer's current prepared-statement documentation explicitly distinguishes protocol-level tracking from SQL
DEALLOCATE, which it forwards. That supports investigating the PDO statement lifecycle; it does not establish the eventual fix by itself. No correction or supported published tuple is claimed yet. The self-hosting guidance will follow the verified product behavior.rmcdaniel commentedon Oct 20, 2026
Source correction and local qualification for #524 are ready in draft #531.
Final source is
2e59f1b38275581d58e0aeb86e4ff8fba858342b/ treec11d5ccb68e013dc5c89e87ec243d3574083ee8e, based on main1638681911a. Native/shared/runner/OpenAPI/Composer/version/catalog/frozen files are untouched.The PostgreSQL PDO connection now uses native parameter binding without session-scoped statement names.
/api/readyadditionally executes a bound read through the primary connection and reports its actual scope; failed reads receivedatabase_query_failed. Write capacity remains explicitly unverified. The lightweight runtime bootstrap guard gains no query on its hot path.Local counterfactual: PHP8.3.35 / PostgreSQL17.11 / PgBouncer1.26.0. Direct and session controls pass on unchanged source. Transaction pooling with tracking0 fails with SQLSTATE26000; tracking200 fails inside a real multi-query transaction with25P02. Official PHP source explains its named statement's SQLDEALLOCATE lifecycle; the correction instead uses native bound execution without that session state. This is a separate tuple from the report's PostgreSQL16/PgBouncer1.26.1 observation.
Final source controls PASS: 50 readiness/ownership-protocol/CI-contract tests,649 assertions; eight actual direct/session/transaction0/transaction200 controls,430 assertions, zero skips. Transaction cells prove that one logical client's physical backend PID actually changes while preserving text/int64/null values, mixed parameter arity and transaction rollback. Four separate ownership process controls pass (SQLite+PostgreSQL),72 assertions; four unavailable MySQL/MariaDB cases are explicit skips. The ordinary PHP feature job now runs all four pooler configurations with its existing PostgreSQL service, pinned PgBouncer image and scoped always-cleanup.
The PDO component also passes40 fresh normally installed Python2.6.1 one-activity originals, ten per direct/session/transaction0/transaction200 endpoint. Each cell uses two real Worker processes and a separate cold Client; all eight Workers exit0 and all40 cold results preserve int64 maximum/null/Unicode. Fresh pooled databases each bootstrap through their own pooler endpoint. Original run30s/execution60s budgets are unchanged. Stopped read-only SQL per cell proves10 completed runs/activities,30 completed first-attempt tasks,60 events/10 completions,zero repairs/registrations and exact stored deadlines. Data-only fingerprints before/after inspection match. This component evidence is distinct from the final source's fresh normal CI and eventual published-image qualification.
Unsuccessful originals stay unsuccessful: the first20-workflow direct observation hit its local20s result wait; a separate ten-workflow original completed business work but its cold observer called the wrong public method. Corrected fresh originals pass without changing runtime source or original workflow budgets. The readiness fixture's initial subclass signature and read-only audit's initial column/table guesses were observer failures corrected before their confirming controls. The PHP built-in API launcher required137/noOOM after SIGTERM; graceful API shutdown is not qualified.
The issue stays open. Final normal CI/Sonar, combined-source qualification and the required exact published-image direct/session/transaction acceptance remain ahead. No published fix, unrestricted proxy support, Cloud adoption or historical common cause is claimed.
rmcdaniel commentedon Oct 20, 2026
Source checkpoint: #531 merged normally at
180019992a52845a79d201fc36b7ec5f38d88b4b, whole tree9f16701bf259f28d112324de3bebcdf310141129, exactly the reviewed/candidate-qualified4e837448. Parents are current-main5084d240and4e837448; native/conformance bytes remain unchanged. The completed remote head is deleted and absent.All applicable candidate PHP/repository/performance/boundary/Sonar checks pass. I independently read the final PHP job:2,911 tests,65,226 assertions,one skip,six deprecations; the pinned PgBouncer cells are provisioned in that job, and its extra release guards pass9/27 and1/12. The original fixture transaction leak and stale expectation remain failed observations, corrected without claiming a production database cause. Sonar's five reviewed locations are resolved by source changes; no quality-profile or threshold change.
Fresh main PHP39418873080, performance39418873098, Sonar39418873138 and boundary39418873275 are independent and pending. Source merge is not published-image delivery. Next is the routine2.9.8/Helm0.1.184 metadata preparation with unchanged Core2.7.2/package references, exact-source qualification and immutable-image direct/session/transaction acceptance, including the reported PostgreSQL16/PgBouncer1.26.1 tuple. #524 stays open. No Server518 routing release, native/shared/frozen adoption or Cloud action is implied.
rmcdaniel commentedon Oct 20, 2026
Reopened until released delivery is verified. The earlier source PR's automatic closure preceded publication; source qualification alone does not give existing users an installable fix.
The source correction #531 and its fresh-main checks pass. Release metadata #537 is now merged at
361a96b905ef927c06bbc85238b34b857e875feb/ treecb8ee0c4c383a7ed0692a06217e2f955f8a52bae, identical to its independently qualified candidate. Its fresh-main checks are running separately. Server2.9.8 / Helm0.1.184 are not yet published.The next required outcomes are immutable publication and actual installed acceptance on direct PostgreSQL and session/transaction poolers, including the reported PostgreSQL16/PgBouncer1.26.1 tuple, before closing this issue as delivered. The existing source/local controls and all unsuccessful original observations remain scoped to their recorded artifacts.
rmcdaniel commentedon Oct 20, 2026
The independent main checks for
361a96b905ef927c06bbc85238b34b857e875feb/ treecb8ee0c4c383a7ed0692a06217e2f955f8a52baehave all passed, including PHP feature/performance, readiness/topology, rolling, Sonar and protected chart/boundary checks. Conditional cancellation/prepared-local qualification remains skipped.Immutable tag
2.9.8now points to that exact commit. The protected release workflow is publishing the image; this is not yet an installed-image acceptance claim. The remaining #524 acceptance is the published image against PostgreSQL 16 / PgBouncer 1.26.1 in direct, session and both transaction-pooling prepared-statement modes. The original failing controls remain retained. #524 stays open until that verification completes.The PHP release preserves the qualified native/shared/frozen bytes; Workflow remains the existing locked 2.7.2 dependency. A later package adoption is a separate source window and qualification.
rmcdaniel commentedon Oct 20, 2026
Published Server 2.9.8 acceptance
Exact source
361a96b905ef927c06bbc85238b34b857e875feb, whole treecb8ee0c4c383a7ed0692a06217e2f955f8a52bae. Independent main checks and protected release 39434334201 pass, including immutable Helm 0.1.184 publication. The PHP package lock remains Workflow 2.7.2; this image does not adopt separately published 2.7.3.sha256:d84df4a0baed50204e821cfe458523aff22f9a3949d48851044442386840e7dbsha256:76c5b6b7cc78aeb02dd1ff9f87016662991c11f9af7dba4aaae7748bc87a1ad1sha256:c509e67817382833471a7a257b4664dd124e511d148c8104b818e81cd964c20dThe actual acceptance used the immutable amd64 child; its image labels independently match source/version. No product-source overlay or patched package was mounted.
max_prepared_statements=0max_prepared_statements=200Each cell has two actual concurrent SDK workers exiting 0, ten exact int64/null/Unicode results and a separate cold reader. Stopped read-only SQL independently confirms 10 completed runs / 10 completed first-attempt activities / 30 completed first-attempt tasks / 60 events / 10 workflow completions / zero repairs or retained registrations, with exact original start+30-second run and start+60-second execution deadlines. No SQLSTATE/PDO/fatal error was found in the four retained API logs.
All four actual readiness replies identify
bound_read_queryand explicitly leave write capacity unverified. A separate unavailable-database original returns 503 /not_ready/backend_unavailable. The first negative observer incorrectly expected areadyfield and failed; the corrected separate readback asserts the real published envelope. No business original, request budget or test assertion was relaxed.All three PgBouncer roles and PostgreSQL exit 0 / no OOM. The four multi-process HTTP fixture roles require 137 / no OOM after the business clients and cold readers complete, so this is not graceful HTTP-shutdown qualification. The original published 2.9.7 failures and source counterfactuals remain separate and failed. No native runtime, frozen actor, TLS, performance or Cloud adoption claim follows.
The remaining #524 follow-through is the self-hosting guide's precise supported-mode/version/readiness note and resource-retirement receipt. The exact source/image handoff is available for independently qualified combined integration; no pass transfers to a changed tree or dependency tuple.
rmcdaniel commentedon Oct 20, 2026
Self-hosting documentation follow-through is PR Website #327. The original
ddaac6efe0candidate passed all six normal checks, including the eight-language MDX/link build at 14:37:29 UTC and Sonar with zero unresolved PR issues / 0.0% new duplication.After that original became terminal, the branch incorporated merged callback documentation main
3dbbf9a888once. The new exact source isd5982f00c74875fe692b6dd76db95580a85ceab8, whole tree408c34b6bd1fa1f758186a0c90dc967cacc11d28. Its diff from current main remains only Deployment EN+7 and the corresponding review keys (15 files). Six translation regressions and the full 105-guide reviewed-source/code inventory pass on this combined source; fresh whole-head normal CI is pending, without carrying the original candidate pass.The guide states the fixed Server 2.9.8+ compatibility floor, links the published PostgreSQL 16.15 / PgBouncer 1.26.1 acceptance, lists direct/session/transaction modes with both tested
max_prepared_statements=0and200, requires ordinary bootstrap, and distinguishes/api/ready's bound read from write capacity. No version-manifest, product, runtime, SDK, schema, native, Cloud or pool-tuning change is included.No additional application/DB cohort was run. The focused ephemeral UID1000 Node checkers auto-removed; all temporary translation/check/posting helpers are gone. Exact-main checks, source publication, actual Pages and eight live Deployment routes remain required before this documentation follow-through is delivered. Server #524 remains open for that final acceptance.
rmcdaniel commentedon Oct 20, 2026
The documentation follow-through is source-merged through Website #327: 14:50:27 UTC, exact main
e36d6c113fefda6c448b628270de396a46e7eec9, whole tree408c34b6bd1fa1f758186a0c90dc967cacc11d28, identical to reviewed candidated5982f00c7.All six fresh candidate checks pass, including full eight-language MDX/link qualification at 14:48:12. Independent Sonar PR review reports zero unresolved findings and 0.0% new duplication under unchanged settings. The original
ddaaccandidate's six passing checks remain separate; the combined head received its own qualification. Remote source head was deleted.Only the Deployment guide EN+7 and its review keys changed. The text now states the Server 2.9.8+ transaction-pooling compatibility floor, links the exact published PG16.15/PgBouncer1.26.1 direct/session/transaction0+200 acceptance, requires ordinary bootstrap and limits readiness to its bound read/configured checks without a write-capacity guarantee. No product, version manifest, SDK, schema, native or Cloud adoption is included. No new application/DB test stack was needed; ephemeral Node checkers and helpers are removed.
Fresh exact-main checks, source publication, actual Pages and eight live Deployment routes remain pending. This is a source checkpoint, not the final #524 documentation acceptance. The issue stays open until those gates complete.
rmcdaniel commentedon Oct 20, 2026
PgBouncer self-hosting documentation delivered
Website PR #327 is merged and publicly verified in all eight offered languages. Exact main
e36d6c113fefda6c448b628270de396a46e7eec9, whole tree408c34b6bd1fa1f758186a0c90dc967cacc11d28, matches qualified candidated5982f00c7.3ac51f138ef5f341718c184c2c50fc518fc6c31dexplicitly deploys that main. Actual Pages 39443730353 passed 14:56:54 UTC.bound_read_queryscope are present in every rendered guide.The published self-hosting guide now states:
max_prepared_statements=0ormax_prepared_statements=200. Other combinations need their own qualification; this is not a capacity promise.server-bootstrapinitialization is required before traffic./api/readyexercises a parameter-bound read and configured checks; it leaves write capacity unverified and does not prove every workflow SQL operation.Scope is exactly Deployment EN+7 and its single review key per locale (15 files). Existing examples, budget assumptions and authoritative-writer requirements remain intact. Six translation regressions, the complete 105-guide reviewed-source/code inventory and focused all-eight tuple/settings/bootstrap/readiness/example checks pass. Independent English technical review approved the wording; exact PR Sonar had zero unresolved issues and 0.0% new duplication under unchanged settings.
No product, Server/SDK source, version manifest, dependency, schema, native, Cloud or pool-tuning edit was made. No additional application/DB qualification stack was started: this follow-through uses the owning published acceptance's 40 genuine SDK originals, preserving its original failures and shutdown limits. Ephemeral UID1000 Node checkers and temporary translation tools are absent; live HTML scratch was removed after each route. The remote source branch is absent.
This completes the required #524 self-hosting documentation acceptance.
rmcdaniel commentedon Oct 20, 2026
Delivered in Server 2.9.8, exact source
361a96b905ef927c06bbc85238b34b857e875feb/ whole treecb8ee0c4c383a7ed0692a06217e2f955f8a52bae, immutable image indexsha256:d84df4a0baed50204e821cfe458523aff22f9a3949d48851044442386840e7dband Helm 0.1.184. Independent main, protected release and actual normally installed consumer acceptance pass. The exact original failed cases remain preserved.The installed PG16.15/PgBouncer1.26.1 cohort completes all 40/40 genuine workflows across direct, session, transaction/max_prepared_statements=0 and transaction/max_prepared_statements=200 modes. Eight workers and the cold readers exit0; stopped read-only SQL confirms original typed values/history/deadlines and zero repairs/registrations. Positive readiness performs an actual bound read; stopped-backend readiness returns503. This does not claim write-capacity or graceful API-shutdown verification. Full installed receipt.
The required self-hosting guidance is now verified live in all eight offered languages, from exact Website
e36d6c113fefda6c448b628270de396a46e7eec9/ tree408c34b6: all candidate and seven independent main checks pass, actual Pages39443730353 passes, and all eight real Deployment pages match generated published bytes with the fixed-version floor, both transaction modes, bootstrap and readiness scope. Final live documentation receipt.All owned runtime/database/compiler/dependency/image/raw resources are retired, latest actual lifecycle14:43:40UTC; documentation helpers/checkouts are retired15:00:13UTC. Borrowed/other-owner resources are preserved. This completes #524; database TLS configuration #540 and service-worker evolution #518/#221 remain separate product obligations.