Skip to content

test: gate healthy-gating on TCP, not the postgres unix socket#57

Merged
lesnik512 merged 1 commit into
mainfrom
fix/healthy-gating-flake
Jul 14, 2026
Merged

test: gate healthy-gating on TCP, not the postgres unix socket#57
lesnik512 merged 1 commit into
mainfrom
fix/healthy-gating-flake

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Fixes the intermittent tests/integration/test_healthy_gating.py failure that has cost two CI reruns. Design: planning/changes/2026-07-14.04-healthy-gating-flake.md.

Not flaky infrastructure — a real defect in the scenario. The gate was asserting a different transport from the one the dependent uses.

  • db's healthcheck: pg_isready -U postgres — no -h, so it probes the Unix socket
  • app: pg_isready -h 127.0.0.1 -p 5432TCP

The mechanism, from the postgres image's own logs

The image runs initdb, then a temporary server to execute its init scripts. That server listens on the Unix socket only — and announces itself as ready:

[36] listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"   <- temp server, socket ONLY
[36] database system is ready to accept connections                  <- pg_isready -U postgres returns 0 HERE
[37] shutting down                                                   <- temp server stops
     PostgreSQL init process complete; ready for start up.
[1]  listening on IPv4 address "0.0.0.0", port 5432                  <- TCP only appears NOW
[1]  database system is ready to accept connections                  <- for real

There is no IPv4 listener at all during init. So the healthcheck reports healthy, wait_healthy returns 0, the gate opens — and app's TCP probe cannot possibly succeed. The window is ~220ms locally (21.697 → 21.917), and wider on a slower, contended CI runner. That is precisely why it failed only in CI and never locally.

Why this fix is deterministic

Probing TCP makes the gate assert exactly what the dependent needs. The temporary server cannot satisfy a TCP healthcheck, because there is nothing listening on TCP to satisfy it against. This is not a widened timeout or a retry — it removes the failure mode by construction.

It also makes the scenario a stricter test of compose2pod: the pod's shared-namespace 127.0.0.1 now has to work for the healthcheck as well as for the target.

Scope

compose2pod itself is unchanged. The tool gated faithfully on the healthcheck it was handed; the defect was in the scenario's compose document.

The 120-second wait budget is untouched and was never relevant — the gate was not timing out, it was opening on the wrong signal.

Integration suite green (10/10) against real podman 6.0.1.

🤖 Generated with Claude Code

The scenario's healthcheck was `pg_isready -U postgres` -- no -h, so it
probes the unix socket -- while the dependent runs `pg_isready -h 127.0.0.1`,
i.e. TCP.

The postgres image runs initdb, then a temporary server that listens on the
unix socket ONLY and announces "database system is ready to accept
connections", then shuts it down and restarts on TCP. So the healthcheck can
pass while no IPv4 listener exists at all, opening the gate before the
dependent can connect. Proven from the image's own logs; the window is ~220ms
locally and wider on a slower CI runner, which is why it flaked only there.

Probing TCP makes the gate assert exactly what the dependent needs: the temp
server cannot satisfy it, because nothing is listening on TCP to satisfy it
against. Deterministic, not a widened timeout.

compose2pod itself is unchanged -- it gated faithfully on the healthcheck it
was handed. The defect was in the scenario's compose document.
@lesnik512 lesnik512 merged commit 7ccdd0c into main Jul 14, 2026
7 checks passed
@lesnik512 lesnik512 deleted the fix/healthy-gating-flake branch July 14, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant