Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions planning/changes/2026-07-14.04-healthy-gating-flake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
summary: The healthy-gating scenario's postgres healthcheck probes TCP rather than the Unix socket, so the gate now opens only once the dependent's actual transport is accepting connections.
---

# Change: Fix the healthy-gating scenario's socket-vs-TCP race

**Lane:** lightweight — one line in one integration scenario.

## Goal

`tests/integration/test_healthy_gating.py` fails intermittently in CI (twice so
far, each time passing on rerun). It is not flaky infrastructure: the scenario
gates on a healthcheck that probes a *different transport* from the one the
dependent service uses.

- `db`'s healthcheck is `pg_isready -U postgres` — no `-h`, so it probes the
**Unix socket**.
- `app` runs `pg_isready -h 127.0.0.1 -p 5432` — **TCP**.

The official postgres image runs `initdb`, then starts a **temporary** server to
run its init scripts. That server listens on the Unix socket *only* — and it
announces itself as ready. From its own logs:

```
[36] listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" <- temp server, socket ONLY
[36] database system is ready to accept connections <- `pg_isready -U postgres` returns 0 HERE
[37] shutting down <- temp server stops
PostgreSQL init process complete; ready for start up.
[1] listening on IPv4 address "0.0.0.0", port 5432 <- TCP only appears NOW
[1] database system is ready to accept connections <- for real
```

There is no IPv4 listener during the init phase, so `pg_isready -U postgres`
(Unix socket) reports ready while `app`'s TCP probe cannot possibly succeed —
and the temp server then shuts down. Measured at 220ms on a developer machine
(21.697 -> 21.917); wider on a slower, contended CI runner, which is why it
fails there and not locally.

A TCP healthcheck cannot pass during that phase, because there is nothing
listening on TCP for it to pass against. That is what makes the fix
deterministic rather than a widened timeout.

The 120-second wait budget is untouched and irrelevant — the gate is not timing
out, it is opening on the wrong signal.

## Approach

Probe the transport the dependent actually uses:

```python
"test": ["CMD-SHELL", "pg_isready -U postgres -h 127.0.0.1 -p 5432"]
```

This is what the scenario meant to assert all along: `service_healthy` should
mean "the thing `app` depends on is ready", and `app` depends on TCP. It also
makes the scenario a stricter test of compose2pod — the pod's shared-namespace
`127.0.0.1` now has to work for the healthcheck as well as for the target.

No change to `compose2pod` itself: the tool gated faithfully on the healthcheck
it was handed. The defect is in the scenario's compose document.

## Files

- `tests/integration/test_healthy_gating.py` — healthcheck probes TCP

## Verification

- [ ] Reproduce the window (socket ready before TCP) against real postgres.
- [ ] Apply the change.
- [ ] Run the scenario repeatedly against real podman — must pass every time.
- [ ] `just test-integration` green; `just test-ci` and `just lint-ci` clean.
7 changes: 6 additions & 1 deletion tests/integration/test_healthy_gating.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,12 @@ def test_healthy_gating_reaches_postgres(run_pod: Callable[..., PodRun]) -> None
"image": "postgres:16-alpine",
"environment": ["POSTGRES_PASSWORD=pw"],
"healthcheck": {
"test": ["CMD-SHELL", "pg_isready -U postgres"],
# Probe TCP, not the Unix socket. The postgres image runs initdb, then a
# temporary server with listen_addresses='' (socket only) for its init
# scripts, and only then restarts on TCP -- so a bare `pg_isready -U postgres`
# reports ready for ~0.5s before 127.0.0.1:5432 accepts anything, opening the
# gate before `app` (which connects over TCP) can reach it.
"test": ["CMD-SHELL", "pg_isready -U postgres -h 127.0.0.1 -p 5432"],
"interval": "1s",
"timeout": "5s",
"retries": 30,
Expand Down