Skip to content

Fix bm_fastapi benchmark failing with httpx.PoolTimeout under load - #509

Open
sendaoYan wants to merge 1 commit into
python:mainfrom
sendaoYan:fix-fastapi-pool-timeout
Open

sendaoYan wants to merge 1 commit into
python:mainfrom
sendaoYan:fix-fastapi-pool-timeout

Conversation

@sendaoYan

Copy link
Copy Markdown
Contributor

Fixes #508

Problem

The fastapi benchmark (bm_fastapi) intermittently fails with
httpx.PoolTimeout when the machine running the benchmark is under heavy load.

The benchmark fires CONCURRENCY = 150 concurrent requests, but the
httpx.AsyncClient uses httpx's default max_connections = 100. The 50 requests
that exceed the pool size wait in the pool's queue for a connection to free up.
httpx's default pool timeout is 5 seconds, so when the in-flight requests take
longer than 5 seconds (which is exactly what happens under load), those 50 queued
requests exceed the pool timeout and the benchmark dies:

httpx.PoolTimeout
RuntimeError: Benchmark died
ERROR: Benchmark fastapi failed: Benchmark died

Fix

Size the connection pool to the benchmark's concurrency so that no request ever
queues:

-        async with httpx.AsyncClient() as client:
+        async with httpx.AsyncClient(limits=httpx.Limits(max_connections=CONCURRENCY)) as client:

With the pool sized to CONCURRENCY, all 150 requests acquire a connection
immediately, so the pool timeout can never be hit regardless of server latency.

Testing

Reproduced deterministically by serving the endpoint with an artificial ~6s
per-request delay and firing 150 concurrent requests:

  • Before: default pool (100 connections) → httpx.PoolTimeout after ~5.1s
  • After: max_connections=150 → completes successfully (~7.1s)

No behavior change in the common case: the benchmark's concurrency is a fixed
constant, so the pool is simply sized to match it.

@sendaoYan

Copy link
Copy Markdown
Contributor Author

Note: this touches the same httpx.AsyncClient(...) line as #505. Once #505 is
merged, the resolved form should be:

async with httpx.AsyncClient(
    trust_env=False,
    limits=httpx.Limits(max_connections=CONCURRENCY),
) as client:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bm_fastapi intermittently fails with httpx.PoolTimeout under load

1 participant