Reverse proxy: 76 allocations per request, 44 of them header handling (follow-up to #1719)

[nicolas-maman]

Follow-up to #1719, which found the LB spends libc 27.3% / own code 9.8% where nginx is 6% / 33.9%, and described the allocation churn as "diffuse ... rather than one hot call". By allocation count it is not diffuse: more than half of it is header handling, in three functions.

Measurement

std.http.server.lb under valgrind --tool=dhat, two Aether backends, 300 proxied requests. Run twice, because the first run gave every request its own connection and so charged per-connection costs to individual requests:

	fresh connection per request	keep-alive, 300 requests over 6 connections
allocations per request	78.2	76.2
total bytes	11.2 MiB	6.5 MiB
conn_serve	3.00/req, 4.8 MiB	2.02/req, 0.1 MiB
the three header sites	21 / 12 / 11	21 / 12 / 11

The read buffer amortises correctly across a connection (it is lazily allocated once and compacted between requests), which is where the byte difference comes from. The header cost does not move, because it is per upstream request either way.

Where the 76 allocations go

site	allocs/req	bytes
http_request_set_header_raw	21.00	95 KiB
http_parse_request_n	12.00	306 KiB
http_response_set_header	11.00	35 KiB
aether_middleware_reverse_proxy	7.00	26 KiB
string_new_with_length	6.03	67 KiB
http_request_internal	5.00	5,297 KiB
http_response_create	4.00	263 KiB
http_request_raw	3.00	34 KiB
conn_serve	2.02	102 KiB
http_extract_response_header	2.00	4 KiB

44 of 76 allocations are header storage and parsing.

The call sites, read from source

1. Three allocations per header, and a quadratic insert — aether_http.c:780, http_request_set_header_raw. Per header: calloc for the node, strdup for the name, strdup for the value. Then the new node is appended by walking the list to its tail, so N headers cost N(N+1)/2 link traversals. Seven forwarded headers is 21 allocations and 28 traversals.

2. Fixed 50-slot arrays plus two strdups per header — aether_http_server.c:1398, http_parse_request_n. Per request: calloc the request, malloc method, path and query separately, strdup the version, then malloc(sizeof(char*) * 50) twice regardless of how many headers arrive, then strdup key and value per header.

3. Two strdups per header — aether_http_server.c:1727, http_response_set_header, with the same 50-slot arrays allocated lazily.

All three store copies of bytes that are already sitting in a buffer the connection owns.

4. A 16 KiB buffer per upstream request — aether_http.c:1941. full_response starts at 16384 bytes and is freed when the request ends, so a pooled connection that serves thousands of requests allocates and frees 16 KiB each time. Plus a 1 KiB header-build buffer at :1687. That is the 5.3 MiB above, and the largest byte figure in the table by an order of magnitude.

5. Per-upstream constants recomputed per request — aether_proxy_middleware.c. extract_authority(u->base_url) mallocs on every request to recover the host:port of a backend whose base URL cannot change; it belongs on the upstream at registration. build_upstream_url genuinely varies, but only in path and query.

Two corrections to what the tree currently says

The picker is not the bottleneck. std/http/server/lb/module.ae lists as its top development direction:

1. Reduce per-request overhead on the hot path (the picker mutates shared state under a lock/atomics; profile it under load).

It does not. pick_round_robin is one atomic_fetch_add on the cursor plus atomic loads in upstream_eligible; the only mutex there is u->rl_lock, guarded by if (u->rl_max_rps > 0) and therefore skipped entirely unless rate limiting is configured. The pthread_mutex_lock/unlock in #1719's profile is the client idle pool's global lock and the worker queue. That note should be replaced with what this census shows.

One poll per response is one syscall more than needed. conn_next_request_imminent polls the socket to decide whether to park, and the caller then reads it — two syscalls where recv(MSG_DONTWAIT) would both detect and consume the next request. The buffered-data short-circuit it needs (conn->write_pos > conn->read_pos) already exists on the line above. This is mine, from #1684, and #1719 measured it at 30% of syscall time.

Suggested order

1. Header storage — one fix covers sites 1–3: a single allocation per header instead of three (name and value in the node's own allocation), plus a tail pointer to kill the quadratic insert. Bigger win available later by pointing into the read buffer rather than copying at all.
2. Reuse the 16 KiB response buffer per transport rather than per request — the biggest byte reduction available.
3. Hoist the per-upstream constants out of the request path.
4. Collapse the poll + recv pair.

Reproduce with valgrind --tool=dhat --dhat-out-file=... on benchmarks/http/lb_reuse_lb.ae against two lb_reuse_backend.ae instances; the numbers above are 300 requests, and the keep-alive column is what a real proxy workload looks like.

[nicolas-maman]

Measured the first item on this list end to end, and it does not hold up. Recording it before anyone spends effort on items 2-4, which rest on the same assumption.

What was built

A per-connection request arena: every string a parsed request owns — method, path, query, version, the header arrays and each key and value — allocated from one 4 KiB block, reset per request, instead of ~30 individual malloc/free pairs. Reusing the runtime's existing arena rather than writing another allocator. It passed everything: keep-alive, parking, streaming upload, actor dispatch, 394 unit tests, no warnings.

What it measured

Six rounds, baseline and candidate alternating, nginx and haproxy in the same session (harness now committed at benchmarks/http/lbbench/):

	rps (median)	CPU per request
main	35,316	50.0 us
main + arena	34,984	51.8 us
	-0.9%	+3.4%

No benefit. Removing thirty allocations per request changed nothing measurable, and if anything cost slightly more.

Allocation count is not what limits this path. glibc handles small short-lived allocations about as well as a bump allocator does, and the census in this issue describes where allocations happen — it was never evidence that removing them buys throughput. That was my inference, and it was wrong.

What that means for the rest of the list

Items 2, 3 and 4 target the same thing:

• the 16 KiB response buffer per upstream request
• extract_authority recomputed per request
• the poll + recv pair

The first two are allocation work and should be assumed worthless until something says otherwise. The third is a syscall, which is a different cost and still worth measuring on its own. extract_authority is worth doing anyway on the grounds that recomputing a per-upstream constant on every request is simply wrong, but not as a performance claim.

A methodology note worth more than the result

The first run of this A/B reported the arena as 17.8% slower on rps and 15.5% worse on CPU. That was the harness, not the code: it ran baseline first and candidate second in every round, so all drift within a round landed on the candidate. Alternating the order brought the same comparison to -0.9% and +3.4%.

The harness now alternates order every round, prints the controls' spread, and says outright when a delta should not be read. Anything measured with a fixed subject order — including numbers I have quoted in this repo before — deserves a second look.

Where the gap actually is

In-container, aether is at 67% of nginx and haproxy is above nginx. The remaining difference is not thirty mallocs. It is architectural: nginx and haproxy run non-blocking event loops with no thread handoff, while an aether request that arrives on a parked connection costs a poller wake, a queue submit and a worker pickup. Profiling should come before the next change, and I would rather profile than guess again.

[nicolas-maman]

Status check against current main, because four of the five call sites named here have since been fixed and someone picking this up would otherwise redo finished work. Read from source, with citations; I could not re-measure, for the reason at the bottom.

1. http_request_set_header_raw — three allocations per header and a quadratic append. Fixed.
std/net/aether_http.c. It is now ONE allocation holding the node, the name and the value, taken from the request's arena when it has one (a pointer bump), and the tail is tracked rather than walked. The code carries the reasoning: "The tail is tracked rather than walked to: appending N headers by walking cost N(N+1)/2 traversals for a list that is only ever appended to." So 21 allocations and 28 traversals for seven headers is now 7 bumps and 7 links.

2. http_parse_request_n — the 50-slot double malloc plus two strdups per header. Fixed.
std/net/aether_http_server.c. Strings come from the request arena via req_alloc, with the worst case reserved up front so the fallback branch is unreachable. The two HTTP_MAX_HEADERS arrays are allocated only when absent, and a reused request arrives with them already there: "the arrays are fixed-size and outlive a request perfectly well."

4. The 16 KiB buffer per upstream request. Changed.
full_response is now the receive buffer rx.buf, grown lazily by doubling rather than starting at a fixed 16384. It is still released per request, so whether the 5.3 MiB figure is gone depends on the response sizes in the benchmark, not on the code alone.

5. extract_authority per request. Fixed.
That function no longer exists. aether_proxy_authority_of replaced it in std/http/proxy/aether_proxy_middleware.c, and its comment names this issue: "Called once per upstream when the pool registers it, not once per forwarded request … this was an allocation and a free on every request to recompute the same bytes (#1739)."

3. http_response_create / http_response_set_header. Still as described.
http_response_create still does malloc(sizeof(char*) * 50) twice for every response, and the per-header strdups remain. This is the one item of the five that is untouched.

What I did not do

I did not re-run the census. This machine has no valgrind/dhat, and the sandbox here blocks the socket binds the LB needs, so I cannot produce a comparable allocations per request number — and quoting the old 76 as if it still held, or "optimising" further without a measurement, would be worse than saying so.

So the headline figures in this issue should be treated as stale rather than current. Whoever picks it up: re-measure first, and expect the remaining work to be item 3 plus whatever the new census actually shows.