Give the content kubo headroom for leaked inbound connection slots

[TakGN]

What was wrong

Since their start on 2026-10-14, every content kubo (sing-1, lith-1, chic-1) has had all 4096 inbound connection slots of the resource manager in use, and has refused new peers: 2.4M to 3.1M refused connections per box (libp2p_rcmgr_blocked_resources{dir="inbound",resource="connection",scope="system"}).

The connection manager is fine. It counts 277 to 359 connections per box, under HighWater 600, so it has nothing to trim.

The slots are leaked reservations, not connections. Read-only snapshot, 2026-10-19:

	sing-1	lith-1	chic-1
resource manager: inbound conns (system)	4096	4096	4096
resource manager: inbound conns (transient, handshaking)	0	0	0
live inbound conns (swarm opened − closed)	154	95	108
resource manager: FD slots	4095	4127	4078
TCP sockets on :4101 (kernel, ss)	156	121	90

About 3,950 inbound slots per box belong to no connection, and each holds an FD slot too. Only TCP (and websocket) connections reserve FDs, so the leak is on go-libp2p's TCP upgrade path, in the kubo 0.43.1 build (go-libp2p v0.49). It sits after the transient scope. In practice the node takes only as many inbound peers as it has unleaked slots, about 150, and refuses the rest.

The bootstrappers leak the same way (same kubo). Since their restart on 2026-10-16 about 21:53 UTC, each has 5,400 to 9,000 more inbound slots in use than live inbound connections: 24,619 against 15,632 on sing-1. Up to 2048 of that can be handshaking connections. That is roughly 2,000–3,200 leaked slots a day, so the 64000 limit from #7 runs out around the end of October unless they are restarted first. Not changed here; see the follow-ups.

Why the limit is 4096. kubo sets System.ConnsInbound to the largest of MaxMemory in MiB (4GiB = 4096), 2 × ConnMgr.HighWater (1200) and a floor of 800 (core/node/libp2p/rcmgr_defaults.go). For the bootstrappers 2 × HighWater was the largest; for content it is MaxMemory.

The change

The leak can't be stopped from the config, and a restart clears it, so this gives the leak room:

• roles/content/files/libp2p-resource-limit-overrides.json: System.ConnsInbound 65536. This file in the repo is kubo's replacement for Swarm.ResourceMgr.Limits, which kubo no longer accepts. It goes into the kubo-config Secret, and the init container writes it into the repo on every start.
• Swarm.ResourceMgr.MaxFileDescriptors and IPFS_FD_MAX: 65536 → 131072, because each leaked slot also holds an FD slot. This is the same value the bootstrappers use.
• Swarm.ConnMgr stays at 600/200, which fits the real load.
• After a deploy, the role fails unless ipfs swarm resources reports the 65536 limit.

A leaked slot holds no socket and no memory, so a higher limit adds no load. Real connections stay capped by ConnMgr, and memory by the 4GiB resource-manager limit and the pod's 8Gi limit. These boxes have 32 cores and 123 GiB, shared with needle and the bootstrappers, and the content pod's limits don't change.

How long 65536 lasts depends on the leak rate once the node stops refusing peers, which can't be measured until it runs unsaturated. Content takes far fewer connections than the bootstrappers, so this should last weeks. The leak metric under Verify shows the real rate.

Rollout (not done; it restarts the content pod)

From the repo root, one box at a time:

ansible-playbook content.yml --limit sing-1 --check --diff

ansible-playbook content.yml --limit sing-1

Then verify (below), then lith-1, then chic-1. The --check --diff should show only the kubo-config Secret (its contents are no_log) and the Deployment's init script and IPFS_FD_MAX.

Verify, on each box after its rollout

sudo k3s kubectl -n content exec deploy/content -c kubo -- ipfs --api=/ip4/127.0.0.1/tcp/5021 swarm resources | grep -E '^system.*(ConnsInbound|FD)'

curl -s 127.0.0.1:5021/debug/metrics/prometheus | grep -E '^libp2p_rcmgr_(connections\{dir="inbound",scope="system"|limit\{.*conns_inbound.*system|blocked_resources\{dir="inbound",resource="connection",scope="system")'

• libp2p_rcmgr_limit{resource="conns_inbound",scope="system"} is 65536, and system FD is 131072.
• libp2p_rcmgr_connections{dir="inbound",scope="system"} is far below 65536.
• libp2p_rcmgr_blocked_resources{…resource="connection",scope="system"} starts again from zero after the restart and stays there; it no longer rises.
• Leak rate: libp2p_rcmgr_connections{dir="inbound",scope="system"} minus live inbound connections (sum(libp2p_swarm_connections_opened_total{dir="inbound"}) - sum(libp2p_swarm_connections_closed_total{dir="inbound"})). This gap grows by the number of leaked slots; check it after a day.

Follow-ups (not in this PR)

• ipni/infra: once this is live on all three boxes, change ContentRefusingConnections and ContentInboundConnectionsAtLimit in kubernetes/manifests/eu/lith1/monitoring/rules/content.yaml (ipni/infra#178) from warning to critical, as rules/bootstrap.yaml does for the bootstrappers. Also add a warning on the leak gap above (content and bootstrappers) at, say, half the limit, so a restart can be planned before it pages.
• Bootstrappers: same leak, about 12 days of headroom left. Restart them before then, or give them the same overrides file.
• Upstream: report the leak to go-libp2p / kubo with these numbers. One candidate, unconfirmed: p2p/net/upgrader/listener.go Accept() drops an upgraded connection that is already closed without calling Close(), so its scope is never released.

🤖 Generated with Claude Code