Give the bootstrappers headroom for leaked inbound connection slots

[TakGN]

What is wrong

The bootstrappers run the same kubo 0.43.1 as the content node, and it leaks inbound connection reservations in the libp2p resource manager the same way (see #10). Since their restart on 2026-10-16 about 21:53 UTC, each instance has 5,400 to 9,000 more inbound slots in use than live inbound connections. On sing-1 on 2026-10-19 that was 24,619 slots against 15,632 live connections. That is roughly 2,000–3,200 leaked slots a day per instance.

Today's limit is 64000. Kubo derives it as 2 × ConnMgr.HighWater (32000, from #7), which is larger than MaxMemory in MiB (8192). At the fastest rate, an instance runs out around 2026-10-31. After that it refuses new peers while the port still accepts TCP, which is the 2026-10-16 incident again.

Only a restart clears a leaked slot, and no config setting stops the leak. So this change gives the leak room, the way #10 does for content.

The change

• roles/bootstrap/files/libp2p-resource-limit-overrides.json: System.ConnsInbound 262144. Kubo dropped Swarm.ResourceMgr.Limits, and this file in the repo replaced it. It goes into each instance's kubo-config Secret, and the init container writes it into $IPFS_PATH on every start. Kubo applies it on top of the limits it computes itself, so everything else stays as it is now, including Transient (ConnsInbound 2048 = MaxMemory MiB / 4).
• Swarm.ResourceMgr.MaxFileDescriptors 131072 → 524288, and IPFS_FD_MAX 65536 → 524288. Each leaked slot also holds an FD slot in the resource manager. Raise the bootstrappers' inbound connection limit to 64000 #7 raised MaxFileDescriptors but left IPFS_FD_MAX at 65536. The two now match again, so the resource manager refuses a connection before an open/accept would fail with EMFILE.
• Swarm.ConnMgr stays at 32000/16000. The task asked whether these can go back to a value based on real load. Not yet:
  • Live inbound is 12k–16k per instance, plus outbound (DHT and peering), and ConnMgr counts both. Lowering HighWater toward the old 8000 would have it trim live peers all the time. That changes how the bootstrappers behave and does nothing about the leak.
  • With ConnsInbound set explicitly, HighWater no longer sets the limit. It can be tuned on its own later, once someone has measured total connections.
  • At 32000, the 64000 limit kubo derives from it stays as the fallback if the overrides file is ever missing.
• Post-deploy assert: for each instance, the role reads ipfs swarm resources --enc=json and fails unless System.ConnsInbound equals the overrides file and System.FD equals MaxFileDescriptors. The success message prints the usage of both.

Sizing (32 cores, 123 GiB per box)

• 262144 inbound slots: (262144 − ~16k live) / 3,200 a day is about 75 days at the fastest rate seen, and about 120 days at the slowest. A leaked slot holds no socket and no resource-manager memory. It is only a counter, so a larger limit adds no load. Real connections stay capped by ConnMgr, memory by MaxMemory 8GiB and the pod's 16Gi limit, and CPU by the 4-core limit.
• 524288 FDs: 2 × ConnsInbound, the same ratio Raise the bootstrappers' inbound connection limit to 64000 #7 chose. Leaked slots plus live TCP connections (in and out) stay well below it, so FD never becomes the limit before ConnsInbound. These are caps, not use, since real FDs track the live TCP connections. The worst case is chic-1: bootstrap and bootstrap-b at 524288 each, plus content at 131072 (Give the content kubo headroom for leaked inbound connection slots #10), comes to about 1.2M. That is under fs.file-max 2097152 and within the 1048576 nofile limit that roles/common and the k3s unit set.

The overrides file is headroom, not a fix: restarts are still needed, about every two months. The ipni/infra follow-up below adds a warning in time to plan one.

Rollout (not done; it restarts every instance)

A restart is a short outage per instance (Recreate, hostNetwork). On chic-1, bootstrap and bootstrap-b restart one after the other.

0. Optional, before the first run. Confirm that the container's hard nofile limit allows 524288. Read-only, on any box:

   sudo k3s kubectl -n bootstrap exec deploy/bootstrap -c kubo -- grep 'open files' /proc/self/limits

   The hard limit (third column) should be at least 524288; it should be 1048576. If it is lower, kubo sets the soft limit to the hard limit, logs an error and still starts.

1. From the repo root, one box at a time, sing-1 first:

   ansible-playbook bootstrap.yml --limit sing-1 --check --diff

   The diff should show only the kubo-config Secret (its contents are no_log) and the Deployment's init script and IPFS_FD_MAX.

   ansible-playbook bootstrap.yml --limit sing-1

2. Verify (below), then repeat for lith-1 and then chic-1, which has two instances.

Verify, on each box after its run

The role already fails if the limits are wrong, and its output shows System.ConnsInbound 262144 (… in use), FD 524288 (… in use) for each instance. To check by hand, with port 5011 for bootstrap and 5012 for bootstrap-b on chic-1:

sudo k3s kubectl -n bootstrap logs deploy/bootstrap -c kubo-config | tail -2

curl -s 127.0.0.1:5011/debug/metrics/prometheus | grep -E '^libp2p_rcmgr_(connections\{dir="inbound",scope="system"|limit\{.*(conns_inbound|fd).*system|blocked_resources\{dir="inbound",resource="connection",scope="system")'

• The init log ends with resource manager overrides: {"System":{"ConnsInbound":262144}}.
• libp2p_rcmgr_limit{resource="conns_inbound",scope="system"} is 262144, and libp2p_rcmgr_limit{resource="fd",scope="system"} is 524288.
• libp2p_rcmgr_blocked_resources{…resource="connection",scope="system"} stays flat.
• After a day, compare libp2p_rcmgr_connections{dir="inbound",scope="system"} with live inbound connections (sum(libp2p_swarm_connections_opened_total{dir="inbound"}) - sum(libp2p_swarm_connections_closed_total{dir="inbound"})). The gap is the leak, and it should grow at about the same 2,000–3,200 a day.
• Externally, the ProbeLab probes and BootstrapRefusingConnections stay quiet.

Follow-ups (not in this PR)

• ipni/infra rules/bootstrap.yaml: the thresholds read libp2p_rcmgr_limit, so they follow the new limit without changes. The header comment says the limit follows ConnMgr.HighWater; it now comes from the overrides file. Also add the leak-gap warning proposed in Give the content kubo headroom for leaked inbound connection slots #10 (above half the limit), so restarts can be planned instead of paged. At 262144, the existing 80%/95% alerts fire only about two months out.
• Upstream: report the leak to go-libp2p / kubo, as in Give the content kubo headroom for leaked inbound connection slots #10.
• ConnMgr: measure total connections per instance, then decide whether HighWater/LowWater should come down.

🤖 Generated with Claude Code