Skip to content

fix(enrich): uplinks skip lo and expand bonds to NICs; warn on private netns mounts - #285

Merged
randomizedcoder merged 4 commits into
mainfrom
fix/enrich-uplinks
Nov 4, 2026
Merged

randomizedcoder merged 4 commits into
mainfrom
fix/enrich-uplinks

Conversation

@randomizedcoder

Copy link
Copy Markdown
Owner

Fixes for the daemon-side gaps in the first v2.0.0 prod data (xtcp-status review, 2026-11-03).

  • UPLINK2 = lo: DefaultUplinks took the kernel's unreachable ::/0 dev lo (RTF_REJECT, not RTF_UP), which is in every host's ipv6_route. Default routes now count only when they're up, not reject, and not on lo/*, and the v4 mask must be 0.
  • Empty NIC and LLDP columns: the prod default route is on bond0, which has no device entry, while LLDP neighbours are keyed by the member ports. A bond/vlan/bridge uplink is now replaced by the physical NICs under its lower_* sysfs links (cycle- and depth-bounded). A virtual device with no NIC beneath it is kept as before.
  • Empty container name/image, cgroupfs runtime, missing pod sockets: the cause is deploy-side. ansible-host mounted the netns dirs with Docker's default rprivate, so pods started after xtcp2 were invisible (one host saw 10 of 17). The fix is runpod/ansible-host#1120 (ro,rslave). Here, xtcp2 now reads /proc/self/mountinfo at startup, warns, and sets xtcp_gauges{function="checkNetnsPropagation",variable="private"} per dir.
  • Docs: the docker example uses ro,rslave; metrics table, design doc, CHANGELOG Unreleased.
  • TODO-SOON §30: root causes, plus open items for link-monitor panic isolation, AI098 veth-churn reconcile warnings, the gRPC Health provisional flag, and Ubuntu guest rows.
  • Tests: tables for both route parsers, the lower walk, DefaultUplinks end to end on fixture proc/sysfs, mountinfo propagation and the gauge. Every guard was mutation-checked. The lint ratchet, metrics-audit, gofmt and vet are clean.
  • Will conflict with fix(linkmonitor): baseline on the prod fleet (bonding_masters, RDMA events fallback, learn deadline) #284 on the CHANGELOG Unreleased heading and the TODO-SOON tail (§29 vs §30). Both are appends; whichever PR merges second resolves it.

🤖 Generated with Claude Code

randomizedcoder and others added 4 commits November 3, 2026 18:09
…NICs

DefaultUplinks took every ::/0 in /proc/net/ipv6_route, including the
kernel's "unreachable ::/0 dev lo" (RTF_REJECT, not RTF_UP) that every host
has, so prod records carried UPLINK2=lo. It also returned bond0 itself, a
virtual device with no "device" entry, so the NIC columns were empty and the
LLDP neighbours (keyed by the physical ports) never matched.

Default routes now count only when up, not reject, and not on lo or "*"
(v4 unreachable/blackhole). The v4 mask must be 0 too. A virtual device
holding the route is replaced by the physical NICs under its lower_<if>
sysfs links (bond, team, vlan, bridge; cycles and depth bounded); one with
no NIC beneath it is kept as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On prod v2.0.0, ansible-host mounted /run/netns and /run/docker/netns with
Docker's default rprivate propagation, so namespaces created after xtcp2
started never appeared in the container: their sockets went uncollected
(setns EINVAL on the stale file) and their container name/image was empty,
with runtime falling back to cgroupfs. One host saw 10 of its 17 netns.

At startup xtcp2 reads /proc/self/mountinfo, finds the mount holding each
netns dir, and if it neither shares nor slaves (no shared:N / master:N) logs
a warning naming the rslave fix and sets
xtcp_gauges{function="checkNetnsPropagation",variable="private",type=<dir>}
to 1 (0 otherwise). The deploy fix is runpod/ansible-host#1120.

Docs: the docker example mounts ro,rslave; metrics table row; design doc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CHANGELOG Unreleased: the uplink fix and the netns propagation check
- TODO-SOON §30: root causes of UPLINK2=lo, empty NIC/LLDP and container
  columns; open items for link-monitor panic isolation, AI098 veth churn,
  gRPC Health provisional flag, and Ubuntu guest rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@randomizedcoder
randomizedcoder merged commit 840e4c3 into main Nov 4, 2026
1 check passed
@randomizedcoder randomizedcoder mentioned this pull request Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant