fix(enrich): uplinks skip lo and expand bonds to NICs; warn on private netns mounts

CHANGELOG.md

@@ -5,10 +5,12 @@
 ### New
 
 - **Link monitor learn deadline.** `-linkMonitorLearnDeadline` (`LINK_MONITOR_LEARN_DEADLINE`, `link_monitor.learn_deadline`; go-link-monitor `-learn-deadline`), default 5m, 0 waits forever. If no baseline is learned by then, the visible up count becomes a provisional baseline, so `up_links_delta` still reports when the count never becomes ready (a stuck RDMA port, a hardware fault). It is kept in memory only, never written to the baseline file, and flagged by the sparse `go_link_monitor_baseline_provisional 1`; a normally learned baseline replaces it and is saved.
+- **Netns mount propagation check.** In a container, `/run/netns` and `/run/docker/netns` must be mounted `rslave` (`-v /run/netns:/run/netns:ro,rslave`). With Docker's default `rprivate`, namespaces created after xtcp2 starts are invisible: their sockets aren't collected and their container name/image stays empty. xtcp2 now logs a warning at startup and sets `xtcp_gauges{function="checkNetnsPropagation",variable="private",type="<dir>"}` to 1 for each such dir.
 
 ### Fixes
 
 - **Link monitor baseline never written on most prod hosts.** Two causes, found on 78 of 105 hosts: the stock Ubuntu `ib_core` answers the RDMA netlink event subscription with `EOPNOTSUPP`, which was retried every 30 s and kept the count unready; and the bonding module's `/sys/class/net/bonding_masters`, a regular file, failed the RDMA netdev walk. A kernel without RDMA events now falls back to polled RDMA state (reported once, as `rdma_events` support `unsupported`), and non-netdev entries are skipped.
+- **Uplink columns: `lo` as UPLINK2, empty NIC and LLDP columns.** Automatic uplink detection took the kernel's `unreachable ::/0 dev lo`, which every host has, as a default route, and used a bond itself as the uplink, which has no NIC attributes and no LLDP neighbour. Only up, non-reject default routes off `lo` count now, and a bond, vlan or bridge holding the route is replaced by the physical NICs beneath it. `-uplinkInterfaces` is unchanged.
 
 ## 0.2.0 (2026-10)
 

TODO-SOON.md

@@ -2440,3 +2440,36 @@ with the same commands as fleet-snapshots' `xtcp2-spot-check.sh`. Open
 questions: whether the OFED dkms `ib_core` can be installed in a box to cover
 the supported-events path, and whether the boxes need nested virtualisation on
 `l`.
+
+## 30. Prod v2.0.0 records: UPLINK2 `lo`, empty NIC/LLDP and container columns — PARTLY FIXED 2026-11-03
+
+**What.** The xtcp-status review of the first v2.0.0 data (2026-11-03) found
+four daemon-side gaps. Root causes, from the fleet check
+(`fleet-snapshots/xtcp2-v2-check`):
+
+| Symptom | Cause | Status |
+|---|---|---|
+| `UPLINK2_IFNAME` = `lo` | `DefaultUplinks` took every `::/0` in `/proc/net/ipv6_route`, including the kernel's `unreachable ::/0 dev lo` (RTF_REJECT, not RTF_UP) | FIXED: up, non-reject, non-`lo` routes only |
+| NIC and LLDP columns empty | the default route is on `bond0`, which has no `device` entry; LLDP neighbours are keyed by the member ports | FIXED: a virtual uplink expands to the NICs under its `lower_*` links |
+| container name/image empty, `CONTAINER_RUNTIME` = `cgroupfs` | ansible-host mounted `/run/netns` and `/run/docker/netns` with Docker's default `rprivate`, so pods started after xtcp2 were invisible (setns EINVAL, stale inode, dockermeta miss); one host saw 10 of 17 netns | deploy fix in runpod/ansible-host#1120 (`ro,rslave`); xtcp2 now warns at startup and sets `xtcp_gauges{function="checkNetnsPropagation",variable="private"}` |
+| those pods' sockets missing entirely | same as above | same as above |
+
+**Still open.**
+
+- **Link monitor panic isolation.** `runLinkMonitor` turns an error into
+  `xtcp_gauges{function="linkMonitor",variable="up"} 0` and collection carries
+  on (fleet data: uploads were unaffected by the baseline bug), but a panic in
+  the link-monitor goroutines takes the whole daemon down. Recover at the
+  goroutine roots, count it, and mark the link monitor down.
+- **AI098 veth churn.** On a host with heavy pod churn the link-monitor
+  reconcile fails transiently: a veth removed mid-walk, "RDMA topology
+  changed", "interrupted dump". Each retry recovers, but the warn rate is
+  high. Treat a vanished netdev during the walk as a normal retry, not a
+  warning, and add a churn row to the reconcile tables.
+- **gRPC Health has no provisional flag.** `baseline_provisional` (the learn
+  deadline, PR #284) is in `/metrics` but not in the admin gRPC Health
+  response.
+- **Ubuntu guest rows.** Add to §29's Vagrant table: a bonded uplink with
+  lldpd (expect UPLINK1/2 = the member NICs with NIC and LLDP columns set),
+  and the container started before a new netns, once with `ro` (gauge 1,
+  netns missed) and once with `ro,rslave` (gauge 0, netns collected).

docs/design-metadata-enrichment.md

@@ -77,7 +77,11 @@ every record on the hot path (they dictionary-compress to ~nothing downstream).
 
 Topology is modeled as **two fixed uplink slots** (`uplink1_*`, `uplink2_*`) since
 hosts are dual-homed. Uplinks default to the interfaces on the default IPv4/IPv6
-routes (`nicinfo.DefaultUplinks`), overridable with `-uplinkInterfaces`.
+routes (`nicinfo.DefaultUplinks`), overridable with `-uplinkInterfaces`. Only
+usable routes count (up, not reject, not on `lo`: every host has the kernel's
+`unreachable ::/0 dev lo`), and a virtual device holding the route (bond, vlan,
+bridge) is replaced by the physical NICs under its `lower_*` sysfs links, since
+those carry the NIC attributes and the LLDP neighbours.
 
 ## 3. Config: flags / env / config protobuf
 
@@ -125,6 +129,14 @@ current deploy **already** mounts `/run/netns:ro` and `/run/docker/netns:ro`, so
 the discovery fix (§1) restores `netns`/`container_id` with **no deploy change**
 once the new image ships.
 
+Those mounts must be `ro,rslave`, not Docker's default `rprivate`: a pod's netns
+is a bind mount made under them after xtcp2 starts, and a private mount never
+shows it. On prod v2.0.0 one host's xtcp2 saw 10 of its 17 netns; the newer
+pods' sockets went uncollected (setns EINVAL on the stale file) and their
+container name/image was empty. xtcp2 warns at startup and sets
+`xtcp_gauges{function="checkNetnsPropagation",variable="private"}` to 1 for
+each such dir.
+
 To enable the *new* enrichers, add (recommended as **opt-in** ansible vars,
 default off, mirroring `xtcp2_resolve_container_id`):
 

docs/metrics.md

@@ -283,6 +283,7 @@ All are in `xtcp_counts`.
 | `xtcp_counts` | `Run` | `start` | counter | the daemon started |
 | `xtcp_gauge` | | | | namespaces tracked, set by `mapCountReporter` |
 | `xtcp_gauges` | `InitPromethus` | `compiledInEnrichers` | enricher name | 1 if the enricher is built into this binary, 0 if not |
+| `xtcp_gauges` | `checkNetnsPropagation` | `private` | netns dir | 1 if the dir is a private mount, so namespaces created after startup are invisible (mount it `rslave`); 0 if it receives them |
 
 ## Link monitor
 

docs/output-and-destinations.md

@@ -112,7 +112,7 @@ sudo ./result/bin/xtcp2 -dest tcp:127.0.0.1:9000 -marshal jsonl -d 1
 sudo ./result/bin/xtcp2 -dest http://127.0.0.1:8080/ingest -marshal jsonl -d 1
 ```
 
-**In Docker** (mount the host namespace dirs so the daemon sees real sockets; records on `stdout`/`stderr` appear in `docker logs`):
+**In Docker** (mount the host namespace dirs so the daemon sees real sockets; `rslave` so it also sees namespaces created after it starts; records on `stdout`/`stderr` appear in `docker logs`):
 
 ```sh
 # one-time: expose the host root netns as a named netns
@@ -121,7 +121,7 @@ sudo mkdir -p /run/netns && sudo ip netns attach xtcp2host 1
 # CSV to docker logs
 docker run -d --name xtcp2 \
   --cap-add NET_ADMIN --cap-add SYS_ADMIN \
-  -v /run/netns:/run/netns:ro -v /run/docker/netns:/run/docker/netns:ro \
+  -v /run/netns:/run/netns:ro,rslave -v /run/docker/netns:/run/docker/netns:ro,rslave \
   xtcp2:min -dest stdout -marshal csv -d 1 -columns hostname,inetDiagMsgSocketSource,inetDiagMsgState
 docker logs xtcp2            # header + humanized rows
 ```

pkg/nicinfo/uplinks.go

@@ -6,15 +6,35 @@ import (
 	"os"
 	"path/filepath"
 	"sort"
+	"strconv"
 	"strings"
 )
 
+// Route flags (include/uapi/linux/route.h, ipv6_route.h). The same bits in
+// /proc/net/route and /proc/net/ipv6_route.
+const (
+	rtfUp     = 0x0001
+	rtfReject = 0x0200
+)
+
+// maxLowerDepth bounds the upper->lower walk (vlan on bond on NIC is 2).
+const maxLowerDepth = 8
+
 // DefaultUplinks returns up to maxN physical interface names that carry the
 // default IPv4/IPv6 routes (best-effort; reads /proc/net/route +
 // /proc/net/ipv6_route). Used when the operator doesn't pin uplink interfaces
 // explicitly. On any read error it returns what it has so far (possibly empty);
 // it never fails.
+//
+// Only usable routes count: lo and reject routes are skipped (the kernel's
+// "unreachable ::/0 dev lo" is in every ipv6_route), as are routes not up. A
+// virtual device holding the route (bond0, a vlan, a bridge) is replaced by the
+// physical NICs beneath it, which is where the NIC and LLDP columns live.
 func DefaultUplinks(maxN int) []string {
+	return defaultUplinks("/proc", "/sys", maxN)
+}
+
+func defaultUplinks(procRoot, sysfsRoot string, maxN int) []string {
 	if maxN <= 0 {
 		return nil
 	}
@@ -33,44 +53,110 @@ func DefaultUplinks(maxN int) []string {
 		return len(out) < maxN
 	}
 
-	for _, name := range defaultRouteIfacesV4() {
-		if !add(name) {
-			return out
-		}
+	var routes []string
+	if b, err := os.ReadFile(filepath.Join(procRoot, "net", "route")); err == nil {
+		routes = append(routes, parseDefaultRoutesV4(b)...)
 	}
-	for _, name := range defaultRouteIfacesV6() {
-		if !add(name) {
-			return out
+	if b, err := os.ReadFile(filepath.Join(procRoot, "net", "ipv6_route")); err == nil {
+		routes = append(routes, parseDefaultRoutesV6(b)...)
+	}
+	for _, route := range routes {
+		for _, name := range physicalLowers(sysfsRoot, route) {
+			if !add(name) {
+				return out
+			}
 		}
 	}
 	return out
 }
 
-// defaultRouteIfacesV4 parses /proc/net/route for interfaces holding the
-// default route (destination 0.0.0.0). The file is whitespace-separated with a
-// header line; fields are: Iface Destination Gateway Flags ...
-func defaultRouteIfacesV4() []string {
-	b, err := os.ReadFile("/proc/net/route")
-	if err != nil {
-		return nil
+// usableRoute reports whether a default route's interface and hex flags name
+// an uplink: up, not a reject route, and not lo ("*" is v4's no-device).
+func usableRoute(iface, flags string) bool {
+	if iface == "" || iface == "lo" || iface == "*" {
+		return false
 	}
+	f, err := strconv.ParseUint(flags, 16, 32)
+	return err == nil && f&rtfUp != 0 && f&rtfReject == 0
+}
+
+// parseDefaultRoutesV4 returns the interfaces holding a usable default route in
+// /proc/net/route: a header line, then whitespace-separated fields
+// Iface Destination Gateway Flags RefCnt Use Metric Mask ...
+func parseDefaultRoutesV4(b []byte) []string {
 	var out []string
 	for i, line := range strings.Split(string(b), "\n") {
 		if i == 0 { // header
 			continue
 		}
 		f := strings.Fields(line)
-		if len(f) < 2 {
+		if len(f) < 8 {
 			continue
 		}
-		// Destination "00000000" is the default route (0.0.0.0).
-		if f[1] == "00000000" {
+		// Destination and Mask 0.0.0.0 is the default route.
+		if f[1] == "00000000" && f[7] == "00000000" && usableRoute(f[0], f[3]) {
 			out = append(out, f[0])
 		}
 	}
 	return out
 }
 
+// parseDefaultRoutesV6 returns the interfaces holding a usable default route in
+// /proc/net/ipv6_route. Each line has ten fields: dest_net dest_prefixlen
+// src_net src_prefixlen next_hop metric refcnt use flags iface.
+func parseDefaultRoutesV6(b []byte) []string {
+	var out []string
+	for _, line := range strings.Split(string(b), "\n") {
+		f := strings.Fields(line)
+		if len(f) != 10 {
+			continue
+		}
+		// dest_net all zeros and prefix length 0 => default route (::/0).
+		if f[0] == "00000000000000000000000000000000" && f[1] == "00" && usableRoute(f[9], f[8]) {
+			out = append(out, f[9])
+		}
+	}
+	return out
+}
+
+// physicalLowers returns the physical NICs beneath name, following the
+// <sysfs>/class/net/<if>/lower_<if> links a bond, team, vlan or bridge has to
+// its lower devices, sorted per level. A NIC (it has a "device" entry) is its
+// own answer. A virtual device with no physical NIC beneath it (a bridge of
+// veths, a tunnel) or with an unreadable sysfs is returned as itself, as
+// before this walk existed.
+func physicalLowers(sysfsRoot, name string) []string {
+	netDir := filepath.Join(sysfsRoot, "class", "net")
+	var out []string
+	visited := make(map[string]bool)
+	var walk func(name string, depth int)
+	walk = func(name string, depth int) {
+		if visited[name] || depth > maxLowerDepth {
+			return
+		}
+		visited[name] = true
+		dir := filepath.Join(netDir, name)
+		if _, err := os.Lstat(filepath.Join(dir, "device")); err == nil {
+			out = append(out, name)
+			return
+		}
+		entries, err := os.ReadDir(dir)
+		if err != nil {
+			return
+		}
+		for _, e := range entries { // ReadDir sorts by name
+			if lower, ok := strings.CutPrefix(e.Name(), "lower_"); ok && lower != "" {
+				walk(lower, depth+1)
+			}
+		}
+	}
+	walk(name, 0)
+	if len(out) == 0 {
+		return []string{name}
+	}
+	return out
+}
+
 // PhysicalInterfaces returns up to maxN physical interface names discovered by
 // scanning <sysfsRoot>/class/net for entries that have a "device" symlink (a
 // real NIC — virtio_net, mlx5, ice, … — as opposed to lo/veth*/docker0/bridges,
@@ -109,25 +195,3 @@ func PhysicalInterfaces(sysfsRoot string, maxN int) []string {
 	}
 	return names
 }
-
-// defaultRouteIfacesV6 parses /proc/net/ipv6_route for interfaces holding the
-// default route (::/0). Each line has: dest_net(32 hex) dest_prefixlen ...
-// with the interface name as the last field.
-func defaultRouteIfacesV6() []string {
-	b, err := os.ReadFile("/proc/net/ipv6_route")
-	if err != nil {
-		return nil
-	}
-	var out []string
-	for _, line := range strings.Split(string(b), "\n") {
-		f := strings.Fields(line)
-		if len(f) < 10 {
-			continue
-		}
-		// dest_net all zeros and prefix length 0 => default route (::/0).
-		if f[0] == "00000000000000000000000000000000" && f[1] == "00" {
-			out = append(out, f[len(f)-1])
-		}
-	}
-	return out
-}