netlab status: show memory usage per node and per lab

[sdargoeuves]

As it has been discussed on the Network Automagic podcast, I wanted to give (AI a) go at showing the memory usage of a lab, per node.

This is the output for a lab with libvirt+clab:

Lab 2 in /home/sa/code/quick-netlab-lab/dns-lab
  status:      started
  topology:    topology.yml
  provider(s): libvirt,clab
  memory used: 1.838GiB

┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ node   ┃ device  ┃ mgmt IP       ┃ provider ┃ status        ┃ memory   ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ r1     │ arubacx │ 10.194.57.101 │ libvirt  │ running       │ 1.768GiB │
│ r2     │ frr     │ 10.194.57.102 │ clab     │ Up 11 minutes │ 30.56MiB │
│ r3     │ frr     │ 10.194.57.103 │ clab     │ Up 11 minutes │ 33.27MiB │
│ dnssrv │ dnsmasq │ 10.194.57.104 │ clab     │ Up 11 minutes │ 7.867MiB │
└────────┴─────────┴───────────────┴──────────┴───────────────┴──────────┘

Disclosure: high risk of AI slop from this point.

Implementation — each provider's get_lab_status enriches its status box with a memory key; fetch_node_status copies it into the node status and sums the total into lab_state.memory (so it also lands in --format json/yaml).

• clab: docker stats --no-stream --format json → MemUsage, "used" half only. The value after the slash is the host total when no limit is set (always, for netlab-generated labs), so it's dropped.
• libvirt: virsh domstats --balloon --list-running → balloon.rss (host RSS of the QEMU process). Falls back to balloon.current suffixed (max) when the guest has no balloon driver. Domain names are matched back to Vagrant machine names via the _<node> suffix; no new keys are added to the status box.
• strings.py: format_memory_size / parse_memory_size helpers, docker's 1.521GiB / 184.5MiB style.

Both collectors are non-fatal — failures are print_verbose only, memory is extra info.

Caveats / things you may want changed:

• Adds ~1.5–2s to netlab status for a clab lab (docker stats --no-stream samples every container on the host, not just the lab's).
• The two numbers aren't the same measure: cgroup footprint vs QEMU RSS, so the total sums slightly different things. It's the right ballpark for "what is this lab eating", not exact host-RAM accounting. With KSM enabled, summing RSS overstates real usage. whatever that means 🫥
• No used / allocated display, which could be useful for libvirt.

[jbemmel]

I contributed a PR to Containerlab over the weekend: srl-labs/containerlab#3325

These cgroups allow you to collect lots of stats directly:

root ➜ /xform $ cat /sys/fs/cgroup/xform/ospf-dual-stack/r1/memory.peak 
62664704
root ➜ /xform $ cat /sys/fs/cgroup/xform/ospf-dual-stack/r1/memory.current 
46010368

I suspect Linux sys/fs could give us all those numbers without going through those separate tools

[ipspace]

These cgroups allow you to collect lots of stats directly:

Just to make sure I understand: when your PR gets into a containerlab release, and when we switch to that release, and (optionally) when we update the clab.yml, we will be able to read these things directly.

Right?

[jbemmel]

These cgroups allow you to collect lots of stats directly:

Just to make sure I understand: when your PR gets into a containerlab release, and when we switch to that release, and (optionally) when we update the clab.yml, we will be able to read these things directly.

Right?

Well, actually we could read it today:

docker inspect --format '{{.State.Pid}}' <container name> gives the PID
cat /proc/<PID>/cgroup gives the cgroup
cat /sys/fs/cgroup/<cgroup>/memory.current gives current memory usage

The cgroup parent grouping can be helpful to get the aggregate, and makes the lookup slightly simpler.

[sdargoeuves]

Aaah, I forgot to remove my Multilab commit... I'll remove it later.

I didn't know about cgroup I recently wanted to check how much RAM were the SDWAN components eating, and started looking into the docker stats command.

[sdargoeuves]

We could also use the cgroup approach for libvirt nodes, if we want to go that direction:

╰─❯ cat /run/libvirt/qemu/ml-2_aruba0.pid -p                                                   
1997096
╰─❯ cat /proc/1997096/cgroup -p                                                                
0::/machine.slice/machine-qemu\x2d2\x2dml\x2d2aruba0.scope/libvirt/emulator
╰─❯ cat /sys/fs/cgroup/machine.slice/machine-qemu\\x2d2\\x2dml\\x2d2aruba0.scope/memory.current -p
1977995264

[ipspace]

Well, actually we could read it today:

Yeah, I found that recipe, and figured out it does not work on my Docker/Ubuntu combo 🤷‍♂️

@sdargoeuves -- I would suggest we use the orchestration tools (Docker/libvirt) to fetch the stats. Let them deal with the underlying OS mess. See also: https://www.youtube.com/watch?v=oebqlzblfyo

[sdargoeuves]

Finally got a chance to start my main lab to look at the memory usage with this branch. The netlab status command takes 46s with this branch, compared to 44s when using the August release.

Now I can see which node(s) are the hungry ones... no surprises, but still nice to see!

Here is an extract:

╰─❯ dnetlab status              
Lab 1 in /home/sa/code/netsim-main-lab
  status:      started
  topology:    topology.yml
  provider(s): libvirt,clab
  memory used: 153.403GiB

┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ node             ┃ device   ┃ image                                      ┃ mgmt IP       ┃ connection  ┃ provider ┃ VM/container               ┃ status                  ┃ memory   ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ d4xfw01          │ linux    │ checkpoint/cloudguard:81.20.5699           │ 10.194.56.99  │ paramiko    │ libvirt  │ ml-1_d4xfw01               │ inaccessible            │ 6.580GiB │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ lb3x01           │ linux    │ f5/bigip                                   │ 10.194.56.101 │ paramiko    │ libvirt  │ ml-1_lb3x01                │ running                 │ 4.009GiB │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ lb3x02           │ linux    │ netlab/netscaler                           │ 10.194.56.102 │ docker      │ clab     │ clab-ml-1-lb3x02           │ Up 51 minutes           │ 567MiB   │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ sdwan-manager    │ linux    │ vrnetlab/cisco_sdwan-manager:20.16.1       │ 10.194.56.130 │ docker      │ clab     │ clab-ml-1-sdwan-manager    │ Up 51 minutes (healthy) │ 32.25GiB │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ sdwan-controller │ linux    │ vrnetlab/cisco_sdwan-controller:20.16.1    │ 10.194.56.131 │ docker      │ clab     │ clab-ml-1-sdwan-controller │ Up 51 minutes (healthy) │ 993.6MiB │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ sdwan-validator  │ linux    │ vrnetlab/cisco_sdwan-validator:20.16.1     │ 10.194.56.132 │ docker      │ clab     │ clab-ml-1-sdwan-validator  │ Up 51 minutes (healthy) │ 831.7MiB │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ edge1x01         │ cat8000v │ vrnetlab/cisco_c8000v:controller-17.15.04c │ 10.194.56.133 │ network_cli │ clab     │ clab-ml-1-edge1x01         │ Up 51 minutes (healthy) │ 4.029GiB │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ edge2x01         │ cat8000v │ vrnetlab/cisco_c8000v:controller-17.15.04c │ 10.194.56.134 │ network_cli │ clab     │ clab-ml-1-edge2x01         │ Up 51 minutes (healthy) │ 4.028GiB │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ edge3x01         │ cat8000v │ vrnetlab/cisco_c8000v:controller-17.15.04c │ 10.194.56.135 │ network_cli │ clab     │ clab-ml-1-edge3x01         │ Up 51 minutes (healthy) │ 4.029GiB │
├──────────────────┼──────────┼────────────────────────────────────────────┼───────────────┼─────────────┼──────────┼────────────────────────────┼─────────────────────────┼──────────┤
│ c0xr01           │ vsrx     │ juniper/vsrx3:21.3R1.9                     │ 10.194.56.66  │ netconf     │ libvirt  │ ml-1_c0xr01                │ running                 │ 3.990GiB │
[...]