Add symlink-based LUN device path resolver for Azure NVMe support

[s4heid]

Problem

Azure v6+ VM sizes (Dv6, Dasv6, Dadsv6, Ev6, etc.) use NVMe disk controllers instead of SCSI. The existing scsiLunDevicePathResolver discovers data disks by scanning /sys/bus/vmbus/devices/ and /sys/class/scsi_host/ paths that do not exist on NVMe VMs. This means the bosh-agent cannot resolve ephemeral or persistent disks on NVMe hardware, blocking adoption of v6+ VM sizes.

Solution

Introduce a configurable, infrastructure-agnostic symlink-based resolver that works for both SCSI and NVMe by leveraging udev-managed symlinks (provided by azure-vm-utils).

Two new resolvers:

• SymlinkLunDevicePathResolver: resolves disks via <basePath>/<LUN> symlinks (e.g. /dev/disk/azure/data/by-lun/1 → /dev/nvme0n3). Polls with 100ms interval until the symlink and its target exist, or times out.
• FallbackDevicePathResolver: generic compositor. Tries a primary resolver first; on failure, delegates to a secondary resolver.

When LunDeviceSymlinkPath is set in LinuxOptions, the symlink resolver wraps the existing DevicePathResolutionType-selected resolver as a fallback, but tries the symlink path first regardless of whether the type is "scsi", or anything else.

Additionally, fixes an NVMe-related regex in findRootDevicePathAndNumber() where the NVMe pattern didn't handle multi-digit numbers (e.g. /dev/nvme0n12p2).

Backward Compatibility

Note

This change is not expected to cause any breaking changes and can be merged without requiring additional modifications.
However, the symlink resolver requires certain dependencies to be in place in order to function properly.

• When LunDeviceSymlinkPath is empty (default), behavior is identical to before.
• When LunDeviceSymlinkPath is set, the existing resolver (e.g. SCSI sysfs scanner) becomes the fallback. If the symlink path doesn't exist or times out, the original resolver is tried.
• The CPI requires no changes. It continues sending lun + host_device_id; the symlink resolver only uses lun.
• Non-Azure infrastructures are unaffected, as they don't set LunDeviceSymlinkPath.

Dependencies

• Stemcell: Requires azure-vm-utils to be installed in the stemcell, which provides the udev rules that create /dev/disk/azure/data/by-lun/<LUN> symlinks for both SCSI and NVMe VMs.
  Add Azure Gen2 and NVMe support for stemcells bosh-linux-stemcell-builder#469
• Stemcell config: A separate PR against the stemcell-builder will configure LunDeviceSymlinkPath in the agent.json.
  TBD

Self-Validation

Tested on Azure VMs with 2 Azure data disks (ephemeral LUN 0 + persistent LUN 1), deployed via BOSH with a zookeeper release:

Standard_D4ads_v6 (NVMe with temp disk)

NVMe-only VM. All data disks on MSFT NVMe Accelerator v1.0 controller. Local temp disk on separate Microsoft NVMe Direct Disk v2 controller.

Device	Controller	Model	LUN	Role
/dev/nvme0n1	nvme0	MSFT NVMe Accelerator v1.0	-	OS disk
/dev/nvme0n2	nvme0	MSFT NVMe Accelerator v1.0	0 (nsid=2)	Ephemeral data disk
/dev/nvme0n3	nvme0	MSFT NVMe Accelerator v1.0	1 (nsid=3)	Persistent data disk
/dev/nvme1n1	nvme1	Microsoft NVMe Direct Disk v2	-	Local temp disk

Agent log confirms symlink resolver handles all disk operations:

[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/0' to real path '/dev/nvme0n2'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:0 ...} as '/dev/nvme0n2'
[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/1' to real path '/dev/nvme0n3'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:1 ...} as '/dev/nvme0n3'

Standard_D4as_v6 (NVMe)

NVMe-only VM without local temp disk. Same controller layout as D4ads_v6 minus the NVMe Direct Disk.

Device	Controller	Model	LUN	Role
/dev/nvme0n1	nvme0	MSFT NVMe Accelerator v1.0	-	OS disk
/dev/nvme0n2	nvme0	MSFT NVMe Accelerator v1.0	0 (nsid=2)	Ephemeral data disk
/dev/nvme0n3	nvme0	MSFT NVMe Accelerator v1.0	1 (nsid=3)	Persistent data disk

Standard_D4as_v5 (SCSI)

SCSI VM. Data disks on VMBus SCSI controller ({f8b3781b-1e82-4818-a1c3-63d806ec15bb}).

Device	Controller	LUN	Role
/dev/sda	SCSI (vmbus)	-	OS disk
/dev/sdb	SCSI (vmbus)	0	Ephemeral data disk
/dev/sdc	SCSI (vmbus)	1	Persistent data disk

Agent log confirms symlink resolver works as fast path on SCSI too. No SCSI sysfs fallback needed:

[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/0' to real path '/dev/sdb'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:0 ...} as '/dev/sdb'
[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/1' to real path '/dev/sdc'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:1 ...} as '/dev/sdc'

Standard_D4as_v4 (SCSI)

SCSI VM with local temp disk. Note the non-sequential device letter assignment typical of v4 VMs. The OS disk is /dev/sdb, temp disk is /dev/sdc, and data disks are /dev/sda and /dev/sdd.

Device	Controller	LUN	Role
/dev/sdb	SCSI (vmbus)	-	OS disk
/dev/sdc	SCSI (vmbus)	-	Temp/resource disk
/dev/sda	SCSI (vmbus)	0	Ephemeral data disk
/dev/sdd	SCSI (vmbus)	1	Persistent data disk

Agent log confirms symlink resolver works as fast path. No SCSI sysfs fallback needed:

[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/0' to real path '/dev/sda'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:0 ...} as '/dev/sda'
[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/1' to real path '/dev/sdd'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:1 ...} as '/dev/sdd'

This demonstrates the symlink resolver handles the notoriously unstable SCSI device letter ordering on older Azure VMs, where the OS disk isn't /dev/sda and data disk LUN 0 isn't /dev/sdb. The symlink path /dev/disk/azure/data/by-lun/0 → /dev/sda is stable regardless of enumeration order.

Standard_E4as_v4 (SCSI)

Memory-optimized E-series SCSI VM with local temp disk. Exhibits the same non-sequential device letter assignment as D4as_v4: OS disk is /dev/sdb, temp disk is /dev/sdc, data disks are /dev/sda and /dev/sdd.

Device	Controller	LUN	Role
/dev/sdb	SCSI (vmbus)	-	OS disk
/dev/sdc	SCSI (vmbus)	-	Temp/resource disk
/dev/sda	SCSI (vmbus)	0	Ephemeral data disk
/dev/sdd	SCSI (vmbus)	1	Persistent data disk

Agent log confirms symlink resolver works as fast path. No SCSI sysfs fallback needed:

[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/0' to real path '/dev/sda'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:0 ...} as '/dev/sda'
[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/1' to real path '/dev/sdd'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:1 ...} as '/dev/sdd'

Standard_E4as_v6 (NVMe)

Memory-optimized E-series NVMe VM without local temp disk. All disks exposed via MSFT NVMe Accelerator v1.0 controller. NVMe namespace IDs map directly: OS = nsid 1, data LUN N = nsid N+2.

Device	Controller	LUN	Role
/dev/nvme0n1	NVMe (nsid=1)	-	OS disk
/dev/nvme0n2	NVMe (nsid=2)	0	Ephemeral data disk
/dev/nvme0n3	NVMe (nsid=3)	1	Persistent data disk

Agent log confirms symlink resolver resolves NVMe devices directly — no SCSI sysfs paths involved:

[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/0' to real path '/dev/nvme0n2'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:0 ...} as '/dev/nvme0n2'
[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/1' to real path '/dev/nvme0n3'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:1 ...} as '/dev/nvme0n3'

Standard_F4as_v6 (NVMe)

Compute-optimized F-series NVMe VM without local temp disk. Same NVMe controller and namespace mapping as E4as_v6, confirming behavior is consistent across VM families.

Device	Controller	LUN	Role
/dev/nvme0n1	NVMe (nsid=1)	-	OS disk
/dev/nvme0n2	NVMe (nsid=2)	0	Ephemeral data disk
/dev/nvme0n3	NVMe (nsid=3)	1	Persistent data disk

Agent log confirms symlink resolver works identically on F-series:

[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/0' to real path '/dev/nvme0n2'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:0 ...} as '/dev/nvme0n2'
[symlinkLunResolver] Resolved LUN symlink '/dev/disk/azure/data/by-lun/1' to real path '/dev/nvme0n3'
[fallbackDevicePathResolver] Primary resolver resolved disk {... Lun:1 ...} as '/dev/nvme0n3'