diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md new file mode 100644 index 000000000..227d03641 --- /dev/null +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -0,0 +1,290 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.3' +--- + +# Provide High-Performance Secondary NICs for KubeVirt VMs with Multus and SR-IOV on ACP + +## Issue + +Users running KubeVirt VMs on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs to VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to VMs, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. + +This article follows the user workflow for completing an end-to-end KubeVirt VM Multus + SR-IOV setup on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, and use the SR-IOV secondary NIC in a KubeVirt VM. The same SR-IOV capability can also be used by application Pods, but this article uses KubeVirt VMs as the example scenario. + +## Environment + +This article applies to the following combination: + +| Component | Version or description | +| --- | --- | +| Alauda Container Platform | 4.3 | +| Plugin | `sriov-network-plugin` | +| Plugin package version | `sriov-network-plugin v4.3.1` | +| Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| Deployment namespace | `cpaas-system` when installed through the ACP marketplace | +| Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | +| Multi-NIC base | ACP provides Multus capability; VMs reference the SR-IOV secondary network through NAD | + +The ACP package enables the SR-IOV CNI path for SR-IOV VF orchestration and Multus secondary-network attachment. + +This article covers installing and using the SR-IOV L5 plugin for KubeVirt VM secondary NICs. It does not cover OVS-DPDK, Userspace CNI, or DPDK application configuration inside containers. + +## Prerequisites + +### Nodes and hardware + +Prepare at least one worker node that meets these requirements: + +- The node has a physical NIC PF that supports SR-IOV. +- IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. +- The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. + +## Resolution + +### Install the plugin + +This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. + +1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.1`. +3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.3.1.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.3.1.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.3.1.tgz`. +4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. +5. Upload the downloaded plugin package to the target ACP platform. + +If the target platform uses `violet` to upload offline packages, use the following command: + +```bash +export PLATFORM_URL="" +export USERNAME="" +export PASSWORD="" +export CLUSTER_NAME="" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" + +violet push "$PACKAGE_FILE" \ + --platform-address "$PLATFORM_URL" \ + --platform-username "$USERNAME" \ + --platform-password "$PASSWORD" \ + --clusters "$CLUSTER_NAME" \ + --target-catalog-source platform +``` + +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.1` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. + +### Confirm the Multus base + +SR-IOV networks are attached to KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the "Install Multus CNI" section in the product documentation for [multiple networks](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. + +After installation, confirm that the operator and config-daemon are running: + +```bash +kubectl get pods -n cpaas-system +``` + +Expected workloads: + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +### Configure VF resources + +A PF is the physical NIC on a node. A VF is a virtual PCI NIC created from a PF and assigned to a VM. + +First list the node states synchronized by the operator, and select the target node from the `NAME` column: + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +On a node with an SR-IOV PF, confirm that the operator discovers the physical NIC: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' +``` + +The operator automatically discovers SR-IOV PFs on nodes and writes them to `SriovNetworkNodeState.status.interfaces`. It does not automatically decide which PF should create VFs, how many VFs to create, which `resourceName` to use, or which VF type to configure. To create VFs and advertise resources through the device plugin, create a `SriovNetworkNodePolicy`. + +Select a PF from the `status.interfaces[*].name` output, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. The following example uses `kubernetes.io/hostname` to select one node, creates four VFs, and advertises a device-plugin resource named `sriov_vf`. + +Save the following content as `sriov-node-policy.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-vf-policy + namespace: cpaas-system +spec: + resourceName: sriov_vf + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: vfio-pci + mtu: 1500 +``` + +`deviceType: vfio-pci` is used for KubeVirt SR-IOV PCI passthrough. The operator configures the VF driver according to this policy and exposes the resource through the device plugin. Do not run `dpdk-devbind.py` inside the VM for host VFs. + +Apply the policy and watch node synchronization: + +```bash +kubectl apply -f sriov-node-policy.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +Confirm that the target node reaches `Succeeded`: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vf}{"\n"}' +``` + +If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. + +### Create an SR-IOV secondary network + +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the VM namespace `kubevirt` and uses Kube-OVN IPAM to assign an address to the SR-IOV secondary NIC. + +Save the following content as `sriov-network.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: sriov_vf + vlan: 0 + ipam: | + { + "type": "kube-ovn", + "server_socket": "/run/openvswitch/kube-ovn-daemon.sock", + "provider": "vm-sriov-net.kubevirt.ovn" + } +``` + +The `provider` uses the `..ovn` format. In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace, so the provider is `vm-sriov-net.kubevirt.ovn`. If the VM runs in another namespace, update `SriovNetwork.spec.networkNamespace`, Subnet `spec.provider`, and VM `metadata.namespace` together. + +Create a Kube-OVN Subnet that uses the same provider. Adjust `cidrBlock`, `gateway`, and `excludeIps` according to the application network plan. + +Save the following content as `sriov-subnet.yaml`: + +```yaml +apiVersion: kubeovn.io/v1 +kind: Subnet +metadata: + name: vm-sriov-net +spec: + protocol: IPv4 + enableDHCP: true + provider: vm-sriov-net.kubevirt.ovn + cidrBlock: 172.22.0.0/16 + gateway: 172.22.0.1 + excludeIps: + - 172.22.0.0..172.22.0.10 +``` + +The KubeVirt VM default network still uses the kube-ovn primary network. The SR-IOV network is attached as a Multus secondary NIC, and its address is allocated by the Kube-OVN Subnet above. + +Apply the objects and confirm that the NAD is generated: + +```bash +kubectl apply -f sriov-network.yaml +kubectl apply -f sriov-subnet.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +Inspect the effective NAD CNI configuration: + +```bash +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . +``` + +### Use the SR-IOV network in a KubeVirt VM + +A VM can reference the same NAD through Multus. The following example shows only the network and interface-related fields: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +In this example, the VM and the NAD are both in the `kubevirt` namespace, so `networkName` can be the NAD name, `vm-sriov-net`. To reference a NAD in another namespace, use the `/` format. + +After creating the VM, confirm that the virt-launcher Pod is scheduled to a node with VF resources and inspect the VMI status: + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +Inside the guest operating system, confirm that an additional NIC appears. The Kube-OVN Subnet handles platform-side address allocation for the secondary network. Whether the address is configured inside the guest still depends on the guest OS DHCP client, cloud-init, or system network configuration. + +### Optional: Bind the service VF to DPDK inside the VM + +If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Prefer the `dpdk-devbind.py` script from the DPDK package inside the VM. If the image does not include the script, get it from the DPDK upstream repository: . + +Inside the VM, identify the PCI NICs: + +```bash +lspci -Dnn | grep -i ethernet +``` + +`dpdk-devbind.py` only binds the VF driver and does not prepare the DPDK application runtime. Configure HugePages, CPU affinity, and DPDK EAL parameters according to the workload image or application documentation. + +Load the VFIO driver: + +```bash +modprobe vfio-pci +``` + +Bind the service VF PCI address as seen inside the VM to `vfio-pci`: + +```bash +python3 dpdk-devbind.py --status +python3 dpdk-devbind.py -b vfio-pci +python3 dpdk-devbind.py --status +``` + +`` is the PCI address seen inside the VM, not the VF address on the host. After binding, the VF is no longer used as a normal guest OS kernel NIC and is instead owned by the DPDK userspace process. If the service VF is owned by the DPDK application, subsequent IP configuration and packet processing are handled by that DPDK application. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md new file mode 100644 index 000000000..7fe740a3f --- /dev/null +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -0,0 +1,290 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.3' +--- + +# 如何在 ACP 上通过 Multus 和 SR-IOV 为 KubeVirt 虚拟机提供高性能辅助网卡 + +## 问题 + +用户在 Alauda Container Platform 4.3 上运行 KubeVirt 虚拟机,并希望通过 Multus 给虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn;Multus 负责把辅助网络接入虚拟机,SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源,并生成 Multus 使用的 `NetworkAttachmentDefinition`。 + +本文从用户使用路径说明如何在 ACP 4.3 上完成 KubeVirt 虚拟机 Multus + SR-IOV 的端到端接入:安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD,并在 KubeVirt 虚拟机中使用 SR-IOV 辅助网卡。同一 SR-IOV 能力也可用于业务 Pod,本文以 KubeVirt 虚拟机场景为例。 + +## 环境 + +本文适用于以下组合: + +| 组件 | 版本或说明 | +| --- | --- | +| Alauda Container Platform | 4.3 | +| 插件 | `sriov-network-plugin` | +| 插件包版本 | `sriov-network-plugin v4.3.1` | +| 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| 部署命名空间 | `cpaas-system`(通过 ACP 市场安装时) | +| 主 CNI | 可使用 kube-ovn;SR-IOV 作为 Multus 辅助网络 | +| 多网卡基座 | ACP 已提供 Multus 能力,虚拟机通过 NAD 引用 SR-IOV 辅助网络 | + +ACP 打包版本启用 SR-IOV CNI 路径,用于完成 SR-IOV VF 编排和 Multus 辅助网络接入。 + +本文覆盖 SR-IOV L5 插件在 KubeVirt 虚拟机辅助网卡场景中的安装与使用;不覆盖 OVS-DPDK、Userspace CNI 或容器内 DPDK 应用的配置。 + +## 先决条件 + +### 节点和硬件 + +至少需要一个满足以下条件的 worker 节点: + +- 节点上有支持 SR-IOV 的物理网卡 PF。 +- BIOS 和操作系统已经启用 IOMMU,例如 Intel VT-d 或 AMD-Vi。 +- PF 驱动支持创建 VF,并且该 PF 未被主 CNI 或 OVS 以不可释放方式占用。 + +## 解决方案 + +### 安装插件 + +该能力作为 ACP 4.3 新功能交付,插件包版本为 `sriov-network-plugin v4.3.1`。用户侧从 AC 应用市场获取插件包,再上传到目标 ACP 平台安装。 + +1. 登录 AC 应用市场,搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.1` 的安装包。 +3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.3.1.tgz`,arm64 平台下载 `sriov-network-plugin.arm64.v4.3.1.tgz`;如果平台不需要区分架构,再下载 `sriov-network-plugin.ALL.v4.3.1.tgz`。 +4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本,重命名可能导致上传失败。 +5. 将下载的插件包上传到目标 ACP 平台。 + +如果目标平台使用 `violet` 上传离线包,可以参考以下命令: + +```bash +export PLATFORM_URL="" +export USERNAME="" +export PASSWORD="" +export CLUSTER_NAME="" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" + +violet push "$PACKAGE_FILE" \ + --platform-address "$PLATFORM_URL" \ + --platform-username "$USERNAME" \ + --platform-password "$PASSWORD" \ + --clusters "$CLUSTER_NAME" \ + --target-catalog-source platform +``` + +上传完成后,进入 **管理员 -> 市场 -> 集群插件**,选择 `sriov-network-plugin` 的 `v4.3.1` 版本并安装到目标业务集群。通过 ACP 市场安装时,SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 + +### 确认 Multus 基座可用 + +SR-IOV 网络通过 Multus 作为辅助网卡接入 KubeVirt 虚拟机。安装 SR-IOV 插件前后,都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装,先参考产品文档[多网络](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) 中的“安装 Multus CNI”章节,再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成,不替代 Multus 元 CNI。 + +安装后确认 operator 和 config-daemon 已运行: + +```bash +kubectl get pods -n cpaas-system +``` + +预期至少看到以下工作负载处于 `Running`: + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +### 配置 VF 资源 + +PF 是节点上的物理网卡,VF 是从 PF 创建出来并分配给虚拟机使用的虚拟 PCI 网卡。 + +先查看 operator 已同步的节点状态,并从输出的 `NAME` 选择目标节点: + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +在带 SR-IOV PF 的节点上,确认 operator 能发现物理网卡: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' +``` + +operator 会自动发现节点上的 SR-IOV PF,并写入 `SriovNetworkNodeState.status.interfaces`;但不会自动决定在哪个 PF 上创建多少 VF、使用什么 `resourceName` 或 VF 类型。要创建 VF 并通过 device plugin 暴露资源,需要创建 `SriovNetworkNodePolicy`。 + +从 `status.interfaces[*].name` 输出中选择一个 PF,例如 `ens5f0`,创建 `SriovNetworkNodePolicy`。`nodeSelector` 只匹配节点上已有的 label;先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点,再使用该节点已有的稳定 label 限制策略作用范围。以下示例使用 `kubernetes.io/hostname` 选中单个节点,创建 4 个 VF,并通过 device plugin 暴露名为 `sriov_vf` 的资源。 + +将以下内容保存为 `sriov-node-policy.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-vf-policy + namespace: cpaas-system +spec: + resourceName: sriov_vf + nodeSelector: + kubernetes.io/hostname: + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: vfio-pci + mtu: 1500 +``` + +`deviceType: vfio-pci` 用于 KubeVirt SR-IOV PCI 直通场景。operator 会根据该策略配置 VF 驱动并通过 device plugin 暴露资源;不要在虚拟机内部对宿主机 VF 执行 `dpdk-devbind.py`。 + +应用后观察节点同步状态: + +```bash +kubectl apply -f sriov-node-policy.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n cpaas-system +``` + +确认目标节点变为 `Succeeded`: + +```bash +kubectl get sriovnetworknodestate -n cpaas-system \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +确认节点资源中出现 SR-IOV device plugin 暴露的资源: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vf}{"\n"}' +``` + +如果输出为正整数,说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 + +### 创建 SR-IOV 辅助网络 + +创建 `SriovNetwork`,由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到虚拟机所在命名空间 `kubevirt`,并使用 kube-ovn IPAM 给 SR-IOV 辅助网卡分配地址。 + +将以下内容保存为 `sriov-network.yaml`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: sriov_vf + vlan: 0 + ipam: | + { + "type": "kube-ovn", + "server_socket": "/run/openvswitch/kube-ovn-daemon.sock", + "provider": "vm-sriov-net.kubevirt.ovn" + } +``` + +其中 `provider` 使用 `..ovn` 格式。上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD,因此 provider 为 `vm-sriov-net.kubevirt.ovn`。如果虚拟机运行在其他命名空间,需要同时替换 `SriovNetwork.spec.networkNamespace`、Subnet `spec.provider` 和 VM `metadata.namespace`。 + +创建与该 provider 匹配的 Kube-OVN Subnet。`cidrBlock`、`gateway` 和 `excludeIps` 按业务网络规划调整。 + +将以下内容保存为 `sriov-subnet.yaml`: + +```yaml +apiVersion: kubeovn.io/v1 +kind: Subnet +metadata: + name: vm-sriov-net +spec: + protocol: IPv4 + enableDHCP: true + provider: vm-sriov-net.kubevirt.ovn + cidrBlock: 172.22.0.0/16 + gateway: 172.22.0.1 + excludeIps: + - 172.22.0.0..172.22.0.10 +``` + +KubeVirt 虚拟机的默认网络仍然使用 kube-ovn 主网络;SR-IOV 网络作为 Multus 辅助网卡接入,并通过上述 kube-ovn Subnet 完成辅助网卡 IPAM。 + +应用后确认 NAD 已生成: + +```bash +kubectl apply -f sriov-network.yaml +kubectl apply -f sriov-subnet.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +查看 NAD 的有效 CNI 配置: + +```bash +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . +``` + +### 在 KubeVirt 虚拟机中使用 SR-IOV 网络 + +虚拟机可以通过 Multus 引用同一个 NAD。以下示例只展示网络和接口相关字段: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +上例中 VM 和 NAD 都在 `kubevirt` 命名空间,因此 `networkName` 可以只写 NAD 名称 `vm-sriov-net`。如果引用其他命名空间的 NAD,使用 `/` 格式。 + +创建 VM 后,确认 virt-launcher Pod 被调度到有 VF 资源的节点,并检查 VMI 状态: + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +进入虚拟机操作系统后,确认出现额外网卡。kube-ovn Subnet 负责平台侧辅助网络地址分配;虚拟机内部是否能拿到该地址,还取决于 guest OS 中的 DHCP 客户端、cloud-init 或系统网络配置。 + +### 在虚拟机内将业务 VF 绑定给 DPDK(可选) + +如果业务需要在虚拟机内使用 DPDK,进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作,不要绑定默认管理网卡。优先使用虚拟机内 DPDK 软件包自带的 `dpdk-devbind.py`;如果镜像中没有该脚本,可以从 DPDK 官方仓库获取:。 + +在虚拟机内确认 PCI 网卡: + +```bash +lspci -Dnn | grep -i ethernet +``` + +`dpdk-devbind.py` 只负责 VF 驱动绑定,不负责准备 DPDK 应用运行环境。HugePages、CPU 亲和性和 DPDK EAL 参数按业务镜像或应用文档配置。 + +加载 VFIO 驱动: + +```bash +modprobe vfio-pci +``` + +将虚拟机内看到的业务 VF PCI 地址绑定到 `vfio-pci`: + +```bash +python3 dpdk-devbind.py --status +python3 dpdk-devbind.py -b vfio-pci +python3 dpdk-devbind.py --status +``` + +`` 是虚拟机内部看到的 PCI 地址,不是宿主机上的 VF 地址。绑定后,该 VF 不再作为 guest OS 的普通内核网卡使用,而是由 DPDK 用户态进程接管。如果业务 VF 被 DPDK 应用接管,后续 IP 配置和报文处理由 DPDK 应用决定。