From 4e8a829e83b5fbaf5c58a26ec2608568404d5360 Mon Sep 17 00:00:00 2001 From: changluyi <47097611+changluyi@users.noreply.github.com> Date: Mon, 6 Jul 2026 15:56:04 +0800 Subject: [PATCH 1/7] docs: add SR-IOV KubeVirt passthrough guide --- AGENTS.md | 4 + ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 421 ++++++++++++++++++ ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 421 ++++++++++++++++++ 3 files changed, 846 insertions(+) create mode 100644 AGENTS.md create mode 100644 docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md create mode 100644 docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..ed9d29adf --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,4 @@ +# Project Memory + +- When adding or updating a solution document, keep the Chinese and English versions in sync. +- For files under `docs/zh/solutions/`, add or update the matching file under `docs/en/solutions/` with the same filename unless the user explicitly asks for Chinese-only work. diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md new file mode 100644 index 000000000..741b6ddb0 --- /dev/null +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -0,0 +1,421 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.4.x' +--- + +# Enable SR-IOV NIC Passthrough for KubeVirt VMs on ACP + +## Issue + +Users running KubeVirt VMs on Alauda Container Platform 4.4.x may need to attach host SR-IOV VFs to the VMs as high-performance secondary NICs. The cluster primary CNI can remain kube-ovn; SR-IOV Network Operator is used to discover SR-IOV PFs, create VFs, advertise VF resources through the device plugin, and generate the corresponding `NetworkAttachmentDefinition` objects for Multus. + +This article follows the user workflow for installing the ACP 4.4.x `sriov-network-plugin`, validating the control plane in an environment without SR-IOV NICs, and completing VF and VM data-plane validation in an environment with SR-IOV hardware. + +## Environment + +This article applies to the following combination: + +| Component | Version or description | +| --- | --- | +| Alauda Container Platform | 4.4.x | +| Plugin | `sriov-network-plugin` | +| Plugin package version | `sriov-network-plugin v4.4.1` | +| Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| Deployment namespace | `cpaas-system` when installed through the ACP marketplace | +| Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | + +The ACP package enables only the SR-IOV CNI path. The image values for `ib-sriov-cni`, `ovs-cni`, and `rdma-cni` are empty, so the operator does not render these init containers in the `config-daemon` DaemonSet. + +## Prerequisites + +### Platform and permissions + +Prepare a kubeconfig that can manage the target cluster. The current user must be able to create the following resources: + +- Namespaces, ServiceAccounts, ClusterRoles, and ClusterRoleBindings +- CRDs +- Deployments and DaemonSets +- Custom resources in the `sriovnetwork.openshift.io` API group +- `NetworkAttachmentDefinition` objects in `k8s.cni.cncf.io/v1` + +### Nodes and hardware + +SR-IOV hardware is not required if you only need to validate the plugin control plane. To complete VF and virtual-machine data-plane validation, prepare at least one worker node that meets these requirements: + +- The node has a physical NIC PF that supports SR-IOV. +- IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. +- The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. +- The target node can enter a maintenance window. Creating VFs or changing VF drivers can trigger node drain or a short network interruption. + +Label the nodes that will be configured for SR-IOV. `SriovNetworkNodePolicy` will use this label to limit its scope: + +```bash +kubectl label node feature.node.kubernetes.io/sriov-capable=true +``` + +## Resolution + +### Install the plugin + +This capability is delivered as an ACP 4.4.x feature. The plugin package version is `sriov-network-plugin v4.4.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. + +1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. +2. Select the package whose compatible platform version is `v4.4` and whose plugin version is `v4.4.1`. +3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.4.1.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.4.1.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.4.1.tgz`. +4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. +5. Upload the downloaded plugin package to the target ACP platform. + +If the target platform uses `violet` to upload offline packages, use the following command: + +```bash +export PLATFORM_URL="" +export USERNAME="" +export PASSWORD="" +export CLUSTER_NAME="" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.4.1.tgz" + +unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY + +violet push "$PACKAGE_FILE" \ + --platform-address "$PLATFORM_URL" \ + --platform-username "$USERNAME" \ + --platform-password "$PASSWORD" \ + --clusters "$CLUSTER_NAME" \ + --target-catalog-source platform +``` + +After the upload succeeds, verify on the global cluster that the plugin version configuration has been generated and is ready to deploy: + +```bash +kubectl get moduleplugin sriov-network-plugin \ + -o jsonpath='{.status.latestVersion}{"\n"}' +kubectl get moduleconfig sriov-network-plugin-v4.4.1 \ + -o jsonpath='{.status.readyForDeploy}{"\n"}' +``` + +The expected output is `v4.4.1` and `true`. If the `ModulePlugin` exists but no `ModuleConfig` is generated, or if the `ModuleConfig` is not `readyForDeploy=true`, the plugin package metadata is incomplete. Common causes are a missing `ModulePlugin.spec.logo` or a missing platform installation config file, `scripts/plugin-config.yaml`, in the package. Upload the complete package published from the AC marketplace instead. + +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. The following commands use this variable: + +```bash +export SRIOV_NAMESPACE="cpaas-system" +``` + +Before installation, confirm that the cluster does not already have a manually installed SR-IOV instance. If a previous test installation exists in another namespace, such as `sriov-network-operator`, uninstall that old instance with the original installation method before installing from the marketplace. Running two operators against the same SR-IOV custom resources is not recommended: + +```bash +kubectl get pods -A | grep sriov-network +kubectl get sriovoperatorconfig -A +kubectl get sriovnetworknodestate -A +``` + +The SR-IOV node daemon requires `hostNetwork`, `hostPID`, `hostPath`, and privileged containers. Before installation, confirm that the deployment namespace allows privileged Pod Security Admission. If the labels are missing, add them: + +```bash +kubectl label namespace "$SRIOV_NAMESPACE" \ + pod-security.kubernetes.io/enforce=privileged \ + pod-security.kubernetes.io/audit=privileged \ + pod-security.kubernetes.io/warn=privileged \ + --overwrite +``` + +After installation, confirm that the operator and config-daemon are running: + +```bash +kubectl get pods -n "$SRIOV_NAMESPACE" +``` + +Expected workloads: + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +Confirm that the CRDs are registered: + +```bash +kubectl get crd | grep sriovnetwork.openshift.io +``` + +Confirm that `config-daemon` has only the `sriov-cni` init container and does not include `ib-sriov-cni`, `ovs-cni`, or `rdma-cni`: + +```bash +kubectl get daemonset sriov-network-config-daemon -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' +``` + +Expected output: + +```text +sriov-cni build-harbor.alauda.cn/3rdparty/sriov/sriov-cni:v2.10.0 +``` + +### Control-plane validation without SR-IOV hardware + +If the current environment has no SR-IOV NIC, the validation goal is to prove that the plugin deploys correctly, CRDs are registered, the operator can synchronize node state, and no extra `ovs-cni`, `rdma-cni`, or `ib-sriov-cni` init containers are deployed. + +Check `SriovNetworkNodeState`: + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n "$SRIOV_NAMESPACE" +``` + +Check a specific node: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}' +``` + +In an environment without SR-IOV hardware, `status.interfaces` can be empty. This is a hardware limitation and does not indicate a plugin deployment failure. If the operator, config-daemon, CRDs, and `syncStatus` are healthy, the control-plane smoke validation is sufficient. + +### VF validation with SR-IOV hardware + +On a node with an SR-IOV PF, first confirm that the operator discovers the physical NIC: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' +``` + +Select a PF, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. The following example creates four VFs and advertises a device-plugin resource named `intel_sriov_netdevice`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: intel-sriov-netdevice + namespace: cpaas-system +spec: + resourceName: intel_sriov_netdevice + nodeSelector: + feature.node.kubernetes.io/sriov-capable: "true" + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: netdevice + mtu: 1500 +``` + +Apply the policy and watch node synchronization: + +```bash +kubectl apply -f sriov-node-policy.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n "$SRIOV_NAMESPACE" +``` + +Confirm that the target node reaches `Succeeded`: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +Confirm that the SR-IOV device-plugin resource appears in node allocatable resources: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' +``` + +If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. + +### Create an SR-IOV secondary network + +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `kubevirt`: + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: intel_sriov_netdevice + vlan: 0 + ipam: | + { + "type": "host-local", + "subnet": "192.168.100.0/24", + "rangeStart": "192.168.100.100", + "rangeEnd": "192.168.100.200", + "gateway": "192.168.100.1" + } +``` + +Apply the object and confirm that the NAD is generated: + +```bash +kubectl apply -f sriov-network.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +Inspect the effective NAD CNI configuration: + +```bash +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . +``` + +### Validate VF allocation with a test Pod + +Create a test Pod in the application namespace. The Pod must request the SR-IOV resource and reference the NAD: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sriov-test + namespace: kubevirt + annotations: + k8s.v1.cni.cncf.io/networks: vm-sriov-net +spec: + restartPolicy: Never + containers: + - name: test + image: + command: + - sleep + - "3600" + resources: + requests: + openshift.io/intel_sriov_netdevice: "1" + limits: + openshift.io/intel_sriov_netdevice: "1" +``` + +Confirm that the Pod is scheduled to an SR-IOV node and enters `Running`: + +```bash +kubectl get pod sriov-test -n kubevirt -o wide +``` + +Enter the Pod and inspect the secondary NIC: + +```bash +kubectl exec -n kubevirt sriov-test -- ip link +kubectl exec -n kubevirt sriov-test -- ip addr +``` + +If the underlay network is ready, continue with ping or application traffic validation: + +```bash +kubectl exec -n kubevirt sriov-test -- ping -c 3 192.168.100.1 +``` + +### Use the SR-IOV network in a KubeVirt VM + +A VM can reference the same NAD through Multus. The following example shows only the network and interface-related fields: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +After creating the VM, confirm that the virt-launcher Pod is scheduled to a node with VF resources and inspect the VMI status: + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +Inside the guest operating system, confirm that an additional NIC appears, then configure an IP address according to the network design or use DHCP if available. + +## Diagnostic steps + +### Operator or config-daemon is not running + +Check Pod status and events: + +```bash +kubectl get pods -n "$SRIOV_NAMESPACE" -o wide +kubectl describe pod -n "$SRIOV_NAMESPACE" +kubectl logs -n "$SRIOV_NAMESPACE" deploy/sriov-network-operator +``` + +If `config-daemon` is stuck in init, first confirm that the init container list contains only `sriov-cni`. If `ovs-cni`, `rdma-cni`, or `ib-sriov-cni` still appears, the installed plugin package or operator image is not the ACP-corrected version. + +### The node does not discover an SR-IOV PF + +Check `SriovNetworkNodeState`: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" -o yaml +``` + +If `status.interfaces` is empty, continue checking hardware and kernel state on the node: + +```bash +lspci -nn | grep -i ethernet +ip link show +``` + +Confirm that BIOS/IOMMU is enabled and that the target PF driver supports SR-IOV. + +### Policy synchronization fails + +Check synchronization status and errors: + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +Common causes include: + +- `nodeSelector` does not match any node. +- `nicSelector.pfNames` is incorrect, or the PF name does not exist on the target node. +- The PF is held by another network component and cannot create VFs. +- IOMMU is not enabled, or the driver does not support the requested number of VFs. + +### Pod or VM cannot allocate a VF + +Confirm that the resource exists in node allocatable: + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' +``` + +Check Pod or virt-launcher events: + +```bash +kubectl describe pod -n kubevirt +``` + +If the event reports insufficient resources, verify that `SriovNetworkNodePolicy.spec.resourceName`, the Pod resource request, and `SriovNetwork.spec.resourceName` all use the same value. + +## Limitations + +- An environment without SR-IOV hardware can only validate the control-plane smoke path. It cannot prove VF creation, device-plugin resource advertisement, or VM data-plane connectivity. +- This solution does not replace the cluster primary CNI. kube-ovn continues to provide the Pod primary network, while SR-IOV is used as a Multus secondary network. +- The current ACP package does not deploy `ovs-cni`, `rdma-cni`, or `ib-sriov-cni`. If the workload requires OVS, RDMA, or InfiniBand SR-IOV, evaluate the required images, operator rendering logic, and validation scope separately. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md new file mode 100644 index 000000000..b0d1747ca --- /dev/null +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -0,0 +1,421 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - '4.4.x' +--- + +# 如何在 ACP 上为 KubeVirt 虚拟机启用 SR-IOV 网卡直通 + +## 问题 + +用户在 Alauda Container Platform 4.4.x 上运行 KubeVirt 虚拟机，并希望给虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；SR-IOV Network Operator 只负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并为 Multus 生成对应的 `NetworkAttachmentDefinition`。 + +本文从用户使用路径说明如何安装 ACP 4.4.x 的 `sriov-network-plugin`，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 VF 和虚拟机数据面验证。 + +## 环境 + +本文适用于以下组合： + +| 组件 | 版本或说明 | +| --- | --- | +| Alauda Container Platform | 4.4.x | +| 插件 | `sriov-network-plugin` | +| 插件包版本 | `sriov-network-plugin v4.4.1` | +| 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | +| 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | +| 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | + +ACP 打包版本只启用 SR-IOV CNI 路径。`ib-sriov-cni`、`ovs-cni`、`rdma-cni` 镜像值为空，operator 渲染 `config-daemon` 时不会部署这些 init containers。 + +## 先决条件 + +### 平台和权限 + +准备一个可管理目标集群的 kubeconfig，并确保当前用户可以创建以下资源： + +- 命名空间、ServiceAccount、ClusterRole、ClusterRoleBinding +- CRD +- Deployment、DaemonSet +- `sriovnetwork.openshift.io` API 组下的自定义资源 +- `k8s.cni.cncf.io/v1` 的 `NetworkAttachmentDefinition` + +### 节点和硬件 + +如果只验证插件控制面，可以没有 SR-IOV 网卡。如果要完成 VF 和虚拟机数据面验证，至少需要一个满足以下条件的 worker 节点： + +- 节点上有支持 SR-IOV 的物理网卡 PF。 +- BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 +- PF 驱动支持创建 VF，并且该 PF 未被主 CNI 或 OVS 以不可释放方式占用。 +- 计划配置 VF 的节点可以进入维护窗口。创建 VF 或切换 VF 驱动可能触发节点 drain 或网络短暂中断。 + +给参与 SR-IOV 配置的节点打标签，后续 `SriovNetworkNodePolicy` 使用该标签限制作用范围： + +```bash +kubectl label node feature.node.kubernetes.io/sriov-capable=true +``` + +## 解决方案 + +### 安装插件 + +该能力作为 ACP 4.4.x 新功能交付，插件包版本为 `sriov-network-plugin v4.4.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 + +1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 +2. 选择适配平台版本为 `v4.4`、插件版本为 `v4.4.1` 的安装包。 +3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.4.1.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.4.1.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.4.1.tgz`。 +4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 +5. 将下载的插件包上传到目标 ACP 平台。 + +如果目标平台使用 `violet` 上传离线包，可以参考以下命令： + +```bash +export PLATFORM_URL="" +export USERNAME="" +export PASSWORD="" +export CLUSTER_NAME="" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.4.1.tgz" + +unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY + +violet push "$PACKAGE_FILE" \ + --platform-address "$PLATFORM_URL" \ + --platform-username "$USERNAME" \ + --platform-password "$PASSWORD" \ + --clusters "$CLUSTER_NAME" \ + --target-catalog-source platform +``` + +上传成功后，在 global 集群确认插件版本配置已经生成并可部署： + +```bash +kubectl get moduleplugin sriov-network-plugin \ + -o jsonpath='{.status.latestVersion}{"\n"}' +kubectl get moduleconfig sriov-network-plugin-v4.4.1 \ + -o jsonpath='{.status.readyForDeploy}{"\n"}' +``` + +预期分别输出 `v4.4.1` 和 `true`。如果 `ModulePlugin` 存在但没有生成 `ModuleConfig`，或者 `ModuleConfig` 不是 `readyForDeploy=true`，说明插件包元数据不完整，常见原因是包内缺少 `ModulePlugin.spec.logo` 或平台安装配置 `scripts/plugin-config.yaml`，需要使用 AC 市场发布后的完整包重新上传。 + +上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。后续命令使用以下变量： + +```bash +export SRIOV_NAMESPACE="cpaas-system" +``` + +安装前确认集群中没有历史手工安装的 SR-IOV 实例。如果曾经在 `sriov-network-operator` 等其他命名空间测试安装过，应先按原安装方式卸载旧实例，避免两个 operator 同时协调同一组 SR-IOV 自定义资源： + +```bash +kubectl get pods -A | grep sriov-network +kubectl get sriovoperatorconfig -A +kubectl get sriovnetworknodestate -A +``` + +SR-IOV 节点守护进程需要使用 `hostNetwork`、`hostPID`、`hostPath` 和 privileged 容器。安装前，确认部署命名空间允许 privileged Pod Security Admission；如果标签缺失，可以补齐： + +```bash +kubectl label namespace "$SRIOV_NAMESPACE" \ + pod-security.kubernetes.io/enforce=privileged \ + pod-security.kubernetes.io/audit=privileged \ + pod-security.kubernetes.io/warn=privileged \ + --overwrite +``` + +安装后确认 operator 和 config-daemon 已运行： + +```bash +kubectl get pods -n "$SRIOV_NAMESPACE" +``` + +预期至少看到以下工作负载处于 `Running`： + +```text +sriov-network-operator-xxxxx 1/1 Running +sriov-network-config-daemon-xxxxx 1/1 Running +``` + +确认 CRD 已注册： + +```bash +kubectl get crd | grep sriovnetwork.openshift.io +``` + +确认 `config-daemon` 只包含 `sriov-cni` init container，不包含 `ib-sriov-cni`、`ovs-cni`、`rdma-cni`： + +```bash +kubectl get daemonset sriov-network-config-daemon -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' +``` + +预期只出现类似输出： + +```text +sriov-cni build-harbor.alauda.cn/3rdparty/sriov/sriov-cni:v2.10.0 +``` + +### 无 SR-IOV 网卡环境的控制面验证 + +如果当前环境没有 SR-IOV 网卡，验证目标是证明插件可以正常部署、CRD 可以注册、operator 可以同步节点状态，并且不会额外部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。 + +检查 `SriovNetworkNodeState`： + +```bash +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n "$SRIOV_NAMESPACE" +``` + +查看具体节点状态： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}' +``` + +在无 SR-IOV 网卡环境中，`status.interfaces` 可能为空；这是硬件条件限制，不代表插件部署失败。只要 operator、config-daemon、CRD 和 `syncStatus` 正常，即可认为控制面 smoke 验证通过。 + +### 有 SR-IOV 网卡环境的 VF 验证 + +在带 SR-IOV PF 的节点上，先确认 operator 能发现物理网卡： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' +``` + +选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。以下示例创建 4 个 VF，并通过 device plugin 暴露名为 `intel_sriov_netdevice` 的资源： + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: intel-sriov-netdevice + namespace: cpaas-system +spec: + resourceName: intel_sriov_netdevice + nodeSelector: + feature.node.kubernetes.io/sriov-capable: "true" + nicSelector: + pfNames: + - ens5f0 + numVfs: 4 + deviceType: netdevice + mtu: 1500 +``` + +应用后观察节点同步状态： + +```bash +kubectl apply -f sriov-node-policy.yaml +kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ + -n "$SRIOV_NAMESPACE" +``` + +确认目标节点变为 `Succeeded`： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +确认节点资源中出现 SR-IOV device plugin 暴露的资源： + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' +``` + +如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 + +### 创建 SR-IOV 辅助网络 + +创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `kubevirt`： + +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: vm-sriov-net + namespace: cpaas-system +spec: + networkNamespace: kubevirt + resourceName: intel_sriov_netdevice + vlan: 0 + ipam: | + { + "type": "host-local", + "subnet": "192.168.100.0/24", + "rangeStart": "192.168.100.100", + "rangeEnd": "192.168.100.200", + "gateway": "192.168.100.1" + } +``` + +应用后确认 NAD 已生成： + +```bash +kubectl apply -f sriov-network.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net +``` + +查看 NAD 的有效 CNI 配置： + +```bash +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . +``` + +### 使用测试 Pod 验证 VF 分配 + +在业务命名空间创建测试 Pod，显式请求 SR-IOV 资源并引用 NAD： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sriov-test + namespace: kubevirt + annotations: + k8s.v1.cni.cncf.io/networks: vm-sriov-net +spec: + restartPolicy: Never + containers: + - name: test + image: + command: + - sleep + - "3600" + resources: + requests: + openshift.io/intel_sriov_netdevice: "1" + limits: + openshift.io/intel_sriov_netdevice: "1" +``` + +确认 Pod 被调度到 SR-IOV 节点并进入 `Running`： + +```bash +kubectl get pod sriov-test -n kubevirt -o wide +``` + +进入 Pod 检查辅助网卡： + +```bash +kubectl exec -n kubevirt sriov-test -- ip link +kubectl exec -n kubevirt sriov-test -- ip addr +``` + +如果网络侧已经配置好二层或三层连通性，可以继续执行 ping 或业务流量验证： + +```bash +kubectl exec -n kubevirt sriov-test -- ping -c 3 192.168.100.1 +``` + +### 在 KubeVirt 虚拟机中使用 SR-IOV 网络 + +虚拟机可以通过 Multus 引用同一个 NAD。以下示例只展示网络和接口相关字段： + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: sriov-vm + namespace: kubevirt +spec: + running: true + template: + spec: + domain: + devices: + interfaces: + - name: default + masquerade: {} + - name: sriov-net + sriov: {} + networks: + - name: default + pod: {} + - name: sriov-net + multus: + networkName: vm-sriov-net +``` + +创建 VM 后，确认 virt-launcher Pod 被调度到有 VF 资源的节点，并检查 VMI 状态： + +```bash +kubectl get vmi -n kubevirt sriov-vm +kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide +``` + +进入虚拟机操作系统后，确认出现额外网卡，并按业务网络规划配置 IP 或使用 DHCP 获取地址。 + +## 诊断步骤 + +### operator 或 config-daemon 未运行 + +检查 Pod 状态和事件： + +```bash +kubectl get pods -n "$SRIOV_NAMESPACE" -o wide +kubectl describe pod -n "$SRIOV_NAMESPACE" +kubectl logs -n "$SRIOV_NAMESPACE" deploy/sriov-network-operator +``` + +如果 `config-daemon` 卡在 init 阶段，先确认 init container 列表是否只有 `sriov-cni`。如果仍然出现 `ovs-cni`、`rdma-cni` 或 `ib-sriov-cni`，说明安装的不是 ACP 修正后的插件包或 operator 镜像版本。 + +### 节点没有发现 SR-IOV PF + +检查 `SriovNetworkNodeState`： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" -o yaml +``` + +如果 `status.interfaces` 为空，继续在节点侧确认硬件和内核状态： + +```bash +lspci -nn | grep -i ethernet +ip link show +``` + +确认 BIOS/IOMMU 已启用，并且目标 PF 驱动支持 SR-IOV。 + +### policy 同步失败 + +查看同步状态和错误： + +```bash +kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ + -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' +``` + +常见原因包括： + +- `nodeSelector` 没有匹配任何节点。 +- `nicSelector.pfNames` 写错，或者 PF 名称在目标节点上不存在。 +- PF 已被其他网络组件占用，无法创建 VF。 +- 节点未启用 IOMMU 或驱动不支持请求的 VF 数量。 + +### Pod 或 VM 无法分配 VF + +确认节点 allocatable 中存在资源： + +```bash +kubectl get node \ + -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' +``` + +确认 Pod 或 virt-launcher 事件： + +```bash +kubectl describe pod -n kubevirt +``` + +如果事件提示资源不足，检查 `SriovNetworkNodePolicy.spec.resourceName`、Pod resource request、`SriovNetwork.spec.resourceName` 三者是否一致。 + +## 限制 + +- 无 SR-IOV 网卡环境只能完成控制面 smoke 验证，不能证明 VF 创建、device plugin 资源暴露或虚拟机数据面连通性。 +- 本方案不替换集群主 CNI。kube-ovn 继续承担 Pod 主网络，SR-IOV 网络作为 Multus 辅助网络使用。 +- 当前 ACP 打包版本不部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。如果业务需要 OVS、RDMA 或 InfiniBand SR-IOV，需要单独评估镜像、operator 渲染逻辑和验证范围。 From 7d337e2373c7948d8ee3ddcca4aa199a21104c27 Mon Sep 17 00:00:00 2001 From: changluyi <47097611+changluyi@users.noreply.github.com> Date: Mon, 6 Jul 2026 18:41:13 +0800 Subject: [PATCH 2/7] ACP-53813 document Multus SR-IOV solution --- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 54 ++++++++++++++----- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 54 ++++++++++++++----- 2 files changed, 80 insertions(+), 28 deletions(-) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 741b6ddb0..4c058fc92 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -4,16 +4,16 @@ kind: products: - Alauda Container Platform ProductsVersion: - - '4.4.x' + - '4.3.x' --- -# Enable SR-IOV NIC Passthrough for KubeVirt VMs on ACP +# Provide High-Performance Secondary NICs for Application Pods and KubeVirt VMs with Multus and SR-IOV on ACP ## Issue -Users running KubeVirt VMs on Alauda Container Platform 4.4.x may need to attach host SR-IOV VFs to the VMs as high-performance secondary NICs. The cluster primary CNI can remain kube-ovn; SR-IOV Network Operator is used to discover SR-IOV PFs, create VFs, advertise VF resources through the device plugin, and generate the corresponding `NetworkAttachmentDefinition` objects for Multus. +Users running KubeVirt VMs or containerized network functions (CNFs) on Alauda Container Platform 4.3.x may need to attach host SR-IOV VFs to Pods or VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to the workload, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. -This article follows the user workflow for installing the ACP 4.4.x `sriov-network-plugin`, validating the control plane in an environment without SR-IOV NICs, and completing VF and VM data-plane validation in an environment with SR-IOV hardware. +This article follows the user workflow for completing an end-to-end Multus + SR-IOV setup on ACP 4.3.x: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, validate the control plane without SR-IOV NICs, and validate Pod plus KubeVirt VM data-plane behavior on SR-IOV hardware. ## Environment @@ -21,14 +21,17 @@ This article applies to the following combination: | Component | Version or description | | --- | --- | -| Alauda Container Platform | 4.4.x | +| Alauda Container Platform | 4.3.x | | Plugin | `sriov-network-plugin` | -| Plugin package version | `sriov-network-plugin v4.4.1` | +| Plugin package version | `sriov-network-plugin v4.3.x` | | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | +| Multi-NIC base | ACP provides Multus capability; workloads reference the SR-IOV secondary network through NAD | -The ACP package enables only the SR-IOV CNI path. The image values for `ib-sriov-cni`, `ovs-cni`, and `rdma-cni` are empty, so the operator does not render these init containers in the `config-daemon` DaemonSet. +The ACP package enables only the SR-IOV CNI path. The `config-daemon` installs the SR-IOV CNI binary into the node CNI bin directory through the `sriov-cni` init container. The image values for `ib-sriov-cni`, `ovs-cni`, and `rdma-cni` are empty, so the operator does not render these init containers in the `config-daemon` DaemonSet. + +This solution delivers SR-IOV VF orchestration and Multus secondary-network attachment. A DPDK userspace data plane can be built on top of SR-IOV VFs, but it still requires additional VF driver binding, HugePages, CPU isolation, NUMA planning, workload image adaptation, and performance testing. These are not provided as out-of-the-box capabilities by this plugin package. ## Prerequisites @@ -61,11 +64,11 @@ kubectl label node feature.node.kubernetes.io/sriov-capable=true ### Install the plugin -This capability is delivered as an ACP 4.4.x feature. The plugin package version is `sriov-network-plugin v4.4.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. +This capability is delivered as an ACP 4.3.x feature. The plugin package version is `sriov-network-plugin v4.3.x`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. 1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. -2. Select the package whose compatible platform version is `v4.4` and whose plugin version is `v4.4.1`. -3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.4.1.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.4.1.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.4.1.tgz`. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.x`. +3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.3.x.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.3.x.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.3.x.tgz`. 4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. 5. Upload the downloaded plugin package to the target ACP platform. @@ -76,7 +79,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.4.1.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.x.tgz" unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY @@ -93,11 +96,11 @@ After the upload succeeds, verify on the global cluster that the plugin version ```bash kubectl get moduleplugin sriov-network-plugin \ -o jsonpath='{.status.latestVersion}{"\n"}' -kubectl get moduleconfig sriov-network-plugin-v4.4.1 \ +kubectl get moduleconfig sriov-network-plugin- \ -o jsonpath='{.status.readyForDeploy}{"\n"}' ``` -The expected output is `v4.4.1` and `true`. If the `ModulePlugin` exists but no `ModuleConfig` is generated, or if the `ModuleConfig` is not `readyForDeploy=true`, the plugin package metadata is incomplete. Common causes are a missing `ModulePlugin.spec.logo` or a missing platform installation config file, `scripts/plugin-config.yaml`, in the package. Upload the complete package published from the AC marketplace instead. +The expected output is `v4.3.x` and `true`. If the `ModulePlugin` exists but no `ModuleConfig` is generated, or if the `ModuleConfig` is not `readyForDeploy=true`, the plugin package metadata is incomplete. Common causes are a missing `ModulePlugin.spec.logo` or a missing platform installation config file, `scripts/plugin-config.yaml`, in the package. Upload the complete package published from the AC marketplace instead. After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. The following commands use this variable: @@ -123,6 +126,16 @@ kubectl label namespace "$SRIOV_NAMESPACE" \ --overwrite ``` +### Confirm the Multus base + +SR-IOV networks are attached to Pods or KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm that the cluster has the NAD CRD: + +```bash +kubectl get crd network-attachment-definitions.k8s.cni.cncf.io +``` + +If this CRD does not exist, the Multus base is not ready. Install or enable ACP Multus capability before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. + After installation, confirm that the operator and config-daemon are running: ```bash @@ -350,6 +363,19 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide Inside the guest operating system, confirm that an additional NIC appears, then configure an IP address according to the network design or use DHCP if available. +### Relationship with DPDK and support boundary + +The upstream `sriov-cni` can assign SR-IOV VFs to workloads. After those VFs are bound to userspace drivers such as `vfio-pci`, they can become part of a DPDK data path. Therefore, this solution provides the SR-IOV attachment base for a DPDK high-performance network path, but it is not a complete DPDK solution by itself. + +If the customer workload explicitly requires DPDK, validate the following items in addition to this solution: + +- Node BIOS settings, IOMMU, VFIO driver, HugePages, CPU isolation, and NUMA affinity. +- Whether VFs use kernel `netdevice` mode or are bound to `vfio-pci` for DPDK userspace processes. +- Whether the DPDK path is direct SR-IOV VF consumption by the workload, kube-ovn OVS-DPDK, Userspace CNI, vhostuser, or another integration pattern. +- Performance baselines for PPS, bandwidth, latency, and jitter using the customer CNF image or tools such as `testpmd` and Trex. + +Do not describe this plugin package as a complete DPDK productized delivery before the hardware, driver, resource-isolation, and performance-validation work is complete. + ## Diagnostic steps ### Operator or config-daemon is not running @@ -418,4 +444,4 @@ If the event reports insufficient resources, verify that `SriovNetworkNodePolicy - An environment without SR-IOV hardware can only validate the control-plane smoke path. It cannot prove VF creation, device-plugin resource advertisement, or VM data-plane connectivity. - This solution does not replace the cluster primary CNI. kube-ovn continues to provide the Pod primary network, while SR-IOV is used as a Multus secondary network. -- The current ACP package does not deploy `ovs-cni`, `rdma-cni`, or `ib-sriov-cni`. If the workload requires OVS, RDMA, or InfiniBand SR-IOV, evaluate the required images, operator rendering logic, and validation scope separately. +- The current ACP package does not deploy `ovs-cni`, `rdma-cni`, or `ib-sriov-cni`. If the workload requires OVS, RDMA, or InfiniBand SR-IOV, evaluate the required images, operator rendering logic, and validation scope separately. \ No newline at end of file diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index b0d1747ca..4a20384d5 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -4,16 +4,16 @@ kind: products: - Alauda Container Platform ProductsVersion: - - '4.4.x' + - '4.3.x' --- -# 如何在 ACP 上为 KubeVirt 虚拟机启用 SR-IOV 网卡直通 +# 如何在 ACP 上通过 Multus 和 SR-IOV 为业务 Pod 和 KubeVirt 虚拟机提供高性能辅助网卡 ## 问题 -用户在 Alauda Container Platform 4.4.x 上运行 KubeVirt 虚拟机，并希望给虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；SR-IOV Network Operator 只负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并为 Multus 生成对应的 `NetworkAttachmentDefinition`。 +用户在 Alauda Container Platform 4.3.x 上运行 KubeVirt 虚拟机或容器化网络功能（CNF），并希望通过 Multus 给业务 Pod 或虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 -本文从用户使用路径说明如何安装 ACP 4.4.x 的 `sriov-network-plugin`，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 VF 和虚拟机数据面验证。 +本文从用户使用路径说明如何在 ACP 4.3.x 上完成 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 Pod 和 KubeVirt 虚拟机数据面验证。 ## 环境 @@ -21,14 +21,17 @@ ProductsVersion: | 组件 | 版本或说明 | | --- | --- | -| Alauda Container Platform | 4.4.x | +| Alauda Container Platform | 4.3.x | | 插件 | `sriov-network-plugin` | -| 插件包版本 | `sriov-network-plugin v4.4.1` | +| 插件包版本 | `sriov-network-plugin v4.3.x` | | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | +| 多网卡基座 | ACP 已提供 Multus 能力，业务通过 NAD 引用 SR-IOV 辅助网络 | -ACP 打包版本只启用 SR-IOV CNI 路径。`ib-sriov-cni`、`ovs-cni`、`rdma-cni` 镜像值为空，operator 渲染 `config-daemon` 时不会部署这些 init containers。 +ACP 打包版本只启用 SR-IOV CNI 路径。`config-daemon` 会通过 `sriov-cni` init container 将 SR-IOV CNI 二进制安装到节点 CNI bin 目录；`ib-sriov-cni`、`ovs-cni`、`rdma-cni` 镜像值为空，operator 渲染 `config-daemon` 时不会部署这些 init containers。 + +本方案交付的是 SR-IOV VF 编排和 Multus 辅助网络接入能力。DPDK 用户态数据面可以基于 SR-IOV VF 继续扩展，但还需要额外完成 VF 驱动绑定、HugePages、CPU 隔离、NUMA 规划、业务镜像适配和性能压测；这些不属于本插件包开箱即用能力。 ## 先决条件 @@ -61,11 +64,11 @@ kubectl label node feature.node.kubernetes.io/sriov-capable=true ### 安装插件 -该能力作为 ACP 4.4.x 新功能交付，插件包版本为 `sriov-network-plugin v4.4.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 +该能力作为 ACP 4.3.x 新功能交付，插件包版本为 `sriov-network-plugin v4.3.x`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 -2. 选择适配平台版本为 `v4.4`、插件版本为 `v4.4.1` 的安装包。 -3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.4.1.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.4.1.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.4.1.tgz`。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.x` 的安装包。 +3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.3.x.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.3.x.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.3.x.tgz`。 4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 5. 将下载的插件包上传到目标 ACP 平台。 @@ -76,7 +79,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.4.1.tgz" +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.x.tgz" unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY @@ -93,11 +96,11 @@ violet push "$PACKAGE_FILE" \ ```bash kubectl get moduleplugin sriov-network-plugin \ -o jsonpath='{.status.latestVersion}{"\n"}' -kubectl get moduleconfig sriov-network-plugin-v4.4.1 \ +kubectl get moduleconfig sriov-network-plugin- \ -o jsonpath='{.status.readyForDeploy}{"\n"}' ``` -预期分别输出 `v4.4.1` 和 `true`。如果 `ModulePlugin` 存在但没有生成 `ModuleConfig`，或者 `ModuleConfig` 不是 `readyForDeploy=true`，说明插件包元数据不完整，常见原因是包内缺少 `ModulePlugin.spec.logo` 或平台安装配置 `scripts/plugin-config.yaml`，需要使用 AC 市场发布后的完整包重新上传。 +预期分别输出 `v4.3.x` 和 `true`。如果 `ModulePlugin` 存在但没有生成 `ModuleConfig`，或者 `ModuleConfig` 不是 `readyForDeploy=true`，说明插件包元数据不完整，常见原因是包内缺少 `ModulePlugin.spec.logo` 或平台安装配置 `scripts/plugin-config.yaml`，需要使用 AC 市场发布后的完整包重新上传。 上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。后续命令使用以下变量： @@ -123,6 +126,16 @@ kubectl label namespace "$SRIOV_NAMESPACE" \ --overwrite ``` +### 确认 Multus 基座可用 + +SR-IOV 网络通过 Multus 作为辅助网卡接入 Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应确认集群已经具备 NAD CRD： + +```bash +kubectl get crd network-attachment-definitions.k8s.cni.cncf.io +``` + +如果该 CRD 不存在，说明 Multus 基座尚未就绪，应先安装或启用 ACP 的 Multus 能力，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 + 安装后确认 operator 和 config-daemon 已运行： ```bash @@ -350,6 +363,19 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide 进入虚拟机操作系统后，确认出现额外网卡，并按业务网络规划配置 IP 或使用 DHCP 获取地址。 +### 与 DPDK 的关系和边界 + +上游 `sriov-cni` 可用于给工作负载分配 SR-IOV VF；VF 进一步绑定到 `vfio-pci` 等用户态驱动后，可以作为 DPDK 数据面的一部分使用。因此，本方案是 DPDK 高性能网络链路中的 SR-IOV 接入基础，但并不等同于完整 DPDK 方案。 + +如果客户业务明确要求 DPDK，需要在本方案之外继续确认并验证以下内容： + +- 节点 BIOS、IOMMU、VFIO 驱动、HugePages、CPU 隔离和 NUMA 亲和性。 +- VF 是使用内核 `netdevice` 模式，还是绑定 `vfio-pci` 供 DPDK 用户态进程使用。 +- DPDK 路径是业务容器直接消费 SR-IOV VF，还是使用 kube-ovn OVS-DPDK、Userspace CNI、vhostuser 等方案。 +- 使用客户真实 CNF 镜像或 `testpmd`、Trex 等工具完成 PPS、带宽、时延和抖动基线压测。 + +在没有完成上述硬件、驱动、资源隔离和压测验证前，不应把本插件包描述为完整的 DPDK 产品化交付。 + ## 诊断步骤 ### operator 或 config-daemon 未运行 @@ -418,4 +444,4 @@ kubectl describe pod -n kubevirt - 无 SR-IOV 网卡环境只能完成控制面 smoke 验证，不能证明 VF 创建、device plugin 资源暴露或虚拟机数据面连通性。 - 本方案不替换集群主 CNI。kube-ovn 继续承担 Pod 主网络，SR-IOV 网络作为 Multus 辅助网络使用。 -- 当前 ACP 打包版本不部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。如果业务需要 OVS、RDMA 或 InfiniBand SR-IOV，需要单独评估镜像、operator 渲染逻辑和验证范围。 +- 当前 ACP 打包版本不部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。如果业务需要 OVS、RDMA 或 InfiniBand SR-IOV，需要单独评估镜像、operator 渲染逻辑和验证范围。 \ No newline at end of file From 5959b3f31a5a7899551d7a520b9a4daf4ee617ca Mon Sep 17 00:00:00 2001 From: changluyi <47097611+changluyi@users.noreply.github.com> Date: Tue, 7 Jul 2026 09:36:11 +0800 Subject: [PATCH 3/7] Remove unintended AGENTS.md from PR --- AGENTS.md | 4 ---- 1 file changed, 4 deletions(-) delete mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index ed9d29adf..000000000 --- a/AGENTS.md +++ /dev/null @@ -1,4 +0,0 @@ -# Project Memory - -- When adding or updating a solution document, keep the Chinese and English versions in sync. -- For files under `docs/zh/solutions/`, add or update the matching file under `docs/en/solutions/` with the same filename unless the user explicitly asks for Chinese-only work. From 42d9e373ecfc6ba351c50b53d6d3911be2830743 Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 7 Jul 2026 11:31:11 +0800 Subject: [PATCH 4/7] docs: refine SR-IOV KubeVirt guide --- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 298 ++++-------------- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 298 ++++-------------- 2 files changed, 118 insertions(+), 478 deletions(-) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 4c058fc92..8aaae5250 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -4,16 +4,16 @@ kind: products: - Alauda Container Platform ProductsVersion: - - '4.3.x' + - '4.3' --- # Provide High-Performance Secondary NICs for Application Pods and KubeVirt VMs with Multus and SR-IOV on ACP ## Issue -Users running KubeVirt VMs or containerized network functions (CNFs) on Alauda Container Platform 4.3.x may need to attach host SR-IOV VFs to Pods or VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to the workload, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. +Users running KubeVirt VMs or containerized network functions (CNFs) on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs to Pods or VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to the workload, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. -This article follows the user workflow for completing an end-to-end Multus + SR-IOV setup on ACP 4.3.x: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, validate the control plane without SR-IOV NICs, and validate Pod plus KubeVirt VM data-plane behavior on SR-IOV hardware. +This article follows the user workflow for completing an end-to-end Multus + SR-IOV setup on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, validate the control plane without SR-IOV NICs, and validate Pod plus KubeVirt VM data-plane behavior on SR-IOV hardware. ## Environment @@ -21,30 +21,20 @@ This article applies to the following combination: | Component | Version or description | | --- | --- | -| Alauda Container Platform | 4.3.x | +| Alauda Container Platform | 4.3 | | Plugin | `sriov-network-plugin` | -| Plugin package version | `sriov-network-plugin v4.3.x` | +| Plugin package version | `sriov-network-plugin v4.3.1` | | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | | Multi-NIC base | ACP provides Multus capability; workloads reference the SR-IOV secondary network through NAD | -The ACP package enables only the SR-IOV CNI path. The `config-daemon` installs the SR-IOV CNI binary into the node CNI bin directory through the `sriov-cni` init container. The image values for `ib-sriov-cni`, `ovs-cni`, and `rdma-cni` are empty, so the operator does not render these init containers in the `config-daemon` DaemonSet. +The ACP package enables the SR-IOV CNI path for SR-IOV VF orchestration and Multus secondary-network attachment. -This solution delivers SR-IOV VF orchestration and Multus secondary-network attachment. A DPDK userspace data plane can be built on top of SR-IOV VFs, but it still requires additional VF driver binding, HugePages, CPU isolation, NUMA planning, workload image adaptation, and performance testing. These are not provided as out-of-the-box capabilities by this plugin package. +This article covers installing and using the SR-IOV L5 plugin for KubeVirt VM secondary NICs. It does not cover OVS-DPDK, Userspace CNI, or DPDK application configuration inside containers. ## Prerequisites -### Platform and permissions - -Prepare a kubeconfig that can manage the target cluster. The current user must be able to create the following resources: - -- Namespaces, ServiceAccounts, ClusterRoles, and ClusterRoleBindings -- CRDs -- Deployments and DaemonSets -- Custom resources in the `sriovnetwork.openshift.io` API group -- `NetworkAttachmentDefinition` objects in `k8s.cni.cncf.io/v1` - ### Nodes and hardware SR-IOV hardware is not required if you only need to validate the plugin control plane. To complete VF and virtual-machine data-plane validation, prepare at least one worker node that meets these requirements: @@ -52,23 +42,16 @@ SR-IOV hardware is not required if you only need to validate the plugin control - The node has a physical NIC PF that supports SR-IOV. - IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. - The PF driver supports VF creation, and the PF is not held by the primary CNI or OVS in a way that prevents VF configuration. -- The target node can enter a maintenance window. Creating VFs or changing VF drivers can trigger node drain or a short network interruption. - -Label the nodes that will be configured for SR-IOV. `SriovNetworkNodePolicy` will use this label to limit its scope: - -```bash -kubectl label node feature.node.kubernetes.io/sriov-capable=true -``` ## Resolution ### Install the plugin -This capability is delivered as an ACP 4.3.x feature. The plugin package version is `sriov-network-plugin v4.3.x`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. +This capability is delivered as an ACP 4.3 feature. The plugin package version is `sriov-network-plugin v4.3.1`. The user workflow is to download the plugin package from the AC application marketplace, upload it to the target ACP platform, and then install it from the platform marketplace. 1. Log in to the AC application marketplace and search for `SR-IOV Network Plugin` or `sriov-network-plugin`. -2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.x`. -3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.3.x.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.3.x.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.3.x.tgz`. +2. Select the package whose compatible platform version is `v4.3` and whose plugin version is `v4.3.1`. +3. Download the package that matches the target platform architecture. For amd64 platforms, download `sriov-network-plugin.amd64.v4.3.1.tgz`; for arm64 platforms, download `sriov-network-plugin.arm64.v4.3.1.tgz`. If the platform does not require an architecture-specific package, download `sriov-network-plugin.ALL.v4.3.1.tgz`. 4. Keep the downloaded `.tgz` filename unchanged. `violet` parses the plugin name, architecture, and version from the filename; renaming the package can make the upload fail. 5. Upload the downloaded plugin package to the target ACP platform. @@ -79,9 +62,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.x.tgz" - -unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -91,55 +72,16 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -After the upload succeeds, verify on the global cluster that the plugin version configuration has been generated and is ready to deploy: - -```bash -kubectl get moduleplugin sriov-network-plugin \ - -o jsonpath='{.status.latestVersion}{"\n"}' -kubectl get moduleconfig sriov-network-plugin- \ - -o jsonpath='{.status.readyForDeploy}{"\n"}' -``` - -The expected output is `v4.3.x` and `true`. If the `ModulePlugin` exists but no `ModuleConfig` is generated, or if the `ModuleConfig` is not `readyForDeploy=true`, the plugin package metadata is incomplete. Common causes are a missing `ModulePlugin.spec.logo` or a missing platform installation config file, `scripts/plugin-config.yaml`, in the package. Upload the complete package published from the AC marketplace instead. - -After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. The following commands use this variable: - -```bash -export SRIOV_NAMESPACE="cpaas-system" -``` - -Before installation, confirm that the cluster does not already have a manually installed SR-IOV instance. If a previous test installation exists in another namespace, such as `sriov-network-operator`, uninstall that old instance with the original installation method before installing from the marketplace. Running two operators against the same SR-IOV custom resources is not recommended: - -```bash -kubectl get pods -A | grep sriov-network -kubectl get sriovoperatorconfig -A -kubectl get sriovnetworknodestate -A -``` - -The SR-IOV node daemon requires `hostNetwork`, `hostPID`, `hostPath`, and privileged containers. Before installation, confirm that the deployment namespace allows privileged Pod Security Admission. If the labels are missing, add them: - -```bash -kubectl label namespace "$SRIOV_NAMESPACE" \ - pod-security.kubernetes.io/enforce=privileged \ - pod-security.kubernetes.io/audit=privileged \ - pod-security.kubernetes.io/warn=privileged \ - --overwrite -``` +After the upload is complete, go to **Administrator -> Marketplace -> Cluster Plugins**, select version `v4.3.1` of `sriov-network-plugin`, and install it into the target business cluster. When installed through the ACP marketplace, the SR-IOV components are deployed in the `cpaas-system` namespace by default. ### Confirm the Multus base -SR-IOV networks are attached to Pods or KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm that the cluster has the NAD CRD: - -```bash -kubectl get crd network-attachment-definitions.k8s.cni.cncf.io -``` - -If this CRD does not exist, the Multus base is not ready. Install or enable ACP Multus capability before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. +SR-IOV networks are attached to Pods or KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the product documentation to [install Multus CNI](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks#%E5%AE%89%E8%A3%85-multus-cni) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. After installation, confirm that the operator and config-daemon are running: ```bash -kubectl get pods -n "$SRIOV_NAMESPACE" +kubectl get pods -n cpaas-system ``` Expected workloads: @@ -149,86 +91,71 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` -Confirm that the CRDs are registered: - -```bash -kubectl get crd | grep sriovnetwork.openshift.io -``` - -Confirm that `config-daemon` has only the `sriov-cni` init container and does not include `ib-sriov-cni`, `ovs-cni`, or `rdma-cni`: - -```bash -kubectl get daemonset sriov-network-config-daemon -n "$SRIOV_NAMESPACE" \ - -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' -``` - -Expected output: - -```text -sriov-cni build-harbor.alauda.cn/3rdparty/sriov/sriov-cni:v2.10.0 -``` - ### Control-plane validation without SR-IOV hardware -If the current environment has no SR-IOV NIC, the validation goal is to prove that the plugin deploys correctly, CRDs are registered, the operator can synchronize node state, and no extra `ovs-cni`, `rdma-cni`, or `ib-sriov-cni` init containers are deployed. +If the current environment has no SR-IOV NIC, the validation goal is to prove that the plugin deploys correctly and the operator can synchronize node state. Check `SriovNetworkNodeState`: ```bash kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n "$SRIOV_NAMESPACE" + -n cpaas-system ``` Check a specific node: ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{.status.syncStatus}{"\n"}' ``` -In an environment without SR-IOV hardware, `status.interfaces` can be empty. This is a hardware limitation and does not indicate a plugin deployment failure. If the operator, config-daemon, CRDs, and `syncStatus` are healthy, the control-plane smoke validation is sufficient. +In an environment without SR-IOV hardware, `status.interfaces` can be empty. If the operator, config-daemon, and `syncStatus` are healthy, the control-plane smoke validation is sufficient. ### VF validation with SR-IOV hardware On a node with an SR-IOV PF, first confirm that the operator discovers the physical NIC: ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' ``` -Select a PF, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. The following example creates four VFs and advertises a device-plugin resource named `intel_sriov_netdevice`: +The operator automatically discovers SR-IOV PFs on nodes and writes them to `SriovNetworkNodeState.status.interfaces`. It does not automatically decide which PF should create VFs, how many VFs to create, which `resourceName` to use, or which VF type to configure. To create VFs and advertise resources through the device plugin, create a `SriovNetworkNodePolicy`. + +Select a PF, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. The following example uses `kubernetes.io/hostname` to select one node, creates four VFs, and advertises a device-plugin resource named `sriov_vf`: ```yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: - name: intel-sriov-netdevice + name: sriov-vf-policy namespace: cpaas-system spec: - resourceName: intel_sriov_netdevice + resourceName: sriov_vf nodeSelector: - feature.node.kubernetes.io/sriov-capable: "true" + kubernetes.io/hostname: nicSelector: pfNames: - ens5f0 numVfs: 4 - deviceType: netdevice + deviceType: vfio-pci mtu: 1500 ``` +`deviceType: vfio-pci` is used for KubeVirt SR-IOV PCI passthrough. The operator configures the VF driver according to this policy and exposes the resource through the device plugin. Do not run `dpdk-devbind.py` inside the VM for host VFs. + Apply the policy and watch node synchronization: ```bash kubectl apply -f sriov-node-policy.yaml kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n "$SRIOV_NAMESPACE" + -n cpaas-system ``` Confirm that the target node reaches `Succeeded`: ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' ``` @@ -236,14 +163,14 @@ Confirm that the SR-IOV device-plugin resource appears in node allocatable resou ```bash kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vf}{"\n"}' ``` If the output is a positive integer, the VF resource is available to the Kubernetes scheduler. ### Create an SR-IOV secondary network -Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `kubevirt`: +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `kubevirt` and uses Kube-OVN IPAM to assign an address to the SR-IOV secondary NIC: ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -253,76 +180,50 @@ metadata: namespace: cpaas-system spec: networkNamespace: kubevirt - resourceName: intel_sriov_netdevice + resourceName: sriov_vf vlan: 0 ipam: | { - "type": "host-local", - "subnet": "192.168.100.0/24", - "rangeStart": "192.168.100.100", - "rangeEnd": "192.168.100.200", - "gateway": "192.168.100.1" + "type": "kube-ovn", + "server_socket": "/run/openvswitch/kube-ovn-daemon.sock", + "provider": "vm-sriov-net.kubevirt.ovn" } ``` -Apply the object and confirm that the NAD is generated: - -```bash -kubectl apply -f sriov-network.yaml -kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net -``` - -Inspect the effective NAD CNI configuration: - -```bash -kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ - -o jsonpath='{.spec.config}{"\n"}' | jq . -``` - -### Validate VF allocation with a test Pod +The `provider` uses the `..ovn` format. In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace, so the provider is `vm-sriov-net.kubevirt.ovn`. -Create a test Pod in the application namespace. The Pod must request the SR-IOV resource and reference the NAD: +Create a Kube-OVN Subnet that uses the same provider. Adjust `cidrBlock`, `gateway`, and `excludeIps` according to the application network plan: ```yaml -apiVersion: v1 -kind: Pod +apiVersion: kubeovn.io/v1 +kind: Subnet metadata: - name: sriov-test - namespace: kubevirt - annotations: - k8s.v1.cni.cncf.io/networks: vm-sriov-net + name: vm-sriov-net spec: - restartPolicy: Never - containers: - - name: test - image: - command: - - sleep - - "3600" - resources: - requests: - openshift.io/intel_sriov_netdevice: "1" - limits: - openshift.io/intel_sriov_netdevice: "1" + protocol: IPv4 + enableDHCP: true + provider: vm-sriov-net.kubevirt.ovn + cidrBlock: 172.22.0.0/16 + gateway: 172.22.0.1 + excludeIps: + - 172.22.0.0..172.22.0.10 ``` -Confirm that the Pod is scheduled to an SR-IOV node and enters `Running`: +The KubeVirt VM default network still uses the kube-ovn primary network. The SR-IOV network is attached as a Multus secondary NIC, and its address is allocated by the Kube-OVN Subnet above. -```bash -kubectl get pod sriov-test -n kubevirt -o wide -``` - -Enter the Pod and inspect the secondary NIC: +Apply the objects and confirm that the NAD is generated: ```bash -kubectl exec -n kubevirt sriov-test -- ip link -kubectl exec -n kubevirt sriov-test -- ip addr +kubectl apply -f sriov-network.yaml +kubectl apply -f sriov-subnet.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net ``` -If the underlay network is ready, continue with ping or application traffic validation: +Inspect the effective NAD CNI configuration: ```bash -kubectl exec -n kubevirt sriov-test -- ping -c 3 192.168.100.1 +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . ``` ### Use the SR-IOV network in a KubeVirt VM @@ -354,6 +255,8 @@ spec: networkName: vm-sriov-net ``` +In this example, the VM and the NAD are both in the `kubevirt` namespace, so `networkName` can be the NAD name, `vm-sriov-net`. To reference a NAD in another namespace, use the `/` format. + After creating the VM, confirm that the virt-launcher Pod is scheduled to a node with VF resources and inspect the VMI status: ```bash @@ -361,87 +264,4 @@ kubectl get vmi -n kubevirt sriov-vm kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ``` -Inside the guest operating system, confirm that an additional NIC appears, then configure an IP address according to the network design or use DHCP if available. - -### Relationship with DPDK and support boundary - -The upstream `sriov-cni` can assign SR-IOV VFs to workloads. After those VFs are bound to userspace drivers such as `vfio-pci`, they can become part of a DPDK data path. Therefore, this solution provides the SR-IOV attachment base for a DPDK high-performance network path, but it is not a complete DPDK solution by itself. - -If the customer workload explicitly requires DPDK, validate the following items in addition to this solution: - -- Node BIOS settings, IOMMU, VFIO driver, HugePages, CPU isolation, and NUMA affinity. -- Whether VFs use kernel `netdevice` mode or are bound to `vfio-pci` for DPDK userspace processes. -- Whether the DPDK path is direct SR-IOV VF consumption by the workload, kube-ovn OVS-DPDK, Userspace CNI, vhostuser, or another integration pattern. -- Performance baselines for PPS, bandwidth, latency, and jitter using the customer CNF image or tools such as `testpmd` and Trex. - -Do not describe this plugin package as a complete DPDK productized delivery before the hardware, driver, resource-isolation, and performance-validation work is complete. - -## Diagnostic steps - -### Operator or config-daemon is not running - -Check Pod status and events: - -```bash -kubectl get pods -n "$SRIOV_NAMESPACE" -o wide -kubectl describe pod -n "$SRIOV_NAMESPACE" -kubectl logs -n "$SRIOV_NAMESPACE" deploy/sriov-network-operator -``` - -If `config-daemon` is stuck in init, first confirm that the init container list contains only `sriov-cni`. If `ovs-cni`, `rdma-cni`, or `ib-sriov-cni` still appears, the installed plugin package or operator image is not the ACP-corrected version. - -### The node does not discover an SR-IOV PF - -Check `SriovNetworkNodeState`: - -```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" -o yaml -``` - -If `status.interfaces` is empty, continue checking hardware and kernel state on the node: - -```bash -lspci -nn | grep -i ethernet -ip link show -``` - -Confirm that BIOS/IOMMU is enabled and that the target PF driver supports SR-IOV. - -### Policy synchronization fails - -Check synchronization status and errors: - -```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ - -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' -``` - -Common causes include: - -- `nodeSelector` does not match any node. -- `nicSelector.pfNames` is incorrect, or the PF name does not exist on the target node. -- The PF is held by another network component and cannot create VFs. -- IOMMU is not enabled, or the driver does not support the requested number of VFs. - -### Pod or VM cannot allocate a VF - -Confirm that the resource exists in node allocatable: - -```bash -kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' -``` - -Check Pod or virt-launcher events: - -```bash -kubectl describe pod -n kubevirt -``` - -If the event reports insufficient resources, verify that `SriovNetworkNodePolicy.spec.resourceName`, the Pod resource request, and `SriovNetwork.spec.resourceName` all use the same value. - -## Limitations - -- An environment without SR-IOV hardware can only validate the control-plane smoke path. It cannot prove VF creation, device-plugin resource advertisement, or VM data-plane connectivity. -- This solution does not replace the cluster primary CNI. kube-ovn continues to provide the Pod primary network, while SR-IOV is used as a Multus secondary network. -- The current ACP package does not deploy `ovs-cni`, `rdma-cni`, or `ib-sriov-cni`. If the workload requires OVS, RDMA, or InfiniBand SR-IOV, evaluate the required images, operator rendering logic, and validation scope separately. \ No newline at end of file +Inside the guest operating system, confirm that an additional NIC appears. The Kube-OVN Subnet handles platform-side address allocation for the secondary network. Whether the address is configured inside the guest still depends on the guest OS DHCP client, cloud-init, or system network configuration. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 4a20384d5..edb1e54d6 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -4,16 +4,16 @@ kind: products: - Alauda Container Platform ProductsVersion: - - '4.3.x' + - '4.3' --- # 如何在 ACP 上通过 Multus 和 SR-IOV 为业务 Pod 和 KubeVirt 虚拟机提供高性能辅助网卡 ## 问题 -用户在 Alauda Container Platform 4.3.x 上运行 KubeVirt 虚拟机或容器化网络功能（CNF），并希望通过 Multus 给业务 Pod 或虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 +用户在 Alauda Container Platform 4.3 上运行 KubeVirt 虚拟机或容器化网络功能（CNF），并希望通过 Multus 给业务 Pod 或虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 -本文从用户使用路径说明如何在 ACP 4.3.x 上完成 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 Pod 和 KubeVirt 虚拟机数据面验证。 +本文从用户使用路径说明如何在 ACP 4.3 上完成 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 Pod 和 KubeVirt 虚拟机数据面验证。 ## 环境 @@ -21,30 +21,20 @@ ProductsVersion: | 组件 | 版本或说明 | | --- | --- | -| Alauda Container Platform | 4.3.x | +| Alauda Container Platform | 4.3 | | 插件 | `sriov-network-plugin` | -| 插件包版本 | `sriov-network-plugin v4.3.x` | +| 插件包版本 | `sriov-network-plugin v4.3.1` | | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | | 多网卡基座 | ACP 已提供 Multus 能力，业务通过 NAD 引用 SR-IOV 辅助网络 | -ACP 打包版本只启用 SR-IOV CNI 路径。`config-daemon` 会通过 `sriov-cni` init container 将 SR-IOV CNI 二进制安装到节点 CNI bin 目录；`ib-sriov-cni`、`ovs-cni`、`rdma-cni` 镜像值为空，operator 渲染 `config-daemon` 时不会部署这些 init containers。 +ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Multus 辅助网络接入。 -本方案交付的是 SR-IOV VF 编排和 Multus 辅助网络接入能力。DPDK 用户态数据面可以基于 SR-IOV VF 继续扩展，但还需要额外完成 VF 驱动绑定、HugePages、CPU 隔离、NUMA 规划、业务镜像适配和性能压测；这些不属于本插件包开箱即用能力。 +本文覆盖 SR-IOV L5 插件在 KubeVirt 虚拟机辅助网卡场景中的安装与使用；不覆盖 OVS-DPDK、Userspace CNI 或容器内 DPDK 应用的配置。 ## 先决条件 -### 平台和权限 - -准备一个可管理目标集群的 kubeconfig，并确保当前用户可以创建以下资源： - -- 命名空间、ServiceAccount、ClusterRole、ClusterRoleBinding -- CRD -- Deployment、DaemonSet -- `sriovnetwork.openshift.io` API 组下的自定义资源 -- `k8s.cni.cncf.io/v1` 的 `NetworkAttachmentDefinition` - ### 节点和硬件 如果只验证插件控制面，可以没有 SR-IOV 网卡。如果要完成 VF 和虚拟机数据面验证，至少需要一个满足以下条件的 worker 节点： @@ -52,23 +42,16 @@ ACP 打包版本只启用 SR-IOV CNI 路径。`config-daemon` 会通过 `sriov-c - 节点上有支持 SR-IOV 的物理网卡 PF。 - BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 - PF 驱动支持创建 VF，并且该 PF 未被主 CNI 或 OVS 以不可释放方式占用。 -- 计划配置 VF 的节点可以进入维护窗口。创建 VF 或切换 VF 驱动可能触发节点 drain 或网络短暂中断。 - -给参与 SR-IOV 配置的节点打标签，后续 `SriovNetworkNodePolicy` 使用该标签限制作用范围： - -```bash -kubectl label node feature.node.kubernetes.io/sriov-capable=true -``` ## 解决方案 ### 安装插件 -该能力作为 ACP 4.3.x 新功能交付，插件包版本为 `sriov-network-plugin v4.3.x`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 +该能力作为 ACP 4.3 新功能交付，插件包版本为 `sriov-network-plugin v4.3.1`。用户侧从 AC 应用市场获取插件包，再上传到目标 ACP 平台安装。 1. 登录 AC 应用市场，搜索 `SR-IOV 网络插件` 或 `sriov-network-plugin`。 -2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.x` 的安装包。 -3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.3.x.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.3.x.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.3.x.tgz`。 +2. 选择适配平台版本为 `v4.3`、插件版本为 `v4.3.1` 的安装包。 +3. 下载与目标平台架构匹配的包。amd64 平台下载 `sriov-network-plugin.amd64.v4.3.1.tgz`，arm64 平台下载 `sriov-network-plugin.arm64.v4.3.1.tgz`；如果平台不需要区分架构，再下载 `sriov-network-plugin.ALL.v4.3.1.tgz`。 4. 保持下载后的 `.tgz` 文件名不变。`violet` 会根据文件名解析插件名、架构和版本，重命名可能导致上传失败。 5. 将下载的插件包上传到目标 ACP 平台。 @@ -79,9 +62,7 @@ export PLATFORM_URL="" export USERNAME="" export PASSWORD="" export CLUSTER_NAME="" -export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.x.tgz" - -unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY +export PACKAGE_FILE="sriov-network-plugin.amd64.v4.3.1.tgz" violet push "$PACKAGE_FILE" \ --platform-address "$PLATFORM_URL" \ @@ -91,55 +72,16 @@ violet push "$PACKAGE_FILE" \ --target-catalog-source platform ``` -上传成功后，在 global 集群确认插件版本配置已经生成并可部署： - -```bash -kubectl get moduleplugin sriov-network-plugin \ - -o jsonpath='{.status.latestVersion}{"\n"}' -kubectl get moduleconfig sriov-network-plugin- \ - -o jsonpath='{.status.readyForDeploy}{"\n"}' -``` - -预期分别输出 `v4.3.x` 和 `true`。如果 `ModulePlugin` 存在但没有生成 `ModuleConfig`，或者 `ModuleConfig` 不是 `readyForDeploy=true`，说明插件包元数据不完整，常见原因是包内缺少 `ModulePlugin.spec.logo` 或平台安装配置 `scripts/plugin-config.yaml`，需要使用 AC 市场发布后的完整包重新上传。 - -上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。后续命令使用以下变量： - -```bash -export SRIOV_NAMESPACE="cpaas-system" -``` - -安装前确认集群中没有历史手工安装的 SR-IOV 实例。如果曾经在 `sriov-network-operator` 等其他命名空间测试安装过，应先按原安装方式卸载旧实例，避免两个 operator 同时协调同一组 SR-IOV 自定义资源： - -```bash -kubectl get pods -A | grep sriov-network -kubectl get sriovoperatorconfig -A -kubectl get sriovnetworknodestate -A -``` - -SR-IOV 节点守护进程需要使用 `hostNetwork`、`hostPID`、`hostPath` 和 privileged 容器。安装前，确认部署命名空间允许 privileged Pod Security Admission；如果标签缺失，可以补齐： - -```bash -kubectl label namespace "$SRIOV_NAMESPACE" \ - pod-security.kubernetes.io/enforce=privileged \ - pod-security.kubernetes.io/audit=privileged \ - pod-security.kubernetes.io/warn=privileged \ - --overwrite -``` +上传完成后，进入 **管理员 -> 市场 -> 集群插件**，选择 `sriov-network-plugin` 的 `v4.3.1` 版本并安装到目标业务集群。通过 ACP 市场安装时，SR-IOV 组件默认部署在 `cpaas-system` 命名空间。 ### 确认 Multus 基座可用 -SR-IOV 网络通过 Multus 作为辅助网卡接入 Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应确认集群已经具备 NAD CRD： - -```bash -kubectl get crd network-attachment-definitions.k8s.cni.cncf.io -``` - -如果该 CRD 不存在，说明 Multus 基座尚未就绪，应先安装或启用 ACP 的 Multus 能力，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 +SR-IOV 网络通过 Multus 作为辅助网卡接入 Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装，先参考产品文档[安装 Multus CNI](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks#%E5%AE%89%E8%A3%85-multus-cni)，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 安装后确认 operator 和 config-daemon 已运行： ```bash -kubectl get pods -n "$SRIOV_NAMESPACE" +kubectl get pods -n cpaas-system ``` 预期至少看到以下工作负载处于 `Running`： @@ -149,86 +91,71 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` -确认 CRD 已注册： - -```bash -kubectl get crd | grep sriovnetwork.openshift.io -``` - -确认 `config-daemon` 只包含 `sriov-cni` init container，不包含 `ib-sriov-cni`、`ovs-cni`、`rdma-cni`： - -```bash -kubectl get daemonset sriov-network-config-daemon -n "$SRIOV_NAMESPACE" \ - -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{"\t"}{.image}{"\n"}{end}' -``` - -预期只出现类似输出： - -```text -sriov-cni build-harbor.alauda.cn/3rdparty/sriov/sriov-cni:v2.10.0 -``` - ### 无 SR-IOV 网卡环境的控制面验证 -如果当前环境没有 SR-IOV 网卡，验证目标是证明插件可以正常部署、CRD 可以注册、operator 可以同步节点状态，并且不会额外部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。 +如果当前环境没有 SR-IOV 网卡，验证目标是证明插件可以正常部署，并且 operator 可以同步节点状态。 检查 `SriovNetworkNodeState`： ```bash kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n "$SRIOV_NAMESPACE" + -n cpaas-system ``` 查看具体节点状态： ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{.status.syncStatus}{"\n"}' ``` -在无 SR-IOV 网卡环境中，`status.interfaces` 可能为空；这是硬件条件限制，不代表插件部署失败。只要 operator、config-daemon、CRD 和 `syncStatus` 正常，即可认为控制面 smoke 验证通过。 +在无 SR-IOV 网卡环境中，`status.interfaces` 可能为空。只要 operator、config-daemon 和 `syncStatus` 正常，即可认为控制面 smoke 验证通过。 ### 有 SR-IOV 网卡环境的 VF 验证 在带 SR-IOV PF 的节点上，先确认 operator 能发现物理网卡： ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{range .status.interfaces[*]}{.name}{"\t"}{.pciAddress}{"\t"}{.vendor}{"\t"}{.deviceID}{"\t"}{.totalVfs}{"\n"}{end}' ``` -选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。以下示例创建 4 个 VF，并通过 device plugin 暴露名为 `intel_sriov_netdevice` 的资源： +operator 会自动发现节点上的 SR-IOV PF，并写入 `SriovNetworkNodeState.status.interfaces`；但不会自动决定在哪个 PF 上创建多少 VF、使用什么 `resourceName` 或 VF 类型。要创建 VF 并通过 device plugin 暴露资源，需要创建 `SriovNetworkNodePolicy`。 + +选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。以下示例使用 `kubernetes.io/hostname` 选中单个节点，创建 4 个 VF，并通过 device plugin 暴露名为 `sriov_vf` 的资源： ```yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: - name: intel-sriov-netdevice + name: sriov-vf-policy namespace: cpaas-system spec: - resourceName: intel_sriov_netdevice + resourceName: sriov_vf nodeSelector: - feature.node.kubernetes.io/sriov-capable: "true" + kubernetes.io/hostname: nicSelector: pfNames: - ens5f0 numVfs: 4 - deviceType: netdevice + deviceType: vfio-pci mtu: 1500 ``` +`deviceType: vfio-pci` 用于 KubeVirt SR-IOV PCI 直通场景。operator 会根据该策略配置 VF 驱动并通过 device plugin 暴露资源；不要在虚拟机内部对宿主机 VF 执行 `dpdk-devbind.py`。 + 应用后观察节点同步状态： ```bash kubectl apply -f sriov-node-policy.yaml kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ - -n "$SRIOV_NAMESPACE" + -n cpaas-system ``` 确认目标节点变为 `Succeeded`： ```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ +kubectl get sriovnetworknodestate -n cpaas-system \ -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' ``` @@ -236,14 +163,14 @@ kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ ```bash kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' + -o jsonpath='{.status.allocatable.openshift\.io/sriov_vf}{"\n"}' ``` 如果输出为正整数，说明 VF 已经被 device plugin 暴露给 Kubernetes 调度器。 ### 创建 SR-IOV 辅助网络 -创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `kubevirt`： +创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `kubevirt`，并使用 kube-ovn IPAM 给 SR-IOV 辅助网卡分配地址： ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -253,76 +180,50 @@ metadata: namespace: cpaas-system spec: networkNamespace: kubevirt - resourceName: intel_sriov_netdevice + resourceName: sriov_vf vlan: 0 ipam: | { - "type": "host-local", - "subnet": "192.168.100.0/24", - "rangeStart": "192.168.100.100", - "rangeEnd": "192.168.100.200", - "gateway": "192.168.100.1" + "type": "kube-ovn", + "server_socket": "/run/openvswitch/kube-ovn-daemon.sock", + "provider": "vm-sriov-net.kubevirt.ovn" } ``` -应用后确认 NAD 已生成： - -```bash -kubectl apply -f sriov-network.yaml -kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net -``` - -查看 NAD 的有效 CNI 配置： - -```bash -kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ - -o jsonpath='{.spec.config}{"\n"}' | jq . -``` - -### 使用测试 Pod 验证 VF 分配 +其中 `provider` 使用 `..ovn` 格式。上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD，因此 provider 为 `vm-sriov-net.kubevirt.ovn`。 -在业务命名空间创建测试 Pod，显式请求 SR-IOV 资源并引用 NAD： +创建与该 provider 匹配的 Kube-OVN Subnet。`cidrBlock`、`gateway` 和 `excludeIps` 按业务网络规划调整： ```yaml -apiVersion: v1 -kind: Pod +apiVersion: kubeovn.io/v1 +kind: Subnet metadata: - name: sriov-test - namespace: kubevirt - annotations: - k8s.v1.cni.cncf.io/networks: vm-sriov-net + name: vm-sriov-net spec: - restartPolicy: Never - containers: - - name: test - image: - command: - - sleep - - "3600" - resources: - requests: - openshift.io/intel_sriov_netdevice: "1" - limits: - openshift.io/intel_sriov_netdevice: "1" + protocol: IPv4 + enableDHCP: true + provider: vm-sriov-net.kubevirt.ovn + cidrBlock: 172.22.0.0/16 + gateway: 172.22.0.1 + excludeIps: + - 172.22.0.0..172.22.0.10 ``` -确认 Pod 被调度到 SR-IOV 节点并进入 `Running`： +KubeVirt 虚拟机的默认网络仍然使用 kube-ovn 主网络；SR-IOV 网络作为 Multus 辅助网卡接入，并通过上述 kube-ovn Subnet 完成辅助网卡 IPAM。 -```bash -kubectl get pod sriov-test -n kubevirt -o wide -``` - -进入 Pod 检查辅助网卡： +应用后确认 NAD 已生成： ```bash -kubectl exec -n kubevirt sriov-test -- ip link -kubectl exec -n kubevirt sriov-test -- ip addr +kubectl apply -f sriov-network.yaml +kubectl apply -f sriov-subnet.yaml +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kubevirt vm-sriov-net ``` -如果网络侧已经配置好二层或三层连通性，可以继续执行 ping 或业务流量验证： +查看 NAD 的有效 CNI 配置： ```bash -kubectl exec -n kubevirt sriov-test -- ping -c 3 192.168.100.1 +kubectl get network-attachment-definitions.k8s.cni.cncf.io vm-sriov-net -n kubevirt \ + -o jsonpath='{.spec.config}{"\n"}' | jq . ``` ### 在 KubeVirt 虚拟机中使用 SR-IOV 网络 @@ -354,6 +255,8 @@ spec: networkName: vm-sriov-net ``` +上例中 VM 和 NAD 都在 `kubevirt` 命名空间，因此 `networkName` 可以只写 NAD 名称 `vm-sriov-net`。如果引用其他命名空间的 NAD，使用 `/` 格式。 + 创建 VM 后，确认 virt-launcher Pod 被调度到有 VF 资源的节点，并检查 VMI 状态： ```bash @@ -361,87 +264,4 @@ kubectl get vmi -n kubevirt sriov-vm kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ``` -进入虚拟机操作系统后，确认出现额外网卡，并按业务网络规划配置 IP 或使用 DHCP 获取地址。 - -### 与 DPDK 的关系和边界 - -上游 `sriov-cni` 可用于给工作负载分配 SR-IOV VF；VF 进一步绑定到 `vfio-pci` 等用户态驱动后，可以作为 DPDK 数据面的一部分使用。因此，本方案是 DPDK 高性能网络链路中的 SR-IOV 接入基础，但并不等同于完整 DPDK 方案。 - -如果客户业务明确要求 DPDK，需要在本方案之外继续确认并验证以下内容： - -- 节点 BIOS、IOMMU、VFIO 驱动、HugePages、CPU 隔离和 NUMA 亲和性。 -- VF 是使用内核 `netdevice` 模式，还是绑定 `vfio-pci` 供 DPDK 用户态进程使用。 -- DPDK 路径是业务容器直接消费 SR-IOV VF，还是使用 kube-ovn OVS-DPDK、Userspace CNI、vhostuser 等方案。 -- 使用客户真实 CNF 镜像或 `testpmd`、Trex 等工具完成 PPS、带宽、时延和抖动基线压测。 - -在没有完成上述硬件、驱动、资源隔离和压测验证前，不应把本插件包描述为完整的 DPDK 产品化交付。 - -## 诊断步骤 - -### operator 或 config-daemon 未运行 - -检查 Pod 状态和事件： - -```bash -kubectl get pods -n "$SRIOV_NAMESPACE" -o wide -kubectl describe pod -n "$SRIOV_NAMESPACE" -kubectl logs -n "$SRIOV_NAMESPACE" deploy/sriov-network-operator -``` - -如果 `config-daemon` 卡在 init 阶段，先确认 init container 列表是否只有 `sriov-cni`。如果仍然出现 `ovs-cni`、`rdma-cni` 或 `ib-sriov-cni`，说明安装的不是 ACP 修正后的插件包或 operator 镜像版本。 - -### 节点没有发现 SR-IOV PF - -检查 `SriovNetworkNodeState`： - -```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" -o yaml -``` - -如果 `status.interfaces` 为空，继续在节点侧确认硬件和内核状态： - -```bash -lspci -nn | grep -i ethernet -ip link show -``` - -确认 BIOS/IOMMU 已启用，并且目标 PF 驱动支持 SR-IOV。 - -### policy 同步失败 - -查看同步状态和错误： - -```bash -kubectl get sriovnetworknodestate -n "$SRIOV_NAMESPACE" \ - -o jsonpath='{.status.syncStatus}{"\n"}{.status.lastSyncError}{"\n"}' -``` - -常见原因包括： - -- `nodeSelector` 没有匹配任何节点。 -- `nicSelector.pfNames` 写错，或者 PF 名称在目标节点上不存在。 -- PF 已被其他网络组件占用，无法创建 VF。 -- 节点未启用 IOMMU 或驱动不支持请求的 VF 数量。 - -### Pod 或 VM 无法分配 VF - -确认节点 allocatable 中存在资源： - -```bash -kubectl get node \ - -o jsonpath='{.status.allocatable.openshift\.io/intel_sriov_netdevice}{"\n"}' -``` - -确认 Pod 或 virt-launcher 事件： - -```bash -kubectl describe pod -n kubevirt -``` - -如果事件提示资源不足，检查 `SriovNetworkNodePolicy.spec.resourceName`、Pod resource request、`SriovNetwork.spec.resourceName` 三者是否一致。 - -## 限制 - -- 无 SR-IOV 网卡环境只能完成控制面 smoke 验证，不能证明 VF 创建、device plugin 资源暴露或虚拟机数据面连通性。 -- 本方案不替换集群主 CNI。kube-ovn 继续承担 Pod 主网络，SR-IOV 网络作为 Multus 辅助网络使用。 -- 当前 ACP 打包版本不部署 `ovs-cni`、`rdma-cni`、`ib-sriov-cni`。如果业务需要 OVS、RDMA 或 InfiniBand SR-IOV，需要单独评估镜像、operator 渲染逻辑和验证范围。 \ No newline at end of file +进入虚拟机操作系统后，确认出现额外网卡。kube-ovn Subnet 负责平台侧辅助网络地址分配；虚拟机内部是否能拿到该地址，还取决于 guest OS 中的 DHCP 客户端、cloud-init 或系统网络配置。 From 06657998fea48a522a67b618f6711ffde23ea2db Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 7 Jul 2026 13:29:21 +0800 Subject: [PATCH 5/7] docs: add VM DPDK binding note --- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 27 +++++++++++++++++++ ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 27 +++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 8aaae5250..8fb36ea34 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -265,3 +265,30 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ``` Inside the guest operating system, confirm that an additional NIC appears. The Kube-OVN Subnet handles platform-side address allocation for the secondary network. Whether the address is configured inside the guest still depends on the guest OS DHCP client, cloud-init, or system network configuration. + +### Bind the service VF to DPDK inside the VM + +If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Use the `dpdk-devbind.py` script from the DPDK package, or download it from the `dpdk-devbind.py` link in the ACP SR-IOV documentation. + +Inside the VM, identify the PCI NICs: + +```bash +lspci -Dnn | grep -i ethernet +``` + +Prepare HugePages according to the workload requirements, and load the VFIO driver: + +```bash +modprobe vfio-pci +modprobe vfio_iommu_type1 +``` + +Bind the service VF PCI address as seen inside the VM to `vfio-pci`: + +```bash +python3 dpdk-devbind.py --status +python3 dpdk-devbind.py -b vfio-pci +python3 dpdk-devbind.py --status +``` + +`` is the PCI address seen inside the VM, not the VF address on the host. After binding, the VF is no longer used as a normal guest OS kernel NIC and is instead owned by the DPDK userspace process. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index edb1e54d6..3d366c1f5 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -265,3 +265,30 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ``` 进入虚拟机操作系统后，确认出现额外网卡。kube-ovn Subnet 负责平台侧辅助网络地址分配；虚拟机内部是否能拿到该地址，还取决于 guest OS 中的 DHCP 客户端、cloud-init 或系统网络配置。 + +### 在虚拟机内将业务 VF 绑定给 DPDK + +如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。可以使用 DPDK 软件包自带的 `dpdk-devbind.py`，也可以从 ACP SR-IOV 文档的 `dpdk-devbind.py` 下载入口获取脚本。 + +在虚拟机内确认 PCI 网卡： + +```bash +lspci -Dnn | grep -i ethernet +``` + +按业务应用要求准备 HugePages，并加载 VFIO 驱动： + +```bash +modprobe vfio-pci +modprobe vfio_iommu_type1 +``` + +将虚拟机内看到的业务 VF PCI 地址绑定到 `vfio-pci`： + +```bash +python3 dpdk-devbind.py --status +python3 dpdk-devbind.py -b vfio-pci +python3 dpdk-devbind.py --status +``` + +`` 是虚拟机内部看到的 PCI 地址，不是宿主机上的 VF 地址。绑定后，该 VF 不再作为 guest OS 的普通内核网卡使用，而是由 DPDK 用户态进程接管。 From 910241656c673512b96fa3e79b528692c07890dd Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 7 Jul 2026 13:45:10 +0800 Subject: [PATCH 6/7] docs: clarify VM DPDK VF binding --- ...eploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md | 7 ++++--- ...eploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 8fb36ea34..2a625c3fa 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -268,7 +268,7 @@ Inside the guest operating system, confirm that an additional NIC appears. The K ### Bind the service VF to DPDK inside the VM -If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Use the `dpdk-devbind.py` script from the DPDK package, or download it from the `dpdk-devbind.py` link in the ACP SR-IOV documentation. +If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Prefer the `dpdk-devbind.py` script from the DPDK package inside the VM. If the image does not include the script, get it from the DPDK upstream repository: . Inside the VM, identify the PCI NICs: @@ -276,11 +276,12 @@ Inside the VM, identify the PCI NICs: lspci -Dnn | grep -i ethernet ``` -Prepare HugePages according to the workload requirements, and load the VFIO driver: +`dpdk-devbind.py` only binds the VF driver and does not prepare the DPDK application runtime. Configure HugePages, CPU affinity, and DPDK EAL parameters according to the workload image or application documentation. + +Load the VFIO driver: ```bash modprobe vfio-pci -modprobe vfio_iommu_type1 ``` Bind the service VF PCI address as seen inside the VM to `vfio-pci`: diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 3d366c1f5..9a4b5ec06 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -268,7 +268,7 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide ### 在虚拟机内将业务 VF 绑定给 DPDK -如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。可以使用 DPDK 软件包自带的 `dpdk-devbind.py`，也可以从 ACP SR-IOV 文档的 `dpdk-devbind.py` 下载入口获取脚本。 +如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。优先使用虚拟机内 DPDK 软件包自带的 `dpdk-devbind.py`；如果镜像中没有该脚本，可以从 DPDK 官方仓库获取：。 在虚拟机内确认 PCI 网卡： @@ -276,11 +276,12 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide lspci -Dnn | grep -i ethernet ``` -按业务应用要求准备 HugePages，并加载 VFIO 驱动： +`dpdk-devbind.py` 只负责 VF 驱动绑定，不负责准备 DPDK 应用运行环境。HugePages、CPU 亲和性和 DPDK EAL 参数按业务镜像或应用文档配置。 + +加载 VFIO 驱动： ```bash modprobe vfio-pci -modprobe vfio_iommu_type1 ``` 将虚拟机内看到的业务 VF PCI 地址绑定到 `vfio-pci`： From 585bf30ea0846f73def685349ea0419538bf2c61 Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 7 Jul 2026 14:13:32 +0800 Subject: [PATCH 7/7] docs: improve SR-IOV VM guide clarity --- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 49 +++++++++---------- ...OV_Network_Operator_for_KubeVirt_on_ACP.md | 49 +++++++++---------- 2 files changed, 44 insertions(+), 54 deletions(-) diff --git a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 2a625c3fa..227d03641 100644 --- a/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/en/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -7,13 +7,13 @@ ProductsVersion: - '4.3' --- -# Provide High-Performance Secondary NICs for Application Pods and KubeVirt VMs with Multus and SR-IOV on ACP +# Provide High-Performance Secondary NICs for KubeVirt VMs with Multus and SR-IOV on ACP ## Issue -Users running KubeVirt VMs or containerized network functions (CNFs) on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs to Pods or VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to the workload, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. +Users running KubeVirt VMs on Alauda Container Platform 4.3 may need to attach host SR-IOV VFs to VMs as high-performance secondary NICs through Multus. The cluster primary CNI can remain kube-ovn. Multus attaches the secondary network to VMs, while SR-IOV Network Operator discovers SR-IOV PFs, creates VFs, advertises VF resources through the device plugin, and generates the `NetworkAttachmentDefinition` objects consumed by Multus. -This article follows the user workflow for completing an end-to-end Multus + SR-IOV setup on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, validate the control plane without SR-IOV NICs, and validate Pod plus KubeVirt VM data-plane behavior on SR-IOV hardware. +This article follows the user workflow for completing an end-to-end KubeVirt VM Multus + SR-IOV setup on ACP 4.3: install `sriov-network-plugin`, confirm the Multus/NAD base, configure `SriovNetworkNodePolicy`, generate `SriovNetwork`/NAD objects, and use the SR-IOV secondary NIC in a KubeVirt VM. The same SR-IOV capability can also be used by application Pods, but this article uses KubeVirt VMs as the example scenario. ## Environment @@ -27,7 +27,7 @@ This article applies to the following combination: | Upstream baseline | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | Deployment namespace | `cpaas-system` when installed through the ACP marketplace | | Primary CNI | kube-ovn can remain the primary CNI; SR-IOV is used as a Multus secondary network | -| Multi-NIC base | ACP provides Multus capability; workloads reference the SR-IOV secondary network through NAD | +| Multi-NIC base | ACP provides Multus capability; VMs reference the SR-IOV secondary network through NAD | The ACP package enables the SR-IOV CNI path for SR-IOV VF orchestration and Multus secondary-network attachment. @@ -37,7 +37,7 @@ This article covers installing and using the SR-IOV L5 plugin for KubeVirt VM se ### Nodes and hardware -SR-IOV hardware is not required if you only need to validate the plugin control plane. To complete VF and virtual-machine data-plane validation, prepare at least one worker node that meets these requirements: +Prepare at least one worker node that meets these requirements: - The node has a physical NIC PF that supports SR-IOV. - IOMMU is enabled in BIOS and the operating system, such as Intel VT-d or AMD-Vi. @@ -76,7 +76,7 @@ After the upload is complete, go to **Administrator -> Marketplace -> Cluster Pl ### Confirm the Multus base -SR-IOV networks are attached to Pods or KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the product documentation to [install Multus CNI](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks#%E5%AE%89%E8%A3%85-multus-cni) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. +SR-IOV networks are attached to KubeVirt VMs by Multus as secondary NICs. Before or after installing the SR-IOV plugin, confirm in **Administrator -> Marketplace -> Cluster Plugins** that Multus CNI is installed in the target business cluster. If it is not installed, follow the "Install Multus CNI" section in the product documentation for [multiple networks](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) before configuring SR-IOV networks. The SR-IOV plugin handles node-side VF orchestration, SR-IOV CNI installation, and SR-IOV-related NAD generation. It does not replace the Multus meta CNI. After installation, confirm that the operator and config-daemon are running: @@ -91,29 +91,18 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` -### Control-plane validation without SR-IOV hardware +### Configure VF resources -If the current environment has no SR-IOV NIC, the validation goal is to prove that the plugin deploys correctly and the operator can synchronize node state. +A PF is the physical NIC on a node. A VF is a virtual PCI NIC created from a PF and assigned to a VM. -Check `SriovNetworkNodeState`: +First list the node states synchronized by the operator, and select the target node from the `NAME` column: ```bash kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ -n cpaas-system ``` -Check a specific node: - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{.status.syncStatus}{"\n"}' -``` - -In an environment without SR-IOV hardware, `status.interfaces` can be empty. If the operator, config-daemon, and `syncStatus` are healthy, the control-plane smoke validation is sufficient. - -### VF validation with SR-IOV hardware - -On a node with an SR-IOV PF, first confirm that the operator discovers the physical NIC: +On a node with an SR-IOV PF, confirm that the operator discovers the physical NIC: ```bash kubectl get sriovnetworknodestate -n cpaas-system \ @@ -122,7 +111,9 @@ kubectl get sriovnetworknodestate -n cpaas-system \ The operator automatically discovers SR-IOV PFs on nodes and writes them to `SriovNetworkNodeState.status.interfaces`. It does not automatically decide which PF should create VFs, how many VFs to create, which `resourceName` to use, or which VF type to configure. To create VFs and advertise resources through the device plugin, create a `SriovNetworkNodePolicy`. -Select a PF, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. The following example uses `kubernetes.io/hostname` to select one node, creates four VFs, and advertises a device-plugin resource named `sriov_vf`: +Select a PF from the `status.interfaces[*].name` output, such as `ens5f0`, and create a `SriovNetworkNodePolicy`. `nodeSelector` only matches labels that already exist on nodes. Use `SriovNetworkNodeState` to identify the node that has the target SR-IOV PF, then use a stable existing node label to limit the policy scope. The following example uses `kubernetes.io/hostname` to select one node, creates four VFs, and advertises a device-plugin resource named `sriov_vf`. + +Save the following content as `sriov-node-policy.yaml`: ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -170,7 +161,9 @@ If the output is a positive integer, the VF resource is available to the Kuberne ### Create an SR-IOV secondary network -Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the application namespace `kubevirt` and uses Kube-OVN IPAM to assign an address to the SR-IOV secondary NIC: +Create a `SriovNetwork`. The operator generates the corresponding `NetworkAttachmentDefinition`. The following example creates the NAD in the VM namespace `kubevirt` and uses Kube-OVN IPAM to assign an address to the SR-IOV secondary NIC. + +Save the following content as `sriov-network.yaml`: ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -190,9 +183,11 @@ spec: } ``` -The `provider` uses the `..ovn` format. In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace, so the provider is `vm-sriov-net.kubevirt.ovn`. +The `provider` uses the `..ovn` format. In this example, the operator generates a NAD named `vm-sriov-net` in the `kubevirt` namespace, so the provider is `vm-sriov-net.kubevirt.ovn`. If the VM runs in another namespace, update `SriovNetwork.spec.networkNamespace`, Subnet `spec.provider`, and VM `metadata.namespace` together. + +Create a Kube-OVN Subnet that uses the same provider. Adjust `cidrBlock`, `gateway`, and `excludeIps` according to the application network plan. -Create a Kube-OVN Subnet that uses the same provider. Adjust `cidrBlock`, `gateway`, and `excludeIps` according to the application network plan: +Save the following content as `sriov-subnet.yaml`: ```yaml apiVersion: kubeovn.io/v1 @@ -266,7 +261,7 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide Inside the guest operating system, confirm that an additional NIC appears. The Kube-OVN Subnet handles platform-side address allocation for the secondary network. Whether the address is configured inside the guest still depends on the guest OS DHCP client, cloud-init, or system network configuration. -### Bind the service VF to DPDK inside the VM +### Optional: Bind the service VF to DPDK inside the VM If the workload needs DPDK inside the VM, operate only on the SR-IOV service VF that is passed through to the guest OS. Do not bind the default management NIC. Prefer the `dpdk-devbind.py` script from the DPDK package inside the VM. If the image does not include the script, get it from the DPDK upstream repository: . @@ -292,4 +287,4 @@ python3 dpdk-devbind.py -b vfio-pci python3 dpdk-devbind.py --status ``` -`` is the PCI address seen inside the VM, not the VF address on the host. After binding, the VF is no longer used as a normal guest OS kernel NIC and is instead owned by the DPDK userspace process. +`` is the PCI address seen inside the VM, not the VF address on the host. After binding, the VF is no longer used as a normal guest OS kernel NIC and is instead owned by the DPDK userspace process. If the service VF is owned by the DPDK application, subsequent IP configuration and packet processing are handled by that DPDK application. diff --git a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md index 9a4b5ec06..7fe740a3f 100644 --- a/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md +++ b/docs/zh/solutions/Deploying_SR_IOV_Network_Operator_for_KubeVirt_on_ACP.md @@ -7,13 +7,13 @@ ProductsVersion: - '4.3' --- -# 如何在 ACP 上通过 Multus 和 SR-IOV 为业务 Pod 和 KubeVirt 虚拟机提供高性能辅助网卡 +# 如何在 ACP 上通过 Multus 和 SR-IOV 为 KubeVirt 虚拟机提供高性能辅助网卡 ## 问题 -用户在 Alauda Container Platform 4.3 上运行 KubeVirt 虚拟机或容器化网络功能（CNF），并希望通过 Multus 给业务 Pod 或虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入工作负载，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 +用户在 Alauda Container Platform 4.3 上运行 KubeVirt 虚拟机，并希望通过 Multus 给虚拟机挂载宿主机 SR-IOV VF 作为高性能辅助网卡。集群主 CNI 仍然可以是 kube-ovn；Multus 负责把辅助网络接入虚拟机，SR-IOV Network Operator 负责发现 SR-IOV PF、创建 VF、通过 device plugin 暴露 VF 资源，并生成 Multus 使用的 `NetworkAttachmentDefinition`。 -本文从用户使用路径说明如何在 ACP 4.3 上完成 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并分别完成无 SR-IOV 网卡环境下的控制面验证，以及有 SR-IOV 网卡环境下的 Pod 和 KubeVirt 虚拟机数据面验证。 +本文从用户使用路径说明如何在 ACP 4.3 上完成 KubeVirt 虚拟机 Multus + SR-IOV 的端到端接入：安装 `sriov-network-plugin`、确认 Multus/NAD 基座、配置 `SriovNetworkNodePolicy`、生成 `SriovNetwork`/NAD，并在 KubeVirt 虚拟机中使用 SR-IOV 辅助网卡。同一 SR-IOV 能力也可用于业务 Pod，本文以 KubeVirt 虚拟机场景为例。 ## 环境 @@ -27,7 +27,7 @@ ProductsVersion: | 上游基线 | `k8snetworkplumbingwg/sriov-network-operator:v1.6.0` | | 部署命名空间 | `cpaas-system`（通过 ACP 市场安装时） | | 主 CNI | 可使用 kube-ovn；SR-IOV 作为 Multus 辅助网络 | -| 多网卡基座 | ACP 已提供 Multus 能力，业务通过 NAD 引用 SR-IOV 辅助网络 | +| 多网卡基座 | ACP 已提供 Multus 能力，虚拟机通过 NAD 引用 SR-IOV 辅助网络 | ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Multus 辅助网络接入。 @@ -37,7 +37,7 @@ ACP 打包版本启用 SR-IOV CNI 路径，用于完成 SR-IOV VF 编排和 Mult ### 节点和硬件 -如果只验证插件控制面，可以没有 SR-IOV 网卡。如果要完成 VF 和虚拟机数据面验证，至少需要一个满足以下条件的 worker 节点： +至少需要一个满足以下条件的 worker 节点： - 节点上有支持 SR-IOV 的物理网卡 PF。 - BIOS 和操作系统已经启用 IOMMU，例如 Intel VT-d 或 AMD-Vi。 @@ -76,7 +76,7 @@ violet push "$PACKAGE_FILE" \ ### 确认 Multus 基座可用 -SR-IOV 网络通过 Multus 作为辅助网卡接入 Pod 或 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装，先参考产品文档[安装 Multus CNI](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks#%E5%AE%89%E8%A3%85-multus-cni)，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 +SR-IOV 网络通过 Multus 作为辅助网卡接入 KubeVirt 虚拟机。安装 SR-IOV 插件前后，都应在 **管理员 -> 市场 -> 集群插件** 中确认目标业务集群已经安装 Multus CNI。如果尚未安装，先参考产品文档[多网络](https://docs.alauda.cn/container_platform/4.3/configure/networking/how_to/kube_ovn/multiple_networks) 中的“安装 Multus CNI”章节，再继续配置 SR-IOV 网络。SR-IOV 插件负责节点侧 VF 编排、SR-IOV CNI 安装和 SR-IOV 相关 NAD 生成，不替代 Multus 元 CNI。 安装后确认 operator 和 config-daemon 已运行： @@ -91,29 +91,18 @@ sriov-network-operator-xxxxx 1/1 Running sriov-network-config-daemon-xxxxx 1/1 Running ``` -### 无 SR-IOV 网卡环境的控制面验证 +### 配置 VF 资源 -如果当前环境没有 SR-IOV 网卡，验证目标是证明插件可以正常部署，并且 operator 可以同步节点状态。 +PF 是节点上的物理网卡，VF 是从 PF 创建出来并分配给虚拟机使用的虚拟 PCI 网卡。 -检查 `SriovNetworkNodeState`： +先查看 operator 已同步的节点状态，并从输出的 `NAME` 选择目标节点： ```bash kubectl get sriovnetworknodestates.sriovnetwork.openshift.io \ -n cpaas-system ``` -查看具体节点状态： - -```bash -kubectl get sriovnetworknodestate -n cpaas-system \ - -o jsonpath='{.status.syncStatus}{"\n"}' -``` - -在无 SR-IOV 网卡环境中，`status.interfaces` 可能为空。只要 operator、config-daemon 和 `syncStatus` 正常，即可认为控制面 smoke 验证通过。 - -### 有 SR-IOV 网卡环境的 VF 验证 - -在带 SR-IOV PF 的节点上，先确认 operator 能发现物理网卡： +在带 SR-IOV PF 的节点上，确认 operator 能发现物理网卡： ```bash kubectl get sriovnetworknodestate -n cpaas-system \ @@ -122,7 +111,9 @@ kubectl get sriovnetworknodestate -n cpaas-system \ operator 会自动发现节点上的 SR-IOV PF，并写入 `SriovNetworkNodeState.status.interfaces`；但不会自动决定在哪个 PF 上创建多少 VF、使用什么 `resourceName` 或 VF 类型。要创建 VF 并通过 device plugin 暴露资源，需要创建 `SriovNetworkNodePolicy`。 -选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。以下示例使用 `kubernetes.io/hostname` 选中单个节点，创建 4 个 VF，并通过 device plugin 暴露名为 `sriov_vf` 的资源： +从 `status.interfaces[*].name` 输出中选择一个 PF，例如 `ens5f0`，创建 `SriovNetworkNodePolicy`。`nodeSelector` 只匹配节点上已有的 label；先根据 `SriovNetworkNodeState` 确认有 SR-IOV PF 的节点，再使用该节点已有的稳定 label 限制策略作用范围。以下示例使用 `kubernetes.io/hostname` 选中单个节点，创建 4 个 VF，并通过 device plugin 暴露名为 `sriov_vf` 的资源。 + +将以下内容保存为 `sriov-node-policy.yaml`： ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -170,7 +161,9 @@ kubectl get node \ ### 创建 SR-IOV 辅助网络 -创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到业务命名空间 `kubevirt`，并使用 kube-ovn IPAM 给 SR-IOV 辅助网卡分配地址： +创建 `SriovNetwork`，由 operator 生成对应的 `NetworkAttachmentDefinition`。以下示例将 NAD 生成到虚拟机所在命名空间 `kubevirt`，并使用 kube-ovn IPAM 给 SR-IOV 辅助网卡分配地址。 + +将以下内容保存为 `sriov-network.yaml`： ```yaml apiVersion: sriovnetwork.openshift.io/v1 @@ -190,9 +183,11 @@ spec: } ``` -其中 `provider` 使用 `..ovn` 格式。上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD，因此 provider 为 `vm-sriov-net.kubevirt.ovn`。 +其中 `provider` 使用 `..ovn` 格式。上例中 operator 会在 `kubevirt` 命名空间生成名为 `vm-sriov-net` 的 NAD，因此 provider 为 `vm-sriov-net.kubevirt.ovn`。如果虚拟机运行在其他命名空间，需要同时替换 `SriovNetwork.spec.networkNamespace`、Subnet `spec.provider` 和 VM `metadata.namespace`。 + +创建与该 provider 匹配的 Kube-OVN Subnet。`cidrBlock`、`gateway` 和 `excludeIps` 按业务网络规划调整。 -创建与该 provider 匹配的 Kube-OVN Subnet。`cidrBlock`、`gateway` 和 `excludeIps` 按业务网络规划调整： +将以下内容保存为 `sriov-subnet.yaml`： ```yaml apiVersion: kubeovn.io/v1 @@ -266,7 +261,7 @@ kubectl get pod -n kubevirt -l kubevirt.io=virt-launcher -o wide 进入虚拟机操作系统后，确认出现额外网卡。kube-ovn Subnet 负责平台侧辅助网络地址分配；虚拟机内部是否能拿到该地址，还取决于 guest OS 中的 DHCP 客户端、cloud-init 或系统网络配置。 -### 在虚拟机内将业务 VF 绑定给 DPDK +### 在虚拟机内将业务 VF 绑定给 DPDK（可选） 如果业务需要在虚拟机内使用 DPDK，进入 guest OS 后只对 SR-IOV 直通进来的业务 VF 操作，不要绑定默认管理网卡。优先使用虚拟机内 DPDK 软件包自带的 `dpdk-devbind.py`；如果镜像中没有该脚本，可以从 DPDK 官方仓库获取：。 @@ -292,4 +287,4 @@ python3 dpdk-devbind.py -b vfio-pci python3 dpdk-devbind.py --status ``` -`` 是虚拟机内部看到的 PCI 地址，不是宿主机上的 VF 地址。绑定后，该 VF 不再作为 guest OS 的普通内核网卡使用，而是由 DPDK 用户态进程接管。 +`` 是虚拟机内部看到的 PCI 地址，不是宿主机上的 VF 地址。绑定后，该 VF 不再作为 guest OS 的普通内核网卡使用，而是由 DPDK 用户态进程接管。如果业务 VF 被 DPDK 应用接管，后续 IP 配置和报文处理由 DPDK 应用决定。