diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml index 42d6472503..2daf7d652a 100644 --- a/_topic_maps/_topic_map.yml +++ b/_topic_maps/_topic_map.yml @@ -1001,6 +1001,8 @@ Topics: File: using-dpdk-and-rdma - Name: Using pod-level bonding for secondary networks File: using-pod-level-bonding + - Name: Configuring hardware offloading + File: configuring-hardware-offloading - Name: OpenShift SDN default CNI network provider Dir: openshift_sdn Topics: diff --git a/modules/nw-operator-cr.adoc b/modules/nw-operator-cr.adoc index 304c36ec5a..9156e3f236 100644 --- a/modules/nw-operator-cr.adoc +++ b/modules/nw-operator-cr.adoc @@ -311,6 +311,7 @@ One of the following additional audit log targets: |==== // end::policy-audit[] +[id="gatewayConfig-object_{context}"] .`gatewayConfig` object [cols=".^2,.^2,.^6a",options="header"] |==== diff --git a/modules/nw-ovn-kubernetes-matrix.adoc b/modules/nw-ovn-kubernetes-matrix.adoc index 00a662b0fc..6eee384c48 100644 --- a/modules/nw-ovn-kubernetes-matrix.adoc +++ b/modules/nw-ovn-kubernetes-matrix.adoc @@ -2,6 +2,7 @@ // // * networking/ovn_kubernetes_network_provider/about-ovn-kubernetes.adoc +:_content-type: REFERENCE [id="nw-ovn-kubernetes-matrix_{context}"] = Supported default CNI network provider feature matrix @@ -27,6 +28,8 @@ ifeval::["{context}" == "about-ovn-kubernetes"] |Kubernetes network policy logs|Supported|Not supported +|Hardware offloading|Supported|Not supported + |Multicast|Supported|Supported endif::[] ifeval::["{context}" == "about-openshift-sdn"] @@ -47,6 +50,8 @@ ifeval::["{context}" == "about-openshift-sdn"] |Kubernetes network policy logs|Not supported|Supported |Multicast|Supported|Supported + +|Hardware offloading|Not supported|Supported endif::[] |=== [.small] diff --git a/modules/nw-sriov-hwol-about-hardware-offloading.adoc b/modules/nw-sriov-hwol-about-hardware-offloading.adoc new file mode 100644 index 0000000000..4c3b505400 --- /dev/null +++ b/modules/nw-sriov-hwol-about-hardware-offloading.adoc @@ -0,0 +1,29 @@ +// Module included in the following assemblies: +// +// * networking/configuring-hardware-offloading.adoc + +:_content-type: CONCEPT +[id="about-hardware-offloading_{context}"] += About hardware offloading + +Open vSwitch hardware offloading is a method of processing network tasks by diverting them away from the CPU and offloading them to a dedicated processor on a network interface controller. +As a result, clusters can benefit from faster data transfer speeds, reduced CPU workloads, and lower computing costs. + +The key element for this feature is a modern class of network interface controllers known as SmartNICs. +A SmartNIC is a network interface controller that is able to handle computationally-heavy network processing tasks. +In the same way that a dedicated graphics card can improve graphics performance, a SmartNIC can improve network performance. +In each case, a dedicated processor improves performance for a specific type of processing task. + +In {product-title}, you can configure hardware offloading for bare metal nodes that have a compatible SmartNIC. +Hardware offloading is configured and enabled by the SR-IOV Network Operator. + +Hardware offloading is not compatible with all workloads or application types. +Only the following two communication types are supported: + +* pod-to-pod +* pod-to-service, where the service is a ClusterIP service backed by a regular pod + +In all cases, hardware offloading takes place only when those pods and services are assigned to nodes that have a compatible SmartNIC. +Suppose, for example, that a pod on a node with hardware offloading tries to communicate with a service on a regular node. +On the regular node, all the processing takes place in the kernel, so the overall performance of the pod-to-service communication is limited to the maximum performance of that regular node. +Hardware offloading is not compatible with DPDK applications. \ No newline at end of file diff --git a/modules/nw-sriov-hwol-adding-network-attachment-definitions-to-pods.adoc b/modules/nw-sriov-hwol-adding-network-attachment-definitions-to-pods.adoc new file mode 100644 index 0000000000..65eaa575de --- /dev/null +++ b/modules/nw-sriov-hwol-adding-network-attachment-definitions-to-pods.adoc @@ -0,0 +1,22 @@ +// Module included in the following assemblies: +// +// * networking/configuring-hardware-offloading.adoc + +:_content-type: PROCEDURE +[id="adding-network-attachment-definition-to-pods_{context}"] += Adding the network attachment definition to your pods + +After you create the machine config pool, the `SriovNetworkPoolConfig` and `SriovNetworkNodePolicy` custom resources, and the network attachment definition, you can apply these configurations to your pods by adding the network attachment definition to your pod specifications. + +.Procedure + +* In the pod specification, add the `.metadata.annotations.k8s.v1.cni.cncf.io/networks` field and specify the network attachment definition you created for hardware offloading: ++ +[source,yaml] +---- +.... +metadata: + annotations: + k8s.v1.cni.cncf.io/default-network: net-attach-def/net-attach-def <.> +---- +<.> The value must be the name and namespace of the network attachment definition you created for hardware offloading. \ No newline at end of file diff --git a/modules/nw-sriov-hwol-configuring-machine-config-pool.adoc b/modules/nw-sriov-hwol-configuring-machine-config-pool.adoc new file mode 100644 index 0000000000..eb046cded0 --- /dev/null +++ b/modules/nw-sriov-hwol-configuring-machine-config-pool.adoc @@ -0,0 +1,104 @@ +// Module included in the following assemblies: +// +// * networking/configuring-hardware-offloading.adoc + +:_content-type: PROCEDURE +[id="configuring-machine-config-pool_{context}"] += Configuring a machine config pool for hardware offloading + +To enable hardware offloading, you must first create a dedicated machine config pool and configure it to work with the SR-IOV Network Operator. + +.Prerequisites + +* You installed the OpenShift CLI (`oc`). +* You have access to the cluster as a user with the `cluster-admin` role. + +.Procedure + +. Create a machine config pool for machines you want to use hardware offloading on. + +.. Create a file, such as `mcp-offloading.yaml`, with content like the following example: ++ +[source,yaml] +---- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfigPool +metadata: + name: mcp-offloading <1> +spec: + machineConfigSelector: + matchExpressions: + - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,mcp-offloading <1>]} + nodeSelector: + matchLabels: + node-role.kubernetes.io/mcp-offloading <2>: "" +---- +<1> The name of your machine config pool for hardware offloading. +<2> This node role label is used to add nodes to the machine config pool. + +.. Apply the configuration for the machine config pool: ++ +[source,terminal] +---- +$ oc create -f mcp-offloading.yaml +---- + +. Add nodes to the machine config pool. Label each node with the node role label of your pool: ++ +[source,terminal] +---- +$ oc label node worker-2 node-role.kubernetes.io/mcp-offloading="" +---- + +. Optional: To verify that the new pool is created, run the following command: ++ +[source,terminal] +---- +$ oc get nodes +---- ++ +-- +.Example output +[source,terminal] +---- +NAME STATUS ROLES AGE VERSION +master-0 Ready master 2d v1.23.3+d99c04f +master-1 Ready master 2d v1.23.3+d99c04f +master-2 Ready master 2d v1.23.3+d99c04f +worker-0 Ready worker 2d v1.23.3+d99c04f +worker-1 Ready worker 2d v1.23.3+d99c04f +worker-2 Ready mcp-offloading,worker 47h v1.23.3+d99c04f +worker-3 Ready mcp-offloading,worker 47h v1.23.3+d99c04f +---- +-- + +. Add this machine config pool to the `SriovNetworkPoolConfig` custom resource: + +.. Create a file, such as `sriov-pool-config.yaml`, with content like the following example: ++ +[source,yaml] +---- +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkPoolConfig +metadata: + name: sriovnetworkpoolconfig-offload + namespace: openshift-sriov-network-operator +spec: + ovsHardwareOffloadConfig: + name: mcp-offloading <1> +---- +<1> The name of your machine config pool for hardware offloading. + +.. Apply the configuration: ++ +[source,terminal] +---- +$ oc create -f .yaml +---- ++ +[NOTE] +===== +When you apply the configuration specified in a `SriovNetworkPoolConfig` object, the SR-IOV Operator drains and restarts the nodes in the machine config pool. + +It might take several minutes for a configuration changes to apply. +===== diff --git a/modules/nw-sriov-hwol-creating-network-attachment-definition.adoc b/modules/nw-sriov-hwol-creating-network-attachment-definition.adoc new file mode 100644 index 0000000000..013856ed56 --- /dev/null +++ b/modules/nw-sriov-hwol-creating-network-attachment-definition.adoc @@ -0,0 +1,57 @@ +// Module included in the following assemblies: +// +// * networking/configuring-hardware-offloading.adoc + +:_content-type: PROCEDURE +[id="create-network-attachment-definition_{context}"] += Creating a network attachment definition + +After you define the machine config pool and the SR-IOV network node policy, you can create a network attachment definition for the network interface card you specified. + +.Prerequisites + +* You installed the OpenShift CLI (`oc`). +* You have access to the cluster as a user with the `cluster-admin` role. + +.Procedure + +. Create a file, such as `net-attach-def.yaml`, with content like the following example: ++ +[source,yaml] +---- +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + name: net-attach-def <.> + namespace: net-attach-def <.> + annotations: + k8s.v1.cni.cncf.io/resourceName: openshift.io/mlxnics <.> +spec: + config: '{"cniVersion":"0.3.1","name":"ovn-kubernetes","type":"ovn-k8s-cni-overlay","ipam":{},"dns":{}}' +---- +<.> The name for your network attachment definition. +<.> The namespace for your network attachment definition. +<.> This is the value of the `spec.resourceName` field you specified in the `SriovNetworkNodePolicy` object. + +. Apply the configuration for the network attachment definition: ++ +[source,terminal] +---- +$ oc create -f net-attach-def.yaml +---- + +.Verification + +* Run the following command to see whether the new definition is present: ++ +[source,terminal] +---- +$ oc get net-attach-def -A +---- ++ +.Example output +[source,terminal] +---- +NAMESPACE NAME AGE +net-attach-def net-attach-def 43h +---- diff --git a/modules/nw-sriov-hwol-creating-sriov-policy.adoc b/modules/nw-sriov-hwol-creating-sriov-policy.adoc new file mode 100644 index 0000000000..3acf048255 --- /dev/null +++ b/modules/nw-sriov-hwol-creating-sriov-policy.adoc @@ -0,0 +1,62 @@ +// Module included in the following assemblies: +// +// * networking/configuring-hardware-offloading.adoc + +:_content-type: PROCEDURE +[id="configure-sriov-node-policy_{context}"] += Configuring the SR-IOV network node policy + +You can create an SR-IOV network device configuration for a node by creating an SR-IOV network node policy. +To enable hardware offloading, you must define the `.spec.eSwitchMode` field with the value `"switchdev"`. + +The following procedure creates an SR-IOV interface for a network interface controller with hardware offloading. + +.Prerequisites + +* You installed the OpenShift CLI (`oc`). +* You have access to the cluster as a user with the `cluster-admin` role. + +.Procedure + +. Create a file, such as `sriov-node-policy.yaml`, with content like the following example: ++ +[source,yaml] +---- +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sriov-node-policy <.> + namespace: openshift-sriov-network-operator +spec: + deviceType: netdevice <.> + eSwitchMode: "switchdev" <.> + nicSelector: + deviceID: "1019" + rootDevices: + - 0000:d8:00.0 + vendor: "15b3" + pfNames: + - ens8f0 + nodeSelector: + feature.node.kubernetes.io/network-sriov.capable: "true" + numVfs: 6 + priority: 5 + resourceName: mlxnics +---- +<.> The name for the custom resource object. +<.> Required. Hardware offloading is not supported with `vfio-pci`. +<.> Required. + +. Apply the configuration for the policy: ++ +[source,terminal] +---- +$ oc create -f sriov-node-policy.yaml +---- ++ +[NOTE] +===== +When you apply the configuration specified in a `SriovNetworkPoolConfig` object, the SR-IOV Operator drains and restarts the nodes in the machine config pool. + +It might take several minutes for a configuration change to apply. +===== diff --git a/modules/nw-sriov-hwol-supported-devices.adoc b/modules/nw-sriov-hwol-supported-devices.adoc new file mode 100644 index 0000000000..e3dee60360 --- /dev/null +++ b/modules/nw-sriov-hwol-supported-devices.adoc @@ -0,0 +1,25 @@ +// Module included in the following assemblies: +// +// * networking/configuring-hardware-offloading.adoc + +:_content-type: REFERENCE +[id="supported_devices_{context}"] += Supported devices + +Hardware offloading is supported on the following network interface controllers: + +.Supported network interface controllers +[cols="1,2,1,1"] +|=== +|Manufacturer |Model |Vendor ID | Device ID + +|Mellanox +|MT27800 Family [ConnectX‑5] +|15b3 +|1017 + +|Mellanox +|MT28880 Family [ConnectX‑5{nbsp}Ex] +|15b3 +|1019 +|=== \ No newline at end of file diff --git a/modules/nw-sriov-networknodepolicy-object.adoc b/modules/nw-sriov-networknodepolicy-object.adoc index 7aa9356c10..a1eb22374a 100644 --- a/modules/nw-sriov-networknodepolicy-object.adoc +++ b/modules/nw-sriov-networknodepolicy-object.adoc @@ -2,6 +2,7 @@ // // * networking/hardware_networks/configuring-sriov-device.adoc +:_content-type: REFERENCE [id="nw-sriov-networknodepolicy-object_{context}"] = SR-IOV network node configuration object @@ -32,7 +33,8 @@ spec: netFilter: "" <14> deviceType: <15> isRdma: false <16> - linkType: <17> + linkType: <17> + eSwitchMode: "switchdev" <18> ---- <1> The name for the custom resource object. @@ -75,6 +77,8 @@ If the `isRdma` parameter is set to `true`, you can continue to use the RDMA-ena Set `isRdma` to `true` and additionally set `needVhostNet` to `true` to configure a Mellanox NIC for use with Fast Datapath DPDK applications. <17> Optional: The link type for the VFs. You can specify one of the following values: `eth` or `ib`. Specify `eth` for Ethernet or `ib` for InfiniBand. The default value is `eth`. + +<18> Optional: To enable hardware offloading, the 'eSwitchMode' field must be set to `"switchdev"`. + When `linkType` is set to `ib`, `isRdma` is automatically set to `true` by the SR-IOV Network Operator webhook. When `linkType` is set to `ib`, `deviceType` should not be set to `vfio-pci`. diff --git a/networking/hardware_networks/configuring-hardware-offloading.adoc b/networking/hardware_networks/configuring-hardware-offloading.adoc new file mode 100644 index 0000000000..e3c03f323e --- /dev/null +++ b/networking/hardware_networks/configuring-hardware-offloading.adoc @@ -0,0 +1,37 @@ +:_content-type: ASSEMBLY +[id="configuring-hardware-offloading"] += Configuring hardware offloading +include::modules/common-attributes.adoc[] +:context: configuring-hardware-offloading + +toc::[] + +As a cluster administrator, you can configure hardware offloading on compatible nodes to increase data processing performance and reduce load on host CPUs. + +//What is this feature, who needs it? + +include::modules/nw-sriov-hwol-about-hardware-offloading.adoc[leveloffset=+1] + +//Which NICs do we support? + +include::modules/nw-sriov-hwol-supported-devices.adoc[leveloffset=+1] + +[id="configuring-hardware-offloading-prerequisites"] +== Prerequisites + +* Your cluster has at least one bare metal machine with a network interface controller that is supported for hardware offloading. +* You xref:./installing-sriov-operator.html#installing-sr-iov-operator_installing-sriov-operator[installed the SR-IOV Network Operator]. +* Your cluster uses the xref:../ovn_kubernetes_network_provider/about-ovn-kubernetes.adoc#about-ovn-kubernetes[OVN-Kubernetes CNI]. +* In your xref:../cluster-network-operator.adoc#gatewayConfig-object_cluster-network-operator[OVN-Kubernetes CNI configuration], the `gatewayConfig.routingViaHost` field is set to `false`. + +//Configure a machine config pool for hardware offloading +include::modules/nw-sriov-hwol-configuring-machine-config-pool.adoc[leveloffset=+1] + +//Configuring the SR-IOV network node policy +include::modules/nw-sriov-hwol-creating-sriov-policy.adoc[leveloffset=+1] + +//Creating a Network Attachment Definition +include::modules/nw-sriov-hwol-creating-network-attachment-definition.adoc[leveloffset=+1] + +//Adding the network attachment definition to your pods +include::modules/nw-sriov-hwol-adding-network-attachment-definitions-to-pods.adoc[leveloffset=+1]