mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
OSDOCS-2231: Hardware offloading for SR-IOV
This commit is contained in:
Christopher Tauchen
committed by
openshift-cherrypick-robot
openshift-cherrypick-robot
parent
81c5bf7db7
commit
f0d9ae240d
@@ -1001,6 +1001,8 @@ Topics:
|
||||
File: using-dpdk-and-rdma
|
||||
- Name: Using pod-level bonding for secondary networks
|
||||
File: using-pod-level-bonding
|
||||
- Name: Configuring hardware offloading
|
||||
File: configuring-hardware-offloading
|
||||
- Name: OpenShift SDN default CNI network provider
|
||||
Dir: openshift_sdn
|
||||
Topics:
|
||||
|
||||
@@ -311,6 +311,7 @@ One of the following additional audit log targets:
|
||||
|====
|
||||
// end::policy-audit[]
|
||||
|
||||
[id="gatewayConfig-object_{context}"]
|
||||
.`gatewayConfig` object
|
||||
[cols=".^2,.^2,.^6a",options="header"]
|
||||
|====
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
//
|
||||
// * networking/ovn_kubernetes_network_provider/about-ovn-kubernetes.adoc
|
||||
|
||||
:_content-type: REFERENCE
|
||||
[id="nw-ovn-kubernetes-matrix_{context}"]
|
||||
= Supported default CNI network provider feature matrix
|
||||
|
||||
@@ -27,6 +28,8 @@ ifeval::["{context}" == "about-ovn-kubernetes"]
|
||||
|
||||
|Kubernetes network policy logs|Supported|Not supported
|
||||
|
||||
|Hardware offloading|Supported|Not supported
|
||||
|
||||
|Multicast|Supported|Supported
|
||||
endif::[]
|
||||
ifeval::["{context}" == "about-openshift-sdn"]
|
||||
@@ -47,6 +50,8 @@ ifeval::["{context}" == "about-openshift-sdn"]
|
||||
|Kubernetes network policy logs|Not supported|Supported
|
||||
|
||||
|Multicast|Supported|Supported
|
||||
|
||||
|Hardware offloading|Not supported|Supported
|
||||
endif::[]
|
||||
|===
|
||||
[.small]
|
||||
|
||||
29
modules/nw-sriov-hwol-about-hardware-offloading.adoc
Normal file
29
modules/nw-sriov-hwol-about-hardware-offloading.adoc
Normal file
@@ -0,0 +1,29 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * networking/configuring-hardware-offloading.adoc
|
||||
|
||||
:_content-type: CONCEPT
|
||||
[id="about-hardware-offloading_{context}"]
|
||||
= About hardware offloading
|
||||
|
||||
Open vSwitch hardware offloading is a method of processing network tasks by diverting them away from the CPU and offloading them to a dedicated processor on a network interface controller.
|
||||
As a result, clusters can benefit from faster data transfer speeds, reduced CPU workloads, and lower computing costs.
|
||||
|
||||
The key element for this feature is a modern class of network interface controllers known as SmartNICs.
|
||||
A SmartNIC is a network interface controller that is able to handle computationally-heavy network processing tasks.
|
||||
In the same way that a dedicated graphics card can improve graphics performance, a SmartNIC can improve network performance.
|
||||
In each case, a dedicated processor improves performance for a specific type of processing task.
|
||||
|
||||
In {product-title}, you can configure hardware offloading for bare metal nodes that have a compatible SmartNIC.
|
||||
Hardware offloading is configured and enabled by the SR-IOV Network Operator.
|
||||
|
||||
Hardware offloading is not compatible with all workloads or application types.
|
||||
Only the following two communication types are supported:
|
||||
|
||||
* pod-to-pod
|
||||
* pod-to-service, where the service is a ClusterIP service backed by a regular pod
|
||||
|
||||
In all cases, hardware offloading takes place only when those pods and services are assigned to nodes that have a compatible SmartNIC.
|
||||
Suppose, for example, that a pod on a node with hardware offloading tries to communicate with a service on a regular node.
|
||||
On the regular node, all the processing takes place in the kernel, so the overall performance of the pod-to-service communication is limited to the maximum performance of that regular node.
|
||||
Hardware offloading is not compatible with DPDK applications.
|
||||
@@ -0,0 +1,22 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * networking/configuring-hardware-offloading.adoc
|
||||
|
||||
:_content-type: PROCEDURE
|
||||
[id="adding-network-attachment-definition-to-pods_{context}"]
|
||||
= Adding the network attachment definition to your pods
|
||||
|
||||
After you create the machine config pool, the `SriovNetworkPoolConfig` and `SriovNetworkNodePolicy` custom resources, and the network attachment definition, you can apply these configurations to your pods by adding the network attachment definition to your pod specifications.
|
||||
|
||||
.Procedure
|
||||
|
||||
* In the pod specification, add the `.metadata.annotations.k8s.v1.cni.cncf.io/networks` field and specify the network attachment definition you created for hardware offloading:
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
....
|
||||
metadata:
|
||||
annotations:
|
||||
k8s.v1.cni.cncf.io/default-network: net-attach-def/net-attach-def <.>
|
||||
----
|
||||
<.> The value must be the name and namespace of the network attachment definition you created for hardware offloading.
|
||||
104
modules/nw-sriov-hwol-configuring-machine-config-pool.adoc
Normal file
104
modules/nw-sriov-hwol-configuring-machine-config-pool.adoc
Normal file
@@ -0,0 +1,104 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * networking/configuring-hardware-offloading.adoc
|
||||
|
||||
:_content-type: PROCEDURE
|
||||
[id="configuring-machine-config-pool_{context}"]
|
||||
= Configuring a machine config pool for hardware offloading
|
||||
|
||||
To enable hardware offloading, you must first create a dedicated machine config pool and configure it to work with the SR-IOV Network Operator.
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You installed the OpenShift CLI (`oc`).
|
||||
* You have access to the cluster as a user with the `cluster-admin` role.
|
||||
|
||||
.Procedure
|
||||
|
||||
. Create a machine config pool for machines you want to use hardware offloading on.
|
||||
|
||||
.. Create a file, such as `mcp-offloading.yaml`, with content like the following example:
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineConfigPool
|
||||
metadata:
|
||||
name: mcp-offloading <1>
|
||||
spec:
|
||||
machineConfigSelector:
|
||||
matchExpressions:
|
||||
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,mcp-offloading <1>]}
|
||||
nodeSelector:
|
||||
matchLabels:
|
||||
node-role.kubernetes.io/mcp-offloading <2>: ""
|
||||
----
|
||||
<1> The name of your machine config pool for hardware offloading.
|
||||
<2> This node role label is used to add nodes to the machine config pool.
|
||||
|
||||
.. Apply the configuration for the machine config pool:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc create -f mcp-offloading.yaml
|
||||
----
|
||||
|
||||
. Add nodes to the machine config pool. Label each node with the node role label of your pool:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc label node worker-2 node-role.kubernetes.io/mcp-offloading=""
|
||||
----
|
||||
|
||||
. Optional: To verify that the new pool is created, run the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc get nodes
|
||||
----
|
||||
+
|
||||
--
|
||||
.Example output
|
||||
[source,terminal]
|
||||
----
|
||||
NAME STATUS ROLES AGE VERSION
|
||||
master-0 Ready master 2d v1.23.3+d99c04f
|
||||
master-1 Ready master 2d v1.23.3+d99c04f
|
||||
master-2 Ready master 2d v1.23.3+d99c04f
|
||||
worker-0 Ready worker 2d v1.23.3+d99c04f
|
||||
worker-1 Ready worker 2d v1.23.3+d99c04f
|
||||
worker-2 Ready mcp-offloading,worker 47h v1.23.3+d99c04f
|
||||
worker-3 Ready mcp-offloading,worker 47h v1.23.3+d99c04f
|
||||
----
|
||||
--
|
||||
|
||||
. Add this machine config pool to the `SriovNetworkPoolConfig` custom resource:
|
||||
|
||||
.. Create a file, such as `sriov-pool-config.yaml`, with content like the following example:
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: sriovnetwork.openshift.io/v1
|
||||
kind: SriovNetworkPoolConfig
|
||||
metadata:
|
||||
name: sriovnetworkpoolconfig-offload
|
||||
namespace: openshift-sriov-network-operator
|
||||
spec:
|
||||
ovsHardwareOffloadConfig:
|
||||
name: mcp-offloading <1>
|
||||
----
|
||||
<1> The name of your machine config pool for hardware offloading.
|
||||
|
||||
.. Apply the configuration:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc create -f <SriovNetworkPoolConfig_name>.yaml
|
||||
----
|
||||
+
|
||||
[NOTE]
|
||||
=====
|
||||
When you apply the configuration specified in a `SriovNetworkPoolConfig` object, the SR-IOV Operator drains and restarts the nodes in the machine config pool.
|
||||
|
||||
It might take several minutes for a configuration changes to apply.
|
||||
=====
|
||||
@@ -0,0 +1,57 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * networking/configuring-hardware-offloading.adoc
|
||||
|
||||
:_content-type: PROCEDURE
|
||||
[id="create-network-attachment-definition_{context}"]
|
||||
= Creating a network attachment definition
|
||||
|
||||
After you define the machine config pool and the SR-IOV network node policy, you can create a network attachment definition for the network interface card you specified.
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You installed the OpenShift CLI (`oc`).
|
||||
* You have access to the cluster as a user with the `cluster-admin` role.
|
||||
|
||||
.Procedure
|
||||
|
||||
. Create a file, such as `net-attach-def.yaml`, with content like the following example:
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: "k8s.cni.cncf.io/v1"
|
||||
kind: NetworkAttachmentDefinition
|
||||
metadata:
|
||||
name: net-attach-def <.>
|
||||
namespace: net-attach-def <.>
|
||||
annotations:
|
||||
k8s.v1.cni.cncf.io/resourceName: openshift.io/mlxnics <.>
|
||||
spec:
|
||||
config: '{"cniVersion":"0.3.1","name":"ovn-kubernetes","type":"ovn-k8s-cni-overlay","ipam":{},"dns":{}}'
|
||||
----
|
||||
<.> The name for your network attachment definition.
|
||||
<.> The namespace for your network attachment definition.
|
||||
<.> This is the value of the `spec.resourceName` field you specified in the `SriovNetworkNodePolicy` object.
|
||||
|
||||
. Apply the configuration for the network attachment definition:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc create -f net-attach-def.yaml
|
||||
----
|
||||
|
||||
.Verification
|
||||
|
||||
* Run the following command to see whether the new definition is present:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc get net-attach-def -A
|
||||
----
|
||||
+
|
||||
.Example output
|
||||
[source,terminal]
|
||||
----
|
||||
NAMESPACE NAME AGE
|
||||
net-attach-def net-attach-def 43h
|
||||
----
|
||||
62
modules/nw-sriov-hwol-creating-sriov-policy.adoc
Normal file
62
modules/nw-sriov-hwol-creating-sriov-policy.adoc
Normal file
@@ -0,0 +1,62 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * networking/configuring-hardware-offloading.adoc
|
||||
|
||||
:_content-type: PROCEDURE
|
||||
[id="configure-sriov-node-policy_{context}"]
|
||||
= Configuring the SR-IOV network node policy
|
||||
|
||||
You can create an SR-IOV network device configuration for a node by creating an SR-IOV network node policy.
|
||||
To enable hardware offloading, you must define the `.spec.eSwitchMode` field with the value `"switchdev"`.
|
||||
|
||||
The following procedure creates an SR-IOV interface for a network interface controller with hardware offloading.
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You installed the OpenShift CLI (`oc`).
|
||||
* You have access to the cluster as a user with the `cluster-admin` role.
|
||||
|
||||
.Procedure
|
||||
|
||||
. Create a file, such as `sriov-node-policy.yaml`, with content like the following example:
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: sriovnetwork.openshift.io/v1
|
||||
kind: SriovNetworkNodePolicy
|
||||
metadata:
|
||||
name: sriov-node-policy <.>
|
||||
namespace: openshift-sriov-network-operator
|
||||
spec:
|
||||
deviceType: netdevice <.>
|
||||
eSwitchMode: "switchdev" <.>
|
||||
nicSelector:
|
||||
deviceID: "1019"
|
||||
rootDevices:
|
||||
- 0000:d8:00.0
|
||||
vendor: "15b3"
|
||||
pfNames:
|
||||
- ens8f0
|
||||
nodeSelector:
|
||||
feature.node.kubernetes.io/network-sriov.capable: "true"
|
||||
numVfs: 6
|
||||
priority: 5
|
||||
resourceName: mlxnics
|
||||
----
|
||||
<.> The name for the custom resource object.
|
||||
<.> Required. Hardware offloading is not supported with `vfio-pci`.
|
||||
<.> Required.
|
||||
|
||||
. Apply the configuration for the policy:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc create -f sriov-node-policy.yaml
|
||||
----
|
||||
+
|
||||
[NOTE]
|
||||
=====
|
||||
When you apply the configuration specified in a `SriovNetworkPoolConfig` object, the SR-IOV Operator drains and restarts the nodes in the machine config pool.
|
||||
|
||||
It might take several minutes for a configuration change to apply.
|
||||
=====
|
||||
25
modules/nw-sriov-hwol-supported-devices.adoc
Normal file
25
modules/nw-sriov-hwol-supported-devices.adoc
Normal file
@@ -0,0 +1,25 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * networking/configuring-hardware-offloading.adoc
|
||||
|
||||
:_content-type: REFERENCE
|
||||
[id="supported_devices_{context}"]
|
||||
= Supported devices
|
||||
|
||||
Hardware offloading is supported on the following network interface controllers:
|
||||
|
||||
.Supported network interface controllers
|
||||
[cols="1,2,1,1"]
|
||||
|===
|
||||
|Manufacturer |Model |Vendor ID | Device ID
|
||||
|
||||
|Mellanox
|
||||
|MT27800 Family [ConnectX‑5]
|
||||
|15b3
|
||||
|1017
|
||||
|
||||
|Mellanox
|
||||
|MT28880 Family [ConnectX‑5{nbsp}Ex]
|
||||
|15b3
|
||||
|1019
|
||||
|===
|
||||
@@ -2,6 +2,7 @@
|
||||
//
|
||||
// * networking/hardware_networks/configuring-sriov-device.adoc
|
||||
|
||||
:_content-type: REFERENCE
|
||||
[id="nw-sriov-networknodepolicy-object_{context}"]
|
||||
= SR-IOV network node configuration object
|
||||
|
||||
@@ -32,7 +33,8 @@ spec:
|
||||
netFilter: "<filter_string>" <14>
|
||||
deviceType: <device_type> <15>
|
||||
isRdma: false <16>
|
||||
linkType: <link_type> <17>
|
||||
linkType: <link_type> <17>
|
||||
eSwitchMode: "switchdev" <18>
|
||||
----
|
||||
<1> The name for the custom resource object.
|
||||
|
||||
@@ -75,6 +77,8 @@ If the `isRdma` parameter is set to `true`, you can continue to use the RDMA-ena
|
||||
Set `isRdma` to `true` and additionally set `needVhostNet` to `true` to configure a Mellanox NIC for use with Fast Datapath DPDK applications.
|
||||
|
||||
<17> Optional: The link type for the VFs. You can specify one of the following values: `eth` or `ib`. Specify `eth` for Ethernet or `ib` for InfiniBand. The default value is `eth`.
|
||||
|
||||
<18> Optional: To enable hardware offloading, the 'eSwitchMode' field must be set to `"switchdev"`.
|
||||
+
|
||||
When `linkType` is set to `ib`, `isRdma` is automatically set to `true` by the SR-IOV Network Operator webhook. When `linkType` is set to `ib`, `deviceType` should not be set to `vfio-pci`.
|
||||
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
:_content-type: ASSEMBLY
|
||||
[id="configuring-hardware-offloading"]
|
||||
= Configuring hardware offloading
|
||||
include::modules/common-attributes.adoc[]
|
||||
:context: configuring-hardware-offloading
|
||||
|
||||
toc::[]
|
||||
|
||||
As a cluster administrator, you can configure hardware offloading on compatible nodes to increase data processing performance and reduce load on host CPUs.
|
||||
|
||||
//What is this feature, who needs it?
|
||||
|
||||
include::modules/nw-sriov-hwol-about-hardware-offloading.adoc[leveloffset=+1]
|
||||
|
||||
//Which NICs do we support?
|
||||
|
||||
include::modules/nw-sriov-hwol-supported-devices.adoc[leveloffset=+1]
|
||||
|
||||
[id="configuring-hardware-offloading-prerequisites"]
|
||||
== Prerequisites
|
||||
|
||||
* Your cluster has at least one bare metal machine with a network interface controller that is supported for hardware offloading.
|
||||
* You xref:./installing-sriov-operator.html#installing-sr-iov-operator_installing-sriov-operator[installed the SR-IOV Network Operator].
|
||||
* Your cluster uses the xref:../ovn_kubernetes_network_provider/about-ovn-kubernetes.adoc#about-ovn-kubernetes[OVN-Kubernetes CNI].
|
||||
* In your xref:../cluster-network-operator.adoc#gatewayConfig-object_cluster-network-operator[OVN-Kubernetes CNI configuration], the `gatewayConfig.routingViaHost` field is set to `false`.
|
||||
|
||||
//Configure a machine config pool for hardware offloading
|
||||
include::modules/nw-sriov-hwol-configuring-machine-config-pool.adoc[leveloffset=+1]
|
||||
|
||||
//Configuring the SR-IOV network node policy
|
||||
include::modules/nw-sriov-hwol-creating-sriov-policy.adoc[leveloffset=+1]
|
||||
|
||||
//Creating a Network Attachment Definition
|
||||
include::modules/nw-sriov-hwol-creating-network-attachment-definition.adoc[leveloffset=+1]
|
||||
|
||||
//Adding the network attachment definition to your pods
|
||||
include::modules/nw-sriov-hwol-adding-network-attachment-definitions-to-pods.adoc[leveloffset=+1]
|
||||
Reference in New Issue
Block a user