mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 21:46:22 +01:00
CNV-21826: Configuring VM for DPDK
This commit is contained in:
Shikha Jhala
committed by
openshift-cherrypick-robot
openshift-cherrypick-robot
parent
7c32c79a9f
commit
4087516c55
123
modules/virt-configuring-cluster-dpdk.adoc
Normal file
123
modules/virt-configuring-cluster-dpdk.adoc
Normal file
@@ -0,0 +1,123 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * virt/virtual_machines/vm_networking/virt-attaching-vm-to-sriov-network.adoc
|
||||
|
||||
:_content-type: PROCEDURE
|
||||
[id="virt-configuring-cluster-dpdk_{context}"]
|
||||
= Configuring a cluster for DPDK workloads
|
||||
|
||||
You can use the following procedure to configure an {product-title} cluster to run Data Plane Development Kit (DPDK) workloads.
|
||||
|
||||
:FeatureName: Configuring a cluster for DPDK workloads
|
||||
include::snippets/technology-preview.adoc[]
|
||||
|
||||
.Prerequisites
|
||||
* You have access to the cluster as a user with `cluster-admin` permissions.
|
||||
* You have installed the OpenShift CLI (`oc`).
|
||||
* You have installed the SR-IOV Network Operator.
|
||||
* You have installed the Node Tuning Operator.
|
||||
|
||||
.Procedure
|
||||
. Map your compute nodes topology to determine which Non-Uniform Memory Access (NUMA) CPUs are isolated for DPDK applications and which ones are reserved for the operating system (OS).
|
||||
. Label a subset of the compute nodes with a custom role; for example, `worker-dpdk`:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc label node <node_name> node-role.kubernetes.io/worker-dpdk=""
|
||||
----
|
||||
|
||||
. Create a new `MachineConfigPool` manifest that contains the `worker-dpdk` label in the `spec.machineConfigSelector` object:
|
||||
+
|
||||
.Example `MachineConfigPool` manifest
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineConfigPool
|
||||
metadata:
|
||||
name: dpdk-workers
|
||||
labels:
|
||||
machineconfiguration.openshift.io/role: dpdk-workers
|
||||
pools.operator.machineconfiguration.openshift.io/dpdk-workers: ""
|
||||
spec:
|
||||
configuration:
|
||||
machineConfigSelector:
|
||||
matchExpressions:
|
||||
- key: machineconfiguration.openshift.io/role
|
||||
operator: In
|
||||
values:
|
||||
- worker
|
||||
- worker-dpdk
|
||||
nodeSelector:
|
||||
matchLabels:
|
||||
node-role.kubernetes.io/worker-dpdk: ""
|
||||
----
|
||||
|
||||
. Create a `PerformanceProfile` manifest that applies to the labeled nodes and the machine config pool that you created in the previous steps. The performance profile specifies the CPUs that are isolated for DPDK applications and the CPUs that are reserved for house keeping.
|
||||
+
|
||||
.Example `PerformanceProfile` manifest
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: performance.openshift.io/v2
|
||||
kind: PerformanceProfile
|
||||
metadata:
|
||||
name: profile-1
|
||||
spec:
|
||||
cpu:
|
||||
isolated: 20,22,24,26,28,30,32,34,36,38,60,62,64,66,68,70,72,74,76,78
|
||||
reserved: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,23,25,27,29,31,33,35,37,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,61,63,65,67,69,71,73,75,77,79
|
||||
hugepages:
|
||||
defaultHugepagesSize: 1G
|
||||
pages:
|
||||
- count: 32
|
||||
size: 1G
|
||||
numa:
|
||||
topologyPolicy: single-numa-node
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/worker-dpdk: ""
|
||||
----
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
The compute nodes automatically restart after you apply the `MachineConfigPool` and `PerformanceProfile` manifests.
|
||||
====
|
||||
|
||||
. Create an `SriovNetworkNodePolicy` object with the `spec.deviceType` field set to `vfio-pci`:
|
||||
+
|
||||
.Example `SriovNetworkNodePolicy` manifest
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: sriovnetwork.openshift.io/v1
|
||||
kind: SriovNetworkNodePolicy
|
||||
metadata:
|
||||
name: policy-1
|
||||
namespace: openshift-sriov-network-operator
|
||||
spec:
|
||||
resourceName: intel_nics_dpdk
|
||||
deviceType: vfio-pci
|
||||
mtu: 9000
|
||||
numVfs: 4
|
||||
priority: 99
|
||||
nicSelector:
|
||||
vendor: "8086"
|
||||
deviceID: "1572"
|
||||
pfNames:
|
||||
- eno3
|
||||
rootDevices:
|
||||
- "0000:19:00.2"
|
||||
nodeSelector:
|
||||
feature.node.kubernetes.io/network-sriov.capable: "true"
|
||||
----
|
||||
|
||||
. Set the container runtime configuration for the `virt-launcher` pod by adding the following annotation to the `HyperConverged` custom resource (CR):
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc annotate --overwrite -n openshift-cnv hco kubevirt-hyperconverged \
|
||||
kubevirt.kubevirt.io/jsonpatch='[{"op": "add", "path": "/spec/configuration/defaultRuntimeClass", "value": <runtimeclass-name>}]'
|
||||
----
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
Adding the annotation to the `HyperConverged` CR changes a global setting that affects all VMs that are created after the annotation is applied.
|
||||
====
|
||||
|
||||
120
modules/virt-configuring-vm-dpdk.adoc
Normal file
120
modules/virt-configuring-vm-dpdk.adoc
Normal file
@@ -0,0 +1,120 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * virt/virtual_machines/vm_networking/virt-attaching-vm-to-sriov-network.adoc
|
||||
|
||||
:_content-type: PROCEDURE
|
||||
[id="virt-configuring-vm-dpdk_{context}"]
|
||||
= Configuring a virtual machine for DPDK workloads
|
||||
|
||||
You can run Data Packet Development Kit (DPDK) workloads on virtual machines (VMs) to achieve lower latency and higher throughput for faster packet processing in the user space. DPDK uses the SR-IOV network for hardware-based I/O sharing.
|
||||
|
||||
.Prerequisites
|
||||
* Your cluster is configured to run DPDK workloads.
|
||||
* You have created and configured the project in which the VM will run.
|
||||
|
||||
.Procedure
|
||||
. Edit the `VirtualMachine` manifest to include information about the SR-IOV network interface, CPU topology, CRI-O annotations, and huge pages:
|
||||
+
|
||||
.Example `VirtualMachine` manifest
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: kubevirt.io/v1
|
||||
kind: VirtualMachine
|
||||
metadata:
|
||||
name: rhel-dpdk-vm
|
||||
spec:
|
||||
running: true
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
cpu-load-balancing.crio.io: disable <1>
|
||||
cpu-quota.crio.io: disable <2>
|
||||
irq-load-balancing.crio.io: disable <3>
|
||||
spec:
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/worker-dpdk: "" <4>
|
||||
domain:
|
||||
cpu:
|
||||
sockets: 1 <5>
|
||||
cores: 5 <6>
|
||||
threads: 2
|
||||
dedicatedCpuPlacement: true
|
||||
isolateEmulatorThread: true
|
||||
interfaces:
|
||||
- masquerade: {}
|
||||
name: default
|
||||
- model: virtio
|
||||
name: nic-east
|
||||
pciAddress: '0000:07:00.0'
|
||||
sriov: {}
|
||||
networkInterfaceMultiqueue: true
|
||||
rng: {}
|
||||
memory:
|
||||
hugepages:
|
||||
pageSize: 1Gi <7>
|
||||
resources:
|
||||
requests:
|
||||
memory: 8Gi
|
||||
networks:
|
||||
- name: default
|
||||
pod: {}
|
||||
- multus:
|
||||
networkName: dpdk-net <8>
|
||||
name: nic-east
|
||||
# ...
|
||||
----
|
||||
<1> This annotation specifies that load balancing is disabled for CPUs that are used by the container.
|
||||
<2> This annotation specifies that the CPU quota is disabled for CPUs that are used by the container.
|
||||
<3> This annotation specifies that Interrupt Request (IRQ) load balancing is disabled for CPUs that are used by the container.
|
||||
<4> The label that is used in the `MachineConfigPool` and `PerformanceProfile` manifests that were created when configuring the cluster for DPDK workloads.
|
||||
<5> The number of sockets inside the VM. This field must be set to `1` for the CPUs to be scheduled from the same Non-Uniform Memory Access (NUMA) node.
|
||||
<6> The number of cores inside the VM. This must be a value greater than or equal to `1`. In this example, the VM is scheduled with 5 hyper-threads or 10 CPUs.
|
||||
<7> The size of the huge pages. The possible values for x86-64 architecture are 1Gi and 2Mi. In this example, the request is for 8 huge pages of size 1Gi.
|
||||
<8> The name of the SR-IOV `NetworkAttachmentDefinition` object.
|
||||
|
||||
. Save and exit the editor.
|
||||
. Apply the `VirtualMachine` manifest:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc apply -f <file_name>.yaml
|
||||
----
|
||||
|
||||
. Configure the guest operating system. The following example shows the configuration steps for {op-system-base} 8 OS:
|
||||
.. Configure isolated VM CPUs and specify huge pages by using the GRUB bootloader command-line interface. In the following example, 8 1G huge pages are specified. The first two CPUs (0 and 1) are set aside for house keeping tasks and the rest are isolated for the DPDK application.
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ grubby --update-kernel=ALL --args="default_hugepagesz=1GB hugepagesz=1G hugepages=8 isolcpus=2-9"
|
||||
----
|
||||
|
||||
.. To achieve low-latency tuning by using the `cpu-partitioning` profile in the TuneD application, run the following commands:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ dnf install -y tuned-profiles-cpu-partitioning
|
||||
----
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ echo isolated_cores=2-9 > /etc/tuned/cpu-partitioning-variables.conf
|
||||
----
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ tuned-adm profile cpu-partitioning
|
||||
----
|
||||
|
||||
.. Override the SR-IOV NIC driver by using the `driverctl` device driver control utility:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ dnf install -y driverctl
|
||||
----
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ driverctl set-override 0000:07:00.0 vfio-pci
|
||||
----
|
||||
|
||||
. Restart the VM to apply the changes.
|
||||
54
modules/virt-configuring-vm-project-dpdk.adoc
Normal file
54
modules/virt-configuring-vm-project-dpdk.adoc
Normal file
@@ -0,0 +1,54 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * virt/virtual_machines/vm_networking/virt-attaching-vm-to-sriov-network.adoc
|
||||
|
||||
:_content-type: PROCEDURE
|
||||
[id="virt-configuring-vm-project-dpdk_{context}"]
|
||||
= Configuring a project for DPDK workloads
|
||||
|
||||
You can configure the project to run DPDK workloads on SR-IOV hardware.
|
||||
|
||||
.Prerequisites
|
||||
* Your cluster is configured to run DPDK workloads.
|
||||
|
||||
.Procedure
|
||||
. Create a namespace for your DPDK applications:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc create ns dpdk-checkup-ns
|
||||
----
|
||||
|
||||
. Create an `SriovNetwork` object that references the `SriovNetworkNodePolicy` object. When you create an `SriovNetwork` object, the SR-IOV Network Operator automatically creates a `NetworkAttachmentDefinition` object.
|
||||
+
|
||||
.Example `SriovNetwork` manifest
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: sriovnetwork.openshift.io/v1
|
||||
kind: SriovNetwork
|
||||
metadata:
|
||||
name: dpdk-sriovnetwork
|
||||
namespace: openshift-sriov-network-operator
|
||||
spec:
|
||||
ipam: |
|
||||
{
|
||||
"type": "host-local",
|
||||
"subnet": "10.56.217.0/24",
|
||||
"rangeStart": "10.56.217.171",
|
||||
"rangeEnd": "10.56.217.181",
|
||||
"routes": [{
|
||||
"dst": "0.0.0.0/0"
|
||||
}],
|
||||
"gateway": "10.56.217.1"
|
||||
}
|
||||
networkNamespace: dpdk-checkup-ns <1>
|
||||
resourceName: intel_nics_dpdk <2>
|
||||
spoofChk: "off"
|
||||
trust: "on"
|
||||
vlan: 1019
|
||||
----
|
||||
<1> The namespace where the `NetworkAttachmentDefinition` object is deployed.
|
||||
<2> The value of the `spec.resourceName` attribute of the `SriovNetworkNodePolicy` object that was created when configuring the cluster for DPDK workloads.
|
||||
|
||||
. Optional: Run the virtual machine latency checkup to verify that the network is properly configured.
|
||||
. Optional: Run the DPDK checkup to verify that the namespace is ready for DPDK workloads.
|
||||
@@ -24,7 +24,27 @@ include::modules/nw-sriov-network-attachment.adoc[leveloffset=+1]
|
||||
|
||||
include::modules/virt-attaching-vm-to-sriov-network.adoc[leveloffset=+1]
|
||||
|
||||
include::modules/virt-configuring-cluster-dpdk.adoc[leveloffset=+1]
|
||||
|
||||
[role="_additional-resources"]
|
||||
.Additional resources
|
||||
* xref:../../../scalability_and_performance/using-cpu-manager.adoc#using-cpu-manager[Using CPU Manager and Topology Manager]
|
||||
* xref:../../../scalability_and_performance/what-huge-pages-do-and-how-they-are-consumed-by-apps.adoc#configuring-huge-pages_huge-pages[Configuring huge pages]
|
||||
* link:https://access.redhat.com/solutions/5688941[Creating a custom machine config pool]
|
||||
|
||||
include::modules/virt-configuring-vm-project-dpdk.adoc[leveloffset=+1]
|
||||
|
||||
[role="_additional-resources"]
|
||||
.Additional resources
|
||||
* xref:../../../applications/projects/working-with-projects.adoc#working-with-projects[Working with projects]
|
||||
* xref:../../../virt/support/monitoring/virt-running-cluster-checkups.adoc#virt-measuring-latency-vm-secondary-network_virt-running-cluster-checkups[Virtual machine latency checkup]
|
||||
* xref:../../../virt/support/monitoring/virt-running-cluster-checkups.adoc#virt-checking-cluster-dpdk-readiness_virt-running-cluster-checkups[DPDK checkup]
|
||||
|
||||
include::modules/virt-configuring-vm-dpdk.adoc[leveloffset=+1]
|
||||
|
||||
|
||||
[id="next-steps_virt-attaching-vm-to-sriov-network"]
|
||||
== Next steps
|
||||
|
||||
* xref:../../../virt/virtual_machines/vm_networking/virt-accessing-vm-secondary-network-fqdn.adoc#virt-accessing-vm-secondary-network-fqdn[Accessing a virtual machine on a secondary network by using the cluster domain name]
|
||||
* xref:../../../virt/virtual_machines/vm_networking/virt-accessing-vm-secondary-network-fqdn.adoc#virt-accessing-vm-secondary-network-fqdn[Accessing a virtual machine on a secondary network by using the cluster domain name]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user