From db7230510f5ee71fb05cceef05672d3d09b6ce24 Mon Sep 17 00:00:00 2001 From: Michael Burke Date: Mon, 12 Dec 2022 15:10:55 -0500 Subject: [PATCH] Documentation is unclear about status of cgroups v2 --- _topic_maps/_topic_map.yml | 6 +- .../nodes-clusters-cgroups-okd-configure.adoc | 126 ++++++++++++++++++ modules/nodes-nodes-kernel-arguments.adoc | 57 ++++++++ nodes/clusters/nodes-cluster-cgroups-okd.adoc | 19 +++ .../cluster-tasks.adoc | 6 +- .../machine-configuration-tasks.adoc | 2 + 6 files changed, 214 insertions(+), 2 deletions(-) create mode 100644 modules/nodes-clusters-cgroups-okd-configure.adoc create mode 100644 nodes/clusters/nodes-cluster-cgroups-okd.adoc diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml index 78dd8a6fad..c99c4790e9 100644 --- a/_topic_maps/_topic_map.yml +++ b/_topic_maps/_topic_map.yml @@ -489,6 +489,7 @@ Topics: File: configuring-firewall - Name: Enabling Linux control group version 2 (cgroup v2) File: enabling-cgroup-v2 + Distros: openshift-enterprise - Name: Validating an installation File: validating-an-installation Distros: openshift-origin,openshift-enterprise @@ -2161,7 +2162,10 @@ Topics: Distros: openshift-enterprise,openshift-origin - Name: Enabling Linux control group version 2 (cgroup v2) File: nodes-cluster-cgroups-2 - Distros: openshift-enterprise,openshift-origin + Distros: openshift-enterprise + - Name: Configuring the Linux cgroup on your nodes + File: nodes-cluster-cgroups-okd + Distros: openshift-origin - Name: Enabling features using FeatureGates File: nodes-cluster-enabling-features Distros: openshift-enterprise,openshift-origin diff --git a/modules/nodes-clusters-cgroups-okd-configure.adoc b/modules/nodes-clusters-cgroups-okd-configure.adoc new file mode 100644 index 0000000000..0779621bb4 --- /dev/null +++ b/modules/nodes-clusters-cgroups-okd-configure.adoc @@ -0,0 +1,126 @@ +// Module included in the following assemblies: +// +// * nodes/clusters/nodes-cluster-cgroups-okd.adoc +// * post_installation_configuration/cluster-tasks.adoc + +ifeval::["{context}" == "nodes-cluster-cgroups-2"] +:node: +endif::[] +ifeval::["{context}" == "post-install-cluster-tasks"] +:post: +endif::[] + +:_content-type: PROCEDURE +[id="nodes-clusters-cgroups-okd-configure_{context}"] += Configuring Linux cgroup + +ifdef::post[] +By default, {product-title} uses link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html[Linux control group version 2] (cgroup v2) in your cluster. You can switch to link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1.html[Linux control group version 1] (cgroup v1), if needed. + +cgroup v2 is the next version of the kernel link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01[control group] and offers multiple improvements. However, it can have some unwanted effects on your nodes. +endif::post[] + +You can configure whether your cluster uses cgroup v1 or cgroup v2 by editing the `node.config` object. Enabling the other version of cgroup in {product-title} disables the current cgroup controllers and hierarchies in your cluster. + +.Prerequisites +* Have administrative privilege to a working {product-title} cluster. + +.Procedure + +. Create a `MachineConfig` object file that identifies the kernel argument (for example, `worker-cgroup-v1.yaml`) ++ +[source,yaml] +---- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker <1> + name: worker-cgroup-v1 <2> +spec: + config: + ignition: + version: 3.2.0 + kernelArguments: + - systemd.unified_cgroup_hierarchy=0 <3> +---- ++ +<1> Applies the new kernel argument only to worker nodes. +<2> Applies a name to the machine config. +<3> Configures cgroup v1 on the associated nodes. + +. Create the new machine config: ++ +[source,terminal] +---- +$ oc create -f 05-worker-cgroup-v1.yaml +---- + +. Check to see that the new machine config was added: ++ +[source,terminal] +---- +$ oc get MachineConfig +---- ++ +.Example output +[source,terminal] +---- +NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE +00-master 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +00-worker 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +01-master-container-runtime 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +01-master-kubelet 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +01-worker-container-runtime 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +01-worker-kubelet 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +99-worker-cgroup-v1 3.2.0 105s +99-master-generated-registries 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +99-master-ssh 3.2.0 40m +99-worker-generated-registries 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +99-worker-ssh 3.2.0 40m +rendered-master-23e785de7587df95a4b517e0647e5ab7 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +rendered-master-c5e92d98103061c4818cfcefcf462770 60746a843e7ef8855ae00f2ffcb655c53e0e8296 3.2.0 115s +rendered-worker-5d596d9293ca3ea80c896a1191735bb1 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +---- + +. Check the nodes: ++ +[source,terminal] +---- +$ oc get nodes +---- ++ +.Example output +[source,terminal] +---- +NAME STATUS ROLES AGE VERSION +ip-10-0-136-161.ec2.internal Ready worker 28m v1.25.0 +ip-10-0-136-243.ec2.internal Ready master 34m v1.25.0 +ip-10-0-141-105.ec2.internal Ready,SchedulingDisabled worker 28m v1.25.0 +ip-10-0-142-249.ec2.internal Ready master 34m v1.25.0 +ip-10-0-153-11.ec2.internal Ready worker 28m v1.25.0 +ip-10-0-153-150.ec2.internal Ready master 34m v1.25.0 +---- ++ +You can see that The command disables scheduling on each worker node. + +. Check that the sys/fs/cgroup/cgroup2fs file has been moved to the `tmpfs` file system: ++ +[source,terminal] +---- +$ stat -c %T -f /sys/fs/cgroup +---- ++ +.Example output ++ +[source,terminal] +---- +tmpfs +---- + +ifeval::["{context}" == "nodes-cluster-cgroups-2"] +:!node: +endif::[] +ifeval::["{context}" == "post-install-cluster-tasks"] +:!post: +endif::[] diff --git a/modules/nodes-nodes-kernel-arguments.adoc b/modules/nodes-nodes-kernel-arguments.adoc index 5f6f3a8f7d..45a9eaff6b 100644 --- a/modules/nodes-nodes-kernel-arguments.adoc +++ b/modules/nodes-nodes-kernel-arguments.adoc @@ -20,7 +20,30 @@ Examples of kernel arguments you could set include: * **nosmt**: Disables symmetric multithreading (SMT) in the kernel. Multithreading allows multiple logical threads for each CPU. You could consider `nosmt` in multi-tenant environments to reduce risks from potential cross-thread attacks. By disabling SMT, you essentially choose security over performance. +ifndef::openshift-origin[] * **systemd.unified_cgroup_hierarchy**: Enables link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html[Linux control group version 2] (cgroup v2). cgroup v2 is the next version of the kernel link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01[control group] and offers multiple improvements. ++ +[IMPORTANT] +==== +[subs="attributes+"] +{product-title} cgroups version 2 support is a Technology Preview feature only. Technology Preview features +are not supported with Red Hat production service level agreements (SLAs) and +might not be functionally complete. Red Hat does not support Technology Preview features for use in production systems. These features provide early access to upcoming product +features, enabling customers to test functionality and provide feedback during +the development process. + +For more information about the support scope of Red Hat Technology Preview +features, see https://access.redhat.com/support/offerings/techpreview/. +==== +endif::openshift-origin[] +ifdef::openshift-origin[] +* **systemd.unified_cgroup_hierarchy**: Configures the version of Linux control group that is installed on your nodes: link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1.html[cgroup v1] or link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html[cgroup v2]. cgroup v2 is the next version of the kernel link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01[control group] and offers multiple improvements. However, it can have some unwanted effects on your nodes. ++ +[NOTE] +==== +cgroup v2 is enabled by default. To disable cgroup v2, use the `systemd.unified_cgroup_hierarchy=0` kernel argument, as shown in the following procedure. +==== +endif::openshift-origin[] See link:https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt[Kernel.org kernel parameters] for a list and descriptions of kernel arguments. @@ -61,6 +84,7 @@ rendered-master-23e785de7587df95a4b517e0647e5ab7 52dd3ba6a9a527fc3ab42afac8d12 rendered-worker-5d596d9293ca3ea80c896a1191735bb1 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m ---- +ifndef::openshift-origin[] . Create a `MachineConfig` object file that identifies the kernel argument (for example, `05-worker-kernelarg-selinuxpermissive.yaml`) + [source,yaml] @@ -83,6 +107,38 @@ spec: <2> Named to identify where it fits among the machine configs (05) and what it does (adds a kernel argument to configure SELinux permissive mode). <3> Identifies the exact kernel argument as `enforcing=0`. +. Create the new machine config: ++ +[source,terminal] +---- +$ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml +---- +endif::openshift-origin[] +ifdef::openshift-origin[] +. Create a `MachineConfig` object file that identifies the kernel argument (for example, `05-worker-kernelarg-selinuxpermissive.yaml`) ++ +[source,yaml] +---- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker <1> + name: 05-worker-kernelarg-selinuxpermissive <2> +spec: + config: + ignition: + version: 3.2.0 + kernelArguments: + - enforcing=0 <3> + systemd.unified_cgroup_hierarchy=0 <4> +---- ++ +<1> Applies the new kernel argument only to worker nodes. +<2> Named to identify where it fits among the machine configs (05) and what it does (adds +a kernel argument to configure SELinux permissive mode). +<3> Identifies the exact kernel argument as `enforcing=0`. +<4> Configures cgroup v1 on the associated nodes. cgroup v2 is the default. . Create the new machine config: + @@ -90,6 +146,7 @@ a kernel argument to configure SELinux permissive mode). ---- $ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml ---- +endif::openshift-origin[] . Check the machine configs to see that the new one was added: + diff --git a/nodes/clusters/nodes-cluster-cgroups-okd.adoc b/nodes/clusters/nodes-cluster-cgroups-okd.adoc new file mode 100644 index 0000000000..77b89bba43 --- /dev/null +++ b/nodes/clusters/nodes-cluster-cgroups-okd.adoc @@ -0,0 +1,19 @@ +:_content-type: ASSEMBLY +:context: nodes-cluster-cgroups-2 +[id="nodes-cluster-cgroups-okd"] += Configuring the Linux cgroup on your nodes +include::_attributes/common-attributes.adoc[] + +toc::[] + +By default, {product-title} uses link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html[Linux control group version 2] (cgroup v2) in your cluster. You can switch to link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1.html[Linux control group version 1] (cgroup v1), if needed. + +cgroup v2 is the next version of the kernel link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01[control group] and offers multiple improvements. However, it can have some unwanted effects on your nodes. + +// The following include statements pull in the module files that comprise +// the assembly. Include any combination of concept, procedure, or reference +// modules required to cover the user story. You can also include other +// assemblies. + +include::modules/nodes-clusters-cgroups-okd-configure.adoc[leveloffset=+1] + diff --git a/post_installation_configuration/cluster-tasks.adoc b/post_installation_configuration/cluster-tasks.adoc index 64a311edd6..dc65168622 100644 --- a/post_installation_configuration/cluster-tasks.adoc +++ b/post_installation_configuration/cluster-tasks.adoc @@ -587,8 +587,12 @@ include::modules/machine-autoscaler-cr.adoc[leveloffset=+2] :FeatureResourceName: MachineAutoscaler include::modules/deploying-resource.adoc[leveloffset=+2] +ifndef::openshift-origin[] include::modules/nodes-clusters-cgroups-2.adoc[leveloffset=+1] - +endif::openshift-origin[] +ifdef::openshift-origin[] +include::modules/nodes-clusters-cgroups-okd-configure.adoc[leveloffset=+1] +endif::openshift-origin[] [id="post-install-tp-tasks"] == Enabling Technology Preview features using FeatureGates diff --git a/post_installation_configuration/machine-configuration-tasks.adoc b/post_installation_configuration/machine-configuration-tasks.adoc index a5141435db..f22b60cbba 100644 --- a/post_installation_configuration/machine-configuration-tasks.adoc +++ b/post_installation_configuration/machine-configuration-tasks.adoc @@ -74,3 +74,5 @@ Besides managing `MachineConfig` objects, the MCO manages two custom resources ( include::modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc[leveloffset=+2] include::modules/create-a-containerruntimeconfig-crd.adoc[leveloffset=+2] include::modules/set-the-default-max-container-root-partition-size-for-overlay-with-crio.adoc[leveloffset=+2] + +