openshift/openshift-docs
1
0
Fork 0
mirror of https://github.com/openshift/openshift-docs.git synced 2026-02-05 12:46:18 +01:00
Code Issues Releases Activity

Merge pull request #105096 from openshift-cherrypick-robot/cherry-pick-103868-to-enterprise-4.21

Browse Source 
[enterprise-4.21] OSDOCS 17429 AutoSizingReserved is enabled by default
This commit is contained in:
Michael Burke
2026-01-20 16:26:44 -05:00
committed by GitHub
parent 68d889f704 447a6add5d
commit dea6097000
4 changed files with 103 additions and 173 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
modules/nodes-nodes-resources-configuring-about.adoc
View File

@@ -6,7 +6,8 @@
[id="nodes-nodes-resources-configuring-about_{context}"]
= Understanding how to allocate resources for nodes
CPU and memory resources reserved for node components in {product-title} are based on two node settings:
[role="_abstract"]
CPU and memory resources reserved for node components in {product-title} are based on two node settings: `kube-reserved` and `system-reserved`. These settings are automatically configured upon node start up, or you can manually set these values as needed.
[options="header",cols="1,2"]
|===
@@ -28,6 +29,39 @@ introduction of allocatable resources.
Any CPUs specifically reserved using the `reservedSystemCPUs` parameter are not available for allocation using `kube-reserved` or `system-reserved`.
====
By default, {product-title} uses a script on each worker node to automatically determine the optimal `system-reserved` CPU and memory resources for nodes. The script runs on node start up and uses the following calculations by default.
Memory reservation::
The memory reservation is weighted. For smaller nodes, {product-title} reserves a higher percentage of memory. For larger nodes, {product-title} reserves a smaller percentage of the remaining capacity.
+
{product-title} uses the following calculations to determine how much memory to reserve for node and system components:
+
--
* 25% of the first 4 GiB of memory
* 20% of the next 4 GiB of memory (up to 8 GiB)
* 10% of the next 8 GiB of memory (up to 16 GiB)
* 6% of the next 112 GiB of memory (up to 128 GiB)
* 2% of any memory above 128 GiB
--
+
For example, on node with 16 GiB of memory, {product-title} reserves 2.6 GiB for node and system components, leaving approximately 13.4 GiB for workloads.
CPU reservation::
{product-title} uses the following logic to determine how much CPU to reserve for node and system components:
+
--
* {product-title} starts with a base allocation for 1 CPU in fractions of a core (60 millicores = 0.06 CPU core).
* Then, it increments 12 millicores (0.012 CPU) for every additional core beyond the first.
* The result is compared against a minimum floor of 0.5 CPU. If the calculated value is less than 0.5, the system enforces a reservation of 0.5 CPU.
--
+
For example, on a 4-core node, 0.5 vCPU is reserved for node and system components, leaving 3.5 vCPUs for workloads. Note that 1000 millicores is equal to 1CPU/vCPU.
+
[IMPORTANT]
====
If you updated your cluster from a version earlier than 4.21, automatic allocation of system resources is disabled by default. To enable the feature, delete the `50-worker-auto-sizing-disabled` machine config.
====
[id="computing-allocated-resources_{context}"]
== How {product-title} computes allocated resources
@@ -44,7 +78,7 @@ The withholding of `Hard-Eviction-Thresholds` from `Allocatable` improves system
If `Allocatable` is negative, it is set to `0`.
Each node reports the system resources that are used by the container runtime and kubelet. To simplify configuring the `system-reserved` parameter, view the resource use for the node by using the node summary API. The node summary is available at `/api/v1/nodes/<node>/proxy/stats/summary`.
Each node reports the system resources that are used by the container runtime and kubelet. To simplify configuring the `system-reserved` parameter, view the resource use for the node by using the node summary API. The node summary is available at `/api/v1/nodes/<node>/proxy/stats/summary`. For more information, use the "Node metrics data" link in the _Additional resources_ section.
[id="allocate-node-enforcement_{context}"]
== How nodes enforce resource constraints

113
modules/nodes-nodes-resources-configuring-auto.adoc
View File

@@ -1,113 +0,0 @@
// Module included in the following assemblies:
//
// * nodes/nodes-nodes-resources-configuring.adoc
:_mod-docs-content-type: PROCEDURE
[id="nodes-nodes-resources-configuring-auto_{context}"]
= Automatically allocating resources for nodes
{product-title} can automatically determine the optimal `system-reserved` CPU and memory resources for nodes associated with a specific machine config pool and update the nodes with those values when the nodes start. By default, the `system-reserved` CPU is `500m` and `system-reserved` memory is `1Gi`.
To automatically determine and allocate the `system-reserved` resources on nodes, create a `KubeletConfig` custom resource (CR) to set the `autoSizingReserved: true` parameter. A script on each node calculates the optimal values for the respective reserved resources based on the installed CPU and memory capacity on each node. The script takes into account that increased capacity requires a corresponding increase in the reserved resources.
Automatically determining the optimal `system-reserved` settings ensures that your cluster is running efficiently and prevents node failure due to resource starvation of system components, such as CRI-O and kubelet, without your needing to manually calculate and update the values.
This feature is disabled by default.
.Prerequisites
. Obtain the label associated with the static `MachineConfigPool` object for the type of node you want to configure by entering the following command:
+
[source,terminal]
----
$ oc edit machineconfigpool <name>
----
+
For example:
+
[source,terminal]
----
$ oc edit machineconfigpool worker
----
+
.Example output
[source,yaml]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
creationTimestamp: "2022-11-16T15:34:25Z"
generation: 4
labels:
pools.operator.machineconfiguration.openshift.io/worker: "" <1>
name: worker
#...
----
<1> The label appears under `Labels`.
+
[TIP]
====
If an appropriate label is not present, add a key/value pair such as:
----
$ oc label machineconfigpool worker custom-kubelet=small-pods
----
====
.Procedure
. Create a custom resource (CR) for your configuration change:
+
.Sample configuration for a resource allocation CR
[source,yaml]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
name: dynamic-node <1>
spec:
autoSizingReserved: true <2>
machineConfigPoolSelector:
matchLabels:
pools.operator.machineconfiguration.openshift.io/worker: "" <3>
#...
----
<1> Assign a name to CR.
<2> Add the `autoSizingReserved` parameter set to `true` to allow {product-title} to automatically determine and allocate the `system-reserved` resources on the nodes associated with the specified label. To disable automatic allocation on those nodes, set this parameter to `false`.
<3> Specify the label from the machine config pool that you configured in the "Prerequisites" section. You can choose any desired labels for the machine config pool, such as `custom-kubelet: small-pods`, or the default label, `pools.operator.machineconfiguration.openshift.io/worker: ""`.
+
The previous example enables automatic resource allocation on all worker nodes. {product-title} drains the nodes, applies the kubelet config, and restarts the nodes.
. Create the CR by entering the following command:
+
[source,terminal]
----
$ oc create -f <file_name>.yaml
----
.Verification
. Log in to a node you configured by entering the following command:
+
[source,terminal]
----
$ oc debug node/<node_name>
----
. Set `/host` as the root directory within the debug shell:
+
[source,terminal]
----
# chroot /host
----
. View the `/etc/node-sizing.env` file:
+
.Example output
[source,terminal]
----
SYSTEM_RESERVED_MEMORY=3Gi
SYSTEM_RESERVED_CPU=0.08
----
+
The kubelet uses the `system-reserved` values in the `/etc/node-sizing.env` file. In the previous example, the worker nodes are allocated `0.08` CPU and 3 Gi of memory. It can take several minutes for the optimal values to appear.

103
modules/nodes-nodes-resources-configuring-setting.adoc
View File

@@ -6,57 +6,30 @@
[id="nodes-nodes-resources-configuring-setting_{context}"]
= Manually allocating resources for nodes
{product-title} supports the CPU and memory resource types for allocation. The `ephemeral-resource` resource type is also supported. For the `cpu` type, you specify the resource quantity in units of cores, such as `200m`, `0.5`, or `1`. For `memory` and `ephemeral-storage`, you specify the resource quantity in units of bytes, such as `200Ki`, `50Mi`, or `5Gi`. By default, the `system-reserved` CPU is `500m` and `system-reserved` memory is `1Gi`.
[role="_abstract"]
As an administrator, you can manually set `system-reserved` CPU and memory resources for your nodes. Setting these values ensures that your cluster is running efficiently and prevents node failure due to resource starvation of system components.
As an administrator, you can set these values by using a kubelet config custom resource (CR) through a set of `<resource_type>=<resource_quantity>` pairs
(e.g., `cpu=200m,memory=512Mi`).
By default, {product-title} uses a script on each worker node to automatically determine the optimal `system-reserved` CPU and memory resources for nodes associated with a specific machine config pool and update the nodes with those values. The script runs on node start up.
[IMPORTANT]
====
You must use a kubelet config CR to manually set resource values. You cannot use a machine config CR.
If you updated your cluster from a version earlier than 4.21, automatic allocation of system resources is disabled by default. To enable the feature, delete the `50-worker-auto-sizing-disabled` machine config.
====
However, you can manually set these values by using a `KubeletConfig` custom resource (CR) that includes a set of `<resource_type>=<resource_quantity>` pairs (for example, `cpu=200m,memory=512Mi`). You must use the `spec.autoSizingReserved: false` parameter in the `KubeletConfig` CR to override the default {product-title} behavior of automatically setting the `systemReserved` values.
For `memory` and `ephemeral-storage`, you specify the resource quantity in units of bytes, such as `200Ki`, `50Mi`, or `5Gi`. By default, the `system-reserved` CPU is `500m` and `system-reserved` memory is `1Gi`. For the `cpu` type, you specify the resource quantity in units of cores, such as `200m`, `0.5`, or `1`. Note that 1000 millicores is equal to 1CPU/vCPU.
For details on the recommended `system-reserved` values, refer to the link:https://access.redhat.com/solutions/5843241[recommended system-reserved values].
[IMPORTANT]
====
To manually set resource values, you must use a kubelet configuration. You cannot use a machine config.
====
.Prerequisites
. Obtain the label associated with the static `MachineConfigPool` CRD for the type of node you want to configure by entering the following command:
+
[source,terminal]
----
$ oc edit machineconfigpool <name>
----
+
For example:
+
[source,terminal]
----
$ oc edit machineconfigpool worker
----
+
.Example output
[source,yaml]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
creationTimestamp: "2022-11-16T15:34:25Z"
generation: 4
labels:
pools.operator.machineconfiguration.openshift.io/worker: "" <1>
name: worker
#...
----
<1> The label appears under Labels.
+
[TIP]
====
If the label is not present, add a key/value pair such as:
----
$ oc label machineconfigpool worker custom-kubelet=small-pods
----
====
* Obtain the label associated with the static `MachineConfigPool` CRD for the type of node you want to configure by entering the following command:
.Procedure
@@ -68,20 +41,26 @@ $ oc label machineconfigpool worker custom-kubelet=small-pods
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
name: set-allocatable <1>
name: set-allocatable
spec:
autoSizingReserved: false
machineConfigPoolSelector:
matchLabels:
pools.operator.machineconfiguration.openshift.io/worker: "" <2>
pools.operator.machineconfiguration.openshift.io/worker: ""
kubeletConfig:
systemReserved: <3>
systemReserved:
cpu: 1000m
memory: 1Gi
memory: 4Gi
ephemeral-storage: 50Mi
#...
----
<1> Assign a name to CR.
<2> Specify the label from the machine config pool.
<3> Specify the resources to reserve for the node components and system components.
+
--
`metadata.name`:: Specifies a name for the CR.
`spec.autoSizingReserved`:: Specify `false` to override the default {product-title} behavior of automatically setting the `systemReserved` values.
`spec.machineConfigPoolSelector.matchLabels`:: Specifies a label from the machine config pool.
`spec.kubeletConfig.systemReserved`:: Specifies the resources to reserve for the node components and system components.
--
. Run the following command to create the CR:
+
@@ -89,3 +68,31 @@ spec:
----
$ oc create -f <file_name>.yaml
----
.Verification
. Log in to a node you configured by entering the following command:
+
[source,terminal]
----
$ oc debug node/<node_name>
----
. Set `/host` as the root directory within the debug shell:
+
[source,terminal]
----
# chroot /host
----
. View the `/etc/node-sizing.env` file:
+
.Example output
[source,terminal]
----
SYSTEM_RESERVED_MEMORY=4Gi
SYSTEM_RESERVED_CPU=1000m
SYSTEM_RESERVED_ES=50Mi
----
+
The kubelet uses the `system-reserved` values in the `/etc/node-sizing.env` file.

22
nodes/nodes/nodes-nodes-resources-configuring.adoc
View File

@@ -6,11 +6,14 @@ include::_attributes/common-attributes.adoc[]
toc::[]
To provide more reliable scheduling and minimize node resource overcommitment, reserve a portion of the CPU and memory resources for use by the underlying node components, such as `kubelet` and `kube-proxy`, and the remaining system components, such as `sshd` and `NetworkManager`. By specifying the resources to reserve, you provide the scheduler with more information about the remaining CPU and memory resources that a node has available for use by pods. You can allow {product-title} to xref:../../nodes/nodes/nodes-nodes-resources-configuring.adoc#nodes-nodes-resources-configuring-auto_nodes-nodes-resources-configuring[automatically determine the optimal `system-reserved` CPU and memory resources] for your nodes or you can xref:../../nodes/nodes/nodes-nodes-resources-configuring.adoc#nodes-nodes-resources-configuring-setting_nodes-nodes-resources-configuring[manually determine and set the best resources] for your nodes.
[role="_abstract"]
By default, upon node start up {product-title} automatically calculates and reserves a portion of the CPU and memory resources for use by the underlying node components, such as `kubelet` and `kube-proxy`, and the remaining system components, such as `sshd` and `NetworkManager`. Review the information in this section to determine if these automatic settings are appropriate for your cluster.
You can modify the CPU and memory resources for these node and system components, as needed, by creating a `Kubelet Config` CR.
[IMPORTANT]
====
To manually set resource values, you must use a kubelet config CR. You cannot use a machine config CR.
If you updated your cluster from a version earlier than 4.21, automatic allocation of system resources is disabled by default. To enable the feature, delete the `50-worker-auto-sizing-disabled` machine config.
====
// The following include statements pull in the module files that comprise
@@ -22,16 +25,8 @@ include::modules/nodes-nodes-resources-configuring-about.adoc[leveloffset=+1]
include::modules/sd-understanding-process-id-limits.adoc[leveloffset=+1]
[role="_additional-resources"]
.Additional resources
* xref:../../machine_configuration/machine-configs-custom.adoc#create-a-kubeletconfig-crd-to-edit-kubelet-parameters_machine-configs-custom[Creating a KubeletConfig CR to edit kubelet parameters]
* xref:../../nodes/nodes/nodes-nodes-resources-configuring.adoc#nodes-nodes-resources-configuring[Allocating resources for nodes in an {product-title} cluster]
include::modules/risks-setting-higher-process-id-limits.adoc[leveloffset=+2]
include::modules/nodes-nodes-resources-configuring-auto.adoc[leveloffset=+1]
include::modules/nodes-nodes-resources-configuring-setting.adoc[leveloffset=+1]
////
@@ -50,3 +45,10 @@ details.
#update these links when the links become available#
////
[role="_additional-resources"]
[id="nodes-nodes-resources-configuring_additional-resources"]
== Additional resources
* xref:../../machine_configuration/machine-configs-custom.adoc#create-a-kubeletconfig-crd-to-edit-kubelet-parameters_machine-configs-custom[Creating a KubeletConfig CR to edit kubelet parameters]
* link:https://kubernetes.io/docs/reference/instrumentation/node-metrics/[Node metrics data (Kubernetes documentation)]