From 5f40dba4ee9f6b33469bdefb39b36076ad673425 Mon Sep 17 00:00:00 2001 From: GroceryBoyJr Date: Tue, 7 Nov 2023 15:14:37 -0500 Subject: [PATCH] resolving merge conflict - new module included --- ...cluster-worker-latency-profiles-about.adoc | 55 ++++++++--------- ...ter-worker-latency-profiles-examining.adoc | 59 +++++++++++++++++++ ...er-latency-profiles-using-at-creation.adoc | 43 ++++++++++++++ ...cluster-worker-latency-profiles-using.adoc | 18 +++--- .../scaling-worker-latency-profiles.adoc | 6 +- snippets/worker-latency-profile-intro.adoc | 19 ++++-- 6 files changed, 154 insertions(+), 46 deletions(-) create mode 100644 modules/nodes-cluster-worker-latency-profiles-examining.adoc create mode 100644 modules/nodes-cluster-worker-latency-profiles-using-at-creation.adoc diff --git a/modules/nodes-cluster-worker-latency-profiles-about.adoc b/modules/nodes-cluster-worker-latency-profiles-about.adoc index a11d01914a..eb223099c5 100644 --- a/modules/nodes-cluster-worker-latency-profiles-about.adoc +++ b/modules/nodes-cluster-worker-latency-profiles-about.adoc @@ -1,34 +1,32 @@ // Module included in the following assemblies: // -// * nodes/clusters/nodes-cluster-worker-latency-profiles -// * nodes/edge/nodes-edge-remote-workers. ?? -// * post_installation_configuration/cluster-tasks ?? +// scalability_and_performance/scaling-worker-latency-profiles.adoc :_mod-docs-content-type: CONCEPT [id="nodes-cluster-worker-latency-profiles-about_{context}"] = Understanding worker latency profiles -Worker latency profiles are multiple sets of carefully-tuned values for the `node-status-update-frequency`, `node-monitor-grace-period`, `default-not-ready-toleration-seconds` and `default-unreachable-toleration-seconds` parameters. These parameters let you control the reaction of the cluster to latency issues without needing to determine the best values manually. +Worker latency profiles are four different categories of carefully-tuned parameters. The four parameters which implement these values are `node-status-update-frequency`, `node-monitor-grace-period`, `default-not-ready-toleration-seconds` and `default-unreachable-toleration-seconds`. These parameters can use values which allow you control the reaction of the cluster to latency issues without needing to determine the best values using manual methods. + +[IMPORTANT] +==== +Setting these parameters manually is not supported. Incorrect parameter settings adversely affect cluster stability. +==== All worker latency profiles configure the following parameters: -- -* `node-status-update-frequency`. Specifies the amount of time in seconds that a kubelet updates its status to the Kubernetes Controller Manager Operator. -* `node-monitor-grace-period`. Specifies the amount of time in seconds that the Kubernetes Controller Manager Operator waits for an update from a kubelet before marking the node unhealthy and adding the `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` taint to the node. -* `default-not-ready-toleration-seconds`. Specifies the amount of time in seconds after marking a node unhealthy that the Kubernetes Controller Manager Operator waits before evicting pods from that node. -* `default-unreachable-toleration-seconds`. Specifies the amount of time in seconds after marking a node unreachable that the Kubernetes Controller Manager Operator waits before evicting pods from that node. +node-status-update-frequency:: Specifies how often the kubelet posts node status to the API server. +node-monitor-grace-period:: Specifies the amount of time in seconds that the Kubernetes Controller Manager waits for an update from a kubelet before marking the node unhealthy and adding the `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` taint to the node. +default-not-ready-toleration-seconds:: Specifies the amount of time in seconds after marking a node unhealthy that the Kube API Server Operator waits before evicting pods from that node. +default-unreachable-toleration-seconds:: Specifies the amount of time in seconds after marking a node unreachable that the Kube API Server Operator waits before evicting pods from that node. -- -[IMPORTANT] -==== -Manually modifying the `node-monitor-grace-period` parameter is not supported. -==== - The following Operators monitor the changes to the worker latency profiles and respond accordingly: * The Machine Config Operator (MCO) updates the `node-status-update-frequency` parameter on the worker nodes. -* The Kubernetes Controller Manager Operator updates the `node-monitor-grace-period` parameter on the control plane nodes. -* The Kubernetes API Server Operator updates the `default-not-ready-toleration-seconds` and `default-unreachable-toleration-seconds` parameters on the control plance nodes. +* The Kubernetes Controller Manager updates the `node-monitor-grace-period` parameter on the control plane nodes. +* The Kubernetes API Server Operator updates the `default-not-ready-toleration-seconds` and `default-unreachable-toleration-seconds` parameters on the control plane nodes. ifndef::openshift-rosa,openshift-dedicated[] Although the default configuration works in most cases, {product-title} offers two other worker latency profiles for situations where the network is experiencing higher latency than usual. The three worker latency profiles are described in the following sections: @@ -37,9 +35,11 @@ ifdef::openshift-rosa,openshift-dedicated[] Although the default configuration works in most cases, {product-title} offers a second worker latency profile for situations where the network is experiencing higher latency than usual. The two worker latency profiles are described in the following sections: endif::openshift-rosa,openshift-dedicated[] -Default worker latency profile:: With the `Default` profile, each kubelet reports its node status to the Kubelet Controller Manager Operator (kube controller) every 10 seconds. The Kubelet Controller Manager Operator checks the kubelet for a status every 5 seconds. +Default worker latency profile:: With the `Default` profile, each `Kubelet` updates it's status every 10 seconds (`node-status-update-frequency`). The `Kube Controller Manager` checks the statuses of `Kubelet` every 5 seconds (`node-monitor-grace-period`). + -The Kubernetes Controller Manager Operator waits 40 seconds for a status update before considering that node unhealthy. It marks the node with the `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` taint and evicts the pods on that node. If a pod on that node has the `NoExecute` toleration, the pod gets evicted in 300 seconds. If the pod has the `tolerationSeconds` parameter, the eviction waits for the period specified by that parameter. +The Kubernetes Controller Manager waits 40 seconds for a status update from `Kubelet` before considering the `Kubelet` unhealthy. If no status is made available to the Kubernetes Controller Manager, it then marks the node with the `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` taint and evicts the pods on that node. ++ +If a pod on that node has the `NoExecute` taint, the pod is run according to `tolerationSeconds`. If the pod has no taint, it will be evicted in 300 seconds (`default-not-ready-toleration-seconds` and `default-unreachable-toleration-seconds` settings of the `Kube API Server`). + [cols="2,1,2,1"] |=== @@ -54,11 +54,11 @@ The Kubernetes Controller Manager Operator waits 40 seconds for a status update | `node-monitor-grace-period` | 40s -| Kubernetes API Server +| Kubernetes API Server Operator | `default-not-ready-toleration-seconds` | 300s -| Kubernetes API Server +| Kubernetes API Server Operator | `default-unreachable-toleration-seconds` | 300s @@ -66,9 +66,9 @@ The Kubernetes Controller Manager Operator waits 40 seconds for a status update Medium worker latency profile:: Use the `MediumUpdateAverageReaction` profile if the network latency is slightly higher than usual. + -The `MediumUpdateAverageReaction` profile reduces the frequency of kubelet updates to 20 seconds and changes the period that the Kubernetes Controller Manager Operator waits for those updates to 2 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the `tolerationSeconds` parameter, the eviction waits for the period specified by that parameter. +The `MediumUpdateAverageReaction` profile reduces the frequency of kubelet updates to 20 seconds and changes the period that the Kubernetes Controller Manager waits for those updates to 2 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the `tolerationSeconds` parameter, the eviction waits for the period specified by that parameter. + -The Kubernetes Controller Manager Operator waits for 2 minutes to consider a node unhealthy. In another minute, the eviction process starts. +The Kubernetes Controller Manager waits for 2 minutes to consider a node unhealthy. In another minute, the eviction process starts. + [cols="2,1,2,1"] |=== @@ -83,22 +83,23 @@ The Kubernetes Controller Manager Operator waits for 2 minutes to consider a nod | `node-monitor-grace-period` | 2m -| Kubernetes API Server +| Kubernetes API Server Operator | `default-not-ready-toleration-seconds` | 60s -| Kubernetes API Server +| Kubernetes API Server Operator | `default-unreachable-toleration-seconds` | 60s |=== ifndef::openshift-rosa,openshift-dedicated[] + Low worker latency profile:: Use the `LowUpdateSlowReaction` profile if the network latency is extremely high. + -The `LowUpdateSlowReaction` profile reduces the frequency of kubelet updates to 1 minute and changes the period that the Kubernetes Controller Manager Operator waits for those updates to 5 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the `tolerationSeconds` parameter, the eviction waits for the period specified by that parameter. +The `LowUpdateSlowReaction` profile reduces the frequency of kubelet updates to 1 minute and changes the period that the Kubernetes Controller Manager waits for those updates to 5 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the `tolerationSeconds` parameter, the eviction waits for the period specified by that parameter. + -The Kubernetes Controller Manager Operator waits for 5 minutes to consider a node unhealthy. In another minute, the eviction process starts. +The Kubernetes Controller Manager waits for 5 minutes to consider a node unhealthy. In another minute, the eviction process starts. + [cols="2,1,2,1"] |=== @@ -113,11 +114,11 @@ The Kubernetes Controller Manager Operator waits for 5 minutes to consider a nod | `node-monitor-grace-period` | 5m -| Kubernetes API Server +| Kubernetes API Server Operator | `default-not-ready-toleration-seconds` | 60s -| Kubernetes API Server +| Kubernetes API Server Operator | `default-unreachable-toleration-seconds` | 60s diff --git a/modules/nodes-cluster-worker-latency-profiles-examining.adoc b/modules/nodes-cluster-worker-latency-profiles-examining.adoc new file mode 100644 index 0000000000..cde1472952 --- /dev/null +++ b/modules/nodes-cluster-worker-latency-profiles-examining.adoc @@ -0,0 +1,59 @@ +// Module included in the following assemblies: +// +// scalability_and_performance/scaling-worker-latency-profiles.adoc + + +:_mod-docs-content-type: PROCEDURE +[id="nodes-cluster-worker-latency-profiles-examining_{context}"] += Example steps for displaying resulting values of workerLatencyProfile + +You can display the values in the `workerLatencyProfile` with the following commands. + +.Verification + +. Check the `default-not-ready-toleration-seconds` and `default-unreachable-toleration-seconds` fields output by the Kube API Server: ++ +[source,terminal] +---- +$ oc get KubeAPIServer -o yaml | grep -A 1 default- +---- ++ +.Example output +[source,terminal] +---- +default-not-ready-toleration-seconds: +- "300" +default-unreachable-toleration-seconds: +- "300" +---- + +. Check the values of the `node-monitor-grace-period` field from the Kube Controller Manager: ++ +[source,terminal] +---- +$ oc get KubeControllerManager -o yaml | grep -A 1 node-monitor +---- ++ +.Example output +[source,terminal] +---- +node-monitor-grace-period: +- 40s +---- + +. Check the `nodeStatusUpdateFrequency` value from the Kubelet. Set the directory `/host` as the root directory within the debug shell. By changing the root directory to `/host`, you can run binaries contained in the host’s executable paths: ++ +[source,terminal] +---- +$ oc debug node/ +$ chroot /host +# cat /etc/kubernetes/kubelet.conf|grep nodeStatusUpdateFrequency +---- ++ +.Example output +[source,terminal] +---- + “nodeStatusUpdateFrequency”: “10s” +---- + +These outputs validate the set of timing variables for the Worker Latency Profile. \ No newline at end of file diff --git a/modules/nodes-cluster-worker-latency-profiles-using-at-creation.adoc b/modules/nodes-cluster-worker-latency-profiles-using-at-creation.adoc new file mode 100644 index 0000000000..7162b08f20 --- /dev/null +++ b/modules/nodes-cluster-worker-latency-profiles-using-at-creation.adoc @@ -0,0 +1,43 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/scaling-worker-latency-profiles.adoc + +:_mod-docs-content-type: PROCEDURE +[id="nodes-cluster-worker-latency-profiles-using-at-creation_{context}"] += Implementing worker latency profiles at cluster creation + +[IMPORTANT] +==== +To edit the configuration of the installer, you will first need to use the command `openshift-install create manifests` to create the default node manifest as well as other manifest YAML files. This file structure must exist before we can add workerLatencyProfile. The platform on which you are installing may have varying requirements. Refer to the Installing section of the documentation for your specific platform. +==== + +The `workerLatencyProfile` must be added to the manifest in the following sequence: + +. Create the manifest needed to build the cluster, using a folder name appropriate for your installation. +. Create a YAML file to define `config.node`. The file must be in the `manifests` directory. +. When defining `workerLatencyProfile` in the manifest for the first time, specify any of the profiles at cluster creation time: `Default`, `MediumUpdateAverageReaction` or `LowUpdateSlowReaction`. + +.Verification +* Here is an example manifest creation showing the `spec.workerLatencyProfile` `Default` value in the manifest file: ++ +[source,terminal] +---- +$ openshift-install create manifests --dir= +---- +* Edit the manifest and add the value. In this example we use `vi` to show an example manifest file with the "Default" `workerLatencyProfile` value added: ++ +[source,terminal] +---- +$ vi /manifests/config-node-default-profile.yaml +---- ++ +.Example output +[source,yaml] +---- +apiVersion: config.openshift.io/v1 +kind: Node +metadata: +name: cluster +spec: +workerLatencyProfile: "Default" +---- \ No newline at end of file diff --git a/modules/nodes-cluster-worker-latency-profiles-using.adoc b/modules/nodes-cluster-worker-latency-profiles-using.adoc index 7a05e8bb71..83e40b8656 100644 --- a/modules/nodes-cluster-worker-latency-profiles-using.adoc +++ b/modules/nodes-cluster-worker-latency-profiles-using.adoc @@ -1,17 +1,14 @@ // Module included in the following assemblies: // -// * nodes/clusters/nodes-cluster-worker-latency-profiles -// * Need to determine if these are good locations: -// * nodes/edge/nodes-edge-remote-workers -// * post_installation_configuration/cluster-tasks +// scalability_and_performance/scaling-worker-latency-profiles.adoc :_mod-docs-content-type: PROCEDURE [id="nodes-cluster-worker-latency-profiles-using_{context}"] -= Using worker latency profiles += Using and changing worker latency profiles -To implement a worker latency profile to deal with network latency, edit the `node.config` object to add the name of the profile. You can change the profile at any time as latency increases or decreases. +To change a worker latency profile to deal with network latency, edit the `node.config` object to add the name of the profile. You can change the profile at any time as latency increases or decreases. -You must move one worker latency profile at a time. For example, you cannot move directly from the `Default` profile to the `LowUpdateSlowReaction` worker latency profile. You must move from the `default` worker latency profile to the `MediumUpdateAverageReaction` profile first, then to `LowUpdateSlowReaction`. Similarly, when returning to the default profile, you must move from the low profile to the medium profile first, then to the default. +You must move one worker latency profile at a time. For example, you cannot move directly from the `Default` profile to the `LowUpdateSlowReaction` worker latency profile. You must move from the `Default` worker latency profile to the `MediumUpdateAverageReaction` profile first, then to `LowUpdateSlowReaction`. Similarly, when returning to the `Default` profile, you must move from the low profile to the medium profile first, then to `Default`. [NOTE] ==== @@ -100,8 +97,8 @@ spec: # ... ---- -<1> Specifies to use the low worker latency policy. -+ +<1> Specifies use of the low worker latency policy. + Scheduling on each worker node is disabled as the change is being applied. .Verification @@ -136,5 +133,4 @@ $ oc get KubeControllerManager -o yaml | grep -i workerlatency -A 5 -B 5 ---- <1> Specifies that the profile is applied and active. -To change the medium profile to default or change the default to medium, edit the `node.config` object and set the `spec.workerLatencyProfile` parameter to the appropriate value. - +To change the medium profile to default or change the default to medium, edit the `node.config` object and set the `spec.workerLatencyProfile` parameter to the appropriate value. \ No newline at end of file diff --git a/scalability_and_performance/scaling-worker-latency-profiles.adoc b/scalability_and_performance/scaling-worker-latency-profiles.adoc index 7ace1f2a64..e02cd95db1 100644 --- a/scalability_and_performance/scaling-worker-latency-profiles.adoc +++ b/scalability_and_performance/scaling-worker-latency-profiles.adoc @@ -16,8 +16,10 @@ toc::[] include::snippets/worker-latency-profile-intro.adoc[] -You can configure worker latency profiles when installing a cluster or at any time you notice increased latency in your cluster network. - include::modules/nodes-cluster-worker-latency-profiles-about.adoc[leveloffset=+1] +include::modules/nodes-cluster-worker-latency-profiles-using-at-creation.adoc[leveloffset=+1] + include::modules/nodes-cluster-worker-latency-profiles-using.adoc[leveloffset=+1] + +include::modules/nodes-cluster-worker-latency-profiles-examining.adoc[leveloffset=+1] \ No newline at end of file diff --git a/snippets/worker-latency-profile-intro.adoc b/snippets/worker-latency-profile-intro.adoc index 5e8c49bbc4..96e8c16062 100644 --- a/snippets/worker-latency-profile-intro.adoc +++ b/snippets/worker-latency-profile-intro.adoc @@ -8,16 +8,23 @@ :_mod-docs-content-type: SNIPPET -All nodes send heartbeats to the Kubernetes Controller Manager Operator (kube controller) in the {product-title} cluster every 10 seconds, by default. If the cluster does not receive heartbeats from a node, {product-title} responds using several default mechanisms. -For example, if the Kubernetes Controller Manager Operator loses contact with a node after a configured period: -. The node controller on the control plane updates the node health to `Unhealthy` and marks the node `Ready` condition as `Unknown`. +If the cluster administrator has performed latency tests for platform verification, they can discover the need to adjust the operation of the cluster to ensure stability in cases of high latency. The cluster administrator need change only one parameter, recorded in a file, which controls four parameters affecting how supervisory processes read status and interpret the health of the cluster. Changing only the one parameter provides cluster tuning in an easy, supportable manner. + +The `Kubelet` process provides the starting point for monitoring cluster health. The `Kubelet` sets status values for all nodes in the {product-title} cluster. The Kubernetes Controller Manager (`kube controller`) reads the status values every 10 seconds, by default. +If the `kube controller` cannot read a node status value, it loses contact with that node after a configured period. The default behavior is: + +. The node controller on the control plane updates the node health to `Unhealthy` and marks the node `Ready` condition`Unknown`. . In response, the scheduler stops scheduling pods to that node. -. The on-premise node controller adds a `node.kubernetes.io/unreachable` taint with a `NoExecute` effect to the node and schedules any pods on the node for eviction after five minutes, by default. +. The Node Lifecycle Controller adds a `node.kubernetes.io/unreachable` taint with a `NoExecute` effect to the node and schedules any pods on the node for eviction after five minutes, by default. -This behavior can cause problems if your network is prone to latency issues, especially if you have nodes at the network edge. In some cases, the Kubernetes Controller Manager Operator might not receive an update from a healthy node due to network latency. The Kubernetes Controller Manager Operator would then evict pods from the node even though the node is healthy. To avoid this problem, you can use _worker latency profiles_ to adjust the frequency that the kubelet and the Kubernetes Controller Manager Operator wait for status updates before taking action. These adjustments help to ensure that your cluster runs properly in the event that network latency between the control plane and the worker nodes is not optimal. +This behavior can cause problems if your network is prone to latency issues, especially if you have nodes at the network edge. In some cases, the Kubernetes Controller Manager might not receive an update from a healthy node due to network latency. The `Kubelet` evicts pods from the node even though the node is healthy. -These worker latency profiles are three sets of parameters that are pre-defined with carefully tuned values that let you control the reaction of the cluster to latency issues without needing to determine the best values manually. +To avoid this problem, you can use _worker latency profiles_ to adjust the frequency that the `Kubelet` and the Kubernetes Controller Manager wait for status updates before taking action. These adjustments help to ensure that your cluster runs properly if network latency between the control plane and the worker nodes is not optimal. + +These worker latency profiles contain three sets of parameters that are pre-defined with carefully tuned values to control the reaction of the cluster to increased latency. No need to experimentally find the best values manually. + +You can configure worker latency profiles when installing a cluster or at any time you notice increased latency in your cluster network. \ No newline at end of file