From 36d9958abf96da570d56f1b56dff6b8138e1c264 Mon Sep 17 00:00:00 2001 From: Michael Burke Date: Wed, 4 May 2022 09:18:19 -0400 Subject: [PATCH] [enterprise-4.11] OSDOCS-3039: Improve kubelet CA update for Paused Machine Config Pools --- modules/compliance-applying.adoc | 5 +++++ modules/troubleshooting-disabling-autoreboot-mco.adoc | 6 +++++- modules/understanding-machine-config-operator.adoc | 2 ++ modules/update-using-custom-machine-config-pools-about.adoc | 4 +++- .../update-using-custom-machine-config-pools-canary.adoc | 6 +++++- modules/update-using-custom-machine-config-pools-pause.adoc | 4 +++- updating/update-using-custom-machine-config-pools.adoc | 6 +++++- 7 files changed, 28 insertions(+), 5 deletions(-) diff --git a/modules/compliance-applying.adoc b/modules/compliance-applying.adoc index 966d0af2f8..ebcccc4a64 100644 --- a/modules/compliance-applying.adoc +++ b/modules/compliance-applying.adoc @@ -16,6 +16,11 @@ After the Compliance Operator processes the applied remediation, the `status.App Note that when the Machine Config Operator applies a new `MachineConfig` object to nodes in a pool, all the nodes belonging to the pool are rebooted. This might be inconvenient when applying multiple remediations, each of which re-renders the composite `75-$scan-name-$suite-name` `MachineConfig` object. To prevent applying the remediation immediately, you can pause the machine config pool by setting the `.spec.paused` attribute of a `MachineConfigPool` object to `true`. +[NOTE] +==== +Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated. +==== + The Compliance Operator can apply remediations automatically. Set `autoApplyRemediations: true` in the `ScanSetting` top-level object. [WARNING] diff --git a/modules/troubleshooting-disabling-autoreboot-mco.adoc b/modules/troubleshooting-disabling-autoreboot-mco.adoc index 045d8c68cc..39488455fd 100644 --- a/modules/troubleshooting-disabling-autoreboot-mco.adoc +++ b/modules/troubleshooting-disabling-autoreboot-mco.adoc @@ -24,7 +24,11 @@ To avoid unwanted disruptions, you can modify the machine config pool (MCP) to p [NOTE] ==== -Pausing an MCP prevents the MCO from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically-rotated certificates from being pushed to the associated nodes, including the automatic rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires, and the MCO attempts to renew the certificate automatically, the new certificate is created but not applied across the nodes in the paused MCP. This causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`. Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only. +Pausing an MCP prevents the MCO from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically rotated certificates from being pushed to the associated nodes, including the automatic rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. + +If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires, and the MCO attempts to renew the certificate automatically, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated. + +Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only. New CA certificates are generated at 292 days from the installation date and removed at 365 days from that date. To determine the next automatic CA certificate rotation, see the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4]. ==== diff --git a/modules/understanding-machine-config-operator.adoc b/modules/understanding-machine-config-operator.adoc index 302e6627ea..9e325876ba 100644 --- a/modules/understanding-machine-config-operator.adoc +++ b/modules/understanding-machine-config-operator.adoc @@ -46,6 +46,8 @@ When changes are made to a machine configuration, the Machine Config Operator (M To prevent the nodes from automatically rebooting after machine configuration changes, before making the changes, you must pause the autoreboot process by setting the `spec.paused` field to `true` in the corresponding machine config pool. When paused, machine configuration changes are not applied until you set the `spec.paused` field to `false` and the nodes have rebooted into the new configuration. +Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated. + The following modifications do not trigger a node reboot: * When the MCO detects any of the following changes, it applies the update without draining or rebooting the node: diff --git a/modules/update-using-custom-machine-config-pools-about.adoc b/modules/update-using-custom-machine-config-pools-about.adoc index d587b28d8b..f382106235 100644 --- a/modules/update-using-custom-machine-config-pools-about.adoc +++ b/modules/update-using-custom-machine-config-pools-about.adoc @@ -26,7 +26,9 @@ Do not remove the default worker label from the nodes. The nodes *must* have a r + [NOTE] ==== -Pausing the MCP also pauses the kube-apiserver-to-kubelet-signer automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4] to find out how much time you have before the next automatic CA certificate rotation. Make sure the pools are unpaused when the CA cert rotation happens. If the MCPs are paused, the cert rotation does not happen, which causes the cluster to become degraded and causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`. +Pausing the MCP also pauses the `kube-apiserver-to-kubelet-signer` automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4] to find out how much time you have before the next automatic CA certificate rotation. + +Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated. ==== . Perform the cluster update. The update process updates the MCPs that are not paused, including the control plane nodes. diff --git a/modules/update-using-custom-machine-config-pools-canary.adoc b/modules/update-using-custom-machine-config-pools-canary.adoc index 2b80a2983f..4dbe979de3 100644 --- a/modules/update-using-custom-machine-config-pools-canary.adoc +++ b/modules/update-using-custom-machine-config-pools-canary.adoc @@ -26,7 +26,11 @@ The rolling update process described in this topic involves: [NOTE] ==== -Pausing an MCP prevents the Machine Config Operator from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically-rotated certificates from being pushed to the associated nodes, including the automatic CA rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires and the MCO attempts to automatically renew the certificate, the new certificate is created but not applied across the nodes in the respective machine config pool. This causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`. Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only. +Pausing an MCP prevents the Machine Config Operator from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically rotated certificates from being pushed to the associated nodes, including the automatic CA rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. + +If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires and the MCO attempts to automatically renew the certificate, the new certificate is created but not applied across the nodes in the respective machine config pool. This causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated. + +Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only. ==== //link that follows is in the assembly: updating-cluster-between-minor diff --git a/modules/update-using-custom-machine-config-pools-pause.adoc b/modules/update-using-custom-machine-config-pools-pause.adoc index 90c5df2fb0..b1a8d9e959 100644 --- a/modules/update-using-custom-machine-config-pools-pause.adoc +++ b/modules/update-using-custom-machine-config-pools-pause.adoc @@ -9,7 +9,9 @@ In this canary rollout update process, after you label the nodes that you do not [NOTE] ==== -Pausing the MCP also pauses the kube-apiserver-to-kubelet-signer automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4] to find out how much time you have before the next automatic CA certificate rotation. Make sure the pools are unpaused when the CA cert rotation happens. If the MCPs are paused, the cert rotation does not happen, which causes the cluster to become degraded and causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`. +Pausing the MCP also pauses the `kube-apiserver-to-kubelet-signer` automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the link:https://access.redhat.com/articles/5651701[Understand CA cert auto renewal in Red Hat OpenShift 4] to find out how much time you have before the next automatic CA certificate rotation. + +Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated. ==== To pause an MCP: diff --git a/updating/update-using-custom-machine-config-pools.adoc b/updating/update-using-custom-machine-config-pools.adoc index 51af3511eb..3233babddd 100644 --- a/updating/update-using-custom-machine-config-pools.adoc +++ b/updating/update-using-custom-machine-config-pools.adoc @@ -28,7 +28,11 @@ This scenario has not been tested and might result in an undefined cluster state [IMPORTANT] ==== -Pausing a machine config pool prevents the Machine Config Operator from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically-rotated certificates from being pushed to the associated nodes, including the automatic CA rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires and the MCO attempts to automatially renew the certificate, the new certificate is created but not applied across the nodes in the respective machine config pool. This causes failure in multiple `oc` commands, including but not limited to `oc debug`, `oc logs`, `oc exec`, and `oc attach`. Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only. +Pausing a machine config pool prevents the Machine Config Operator from applying any configuration changes on the associated nodes. Pausing an MCP also prevents any automatically rotated certificates from being pushed to the associated nodes, including the automatic CA rotation of the `kube-apiserver-to-kubelet-signer` CA certificate. + +If the MCP is paused when the `kube-apiserver-to-kubelet-signer` CA certificate expires and the MCO attempts to automatically renew the certificate, the MCO cannot push the newly rotated certificates to those nodes. This causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated. + +Pausing an MCP should be done with careful consideration about the `kube-apiserver-to-kubelet-signer` CA certificate expiration and for short periods of time only. ==== [id="update-using-custom-machine-config-pools-about-mcp_{context}"]