mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
255 lines
16 KiB
Plaintext
255 lines
16 KiB
Plaintext
// Module included in the following assemblies:
|
|
//
|
|
// * machine_configuration/machine-config-index.adoc
|
|
|
|
:_mod-docs-content-type: CONCEPT
|
|
[id="checking-mco-node-status_{context}"]
|
|
= About checking machine config node status
|
|
|
|
If you make changes to the default control plane or worker nodes in your cluster, for example by using a machine config or kubelet config, you can get detailed information about the progress of the node updates by using the machine config nodes custom resource. Any change that results in a new machine config triggers the node update process.
|
|
|
|
You can view this detailed information for nodes in the `control plane` and `worker` machine config pools that were created upon {product-title} installation. You cannot use custom machine config pools.
|
|
|
|
The `MachineConfigNode` custom resource allows you to monitor the progress of individual node updates as they move through the upgrade phases. This information can be helpful with troubleshooting if one of the nodes has an issue during the update. The custom resource reports where in the update process the node is at the moment, the phases that have completed, and the phases that are remaining.
|
|
|
|
:FeatureName: The machine config node custom resource
|
|
include::snippets/technology-preview.adoc[]
|
|
|
|
The node update process consists of the following phases and subphases that are tracked by the machine config node custom resource, explained with more detail later in this section:
|
|
|
|
* *Update Prepared*. The MCO stops the configuration drift monitoring process and verifies that the newly-created machine config can be applied to a node.
|
|
** *UpdateCompatible*
|
|
* *Update Executed*. The MCO cordons and drains the node and applies the new machine config to the node's files and operating system, as needed. It contains the following sub-phases:
|
|
** *Cordoned*
|
|
** *Drained*
|
|
** *AppliedFilesAndOS*
|
|
* *PinnedImageSetsProgressing* The MCO is performing the steps needed to pin and pre-load container images.
|
|
* *PinnedImageSetsDegraded* The pinned image process failed. You can view the reason for the failure by using the `oc describe machineconfignode` command, as described later in this section.
|
|
* *Update Post update action*. The MCO reboots the node or reloads CRI-O, as needed.
|
|
** *RebootedNode*
|
|
** *ReloadedCRIO*
|
|
* *Update Complete*. The MCO uncordons the nodes, updates the node state to the cluster, and resumes producing node metrics.
|
|
** *Updated*
|
|
** *Uncordoned*
|
|
* *Resumed*. The MCO restarts the config drift monitor process and the node returns to operational state.
|
|
|
|
As the update moves through these phases, you can query the `MachineConfigNode` custom resource, which reports one of the following conditions for each phase:
|
|
|
|
* `True`. The phase is complete on that node or the MCO has started that phase on that node.
|
|
* `False`. The phase is either being executed or will not be executed on that node.
|
|
* `Unknown`. The phase is either being executed on that node or has an error. If the phase has an error, you can use the `oc describe machineconfignodes` command for more information, as described later in this section.
|
|
|
|
For example, consider a cluster with a newly-created machine config:
|
|
|
|
[source,terminal]
|
|
----
|
|
$ oc get machineconfig
|
|
----
|
|
|
|
.Example output
|
|
[source,text]
|
|
----
|
|
# ...
|
|
rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 6d15h
|
|
rendered-master-a386c2d1550b927d274054124f58be68 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 7m26s
|
|
# ...
|
|
rendered-worker-01f27f752eb84eba917450e43636b210 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 6d15h <1>
|
|
rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 7m26s <2>
|
|
# ...
|
|
----
|
|
<1> The current machine config for the control plane and worker nodes.
|
|
<2> A newly-created machine config that is being applied to the control plane and worker nodes.
|
|
|
|
You can watch as the nodes are updated with the new machine config:
|
|
|
|
[source,terminal]
|
|
----
|
|
$ oc get machineconfignodes
|
|
----
|
|
|
|
.Example output
|
|
[source,text]
|
|
----
|
|
NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED
|
|
ci-ln-ds73n5t-72292-9xsm9-master-0 master rendered-master-a386c2d1550b927d274054124f58be68 rendered-master-a386c2d1550b927d274054124f58be68 True
|
|
ci-ln-ds73n5t-72292-9xsm9-master-1 master rendered-master-a386c2d1550b927d274054124f58be68 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 False
|
|
ci-ln-ds73n5t-72292-9xsm9-master-2 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True
|
|
ci-ln-ds73n5t-72292-9xsm9-worker-a-2d8tz worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 True <1>
|
|
ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-01f27f752eb84eba917450e43636b210 False <2>
|
|
ci-ln-ds73n5t-72292-9xsm9-worker-c-t227w worker rendered-worker-01f27f752eb84eba917450e43636b210 rendered-worker-01f27f752eb84eba917450e43636b210 True <3>
|
|
----
|
|
<1> This node has been updated. The new machine config, `rendered-worker-f351f6947f15cd0380514f4b1c89f8f2`, is shown as the desired and current machine configs.
|
|
<2> This node is currently being updated to the new machine config. The previous and new machine configs are shown as the desired and current machine configs, respectively.
|
|
<3> This node has not yet been updated to the new machine config. The previous machine config is shown as the desired and current machine configs.
|
|
|
|
.Basic machine config node fields
|
|
[cols="1,4",options="header"]
|
|
|===
|
|
|Field |Meaning
|
|
|`NAME` |The name of the node.
|
|
|`POOLNAME` |The name of the machine config pool associated with that node.
|
|
|`DESIREDCONFIG` |The name of the new machine config that the node updates to.
|
|
|`CURRENTCONFIG` |The name of the current machine configuration on that node.
|
|
|`UPDATED` a|Indicates if the node has been updated with one of the following conditions:
|
|
|
|
* If `False`, the node is being updated to the new machine configuration shown in the `DESIREDCONFIG` field.
|
|
* If `True`, and the `CURRENTCONFIG` matches the new machine configuration shown in the `DESIREDCONFIG` field, the node has been updated.
|
|
* If `True`, and the `CURRENTCONFIG` matches the old machine configuration shown in the `DESIREDCONFIG` field, that node has not been updated yet.
|
|
|===
|
|
// Field definitions based on https://github.com/openshift/api/pull/1596
|
|
|
|
You can use the `-o wide` flag to display additional information about the updates:
|
|
|
|
[source,terminal]
|
|
----
|
|
$ oc get machineconfignodes -o wide
|
|
----
|
|
|
|
.Example output
|
|
[source,text]
|
|
----
|
|
$ oc get machineconfignode -o wide
|
|
NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED UPDATEPREPARED UPDATEEXECUTED UPDATEPOSTACTIONCOMPLETE UPDATECOMPLETE RESUMED UPDATECOMPATIBLE UPDATEDFILESANDOS CORDONEDNODE DRAINEDNODE REBOOTEDNODE RELOADEDCRIO UNCORDONEDNODE
|
|
ci-ln-ds73n5t-72292-9xsm9-master-0 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False
|
|
ci-ln-ds73n5t-72292-9xsm9-master-1 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False
|
|
ci-ln-ds73n5t-72292-9xsm9-master-2 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False
|
|
ci-ln-ds73n5t-72292-9xsm9-worker-a-2d8tz worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 True False False False False False False False False False False False False
|
|
ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-01f27f752eb84eba917450e43636b210 False True True Unknown False False True True True True Unknown False False
|
|
ci-ln-ds73n5t-72292-9xsm9-worker-c-t227w worker rendered-worker-01f27f752eb84eba917450e43636b210 rendered-worker-01f27f752eb84eba917450e43636b210 True False False False False False False False False False False False False
|
|
----
|
|
|
|
In addition to the fields defined in the previous table, the `-o wide` output displays the following fields:
|
|
|
|
.Machine config node fields in the `-o wide` output
|
|
[cols="1,4",options="header"]
|
|
|===
|
|
|Phase Name |Definition
|
|
|`UPDATEPREPARED` |Indicates if the MCO is preparing to update the node.
|
|
|`UPDATEEXECUTED` |Indicates if the MCO has completed the body of the update on the node.
|
|
|`UPDATEPOSTACTIONCOMPLETE` |Indicates if the MCO has executed the post-update actions on the node.
|
|
|`UPDATECOMPLETE` |Indicates if the MCO has completed the update on the node.
|
|
|`RESUMED` |Indicates if the node has resumed normal processes.
|
|
|`UPDATECOMPATIBLE` |Indicates if the MCO has determined it can execute the update on the node.
|
|
|`UPDATEDFILESANDOS` |Indicates if the MCO has updated the node files and operating system.
|
|
|`CORDONEDNODE` |Indicates if the MCO has marked the node as not schedulable.
|
|
|`DRAINEDNODE` |Indicates if the MCO has drained the node.
|
|
|`REBOOTEDNODE` |Indicates if the MCO has restarted the node.
|
|
|`RELOADEDCRIO` |Indicates if the MCO has restarted the CRI-O service.
|
|
|`UNCORDONEDNODE` |Indicates if the MCO has marked the node as schedulable.
|
|
|===
|
|
|
|
For more details on the update status, you can use the `oc describe machineconfignode` command:
|
|
|
|
[source,terminal]
|
|
----
|
|
$ oc describe machineconfignode/<machine_config_node_name> -o yaml
|
|
----
|
|
|
|
.Example output
|
|
[source,text]
|
|
----
|
|
Name: <machine_config_node_name> <1>
|
|
Namespace:
|
|
Labels: <none>
|
|
Annotations: <none>
|
|
API Version: machineconfiguration.openshift.io/v1alpha1
|
|
Kind: MachineConfigNode
|
|
Metadata:
|
|
Creation Timestamp: 2023-10-17T13:08:58Z
|
|
Generation: 1
|
|
Resource Version: 49443
|
|
UID: 4bd758ab-2187-413c-ac42-882e61761b1d
|
|
Spec:
|
|
Node Ref:
|
|
Name: <node_name>
|
|
Pool:
|
|
Name: worker
|
|
ConfigVersion:
|
|
Desired: rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 <2>
|
|
Status:
|
|
Conditions:
|
|
Last Transition Time: 2025-01-14T17:01:16Z
|
|
Message: Node ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd needs an update
|
|
Reason: Updated
|
|
Status: False
|
|
Type: Updated
|
|
Last Transition Time: 2025-01-14T17:01:18Z
|
|
Message: Update is Compatible.
|
|
Reason: UpdateCompatible
|
|
Status: True
|
|
Type: UpdatePrepared
|
|
Last Transition Time: 2025-01-14T17:04:08Z
|
|
Message: Updated the Files and OS on disk as a part of the in progress phase
|
|
Reason: AppliedFilesAndOS
|
|
Status: True
|
|
Type: UpdateExecuted
|
|
Last Transition Time: 2025-01-14T17:04:08Z
|
|
Message: Node will reboot into config rendered-worker-db01b33f959e5645a721da50a6db1fbb
|
|
Reason: RebootedNode
|
|
Status: Unknown
|
|
Type: UpdatePostActionComplete
|
|
Last Transition Time: 2025-01-14T16:04:27Z
|
|
Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: UnCordoned Node as part of completing upgrade phase
|
|
Reason: Uncordoned
|
|
Status: False
|
|
Type: UpdateComplete
|
|
Last Transition Time: 2025-01-14T16:04:27Z
|
|
Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: In desired config rendered-worker-01f27f752eb84eba917450e43636b210. Resumed normal operations.
|
|
Reason: Resumed
|
|
Status: False
|
|
Type: Resumed
|
|
Last Transition Time: 2025-01-14T17:01:18Z
|
|
Message: Update Compatible. Post Cfg Actions []: Drain Required: true
|
|
Reason: UpdatePreparedUpdateCompatible
|
|
Status: True
|
|
Type: UpdateCompatible
|
|
Last Transition Time: 2025-01-14T17:03:57Z
|
|
Message: Drained node. The drain is complete as the desired drainer matches current drainer: drain-rendered-worker-db01b33f959e5645a721da50a6db1fbb
|
|
Reason: UpdateExecutedDrained
|
|
Status: True
|
|
Type: Drained
|
|
Last Transition Time: 2025-01-14T17:04:08Z
|
|
Message: Applied files and new OS config to node. OS did not need an update. SSH Keys did not need an update
|
|
Reason: UpdateExecutedAppliedFilesAndOS
|
|
Status: True
|
|
Type: AppliedFilesAndOS
|
|
Last Transition Time: 2025-01-14T17:01:23Z
|
|
Message: Cordoned node. The node is reporting Unschedulable = true
|
|
Reason: UpdateExecutedCordoned
|
|
Status: True
|
|
Type: Cordoned
|
|
Last Transition Time: 2025-01-14T17:04:08Z
|
|
Message: Upgrade requires a reboot. Currently doing this as the post update action.
|
|
Reason: UpdatePostActionCompleteRebootedNode
|
|
Status: Unknown
|
|
Type: RebootedNode
|
|
Last Transition Time: 2025-01-14T15:30:57Z
|
|
Message: This node has not yet entered the ReloadedCRIO phase
|
|
Reason: NotYetOccured
|
|
Status: False
|
|
Type: ReloadedCRIO
|
|
Last Transition Time: 2025-01-14T16:04:27Z
|
|
Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: UnCordoned node. The node is reporting Unschedulable = false
|
|
Reason: UpdateCompleteUncordoned
|
|
Status: False
|
|
Type: Uncordoned
|
|
Last Transition Time: 2025-01-14T16:04:07Z
|
|
Message: All is good
|
|
Reason: AsExpected
|
|
Status: False
|
|
Type: PinnedImageSetsDegraded
|
|
Last Transition Time: 2025-01-14T16:04:07Z
|
|
Message: All pinned image sets complete
|
|
Reason: AsExpected
|
|
Status: False
|
|
Type: PinnedImageSetsProgressing
|
|
Config Version:
|
|
Current: rendered-worker-01f27f752eb84eba917450e43636b210 <3>
|
|
Desired: rendered-worker-f351f6947f15cd0380514f4b1c89f8f2
|
|
Observed Generation: 6
|
|
# ...
|
|
----
|
|
<1> The `MachineConfigNode` object name.
|
|
<2> The new machine configuration. This field updates after the MCO validates the machine config in the `UPDATEPREPARED` phase, then the status adds the new configuration.
|
|
<3> The current machine config on the node.
|