mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
79 lines
2.0 KiB
Plaintext
79 lines
2.0 KiB
Plaintext
// Module included in the following assemblies:
|
|
//
|
|
// * disaster_recovery/about-disaster-recovery.adoc
|
|
// * etcd/etcd-backup-restore/etcd-disaster-recovery.adoc
|
|
|
|
:_mod-docs-content-type: PROCEDURE
|
|
[id="dr-testing-restore-procedures_{context}"]
|
|
= Testing restore procedures
|
|
|
|
Testing the restore procedure is important to ensure that your automation and workload handle the new cluster state gracefully. Due to the complex nature of etcd quorum and the etcd Operator attempting to mend automatically, it is often difficult to correctly bring your cluster into a broken enough state that it can be restored.
|
|
|
|
[WARNING]
|
|
====
|
|
You **must** have SSH access to the cluster. Your cluster might be entirely lost without SSH access.
|
|
====
|
|
|
|
.Prerequisites
|
|
|
|
* You have SSH access to control plane hosts.
|
|
* You have installed the {oc-first}.
|
|
|
|
.Procedure
|
|
|
|
. Use SSH to connect to each of your nonrecovery nodes and run the following commands to disable etcd and the `kubelet` service:
|
|
|
|
.. Disable etcd by running the following command:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ sudo /usr/local/bin/disable-etcd.sh
|
|
----
|
|
|
|
.. Delete variable data for etcd by running the following command:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ sudo rm -rf /var/lib/etcd
|
|
----
|
|
|
|
.. Disable the `kubelet` service by running the following command:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ sudo systemctl disable kubelet.service
|
|
----
|
|
|
|
. Exit every SSH session.
|
|
|
|
. Run the following command to ensure that your nonrecovery nodes are in a `NOT READY` state:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ oc get nodes
|
|
----
|
|
|
|
. Follow the steps in "Restoring to a previous cluster state" to restore your cluster.
|
|
|
|
. After you restore the cluster and the API responds, use SSH to connect to each nonrecovery node and enable the `kubelet` service:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ sudo systemctl enable kubelet.service
|
|
----
|
|
|
|
. Exit every SSH session.
|
|
|
|
. Run the following command to observe your nodes coming back into the `READY` state:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ oc get nodes
|
|
----
|
|
|
|
. Run the following command to verify that etcd is available:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ oc get pods -n openshift-etcd
|
|
---- |