mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
OSDOCS-11312: adds enhancements and nit fixes to auto recovery MicroShift
This commit is contained in:
“Shauna Diaz”
@@ -6,25 +6,16 @@ include::_attributes/attributes-microshift.adoc[]
|
||||
|
||||
toc::[]
|
||||
|
||||
You can automatically restore data from manual backups when {microshift-short} fails to start by using the `auto-recovery` feature.
|
||||
You can automatically restore data from manual backups when {microshift-short} fails to start by configuring automatic recovery.
|
||||
|
||||
You can use the following options with the existing `backup` and `restore` commands in this feature:
|
||||
include::modules/microshift-auto-recovery-manual-backups.adoc[leveloffset=+1]
|
||||
|
||||
* `--auto-recovery`: Selects the most recent version of the backup, and then restores it. This option treats the `PATH` argument as a path to a directory that holds all the backups for automated recovery, and not just as a path to a particular backup file.
|
||||
* `--dont-save-failed`: Disables the backup of failed {microshift-short} data.
|
||||
include::modules/microshift-creating-backups-auto-recovery.adoc[leveloffset=+1]
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
* You can use the `--auto-recovery` option with both the `backup` and `restore` commands.
|
||||
* You can use the `--dont-save-failed` option only with the `restore` command.
|
||||
====
|
||||
include::modules/microshift-restoring-backups-auto-recovery.adoc[leveloffset=+1]
|
||||
|
||||
include::modules/microshift-creating-backups.adoc[leveloffset=+1]
|
||||
include::modules/microshift-auto-recovery-example-rpm-systems.adoc[leveloffset=+2]
|
||||
|
||||
include::modules/microshift-restoring-backups.adoc[leveloffset=+1]
|
||||
include::modules/microshift-auto-recovery-example-ostree-systems.adoc[leveloffset=+2]
|
||||
|
||||
include::modules/microshift-automation-example-rpm-systems.adoc[leveloffset=+1]
|
||||
|
||||
include::modules/microshift-automation-example-ostree-systems.adoc[leveloffset=+1]
|
||||
|
||||
include::modules/microshift-automation-example-bootc-systems.adoc[leveloffset=+1]
|
||||
include::modules/microshift-auto-recovery-example-bootc-systems.adoc[leveloffset=+2]
|
||||
|
||||
93
modules/microshift-auto-recovery-example-bootc-systems.adoc
Normal file
93
modules/microshift-auto-recovery-example-bootc-systems.adoc
Normal file
@@ -0,0 +1,93 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc
|
||||
|
||||
:_mod-docs-content-type: PROCEDURE
|
||||
[id="microshift-auto-recovery-example-bootc-systems_{context}"]
|
||||
= Using automatic recovery in image mode for {op-system-base} systems
|
||||
|
||||
:FeatureName: Image mode for {op-system-base}
|
||||
|
||||
include::snippets/technology-preview.adoc[]
|
||||
|
||||
As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for image mode for {op-system-base-full} systems that use the systemd service.
|
||||
|
||||
[IMPORTANT]
|
||||
====
|
||||
You must include the entire `auto-recovery` process for {op-system-image} systems that use `systemd` in the container file.
|
||||
====
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You created a Containerfile as instructed in link:https://docs.redhat.com/en/documentation/red_hat_build_of_microshift/4.18/html/installing_with_rhel_image_mode/installing-with-rhel-image-mode#microshift-rhel-image-mode-build-image_microshift-install-rhel-image-mode[Building the bootc image].
|
||||
|
||||
* You created the `10-auto-recovery.conf` and `microshift-auto-recovery.service` files as explained in the "Using auto-recovery in RPM systems" section.
|
||||
+
|
||||
[IMPORTANT]
|
||||
====
|
||||
The location of the the `10-auto-recovery.conf` and `microshift-auto-recovery.service` must be relative to the Containerfile.
|
||||
|
||||
For example, if the path to the Containerfile is `/home/microshift/my-build/Containerfile`, the systemd files need to be adjacent for proper embedding. The following paths are correct for this example:
|
||||
|
||||
* `/home/microshift/my-build/auto-rec/10-auto-recovery.conf`
|
||||
* `/home/microshift/my-build/auto-rec/microshift-auto-recovery.service`
|
||||
* `/home/microshift/my-build/auto-rec/microshift-auto-recovery`
|
||||
====
|
||||
|
||||
* You created the `microshift-auto-recovery` script as explained in the "Using auto-recovery in RPM systems" section.
|
||||
|
||||
.Procedure
|
||||
|
||||
. Use the following example snippet to update the container file that you use to prepare the {op-system-image} image.
|
||||
+
|
||||
[source,text]
|
||||
----
|
||||
RUN mkdir -p /usr/lib/systemd/system/microshift.service.d
|
||||
COPY ./auto-rec/10-auto-recovery.conf /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf
|
||||
COPY ./auto-rec/microshift-auto-recovery.service /usr/lib/systemd/system/
|
||||
COPY ./auto-rec/microshift-auto-recovery /usr/bin/
|
||||
RUN chmod +x /usr/bin/microshift-auto-recovery
|
||||
----
|
||||
+
|
||||
[IMPORTANT]
|
||||
====
|
||||
Podman uses the host subscription information and repositories inside the container when building the container image. If the `rhocp` and `fast-datapath` repositories are not available on the host, the build fails.
|
||||
====
|
||||
|
||||
. Rebuild your local bootc image by running the following image build command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
PULL_SECRET=~/.pull-secret.json
|
||||
USER_PASSWD=<your_redhat_user_password>
|
||||
IMAGE_NAME=microshift-4.18-bootc
|
||||
|
||||
sudo podman build --authfile "${PULL_SECRET}" -t "${IMAGE_NAME}" \
|
||||
--build-arg USER_PASSWD="${USER_PASSWD}" \
|
||||
-f Containerfile
|
||||
----
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
Secrets are used during the image build in the following ways:
|
||||
|
||||
* The podman `--authfile` argument is required to pull the base `rhel-bootc:9.4` image from the `registry.redhat.io` registry.
|
||||
|
||||
* The build `USER_PASSWD` argument is used to set a password for the `redhat user`.
|
||||
====
|
||||
|
||||
.Verification
|
||||
|
||||
* Verify that the local bootc image was created by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ sudo podman images "${IMAGE_NAME}"
|
||||
----
|
||||
+
|
||||
.Example output
|
||||
[source,text]
|
||||
----
|
||||
REPOSITORY TAG IMAGE ID CREATED SIZE
|
||||
localhost/microshift-4.18-bootc latest 193425283c00 2 minutes ago 2.31 GB
|
||||
----
|
||||
@@ -3,50 +3,60 @@
|
||||
// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc
|
||||
|
||||
:_mod-docs-content-type: PROCEDURE
|
||||
[id="microshift-automation-example-ostree-systems_{context}"]
|
||||
= Automating the integration process with systemd for OSTree systems
|
||||
[id="microshift-auto-recovery-ostree-systems_{context}"]
|
||||
= Using automatic recovery with {op-system-ostree}
|
||||
|
||||
As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for {op-system-ostree-first} systems that use systemd in the blueprint file.
|
||||
|
||||
[IMPORTANT]
|
||||
====
|
||||
You must include the entire `auto-recovery` process for OSTree systems that use `systemd` in the blueprint file.
|
||||
You must include the entire `auto-recovery` process for {op-system-ostree} systems that use `systemd` in the blueprint file.
|
||||
====
|
||||
|
||||
As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for OSTree systems that use systemd.
|
||||
.Prerequisites
|
||||
|
||||
* You installed Podman.
|
||||
* You installed the command-line `composer-cli` tool.
|
||||
|
||||
.Procedure
|
||||
|
||||
. Optional: Because the `composer-cli` can only create files in the `/etc` directory, package your files into an RPM that you include the blueprint.
|
||||
|
||||
. Use the following example to create your blueprint file:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
[[customizations.directories]]
|
||||
path = "/etc/systemd/system/microshift.service.d"
|
||||
|
||||
[[customizations.directories]]
|
||||
path = "/etc/bin"
|
||||
|
||||
[[customizations.files]]
|
||||
path = "/usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf"
|
||||
path = "/etc/systemd/system/microshift.service.d/10-auto-recovery.conf"
|
||||
data = """
|
||||
[Unit]
|
||||
OnFailure=microshift-auto-recovery.service
|
||||
"""
|
||||
|
||||
[[customizations.files]]
|
||||
path = "/usr/lib/systemd/system/microshift-auto-recovery.service"
|
||||
path = "/etc/systemd/system/microshift-auto-recovery.service"
|
||||
data = """
|
||||
[Unit]
|
||||
Description=MicroShift auto-recovery
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/microshift-auto-recovery
|
||||
|
||||
ExecStart=/etc/bin/microshift-auto-recovery
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
"""
|
||||
|
||||
[[customizations.files]]
|
||||
path = "/usr/bin/microshift-auto-recovery"
|
||||
path = "/etc/bin/microshift-auto-recovery"
|
||||
mode = "0755"
|
||||
data = """
|
||||
#!/usr/bin/env bash
|
||||
set -xeuo pipefail
|
||||
|
||||
# If greenboot uses a non-default file for clearing boot_counter, use boot_success instead.
|
||||
if grep -q "/boot/grubenv" /usr/libexec/greenboot/greenboot-grub2-set-success; then
|
||||
if grub2-editenv - list | grep -q ^boot_success=0; then
|
||||
@@ -61,12 +71,11 @@ else
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
/usr/bin/microshift restore --auto-recovery /var/lib/microshift-auto-recovery
|
||||
/usr/bin/systemctl reset-failed microshift
|
||||
/usr/bin/systemctl start microshift
|
||||
|
||||
echo "DONE"
|
||||
"""
|
||||
----
|
||||
|
||||
. For the next steps, see link:https://docs.redhat.com/en/documentation/red_hat_build_of_microshift/{ocp-version}/html/embedding_in_a_rhel_for_edge_image/microshift-embed-in-rpm-ostree#preparing-for-image-building_microshift-embed-in-rpm-ostree[Preparing for image building].
|
||||
@@ -3,19 +3,16 @@
|
||||
// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc
|
||||
|
||||
:_mod-docs-content-type: PROCEDURE
|
||||
[id="microshift-automation-example-rpm-systems_{context}"]
|
||||
= Automating the integration process with systemd for RPM systems
|
||||
[id="microshift-auto-recovery-rpm-systems_{context}"]
|
||||
= Using automatic recovery in RPM systems
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
When the `microshift.service` enters a failed state, `systemd` starts the `microshift-auto-recovery.service` unit. This unit executes the `auto-recovery` restore process and restarts {microshift-short}.
|
||||
====
|
||||
When {microshift-short} enters a failed state, the systemd service starts the `microshift-auto-recovery.service` unit. This unit executes the `auto-recovery` restore process.
|
||||
|
||||
As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for RPM systems that use systemd.
|
||||
As a use case, consider the following example situation in which you want to automate the automatic recovery process for RPM systems that use the systemd service.
|
||||
|
||||
.Procedure
|
||||
|
||||
. Create a directory for the `microshift.service` by running the following command:
|
||||
. Create a directory for the `microshift` systemd service by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
@@ -28,8 +25,11 @@ $ sudo mkdir -p /usr/lib/systemd/system/microshift.service.d
|
||||
$ sudo tee /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf > /dev/null <<'EOF'
|
||||
[Unit]
|
||||
OnFailure=microshift-auto-recovery.service
|
||||
StartLimitIntervalSec=25s # <1>
|
||||
EOF
|
||||
----
|
||||
<1> Increase the `StartLimitInterval` value from the default `10s` to a larger value for slower systems. A value that is too low can result in systemd never marking the `microshift` systemd service as failed, which means that the `OnFailure=` service does not get triggered.
|
||||
|
||||
. Create the `microshift-auto-recovery.service` file by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
@@ -46,6 +46,7 @@ ExecStart=/usr/bin/microshift-auto-recovery
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
----
|
||||
|
||||
. Create the `microshift-auto-recovery` script by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
@@ -76,12 +77,14 @@ fi
|
||||
echo "DONE"
|
||||
EOF
|
||||
----
|
||||
|
||||
. Make the script executable by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ sudo chmod +x /usr/bin/microshift-auto-recovery
|
||||
----
|
||||
|
||||
. Reload the system configuration by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
22
modules/microshift-auto-recovery-manual-backups.adoc
Normal file
22
modules/microshift-auto-recovery-manual-backups.adoc
Normal file
@@ -0,0 +1,22 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc
|
||||
|
||||
:_mod-docs-content-type: CONCEPT
|
||||
[id="microshift-auto-recovery-manual-backups_{context}"]
|
||||
= Modifying backup and restore commands to automate data recovery
|
||||
|
||||
Use automatic recovery options to store all of your backups in a single directory, then automatically select the latest one to restore. Modifying existing `backup` and `restore` commands enables you to set up automatic recovery.
|
||||
|
||||
The `--auto-recovery` option treats the `PATH` argument as a path to a directory that holds all the backups for automated recovery, and not just as a path to a particular backup file. You can use the `--auto-recovery` option with both `backup` and `restore` commands.
|
||||
|
||||
* For example, if you use the automatic recovery option with `restore`, such as in `microshift restore --auto-recovery PATH`, running the modified command automatically selects and restores the most recent backup.
|
||||
|
||||
* If you use the same option in the `microshift backup` command, such as in `microshift backup --auto-recovery PATH`, a new backup is created in the PATH.
|
||||
|
||||
* By default, `microshift restore --auto-recovery PATH` creates a backup of the failed {microshift-short} data in `PATH/failed`. You can add the `--dont-save-failed` option to disable the creation of failed backup data.
|
||||
|
||||
[IMPORTANT]
|
||||
====
|
||||
You can only use the `--dont-save-failed` option with the `restore` command.
|
||||
====
|
||||
@@ -1,32 +0,0 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc
|
||||
|
||||
:_mod-docs-content-type: PROCEDURE
|
||||
[id="microshift-automation-example-bootc-systems_{context}"]
|
||||
= Automating the integration process with systemd for bootc systems
|
||||
|
||||
[IMPORTANT]
|
||||
====
|
||||
You must include the entire `auto-recovery` process for bootc systems that use `systemd` in the container file.
|
||||
====
|
||||
|
||||
As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for bootc systems that use systemd.
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You have created the `10-auto-recovery.conf` and `microshift-auto-recovery.service` files as explained in the "Automating the integration process with systemd for RPM systems" section.
|
||||
* You have created the `microshift-auto-recovery` script as explained in the "Automating the integration process with systemd for RPM systems" section.
|
||||
|
||||
.Procedure
|
||||
|
||||
* Use the following example to update your Containerfile that you use to prepare the bootc image.
|
||||
+
|
||||
[source,text]
|
||||
----
|
||||
RUN mkdir -p /usr/lib/systemd/system/microshift.service.d
|
||||
COPY ./auto-rec/10-auto-recovery.conf /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf
|
||||
COPY ./auto-rec/microshift-auto-recovery.service /usr/lib/systemd/system/
|
||||
COPY ./auto-rec/microshift-auto-recovery /usr/bin/
|
||||
RUN chmod +x /usr/bin/microshift-auto-recovery && systemctl daemon-reload
|
||||
----
|
||||
@@ -3,10 +3,10 @@
|
||||
// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc
|
||||
|
||||
:_mod-docs-content-type: PROCEDURE
|
||||
[id="microshift-creating-backups_{context}"]
|
||||
[id="microshift-creating-backups-auto-recovery_{context}"]
|
||||
= Creating backups using the auto-recovery feature
|
||||
|
||||
Use the following procedure to create backups.
|
||||
Use the following procedure to create backups using automatic recovery options.
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
@@ -15,14 +15,13 @@ Creating backups require stopping {microshift-short}, so you must determine the
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You have stopped {microshift-short}.
|
||||
* You stopped {microshift-short}.
|
||||
|
||||
.Procedure
|
||||
|
||||
* Create and store backups in the directory you choose by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
[subs="+quotes"]
|
||||
[source,terminal,subs="+quotes"]
|
||||
----
|
||||
$ sudo microshift backup --auto-recovery _<path_of_directory>_ <1>
|
||||
----
|
||||
@@ -30,11 +29,10 @@ $ sudo microshift backup --auto-recovery _<path_of_directory>_ <1>
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
The `--auto-recovery` option modifies the interpretation of the `PATH` argument from the final backup path to a directory that holds all the backups for automated recovery.
|
||||
The `--auto-recovery` option modifies the interpretation of the `PATH` argument from the final backup path to a directory that holds all of the backups for automated recovery.
|
||||
====
|
||||
+
|
||||
.Example output
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
??? I1104 09:18:52.100725 8906 system.go:58] "OSTree deployments" deployments=[{"id":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","booted":true,"staged":false,"pinned":false},{"id":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","booted":false,"staged":false,"pinned":false}]
|
||||
@@ -49,11 +47,10 @@ The `--auto-recovery` option modifies the interpretation of the `PATH` argument
|
||||
|
||||
.Verification
|
||||
|
||||
* To verify that the backup has been created, view the directory you chose to store backups by running the following command:
|
||||
* Verify that the backup you created exists in your customized storage directory by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
[subs="+quotes"]
|
||||
[source,terminal,subs="+quotes"]
|
||||
----
|
||||
$ ls -la _<path_of_directory>_ <1>
|
||||
$ sudo ls -la _<path_of_directory>_ <1>
|
||||
----
|
||||
<1> Replace `_<path_of_directory>_` with the path of the directory that stores backups. For example, `/var/lib/microshift-auto-recovery`.
|
||||
@@ -3,12 +3,10 @@
|
||||
// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc
|
||||
|
||||
:_mod-docs-content-type: PROCEDURE
|
||||
[id="microshift-restoring-backups_{context}"]
|
||||
[id="microshift-restoring-backups-auto-recovery_{context}"]
|
||||
= Restoring backups using the auto-recovery feature
|
||||
|
||||
You can restore backups after system events that remove or damage required data.
|
||||
|
||||
Use the following procedure to restore backups.
|
||||
You can restore backups after system events that remove or damage required data. Use the following procedure to restore backups using automatic recovery. Automatic recovery selects the most recent backup and restores it. Previously restored backups that used automatic recovery are moved to your `PATH/restored` directory.
|
||||
|
||||
.Prerequisites
|
||||
|
||||
@@ -16,10 +14,9 @@ Use the following procedure to restore backups.
|
||||
|
||||
.Procedure
|
||||
|
||||
* Restore the backup from the directory in which you have stored the backups by running the following command:
|
||||
. Restore the latest backup from your backups directory by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
[subs="+quotes"]
|
||||
[source,terminal,subs="+quotes"]
|
||||
----
|
||||
$ sudo microshift restore --auto-recovery _<path_of_directory>_ <1>
|
||||
----
|
||||
@@ -27,12 +24,12 @@ $ sudo microshift restore --auto-recovery _<path_of_directory>_ <1>
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
The `--auto-recovery` option copies the {microshift-short} data to `/var/lib/microshift-auto-recovery/failed/` for later investigation, selects the most recent backup, and restores it.
|
||||
The `--dont-save-failed` option disables the backing up of failed {microshift-short} data.
|
||||
* The `--auto-recovery` option copies the {microshift-short} data to `/var/lib/microshift-auto-recovery/failed/` for later investigation, selects the most recent backup, and restores it.
|
||||
|
||||
* The `--dont-save-failed` option disables the backing up of failed {microshift-short} data.
|
||||
====
|
||||
+
|
||||
.Example output
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
??? I1104 09:19:28.617225 8950 state.go:80] "Read state from the disk" state={"LastBackup":"20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"}
|
||||
@@ -57,12 +54,24 @@ The `--dont-save-failed` option disables the backing up of failed {microshift-sh
|
||||
??? I1104 09:19:28.662983 8950 restore.go:141] "Auto-recovery restore completed".
|
||||
----
|
||||
+
|
||||
[NOTE]
|
||||
[IMPORTANT]
|
||||
====
|
||||
* The `restore` command does not start {microshift-short} after restoration. When you execute this command, {microshift-short} service has already failed or you need to stop it.
|
||||
* {microshift-short} does not monitor the disk space of any filesystem. You need to ensure your automation handles old backup removal.
|
||||
* The `restore` command does not restart {microshift-short} after restoration. When you execute this command, {microshift-short} service has already failed or you stopped it.
|
||||
|
||||
* {microshift-short} does not monitor the disk space of any filesystem. You must ensure that your automation handles old backup removal. For example, you can add this process to the auto-recovery service or add another service that runs periodically.
|
||||
====
|
||||
|
||||
. Restart {microshift-short} by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ sudo systemctl restart microshift
|
||||
----
|
||||
|
||||
.Verification
|
||||
|
||||
* Verify that {microshift-short} has started successfully.
|
||||
* Verify that {microshift-short} has started successfully by running the following command:
|
||||
+
|
||||
--
|
||||
include::snippets/microshift-healthy-pods-snip.adoc[leveloffset=+1]
|
||||
--
|
||||
Reference in New Issue
Block a user