From 4516fa99643804f6d1b2dad464c9e45f92cdadef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CShauna=20Diaz=E2=80=9D?= Date: Thu, 6 Feb 2025 08:39:31 -0500 Subject: [PATCH] OSDOCS-11312: adds enhancements and nit fixes to auto recovery MicroShift --- ...microshift-auto-recover-manual-backup.adoc | 23 ++--- ...t-auto-recovery-example-bootc-systems.adoc | 93 +++++++++++++++++++ ...auto-recovery-example-ostree-systems.adoc} | 35 ++++--- ...ft-auto-recovery-example-rpm-systems.adoc} | 19 ++-- ...croshift-auto-recovery-manual-backups.adoc | 22 +++++ ...hift-automation-example-bootc-systems.adoc | 32 ------- ...shift-creating-backups-auto-recovery.adoc} | 19 ++-- ...hift-restoring-backups-auto-recovery.adoc} | 37 +++++--- 8 files changed, 186 insertions(+), 94 deletions(-) create mode 100644 modules/microshift-auto-recovery-example-bootc-systems.adoc rename modules/{microshift-automation-example-ostree-systems.adoc => microshift-auto-recovery-example-ostree-systems.adoc} (65%) rename modules/{microshift-automation-example-rpm-systems.adoc => microshift-auto-recovery-example-rpm-systems.adoc} (75%) create mode 100644 modules/microshift-auto-recovery-manual-backups.adoc delete mode 100644 modules/microshift-automation-example-bootc-systems.adoc rename modules/{microshift-creating-backups.adoc => microshift-creating-backups-auto-recovery.adoc} (86%) rename modules/{microshift-restoring-backups.adoc => microshift-restoring-backups-auto-recovery.adoc} (82%) diff --git a/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc b/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc index e2aab42ce7..87048afbbe 100644 --- a/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc +++ b/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc @@ -6,25 +6,16 @@ include::_attributes/attributes-microshift.adoc[] toc::[] -You can automatically restore data from manual backups when {microshift-short} fails to start by using the `auto-recovery` feature. +You can automatically restore data from manual backups when {microshift-short} fails to start by configuring automatic recovery. -You can use the following options with the existing `backup` and `restore` commands in this feature: +include::modules/microshift-auto-recovery-manual-backups.adoc[leveloffset=+1] -* `--auto-recovery`: Selects the most recent version of the backup, and then restores it. This option treats the `PATH` argument as a path to a directory that holds all the backups for automated recovery, and not just as a path to a particular backup file. -* `--dont-save-failed`: Disables the backup of failed {microshift-short} data. +include::modules/microshift-creating-backups-auto-recovery.adoc[leveloffset=+1] -[NOTE] -==== -* You can use the `--auto-recovery` option with both the `backup` and `restore` commands. -* You can use the `--dont-save-failed` option only with the `restore` command. -==== +include::modules/microshift-restoring-backups-auto-recovery.adoc[leveloffset=+1] -include::modules/microshift-creating-backups.adoc[leveloffset=+1] +include::modules/microshift-auto-recovery-example-rpm-systems.adoc[leveloffset=+2] -include::modules/microshift-restoring-backups.adoc[leveloffset=+1] +include::modules/microshift-auto-recovery-example-ostree-systems.adoc[leveloffset=+2] -include::modules/microshift-automation-example-rpm-systems.adoc[leveloffset=+1] - -include::modules/microshift-automation-example-ostree-systems.adoc[leveloffset=+1] - -include::modules/microshift-automation-example-bootc-systems.adoc[leveloffset=+1] +include::modules/microshift-auto-recovery-example-bootc-systems.adoc[leveloffset=+2] diff --git a/modules/microshift-auto-recovery-example-bootc-systems.adoc b/modules/microshift-auto-recovery-example-bootc-systems.adoc new file mode 100644 index 0000000000..e89da20976 --- /dev/null +++ b/modules/microshift-auto-recovery-example-bootc-systems.adoc @@ -0,0 +1,93 @@ +// Module included in the following assemblies: +// +// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc + +:_mod-docs-content-type: PROCEDURE +[id="microshift-auto-recovery-example-bootc-systems_{context}"] += Using automatic recovery in image mode for {op-system-base} systems + +:FeatureName: Image mode for {op-system-base} + +include::snippets/technology-preview.adoc[] + +As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for image mode for {op-system-base-full} systems that use the systemd service. + +[IMPORTANT] +==== +You must include the entire `auto-recovery` process for {op-system-image} systems that use `systemd` in the container file. +==== + +.Prerequisites + +* You created a Containerfile as instructed in link:https://docs.redhat.com/en/documentation/red_hat_build_of_microshift/4.18/html/installing_with_rhel_image_mode/installing-with-rhel-image-mode#microshift-rhel-image-mode-build-image_microshift-install-rhel-image-mode[Building the bootc image]. + +* You created the `10-auto-recovery.conf` and `microshift-auto-recovery.service` files as explained in the "Using auto-recovery in RPM systems" section. ++ +[IMPORTANT] +==== +The location of the the `10-auto-recovery.conf` and `microshift-auto-recovery.service` must be relative to the Containerfile. + +For example, if the path to the Containerfile is `/home/microshift/my-build/Containerfile`, the systemd files need to be adjacent for proper embedding. The following paths are correct for this example: + +* `/home/microshift/my-build/auto-rec/10-auto-recovery.conf` +* `/home/microshift/my-build/auto-rec/microshift-auto-recovery.service` +* `/home/microshift/my-build/auto-rec/microshift-auto-recovery` +==== + +* You created the `microshift-auto-recovery` script as explained in the "Using auto-recovery in RPM systems" section. + +.Procedure + +. Use the following example snippet to update the container file that you use to prepare the {op-system-image} image. ++ +[source,text] +---- +RUN mkdir -p /usr/lib/systemd/system/microshift.service.d +COPY ./auto-rec/10-auto-recovery.conf /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf +COPY ./auto-rec/microshift-auto-recovery.service /usr/lib/systemd/system/ +COPY ./auto-rec/microshift-auto-recovery /usr/bin/ +RUN chmod +x /usr/bin/microshift-auto-recovery +---- ++ +[IMPORTANT] +==== +Podman uses the host subscription information and repositories inside the container when building the container image. If the `rhocp` and `fast-datapath` repositories are not available on the host, the build fails. +==== + +. Rebuild your local bootc image by running the following image build command: ++ +[source,terminal] +---- +PULL_SECRET=~/.pull-secret.json +USER_PASSWD= +IMAGE_NAME=microshift-4.18-bootc + +sudo podman build --authfile "${PULL_SECRET}" -t "${IMAGE_NAME}" \ + --build-arg USER_PASSWD="${USER_PASSWD}" \ + -f Containerfile +---- ++ +[NOTE] +==== +Secrets are used during the image build in the following ways: + +* The podman `--authfile` argument is required to pull the base `rhel-bootc:9.4` image from the `registry.redhat.io` registry. + +* The build `USER_PASSWD` argument is used to set a password for the `redhat user`. +==== + +.Verification + +* Verify that the local bootc image was created by running the following command: ++ +[source,terminal] +---- +$ sudo podman images "${IMAGE_NAME}" +---- ++ +.Example output +[source,text] +---- +REPOSITORY TAG IMAGE ID CREATED SIZE +localhost/microshift-4.18-bootc latest 193425283c00 2 minutes ago 2.31 GB +---- \ No newline at end of file diff --git a/modules/microshift-automation-example-ostree-systems.adoc b/modules/microshift-auto-recovery-example-ostree-systems.adoc similarity index 65% rename from modules/microshift-automation-example-ostree-systems.adoc rename to modules/microshift-auto-recovery-example-ostree-systems.adoc index ee1e6bcae3..ef6e8416bb 100644 --- a/modules/microshift-automation-example-ostree-systems.adoc +++ b/modules/microshift-auto-recovery-example-ostree-systems.adoc @@ -3,50 +3,60 @@ // * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc :_mod-docs-content-type: PROCEDURE -[id="microshift-automation-example-ostree-systems_{context}"] -= Automating the integration process with systemd for OSTree systems +[id="microshift-auto-recovery-ostree-systems_{context}"] += Using automatic recovery with {op-system-ostree} + +As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for {op-system-ostree-first} systems that use systemd in the blueprint file. [IMPORTANT] ==== -You must include the entire `auto-recovery` process for OSTree systems that use `systemd` in the blueprint file. +You must include the entire `auto-recovery` process for {op-system-ostree} systems that use `systemd` in the blueprint file. ==== -As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for OSTree systems that use systemd. +.Prerequisites + +* You installed Podman. +* You installed the command-line `composer-cli` tool. .Procedure +. Optional: Because the `composer-cli` can only create files in the `/etc` directory, package your files into an RPM that you include the blueprint. + . Use the following example to create your blueprint file: + [source,terminal] ---- +[[customizations.directories]] +path = "/etc/systemd/system/microshift.service.d" + +[[customizations.directories]] +path = "/etc/bin" + [[customizations.files]] -path = "/usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf" +path = "/etc/systemd/system/microshift.service.d/10-auto-recovery.conf" data = """ [Unit] OnFailure=microshift-auto-recovery.service """ [[customizations.files]] -path = "/usr/lib/systemd/system/microshift-auto-recovery.service" +path = "/etc/systemd/system/microshift-auto-recovery.service" data = """ [Unit] Description=MicroShift auto-recovery - [Service] Type=oneshot -ExecStart=/usr/bin/microshift-auto-recovery - +ExecStart=/etc/bin/microshift-auto-recovery [Install] WantedBy=multi-user.target """ [[customizations.files]] -path = "/usr/bin/microshift-auto-recovery" +path = "/etc/bin/microshift-auto-recovery" mode = "0755" data = """ #!/usr/bin/env bash set -xeuo pipefail - # If greenboot uses a non-default file for clearing boot_counter, use boot_success instead. if grep -q "/boot/grubenv" /usr/libexec/greenboot/greenboot-grub2-set-success; then if grub2-editenv - list | grep -q ^boot_success=0; then @@ -61,12 +71,11 @@ else exit 0 fi fi - /usr/bin/microshift restore --auto-recovery /var/lib/microshift-auto-recovery /usr/bin/systemctl reset-failed microshift /usr/bin/systemctl start microshift - echo "DONE" """ ---- + . For the next steps, see link:https://docs.redhat.com/en/documentation/red_hat_build_of_microshift/{ocp-version}/html/embedding_in_a_rhel_for_edge_image/microshift-embed-in-rpm-ostree#preparing-for-image-building_microshift-embed-in-rpm-ostree[Preparing for image building]. \ No newline at end of file diff --git a/modules/microshift-automation-example-rpm-systems.adoc b/modules/microshift-auto-recovery-example-rpm-systems.adoc similarity index 75% rename from modules/microshift-automation-example-rpm-systems.adoc rename to modules/microshift-auto-recovery-example-rpm-systems.adoc index e670359260..81f8a77ab6 100644 --- a/modules/microshift-automation-example-rpm-systems.adoc +++ b/modules/microshift-auto-recovery-example-rpm-systems.adoc @@ -3,19 +3,16 @@ // * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc :_mod-docs-content-type: PROCEDURE -[id="microshift-automation-example-rpm-systems_{context}"] -= Automating the integration process with systemd for RPM systems +[id="microshift-auto-recovery-rpm-systems_{context}"] += Using automatic recovery in RPM systems -[NOTE] -==== -When the `microshift.service` enters a failed state, `systemd` starts the `microshift-auto-recovery.service` unit. This unit executes the `auto-recovery` restore process and restarts {microshift-short}. -==== +When {microshift-short} enters a failed state, the systemd service starts the `microshift-auto-recovery.service` unit. This unit executes the `auto-recovery` restore process. -As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for RPM systems that use systemd. +As a use case, consider the following example situation in which you want to automate the automatic recovery process for RPM systems that use the systemd service. .Procedure -. Create a directory for the `microshift.service` by running the following command: +. Create a directory for the `microshift` systemd service by running the following command: + [source,terminal] ---- @@ -28,8 +25,11 @@ $ sudo mkdir -p /usr/lib/systemd/system/microshift.service.d $ sudo tee /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf > /dev/null <<'EOF' [Unit] OnFailure=microshift-auto-recovery.service +StartLimitIntervalSec=25s # <1> EOF ---- +<1> Increase the `StartLimitInterval` value from the default `10s` to a larger value for slower systems. A value that is too low can result in systemd never marking the `microshift` systemd service as failed, which means that the `OnFailure=` service does not get triggered. + . Create the `microshift-auto-recovery.service` file by running the following command: + [source,terminal] @@ -46,6 +46,7 @@ ExecStart=/usr/bin/microshift-auto-recovery WantedBy=multi-user.target EOF ---- + . Create the `microshift-auto-recovery` script by running the following command: + [source,terminal] @@ -76,12 +77,14 @@ fi echo "DONE" EOF ---- + . Make the script executable by running the following command: + [source,terminal] ---- $ sudo chmod +x /usr/bin/microshift-auto-recovery ---- + . Reload the system configuration by running the following command: + [source,terminal] diff --git a/modules/microshift-auto-recovery-manual-backups.adoc b/modules/microshift-auto-recovery-manual-backups.adoc new file mode 100644 index 0000000000..aee6a8db65 --- /dev/null +++ b/modules/microshift-auto-recovery-manual-backups.adoc @@ -0,0 +1,22 @@ +// Module included in the following assemblies: +// +// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc + +:_mod-docs-content-type: CONCEPT +[id="microshift-auto-recovery-manual-backups_{context}"] += Modifying backup and restore commands to automate data recovery + +Use automatic recovery options to store all of your backups in a single directory, then automatically select the latest one to restore. Modifying existing `backup` and `restore` commands enables you to set up automatic recovery. + +The `--auto-recovery` option treats the `PATH` argument as a path to a directory that holds all the backups for automated recovery, and not just as a path to a particular backup file. You can use the `--auto-recovery` option with both `backup` and `restore` commands. + +* For example, if you use the automatic recovery option with `restore`, such as in `microshift restore --auto-recovery PATH`, running the modified command automatically selects and restores the most recent backup. + +* If you use the same option in the `microshift backup` command, such as in `microshift backup --auto-recovery PATH`, a new backup is created in the PATH. + +* By default, `microshift restore --auto-recovery PATH` creates a backup of the failed {microshift-short} data in `PATH/failed`. You can add the `--dont-save-failed` option to disable the creation of failed backup data. + +[IMPORTANT] +==== +You can only use the `--dont-save-failed` option with the `restore` command. +==== \ No newline at end of file diff --git a/modules/microshift-automation-example-bootc-systems.adoc b/modules/microshift-automation-example-bootc-systems.adoc deleted file mode 100644 index 302cf4e099..0000000000 --- a/modules/microshift-automation-example-bootc-systems.adoc +++ /dev/null @@ -1,32 +0,0 @@ -// Module included in the following assemblies: -// -// * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc - -:_mod-docs-content-type: PROCEDURE -[id="microshift-automation-example-bootc-systems_{context}"] -= Automating the integration process with systemd for bootc systems - -[IMPORTANT] -==== -You must include the entire `auto-recovery` process for bootc systems that use `systemd` in the container file. -==== - -As a use case, consider the following example situation in which you want to automate the `auto-recovery` process for bootc systems that use systemd. - -.Prerequisites - -* You have created the `10-auto-recovery.conf` and `microshift-auto-recovery.service` files as explained in the "Automating the integration process with systemd for RPM systems" section. -* You have created the `microshift-auto-recovery` script as explained in the "Automating the integration process with systemd for RPM systems" section. - -.Procedure - -* Use the following example to update your Containerfile that you use to prepare the bootc image. -+ -[source,text] ----- -RUN mkdir -p /usr/lib/systemd/system/microshift.service.d -COPY ./auto-rec/10-auto-recovery.conf /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf -COPY ./auto-rec/microshift-auto-recovery.service /usr/lib/systemd/system/ -COPY ./auto-rec/microshift-auto-recovery /usr/bin/ -RUN chmod +x /usr/bin/microshift-auto-recovery && systemctl daemon-reload ----- diff --git a/modules/microshift-creating-backups.adoc b/modules/microshift-creating-backups-auto-recovery.adoc similarity index 86% rename from modules/microshift-creating-backups.adoc rename to modules/microshift-creating-backups-auto-recovery.adoc index b683a9c1e1..8e70d579d6 100644 --- a/modules/microshift-creating-backups.adoc +++ b/modules/microshift-creating-backups-auto-recovery.adoc @@ -3,10 +3,10 @@ // * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc :_mod-docs-content-type: PROCEDURE -[id="microshift-creating-backups_{context}"] +[id="microshift-creating-backups-auto-recovery_{context}"] = Creating backups using the auto-recovery feature -Use the following procedure to create backups. +Use the following procedure to create backups using automatic recovery options. [NOTE] ==== @@ -15,14 +15,13 @@ Creating backups require stopping {microshift-short}, so you must determine the .Prerequisites -* You have stopped {microshift-short}. +* You stopped {microshift-short}. .Procedure * Create and store backups in the directory you choose by running the following command: + -[source,terminal] -[subs="+quotes"] +[source,terminal,subs="+quotes"] ---- $ sudo microshift backup --auto-recovery __ <1> ---- @@ -30,11 +29,10 @@ $ sudo microshift backup --auto-recovery __ <1> + [NOTE] ==== -The `--auto-recovery` option modifies the interpretation of the `PATH` argument from the final backup path to a directory that holds all the backups for automated recovery. +The `--auto-recovery` option modifies the interpretation of the `PATH` argument from the final backup path to a directory that holds all of the backups for automated recovery. ==== + .Example output -+ [source,terminal] ---- ??? I1104 09:18:52.100725 8906 system.go:58] "OSTree deployments" deployments=[{"id":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","booted":true,"staged":false,"pinned":false},{"id":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","booted":false,"staged":false,"pinned":false}] @@ -49,11 +47,10 @@ The `--auto-recovery` option modifies the interpretation of the `PATH` argument .Verification -* To verify that the backup has been created, view the directory you chose to store backups by running the following command: +* Verify that the backup you created exists in your customized storage directory by running the following command: + -[source,terminal] -[subs="+quotes"] +[source,terminal,subs="+quotes"] ---- -$ ls -la __ <1> +$ sudo ls -la __ <1> ---- <1> Replace `__` with the path of the directory that stores backups. For example, `/var/lib/microshift-auto-recovery`. \ No newline at end of file diff --git a/modules/microshift-restoring-backups.adoc b/modules/microshift-restoring-backups-auto-recovery.adoc similarity index 82% rename from modules/microshift-restoring-backups.adoc rename to modules/microshift-restoring-backups-auto-recovery.adoc index ab20009bae..4d4a8247fb 100644 --- a/modules/microshift-restoring-backups.adoc +++ b/modules/microshift-restoring-backups-auto-recovery.adoc @@ -3,12 +3,10 @@ // * microshift/microshift_backup_and_restore/microshift-auto-recover-manual-backup.adoc :_mod-docs-content-type: PROCEDURE -[id="microshift-restoring-backups_{context}"] +[id="microshift-restoring-backups-auto-recovery_{context}"] = Restoring backups using the auto-recovery feature -You can restore backups after system events that remove or damage required data. - -Use the following procedure to restore backups. +You can restore backups after system events that remove or damage required data. Use the following procedure to restore backups using automatic recovery. Automatic recovery selects the most recent backup and restores it. Previously restored backups that used automatic recovery are moved to your `PATH/restored` directory. .Prerequisites @@ -16,10 +14,9 @@ Use the following procedure to restore backups. .Procedure -* Restore the backup from the directory in which you have stored the backups by running the following command: +. Restore the latest backup from your backups directory by running the following command: + -[source,terminal] -[subs="+quotes"] +[source,terminal,subs="+quotes"] ---- $ sudo microshift restore --auto-recovery __ <1> ---- @@ -27,12 +24,12 @@ $ sudo microshift restore --auto-recovery __ <1> + [NOTE] ==== -The `--auto-recovery` option copies the {microshift-short} data to `/var/lib/microshift-auto-recovery/failed/` for later investigation, selects the most recent backup, and restores it. -The `--dont-save-failed` option disables the backing up of failed {microshift-short} data. +* The `--auto-recovery` option copies the {microshift-short} data to `/var/lib/microshift-auto-recovery/failed/` for later investigation, selects the most recent backup, and restores it. + +* The `--dont-save-failed` option disables the backing up of failed {microshift-short} data. ==== + .Example output -+ [source,terminal] ---- ??? I1104 09:19:28.617225 8950 state.go:80] "Read state from the disk" state={"LastBackup":"20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"} @@ -57,12 +54,24 @@ The `--dont-save-failed` option disables the backing up of failed {microshift-sh ??? I1104 09:19:28.662983 8950 restore.go:141] "Auto-recovery restore completed". ---- + -[NOTE] +[IMPORTANT] ==== -* The `restore` command does not start {microshift-short} after restoration. When you execute this command, {microshift-short} service has already failed or you need to stop it. -* {microshift-short} does not monitor the disk space of any filesystem. You need to ensure your automation handles old backup removal. +* The `restore` command does not restart {microshift-short} after restoration. When you execute this command, {microshift-short} service has already failed or you stopped it. + +* {microshift-short} does not monitor the disk space of any filesystem. You must ensure that your automation handles old backup removal. For example, you can add this process to the auto-recovery service or add another service that runs periodically. ==== +. Restart {microshift-short} by running the following command: ++ +[source,terminal] +---- +$ sudo systemctl restart microshift +---- + .Verification -* Verify that {microshift-short} has started successfully. \ No newline at end of file +* Verify that {microshift-short} has started successfully by running the following command: ++ +-- +include::snippets/microshift-healthy-pods-snip.adoc[leveloffset=+1] +-- \ No newline at end of file