From ac054166cc6e27a5e61a7a9764f3926a5b6a515b Mon Sep 17 00:00:00 2001 From: Pan Ousley Date: Fri, 24 Sep 2021 18:01:25 -0400 Subject: [PATCH] CNV-9800: New upgrade behavior for 4.9+ --- modules/virt-about-upgrading-virt.adoc | 20 +++++--- ...t-configuring-workload-update-methods.adoc | 50 +++++++++++++++++++ modules/virt-document-attributes.adoc | 2 +- modules/virt-monitoring-upgrade-status.adoc | 10 ++-- modules/virt-opting-out-workload-updates.adoc | 42 ++++++++++++++++ .../virt-understanding-live-migration.adoc | 21 ++++---- modules/virt-upgrade-pathways.adoc | 25 ---------- modules/virt-viewing-outdated-workloads.adoc | 22 ++++++++ virt/upgrading-virt.adoc | 31 ++++++++++-- 9 files changed, 171 insertions(+), 52 deletions(-) create mode 100644 modules/virt-configuring-workload-update-methods.adoc create mode 100644 modules/virt-opting-out-workload-updates.adoc delete mode 100644 modules/virt-upgrade-pathways.adoc create mode 100644 modules/virt-viewing-outdated-workloads.adoc diff --git a/modules/virt-about-upgrading-virt.adoc b/modules/virt-about-upgrading-virt.adoc index a0161322f9..e939ff1077 100644 --- a/modules/virt-about-upgrading-virt.adoc +++ b/modules/virt-about-upgrading-virt.adoc @@ -24,10 +24,7 @@ connection. Most automatic updates complete within fifteen minutes. [id="how-upgrades-affect-cluster_{context}"] == How {VirtProductName} upgrades affect your cluster -* Upgrading does not interrupt virtual machine workloads. -** Virtual machine pods are not restarted or migrated during an upgrade. If you -need to update the `virt-launcher` pod, you must restart or live migrate the -virtual machine. +* When you upgrade {VirtProductName}, virtual machine workloads including `libvirt`, `virt-launcher`, and `qemu` update automatically if they support live migration. + [NOTE] ==== @@ -36,10 +33,21 @@ instance. The `virt-launcher` pod runs an instance of `libvirt`, which is used to manage the virtual machine process. ==== +** There are two supported workload update methods: `LiveMigrate` and `Evict`. By default, only `LiveMigrate` is enabled, which means that: + +*** Virtual machine instances (VMIs) that support live migration are migrated during the upgrade process. + +*** VMIs are not updated at all if the following conditions are both true: + +**** The VMI does not support live migration. + +**** The VMI is configured to use the `LiveMigrate` eviction strategy. + +** You can configure how workloads are updated by editing the `spec.workloadUpdateStrategy` stanza of the `HyperConverged` custom resource (CR). + * Upgrading does not interrupt network connections. -* Data volumes and their associated persistent volume claims are preserved during -upgrade. +* Data volumes and their associated persistent volume claims are preserved during upgrade. + [IMPORTANT] ==== diff --git a/modules/virt-configuring-workload-update-methods.adoc b/modules/virt-configuring-workload-update-methods.adoc new file mode 100644 index 0000000000..6cda1274a5 --- /dev/null +++ b/modules/virt-configuring-workload-update-methods.adoc @@ -0,0 +1,50 @@ +// Module included in the following assemblies: +// +// * virt/upgrading-virt.adoc + +[id="virt-configuring-workload-update-methods_{context}"] += Configuring workload update methods + +When you upgrade {VirtProductName}, virtual machine instance (VMI) workloads, such as `virt-launcher` pods, update automatically if they support live migration. You can configure workload update strategies or opt out of workload updates by editing the `HyperConverged` custom resource (CR). + +.Prerequisites + +* To use live migration as an update method, you must first enable live migration in the cluster. ++ +[NOTE] +==== +If a `VirtualMachineInstance` CR contains `evictionStrategy: LiveMigrate` and the VMI does not support live migration, the VMI will not update. +==== + +.Procedure + +. To open the `HyperConverged` CR in your default editor, run the following command: ++ +[source,terminal] +---- +$ oc edit hco -n openshift-cnv kubevirt-hyperconverged +---- + +. Edit the `workloadUpdateStrategy` stanza of the `HyperConverged` CR. For example: ++ +[source,yaml] +---- +apiVersion: hco.kubevirt.io/v1beta1 +kind: HyperConverged +metadata: + name: kubevirt-hyperconverged +spec: + workloadUpdateStrategy: + workloadUpdateMethods: <1> + - LiveMigrate <2> + - Evict <3> + batchEvictionSize: 10 <4> + batchEvictionInterval: "1m0s" <5> +---- +<1> The methods that can be used to perform automated workload updates. The supported values are `LiveMigrate` and `Evict`. By default, only `LiveMigrate` is enabled. If you enable both options as shown in this example, updates use `LiveMigrate` for VMIs that support live migration and `Evict` for any VMIs that do not support live migration. To disable automatic workload updates, set `workloadUpdateMethods: []` to leave the array empty. +<2> The least disruptive update method. VMIs that support live migration are updated by migrating the virtual machine (VM) guest into a new pod with the updated components enabled. If `LiveMigrate` is the only workload update method listed, VMIs that do not support live migration are not disrupted or updated. `LiveMigrate` is enabled by default. +<3> A disruptive method that shuts down VMI pods during upgrade. `Evict` is the only update method available if live migration is not enabled in the cluster. If a VMI is controlled by a `VirtualMachine` object that has `runStrategy: always` configured, a new VMI is created in a new pod with updated components. `Evict` is not enabled by default. +<4> The number of VMIs that can be forced to be updated at a time. The default value is `10`. +<5> The interval to wait before evicting the next batch of workloads. The default value is `"1m0s"`. + +. To apply your changes, save and exit the editor. \ No newline at end of file diff --git a/modules/virt-document-attributes.adoc b/modules/virt-document-attributes.adoc index 7ab88ab2bc..6cd2187242 100644 --- a/modules/virt-document-attributes.adoc +++ b/modules/virt-document-attributes.adoc @@ -12,7 +12,7 @@ :VirtProductName: OpenShift Virtualization :ProductRelease: :ProductVersion: -:VirtVersion: 4.8 +:VirtVersion: 4.9 :KubeVirtVersion: v0.41.0 :HCOVersion: 4.8.2 // :LastHCOVersion: diff --git a/modules/virt-monitoring-upgrade-status.adoc b/modules/virt-monitoring-upgrade-status.adoc index e11a4cd083..f7618c4191 100644 --- a/modules/virt-monitoring-upgrade-status.adoc +++ b/modules/virt-monitoring-upgrade-status.adoc @@ -3,11 +3,9 @@ // * virt/upgrading-virt.adoc [id="virt-monitoring-upgrade-status_{context}"] -= Monitoring upgrade status += Monitoring {VirtProductName} upgrade status -The best way to monitor {VirtProductName} upgrade status is to watch the -cluster service version (CSV) `PHASE`. You can also monitor the CSV conditions -in the web console or by running the command provided here. +To monitor the status of a {VirtProductName} Operator upgrade, watch the cluster service version (CSV) `PHASE`. You can also monitor the CSV conditions in the web console or by running the command provided here. [NOTE] ==== @@ -35,8 +33,8 @@ $ oc get csv -n openshift-cnv [source,terminal,subs="attributes+"] ---- VERSION REPLACES PHASE -4.8.0 kubevirt-hyperconverged-operator.v2.6.5 Installing -4.8.1 kubevirt-hyperconverged-operator.v4.8.0 Replacing +4.9.0 kubevirt-hyperconverged-operator.v4.8.2 Installing +4.9.0 kubevirt-hyperconverged-operator.v4.9.0 Replacing ---- . Optional: Monitor the aggregated status of all {VirtProductName} component diff --git a/modules/virt-opting-out-workload-updates.adoc b/modules/virt-opting-out-workload-updates.adoc new file mode 100644 index 0000000000..bfb1bcedd3 --- /dev/null +++ b/modules/virt-opting-out-workload-updates.adoc @@ -0,0 +1,42 @@ +// Module included in the following assemblies: +// +// * virt/upgrading-virt.adoc + +[id="virt-opting-out-workload-updates_{context}"] += Opting out of automatic workload updates before you upgrade to {VirtProductName} {VirtVersion} + +Automatic workload updates are enabled by default as of {VirtProductName} {VirtVersion}. If you plan to disable this feature, you can do so _before_ upgrading by following this workaround. + +[NOTE] +==== +If you already upgraded to {VirtProductName} {VirtVersion}, you can disable automatic workload updates by setting the value of `spec.workloadUpdateStrategy.workloadUpdateMethods` to `[]` in the `HyperConverged` CR. +==== + +.Prerequisites + +* You have not yet upgraded to {VirtProductName} {VirtVersion}. + +.Procedure + +. Annotate the `HyperConverged` CR by running the following command: ++ +[source,terminal] +---- +$ oc -n openshift-cnv annotate hco kubevirt-hyperconverged --overwrite kubevirt.kubevirt.io/jsonpatch='[{"op": "replace", "path": "/spec/workloadUpdateStrategy/workloadUpdateMethods", "value": []}]' +---- + +. Upgrade {VirtProductName} to version {VirtVersion}. If you use the *Automatic* approval strategy, upgrading {product-title} triggers the {VirtProductName} update. Ensure that the {VirtProductName} upgrade completes before moving to the next step. + +. Patch the `HyperConverged` CR by running the following command: ++ +[source,terminal] +---- +$ oc -n openshift-cnv patch hco kubevirt-hyperconverged --type='json' -p='[{"op": "replace", "path": "/spec/workloadUpdateStrategy/workloadUpdateMethods", "value":[]}]' +---- + +. Delete the annotation by running the following command: ++ +[source,terminal] +---- +$ oc -n kubevirt-hyperconverged annotate hco kubevirt-hyperconverged kubevirt.kubevirt.io/jsonpatch- +---- \ No newline at end of file diff --git a/modules/virt-understanding-live-migration.adoc b/modules/virt-understanding-live-migration.adoc index ed954e2f31..316ad82992 100644 --- a/modules/virt-understanding-live-migration.adoc +++ b/modules/virt-understanding-live-migration.adoc @@ -1,20 +1,19 @@ // Module included in the following assemblies: // // * virt/live_migration/virt-live-migration.adoc +// * virt/upgrading-virt.adoc [id="virt-understanding-live-migration_{context}"] = Understanding live migration -Live migration is the process of moving a running virtual machine instance (VMI) to -another node in the cluster without interrupting the virtual workload or access. -If a VMI uses the `LiveMigrate` eviction strategy, it automatically migrates -when the node that the VMI runs on is placed into maintenance mode. You can also -manually start live migration by selecting a VMI to migrate. +Live migration is the process of moving a running virtual machine instance (VMI) to another node in the cluster without interrupting the virtual workload or access. If a VMI uses the `LiveMigrate` eviction strategy, it automatically migrates when the node that the VMI runs on is placed into maintenance mode. You can also manually start live migration by selecting a VMI to migrate. -Virtual machines must have a persistent volume claim (PVC) with a shared -ReadWriteMany (RWX) access mode to be live migrated. +You can use live migration if the following conditions are met: -[NOTE] -==== -Live migration is supported for virtual machines that are attached to an SR-IOV network interface only if the `sriovLiveMigration` feature gate is enabled in the HyperConverged Cluster custom resource (CR). When the `spec.featureGates.sriovLiveMigration` field is set to `true`, the `virt-launcher` pod runs with the `SYS_RESOURCE` capability. This might degrade its security. -==== +* Virtual machines must have a persistent volume claim (PVC) with a shared `ReadWriteMany` (RWX) access mode to be live migrated. + +* The pod network binding must not be of the bridge interface type `()`. + +* Ports `49152` and `49153` must be available in the `virt-launcher` pod. If these ports are specified in the `masquerade` interface, live migration does not function. + +* Live migration is supported for virtual machines that are attached to an SR-IOV network interface only if the `sriovLiveMigration` feature gate is enabled in the `HyperConverged` custom resource (CR). When the `spec.featureGates.sriovLiveMigration` field is set to `true`, the `virt-launcher` pod runs with the `SYS_RESOURCE` capability. This might degrade its security. diff --git a/modules/virt-upgrade-pathways.adoc b/modules/virt-upgrade-pathways.adoc deleted file mode 100644 index f787953177..0000000000 --- a/modules/virt-upgrade-pathways.adoc +++ /dev/null @@ -1,25 +0,0 @@ -// Module included in the following assemblies: -// -// * virt/upgrading-virt.adoc - -[id="virt-upgrade-pathways_{context}"] -= Upgrade pathways for minor releases - -The upgrade pathway is different depending on the 2.4.z version of {VirtProductName} that you have installed. - -[IMPORTANT] -==== -You must upgrade your {product-title} to 4.8 before upgrading the minor release of {VirtProductName}. -==== - -[id="virt-upgrade-pathways-2.4.3_{context}"] -== Upgrading from 2.4.3 to 2.5.2 - -You must first upgrade to 2.5.0 before you can upgrade the z-stream. You can then upgrade from 2.5.0 to 2.5.1 and then to 2.5.2. - -If your *Approval Strategy* is *Automatic*, which is the default, {VirtProductName} upgrades the z-stream automatically once you have upgraded to 2.5.0. - -[id="virt-upgrade-pathways-2.4.4_{context}"] -== Upgrading from 2.4.4 to 2.5.2 - -You can upgrade from 2.4.4 directly to 2.5.2. diff --git a/modules/virt-viewing-outdated-workloads.adoc b/modules/virt-viewing-outdated-workloads.adoc new file mode 100644 index 0000000000..4df911646b --- /dev/null +++ b/modules/virt-viewing-outdated-workloads.adoc @@ -0,0 +1,22 @@ +// Module included in the following assemblies: +// +// * virt/upgrading-virt.adoc + +[id="virt-viewing-outdated-workloads_{context}"] += Viewing outdated {VirtProductName} workloads + +You can view a list of outdated workloads by using the CLI. + +[NOTE] +==== +If there are outdated virtualization pods in your cluster, the `OutdatedVirtualMachineInstanceWorkloads` alert fires. +==== + +.Procedure + +* To view a list of outdated virtual machine instances (VMIs), run the following command: ++ +[source,terminal] +---- +$ kubectl get vmi -l kubevirt.io/outdatedLauncherImage --all-namespaces +---- \ No newline at end of file diff --git a/virt/upgrading-virt.adoc b/virt/upgrading-virt.adoc index b829da2e70..28c7537b53 100644 --- a/virt/upgrading-virt.adoc +++ b/virt/upgrading-virt.adoc @@ -10,11 +10,34 @@ Learn how Operator Lifecycle Manager (OLM) delivers z-stream and minor version u include::modules/virt-about-upgrading-virt.adoc[leveloffset=+1] -//include::modules/virt-upgrade-pathways.adoc[leveloffset=+1] +[id="configuring-workload-updates_upgrading-virt"] +== Configuring workload updates -include::modules/olm-approving-pending-upgrade.adoc[leveloffset=+1] +[NOTE] +==== +Automatic workload updates are enabled by default in {VirtProductName} {VirtVersion}. You can opt out by following a xref:../virt/upgrading-virt.adoc#virt-opting-out-workload-updates_upgrading-virt[workaround to disable this feature] before you upgrade to {VirtVersion}. +==== -include::modules/virt-monitoring-upgrade-status.adoc[leveloffset=+1] +include::modules/virt-configuring-workload-update-methods.adoc[leveloffset=+2] + +include::modules/virt-opting-out-workload-updates.adoc[leveloffset=+2] + +[id="approving-operator-upgrades_upgrading-virt"] +== Approving pending Operator upgrades + +include::modules/olm-approving-pending-upgrade.adoc[leveloffset=+2] + +[id="monitoring-upgrade-status_upgrading-virt"] +== Monitoring upgrade status + +include::modules/virt-monitoring-upgrade-status.adoc[leveloffset=+2] + +include::modules/virt-viewing-outdated-workloads.adoc[leveloffset=+2] + +[NOTE] +==== +xref:../virt/upgrading-virt.adoc#configuring-workload-updates_upgrading-virt[Configure workload updates] to ensure that VMIs update automatically. +==== [id="additional-resources_upgrading-virt"] == Additional resources @@ -25,4 +48,6 @@ include::modules/virt-monitoring-upgrade-status.adoc[leveloffset=+1] * xref:../operators/understanding/olm/olm-understanding-olm.adoc#olm-csv_olm-understanding-olm[Cluster service versions (CSVs)] +* xref:../virt/live_migration/virt-live-migration.adoc#virt-live-migration[Virtual machine live migration] + * xref:../virt/live_migration/virt-configuring-vmi-eviction-strategy.adoc#virt-configuring-vmi-eviction-strategy[Configuring virtual machine eviction strategy]