diff --git a/installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc b/installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc index 0402a760da..54173d091a 100644 --- a/installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc +++ b/installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc @@ -131,6 +131,8 @@ include::modules/installation-user-infra-machines-advanced.adoc[leveloffset=+2] include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] +include::modules/rhcos-enabling-multipath.adoc[leveloffset=+2] + include::modules/architecture-rhcos-updating-bootloader.adoc[leveloffset=+2] include::modules/installation-installing-bare-metal.adoc[leveloffset=+1] diff --git a/installing/installing_bare_metal/installing-bare-metal.adoc b/installing/installing_bare_metal/installing-bare-metal.adoc index 3c129eab90..9c3504ff14 100644 --- a/installing/installing_bare_metal/installing-bare-metal.adoc +++ b/installing/installing_bare_metal/installing-bare-metal.adoc @@ -32,7 +32,7 @@ include::modules/cluster-entitlements.adoc[leveloffset=+1] .Additional resources -* See xref:../../installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc#installing-restricted-networks-bare-metal[Installing a user-provisioned bare metal cluster on a restricted network] for more information about performing a restricted network installation on bare metal infrastructure that you provision. +* See xref:../../installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc#installing-restricted-networks-bare-metal[Installing a user-provisioned bare metal cluster on a restricted network] for more information about performing a restricted network installation on bare metal infrastructure that you provision. include::modules/installation-requirements-user-infra.adoc[leveloffset=+1] @@ -150,6 +150,12 @@ include::modules/installation-user-infra-machines-advanced.adoc[leveloffset=+2] include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] +include::modules/rhcos-enabling-multipath.adoc[leveloffset=+2] + +.Additional resources + +* See xref:../../installing/installing_bare_metal/installing-bare-metal.adoc#creating-machines-bare-metal[Installing {op-system} and starting the {product-title} bootstrap process] for more information on using special `coreos.inst.*` arguments to direct the live installer. + include::modules/architecture-rhcos-updating-bootloader.adoc[leveloffset=+2] include::modules/installation-installing-bare-metal.adoc[leveloffset=+1] diff --git a/installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc b/installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc index 6ad1b6427b..a01fbb14fa 100644 --- a/installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc +++ b/installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc @@ -152,6 +152,8 @@ include::modules/installation-user-infra-machines-advanced.adoc[leveloffset=+2] include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] +include::modules/rhcos-enabling-multipath.adoc[leveloffset=+2] + include::modules/architecture-rhcos-updating-bootloader.adoc[leveloffset=+2] include::modules/installation-installing-bare-metal.adoc[leveloffset=+1] diff --git a/modules/installation-complete-user-infra.adoc b/modules/installation-complete-user-infra.adoc index a769e5ab6d..43ec6b43e6 100644 --- a/modules/installation-complete-user-infra.adoc +++ b/modules/installation-complete-user-infra.adoc @@ -149,12 +149,17 @@ cluster machines. ifndef::ibm-power[] . For an installation with Fibre Channel Protocol (FCP), additional steps are required to enable multipathing. Do not enable multipathing during installation. + -See "Enabling multipathing with kernel arguments on RHCOS" in the _Post-installation configuration_ documentation for more information. +[NOTE] +==== +When installing with multipath, it is strongly recommended to enable it at installation time, and not at a later time, which can cause problems. +==== ++ +See "Enabling multipathing with kernel arguments on {op-system}" in the _Installing on bare metal_ documentation for more information. endif::ibm-power[] ifdef::ibm-power[] . Additional steps are required to enable multipathing. Do not enable multipathing during installation. + -See "Enabling multipathing with kernel arguments on RHCOS" in the _Post-installation configuration_ documentation for more information. +See the _Installing {op-system} and starting the {product-title} bootstrap process_ documentation for more information. .. To display a boot list and specify the possible boot devices if the system is booted in normal mode, enter the following command: + diff --git a/modules/rhcos-enabling-multipath-day-2.adoc b/modules/rhcos-enabling-multipath-day-2.adoc new file mode 100644 index 0000000000..ee96a11de7 --- /dev/null +++ b/modules/rhcos-enabling-multipath-day-2.adoc @@ -0,0 +1,139 @@ +// Module included in the following assemblies: +// +// * post_installation_configuration/machine-configuration-tasks.adoc + +[id="rhcos-enabling-multipath-day-2_{context}"] += Enabling multipathing with kernel arguments on {op-system} + +{op-system-first} supports multipathing on the primary disk, allowing stronger resilience to hardware failure to achieve higher host availability. Post-installation support is available by activating multipathing via the machine config. + +[IMPORTANT] +==== +Enabling multipathing during installation is supported and recommended for nodes provisioned in {product-title} 4.8 or higher. In setups where any I/O to non-optimized paths results in I/O system errors, you must enable multipathing at installation time. For more information about enabling multipathing during installation time, see "Enabling multipathing with kernel arguments on RHCOS" in the _Installing on bare metal_ documentation. +==== + +[IMPORTANT] +==== +On IBM Z and LinuxONE, you can enable multipathing only if you configured your cluster for it during installation. For more information, see "Installing {op-system} and starting the {product-title} bootstrap process" in _Installing a cluster with z/VM on IBM Z and LinuxONE_. +==== +// Add xref once it's allowed. + +.Prerequisites +* You have a running {product-title} cluster that uses version 4.7 or later. +* You are logged in to the cluster as a user with administrative privileges. + +.Procedure + +. To enable multipathing post-installation on control plane nodes: + +* Create a machine config file, such as `99-master-kargs-mpath.yaml`, that instructs the cluster to add the `master` label and that identifies the multipath kernel argument, for example: ++ +[source,yaml] +---- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: "master" + name: 99-master-kargs-mpath +spec: + kernelArguments: + - 'rd.multipath=default' + - 'root=/dev/disk/by-label/dm-mpath-root' +---- + +. To enable multipathing post-installation on worker nodes: + +* Create a machine config file, such as `99-worker-kargs-mpath.yaml`, that instructs the cluster to add the `worker` label and that identifies the multipath kernel argument, for example: ++ +[source,yaml] +---- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: "worker" + name: 99-worker-kargs-mpath +spec: + kernelArguments: + - 'rd.multipath=default' + - 'root=/dev/disk/by-label/dm-mpath-root' +---- + +. Create the new machine config by using either the master or worker YAML file you previously created: ++ +[source,terminal] +---- +$ oc create -f ./99-master-kargs-mpath.yaml +---- + +. Check the machine configs to see that the new one was added: ++ +[source,terminal] +---- +$ oc get MachineConfig +---- ++ +.Example output +[source,terminal] +---- +NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE +00-master 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +00-worker 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +01-master-container-runtime 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +01-master-kubelet 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +01-worker-container-runtime 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +01-worker-kubelet 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +99-master-kargs-mpath 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 105s +99-master-generated-registries 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +99-master-ssh 3.2.0 40m +99-worker-generated-registries 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +99-worker-ssh 3.2.0 40m +rendered-master-23e785de7587df95a4b517e0647e5ab7 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +rendered-worker-5d596d9293ca3ea80c896a1191735bb1 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +---- + +. Check the nodes: ++ +[source,terminal] +---- +$ oc get nodes +---- ++ +.Example output +[source,terminal] +---- +NAME STATUS ROLES AGE VERSION +ip-10-0-136-161.ec2.internal Ready worker 28m v1.20.0 +ip-10-0-136-243.ec2.internal Ready master 34m v1.20.0 +ip-10-0-141-105.ec2.internal Ready,SchedulingDisabled worker 28m v1.20.0 +ip-10-0-142-249.ec2.internal Ready master 34m v1.20.0 +ip-10-0-153-11.ec2.internal Ready worker 28m v1.20.0 +ip-10-0-153-150.ec2.internal Ready master 34m v1.20.0 +---- ++ +You can see that scheduling on each worker node is disabled as the change is being applied. + +. Check that the kernel argument worked by going to one of the worker nodes and listing +the kernel command line arguments (in `/proc/cmdline` on the host): ++ +[source,terminal] +---- +$ oc debug node/ip-10-0-141-105.ec2.internal +---- ++ +.Example output +[source,terminal] +---- +Starting pod/ip-10-0-141-105ec2internal-debug ... +To use host binaries, run `chroot /host` + +sh-4.2# cat /host/proc/cmdline +... +rd.multipath=default root=/dev/disk/by-label/dm-mpath-root +... + +sh-4.2# exit +---- ++ +You should see the added kernel arguments. diff --git a/modules/rhcos-enabling-multipath.adoc b/modules/rhcos-enabling-multipath.adoc index 0dfd10c01e..8e8a9dd0e0 100644 --- a/modules/rhcos-enabling-multipath.adoc +++ b/modules/rhcos-enabling-multipath.adoc @@ -1,122 +1,65 @@ // Module included in the following assemblies: // -// * post_installation_configuration/machine-configuration-tasks.adoc +// * installing/installing_bare_metal/installing-bare-metal.adoc +// * installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc +// * installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc [id="rhcos-enabling-multipath_{context}"] = Enabling multipathing with kernel arguments on {op-system} {op-system} supports multipathing on the primary disk, allowing stronger resilience to hardware failure to achieve higher host availability. -[IMPORTANT] -==== -Multipathing is only supported when it is activated using the machine config as documented in the following procedure. It must be enabled after {op-system} installation. -==== +You can enable multipathing at installation time for nodes that were provisioned in {product-title} 4.8 or later. While post-installation support is available by activating multipathing via the machine config, enabling multipathing during installation is recommended. + +In setups where any I/O to non-optimized paths results in I/O system errors, you must enable multipathing at installation time. [IMPORTANT] ==== -On IBM Z and LinuxONE, you can enable multipathing only if you configured your cluster for it during installation. For more information, see "Creating {op-system-first} machines" in _Installing a cluster with z/VM on IBM Z and LinuxONE_. +On IBM Z and LinuxONE, you can enable multipathing only if you configured your cluster for it during installation. For more information, see "Installing {op-system} and starting the {product-title} bootstrap process" in _Installing a cluster with z/VM on IBM Z and LinuxONE_. ==== // Add xref once it's allowed. +The following procedure enables multipath at installation time and appends kernel arguments to the `coreos-installer install` command so that the installed system itself will use multipath beginning from the first boot. + .Prerequisites -* You have a running {product-title} cluster that uses version 4.7 or later. +* You have a running {product-title} cluster that uses version 4.8 or later. * You are logged in to the cluster as a user with administrative privileges. .Procedure -. To enable multipathing on control plane nodes (also known as the master nodes): - -* Create a machine config file, such as `99-master-kargs-mpath.yaml`, that instructs the cluster to add the `master` label and that identifies the multipath kernel argument, for example: - -+ -[source,yaml] ----- -apiVersion: machineconfiguration.openshift.io/v1 -kind: MachineConfig -metadata: - labels: - machineconfiguration.openshift.io/role: "master" - name: 99-master-kargs-mpath -spec: - kernelArguments: - - rd.multipath=default - - root=/dev/disk/by-label/dm-mpath-root ----- - -. To enable multipathing on worker nodes: - -* Create a machine config file, such as `99-worker-kargs-mpath.yaml`, that instructs the cluster to add the `worker` label and that identifies the multipath kernel argument, for example: -+ -[source,yaml] ----- -apiVersion: machineconfiguration.openshift.io/v1 -kind: MachineConfig -metadata: - labels: - machineconfiguration.openshift.io/role: "worker" - name: 99-worker-kargs-mpath -spec: - kernelArguments: - - rd.multipath=default - - root=/dev/disk/by-label/dm-mpath-root ----- - -. Create the new machine config by using either the master or worker YAML file you previously created: +. To enable multipath and start the `multipathd` daemon, run the following command: + [source,terminal] ---- -$ oc create -f ./99-master-kargs-mpath.yaml +$ mpathconf --enable && systemctl start multipathd.service ---- +** Optional: If booting the PXE or ISO, you can instead enable multipath by adding `rd.multipath=default` from the kernel command line. -. Check the machine configs to see that the new one was added: +. Append the kernel arguments by invoking the `coreos-installer` program: ++ +* If there is only one multipath device connected to the machine, it should be available at path `/dev/mapper/mpatha`. For example: + [source,terminal] ---- -$ oc get MachineConfig +$ coreos-installer install /dev/mapper/mpatha \ <1> +--append-karg rd.multipath=default \ +--append-karg root=/dev/disk/by-label/dm-mpath-root ---- +<1> Indicates the path of the single multipathed device. ++ +* If there are multiple multipath devices connected to the machine, or to be more explicit, instead of using `/dev/mapper/mpatha`, it is recommended to use the World Wide Name (WWN) symlink available in `/dev/disk/by-id`. For example: + -.Example output [source,terminal] ---- -NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE -00-master 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -00-worker 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -01-master-container-runtime 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -01-master-kubelet 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -01-worker-container-runtime 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -01-worker-kubelet 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -99-master-kargs-mpath 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 105s -99-master-generated-registries 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -99-master-ssh 3.2.0 40m -99-worker-generated-registries 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -99-worker-ssh 3.2.0 40m -rendered-master-23e785de7587df95a4b517e0647e5ab7 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m -rendered-worker-5d596d9293ca3ea80c896a1191735bb1 52dd3ba6a9a527fc3ab42afac8d12b693534c8c9 3.2.0 33m +$ coreos-installer install /dev/disk/by-id/wwn- \ <1> +--append-karg rd.multipath=default \ +--append-karg root=/dev/disk/by-label/dm-mpath-root ---- +<1> Indicates the WWN ID of the target multipathed device. For example, `0xx194e957fcedb4841`. ++ +This symlink can also be used as the `coreos.inst.install_dev` kernel argument when using special `coreos.inst.*` arguments to direct the live installer. For more information, see "Installing {op-system} and starting the {product-title} bootstrap process". -. Check the nodes: -+ -[source,terminal] ----- -$ oc get nodes ----- -+ -.Example output -[source,terminal] ----- -NAME STATUS ROLES AGE VERSION -ip-10-0-136-161.ec2.internal Ready worker 28m v1.20.0 -ip-10-0-136-243.ec2.internal Ready master 34m v1.20.0 -ip-10-0-141-105.ec2.internal Ready,SchedulingDisabled worker 28m v1.20.0 -ip-10-0-142-249.ec2.internal Ready master 34m v1.20.0 -ip-10-0-153-11.ec2.internal Ready worker 28m v1.20.0 -ip-10-0-153-150.ec2.internal Ready master 34m v1.20.0 ----- -+ -You can see that scheduling on each worker node is disabled as the change is being applied. - -. Check that the kernel argument worked by going to one of the worker nodes and listing -the kernel command line arguments (in `/proc/cmdline` on the host): +. Check that the kernel arguments worked by going to one of the worker nodes and listing the kernel command line arguments (in `/proc/cmdline` on the host): + [source,terminal] ---- diff --git a/post_installation_configuration/machine-configuration-tasks.adoc b/post_installation_configuration/machine-configuration-tasks.adoc index 0e13840440..a5e02e1eaa 100644 --- a/post_installation_configuration/machine-configuration-tasks.adoc +++ b/post_installation_configuration/machine-configuration-tasks.adoc @@ -24,11 +24,16 @@ include::modules/checking-mco-status.adoc[leveloffset=+2] You can use the tasks in this section to create `MachineConfig` objects that modify files, systemd unit files, and other operating system features running on {product-title} nodes. For more ideas on working with machine configs, see content related to link:https://access.redhat.com/solutions/5096731[adding] or link:https://access.redhat.com/solutions/4510281[updating] SSH authorized keys, link:https://access.redhat.com/verify-images-ocp4[verifying image signatures], link:https://access.redhat.com/solutions/4727321[enabling SCTP], and link:https://access.redhat.com/solutions/5170251[configuring iSCSI initiatornames] for {product-title}. -{product-title} supports link:https://coreos.github.io/ignition/configuration-v3_2/[Ignition specification version 3.2]. All new machine configs you create going forward should be based on Ignition specification version 3.2. If you are upgrading your {product-title} cluster, any existing Ignition specification version 2.x machine configs will be translated automatically to specification version 3.2. +{product-title} supports link:https://coreos.github.io/ignition/configuration-v3_2/[Ignition specification version 3.2]. All new machine configs you create going forward should be based on Ignition specification version 3.2. If you are upgrading your {product-title} cluster, any existing Ignition specification version 2.x machine configs will be translated automatically to specification version 3.2. include::modules/installation-special-config-chrony.adoc[leveloffset=+2] include::modules/nodes-nodes-kernel-arguments.adoc[leveloffset=+2] -include::modules/rhcos-enabling-multipath.adoc[leveloffset=+2] +include::modules/rhcos-enabling-multipath-day-2.adoc[leveloffset=+2] + +.Additional resources + +* See xref:../installing/installing_bare_metal/installing-bare-metal.adoc#rhcos-enabling-multipath_installing-bare-metal[Enabling multipathing with kernel arguments on RHCOS] for more information about enabling multipathing during installation time. + include::modules/nodes-nodes-rtkernel-arguments.adoc[leveloffset=+2] include::modules/machineconfig-modify-journald.adoc[leveloffset=+2] include::modules/rhcos-add-extensions.adoc[leveloffset=+2]