openshift-docs/modules/move-etcd-different-disk.adoc

// Module included in the following assemblies:
//
// * scalability_and_performance/recommended-performance-scale-practices/recommended-etcd-practices.adoc

:_mod-docs-content-type: PROCEDURE
[id="move-etcd-different-disk_{context}"]
= Moving etcd to a different disk

You can move etcd from a shared disk to a separate disk to prevent or resolve performance issues.

The Machine Config Operator (MCO) is responsible for mounting a secondary disk for {product-title} {product-version} container storage.

[NOTE]
====
This procedure does not move parts of the root file system, such as `/var/`, to another disk or partition on an installed node.
====

.Prerequisites

* You have installed the {oc-first}.
* You have access to the cluster with `cluster-admin` privileges.
* The `MachineConfigPool` must match `metadata.labels[machineconfiguration.openshift.io/role]`. This applies to a controller, worker, or a custom pool.

.Procedure

. Attach the new disk to the cluster and verify that the disk is detected in the node by using the `lsblk` command in a debug shell:
+
[source,terminal]
----
$ oc debug node/<node_name>
----
+
[source,terminal]
----
# lsblk
----
+
Note the device name of the new disk reported by the `lsblk` command.

. Create a `MachineConfig` YAML file named `etcd-mc.yml` with contents such as the following, replacing instances of `<new_disk_name>` with the noted device name:
+
[source,yaml]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-var-lib-etcd
spec:
  config:
    ignition:
      version: 3.4.0
    systemd:
      units:
      - contents: |
          [Unit]
          Description=Make File System on /dev/<new_disk_name>
          DefaultDependencies=no
          BindsTo=dev-<new_disk_name>.device
          After=dev-<new_disk_name>.device var.mount
          Before=systemd-fsck@dev-<new_disk_name>.service

          [Service]
          Type=oneshot
          RemainAfterExit=yes
          ExecStart=/usr/lib/systemd/systemd-makfs.xfs -f /dev/<new_disk_name>
          TimeoutSec=0

          [Install]
          WantedBy=var-lib-containers.mount
        enabled: true
        name: systemd-mkfs@dev-<new_disk_name>.service
      - contents: |
          [Unit]
          Description=Mount /dev/<new_disk_name> to /var/lib/etcd
          Before=local-fs.target
          Requires=systemd-mkfs@dev-<new_disk_name>.service
          After=systemd-mkfs@dev-<new_disk_name>.service var.mount

          [Mount]
          What=/dev/<new_disk_name>
          Where=/var/lib/etcd
          Type=xfs
          Options=defaults,prjquota

          [Install]
          WantedBy=local-fs.target
        enabled: true
        name: var-lib-etcd.mount
      - contents: |
          [Unit]
          Description=Sync etcd data if new mount is empty
          DefaultDependencies=no
          After=var-lib-etcd.mount var.mount
          Before=crio.service

          [Service]
          Type=oneshot
          RemainAfterExit=yes
          ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
          ExecStart=semanage fcontext -a -e /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
          ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
          TimeoutSec=0

          [Install]
          WantedBy=multi-user.target graphical.target
        enabled: true
        name: sync-var-lib-etcd-to-etcd.service
      - contents: |
          [Unit]
          Description=Restore recursive SELinux security contexts
          DefaultDependencies=no
          After=var-lib-etcd.mount
          Before=crio.service

          [Service]
          Type=oneshot
          RemainAfterExit=yes
          ExecStart=/sbin/restorecon -R /var/lib/etcd/
          TimeoutSec=0

          [Install]
          WantedBy=multi-user.target graphical.target
        enabled: true
        name: restorecon-var-lib-etcd.service
----

. Log in to the cluster as a user with `cluster-admin` privileges and create the machine configuration:
+
[source,terminal]
----
$ oc login -u <username> -p <password>
----
+
[source,terminal]
----
$ oc create -f etcd-mc.yml
----
+
The nodes are updated and rebooted. After the reboot completes, the following events occur:
+
* An XFS file system is created on the specified disk.
* The disk mounts to `/var/lib/etcd`.
* The content from `/sysroot/ostree/deploy/rhcos/var/lib/etcd` syncs to `/var/lib/etcd`.
* A restore of `SELinux` labels is forced for `/var/lib/etcd`.
* The old content is not removed.

. After the nodes are on a separate disk, update the `etcd-mc.yml` file with contents such as the following, replacing instances of `<new_disk_name>` with the noted device name:
+
[source,yaml]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-var-lib-etcd
spec:
  config:
    ignition:
      version: 3.4.0
    systemd:
      units:
      - contents: |
          [Unit]
          Description=Mount /dev/<new_disk_name> to /var/lib/etcd
          Before=local-fs.target
          After=var.mount

          [Mount]
          What=/dev/<new_disk_name>
          Where=/var/lib/etcd
          Type=xfs
          Options=defaults,prjquota

          [Install]
          WantedBy=local-fs.target
        enabled: true
        name: var-lib-etcd.mount
----

. Apply the modified version that removes the logic for creating and syncing the device to prevent the nodes from rebooting:
+
[source,terminal]
----
$ oc replace -f etcd-mc.yml
----

.Verification steps

* Run the `grep <new_disk_name> /proc/mounts` command in a debug shell for the node to ensure that the disk mounted:
+
[source,terminal]
----
$ oc debug node/<node_name>
----
+
[source,terminal]
----
# grep <new_disk_name> /proc/mounts
----
+
.Example output
+
[source,terminal]
----
/dev/nvme1n1 /var/lib/etcd xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota 0 0
----