mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
212 lines
6.1 KiB
Plaintext
212 lines
6.1 KiB
Plaintext
// Module included in the following assemblies:
|
|
//
|
|
// * etcd/etcd-performance.adoc
|
|
|
|
:_mod-docs-content-type: PROCEDURE
|
|
[id="move-etcd-different-disk_{context}"]
|
|
= Moving etcd to a different disk
|
|
|
|
You can move etcd from a shared disk to a separate disk to prevent or resolve performance issues.
|
|
|
|
The Machine Config Operator (MCO) is responsible for mounting a secondary disk for {product-title} {product-version} container storage.
|
|
|
|
[NOTE]
|
|
====
|
|
This encoded script only supports device names for the following device types:
|
|
|
|
SCSI or SATA:: `/dev/sd*`
|
|
Virtual device:: `/dev/vd*`
|
|
NVMe:: `/dev/nvme*[0-9]\*n*`
|
|
====
|
|
|
|
.Limitations
|
|
|
|
* When the new disk is attached to the cluster, the etcd database is part of the root mount. It is not part of the secondary disk or the intended disk when the primary node is recreated. As a result, the primary node will not create a separate `/var/lib/etcd` mount.
|
|
|
|
.Prerequisites
|
|
|
|
* You have a backup of your cluster's etcd data.
|
|
* You have installed the {oc-first}.
|
|
* You have access to the cluster with `cluster-admin` privileges.
|
|
* Add additional disks before uploading the machine configuration.
|
|
* The `MachineConfigPool` must match `metadata.labels[machineconfiguration.openshift.io/role]`. This applies to a controller, worker, or a custom pool.
|
|
|
|
[NOTE]
|
|
====
|
|
This procedure does not move parts of the root file system, such as `/var/`, to another disk or partition on an installed node.
|
|
====
|
|
|
|
[IMPORTANT]
|
|
====
|
|
This procedure is not supported when using control plane machine sets.
|
|
====
|
|
|
|
.Procedure
|
|
|
|
. Attach the new disk to the cluster and verify that the disk is detected in the node by running the `lsblk` command in a debug shell:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ oc debug node/<node_name>
|
|
----
|
|
+
|
|
[source,terminal]
|
|
----
|
|
# lsblk
|
|
----
|
|
+
|
|
Note the device name of the new disk reported by the `lsblk` command.
|
|
|
|
. Create the following script and name it `etcd-find-secondary-device.sh`:
|
|
+
|
|
[source,bash]
|
|
----
|
|
#!/bin/bash
|
|
set -uo pipefail
|
|
|
|
for device in <device_type_glob>; do # <1>
|
|
/usr/sbin/blkid "${device}" &> /dev/null
|
|
if [ $? == 2 ]; then
|
|
echo "secondary device found ${device}"
|
|
echo "creating filesystem for etcd mount"
|
|
mkfs.xfs -L var-lib-etcd -f "${device}" &> /dev/null
|
|
udevadm settle
|
|
touch /etc/var-lib-etcd-mount
|
|
exit
|
|
fi
|
|
done
|
|
echo "Couldn't find secondary block device!" >&2
|
|
exit 77
|
|
----
|
|
<1> Replace `<device_type_glob>` with a shell glob for your block device type. For SCSI or SATA drives, use `/dev/sd*`; for virtual drives, use `/dev/vd*`; for NVMe drives, use `/dev/nvme*[0-9]\*n*`.
|
|
|
|
. Create a base64-encoded string from the `etcd-find-secondary-device.sh` script and note its contents:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ base64 -w0 etcd-find-secondary-device.sh
|
|
----
|
|
|
|
. Create a `MachineConfig` YAML file named `etcd-mc.yml` with contents such as the following:
|
|
+
|
|
[source,yaml]
|
|
----
|
|
apiVersion: machineconfiguration.openshift.io/v1
|
|
kind: MachineConfig
|
|
metadata:
|
|
labels:
|
|
machineconfiguration.openshift.io/role: master
|
|
name: 98-var-lib-etcd
|
|
spec:
|
|
config:
|
|
ignition:
|
|
version: 3.5.0
|
|
storage:
|
|
files:
|
|
- path: /etc/find-secondary-device
|
|
mode: 0755
|
|
contents:
|
|
source: data:text/plain;charset=utf-8;base64,<encoded_etcd_find_secondary_device_script> # <1>
|
|
systemd:
|
|
units:
|
|
- name: find-secondary-device.service
|
|
enabled: true
|
|
contents: |
|
|
[Unit]
|
|
Description=Find secondary device
|
|
DefaultDependencies=false
|
|
After=systemd-udev-settle.service
|
|
Before=local-fs-pre.target
|
|
ConditionPathExists=!/etc/var-lib-etcd-mount
|
|
|
|
[Service]
|
|
RemainAfterExit=yes
|
|
ExecStart=/etc/find-secondary-device
|
|
|
|
RestartForceExitStatus=77
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
- name: var-lib-etcd.mount
|
|
enabled: true
|
|
contents: |
|
|
[Unit]
|
|
Before=local-fs.target
|
|
|
|
[Mount]
|
|
What=/dev/disk/by-label/var-lib-etcd
|
|
Where=/var/lib/etcd
|
|
Type=xfs
|
|
TimeoutSec=120s
|
|
|
|
[Install]
|
|
RequiredBy=local-fs.target
|
|
- name: sync-var-lib-etcd-to-etcd.service
|
|
enabled: true
|
|
contents: |
|
|
[Unit]
|
|
Description=Sync etcd data if new mount is empty
|
|
DefaultDependencies=no
|
|
After=var-lib-etcd.mount var.mount
|
|
Before=crio.service
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
RemainAfterExit=yes
|
|
ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
|
|
ExecStart=/usr/sbin/setsebool -P rsync_full_access 1
|
|
ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
|
|
ExecStart=/usr/sbin/semanage fcontext -a -t container_var_lib_t '/var/lib/etcd(/.*)?'
|
|
ExecStart=/usr/sbin/setsebool -P rsync_full_access 0
|
|
TimeoutSec=0
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target graphical.target
|
|
- name: restorecon-var-lib-etcd.service
|
|
enabled: true
|
|
contents: |
|
|
[Unit]
|
|
Description=Restore recursive SELinux security contexts
|
|
DefaultDependencies=no
|
|
After=var-lib-etcd.mount
|
|
Before=crio.service
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
RemainAfterExit=yes
|
|
ExecStart=/sbin/restorecon -R /var/lib/etcd/
|
|
TimeoutSec=0
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target graphical.target
|
|
----
|
|
<1> Replace `<encoded_etcd_find_secondary_device_script>` with the encoded script contents that you noted.
|
|
|
|
. Apply the created `MachineConfig` YAML file:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ oc create -f etcd-mc.yml
|
|
----
|
|
|
|
.Verification steps
|
|
|
|
* Run the `grep /var/lib/etcd /proc/mounts` command in a debug shell for the node to ensure that the disk is mounted:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ oc debug node/<node_name>
|
|
----
|
|
+
|
|
[source,terminal]
|
|
----
|
|
# grep -w "/var/lib/etcd" /proc/mounts
|
|
----
|
|
+
|
|
.Example output
|
|
+
|
|
[source,terminal]
|
|
----
|
|
/dev/sdb /var/lib/etcd xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
|
|
----
|