From aba395e183a3eaa823c46d75dedce577cffa86de Mon Sep 17 00:00:00 2001 From: Nadia Pinaeva Date: Mon, 13 Sep 2021 16:14:03 +0200 Subject: [PATCH] OSDOCS-2479: how to change ovs log level Revised to fit doc guidelines by Mike McKiernan. Feedback from Ross--limit the 20-ovs-vswitchd-restart.conf change to setting the log level only instead of the log level and repeating the series of chown commands that are already contributed by 10-ovs-vswitchd-restart.conf. Signed-off-by: Nadia Pinaeva --- ...configuring-ovs-log-level-permanently.adoc | 56 +++++++++++ modules/configuring-ovs-log-level-temp.adoc | 94 +++++++++++++++++++ modules/displaying-ovs-logs.adoc | 31 ++++++ modules/nw-troubleshoot-ovs.adoc | 14 +++ .../troubleshooting-network-issues.adoc | 15 +++ 5 files changed, 210 insertions(+) create mode 100644 modules/configuring-ovs-log-level-permanently.adoc create mode 100644 modules/configuring-ovs-log-level-temp.adoc create mode 100644 modules/displaying-ovs-logs.adoc create mode 100644 modules/nw-troubleshoot-ovs.adoc diff --git a/modules/configuring-ovs-log-level-permanently.adoc b/modules/configuring-ovs-log-level-permanently.adoc new file mode 100644 index 0000000000..0322d1fc0b --- /dev/null +++ b/modules/configuring-ovs-log-level-permanently.adoc @@ -0,0 +1,56 @@ +:ign-config-version: 3.1.0 +ifeval::[{product-version} > 4.6] +:ign-config-version: 3.2.0 +endif::[] + +[id="configuring-ovs-log-level-permanently_{context}"] += Configuring the Open vSwitch log level permanently + +For long-term changes to the Open vSwitch (OVS) log level, you can change the log level permanently. + +.Prerequisites + +* You have access to the cluster as a user with the `cluster-admin` role. + +* You have installed the OpenShift CLI (`oc`). + +.Procedure + +. Create a file, such as `99-change-ovs-loglevel.yaml`, with a `MachineConfig` object like the following example: ++ +[source,yaml,subs="attributes+"] +---- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: master <.> + name: 99-change-ovs-loglevel +spec: + config: + ignition: + version: {ign-config-version} + systemd: + units: + - dropins: + - contents: | + [Service] + ExecStartPost=-/usr/bin/ovs-appctl vlog/set syslog:dbg <.> + ExecReload=-/usr/bin/ovs-appctl vlog/set syslog:dbg + name: 20-ovs-vswitchd-restart.conf + name: ovs-vswitchd.service +---- +<.> After you perform this procedure to configure control plane nodes, repeat the procedure and set the role to `worker` to configure worker nodes. +<.> Set the `syslog:` value. Log levels are `off`, `emer`, `err`, `warn`, `info`, or `dbg`. Setting the value to `off` filters out all log messages. + +. Apply the machine config: ++ +[source,terminal] +---- +$ oc apply -f 99-change-ovs-loglevel.yaml +---- + +ifdef::ign-config-version[] +:!ign-config-version: +endif::[] + diff --git a/modules/configuring-ovs-log-level-temp.adoc b/modules/configuring-ovs-log-level-temp.adoc new file mode 100644 index 0000000000..e57732e37a --- /dev/null +++ b/modules/configuring-ovs-log-level-temp.adoc @@ -0,0 +1,94 @@ +[id="configuring-ovs-log-level-temp_{context}"] += Configuring the Open vSwitch log level temporarily + +For short-term troubleshooting, you can configure the Open vSwitch (OVS) log level temporarily. +The following procedure does not require rebooting the node. +In addition, the configuration change does not persist whenever you reboot the node. + +After you perform this procedure to change the log level, you can receive log messages from the machine config daemon that indicate a content mismatch for the `ovs-vswitchd.service`. +To avoid the log messages, repeat this procedure and set the log level to the original value. + +.Prerequisites + +* You have access to the cluster as a user with the `cluster-admin` role. + +* You have installed the OpenShift CLI (`oc`). + +.Procedure + +. Start a debug pod for a node: ++ +[source,terminal] +---- +$ oc debug node/ +---- + +. Set `/host` as the root directory within the debug shell. The debug pod mounts the root file system from the host in `/host` within the pod. By changing the root directory to `/host`, you can run binaries from the host file system: ++ +[source,terminal] +---- +# chroot /host +---- + +. View the current syslog level for OVS modules: ++ +[source,terminal] +---- +# ovs-appctl vlog/list +---- ++ +The following example output shows the log level for syslog set to `info`. ++ +.Example output +[source,terminal] +---- + console syslog file + ------- ------ ------ +backtrace OFF INFO INFO +bfd OFF INFO INFO +bond OFF INFO INFO +bridge OFF INFO INFO +bundle OFF INFO INFO +bundles OFF INFO INFO +cfm OFF INFO INFO +collectors OFF INFO INFO +command_line OFF INFO INFO +connmgr OFF INFO INFO +conntrack OFF INFO INFO +conntrack_tp OFF INFO INFO +coverage OFF INFO INFO +ct_dpif OFF INFO INFO +daemon OFF INFO INFO +daemon_unix OFF INFO INFO +dns_resolve OFF INFO INFO +dpdk OFF INFO INFO +... +---- + +. Specify the log level in the `/etc/systemd/system/ovs-vswitchd.service.d/10-ovs-vswitchd-restart.conf` file: ++ +[source,text] +---- +Restart=always +ExecStartPre=-/bin/sh -c '/usr/bin/chown -R :$${OVS_USER_ID##*:} /var/lib/openvswitch' +ExecStartPre=-/bin/sh -c '/usr/bin/chown -R :$${OVS_USER_ID##*:} /etc/openvswitch' +ExecStartPre=-/bin/sh -c '/usr/bin/chown -R :$${OVS_USER_ID##*:} /run/openvswitch' +ExecStartPost=-/usr/bin/ovs-appctl vlog/set syslog:dbg +ExecReload=-/usr/bin/ovs-appctl vlog/set syslog:dbg +---- ++ +In the preceding example, the log level is set to `dbg`. +Change the last two lines by setting `syslog:` to `off`, `emer`, `err`, `warn`, `info`, or `dbg`. The `off` log level filters out all log messages. + +. Restart the service: ++ +[source,terminal] +---- +# systemctl daemon-reload +---- ++ +[source,terminal] +---- +# systemctl restart ovs-vswitchd +---- + diff --git a/modules/displaying-ovs-logs.adoc b/modules/displaying-ovs-logs.adoc new file mode 100644 index 0000000000..90ec8c5d6c --- /dev/null +++ b/modules/displaying-ovs-logs.adoc @@ -0,0 +1,31 @@ +[id="displaying-ovs-logs_{context}"] += Displaying Open vSwitch logs + +Use the following procedure to display Open vSwitch (OVS) logs. + +.Prerequisites + +* You have access to the cluster as a user with the `cluster-admin` role. + +* You have installed the OpenShift CLI (`oc`). + +.Procedure + +* Run one of the following commands: + +** Display the logs by using the `oc` command from outside the cluster: ++ +[source,terminal] +---- +$ oc adm node-logs -u ovs-vswitchd +---- + +** Display the logs after logging on to a node in the cluster: ++ +[source,terminal] +---- +# journalctl -b -f -u ovs-vswitchd.service +---- ++ +One way to log on to a node is by using the `oc debug node/` command. + diff --git a/modules/nw-troubleshoot-ovs.adoc b/modules/nw-troubleshoot-ovs.adoc new file mode 100644 index 0000000000..309bae0abb --- /dev/null +++ b/modules/nw-troubleshoot-ovs.adoc @@ -0,0 +1,14 @@ +[id="nw-troubleshoot-ovs_{context}"] += Troubleshooting Open vSwitch issues + +To troubleshoot some Open vSwitch (OVS) issues, you might need to configure the log level to include more information. + +If you modify the log level on a node temporarily, be aware that you can receive log messages from the machine config daemon on the node like the following example: + +[source,terminal] +---- +E0514 12:47:17.998892 2281 daemon.go:1350] content mismatch for file /etc/systemd/system/ovs-vswitchd.service: [Unit] +---- + +To avoid the log messages related to the mismatch, revert the log level change after you complete your troubleshooting. + diff --git a/support/troubleshooting/troubleshooting-network-issues.adoc b/support/troubleshooting/troubleshooting-network-issues.adoc index c3d6e37c45..c56dbdef84 100644 --- a/support/troubleshooting/troubleshooting-network-issues.adoc +++ b/support/troubleshooting/troubleshooting-network-issues.adoc @@ -7,3 +7,18 @@ toc::[] // How the network interface is selected include::modules/nw-how-nw-iface-selected.adoc[leveloffset=+1] + +// Troubleshooting OVS issues +include::modules/nw-troubleshoot-ovs.adoc[leveloffset=+1] + +include::modules/configuring-ovs-log-level-temp.adoc[leveloffset=+2] + +include::modules/configuring-ovs-log-level-permanently.adoc[leveloffset=+2] +.Additional resources + +* xref:../../post_installation_configuration/machine-configuration-tasks.adoc#understanding-the-machine-config-operator[Understanding the Machine Config Operator] + +* xref:../../post_installation_configuration/machine-configuration-tasks.adoc#checking-mco-status_post-install-machine-configuration-tasks[Checking machine config pool status] + +include::modules/displaying-ovs-logs.adoc[leveloffset=+2] +