mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
OCPBUGS-34611-new: Documented the additional br-ex
This commit is contained in:
dfitzmau
committed by
openshift-cherrypick-robot
openshift-cherrypick-robot
parent
9ea31ad087
commit
49eec42d1b
296
modules/configuring-ovnk-use-second-ovs-bridge.adoc
Normal file
296
modules/configuring-ovnk-use-second-ovs-bridge.adoc
Normal file
@@ -0,0 +1,296 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * support/troubleshooting/troubleshooting-network-issues.adoc
|
||||
|
||||
:_mod-docs-content-type: PROCEDURE
|
||||
[id="configuring-ovnk-use-second-ovs-bridge_{context}"]
|
||||
= Configuring OVN-Kubernetes to use a secondary OVS bridge
|
||||
|
||||
You can create an additional or _secondary_ Open vSwitch (OVS) bridge, `br-ex1`, that OVN-Kubernetes manages and the Multiple External Gateways (MEG) implementation uses for defining external gateways for an {product-title} node. You can define a MEG in an `AdminPolicyBasedExternalRoute` custom resource (CR). The MEG implementation provides a pod with access to multiple gateways, equal-cost multipath (ECMP) routes, and the Bidirectional Forwarding Detection (BFD) implementation.
|
||||
|
||||
Consider a use case for pods impacted by the Multiple External Gateways (MEG) feature and you want to egress traffic to a different interface, for example `br-ex1`, on a node. Egress traffic for pods not impacted by MEG get routed to the default OVS `br-ex` bridge.
|
||||
|
||||
You must define the additional bridge in an interface definition of a machine configuration manifest file. The Machine Config Operator uses the manifest to create a new file at `/etc/ovnk/extra_bridge` on the host. The new file includes the name of the network interface that the additional OVS bridge configures for a node.
|
||||
|
||||
After you create and edit the manifest file, the Machine Config Operator completes tasks in the following order:
|
||||
|
||||
. Drains nodes in singular order based on the selected machine configuration pool.
|
||||
. Injects Ignition configuration files into each node, so that each node receives the additional `br-ex1` bridge network configuration.
|
||||
. Verify that the `br-ex` MAC address matches the MAC address for the interface that `br-ex` uses for the network connection.
|
||||
. Executes the `configure-ovs.sh` shell script that references the new interface definition.
|
||||
. Adds `br-ex` and `br-ex1` to the host node.
|
||||
. Uncordons the nodes.
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
After all the nodes return to the `Ready` state and the OVN-Kubernetes Operator detects and configures `br-ex` and `br-ex1`, the Operator applies the `k8s.ovn.org/l3-gateway-config` annotation to each node.
|
||||
====
|
||||
|
||||
For more information about useful situations for the additional `br-ex1` bridge and a situation that always requires the default `br-ex` bridge, see "Configuration for a localnet topology".
|
||||
|
||||
.Procedure
|
||||
|
||||
. Optional: Create an interface connection that your additional bridge, `br-ex1`, can use by completing the following steps. The example steps show the creation of a new bond and its dependent interfaces that are all defined in a machine configuration manifest file. The additional bridge uses the `MachineConfig` object to form a additional bond interface.
|
||||
+
|
||||
[IMPORTANT]
|
||||
====
|
||||
Do not use the Kubernetes NMState Operator to define or a `NodeNetworkConfigurationPolicy` (NNCP) manifest file to define the additional interface.
|
||||
|
||||
Also ensure that the additional interface or sub-interfaces when defining a `bond` interface are not used by an existing `br-ex` OVN Kubernetes network deployment.
|
||||
====
|
||||
+
|
||||
.. Create the following interface definition files. These files get added to a machine configuration manifest file so that host nodes can access the definition files.
|
||||
+
|
||||
.Example of the first interface definition file that is named `eno1.config`
|
||||
[source,ini]
|
||||
----
|
||||
[connection]
|
||||
id=eno1
|
||||
type=ethernet
|
||||
interface-name=eno1
|
||||
master=bond1
|
||||
slave-type=bond
|
||||
autoconnect=true
|
||||
autoconnect-priority=20
|
||||
----
|
||||
+
|
||||
.Example of the second interface definition file that is named `eno2.config`
|
||||
[source,ini]
|
||||
----
|
||||
[connection]
|
||||
id=eno2
|
||||
type=ethernet
|
||||
interface-name=eno2
|
||||
master=bond1
|
||||
slave-type=bond
|
||||
autoconnect=true
|
||||
autoconnect-priority=20
|
||||
----
|
||||
+
|
||||
.Example of the second bond interface definition file that is named `bond1.config`
|
||||
[source,ini]
|
||||
----
|
||||
[connection]
|
||||
id=bond1
|
||||
type=bond
|
||||
interface-name=bond1
|
||||
autoconnect=true
|
||||
connection.autoconnect-slaves=1
|
||||
autoconnect-priority=20
|
||||
|
||||
[bond]
|
||||
mode=802.3ad
|
||||
miimon=100
|
||||
xmit_hash_policy="layer3+4"
|
||||
|
||||
[ipv4]
|
||||
method=auto
|
||||
----
|
||||
+
|
||||
.. Convert the definition files to Base64 encoded strings by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ base64 <directory_path>/en01.config
|
||||
----
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ base64 <directory_path>/eno2.config
|
||||
----
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ base64 <directory_path>/bond1.config
|
||||
----
|
||||
|
||||
. Prepare the environment variables. Replace `<machine_role>` with the node role, such as `worker`, and replace `<interface_name>` with the name of your additional `br-ex` bridge name.
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ export ROLE=<machine_role>
|
||||
----
|
||||
|
||||
. Define each interface definition in a machine configuration manifest file:
|
||||
+
|
||||
.Example of a machine configuration file with definitions added for `bond1`, `eno1`, and `en02`
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineConfig
|
||||
metadata:
|
||||
labels:
|
||||
machineconfiguration.openshift.io/role: ${worker}
|
||||
name: 12-${ROLE}-sec-bridge-cni
|
||||
spec:
|
||||
config:
|
||||
ignition:
|
||||
version: 3.2.0
|
||||
storage:
|
||||
files:
|
||||
- contents:
|
||||
source: data:;base64,<base-64-encoded-contents-for-bond1.conf>
|
||||
path: /etc/NetworkManager/system-connections/bond1.nmconnection
|
||||
filesystem: root
|
||||
mode: 0600
|
||||
- contents:
|
||||
source: data:;base64,<base-64-encoded-contents-for-eno1.conf>
|
||||
path: /etc/NetworkManager/system-connections/eno1.nmconnection
|
||||
filesystem: root
|
||||
mode: 0600
|
||||
- contents:
|
||||
source: data:;base64,<base-64-encoded-contents-for-eno2.conf>
|
||||
path: /etc/NetworkManager/system-connections/eno2.nmconnection
|
||||
filesystem: root
|
||||
mode: 0600
|
||||
# ...
|
||||
----
|
||||
|
||||
. Create a machine configuration manifest file for configuring the network plugin by entering the following command in your terminal:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc create -f <machine_config_file_name>
|
||||
----
|
||||
|
||||
. Create an Open vSwitch (OVS) bridge, `br-ex1`, on nodes by using the OVN-Kubernetes network plugin to create an `extra_bridge` file`. Ensure that you save the file in the `/etc/ovnk/extra_bridge` path of the host. The file must state the interface name that supports the additional bridge and not the default interface that supports `br-ex`, which holds the primary IP address of the node.
|
||||
+
|
||||
.Example configuration for the `extra_bridge` file, `/etc/ovnk/extra_bridge`, that references a additional interface
|
||||
[source,text]
|
||||
----
|
||||
bond1
|
||||
----
|
||||
|
||||
. Create a machine configuration manifest file that defines the existing static interface that hosts `br-ex1` on any nodes restarted on your cluster:
|
||||
+
|
||||
.Example of a machine configuration file that defines `bond1` as the interface for hosting `br-ex1`
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineConfig
|
||||
metadata:
|
||||
labels:
|
||||
machineconfiguration.openshift.io/role: ${worker}
|
||||
name: 12-worker-extra-bridge
|
||||
spec:
|
||||
config:
|
||||
ignition:
|
||||
version: 3.2.0
|
||||
storage:
|
||||
files:
|
||||
- path: /etc/ovnk/extra_bridge
|
||||
mode: 0420
|
||||
overwrite: true
|
||||
contents:
|
||||
source: data:text/plain;charset=utf-8,bond1
|
||||
filesystem: root
|
||||
----
|
||||
|
||||
. Apply the machine-configuration to your selected nodes:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc create -f <machine_config_file_name>
|
||||
----
|
||||
|
||||
. Optional: You can override the `br-ex` selection logic for nodes by creating a machine configuration file that in turn creates a `/var/lib/ovnk/iface_default_hint` resource.
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
The resource lists the name of the interface that `br-ex` selects for your cluster. By default, `br-ex` selects the primary interface for a node based on boot order and the IP address subnet in the machine network. Certain machine network configurations might require that `br-ex` continues to select the default interfaces or bonds for a host node.
|
||||
====
|
||||
+
|
||||
.. Create a machine configuration file on the host node to override the default interface.
|
||||
+
|
||||
[IMPORTANT]
|
||||
====
|
||||
Only create this machine configuration file for the purposes of changing the `br-ex` selection logic. Using this file to change the IP addresses of existing nodes in your cluster is not supported.
|
||||
====
|
||||
+
|
||||
.Example of a machine configuration file that overrides the default interface
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineConfig
|
||||
metadata:
|
||||
labels:
|
||||
machineconfiguration.openshift.io/role: ${worker}
|
||||
name: 12-worker-br-ex-override
|
||||
spec:
|
||||
config:
|
||||
ignition:
|
||||
version: 3.2.0
|
||||
storage:
|
||||
files:
|
||||
- path: /var/lib/ovnk/iface_default_hint
|
||||
mode: 0420
|
||||
overwrite: true
|
||||
contents:
|
||||
source: data:text/plain;charset=utf-8,bond0 <1>
|
||||
filesystem: root
|
||||
----
|
||||
<1> Ensure `bond0` exists on the node before you apply the machine configuration file to the node.
|
||||
+
|
||||
.. Before you apply the configuration to all new nodes in your cluster, reboot the host node to verify that `br-ex` selects the intended interface and does not conflict with the new interfaces that you defined on `br-ex1`.
|
||||
+
|
||||
.. Apply the machine configuration file to all new nodes in your cluster:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc create -f <machine_config_file_name>
|
||||
----
|
||||
|
||||
.Verification
|
||||
|
||||
. Identify the IP addresses of nodes with the `exgw-ip-addresses` label in your cluster to verify that the nodes use the additional bridge instead of the default bridge:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc get nodes -o json | grep --color exgw-ip-addresses
|
||||
----
|
||||
+
|
||||
.Example output
|
||||
[source,terminal]
|
||||
----
|
||||
"k8s.ovn.org/l3-gateway-config":
|
||||
\"exgw-ip-address\":\"172.xx.xx.yy/24\",\"next-hops\":[\"xx.xx.xx.xx\"],
|
||||
----
|
||||
|
||||
. Observe that the additional bridge exists on target nodes by reviewing the network interface names on the host node:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc debug node/<node_name> -- chroot /host sh -c "ip a | grep mtu | grep br-ex"
|
||||
----
|
||||
+
|
||||
.Example output
|
||||
[source,terminal]
|
||||
----
|
||||
Starting pod/worker-1-debug ...
|
||||
To use host binaries, run `chroot /host`
|
||||
# ...
|
||||
5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
|
||||
6: br-ex1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
|
||||
----
|
||||
|
||||
. Optional: If you use `/var/lib/ovnk/iface_default_hint`, check that the MAC address of `br-ex` matches the MAC address of the primary selected interface:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc debug node/<node_name> -- chroot /host sh -c "ip a | grep -A1 -E 'br-ex|bond0'
|
||||
----
|
||||
+
|
||||
.Example output that shows the primary interface for `br-ex` as `bond0`
|
||||
[source,terminal]
|
||||
----
|
||||
Starting pod/worker-1-debug ...
|
||||
To use host binaries, run `chroot /host`
|
||||
# ...
|
||||
sh-5.1# ip a | grep -A1 -E 'br-ex|bond0'
|
||||
2: bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
|
||||
link/ether fa:16:3e:47:99:98 brd ff:ff:ff:ff:ff:ff
|
||||
--
|
||||
5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
|
||||
link/ether fa:16:3e:47:99:98 brd ff:ff:ff:ff:ff:ff
|
||||
inet 10.xx.xx.xx/21 brd 10.xx.xx.255 scope global dynamic noprefixroute br-ex
|
||||
----
|
||||
@@ -11,6 +11,14 @@ include::modules/nw-how-nw-iface-selected.adoc[leveloffset=+1]
|
||||
|
||||
include::modules/overriding-default-node-ip-selection-logic.adoc[leveloffset=+2]
|
||||
|
||||
// Configuring OVN-Kubernetes to use a secondary OVS bridge
|
||||
include::modules/configuring-ovnk-use-second-ovs-bridge.adoc[leveloffset=+2]
|
||||
|
||||
[role="_additional-resources"]
|
||||
.Additional resources
|
||||
|
||||
* xref:../../networking/ovn_kubernetes_network_provider/configuring-secondary-external-gateway.adoc#configuring-secondary-external-gateway[Configure an external gateway on the default network]
|
||||
|
||||
// Troubleshooting OVS issues
|
||||
include::modules/nw-troubleshoot-ovs.adoc[leveloffset=+1]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user