openshift/openshift-docs
1
0
Fork 0
mirror of https://github.com/openshift/openshift-docs.git synced 2026-02-05 12:46:18 +01:00
Code Issues Releases Activity

OSDOCS-10810: Add additional routing capabilities

Browse Source 
- https://issues.redhat.com/browse/OSDOCS-10810
This commit is contained in:
Jason Boxman
2024-08-21 22:22:01 -04:00
committed by openshift-cherrypick-robot
parent 453e96c271
commit e002e7d99b
18 changed files with 1106 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
_topic_maps/_topic_map.yml
View File

@@ -1761,6 +1761,17 @@ Topics:
File: disabling-bgp-routing
- Name: Migrating FRR-K8s custom resources
File: migrating-frr-k8s-resources
- Name: Route advertisements
Dir: route_advertisements
Topics:
- Name: About route advertisements
File: about-route-advertisements
- Name: Enabling route advertisements
File: enabling-route-advertisements
- Name: Disabling route advertisements
File: disabling-route-advertisements
- Name: Example route advertisements setup
File: example-route-advertisement-setup
- Name: Using PTP hardware
Dir: ptp
Distros: openshift-enterprise,openshift-origin

BIN
images/524-openshift-bgp-ovn-k8s-no-vpn-0325.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 72 KiB

BIN
images/524-openshift-bgp-ovn-k8s-vrf-lite-0325.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 91 KiB

8
modules/nw-operator-cr.adoc
View File

@@ -220,6 +220,14 @@ endif::operator[]
|`object`
|Specify a configuration object for customizing network policy audit logging. If unset, the defaults audit log settings are used.
|`routeAdvertisements`
|`string`
a|Specifies whether to advertise cluster network routes. The default value is `Disabled`.
--
- `Enabled`: Import routes to the cluster network and advertise cluster network routes as configured in `RouteAdvertisements` objects.
- `Disabled`: Do not import routes to the cluster network or advertise cluster network routes.
--
|`gatewayConfig`
|`object`
|Optional: Specify a configuration object for customizing how egress traffic is sent to the node gateway. Valid values are `Shared` and `Local`. The default value is `Shared`. In the default setting, the Open vSwitch (OVS) outputs traffic directly to the node IP interface. In the `Local` setting, it traverses the host network; consequently, it gets applied to the routing table of the host.

32
modules/nw-route-advertisements-disable.adoc Normal file
View File

@@ -0,0 +1,32 @@
// Module included in the following assemblies:
//
// * networking/route_advertisements/disabling-route-advertisements.adoc
:_mod-docs-content-type: PROCEDURE
[id="nw-route-advertisements-disable_{context}"]
= Disabling route advertisements
As a cluster administrator, you can disable additional route advertisements for your cluster.
.Prerequisites
* You have installed the {oc-first}.
* You are logged in to the cluster as a user with the `cluster-admin` role.
* The cluster is installed on compatible infrastructure.
.Procedure
* To disable additional routing support, enter the following command:
+
[source,terminal]
----
$ oc patch network.operator cluster -p '{
"spec": {
"defaultNetwork": {
"ovnKubernetesConfig": {
"routeAdvertisements": "Disabled"
}
}
}
}'
----

33
modules/nw-route-advertisements-enable.adoc Normal file
View File

@@ -0,0 +1,33 @@
// Module included in the following assemblies:
//
// * networking/route_advertisements/enabling-route-advertisements.adoc
:_mod-docs-content-type: PROCEDURE
[id="nw-route-advertisements-enable_{context}"]
= Enabling route advertisements
As a cluster administrator, you can enable additional routing support for your cluster.
.Prerequisites
* You have installed the {oc-first}.
* You are logged in to the cluster as a user with the `cluster-admin` role.
* The cluster is installed on compatible infrastructure.
.Procedure
* To enable a routing provider and additional route advertisements, enter the following command:
+
[source,terminal]
----
$ oc patch Network.operator.openshift.io cluster --type=merge \
-p='{
"spec": {
"additionalRoutingCapabilities": {
"providers": ["FRR"]
},
"defaultNetwork": {
"ovnKubernetesConfig": {
"routeAdvertisements": "Enabled"
}}}}'
----

106
modules/nw-routeadvertisements-about.adoc Normal file
View File

@@ -0,0 +1,106 @@
// Module included in the following assemblies:
//
// * networking/route_advertisements/about-route-advertisements.adoc
////
Terminology -
Cluster network routes: Both pod network routes and/or EgressIP routes
Pod network routes: Routes to pod IPs
EgressIP routes: Routes to EgressIPs
////
:_mod-docs-content-type: CONCEPT
[id="nw-routeadvertisements-about_{context}"]
= Advertise cluster network routes with Border Gateway Protocol
With route advertisements enabled, the OVN-Kubernetes network plugin supports advertising network routes for the default pod network and cluster user-defined (CUDN) networks to the provider network, including EgressIPs, and importing routes from the provider network to the default pod network and CUDNs. From the provider network, IP addresses advertised from the default pod network and CUDNs can be reached directly.
For example, you can import routes to the default pod network so you no longer need to manually configure routes on each node. Previously, you might have been using local gateway mode (`RoutingViaHost=true`) and manually configuring routes on each node to approximate a similar configuration. With route advertisements you can accomplish this seamlessly and you can use shared gateway mode (`RoutingViaHost=false`) as well.
Route reflectors on the provider network are supported and can reduce the number of BGP connections required to advertise routes on large networks.
If you use EgressIPs with route advertisements enabled, the layer 3 provider network is aware of EgressIP failovers. This allows you to locate cluster nodes that host EgressIPs on different layer 2 segments whereas before only the layer 2 provider network was aware so that required all the egress nodes to be on the same layer 2 segment.
[id="supported-platforms_{context}"]
== Supported platforms
Advertising routes with border gateway protocol (BGP) is supported on the following infrastructure types:
- Bare-metal
//- {vmw-full} on-premise
[id="infrastructure-requirements_{context}"]
== Infrastructure requirements
To use route advertisements, you must have configured BGP for your network infrastructure. Outages or misconfigurations of your network infrastructure might cause disruptions to your cluster network.
[id="compatibility-with-other-networking-features_{context}"]
== Compatibility with other networking features
Route advertisements support the following {product-title} Networking features:
Multiple external gateways (MEG)::
MEG is not supported with this feature.
EgressIPs::
--
Supports the use and advertisement of EgressIPs. The node where an egress IP address resides advertises the EgressIP. An egress IP address must be on the same layer 2 network subnet as the egress node. The following limitations apply:
- Advertising EgressIPs from a user-defined network (CUDN) operating in layer 2 mode are not supported.
- Advertising EgressIPs for a network that has both egress IP addresses assigned to the primary network interface and egress IP addresses assigned to additional network interfaces is impractical. All EgressIPs are advertised on all of the BGP sessions of the selected FRRConfiguration instances, regardless of whether these sessions are established over the same interface that the EgressIP is assigned to or not, potentially leading to unwanted advertisements.
--
Services::
Works with the MetalLB Operator to advertise services to the provider network.
Egress service::
Full support.
Egress firewall::
Full support.
Egress QoS::
Full support.
Network policies::
Full support.
Direct pod ingress::
Full support for the default cluster network and cluster user-defined (CUDN) networks.
[id="considerations-for-use-with-the-metallb-operator_{context}"]
== Considerations for use with the MetalLB Operator
The MetalLB Operator is installed as an add-on to the cluster. Deployment of the MetalLB Operator automatically enables FRR-K8s as an additional routing capability provider. This feature and the MetalLB Operator use the same FRR-K8s deployment.
[id="considerations-for-naming-cluster-user-defined-networks_{context}"]
== Considerations for naming cluster user-defined networks (CUDNs)
When referencing a VRF device in a `FRRConfiguration` CR, the VRF name is the same as the CUDN name for VRF names that are less than or equal to 15 characters. It is recommended to use a VRF name no longer than 15 characters so that the VRF name can be inferred from the CUDN name.
[id="bgp-routing-custom-resources_{context}"]
== BGP routing custom resources
The following custom resources (CRs) are used to configure route advertisements with BGP:
`RouteAdvertisements`::
This CR defines the advertisements for the BGP routing. From this CR, the OVN-Kubernetes controller generates a `FRRConfiguration` object that configures the FRR daemon to advertise cluster network routes. This CR is cluster scoped.
`FRRConfiguration`::
This CR is used to define BGP peers and to configure route imports from the provider network into the cluster network. Before applying `RouteAdvertisements` objects, at least one FRRConfiguration object must be initially defined to configure the BGP peers. This CR is namespaced.
[id="ovn-kubernetes-controller-generation-of-frrconfiguration-objects_{context}"]
== OVN-Kubernetes controller generation of `FRRConfiguration` objects
An `FRRConfiguration` object is generated for each network and node selected by a `RouteAdvertisements` CR with the appropriate advertised prefixes that apply to each node. The OVN-Kubernetes controller checks whether the `RouteAdvertisements`-CR-selected nodes are a subset of the nodes that are selected by the `RouteAdvertisements`-CR-selected FRR configurations.
Any filtering or selection of prefixes to receive are not considered in `FRRConfiguration` objects that are generated from the `RouteAdvertisement` CRs. Configure any prefixes to receive on other `FRRConfiguration` objects. OVN-Kubernetes imports routes from the VRF into the appropriate network.
[id="cluster-network-operator_{context}"]
== Cluster Network Operator configuration
The Cluster Network Operator (CNO) API exposes several fields to configure route advertisements:
- `spec.additionalRoutingCapabilities.providers`: Specifies an additional routing provider, which is required to advertise routes. The only supported value is `FRR`, which enables deployment of the FRR-K8S daemon for the cluster. When enabled, the FRR-K8S daemon is deployed on all nodes.
- `spec.defaultNetwork.ovnKubernetesConfig.routeAdvertisements`: Enables route advertisements for the default cluster network and CUDN networks. The `spec.additionalRoutingCapabilities` field must be set to `FRR` to enable this feature.

323
modules/nw-routeadvertisements-example.adoc Normal file
View File

@@ -0,0 +1,323 @@
// Module included in the following assemblies:
//
// * networking/route_advertisements/about-bgp-routing.adoc
:_mod-docs-content-type: CONCEPT
[id="nw-bgp-examples_{context}"]
= Examples advertising pod IP addresses with BGP
The following examples describe several configurations for advertising pod IP addresses and EgressIPs with Border Gateway Protocol (BGP). The external network border router has the `172.18.0.5` IP address. These configures assume that you have configured an external route reflector that can relay routes to all nodes on the cluster network.
////
A route advertisements object selects existing FRRConfiguration objects, and along with the selected nodes, OVN-Kubernetes controller generates further FRRConfiguration objects with the appropriate advertised prefixes that apply to each node. The OVN-Kubernetes controller checks whether the `RouteAdvertisements`-CR-selected nodes are a subset of the nodes that are selected by the `RouteAdvertisements`-CR-selected FRR configurations.
The FRRConfiguration objects generated by OVN-Kubernetes controller do not configure routes to be imported from the provider network. Any filtering or selection of prefixes to import are configured by a cluster administrator in different FRRConfiguration objects that those managed by OVN-Kubernetes controller. Imported routes will take effect for a cluster network as long as they are imported to that network VRF or a VRF that the network advertises to as specified by the targetVRF field of a routeAdvertisements object applying to it.
////
[id="advertising-the-default-cluster-network_{context}"]
== Advertising the default cluster network
In this scenario, the default cluster network is exposed to the external network so that pod IP addresses and EgressIPs are advertised to the provider network.
This scenario relies upon the following `FRRConfiguration` object:
.`FRRConfiguration` CR
[source,yaml]
----
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
name: default
spec:
advertisements:
- PodNetwork
- EgressIP
networkSelectors:
- networkSelectionType: DefaultNetwork
frrConfigurationSelector:
matchLabels:
routeAdvertisements: receive-all
nodeSelector: {}
----
When the OVN-Kubernetes controller sees this `RouteAdvertisements` CR, it generates further `FRRConfiguration` objects based on the selected ones that configure the FRR daemon to advertise the routes for the default cluster network.
.An example of a `FRRConfiguration` CR generated by OVN-Kubernetes
[source,yaml]
----
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: ovnk-generated-abcdef
namespace: openshift-frr-k8s
spec:
bgp:
routers:
- asn: 64512
neighbors:
- address: 172.18.0.5
asn: 64512
toReceive:
allowed:
mode: filtered
toAdvertise:
allowed:
prefixes:
- <default_network_host_subnet>
prefixes:
- <default_network_host_subnet>
nodeSelector:
matchLabels:
kubernetes.io/hostname: ovn-worker
----
In the example generated `FRRConfiguration` object, `<default_network_host_subnet>` is the subnet of the default cluster network that is advertised to the provider network.
[id="advertising-pod-ips-from-a-user-defined-network-over-bgp_{context}"]
== Advertising pod IPs from a cluster user-defined network over BGP
In this scenario, the blue cluster user-defined network (CUDN) is exposed to the external network so that the network's pod IP addresses and EgressIPs are advertised to the provider network.
This scenario relies upon the following `FRRConfiguration` object:
.`FRRConfiguration` CR
[source,yaml]
----
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: receive-all
namespace: openshift-frr-k8s
labels:
routeAdvertisements: receive-all
spec:
bgp:
routers:
- asn: 64512
neighbors:
- address: 172.18.0.5
asn: 64512
disableMP: true
toReceive:
allowed:
mode: all
----
With this `FRRConfiguration` object, routes will be imported from neighbor `172.18.0.5` into the default VRF and are available to the default cluster network.
The CUDNs are advertised over the default VRF as illustrated in the following diagram:
image::524-openshift-bgp-ovn-k8s-no-vpn-0325.png[Advertising pod IPs from a cluster user-defined network over BGP]
Red CUDN::
- A VRF named `red` associated with a CUDN named `red`
- A subnet of `10.0.0.0/24`
Blue CUDN::
- A VRF named `blue` associated with a CUDN named `blue`
- A subnet of `10.0.1.0/24`
In this configuration, two separate CUDNs are defined. The red network covers the `10.0.0.0/24` subnet and the blue network covers the `10.0.1.0/24` subnet. The red and blue networks are labeled as `export: true`.
The following `RouteAdvertisements` CR describes the configuration for the red and blue tenants:
.`RouteAdvertisements` CR for the red and blue tenants
[source,yaml]
----
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
name: advertise-cudns
spec:
advertisements:
- PodNetwork
- EgressIP
networkSelectors:
- networkSelectionType: ClusterUserDefinedNetworks
clusterUserDefinedNetworkSelector:
networkSelector:
matchLabels:
export: "true"
frrConfigurationSelector:
matchLabels:
routeAdvertisements: receive-all
nodeSelector: {}
----
When the OVN-Kubernetes controller sees this `RouteAdvertisements` CR, it generates generates further `FRRConfiguration` objects based on the selected ones that configure the FRR daemon to advertise the routes. The following example is of one such configuration object, with the number of `FRRConfiguration` objects created depending on the node and networks selected.
.An example of a `FRRConfiguration` CR generated by OVN-Kubernetes
[source,yaml]
----
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: ovnk-generated-abcdef
namespace: openshift-frr-k8s
spec:
bgp:
routers:
- asn: 64512
vrf: blue
imports:
- vrf: default
- asn: 64512
neighbors:
- address: 172.18.0.5
asn: 64512
toReceive:
allowed:
mode: filtered
toAdvertise:
allowed:
prefixes:
- 10.0.1.0/24
prefixes:
- 10.0.1.0/24
imports:
- vrf: blue
nodeSelector:
matchLabels:
kubernetes.io/hostname: ovn-worker
----
The generated `FRRConfiguration` object configures the subnet `10.0.1.0/24`, which belongs to network blue, to be imported into the default VRF and advertised to the `172.18.0.5` neighbor. An `FRRConfiguration` object is generated for each network and nodes selected by a `RouteAdvertisements` CR with the appropriate prefixes that apply to each node.
When the `targetVRF` field is omitted, the routes are leaked and advertised over the default VRF. Additionally, routes that were imported to the default VRF after the definition of the initial FRRConfiguration object are also imported into the blue VRF.
[id="advertising-pod-ips-from-a-user-defined-network-over-bgp-with-vpn_{context}"]
== Advertising pod IPs from a cluster user-defined network over BGP with VPN
In this scenario, a VLAN interface is attached to the VRF device associated with the blue network. This setup provides a _VRF lite_ design, where FRR-K8S is used to advertise the blue network only over the corresponding BGP session on the blue network VRF/VLAN link to the next hop Provide Edge (PE) router. The red tenant uses the same configuration. The blue and red networks are labeled as `export: true`.
[IMPORTANT]
====
This scenario does not support the use of EgressIPs.
====
The following diagram illustrates this configuration:
image::524-openshift-bgp-ovn-k8s-vrf-lite-0325.png[Advertising pod IPs from a cluster user-defined network over BGP with VPN]
Red CUDN::
- A VRF named `red` associated with a CUDN named `red`
- A VLAN interface attached to the VRF device and connected to the external PE router
- An assigned subnet of `10.0.2.0/24`
Blue CUDN::
- A VRF named `blue` associated with a CUDN named `blue`
- A VLAN interface attached to the VRF device and connected to the external PE router
- An assigned subnet of `10.0.1.0/24`
[NOTE]
====
This approach is available only when you use OVN-Kubernetes in local gateway mode by setting `routingViaHost=true`.
====
In the following configuration, an additional `FRRConfiguration` CR configures peering with the PE router on the blue and red VLANs:
.`FRRConfiguration` CR manually configured for BGP VPN setup
[source,yaml]
----
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: vpn-blue-red
namespace: openshift-frr-k8s
labels:
routeAdvertisements: vpn-blue-red
spec:
bgp:
routers:
- asn: 64512
vrf: blue
neighbors:
- address: 182.18.0.5
asn: 64512
toReceive:
allowed:
mode: filtered
- asn: 64512
vrf: red
neighbors:
- address: 192.18.0.5
asn: 64512
toReceive:
allowed:
mode: filtered
----
The following `RouteAdvertisements` CR describes the configuration for the blue and red tenants:
.`RouteAdvertisements` CR for the blue and red tenants
[source,yaml]
----
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
name: advertise-vrf-lite
spec:
targetVRF: auto
advertisements:
- "PodNetwork"
nodeSelector: {}
frrConfigurationSelector:
matchLabels:
routeAdvertisements: vpn-blue-red
networkSelectors:
- networkSelectionType: ClusterUserDefinedNetworks
clusterUserDefinedNetworkSelector:
networkSelector:
matchLabels:
export: "true"
----
In the `RouteAdvertisements` CR, the `targetVRF` is set to `auto` so that advertisements occur within the VRF device that corresponds to the individual networks that are selected. In this scenario, the pod subnet for blue is advertised over the blue VRF device, and the pod subnet for red is advertised over the red VRF device. Additionally, each BGP session imports routes to only the corresponding CUDN VRF as defined by the initial `FRRConfiguration` object.
When the OVN-Kubernetes controller sees this `RouteAdvertisements` CR, it generates further `FRRConfiguration` objects based on the selected ones that configure the FRR daemon to advertise the routes for the blue and red tenants.
.`FRRConfiguration` CR generated by OVN-Kubernetes for blue and red tenants
[source,yaml]
----
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: ovnk-generated-abcde
namespace: openshift-frr-k8s
spec:
bgp:
routers:
- asn: 64512
neighbors:
- address: 182.18.0.5
asn: 64512
toReceive:
allowed:
mode: filtered
toAdvertise:
allowed:
prefixes:
- 10.0.1.0/24
vrf: blue
prefixes:
- 10.0.1.0/24
- asn: 64512
neighbors:
- address: 192.18.0.5
asn: 64512
toReceive:
allowed:
mode: filtered
toAdvertise:
allowed:
prefixes:
- 10.0.2.0/24
vrf: red
prefixes:
- 10.0.2.0/24
nodeSelector:
matchLabels:
kubernetes.io/hostname: ovn-worker
----
In this scenario, any filtering or selection of routes to receive must be done in the `FRRConfiguration` CR that defines peering relationships.

42
modules/nw-routeadvertisements-object.adoc Normal file
View File

@@ -0,0 +1,42 @@
// Module included in the following assemblies:
//
// * networking/route_advertisements/about-route-advertisements.adoc
:_mod-docs-content-type: CONCEPT
[id="nw-bgp-routeadvertisements-object_{context}"]
= RouteAdvertisements object configuration
You can define an `RouteAdvertisements` object, which is cluster scoped, with the following properties.
The fields for the `RouteAdvertisements` custom resource (CR) are described in the following table:
.`RouteAdvertisements` object
[cols=".^2,.^2,.^6",options="header"]
|====
|Field|Type|Description
|`metadata.name`
|`string`
|Specifies the name of the `RouteAdvertisements` object.
|`advertisements`
|`array`
|Specifies an array that can contain a list of different types of networks to advertise. Supports only the `"PodNetwork"` and `"EgressIP"` values.
|`frrConfigurationSelector`
|`object`
|Determines which `FRRConfiguration` CR the OVN-Kubernetes driven `FRRConfiguration` CR is based on.
|`networkSelector`
|`object`
|Specifies which networks to advertise among default cluster network and cluster user defined networks (CUDNs).
|`nodeSelector`
|`object`
|
Limits the advertisements to selected nodes. When `advertisements="PodNetwork"` is selected, all nodes must be selected. When `advertisements="EgressIP"` is selected, only the egress IP addresses assigned to the selected nodes are advertised.
|`targetVRF`
|`string`
|Determines which router to advertise the routes in. Routes are advertised on the routers associated with this virtual routing and forwarding (VRF) target, as specified on the selected `FRRConfiguration` CR. When omitted, the default VRF is used as the target. When specified as `auto`, a VRF with the same name as the network name is used as the target.
|====

491
modules/nw-routeadvertisements-setup.adoc Normal file
View File

@@ -0,0 +1,491 @@
// Module included in the following assemblies:
//
// * networking/route_advertisements/example-route-advertisement-setup.adoc
:_mod-docs-content-type: PROCEDURE
[id="nw-route-advertisements-setup_{context}"]
= Sample route advertisements setup
As a cluster administrator, you can enable Border Gateway Protocol (BGP) routing support for your cluster. This configuration is intended as a sample that demonstrates how to configure route advertisements. The configuration uses route reflection rather than a full mesh setup.
[NOTE]
====
BGP routing is supported only on bare-metal infrastructure.
====
.Prerequisites
* You installed the {oc-first}.
* You are logged in to the cluster as a user with `cluster-admin` privileges.
* The cluster is installed on bare-metal infrastructure.
* You have a bare-metal system with access to the cluster where you plan to run the FRR daemon container.
.Procedure
. Confirm that the `RouteAdvertisements` feature gate is enabled by running the following command:
+
[source,terminal]
----
$ oc get featuregate -oyaml | grep -i routeadvertisement
----
+
.Example output
[source,yaml]
----
- name: RouteAdvertisements
----
. Configure the Cluster Network Operator (CNO) by running the following command:
+
[source,terminal]
----
$ oc patch Network.operator.openshift.io cluster --type=merge \
-p='
{"spec":{
"additionalRoutingCapabilities": {
"providers": ["FRR"]},
"defaultNetwork":{"ovnKubernetesConfig"{
"routeAdvertisements":"Enabled"
}}}}'
----
+
It might take a few minutes for the CNO to restart all nodes.
. Get the IP addresses of the nodes by running the following command:
+
[source,terminal]
----
$ oc get node -owide
----
+
.Example output
[source,text]
----
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master-0 Ready control-plane,master 27h v1.31.3 192.168.111.20 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
master-1 Ready control-plane,master 27h v1.31.3 192.168.111.21 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
master-2 Ready control-plane,master 27h v1.31.3 192.168.111.22 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
worker-0 Ready worker 27h v1.31.3 192.168.111.23 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
worker-1 Ready worker 27h v1.31.3 192.168.111.24 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
worker-2 Ready worker 27h v1.31.3 192.168.111.25 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
----
. Get the default pod network of each node by running the following command:
+
[source,terminal]
----
$ oc get node <node_name> -o=jsonpath={.metadata.annotations.k8s\\.ovn\\.org/node-subnets}
----
+
.Example output
[source,text]
----
{"default":["10.129.0.0/23"],"ns1.udn-network-primary-layer3":["10.150.6.0/24"]}
----
. On the bare-metal hypervisor, get the IP address for the external FRR container to use by running the following command:
+
[source,terminal]
----
$ ip -j -d route get <a cluster node's IP> | jq -r '.[] | .dev' | xargs ip -d -j address show | jq -r '.[] | .addr_info[0].local'
----
. Create a `frr.conf` file for FRR that includes each node's IP address, as shown in the following example:
+
.Example `frr.conf` configuration file
[source,text]
----
router bgp 64512
no bgp default ipv4-unicast
no bgp default ipv6-unicast
no bgp network import-check
neighbor 192.168.111.20 remote-as 64512
neighbor 192.168.111.20 route-reflector-client
neighbor 192.168.111.21 remote-as 64512
neighbor 192.168.111.21 route-reflector-client
neighbor 192.168.111.22 remote-as 64512
neighbor 192.168.111.22 route-reflector-client
neighbor 192.168.111.40 remote-as 64512
neighbor 192.168.111.40 route-reflector-client
neighbor 192.168.111.47 remote-as 64512
neighbor 192.168.111.47 route-reflector-client
neighbor 192.168.111.23 remote-as 64512
neighbor 192.168.111.23 route-reflector-client
neighbor 192.168.111.24 remote-as 64512
neighbor 192.168.111.24 route-reflector-client
neighbor 192.168.111.25 remote-as 64512
neighbor 192.168.111.25 route-reflector-client
address-family ipv4 unicast
network 192.168.1.0/24
network 192.169.1.1/32
exit-address-family
address-family ipv4 unicast
neighbor 192.168.111.20 activate
neighbor 192.168.111.20 next-hop-self
neighbor 192.168.111.21 activate
neighbor 192.168.111.21 next-hop-self
neighbor 192.168.111.22 activate
neighbor 192.168.111.22 next-hop-self
neighbor 192.168.111.40 activate
neighbor 192.168.111.40 next-hop-self
neighbor 192.168.111.47 activate
neighbor 192.168.111.47 next-hop-self
neighbor 192.168.111.23 activate
neighbor 192.168.111.23 next-hop-self
neighbor 192.168.111.24 activate
neighbor 192.168.111.24 next-hop-self
neighbor 192.168.111.25 activate
neighbor 192.168.111.25 next-hop-self
exit-address-family
neighbor remote-as 64512
neighbor route-reflector-client
address-family ipv6 unicast
network 2001:db8::/128
exit-address-family
address-family ipv6 unicast
neighbor activate
neighbor next-hop-self
exit-address-family
----
. Create a file named `daemons` that includes the following content:
+
.Example `daemons` configuration file
[source,text]
----
# This file tells the frr package which daemons to start.
#
# Sample configurations for these daemons can be found in
# /usr/share/doc/frr/examples/.
#
# ATTENTION:
#
# When activating a daemon for the first time, a config file, even if it is
# empty, has to be present *and* be owned by the user and group "frr", else
# the daemon will not be started by /etc/init.d/frr. The permissions should
# be u=rw,g=r,o=.
# When using "vtysh" such a config file is also needed. It should be owned by
# group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
#
# The watchfrr and zebra daemons are always started.
#
bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=yes
fabricd=no
vrrpd=no
#
# If this option is set the /etc/init.d/frr script automatically loads
# the config via "vtysh -b" when the servers are started.
# Check /etc/pam.d/frr if you intend to use "vtysh"!
#
vtysh_enable=yes
zebra_options=" -A 127.0.0.1 -s 90000000"
bgpd_options=" -A 127.0.0.1"
ospfd_options=" -A 127.0.0.1"
ospf6d_options=" -A ::1"
ripd_options=" -A 127.0.0.1"
ripngd_options=" -A ::1"
isisd_options=" -A 127.0.0.1"
pimd_options=" -A 127.0.0.1"
ldpd_options=" -A 127.0.0.1"
nhrpd_options=" -A 127.0.0.1"
eigrpd_options=" -A 127.0.0.1"
babeld_options=" -A 127.0.0.1"
sharpd_options=" -A 127.0.0.1"
pbrd_options=" -A 127.0.0.1"
staticd_options="-A 127.0.0.1"
bfdd_options=" -A 127.0.0.1"
fabricd_options="-A 127.0.0.1"
vrrpd_options=" -A 127.0.0.1"
# configuration profile
#
#frr_profile="traditional"
#frr_profile="datacenter"
#
# This is the maximum number of FD's that will be available.
# Upon startup this is read by the control files and ulimit
# is called. Uncomment and use a reasonable value for your
# setup if you are expecting a large number of peers in
# say BGP.
#MAX_FDS=1024
# The list of daemons to watch is automatically generated by the init script.
#watchfrr_options=""
# for debugging purposes, you can specify a "wrap" command to start instead
# of starting the daemon directly, e.g. to use valgrind on ospfd:
# ospfd_wrap="/usr/bin/valgrind"
# or you can use "all_wrap" for all daemons, e.g. to use perf record:
# all_wrap="/usr/bin/perf record --call-graph -"
# the normal daemon command is added to this at the end.
----
. Save both the `frr.conf` and `daemons` files in the same directory, such as `/tmp/frr`.
. Create an external FRR container by running the following command:
+
[source,terminal]
----
$ sudo podman run -d --privileged --network host --rm --ulimit core=-1 --name frr --volume /tmp/frr:/etc/frr quay.io/frrouting/frr:9.1.0
----
. Create the following `FRRConfiguration` and `RouteAdvertisements` configurations:
.. Create a `receive_all.yaml` file that includes the following content:
+
.Example `receive_all.yaml` configuration file
[source,yaml]
----
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: receive-all
namespace: openshift-frr-k8s
spec:
bgp:
routers:
- asn: 64512
neighbors:
- address: 192.168.111.1
asn: 64512
toReceive:
allowed:
mode: all
----
.. Create a `ra.yaml` file that includes the following content:
+
.Example `ra.yaml` configuration file
[source,yaml]
----
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
name: default
spec:
nodeSelector: {}
frrConfigurationSelector: {}
networkSelectors:
- networkSelectionType: DefaultNetwork
advertisements:
- "PodNetwork"
- "EgressIP"
----
. Apply the `receive_all.yaml` and `ra.yaml` files by running the following command:
+
[source,terminal]
----
$ for f in receive_all.yaml ra.yaml; do oc apply -f $f; done
----
.Verification
. Verify that the configurations were applied:
.. Verify that the `FRRConfiguration` configurations were created by running the following command:
+
[source,terminal]
----
$ oc get frrconfiguration -A
----
+
.Example output
[source,text]
----
NAMESPACE NAME AGE
openshift-frr-k8s ovnk-generated-6lmfb 4h47m
openshift-frr-k8s ovnk-generated-bhmnm 4h47m
openshift-frr-k8s ovnk-generated-d2rf5 4h47m
openshift-frr-k8s ovnk-generated-f958l 4h47m
openshift-frr-k8s ovnk-generated-gmsmw 4h47m
openshift-frr-k8s ovnk-generated-kmnqg 4h47m
openshift-frr-k8s ovnk-generated-wpvgb 4h47m
openshift-frr-k8s ovnk-generated-xq7v6 4h47m
openshift-frr-k8s receive-all 4h47m
----
.. Verify that the `RouteAdvertisements` configurations were created by running the following command:
+
[source,terminal]
----
$ oc get ra -A
----
+
.Example output
[source,text]
----
NAME STATUS
default Accepted
----
. Get the external FRR container ID by running the following command:
+
[source,terminal]
----
$ sudo podman ps | grep frr
----
+
.Example output
[source,text]
----
22cfc713890e quay.io/frrouting/frr:9.1.0 /usr/lib/frr/dock... 5 hours ago Up 5 hours ago frr
----
. Use the container ID that you obtained in the previous step to check the BGP neighbor and routes in the external FRR container's `vtysh` session. Run the following command:
+
[source,terminal]
----
$ sudo podman exec -it <container_id> vtysh -c "show ip bgp"
----
+
.Example output
[source,text]
----
BGP table version is 10, local router ID is 192.168.111.1, vrf id 0
Default local pref 100, local AS 64512
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*>i10.128.0.0/23 192.168.111.22 0 100 0 i
*>i10.128.2.0/23 192.168.111.23 0 100 0 i
*>i10.129.0.0/23 192.168.111.20 0 100 0 i
*>i10.129.2.0/23 192.168.111.24 0 100 0 i
*>i10.130.0.0/23 192.168.111.21 0 100 0 i
*>i10.130.2.0/23 192.168.111.40 0 100 0 i
*>i10.131.0.0/23 192.168.111.25 0 100 0 i
*>i10.131.2.0/23 192.168.111.47 0 100 0 i
*> 192.168.1.0/24 0.0.0.0 0 32768 i
*> 192.169.1.1/32 0.0.0.0 0 32768 i
----
. Find the `frr-k8s` pod for each cluster node by running the following command:
+
[source,terminal]
----
$ oc -n openshift-frr-k8s get pod -owide
----
+
.Example output
[source,text]
----
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
frr-k8s-86wmq 6/6 Running 0 25h 192.168.111.20 master-0 <none> <none>
frr-k8s-h2wl6 6/6 Running 0 25h 192.168.111.21 master-1 <none> <none>
frr-k8s-jlbgs 6/6 Running 0 25h 192.168.111.40 node1.example.com <none> <none>
frr-k8s-qc6l5 6/6 Running 0 25h 192.168.111.25 worker-2 <none> <none>
frr-k8s-qtxdc 6/6 Running 0 25h 192.168.111.47 node2.example.com <none> <none>
frr-k8s-s5bxh 6/6 Running 0 25h 192.168.111.24 worker-1 <none> <none>
frr-k8s-szgj9 6/6 Running 0 25h 192.168.111.22 master-2 <none> <none>
frr-k8s-webhook-server-6cd8b8d769-kmctw 1/1 Running 0 25h 10.131.2.9 node3.example.com <none> <none>
frr-k8s-zwmgh 6/6 Running 0 25h 192.168.111.23 worker-0 <none> <none>
----
. From the {product-title} cluster, check BGP routes on the cluster node's `frr-k8s` pod in the FRR container by running the following command:
+
[source,terminal]
----
$ oc -n openshift-frr-k8s -c frr rsh frr-k8s-86wmq
----
. Check the IP routes from the cluster node by running the following command:
+
[source,terminal]
----
sh-5.1# vtysh
----
+
.Example output
[source,text]
----
Hello, this is FRRouting (version 8.5.3).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
----
. Check the IP routes by running the following command:
+
[source,terminal]
----
worker-2# show ip bgp
----
+
.Example output
[source,text]
----
BGP table version is 10, local router ID is 192.168.111.25, vrf id 0
Default local pref 100, local AS 64512
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*>i10.128.0.0/23 192.168.111.22 0 100 0 i
*>i10.128.2.0/23 192.168.111.23 0 100 0 i
*>i10.129.0.0/23 192.168.111.20 0 100 0 i
*>i10.129.2.0/23 192.168.111.24 0 100 0 i
*>i10.130.0.0/23 192.168.111.21 0 100 0 i
*>i10.130.2.0/23 192.168.111.40 0 100 0 i
*> 10.131.0.0/23 0.0.0.0 0 32768 i
*>i10.131.2.0/23 192.168.111.47 0 100 0 i
*>i192.168.1.0/24 192.168.111.1 0 100 0 i
*>i192.169.1.1/32 192.168.111.1 0 100 0 i
Displayed 10 routes and 10 total paths
----
. From the {product-title} cluster, debug the node by running the following command:
+
[source,terminal]
----
$ oc debug node/<node_name>
----
+
.Example output
[source,text]
----
Temporary namespace openshift-debug-lbtgh is created for debugging node...
Starting pod/worker-2-debug-zrg4v ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.111.25
If you don't see a command prompt, try pressing enter.
----
. Confirm that the BGP routes are being advertised by running the following command:
+
[source,terminal]
----
sh-5.1# ip route show | grep bgp
----
+
.Example output
[source,text]
----
10.128.0.0/23 nhid 268 via 192.168.111.22 dev br-ex proto bgp metric 20
10.128.2.0/23 nhid 259 via 192.168.111.23 dev br-ex proto bgp metric 20
10.129.0.0/23 nhid 260 via 192.168.111.20 dev br-ex proto bgp metric 20
10.129.2.0/23 nhid 261 via 192.168.111.24 dev br-ex proto bgp metric 20
10.130.0.0/23 nhid 266 via 192.168.111.21 dev br-ex proto bgp metric 20
10.130.2.0/23 nhid 262 via 192.168.111.40 dev br-ex proto bgp metric 20
10.131.2.0/23 nhid 263 via 192.168.111.47 dev br-ex proto bgp metric 20
192.168.1.0/24 nhid 264 via 192.168.111.1 dev br-ex proto bgp metric 20
192.169.1.1 nhid 264 via 192.168.111.1 dev br-ex proto bgp metric 20
----

1
networking/advanced_networking/route_advertisements/_attributes Symbolic link
View File

@@ -0,0 +1 @@
../../_attributes

23
networking/advanced_networking/route_advertisements/about-route-advertisements.adoc Normal file
View File

@@ -0,0 +1,23 @@
:_mod-docs-content-type: ASSEMBLY
[id="about-route-advertisements"]
= About route advertisements
include::_attributes/common-attributes.adoc[]
:context: about-route-advertisements
toc::[]
This feature provides route advertisement capabilities for the OVN-Kubernetes network plugin. A Border Gateway Router (BGP) provider is required. For more information, see xref:../../../networking/advanced_networking/bgp_routing/about-bgp-routing.adoc#about-bgp-routing[About BGP routing].
include::modules/nw-routeadvertisements-about.adoc[leveloffset=+1]
include::modules/nw-routeadvertisements-object.adoc[leveloffset=+1]
include::modules/nw-routeadvertisements-example.adoc[leveloffset=+1]
[role="_additional-resources"]
[id="additional-resources_about-route-advertisements"]
== Additional resources
- xref:../../../networking/ingress_load_balancing/metallb/metallb-frr-k8s.adoc#nw-metallb-frrconfiguration-crd_configure-metallb-frr-k8s[Configuring the FRRConfiguration CRD]
- link:https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/assembly_starting-a-service-within-an-isolated-vrf-network_configuring-and-managing-networking[Starting a service within an isolated VRF network]
- link:https://docs.frrouting.org/en/latest/bgp.html[FRRouting User Guide: BGP]

11
networking/advanced_networking/route_advertisements/disabling-route-advertisements.adoc Normal file
View File

@@ -0,0 +1,11 @@
:_mod-docs-content-type: ASSEMBLY
[id="disabling-route-advertisements"]
= Disabling route advertisements
include::_attributes/common-attributes.adoc[]
:context: disabling-route-advertisements
toc::[]
As a cluster administrator, you can disable additional route advertisements for your cluster.
include::modules/nw-route-advertisements-disable.adoc[leveloffset=+1]

11
networking/advanced_networking/route_advertisements/enabling-route-advertisements.adoc Normal file
View File

@@ -0,0 +1,11 @@
:_mod-docs-content-type: ASSEMBLY
[id="enabling-route-advertisements"]
= Enabling route advertisements
include::_attributes/common-attributes.adoc[]
:context: enabling-route-advertisements
toc::[]
As a cluster administrator, you can configure additional route advertisements for your cluster. You must use the OVN-Kubernetes network plugin.
include::modules/nw-route-advertisements-enable.adoc[leveloffset=+1]

11
networking/advanced_networking/route_advertisements/example-route-advertisement-setup.adoc Normal file
View File

@@ -0,0 +1,11 @@
:_mod-docs-content-type: ASSEMBLY
[id="example-route-advertisements-setup"]
= Example route advertisements setup
include::_attributes/common-attributes.adoc[]
:context: example-route-advertisements-setup
toc::[]
As a cluster administrator, you can configure the following example route advertisements setup for your cluster. This configuration is intended as a sample that demonstrates how to configure route advertisements.
include::modules/nw-routeadvertisements-setup.adoc[leveloffset=+1]

1
networking/advanced_networking/route_advertisements/images Symbolic link
View File

@@ -0,0 +1 @@
../../images

1
networking/advanced_networking/route_advertisements/modules Symbolic link
View File

@@ -0,0 +1 @@
../../modules

1
networking/advanced_networking/route_advertisements/snippets Symbolic link
View File

@@ -0,0 +1 @@
../../snippets