openshift/openshift-docs
1
0
Fork 0
mirror of https://github.com/openshift/openshift-docs.git synced 2026-02-05 12:46:18 +01:00
Code Issues Releases Activity
Files
enterprise-4.21
openshift-docs/modules/nw-routeadvertisements-setup.adoc

492 lines
17 KiB
Plaintext
Raw Permalink Blame History

// Module included in the following assemblies:
//
// * networking/route_advertisements/example-route-advertisement-setup.adoc
:_mod-docs-content-type: PROCEDURE
[id="nw-route-advertisements-setup_{context}"]
= Sample route advertisements setup
As a cluster administrator, you can enable Border Gateway Protocol (BGP) routing support for your cluster. This configuration is intended as a sample that demonstrates how to configure route advertisements. The configuration uses route reflection rather than a full mesh setup.
[NOTE]
====
BGP routing is supported only on bare-metal infrastructure.
====
.Prerequisites
* You installed the {oc-first}.
* You are logged in to the cluster as a user with `cluster-admin` privileges.
* The cluster is installed on bare-metal infrastructure.
* You have a bare-metal system with access to the cluster where you plan to run the FRR daemon container.
.Procedure
. Confirm that the `RouteAdvertisements` feature gate is enabled by running the following command:
+
[source,terminal]
----
$ oc get featuregate -oyaml | grep -i routeadvertisement
----
+
.Example output
[source,yaml]
----
- name: RouteAdvertisements
----
. Configure the Cluster Network Operator (CNO) by running the following command:
+
[source,terminal]
----
$ oc patch Network.operator.openshift.io cluster --type=merge \
-p='
{"spec":{
"additionalRoutingCapabilities": {
"providers": ["FRR"]},
"defaultNetwork":{"ovnKubernetesConfig"{
"routeAdvertisements":"Enabled"
}}}}'
----
+
It might take a few minutes for the CNO to restart all nodes.
. Get the IP addresses of the nodes by running the following command:
+
[source,terminal]
----
$ oc get node -owide
----
+
.Example output
[source,text]
----
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master-0 Ready control-plane,master 27h v1.31.3 192.168.111.20 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
master-1 Ready control-plane,master 27h v1.31.3 192.168.111.21 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
master-2 Ready control-plane,master 27h v1.31.3 192.168.111.22 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
worker-0 Ready worker 27h v1.31.3 192.168.111.23 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
worker-1 Ready worker 27h v1.31.3 192.168.111.24 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
worker-2 Ready worker 27h v1.31.3 192.168.111.25 <none> Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
----
. Get the default pod network of each node by running the following command:
+
[source,terminal]
----
$ oc get node <node_name> -o=jsonpath={.metadata.annotations.k8s\\.ovn\\.org/node-subnets}
----
+
.Example output
[source,text]
----
{"default":["10.129.0.0/23"],"ns1.udn-network-primary-layer3":["10.150.6.0/24"]}
----
. On the bare-metal hypervisor, get the IP address for the external FRR container to use by running the following command:
+
[source,terminal]
----
$ ip -j -d route get <a cluster node's IP> | jq -r '.[] | .dev' | xargs ip -d -j address show | jq -r '.[] | .addr_info[0].local'
----
. Create a `frr.conf` file for FRR that includes each node's IP address, as shown in the following example:
+
.Example `frr.conf` configuration file
[source,text]
----
router bgp 64512
no bgp default ipv4-unicast
no bgp default ipv6-unicast
no bgp network import-check
neighbor 192.168.111.20 remote-as 64512
neighbor 192.168.111.20 route-reflector-client
neighbor 192.168.111.21 remote-as 64512
neighbor 192.168.111.21 route-reflector-client
neighbor 192.168.111.22 remote-as 64512
neighbor 192.168.111.22 route-reflector-client
neighbor 192.168.111.40 remote-as 64512
neighbor 192.168.111.40 route-reflector-client
neighbor 192.168.111.47 remote-as 64512
neighbor 192.168.111.47 route-reflector-client
neighbor 192.168.111.23 remote-as 64512
neighbor 192.168.111.23 route-reflector-client
neighbor 192.168.111.24 remote-as 64512
neighbor 192.168.111.24 route-reflector-client
neighbor 192.168.111.25 remote-as 64512
neighbor 192.168.111.25 route-reflector-client
address-family ipv4 unicast
network 192.168.1.0/24
network 192.169.1.1/32
exit-address-family
address-family ipv4 unicast
neighbor 192.168.111.20 activate
neighbor 192.168.111.20 next-hop-self
neighbor 192.168.111.21 activate
neighbor 192.168.111.21 next-hop-self
neighbor 192.168.111.22 activate
neighbor 192.168.111.22 next-hop-self
neighbor 192.168.111.40 activate
neighbor 192.168.111.40 next-hop-self
neighbor 192.168.111.47 activate
neighbor 192.168.111.47 next-hop-self
neighbor 192.168.111.23 activate
neighbor 192.168.111.23 next-hop-self
neighbor 192.168.111.24 activate
neighbor 192.168.111.24 next-hop-self
neighbor 192.168.111.25 activate
neighbor 192.168.111.25 next-hop-self
exit-address-family
neighbor remote-as 64512
neighbor route-reflector-client
address-family ipv6 unicast
network 2001:db8::/128
exit-address-family
address-family ipv6 unicast
neighbor activate
neighbor next-hop-self
exit-address-family
----
. Create a file named `daemons` that includes the following content:
+
.Example `daemons` configuration file
[source,text]
----
# This file tells the frr package which daemons to start.
#
# Sample configurations for these daemons can be found in
# /usr/share/doc/frr/examples/.
#
# ATTENTION:
#
# When activating a daemon for the first time, a config file, even if it is
# empty, has to be present *and* be owned by the user and group "frr", else
# the daemon will not be started by /etc/init.d/frr. The permissions should
# be u=rw,g=r,o=.
# When using "vtysh" such a config file is also needed. It should be owned by
# group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
#
# The watchfrr and zebra daemons are always started.
#
bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=yes
fabricd=no
vrrpd=no
#
# If this option is set the /etc/init.d/frr script automatically loads
# the config via "vtysh -b" when the servers are started.
# Check /etc/pam.d/frr if you intend to use "vtysh"!
#
vtysh_enable=yes
zebra_options=" -A 127.0.0.1 -s 90000000"
bgpd_options=" -A 127.0.0.1"
ospfd_options=" -A 127.0.0.1"
ospf6d_options=" -A ::1"
ripd_options=" -A 127.0.0.1"
ripngd_options=" -A ::1"
isisd_options=" -A 127.0.0.1"
pimd_options=" -A 127.0.0.1"
ldpd_options=" -A 127.0.0.1"
nhrpd_options=" -A 127.0.0.1"
eigrpd_options=" -A 127.0.0.1"
babeld_options=" -A 127.0.0.1"
sharpd_options=" -A 127.0.0.1"
pbrd_options=" -A 127.0.0.1"
staticd_options="-A 127.0.0.1"
bfdd_options=" -A 127.0.0.1"
fabricd_options="-A 127.0.0.1"
vrrpd_options=" -A 127.0.0.1"
# configuration profile
#
#frr_profile="traditional"
#frr_profile="datacenter"
#
# This is the maximum number of FD's that will be available.
# Upon startup this is read by the control files and ulimit
# is called. Uncomment and use a reasonable value for your
# setup if you are expecting a large number of peers in
# say BGP.
#MAX_FDS=1024
# The list of daemons to watch is automatically generated by the init script.
#watchfrr_options=""
# for debugging purposes, you can specify a "wrap" command to start instead
# of starting the daemon directly, e.g. to use valgrind on ospfd:
# ospfd_wrap="/usr/bin/valgrind"
# or you can use "all_wrap" for all daemons, e.g. to use perf record:
# all_wrap="/usr/bin/perf record --call-graph -"
# the normal daemon command is added to this at the end.
----
. Save both the `frr.conf` and `daemons` files in the same directory, such as `/tmp/frr`.
. Create an external FRR container by running the following command:
+
[source,terminal]
----
$ sudo podman run -d --privileged --network host --rm --ulimit core=-1 --name frr --volume /tmp/frr:/etc/frr quay.io/frrouting/frr:9.1.0
----
. Create the following `FRRConfiguration` and `RouteAdvertisements` configurations:
.. Create a `receive_all.yaml` file that includes the following content:
+
.Example `receive_all.yaml` configuration file
[source,yaml]
----
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: receive-all
namespace: openshift-frr-k8s
spec:
bgp:
routers:
- asn: 64512
neighbors:
- address: 192.168.111.1
asn: 64512
toReceive:
allowed:
mode: all
----
.. Create a `ra.yaml` file that includes the following content:
+
.Example `ra.yaml` configuration file
[source,yaml]
----
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
name: default
spec:
nodeSelector: {}
frrConfigurationSelector: {}
networkSelectors:
- networkSelectionType: DefaultNetwork
advertisements:
- "PodNetwork"
- "EgressIP"
----
. Apply the `receive_all.yaml` and `ra.yaml` files by running the following command:
+
[source,terminal]
----
$ for f in receive_all.yaml ra.yaml; do oc apply -f $f; done
----
.Verification
. Verify that the configurations were applied:
.. Verify that the `FRRConfiguration` configurations were created by running the following command:
+
[source,terminal]
----
$ oc get frrconfiguration -A
----
+
.Example output
[source,text]
----
NAMESPACE NAME AGE
openshift-frr-k8s ovnk-generated-6lmfb 4h47m
openshift-frr-k8s ovnk-generated-bhmnm 4h47m
openshift-frr-k8s ovnk-generated-d2rf5 4h47m
openshift-frr-k8s ovnk-generated-f958l 4h47m
openshift-frr-k8s ovnk-generated-gmsmw 4h47m
openshift-frr-k8s ovnk-generated-kmnqg 4h47m
openshift-frr-k8s ovnk-generated-wpvgb 4h47m
openshift-frr-k8s ovnk-generated-xq7v6 4h47m
openshift-frr-k8s receive-all 4h47m
----
.. Verify that the `RouteAdvertisements` configurations were created by running the following command:
+
[source,terminal]
----
$ oc get ra -A
----
+
.Example output
[source,text]
----
NAME STATUS
default Accepted
----
. Get the external FRR container ID by running the following command:
+
[source,terminal]
----
$ sudo podman ps | grep frr
----
+
.Example output
[source,text]
----
22cfc713890e quay.io/frrouting/frr:9.1.0 /usr/lib/frr/dock... 5 hours ago Up 5 hours ago frr
----
. Use the container ID that you obtained in the previous step to check the BGP neighbor and routes in the external FRR container's `vtysh` session. Run the following command:
+
[source,terminal]
----
$ sudo podman exec -it <container_id> vtysh -c "show ip bgp"
----
+
.Example output
[source,text]
----
BGP table version is 10, local router ID is 192.168.111.1, vrf id 0
Default local pref 100, local AS 64512
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*>i10.128.0.0/23 192.168.111.22 0 100 0 i
*>i10.128.2.0/23 192.168.111.23 0 100 0 i
*>i10.129.0.0/23 192.168.111.20 0 100 0 i
*>i10.129.2.0/23 192.168.111.24 0 100 0 i
*>i10.130.0.0/23 192.168.111.21 0 100 0 i
*>i10.130.2.0/23 192.168.111.40 0 100 0 i
*>i10.131.0.0/23 192.168.111.25 0 100 0 i
*>i10.131.2.0/23 192.168.111.47 0 100 0 i
*> 192.168.1.0/24 0.0.0.0 0 32768 i
*> 192.169.1.1/32 0.0.0.0 0 32768 i
----
. Find the `frr-k8s` pod for each cluster node by running the following command:
+
[source,terminal]
----
$ oc -n openshift-frr-k8s get pod -owide
----
+
.Example output
[source,text]
----
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
frr-k8s-86wmq 6/6 Running 0 25h 192.168.111.20 master-0 <none> <none>
frr-k8s-h2wl6 6/6 Running 0 25h 192.168.111.21 master-1 <none> <none>
frr-k8s-jlbgs 6/6 Running 0 25h 192.168.111.40 node1.example.com <none> <none>
frr-k8s-qc6l5 6/6 Running 0 25h 192.168.111.25 worker-2 <none> <none>
frr-k8s-qtxdc 6/6 Running 0 25h 192.168.111.47 node2.example.com <none> <none>
frr-k8s-s5bxh 6/6 Running 0 25h 192.168.111.24 worker-1 <none> <none>
frr-k8s-szgj9 6/6 Running 0 25h 192.168.111.22 master-2 <none> <none>
frr-k8s-webhook-server-6cd8b8d769-kmctw 1/1 Running 0 25h 10.131.2.9 node3.example.com <none> <none>
frr-k8s-zwmgh 6/6 Running 0 25h 192.168.111.23 worker-0 <none> <none>
----
. From the {product-title} cluster, check BGP routes on the cluster node's `frr-k8s` pod in the FRR container by running the following command:
+
[source,terminal]
----
$ oc -n openshift-frr-k8s -c frr rsh frr-k8s-86wmq
----
. Check the IP routes from the cluster node by running the following command:
+
[source,terminal]
----
sh-5.1# vtysh
----
+
.Example output
[source,text]
----
Hello, this is FRRouting (version 8.5.3).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
----
. Check the IP routes by running the following command:
+
[source,terminal]
----
worker-2# show ip bgp
----
+
.Example output
[source,text]
----
BGP table version is 10, local router ID is 192.168.111.25, vrf id 0
Default local pref 100, local AS 64512
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*>i10.128.0.0/23 192.168.111.22 0 100 0 i
*>i10.128.2.0/23 192.168.111.23 0 100 0 i
*>i10.129.0.0/23 192.168.111.20 0 100 0 i
*>i10.129.2.0/23 192.168.111.24 0 100 0 i
*>i10.130.0.0/23 192.168.111.21 0 100 0 i
*>i10.130.2.0/23 192.168.111.40 0 100 0 i
*> 10.131.0.0/23 0.0.0.0 0 32768 i
*>i10.131.2.0/23 192.168.111.47 0 100 0 i
*>i192.168.1.0/24 192.168.111.1 0 100 0 i
*>i192.169.1.1/32 192.168.111.1 0 100 0 i
Displayed 10 routes and 10 total paths
----
. From the {product-title} cluster, debug the node by running the following command:
+
[source,terminal]
----
$ oc debug node/<node_name>
----
+
.Example output
[source,text]
----
Temporary namespace openshift-debug-lbtgh is created for debugging node...
Starting pod/worker-2-debug-zrg4v ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.111.25
If you don't see a command prompt, try pressing enter.
----
. Confirm that the BGP routes are being advertised by running the following command:
+
[source,terminal]
----
sh-5.1# ip route show | grep bgp
----
+
.Example output
[source,text]
----
10.128.0.0/23 nhid 268 via 192.168.111.22 dev br-ex proto bgp metric 20
10.128.2.0/23 nhid 259 via 192.168.111.23 dev br-ex proto bgp metric 20
10.129.0.0/23 nhid 260 via 192.168.111.20 dev br-ex proto bgp metric 20
10.129.2.0/23 nhid 261 via 192.168.111.24 dev br-ex proto bgp metric 20
10.130.0.0/23 nhid 266 via 192.168.111.21 dev br-ex proto bgp metric 20
10.130.2.0/23 nhid 262 via 192.168.111.40 dev br-ex proto bgp metric 20
10.131.2.0/23 nhid 263 via 192.168.111.47 dev br-ex proto bgp metric 20
192.168.1.0/24 nhid 264 via 192.168.111.1 dev br-ex proto bgp metric 20
192.169.1.1 nhid 264 via 192.168.111.1 dev br-ex proto bgp metric 20
----
View Git Blame Copy Permalink