BGP ,Cilium, and FRR: Top of Rack For All!
I recently came across a LinkedIn post talking about the above concepts and its trivialness to setup. The goal: Use Cilium's BGP capabilities to either expose a service or export the pod cidr and advertise its range to a peer. We are all on different chapters of our life's book, so I wanted to explain the setup a little more in order to possibly help someone out there add a feather to their hat!
Why would you want to expose a pod network directly using BGP? The concepts are relatively simple. The idea of ToR or top of rack is the idea that in the data center a rack has multiple servers in it and at the top is a switch that they all connect to. Then off to an aggregate it goes. In this scenario, we have no load balancers in between as Kubernetes is keen to do, nor do we need to expose node ports. Just straight connections via advertised routes directly to the applications. Why set this up at home? Its likely that any services you may be running are one-offs serving things like Plex, Pihole, etc. This makes it incredibly easy to connect to the applications directly.
The Setup
In my setup I will be using FRR on a UDM-SE and a Raspberry Pi running K3S and Cilium. Feel free to use a standalone nix box to setup FRR but know that you will also need to add some static routes to it. For my UDM, the FRR package was already installed! Under the hood, Ubiquiti uses it for its Magic VPN feature. I don't use it, so it was straight forward to enable the systemd service with my own custom configuration I will show below. For more details, Chris's blog here can show you everything you need to do.
hostname UDM-SE
frr defaults datacenter
log file stdout
service integrated-vtysh-config
!
!
router bgp 65001
bgp router-id 192.168.120.254
neighbor 192.168.120.11 remote-as 65000 #raspberry pi
neighbor 192.168.120.11 default-originate #raspberry pi
!
address-family ipv4 unicast
redistribute connected
redistribute kernel
neighbor V4 soft-reconfiguration inbound
neighbor V4 route-map ALLOW-ALL in
neighbor V4 route-map ALLOW-ALL out
exit-address-family
!
route-map ALLOW-ALL permit 10
!
line vty
!
The above is my configuration for FRR. Straight forward besides my comments about where the raspberry pi single node k3s lives.
Now its on to Cilium and K3s. Note that you will need to disable flannel, servicelb, and network-policy. You can do this with a fresh install, setting environment variables, or by editing the systemd service. If you are running this on an existing installation, you will likely also need to remove the flannel vxlan. Run ip link show to verify its presence if you encounter a crash loop with Cilium.
ExecStart=/usr/local/bin/k3s \
server \
'--flannel-backend=none' \
'--disable-network-policy' \
'--disable=servicelb' \
Installing cilium with the binary and single flag for this use case is as follows cilium install --set bgpControlPlane.enabled=true . After a successful installation, its time to create the CiliumBGPPeeringPolicy . Below is my example with notes.
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
name: custom-policy
spec:
virtualRouters:
- exportPodCIDR: true # allows the pod CIDR to be advertised
localASN: 65000
neighbors:
- connectRetryTimeSeconds: 120
eBGPMultihopTTL: 1
holdTimeSeconds: 90
keepAliveTimeSeconds: 30
peerASN: 65001 #FRR ASN
peerAddress: 192.168.120.1/32 # FRR address
peerPort: 179
Validation
On to validation. From the FRR side I run vtysh -c 'show ip bgp' and receive
Network Next Hop Metric LocPrf Weight Path
*> 10.0.0.0/24 192.168.120.11
From where my Cilium binary is installed with access to my K3s cluster I run cilium bgp peers and receive
Node Local AS Peer AS Peer Address Session State Uptime Family Received Advertised
pi 65000 65001 192.168.120.1 established 12h49m3s ipv4/unicast 7 1
ipv6/unicast 0 0
From here, if you had flannel installed previously you will likely need to restart the pods in order to get new Cilium CNI range. Run a curl or whatever you see fit and verify its working!
$curl http://10.0.0.83:8080
Hello World!
