Sometimes we want to have control loops that watch for state, and make changes based on that state. In Kubernetes that's a controller, but writing a Kubernetes controller in Go is a non-trivial task. It requires knowledge of Kubernetes, golang, and sometimes lots of complex logic. What if there were a way we could author these controllers without code? We want to solve problems that are sometimes wildly complex to very trivial but either lack the knowledge of authoring and/or the time to complete the task. Imagine the example below as a thought experiment rather than a direct use case. Let's jump on the AI hype train together if not for just a moment to explore how it might solve our daily challenges.

New Crossplane Primitives

Recently Crossplane released v2 which among things like namespace scoped resources also included operations. Operations were made to solve a lot of day 2 problems like backups or even configuration validation. They can be extended to an enormous amount of use cases but today, we will use the watch operation to monitor and change a deployment based on my requirements. I will be using Ollama to run a locally running LLM, namely gpt-oss:20b in combination with the open-ai function as it has been extended to allow calling any AI API that uses OpenAI’s API format. As grandpa used to say “a token saved is a token earned” or something like that.

The Controller

My example written at https://github.com/jaymiracola/configuration-english-controller will allow you to run an operation that will watch deployments in the default namespace and regardless of the deployment applied, ensure that they are all scaled to 3 replicas. As promised, all the plain english.

          systemPrompt: |-
            You are a Kubernetes controller implemented as an LLM.

            Goal:
            - Ensure any watched Deployment always runs at least 3 replicas.

            Rules:
            - If spec.replicas is missing or less than 3, set it to 3.
            - Otherwise, make no changes.
            - Output one or more fully-specified Kubernetes manifests as YAML.
            - Include metadata.name and metadata.namespace from the original resource.
            - Only output YAML manifests, separated by "---" if there are multiple.
          userPrompt: "Inspect the watched nginx Deployment resource and adjust it to satisfy the rules above."
          systemPrompt: |-

In order to run it, most of the steps are taken care of in the repository provided other than needing an LLM (Ollama, OpenAI, etc) to connect to. My example is setup currently for Ollama locally with no auth. Past that, all you need to do is the following:

git clone https://github.com/jaymiracola/configuration-english-controller.git && cd configuration-english-controller

Edit the secret in the example folder with your credentials for your LLM.

up project run —local

The configuration will be packaged and applied to a locally created kind cluster

kubectl apply -f examples/secret.yaml

Now everything should be ready to go! Apply the example deployment with a single replica and watch the magic happen. The operation should show the deployment with a single replica, the LLM picks that up, makes a change to 3, and the process is complete. Change it again manually if you’d like to see it in action again.

kubectl apply -f examples/deployment.yaml

Some Caveats

As I stated before, this is simply a thought experiment and I have certainly taken some creative liberties around calling it a controller. It is in its simplest form something for you to look at and think about what else could be possible. Maybe an operation that denies changes to be applied on Fridays? A step in your Infrastructure that watches for database resource utilization and scales as needed? Hallucinations and non-deterministic behaviours become less problematic as LLMs and prompts mature. A world where we solve real problems while applying our institutional knowledge in organizations may be closer than once thought.

Not today! I have an old HP Z600 (2009!) and GPU that I wanted to use to run #Kubernetes, #Ollama, Open WebUI, and utilize NVIDIA’s gpu-operator. It has been a solid machine through the years with dual socket Xeons, loads of ECC ram, and simply wont quit. It has run several hypervisors, OpenStack, OpenShift, and more! When I decided to plug in a GPU, and load up my AI stack, I had no idea the rabbit hole I would go down. Here is the short story; Ollama’s GPU runner by default uses the AVX instruction set which is not available in old CPUs. I briefly thought it was time to retire my old machine and buy something a little newer, but no! The kind Ollama devs added a build argument in their Dockerfile --build-arg CUSTOM_CPU_FLAGS= . Leaving the flag’s values empty builds the GPU runner without AVX allowing my beloved Z600 to live on, continuing to serve modern workloads.

Moral of the story? With a little ingenuity (and a helpful open-source community), old hardware can still punch above its weight in the AI era!

Why would you want to expose a pod network directly using BGP? The concepts are relatively simple. The idea of ToR or top of rack is the idea that in the data center a rack has multiple servers in it and at the top is a switch that they all connect to. Then off to an aggregate it goes. In this scenario, we have no load balancers in between as Kubernetes is keen to do, nor do we need to expose node ports. Just straight connections via advertised routes directly to the applications. Why set this up at home? Its likely that any services you may be running are one-offs serving things like Plex, Pihole, etc. This makes it incredibly easy to connect to the applications directly.

The Setup

In my setup I will be using FRR on a UDM-SE and a Raspberry Pi running K3S and Cilium. Feel free to use a standalone nix box to setup FRR but know that you will also need to add some static routes to it. For my UDM, the FRR package was already installed! Under the hood, Ubiquiti uses it for its Magic VPN feature. I don't use it, so it was straight forward to enable the systemd service with my own custom configuration I will show below. For more details, Chris's blog here can show you everything you need to do.

hostname UDM-SE
frr defaults datacenter
log file stdout
service integrated-vtysh-config
!
!
router bgp 65001
 bgp router-id 192.168.120.254
 neighbor 192.168.120.11 remote-as 65000 #raspberry pi
 neighbor 192.168.120.11 default-originate #raspberry pi
 !
 address-family ipv4 unicast
  redistribute connected
  redistribute kernel
  neighbor V4 soft-reconfiguration inbound
  neighbor V4 route-map ALLOW-ALL in
  neighbor V4 route-map ALLOW-ALL out
 exit-address-family
 !
route-map ALLOW-ALL permit 10
!
line vty
!

The above is my configuration for FRR. Straight forward besides my comments about where the raspberry pi single node k3s lives.

Now its on to Cilium and K3s. Note that you will need to disable flannel, servicelb, and network-policy. You can do this with a fresh install, setting environment variables, or by editing the systemd service. If you are running this on an existing installation, you will likely also need to remove the flannel vxlan. Run ip link show to verify its presence if you encounter a crash loop with Cilium.

ExecStart=/usr/local/bin/k3s \
    server \
        '--flannel-backend=none' \
        '--disable-network-policy' \
        '--disable=servicelb' \

Installing cilium with the binary and single flag for this use case is as follows cilium install --set bgpControlPlane.enabled=true . After a successful installation, its time to create the CiliumBGPPeeringPolicy . Below is my example with notes.

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: custom-policy
spec:
  virtualRouters:
  - exportPodCIDR: true # allows the pod CIDR to be advertised
    localASN: 65000
    neighbors:
    - connectRetryTimeSeconds: 120
      eBGPMultihopTTL: 1
      holdTimeSeconds: 90
      keepAliveTimeSeconds: 30
      peerASN: 65001 #FRR ASN
      peerAddress: 192.168.120.1/32 # FRR address
      peerPort: 179

Validation

On to validation. From the FRR side I run vtysh -c 'show ip bgp' and receive

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0/24      192.168.120.11

From where my Cilium binary is installed with access to my K3s cluster I run cilium bgp peers and receive

Node     Local AS   Peer AS   Peer Address    Session State   Uptime     Family         Received   Advertised
pi       65000      65001     192.168.120.1   established     12h49m3s   ipv4/unicast   7          1
                                                                         ipv6/unicast   0          0

From here, if you had flannel installed previously you will likely need to restart the pods in order to get new Cilium CNI range. Run a curl or whatever you see fit and verify its working!

 $curl http://10.0.0.83:8080
  Hello World!

For those uninitiated, conntrack, put simply, is a subsystem of the Linux kernel that tracks all network connections entering, exiting, or passing through the system, allowing it to monitor and manage the state of each connection, which is crucial for tasks like NAT (Network Address Translation), firewalling, and maintaining session continuity. It operates as part of Netfilter, the Linux kernel's framework for network packet filtering, which provides the underlying infrastructure for connection tracking, packet filtering, and network address translation. A quick explanation of the problem is if connection tracking reaches over the conntrack_max value (found with sysctl net.netfilter.nf_conntrack_max ) by long lived, stale, or an inundation of requests, your CPU and memory headroom will look fine but requests will be dropped.

$ sysctl net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_max = 131072

How to monitor

How can we monitor for this type of event? Or really many of the hardware and OS level metrics that are important to collect? Prometheus ships a collector called node_exporter . By utilizing this, you will be able to track and monitor events related to conntrack as defined by the project Shows conntrack statistics (does nothing if no /proc/sys/net/netfilter/ present). If running on AWS, using a Nitro based image instance type, and using the ENA driver version 2.8.1 or newer, AWS has the capability of gathering these metrics into CloudWatch should you prefer.

Ways around the problem

So how can we get around the issue? The most straight forward answer, in the context of AWS, would be to upgrade your EC2 size. AWS calculates and sets the conntrack table size based on a calculation of CPU, memory, and OS (32/64 bit). Throw more power at it! But wait, isn't that a monolith mentality?

root@ip-192-168-30-28:/# sysctl net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_max = 262144 # m5.large

root@ip-192-168-98-251:/# sysctl net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_max = 131072 # t2.micro

Another way is simply knowing your machine, traffic type, what it can handle through performance tests, and setting your conntrack_max accordingly. Using the kube-proxy ConfigMap, we can declaratively set the max as seen in the Kubernetes docs.

    conntrack:
      maxPerCore: 32768
      min: 131072
      tcpCloseWaitTimeout: 1h0m0s
      tcpEstablishedTimeout: 24h0m0s

A less likely, possibly excessive for this issue, design change that would need consideration in various elements of your architecture is the switch from iptables to IPVS. Shifting from iptables to IPVS for load balancing addresses the bottleneck of hitting the maximum connection tracking capacity. Unlike iptables, which filters and inspects packets and relies on connection tracking, IPVS efficiently routes traffic to backends with load balancing algorithms, bypassing the exhaustive state tracking.

Least likely and last, the use of tunnels. By using an IP tunnel and encapsulating the traffic, the far end only sees the tunnel as the tracked connection. This is more feasible of an option in public facing proxies hosted externally of the Kubernetes cluster (with the same conntrack adjustments), and/or not serving a public gateway.

Wrapping up

Like anything in IT, its important to monitor everything you can while being mindful of the pitfalls of cardinality and the inundation of metrics that brings. Conntrack only being a small piece of the larger puzzle of issues we solve everyday!

Getting started, I needed to choose BGP or ARP. I chose Layer 2 as the router in question isn't capable out of the box with BGP. Next I needed to configure the server interface with the second, non-default VLAN tag 12 and map it to a port.

auto eth0.12
iface eth0.12 inet static
    address 172.16.12.10
    netmask 255.255.255.0
    network 172.16.12.1
    broadcast 172.16.12.255
    gateway 172.16.12.1
    dns-nameservers 1.1.1.1 8.8.4.4
    vlan_raw_device eth0

Now the server should be able to reach out of the same physical interface and reach both the default VLAN and the newly configured VLAN 12 via interface eth0.12 .

$ arping 172.16.12.5 -I eth0.12
ARPING 172.16.12.5
60 bytes from 9c:30:5b:06:6d:f5 (172.16.12.5): index=0 time=117.228 msec
56 bytes from 9c:30:5b:06:6d:f5 (172.16.12.5): index=1 time=39.202 msec
56 bytes from 9c:30:5b:06:6d:f5 (172.16.12.5): index=2 time=132.372 msec
56 bytes from 9c:30:5b:06:6d:f5 (172.16.12.5): index=3 time=72.073 msec

$ ping 172.16.12.5 -I eth0.12
PING 172.16.12.5 (172.16.12.5) from 172.16.12.10 eth0.12: 56(84) bytes of data.
64 bytes from 172.16.12.5: icmp_seq=1 ttl=255 time=56.1 ms
64 bytes from 172.16.12.5: icmp_seq=2 ttl=255 time=8.98 ms
64 bytes from 172.16.12.5: icmp_seq=3 ttl=255 time=13.5 ms

First I need to tell metalLB about the IP pool reservations I've carved out for it on my network so it only pulls from the selected range. Ill setup two pools instead of adding another string so I can appropriately tie them to the L2 advertisements in the next step.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: public-pool
  namespace: metallb
spec:
  addresses:
  - 172.16.12.10-172.16.12.15
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: private-pool
  namespace: metallb
spec:
  addresses:
  - 192.168.120.20-192.168.120.30

Now I need to setup the L2 advertisements and tie them to the correct interfaces. Ill also reference the IP Pools above to the correct interface.

apiVersion: v1
items:
- apiVersion: metallb.io/v1beta1
  kind: L2Advertisement
  metadata:
    name: public-l2
    namespace: metallb
  spec:
    interfaces:
    - eth0.12
    ipAddressPools:
    - public-pool
kind: List
---
apiVersion: v1
items:
- apiVersion: metallb.io/v1beta1
  kind: L2Advertisement
  metadata:
    name: private-l2
    namespace: metallb
  spec:
    interfaces:
    - eth0
    ipAddressPools:
    - private-pool
kind: List

Now MetalLB's speaker logs should show the appropriate pools and advertisements tied to the correct interface. Without this , the speaker logs indicate fuzzy logic is used and it doesn't work out well in this configuration. Moving on, I need to now configure my services so that the use the IP range when calling type: LoadBalancer on the separate interfaces. For brevity, I'll only show an example of the VLAN tagged interface. The other service also matches on the same application, ports, etc but serves the default VLAN.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: dns-server
  name: dns-public
  namespace: dns-server
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Local
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  loadBalancerIP: 172.16.12.10
  ports:
  - name: dns
    port: 53
    targetPort: dns
  - name: dns-udp
    port: 53
    protocol: UDP
    targetPort: dns-udp
  selector:
    app: dns-server
    release: dns-server
  type: LoadBalancer

From here, everything is up and working. Verifying would be as simple as using nslookup to the appropriate IPs or similar with arping again to be certain our L2 advertisements are working. Instead, lets use a Kubernetes tool called ksniff to observe the traffic on the speaker in cluster. Here is an example of what that command might look like.

kubectl sniff -n metallb metallb-speaker -p

In my scenario, it was a bit more involved. Because I am running k3s on ARM I needed to specify the containerd socket and use arm compatible images. Im also going to specify the traffic Im interested in. Just in case you're interested, heres what that looks like!

kubectl sniff -n metallb argo-metallb-speaker-nprwp --socket /run/k3s/containerd/containerd.sock -p --image ghcr.io/nefelim4ag/ksniff-helper:v4 --tcpdump-image ghcr.io/nefelim4ag/tcpdump:latest -f "arp"

Above we can see ARP being successfully requested and replied to for the appropriate IP and MAC. Now the speaker is successfully routing traffic via the appropriate services to the same DNS server in Kubernetes so all external requests can be observed in a single place.

First lets stand up our bootstrap cluster using Kind

kind create cluster --name bootstrap

Next go ahead and fork, copy, whatever you wish in order to get the necessary files I've shared in this repository https://gitlab.com/jaymiracola/app-of-infra. After you've done so we will install ArgoCD followed by kicking off our app-of-apps pattern by applying the first application which is Argo itself. From there forward it will be managed via git.

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argo argo/argo-cd -n argocd --create-namespace --set version=5.51.4

kubectl apply -f applications.yaml

Now ArgoCD and Crossplane have been installed and are being defined entirely from our git repository. It's time to define the infrastructure.

kubectl get applications -A

NAMESPACE   NAME           SYNC STATUS   HEALTH STATUS
argocd      applications   Synced        Healthy
argocd      argocd         Synced        Healthy
argocd      crossplane     Synced        Healthy
argocd      infra          Synced        Healthy

Next, we will define some infrastructure in Digital Ocean. Why Digital Ocean vs AWS, GCP, or Azure? Simple, I use them all the time for low cost services I use personally! Of course, from here you could define whatever you'd like with Crossplane so the cloud and infrastructure is up to you.

Now we need to start defining Providers and Provider Configurations to Crossplane to configure which Cloud Providers we want and the keys to allow Crossplane to configure them on our behalf. I've already added the Digital Ocean Provider so now we need to configure access.

I'll add the following manifest to my /app-of-infra/applications/infra/manifests/ folder as I've already defined and declared it as an application that ArgoCD is tracking. You can get a Digital Ocean token here.

apiVersion: v1
kind: Secret
metadata:
  namespace: crossplane
  name: provider-do-secret
type: Opaque
data:
  token: BASE64ENCODED_PROVIDER_CREDS
---
apiVersion: do.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: do-config
  namespace: crossplane
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane
      name: provider-do-secret
      key: token

Important! Do NOT have this in a public repo. There is a reason this part of the instruction is omitted from my demonstration repository.

Last, it's time to declare infrastructure! As an example I will add the following configuration in /app-of-infra/applications/infra/manifests/ to create a Kubernetes cluster in my Digital Ocean account. More examples of different Digital Ocean infrastructure can be found here.

apiVersion: kubernetes.do.crossplane.io/v1alpha1
kind: DOKubernetesCluster
metadata:
  name: do-cluster
  namespace: infra
spec:
  providerConfigRef:
    name: do-config
  forProvider:
    region: nyc3
    version: 1.29.0-do.0
    nodePools:
      - size: s-1vcpu-2gb #lowest tier
        count: 1 #cost cutting for demo
        name: worker-pool
    maintenancePolicy:
      startTime: "03:00"
      day: sunday
    autoUpgrade: true
    surgeUpgrade: false
    highlyAvailable: false

Now you have successfully declared the entirety of your bootstrap cluster all the way into platform infrastructure in a not only declarative but also idempotent pattern! From here you can create more infrastructure as needed. You may also be interested in extending the control plane using Crossplane's Composite Resource Definitions allowing SRE teams to abstract infra with apps (and more) to easily consumable APIs.