Using OpenBSD's OpenBGPD and Cilium for k0s Networking

Using OpenBSD’s OpenBGPD and Cilium for k0s Networking

If you’re running a bare-metal or on-premises Kubernetes cluster, one of the biggest challenges is making your services accessible to the rest of your network. While NodePorts and traditional reverse proxies work, running BGP (Border Gateway Protocol) natively gives you a much more elegant, scalable, and true-to-cloud experience.

In this post, we’ll walk through how to configure an OpenBSD router running OpenBGPD to peer with a k0s Kubernetes cluster using Cilium as the CNI (Container Network Interface). This allows Cilium to announce LoadBalancer IP pools and Pod CIDRs directly to your network’s core router.

The Architecture

Our lab setup uses the following topology:

Gateway / Router: OpenBSD 8.0 on a Protectli v1210 (pf, OpenBGPD, unbound and isc-dhcpd all on one box)
Kubernetes Distro: k0s v1.36.4 on Minisforum MS-A2 mini PCs (three control plane nodes, also running workloads, plus three workers), CentOS Stream 10
CNI: Cilium 1.20.3 (kube-proxy replacement, BGP control plane enabled)
k0s Subnet: 10.30.0.0/24 (VLAN 30)
OpenBSD VLAN30 Interface IP: 10.30.0.1
OpenBSD ASN: 65000
Cilium Nodes ASN: 65030

Each node is dual-homed: it has an address on the LAN (172.16.0.0/22) where kubectl’s InternalIP comes from, and an address on the Kubernetes VLAN where BGP peering and LoadBalancer IPs live. The VLAN octets deliberately mirror the LAN: control plane is 10.30.0.41-10.30.0.43, workers are 10.30.0.31–10.30.0.33 which makes debugging on either side much easier.

Let’s break down the OpenBSD configurations needed to make this routing magic happen.

1. Setting Up the DHCP Scope (`dhcpd.conf`)

First, we need to ensure our k0s nodes have stable IP addresses within our designated Kubernetes VLAN (VLAN 30). We define the subnet in /etc/dhcpd.conf, assigning the OpenBSD box (10.30.0.1) as both the default gateway and the DNS server.

# VLAN 30 - k0s
subnet 10.30.0.0 netmask 255.255.255.192 {
    option routers 10.30.0.1;
    option domain-name-servers 10.30.0.1;
    range 10.30.0.2 10.30.0.62;

    host k0s-w01 {
      hardware ethernet fc:aa:14:00:11:22;
      fixed-address 172.16.0.31;
      option host-name "k0s-w01";
    }
    ...
}

This allocates a generous /26 block for our cluster nodes, providing plenty of room for horizontal scaling.

2. Firewall and Routing Policies (`pf.conf`)

OpenBSD’s PF (Packet Filter) is incredibly powerful, but to get BGP working, we need to explicitly allow the BGP protocol (TCP port 179) to pass between the k0s nodes and the router.

# Interfaces and Networks
k8s_if    = "vlan30"
k8s_net   = "10.30.0.0/24"

# k0s Node IP Definitions (control plane + workers — the CP peers too!)
k8s_nodes = "{ 10.30.0.31, 10.30.0.32, 10.30.0.33, 10.30.0.41, 10.30.0.42, 10.30.0.43 }"

# Tables
table <k8s> const { $k8s_net }
table <puffy> const { 10.30.0.1 }

### NAT ###
# Ensure outbound traffic from internal networks can route correctly
match out on $k8s_if from 172.16.0.0/22 to any nat-to 10.30.0.1

### Allow BGP Peering ###
# Pass BGP (TCP 179) traffic between k0s nodes and the OpenBSD router
pass in log on $k8s_if proto tcp from $k8s_nodes to 10.30.0.1 port 179
pass out log on $k8s_if proto tcp from 10.30.0.1 to $k8s_nodes port 179

# General Node Connectivity
pass out log on $k8s_if from 10.30.0.1 to any
pass in log on $k8s_if from $k8s_nodes to any

### DNS Redirect to Unbound ###
# Transparently redirect cluster DNS queries to local unbound instance
pass in log quick on $k8s_if proto { tcp udp } from <k8s> to <puffy> port domain rdr-to 127.0.0.1
pass in log quick on $k8s_if proto tcp from <k8s> to <puffy> port 853 rdr-to 127.0.0.1

Key Takeaways from the PF config:

BGP Rules: We explicitly pass TCP port 179 in both directions. Without this, OpenBGPD will never establish a neighbor relationship with Cilium.
DNS Redirection: Nodes pointing to the gateway for DNS are seamlessly routed to the local unbound resolver on OpenBSD.

3. Configuring OpenBGPD (`bgpd.conf`)

Now for the star of the show. We need to configure OpenBGPD to listen on the Kubernetes interface and establish peering sessions with the Cilium daemon running on each k0s node.

Since the OpenBSD router and the Cilium nodes have different ASNs (65000 vs 65030), this will be an eBGP (External BGP) peering setup.

# /etc/bgpd.conf

AS 65000
router-id 10.30.0.1
listen on 10.30.0.1
network 10.30.0.0/24

# Define our k0s cluster workers
group "cilium-nodes" {
    remote-as 65030
    neighbor 10.30.0.41 { descr "k0s-m01" }
    neighbor 10.30.0.42 { descr "k0s-m02" }
    neighbor 10.30.0.43 { descr "k0s-m03" }
    neighbor 10.30.0.31 { descr "k0s-w01" }
    neighbor 10.30.0.32 { descr "k0s-w02" }
    neighbor 10.30.0.33 { descr "k0s-w03" }
}

# Filtering: Accept the Cilium LoadBalancer IP Pool
allow from group "cilium-nodes"
allow to group "cilium-nodes"
allow from AS 65030

What’s happening here?

listen on 10.30.0.1: Instructs the BGP daemon to listen for connections on the router’s VLAN 30 IP.
group "cilium-nodes": Groups all the k0s nodes together. Setting remote-as 65030 ensures the router expects the incoming BGP requests from Cilium to identify as AS 65030.
allow from AS 65030: This filter is crucial. It tells OpenBSD to accept the route advertisements that Cilium pushes (e.g., your Kubernetes LoadBalancer IP addresses).

4. The Cilium side (the part that actually advertises)

The current Cilium BGP control plane (enabled with bgpControlPlane.enabled: true in helm values) speaks in four cooperating CRDs.

The IP pool. Cilium hands out LoadBalancer addresses from an explicit pool — pick a block inside your VLAN range that DHCP won’t hand out, and clear of the router and node addresses:

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: k0s-lb-pool
spec:
  blocks:
    - cidr: 10.30.0.128/26 # range .129 - .190

What gets advertised. A CiliumBGPAdvertisement says which route classes leave the cluster — I announce both Pod CIDRs (which pairs nicely with Cilium’s native routing mode) and every Service LoadBalancerIP:

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisement
  labels:
    advertise: bgp-openbsd # peer configs select on this label
spec:
  advertisements:
    - advertisementType: PodCIDR
    - advertisementType: Service
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchLabels: {} # every Service, no filter

Session behavior. Timers and graceful restart live in a CiliumBGPPeerConfig, and the label selector above binds it to the advertisement:

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: openbsd-peer-config
spec:
  timers:
    holdTimeSeconds: 90
    keepAliveTimeSeconds: 30
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "bgp-openbsd"

The peering itself is a CiliumBGPClusterConfig: my ASN, a node selector, and one peer pointing back at the router from section 3. Note how it closes the loop with the gateway’s bgpd.conf (ASNs in mirror image, and every node in the gateway’s neighbor list must also match this nodeSelector):

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux # every node — including the control plane
  bgpInstances:
    - name: cilium-bgp
      localASN: 65030
      peers:
        - name: puffy # OpenBSD's mascot earns its place
          peerAddress: 10.30.0.1
          peerASN: 65000
          peerConfigRef:
            name: openbsd-peer-config

One gotcha worth knowing up front: these CRDs ship with schema defaults (ebgpMultihop, connectRetryTimeSeconds, and the timer/graceful-restart values). The API server injects them into your live objects, so kubectl get -o yaml shows fields you never wrote.

Seeing it work

After applying, the router tells the whole story:

# doas bgpctl show
Neighbor                   AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
k0s-m01                 65030        169        173     0 01:23:00      4
k0s-m02                 65030        170        176     0 01:23:06      4
k0s-m03                 65030        170        179     0 01:23:07      4
k0s-w01                 65030        170        173     0 01:23:02      4
k0s-w02                 65030        170        179     0 01:23:10      4
k0s-w03                 65030          0          0     0 01:24:50 Connect

Six established sessions (my control plane peers too, see the k8s_nodes macro above), each with a fistful of prefixes received from Cilium. On the cluster side:

# kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- cilium bgp peers
Local AS   Peer AS   Peer Address    Session       Uptime     Family         Received   Advertised
65030      65000     10.30.0.1:179   established   1h26m13s   ipv4/unicast   8          4

And in the pool, the addresses my LoadBalancer Services actually pulled: .129 for the shared-mode ingress, .130 for the Gateway API entrypoint that serves every *.neozeon.ca hostname, .131 for the pf-dashboard stack. Create a new LoadBalancer Service and a new IP appears in the pool, a route appears in bgpctl show rib. And because the gateway’s routes cover the whole /24, a curl from any machine on the network works as a smoke-test. No port forwarding, no reverse-proxy hop just to reach a cluster service from the LAN.

Wrapping Up

The complete chain is: pf passes TCP 179, OpenBGPD accepts the sessions and the advertised prefixes, and four small CRDs make every Cilium node a tiny router that knows exactly which IPs it owns. OpenBSD’s minimalist, secure-by-default philosophy combined with Cilium’s eBPF datapath makes homelab networking behave like a cloud VPC and every piece of it fits in about a page of Git.

Using OpenBSD’s OpenBGPD and Cilium for k0s Networking