Somewhere between running my first app on Kubernetes and running my thirtieth, the problem changed shape. It stopped being “how do I deploy this” and became “how do I know what my cluster is, and how do I change it without nuking & paving.” My answer is a single GitLab repo gitlab.com/caseyrobb/iac: one ApplicationSet generator, one allowlist per cluster, one thin wrapper chart per app, and a handful of infrastructure operators doing the rest. The production cluster, chaos (k0s, six nodes on mini PCs at home — no, the name is not prophetic; mostly), runs thirty-two apps out of that repo and has never had a kubectl apply from a human.

This post is a tour of the five design choices that make that work and a few scars, because every ignoreDifferences rule in my repo is the fossil of a fight ArgoCD and I had.

1. The monorepo, and an ApplicationSet that treats absence as policy

Everything (ArgoCD’s own bootstrap, all chart wrappers, all per-cluster values) lives in one repo. The tempting alternative is a chart repo per app or a “platform” repo plus an “apps” repo, and I tried fragments of both. The monorepo won for one reason: a change and its consequences get reviewed in one diff. Adding an app means a new directory and its allowlist entry in the same merge request; CI fails if either half is missing.

The entire fleet is generated by one ApplicationSet, whose generator is a matrix of clusters × that cluster’s app list:

generators:
  - matrix:
      generators:
        - list:
            elements:
              - clusterName: PATCH_ME # bootstrap patches in the real name
        - git:
            repoURL: https://gitlab.com/caseyrobb/iac.git
            revision: HEAD
            files:
              - path: "clusters/{{ .clusterName }}/apps.yaml"
template:
  spec:
    source:
      path: "apps/{{ .app }}"
      helm:
        valueFiles:
          - values.yaml
          - "../../clusters/{{ .clusterName }}/{{ .app }}.yaml"

The file that generator reads is the most important opinion in the repo:

# clusters/chaos/apps.yaml — app deploys ONLY if listed here.
- app: argocd
- app: cilium
  namespaceOverride: kube-system
- app: immich
- app: keycloak
  namespaceOverride: keycloak

The allowlist is the contract. Presence deploys; absence is a deliberate decision, not an oversight, and CI enforces the inverse direction too. Every chart under apps/ must be claimed by at least one cluster, so dead apps can’t accrete. The sandbox cluster is the same generator pointed at a different file, which means staging an app in prod is one line in a merge request, reviewed like everything else.

Values layering follows the same philosophy: each wrapper chart carries its values.yaml (defaults for all clusters) and an optional per-cluster overlay. ignoreMissingValueFiles: true means overlays exist only where they earn their place.

One honesty corner, because every GitOps blog pretends ignoreDifferences is for cowards: I have exactly three, and each one is a documented truce. Two Secrets whose data is written by the External Secrets controller (which ArgoCD can never observe as Synced, so the alternative is a permanent red badge and selfHeal ping-pong), and ServiceMonitors, where the prometheus-operator defaulting webhook rewrites fields the chart never sent, a perpetual OutOfSync that SSA applies and the webhook un-applies, forever. Each rule states its reason in a comment. IgnoreDifferences without a written reason is just drift with extra steps.

And because syncPolicy is automated + selfHeal, there’s one operational rule the whole cluster lives by: nothing is hot-patched. The fix for anything is a revert-MR to master. The emergency lever is git revert, and it’s the same lever you use on a normal Tuesday.

2. CNI is the platform: Cilium, Gateway API, and one door

Cilium does far more than pod networking on this cluster: kube-proxy replacement, native routing (which makes the BGP setup a post of its own, see the OpenBGPD writeup), Hubble for observability, Envoy as the L7 data-path, and the Gateway API implementation. It installs via the appset like any app, into kube-system via namespaceOverride, from a wrapper chart that just pins the upstream release.

Every public service on the cluster passes through one Gateway and its own one-screen HTTPRoute:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: neozeon-gateway
  namespace: kube-system
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.neozeon.ca"
      allowedRoutes:
        namespaces: { from: All }
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-neozeon-tls-secret

Adding an app’s public face is then pure declaration, an HTTPRoute in the wrapper chart mapping hostname + backend service to the shared gateway. No Ingress annotations zoo, no per-app TLS config, no load-balancer provisioning: the gateway’s LoadBalancer IP already exists (it’s one of the addresses Cilium advertises over BGP), and cert-manager already owns the wildcard cert. The route is the only per-app decision left, which is exactly how it should be.

3. Secrets: Vault as the source of truth that never touches git

The secret-management posture fits in one sentence, and that’s the point: no secret material is ever committed to git or posted to chat. What lives in git is references.

The chain: a HashiCorp Vault (one node, it’s a homelab :p) runs in-cluster on persistent storage; every credential lives under a chaos/ KV v2 mount, chaos/rustfs/rustfs-secret, chaos/keycloak/operator-client, and so on. External Secrets Operator syncs Vault into Secrets through a single ClusterSecretStore:

provider:
  vault:
    server: https://vault.neozeon.ca
    path: chaos
    version: v2
    auth:
      kubernetes:
        mountPath: kubernetes
        role: external-secrets
        serviceAccountRef:
          name: external-secrets

That auth.kubernetes block is the quiet hero. ESO authenticates to Vault by exchanging the controller’s own projected service-account token. No static token in git, no static token in a Secret, no rotation ceremony. When I want a new credential in the cluster, the flow is: put it in Vault, chart’s ExternalSecret template names the path, ArgoCD syncs. The cluster learns secrets the same way it learns everything else: from a merge request.

The hygiene rules that grew around this are boring and absolute. Secrets that ESO owns get /data ignored in ArgoCD (above), so selfHeal and ESO never fight. Backups (CNPG’s ScheduledBackup to an in-cluster RustFS S3 bucket) read per-namespace ExternalSecrets for their S3 credentials, so the backup path and the application path share one Vault source. One rotation, both fixed. Finally, the credential for the notification system, pushover, follows the same discipline living in Vault.

4. Storage: ZFS on the NAS, iSCSI into the cluster, and the longhorn war I retired

Storage is the least fashionable infrastructure decision and the one that decides whether you keep the cluster. Mine is deliberately old-fashioned: a Debian 13 NAS with a raidz2 ZFS pool exports volumes over iSCSI, and democratic-csi in the cluster converts iSCSI targets into PersistentVolumes with a StorageClass nas-iscsi (zfs-generic-iscsi under the hood) that everything uses:

driver:
  existingConfigSecret: democratic-csi-iscsi
  config:
    driver: zfs-generic-iscsi
storageClasses:
  - name: nas-iscsi
    reclaimPolicy: Delete
    allowVolumeExpansion: true

The cluster previously ran Longhorn, the Kubernetes-native block storage product with the lovely UI and the replica-rebalancing opinions. On homelab hardware, it was a second distributed system to babysit in order to run the other distributed systems: upgrade windows for the storage layer, replica anti-affinity drama whenever a node rebooted for kernel patches, and a recovery procedure that required the cluster to be healthy to help the cluster be healthy. I decommissioned it, migrated the volumes, and the config above is what replaced it: ZFS snapshots and sends on the NAS, iSCSI to the nodes, plain TCP over the switch (homelab-fast enough), and one k0s quirk: the node.kubeletHostPath has to point at /var/lib/k0s/kubelet, because k0s doesn’t pretend to be kubeadm.

The trade is honest: no replicated volumes, so the NAS is a single point. I accepted it the same way I accepted a single gateway router, with the same upgrade path in my back pocket: a second NAS and DRBD or just very good snapshots. What I got back is that storage now fails in ways I understand in one layer instead of three. RustFS (an S3-compatible server, itself running on these very volumes) then serves as the backup target for the Postgres clusters (CloudNativePG writes Barman-format base backups and WAL straight into it nightly), which means the storage stack is: ZFS → iSCSI → PVs → apps and app-backups, with each hop boring enough to debug at 23:00.

5. Certificates and DNS: the wildcard that makes the edges line up

Two small operators finish the perimeter, and they’re best explained together because they solve one problem from both ends: an app should become reachable by writing a route, not by filing tickets with yourself.

cert-manager holds the ACME relationship with Let’s Encrypt via a ClusterIssuer in the wrapper chart, using the DNS-01 challenge through Cloudflare, which is what makes the *.neozeon.ca wildcard certificate in section 2 renewable forever without exposing an HTTP path:

solvers:
  - dns01:
      cloudflare:
        apiTokenSecretRef:
          name: cloudflare-api-token-secret # from Vault, via ESO
          key: api-token

Yes: even cert-manager’s Cloudflare API token is an ExternalSecret. There is no bootstrap exception, which is why this section is short.

external-dns reads Gateway API objects: gateway-httproute, gateway-tlsroute, gateway-tcproute, policy sync and maintains the Cloudflare DNS records that mirror them. Create a route in git, get a DNS record through the same tunnel the app uses; delete the route, the record goes away. The cluster’s public surface and its DNS are the same list, which is the kind of statement you should distrust in any blog post, including this one.

The asterisk: the Cloudflare zone deliberately has no wildcard record. Every hostname is a real DNS entry (external-dns’s doing) plus a manual Cloudflare Tunnel route (mine). That manual step allows me to selectively expose an app over the internet versus just being available via a RFC1918 address on my LAN.

Wrapping Up

Step back and the architecture is almost aggressively boring: one repo that a generator matrix fans out into thirty-plus apps, one CNI wearing four hats, one Gateway, one Vault, one NAS, one wildcard certificate. The interesting parts aren’t components, they’re the contracts: the allowlist that makes absence mean something, the wrapper chart that pins one version while upstream drifts, the ExternalSecret that makes a credential deployable by ArgoCD, and the selfHeal that makes every operator habit (like hot-patching) impossible to form.

Boring is the goal. Boring is what “production” costs when you run it on mini-PCs in my basement and want the weekend back.

OpenShift is unusual among Kubernetes distros in how much it demands from your network before it will install itself: three virtual IPs (API, internal API, and wildcard ingress), DNS that resolves all three before the first node boots, and a healthcheck contract for your load balancer that is spelled out almost to the millisecond in the install docs. In the cloud this is someone else’s problem, on bare metal it is yours.

In this post I’ll build the entire gateway side of an OpenShift compact cluster: the pf rules, a validated relayd.conf that satisfies OpenShift’s health-check spec exactly, a tiny shell probe for the ingress routers, and the unbound DNS entries the installer will nag you about. The cluster deployment itself (ignition, bootstrap, all that) is out of scope, this is the part that lives on your router forever.

My platform is a Protectli v1210 running OpenBSD 8.0, the same little box that already routes my LAN (igc1, 172.16.0.0/22) and peers BGP with my other clusters.

The Architecture

OpenShift needs three addresses fronting a three-node compact cluster:

• API VIP (172.16.0.70) — fronts port 6443 (kube API), 22623 (Machine Config Server / Ignition), and 22624 (MCS internal)
• Ingress VIP (172.16.0.75) — fronts ports 80/443 for every *.apps.ocp.redcomet.ca route
• The three nodes (ocp1–ocp3, 172.16.0.71–.73) — masters that also run workloads

 clients ──6443/22623──►  ┌────────────────────────────────┐
                          │ OpenBSD gateway (Protectli)    │
 clients ────80/443───►   │  pf (rdr) + relayd + unbound   │
                          │  VIPs aliased on igc1          │
                          └──────────┬─────────────────────┘
                          ┌──────────┴──────────┐
                       ocp1 .71 ── ocp2 .72 ── ocp3 .73

The key decision: relayd runs as an L4 redirect, not an L7 proxy. In redirect mode relayd arranges a pf rdr-to, meaning packets are redirected, not terminated. The client’s source IP arrives intact at the router pods, and TLS passes through untouched. relayd never sees a key, and your apps never need PROXY protocol or X-Forwarded-For gymnastics just because a load balancer exists.

1. Aliases and pf

First the VIPs must exist on the wire. They’re aliases on the LAN interface, persisted in /etc/hostname.igc1:

inet 172.16.0.1 255.255.252.0 172.16.3.255
inet 172.16.0.70 alias
inet 172.16.0.75 alias

Then pf. relayd injects its own rdr-to rules into an anchor, so the anchor line is the one requirement; everything else is ordinary filter policy:

# --- OpenShift external LB (gateway side) ---
ocp_if     = "igc1"
ocp_nodes  = "{ 172.16.0.71, 172.16.0.72, 172.16.0.73 }"
ocp_apivip = "172.16.0.70"
ocp_ingvip = "172.16.0.75"

# relayd's redirect rules land here automatically
anchor "relayd/*"

# Explicit pass rules so ordering stays sane in a deny-by-default ruleset
pass in quick on $ocp_if inet proto tcp from any to $ocp_apivip port { 6443 22623 22624 }
pass in quick on $ocp_if inet proto tcp from any to $ocp_ingvip port { 80 443 }

# ... and never, ever forward 1936 (router stats/metrics)

Two details worth calling out. The pass rules are technically redundant, a relayd redirect passes the traffic it redirects but being explicit keeps your rule ordering auditable when the ruleset grows. And the 1936 comment deserves emphasis: that’s the HAProxy stats port on every router pod, and if it’s reachable from outside the cluster you’re leaking your whole routing table as a metric. The health check for it runs from the gateway itself, so no forward rule is ever needed.

2. relayd: OpenShift’s health-check contract, in 60 lines

OpenShift documents its load balancer expectations precisely: probe the kube API with a TLS GET /readyz expecting 200, the MCS with a TLS GET /healthz, probe every 5–10 seconds, and a failed node must leave the pool within 30 seconds. Long-lived connections (oc exec, oc port-forward, watches, websockets) must also survive. relayd can satisfy every clause. Here’s the config:

# /etc/relayd.conf — OpenShift external LB

apivip="172.16.0.70"
ingvip="172.16.0.75"

interval 10           # spec: probe every 5-10s
timeout 3000          # check timeout ms; generous for TLS handshakes
log state changes

table <ocp_api>   { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }
table <ocp_mcs>   { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }
table <ocp_mcs24> { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }
table <ocp_http>  { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }
table <ocp_https> { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }

# kube API: TLS GET /readyz -> 200 (docs-compliant unauthenticated readiness)
redirect "ocp-api" {
 listen on $apivip port 6443
 forward to <ocp_api> port 6443 timeout 1500 \
  check send "GET /readyz HTTP/1.0\r\n\r\n" expect "HTTP/1.[01] 200*" tls
 session timeout 3600
}

# Machine Config Server / Ignition: TLS GET /healthz -> 200
redirect "ocp-mcs" {
 listen on $apivip port 22623
 forward to <ocp_mcs> port 22623 timeout 1500 \
  check send "GET /healthz HTTP/1.0\r\n\r\n" expect "HTTP/1.[01] 200*" tls
 session timeout 3600
}

# MCS internal (kubelet config-driver): TCP only
redirect "ocp-mcs24" {
 listen on $apivip port 22624
 forward to <ocp_mcs24> port 22624 timeout 1500 check tcp
 session timeout 3600
}

# Ingress: L4 passthrough on 80/443, readiness probed on router stats port 1936
redirect "ocp-http" {
 listen on $ingvip port 80
 forward to <ocp_http> port 80 timeout 1500 \
  check script "/usr/local/bin/ocp_router_check"
 session timeout 3600
}

redirect "ocp-https" {
 listen on $ingvip port 443
 forward to <ocp_https> port 443 timeout 1500 \
  check script "/usr/local/bin/ocp_router_check"
 session timeout 3600
}

Reading the highlights:

• check send/expect ... tls speaks just enough HTTP over TLS to ask the API server if it’s ready. No certificates, no client auth, no dependencies. Interval 10 with per-host retry 2 puts a dead node out of the pool in roughly 30 seconds: exactly the documented maximum, no tighter.
• session timeout 3600 beats relayd’s 600-second default so an oc rsh into a debugging pod doesn’t evaporate mid-sentence.
• Five tables, not one — the ingress table membership should differ from the API table the moment you add dedicated infra nodes; starting with five identical tables means adding a node later is one line, not a redesign.

The ingress check: where relayd gets weird

The router readiness probe lives on port 1936 (GET /healthz/ready on the HAProxy stats interface) but relayd health checks always target the forwarded port, and there’s no check-port override (haproxy has one; relayd doesn’t). The escape hatch is check script, which also comes with the single most counterintuitive rule in relayd.conf(5):

the script “expects a positive return value on success and zero on failure”

Exit nonzero = healthy. Exit 0 = down. It’s inverted from every monitoring convention you know, and no syntax error will warn you. Misread it and every node reports down forever. Here’s the probe, including two more traps it dodges:

#!/bin/sh
# /usr/local/bin/ocp_router_check <ip>
# relayd 'check script' probe: router readiness on stats port 1936
h="$1"
[ -n "$h" ] || exit 0
r=$( (printf 'GET /healthz/ready HTTP/1.0\r\n\r\n'; sleep 0.2) \
     | /usr/bin/nc -w 1 "$h" 1936 2>/dev/null | /usr/bin/head -n1 )
case "$r" in
  *" 200"*) exit 1 ;;   # positive -> relayd marks host UP
  *)        exit 0 ;;   # zero    -> relayd marks host DOWN
esac

• The script runs as _relayd with a minimal PATH.
• It requests a plain path because httpd’s GET /healthz/ready on a directory returns a 301 redirect, and this probe doesn’t follow redirects.

Install both files, then rcctl enable relayd && rcctl restart relayd, and watch it think:

# relayctl show hosts
172.16.0.71   100%  6/6   up    (3.12s)
172.16.0.72   100%  6/6   up    (2.87s)
172.16.0.73   100%  6/6   up    (3.01s)

3. unbound: the DNS the installer will hold you hostage over

OpenShift resolves api.<cluster-domain> and api-int.<cluster-domain> from inside the cluster. MCO and etcd will not tolerate api-int failing to resolve and every application route hangs off a wildcard for the apps domain. The three records that matter, in /var/unbound/etc/unbound.conf:

server:
    # ... usual homelab server config ...

    local-zone: "ocp.redcomet.ca." redirect
    local-data: "api.ocp.redcomet.ca.       3600 IN A 172.16.0.70"
    local-data: "api-int.ocp.redcomet.ca.   3600 IN A 172.16.0.70"
    local-data: "*.apps.ocp.redcomet.ca.    3600 IN A 172.16.0.75"

Plus the boring-but-vital node records (ocp1–ocp3 → .71–.73) in the regular LAN zone so the hostnames, the DHCP leases, and relayctl output all agree.

The wildcard is the one everyone forgets: without *.apps pointing at the ingress VIP, your first oc expose “works” perfectly and then times out, and the error message will not mention DNS. Note that redirect is what makes the wildcard legal, unbound only synthesizes wildcard answers inside a declared redirect zone. If the records silently don’t resolve, that’s usually why.

Verifying the whole path before OpenShift exists

The beauty of doing the LB first is that you can prove it before a single RHCOS disk is written:

# relayctl show hosts        # all nodes up in all five tables
# relayctl show redirects    # all five redirects "active"
$ curl -k https://172.16.0.70:6443/readyz      # mock 200 (on the node!)
$ dig +short api.ocp.redcomet.ca               # 172.16.0.70
$ dig +short foo.apps.ocp.redcomet.ca          # 172.16.0.75

The failover test is the confidence-builder: pull one node’s IP off the wire and watch relayctl show hosts drop it to down in all five tables within 30 seconds while the redirects stay active with two hosts. Then re-add it and watch the check counts climb back.

One OpenShift-specific dance remains, and it’s easy to trip over on install day: the bootstrap host serves Ignition on port 22623 to the masters. If your MCS table only contains the three nodes, the masters boot, can’t fetch their ignition, and everyone stares at a timeout. Add the bootstrap host’s IP to <ocp_api> and <ocp_mcs> before you start (relayctl or edit and reload), and remove it when the install completes.

Wrapping Up

Three small pieces on one unassuming x86 box: pf aliases and rules, a 60-line relayd config with one inverted-exit-code shell probe, and three unbound records. An OpenShift installer that normally assumes it’s running in a cloud now finds a compliant enterprise-style load balancer waiting on the LAN. relayd’s redirect mode means OpenShift never learns a proxy exists: real source IPs, untouched TLS, no application changes.

Is the single gateway a load-balancer SPOF? Yes, and for a homelab that’s a fine trade with a clean upgrade path: a second OpenBSD, carp(4) VIPs, relayd on both if I ever choose to go down that path.

If you’re running a bare-metal or on-premises Kubernetes cluster, one of the biggest challenges is making your services accessible to the rest of your network. While NodePorts and traditional reverse proxies work, running BGP (Border Gateway Protocol) natively gives you a much more elegant, scalable, and true-to-cloud experience.

In this post, we’ll walk through how to configure an OpenBSD router running OpenBGPD to peer with a k0s Kubernetes cluster using Cilium as the CNI (Container Network Interface). This allows Cilium to announce LoadBalancer IP pools and Pod CIDRs directly to your network’s core router.

The Architecture

Our lab setup uses the following topology:

• Gateway / Router: OpenBSD 8.0 on a Protectli v1210 (pf, OpenBGPD, unbound and isc-dhcpd all on one box)
• Kubernetes Distro: k0s v1.36.4 on Minisforum MS-A2 mini PCs (three control plane nodes, also running workloads, plus three workers), CentOS Stream 10
• CNI: Cilium 1.20.3 (kube-proxy replacement, BGP control plane enabled)
• k0s Subnet: 10.30.0.0/24 (VLAN 30)
• OpenBSD VLAN30 Interface IP: 10.30.0.1
• OpenBSD ASN: 65000
• Cilium Nodes ASN: 65030

Each node is dual-homed: it has an address on the LAN (172.16.0.0/22) where kubectl’s InternalIP comes from, and an address on the Kubernetes VLAN where BGP peering and LoadBalancer IPs live. The VLAN octets deliberately mirror the LAN: control plane is 10.30.0.41-10.30.0.43, workers are 10.30.0.31–10.30.0.33 which makes debugging on either side much easier.

Let’s break down the OpenBSD configurations needed to make this routing magic happen.

1. Setting Up the DHCP Scope (dhcpd.conf)

First, we need to ensure our k0s nodes have stable IP addresses within our designated Kubernetes VLAN (VLAN 30). We define the subnet in /etc/dhcpd.conf, assigning the OpenBSD box (10.30.0.1) as both the default gateway and the DNS server.

# VLAN 30 - k0s
subnet 10.30.0.0 netmask 255.255.255.192 {
    option routers 10.30.0.1;
    option domain-name-servers 10.30.0.1;
    range 10.30.0.2 10.30.0.62;

    host k0s-w01 {
      hardware ethernet fc:aa:14:00:11:22;
      fixed-address 172.16.0.31;
      option host-name "k0s-w01";
    }
    ...
}

This allocates a generous /26 block for our cluster nodes, providing plenty of room for horizontal scaling.

2. Firewall and Routing Policies (pf.conf)

OpenBSD’s PF (Packet Filter) is incredibly powerful, but to get BGP working, we need to explicitly allow the BGP protocol (TCP port 179) to pass between the k0s nodes and the router.

# Interfaces and Networks
k8s_if    = "vlan30"
k8s_net   = "10.30.0.0/24"

# k0s Node IP Definitions (control plane + workers — the CP peers too!)
k8s_nodes = "{ 10.30.0.31, 10.30.0.32, 10.30.0.33, 10.30.0.41, 10.30.0.42, 10.30.0.43 }"

# Tables
table <k8s> const { $k8s_net }
table <puffy> const { 10.30.0.1 }

### NAT ###
# Ensure outbound traffic from internal networks can route correctly
match out on $k8s_if from 172.16.0.0/22 to any nat-to 10.30.0.1

### Allow BGP Peering ###
# Pass BGP (TCP 179) traffic between k0s nodes and the OpenBSD router
pass in log on $k8s_if proto tcp from $k8s_nodes to 10.30.0.1 port 179
pass out log on $k8s_if proto tcp from 10.30.0.1 to $k8s_nodes port 179

# General Node Connectivity
pass out log on $k8s_if from 10.30.0.1 to any
pass in log on $k8s_if from $k8s_nodes to any

### DNS Redirect to Unbound ###
# Transparently redirect cluster DNS queries to local unbound instance
pass in log quick on $k8s_if proto { tcp udp } from <k8s> to <puffy> port domain rdr-to 127.0.0.1
pass in log quick on $k8s_if proto tcp from <k8s> to <puffy> port 853 rdr-to 127.0.0.1

Key Takeaways from the PF config:

1. BGP Rules: We explicitly pass TCP port 179 in both directions. Without this, OpenBGPD will never establish a neighbor relationship with Cilium.
2. DNS Redirection: Nodes pointing to the gateway for DNS are seamlessly routed to the local unbound resolver on OpenBSD.

3. Configuring OpenBGPD (bgpd.conf)

Now for the star of the show. We need to configure OpenBGPD to listen on the Kubernetes interface and establish peering sessions with the Cilium daemon running on each k0s node.

Since the OpenBSD router and the Cilium nodes have different ASNs (65000 vs 65030), this will be an eBGP (External BGP) peering setup.

# /etc/bgpd.conf

AS 65000
router-id 10.30.0.1
listen on 10.30.0.1
network 10.30.0.0/24

# Define our k0s cluster workers
group "cilium-nodes" {
    remote-as 65030
    neighbor 10.30.0.41 { descr "k0s-m01" }
    neighbor 10.30.0.42 { descr "k0s-m02" }
    neighbor 10.30.0.43 { descr "k0s-m03" }
    neighbor 10.30.0.31 { descr "k0s-w01" }
    neighbor 10.30.0.32 { descr "k0s-w02" }
    neighbor 10.30.0.33 { descr "k0s-w03" }
}

# Filtering: Accept the Cilium LoadBalancer IP Pool
allow from group "cilium-nodes"
allow to group "cilium-nodes"
allow from AS 65030

What’s happening here?

• listen on 10.30.0.1: Instructs the BGP daemon to listen for connections on the router’s VLAN 30 IP.
• group "cilium-nodes": Groups all the k0s nodes together. Setting remote-as 65030 ensures the router expects the incoming BGP requests from Cilium to identify as AS 65030.
• allow from AS 65030: This filter is crucial. It tells OpenBSD to accept the route advertisements that Cilium pushes (e.g., your Kubernetes LoadBalancer IP addresses).

4. The Cilium side (the part that actually advertises)

The current Cilium BGP control plane (enabled with bgpControlPlane.enabled: true in helm values) speaks in four cooperating CRDs.

The IP pool. Cilium hands out LoadBalancer addresses from an explicit pool — pick a block inside your VLAN range that DHCP won’t hand out, and clear of the router and node addresses:

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: k0s-lb-pool
spec:
  blocks:
    - cidr: 10.30.0.128/26 # range .129 - .190

What gets advertised. A CiliumBGPAdvertisement says which route classes leave the cluster — I announce both Pod CIDRs (which pairs nicely with Cilium’s native routing mode) and every Service LoadBalancerIP:

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisement
  labels:
    advertise: bgp-openbsd # peer configs select on this label
spec:
  advertisements:
    - advertisementType: PodCIDR
    - advertisementType: Service
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchLabels: {} # every Service, no filter

Session behavior. Timers and graceful restart live in a CiliumBGPPeerConfig, and the label selector above binds it to the advertisement:

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: openbsd-peer-config
spec:
  timers:
    holdTimeSeconds: 90
    keepAliveTimeSeconds: 30
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "bgp-openbsd"

The peering itself is a CiliumBGPClusterConfig: my ASN, a node selector, and one peer pointing back at the router from section 3. Note how it closes the loop with the gateway’s bgpd.conf (ASNs in mirror image, and every node in the gateway’s neighbor list must also match this nodeSelector):

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux # every node — including the control plane
  bgpInstances:
    - name: cilium-bgp
      localASN: 65030
      peers:
        - name: puffy # OpenBSD's mascot earns its place
          peerAddress: 10.30.0.1
          peerASN: 65000
          peerConfigRef:
            name: openbsd-peer-config

One gotcha worth knowing up front: these CRDs ship with schema defaults (ebgpMultihop, connectRetryTimeSeconds, and the timer/graceful-restart values). The API server injects them into your live objects, so kubectl get -o yaml shows fields you never wrote.

Seeing it work

After applying, the router tells the whole story:

# doas bgpctl show
Neighbor                   AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
k0s-m01                 65030        169        173     0 01:23:00      4
k0s-m02                 65030        170        176     0 01:23:06      4
k0s-m03                 65030        170        179     0 01:23:07      4
k0s-w01                 65030        170        173     0 01:23:02      4
k0s-w02                 65030        170        179     0 01:23:10      4
k0s-w03                 65030          0          0     0 01:24:50 Connect

Six established sessions (my control plane peers too, see the k8s_nodes macro above), each with a fistful of prefixes received from Cilium. On the cluster side:

# kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- cilium bgp peers
Local AS   Peer AS   Peer Address    Session       Uptime     Family         Received   Advertised
65030      65000     10.30.0.1:179   established   1h26m13s   ipv4/unicast   8          4

And in the pool, the addresses my LoadBalancer Services actually pulled: .129 for the shared-mode ingress, .130 for the Gateway API entrypoint that serves every *.neozeon.ca hostname, .131 for the pf-dashboard stack. Create a new LoadBalancer Service and a new IP appears in the pool, a route appears in bgpctl show rib. And because the gateway’s routes cover the whole /24, a curl from any machine on the network works as a smoke-test. No port forwarding, no reverse-proxy hop just to reach a cluster service from the LAN.

Further reading

• Cilium’s BGP control plane guide and LB IPAM docs — the canonical reference for the four CRDs above (and the ECMP rabbit hole you’ll want before advertising Service IPs from every node).
• OpenBSD’s bgpd.conf(5) and bgpd(8) (which covers the bgpctl client) — the filtering language deserves a read even for a setup this small.
• The pf docs if you want to understand why those BGP pass rules are written the way they are.
• k0s documentation for the distro itself — BGP lives entirely in the CNI, so any conformant cluster works.

Wrapping Up

The complete chain is: pf passes TCP 179, OpenBGPD accepts the sessions and the advertised prefixes, and four small CRDs make every Cilium node a tiny router that knows exactly which IPs it owns. OpenBSD’s minimalist, secure-by-default philosophy combined with Cilium’s eBPF datapath makes homelab networking behave like a cloud VPC and every piece of it fits in about a page of Git.