Serving OpenShift VIPs from OpenBSD: relayd, pf, and unbound

OpenShift is unusual among Kubernetes distros in how much it demands from your network before it will install itself: three virtual IPs (API, internal API, and wildcard ingress), DNS that resolves all three before the first node boots, and a healthcheck contract for your load balancer that is spelled out almost to the millisecond in the install docs. In the cloud this is someone else’s problem, on bare metal it is yours.

In this post I’ll build the entire gateway side of an OpenShift compact cluster: the pf rules, a validated relayd.conf that satisfies OpenShift’s health-check spec exactly, a tiny shell probe for the ingress routers, and the unbound DNS entries the installer will nag you about. The cluster deployment itself (ignition, bootstrap, all that) is out of scope, this is the part that lives on your router forever.

My platform is a Protectli v1210 running OpenBSD 8.0, the same little box that already routes my LAN (igc1, 172.16.0.0/22) and peers BGP with my other clusters.

The Architecture

OpenShift needs three addresses fronting a three-node compact cluster:

  • API VIP (172.16.0.70) — fronts port 6443 (kube API), 22623 (Machine Config Server / Ignition), and 22624 (MCS internal)
  • Ingress VIP (172.16.0.75) — fronts ports 80/443 for every *.apps.ocp.redcomet.ca route
  • The three nodes (ocp1–ocp3, 172.16.0.71–.73) — masters that also run workloads
 clients ──6443/22623──►  ┌────────────────────────────────┐
                          │ OpenBSD gateway (Protectli)    │
 clients ────80/443───►   │  pf (rdr) + relayd + unbound   │
                          │  VIPs aliased on igc1          │
                          └──────────┬─────────────────────┘
                          ┌──────────┴──────────┐
                       ocp1 .71 ── ocp2 .72 ── ocp3 .73

The key decision: relayd runs as an L4 redirect, not an L7 proxy. In redirect mode relayd arranges a pf rdr-to, meaning packets are redirected, not terminated. The client’s source IP arrives intact at the router pods, and TLS passes through untouched. relayd never sees a key, and your apps never need PROXY protocol or X-Forwarded-For gymnastics just because a load balancer exists.

1. Aliases and pf

First the VIPs must exist on the wire. They’re aliases on the LAN interface, persisted in /etc/hostname.igc1:

inet 172.16.0.1 255.255.252.0 172.16.3.255
inet 172.16.0.70 alias
inet 172.16.0.75 alias

Then pf. relayd injects its own rdr-to rules into an anchor, so the anchor line is the one requirement; everything else is ordinary filter policy:

# --- OpenShift external LB (gateway side) ---
ocp_if     = "igc1"
ocp_nodes  = "{ 172.16.0.71, 172.16.0.72, 172.16.0.73 }"
ocp_apivip = "172.16.0.70"
ocp_ingvip = "172.16.0.75"

# relayd's redirect rules land here automatically
anchor "relayd/*"

# Explicit pass rules so ordering stays sane in a deny-by-default ruleset
pass in quick on $ocp_if inet proto tcp from any to $ocp_apivip port { 6443 22623 22624 }
pass in quick on $ocp_if inet proto tcp from any to $ocp_ingvip port { 80 443 }

# ... and never, ever forward 1936 (router stats/metrics)

Two details worth calling out. The pass rules are technically redundant, a relayd redirect passes the traffic it redirects but being explicit keeps your rule ordering auditable when the ruleset grows. And the 1936 comment deserves emphasis: that’s the HAProxy stats port on every router pod, and if it’s reachable from outside the cluster you’re leaking your whole routing table as a metric. The health check for it runs from the gateway itself, so no forward rule is ever needed.

2. relayd: OpenShift’s health-check contract, in 60 lines

OpenShift documents its load balancer expectations precisely: probe the kube API with a TLS GET /readyz expecting 200, the MCS with a TLS GET /healthz, probe every 5–10 seconds, and a failed node must leave the pool within 30 seconds. Long-lived connections (oc exec, oc port-forward, watches, websockets) must also survive. relayd can satisfy every clause. Here’s the config:

# /etc/relayd.conf — OpenShift external LB

apivip="172.16.0.70"
ingvip="172.16.0.75"

interval 10           # spec: probe every 5-10s
timeout 3000          # check timeout ms; generous for TLS handshakes
log state changes

table <ocp_api>   { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }
table <ocp_mcs>   { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }
table <ocp_mcs24> { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }
table <ocp_http>  { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }
table <ocp_https> { 172.16.0.71 retry 2, 172.16.0.72 retry 2, 172.16.0.73 retry 2 }

# kube API: TLS GET /readyz -> 200 (docs-compliant unauthenticated readiness)
redirect "ocp-api" {
 listen on $apivip port 6443
 forward to <ocp_api> port 6443 timeout 1500 \
  check send "GET /readyz HTTP/1.0\r\n\r\n" expect "HTTP/1.[01] 200*" tls
 session timeout 3600
}

# Machine Config Server / Ignition: TLS GET /healthz -> 200
redirect "ocp-mcs" {
 listen on $apivip port 22623
 forward to <ocp_mcs> port 22623 timeout 1500 \
  check send "GET /healthz HTTP/1.0\r\n\r\n" expect "HTTP/1.[01] 200*" tls
 session timeout 3600
}

# MCS internal (kubelet config-driver): TCP only
redirect "ocp-mcs24" {
 listen on $apivip port 22624
 forward to <ocp_mcs24> port 22624 timeout 1500 check tcp
 session timeout 3600
}

# Ingress: L4 passthrough on 80/443, readiness probed on router stats port 1936
redirect "ocp-http" {
 listen on $ingvip port 80
 forward to <ocp_http> port 80 timeout 1500 \
  check script "/usr/local/bin/ocp_router_check"
 session timeout 3600
}

redirect "ocp-https" {
 listen on $ingvip port 443
 forward to <ocp_https> port 443 timeout 1500 \
  check script "/usr/local/bin/ocp_router_check"
 session timeout 3600
}

Reading the highlights:

  • check send/expect ... tls speaks just enough HTTP over TLS to ask the API server if it’s ready. No certificates, no client auth, no dependencies. Interval 10 with per-host retry 2 puts a dead node out of the pool in roughly 30 seconds: exactly the documented maximum, no tighter.
  • session timeout 3600 beats relayd’s 600-second default so an oc rsh into a debugging pod doesn’t evaporate mid-sentence.
  • Five tables, not one — the ingress table membership should differ from the API table the moment you add dedicated infra nodes; starting with five identical tables means adding a node later is one line, not a redesign.

The ingress check: where relayd gets weird

The router readiness probe lives on port 1936 (GET /healthz/ready on the HAProxy stats interface) but relayd health checks always target the forwarded port, and there’s no check-port override (haproxy has one; relayd doesn’t). The escape hatch is check script, which also comes with the single most counterintuitive rule in relayd.conf(5):

the script “expects a positive return value on success and zero on failure”

Exit nonzero = healthy. Exit 0 = down. It’s inverted from every monitoring convention you know, and no syntax error will warn you. Misread it and every node reports down forever. Here’s the probe, including two more traps it dodges:

#!/bin/sh
# /usr/local/bin/ocp_router_check <ip>
# relayd 'check script' probe: router readiness on stats port 1936
h="$1"
[ -n "$h" ] || exit 0
r=$( (printf 'GET /healthz/ready HTTP/1.0\r\n\r\n'; sleep 0.2) \
     | /usr/bin/nc -w 1 "$h" 1936 2>/dev/null | /usr/bin/head -n1 )
case "$r" in
  *" 200"*) exit 1 ;;   # positive -> relayd marks host UP
  *)        exit 0 ;;   # zero    -> relayd marks host DOWN
esac
  • The script runs as _relayd with a minimal PATH.
  • It requests a plain path because httpd’s GET /healthz/ready on a directory returns a 301 redirect, and this probe doesn’t follow redirects.

Install both files, then rcctl enable relayd && rcctl restart relayd, and watch it think:

# relayctl show hosts
172.16.0.71   100%  6/6   up    (3.12s)
172.16.0.72   100%  6/6   up    (2.87s)
172.16.0.73   100%  6/6   up    (3.01s)

3. unbound: the DNS the installer will hold you hostage over

OpenShift resolves api.<cluster-domain> and api-int.<cluster-domain> from inside the cluster. MCO and etcd will not tolerate api-int failing to resolve and every application route hangs off a wildcard for the apps domain. The three records that matter, in /var/unbound/etc/unbound.conf:

server:
    # ... usual homelab server config ...

    local-zone: "ocp.redcomet.ca." redirect
    local-data: "api.ocp.redcomet.ca.       3600 IN A 172.16.0.70"
    local-data: "api-int.ocp.redcomet.ca.   3600 IN A 172.16.0.70"
    local-data: "*.apps.ocp.redcomet.ca.    3600 IN A 172.16.0.75"

Plus the boring-but-vital node records (ocp1–ocp3 → .71–.73) in the regular LAN zone so the hostnames, the DHCP leases, and relayctl output all agree.

The wildcard is the one everyone forgets: without *.apps pointing at the ingress VIP, your first oc expose “works” perfectly and then times out, and the error message will not mention DNS. Note that redirect is what makes the wildcard legal, unbound only synthesizes wildcard answers inside a declared redirect zone. If the records silently don’t resolve, that’s usually why.

Verifying the whole path before OpenShift exists

The beauty of doing the LB first is that you can prove it before a single RHCOS disk is written:

# relayctl show hosts        # all nodes up in all five tables
# relayctl show redirects    # all five redirects "active"
$ curl -k https://172.16.0.70:6443/readyz      # mock 200 (on the node!)
$ dig +short api.ocp.redcomet.ca               # 172.16.0.70
$ dig +short foo.apps.ocp.redcomet.ca          # 172.16.0.75

The failover test is the confidence-builder: pull one node’s IP off the wire and watch relayctl show hosts drop it to down in all five tables within 30 seconds while the redirects stay active with two hosts. Then re-add it and watch the check counts climb back.

One OpenShift-specific dance remains, and it’s easy to trip over on install day: the bootstrap host serves Ignition on port 22623 to the masters. If your MCS table only contains the three nodes, the masters boot, can’t fetch their ignition, and everyone stares at a timeout. Add the bootstrap host’s IP to <ocp_api> and <ocp_mcs> before you start (relayctl or edit and reload), and remove it when the install completes.

Wrapping Up

Three small pieces on one unassuming x86 box: pf aliases and rules, a 60-line relayd config with one inverted-exit-code shell probe, and three unbound records. An OpenShift installer that normally assumes it’s running in a cloud now finds a compliant enterprise-style load balancer waiting on the LAN. relayd’s redirect mode means OpenShift never learns a proxy exists: real source IPs, untouched TLS, no application changes.

Is the single gateway a load-balancer SPOF? Yes, and for a homelab that’s a fine trade with a clean upgrade path: a second OpenBSD, carp(4) VIPs, relayd on both if I ever choose to go down that path.