Thirty Apps, One Repo: Anatomy of a GitOps Homelab Cluster

Thirty Apps, One Repo: Anatomy of a GitOps Homelab Cluster

Somewhere between running my first app on Kubernetes and running my thirtieth, the problem changed shape. It stopped being “how do I deploy this” and became “how do I know what my cluster is, and how do I change it without nuking & paving.” My answer is a single GitLab repo gitlab.com/caseyrobb/iac: one ApplicationSet generator, one allowlist per cluster, one thin wrapper chart per app, and a handful of infrastructure operators doing the rest. The production cluster, chaos (k0s, six nodes on mini PCs at home — no, the name is not prophetic; mostly), runs thirty-two apps out of that repo and has never had a kubectl apply from a human.

This post is a tour of the five design choices that make that work and a few scars, because every ignoreDifferences rule in my repo is the fossil of a fight ArgoCD and I had.

1. The monorepo, and an ApplicationSet that treats absence as policy

Everything (ArgoCD’s own bootstrap, all chart wrappers, all per-cluster values) lives in one repo. The tempting alternative is a chart repo per app or a “platform” repo plus an “apps” repo, and I tried fragments of both. The monorepo won for one reason: a change and its consequences get reviewed in one diff. Adding an app means a new directory and its allowlist entry in the same merge request; CI fails if either half is missing.

The entire fleet is generated by one ApplicationSet, whose generator is a matrix of clusters × that cluster’s app list:

generators:
  - matrix:
      generators:
        - list:
            elements:
              - clusterName: PATCH_ME # bootstrap patches in the real name
        - git:
            repoURL: https://gitlab.com/caseyrobb/iac.git
            revision: HEAD
            files:
              - path: "clusters/{{ .clusterName }}/apps.yaml"
template:
  spec:
    source:
      path: "apps/{{ .app }}"
      helm:
        valueFiles:
          - values.yaml
          - "../../clusters/{{ .clusterName }}/{{ .app }}.yaml"

The file that generator reads is the most important opinion in the repo:

# clusters/chaos/apps.yaml — app deploys ONLY if listed here.
- app: argocd
- app: cilium
  namespaceOverride: kube-system
- app: immich
- app: keycloak
  namespaceOverride: keycloak

The allowlist is the contract. Presence deploys; absence is a deliberate decision, not an oversight, and CI enforces the inverse direction too. Every chart under apps/ must be claimed by at least one cluster, so dead apps can’t accrete. The sandbox cluster is the same generator pointed at a different file, which means staging an app in prod is one line in a merge request, reviewed like everything else.

Values layering follows the same philosophy: each wrapper chart carries its values.yaml (defaults for all clusters) and an optional per-cluster overlay. ignoreMissingValueFiles: true means overlays exist only where they earn their place.

One honesty corner, because every GitOps blog pretends ignoreDifferences is for cowards: I have exactly three, and each one is a documented truce. Two Secrets whose data is written by the External Secrets controller (which ArgoCD can never observe as Synced, so the alternative is a permanent red badge and selfHeal ping-pong), and ServiceMonitors, where the prometheus-operator defaulting webhook rewrites fields the chart never sent, a perpetual OutOfSync that SSA applies and the webhook un-applies, forever. Each rule states its reason in a comment. IgnoreDifferences without a written reason is just drift with extra steps.

And because syncPolicy is automated + selfHeal, there’s one operational rule the whole cluster lives by: nothing is hot-patched. The fix for anything is a revert-MR to master. The emergency lever is git revert, and it’s the same lever you use on a normal Tuesday.

2. CNI is the platform: Cilium, Gateway API, and one door

Cilium does far more than pod networking on this cluster: kube-proxy replacement, native routing (which makes the BGP setup a post of its own, see the OpenBGPD writeup), Hubble for observability, Envoy as the L7 data-path, and the Gateway API implementation. It installs via the appset like any app, into kube-system via namespaceOverride, from a wrapper chart that just pins the upstream release.

Every public service on the cluster passes through one Gateway and its own one-screen HTTPRoute:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: neozeon-gateway
  namespace: kube-system
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.neozeon.ca"
      allowedRoutes:
        namespaces: { from: All }
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-neozeon-tls-secret

Adding an app’s public face is then pure declaration, an HTTPRoute in the wrapper chart mapping hostname + backend service to the shared gateway. No Ingress annotations zoo, no per-app TLS config, no load-balancer provisioning: the gateway’s LoadBalancer IP already exists (it’s one of the addresses Cilium advertises over BGP), and cert-manager already owns the wildcard cert. The route is the only per-app decision left, which is exactly how it should be.

3. Secrets: Vault as the source of truth that never touches git

The secret-management posture fits in one sentence, and that’s the point: no secret material is ever committed to git or posted to chat. What lives in git is references.

The chain: a HashiCorp Vault (one node, it’s a homelab :p) runs in-cluster on persistent storage; every credential lives under a chaos/ KV v2 mount, chaos/rustfs/rustfs-secret, chaos/keycloak/operator-client, and so on. External Secrets Operator syncs Vault into Secrets through a single ClusterSecretStore:

provider:
  vault:
    server: https://vault.neozeon.ca
    path: chaos
    version: v2
    auth:
      kubernetes:
        mountPath: kubernetes
        role: external-secrets
        serviceAccountRef:
          name: external-secrets

That auth.kubernetes block is the quiet hero. ESO authenticates to Vault by exchanging the controller’s own projected service-account token. No static token in git, no static token in a Secret, no rotation ceremony. When I want a new credential in the cluster, the flow is: put it in Vault, chart’s ExternalSecret template names the path, ArgoCD syncs. The cluster learns secrets the same way it learns everything else: from a merge request.

The hygiene rules that grew around this are boring and absolute. Secrets that ESO owns get /data ignored in ArgoCD (above), so selfHeal and ESO never fight. Backups (CNPG’s ScheduledBackup to an in-cluster RustFS S3 bucket) read per-namespace ExternalSecrets for their S3 credentials, so the backup path and the application path share one Vault source. One rotation, both fixed. Finally, the credential for the notification system, pushover, follows the same discipline living in Vault.

4. Storage: ZFS on the NAS, iSCSI into the cluster, and the longhorn war I retired

Storage is the least fashionable infrastructure decision and the one that decides whether you keep the cluster. Mine is deliberately old-fashioned: a Debian 13 NAS with a raidz2 ZFS pool exports volumes over iSCSI, and democratic-csi in the cluster converts iSCSI targets into PersistentVolumes with a StorageClass nas-iscsi (zfs-generic-iscsi under the hood) that everything uses:

driver:
  existingConfigSecret: democratic-csi-iscsi
  config:
    driver: zfs-generic-iscsi
storageClasses:
  - name: nas-iscsi
    reclaimPolicy: Delete
    allowVolumeExpansion: true

The cluster previously ran Longhorn, the Kubernetes-native block storage product with the lovely UI and the replica-rebalancing opinions. On homelab hardware, it was a second distributed system to babysit in order to run the other distributed systems: upgrade windows for the storage layer, replica anti-affinity drama whenever a node rebooted for kernel patches, and a recovery procedure that required the cluster to be healthy to help the cluster be healthy. I decommissioned it, migrated the volumes, and the config above is what replaced it: ZFS snapshots and sends on the NAS, iSCSI to the nodes, plain TCP over the switch (homelab-fast enough), and one k0s quirk: the node.kubeletHostPath has to point at /var/lib/k0s/kubelet, because k0s doesn’t pretend to be kubeadm.

The trade is honest: no replicated volumes, so the NAS is a single point. I accepted it the same way I accepted a single gateway router, with the same upgrade path in my back pocket: a second NAS and DRBD or just very good snapshots. What I got back is that storage now fails in ways I understand in one layer instead of three. RustFS (an S3-compatible server, itself running on these very volumes) then serves as the backup target for the Postgres clusters (CloudNativePG writes Barman-format base backups and WAL straight into it nightly), which means the storage stack is: ZFS → iSCSI → PVs → apps and app-backups, with each hop boring enough to debug at 23:00.

5. Certificates and DNS: the wildcard that makes the edges line up

Two small operators finish the perimeter, and they’re best explained together because they solve one problem from both ends: an app should become reachable by writing a route, not by filing tickets with yourself.

cert-manager holds the ACME relationship with Let’s Encrypt via a ClusterIssuer in the wrapper chart, using the DNS-01 challenge through Cloudflare, which is what makes the *.neozeon.ca wildcard certificate in section 2 renewable forever without exposing an HTTP path:

solvers:
  - dns01:
      cloudflare:
        apiTokenSecretRef:
          name: cloudflare-api-token-secret # from Vault, via ESO
          key: api-token

Yes: even cert-manager’s Cloudflare API token is an ExternalSecret. There is no bootstrap exception, which is why this section is short.

external-dns reads Gateway API objects: gateway-httproute, gateway-tlsroute, gateway-tcproute, policy sync and maintains the Cloudflare DNS records that mirror them. Create a route in git, get a DNS record through the same tunnel the app uses; delete the route, the record goes away. The cluster’s public surface and its DNS are the same list, which is the kind of statement you should distrust in any blog post, including this one.

The asterisk: the Cloudflare zone deliberately has no wildcard record. Every hostname is a real DNS entry (external-dns’s doing) plus a manual Cloudflare Tunnel route (mine). That manual step allows me to selectively expose an app over the internet versus just being available via a RFC1918 address on my LAN.

Wrapping Up

Step back and the architecture is almost aggressively boring: one repo that a generator matrix fans out into thirty-plus apps, one CNI wearing four hats, one Gateway, one Vault, one NAS, one wildcard certificate. The interesting parts aren’t components, they’re the contracts: the allowlist that makes absence mean something, the wrapper chart that pins one version while upstream drifts, the ExternalSecret that makes a credential deployable by ArgoCD, and the selfHeal that makes every operator habit (like hot-patching) impossible to form.

Boring is the goal. Boring is what “production” costs when you run it on mini-PCs in my basement and want the weekend back.