Bare-metal Kubernetes in HA, without kidding yourself

On a cloud, Kubernetes high availability is bought: the control plane is managed, the load balancer is an API object, storage is a class you reference. On bare metal, each of those three conveniences becomes something you have to build yourself and that is where most tutorials stack seven components to cover three jobs. This post builds an HA cluster with the defensible minimum: a control plane that survives the loss of a node, an API VIP with no external load balancer, a network that also carries the LoadBalancer and storage replicated across the nodes’ own disks. Three pieces covering the three single points of failure and not one more, because a cluster you do not understand is not available, it is on borrowed time.

The three things that must not be singular

HA is not a property of the cluster, it is the absence of a single point of failure on three precise axes, each with its own mechanics. The first is the control plane: the API server and the etcd that holds the state, which must tolerate the loss of a node without the cluster becoming unreadable. The second is the entrance: the address you talk to the API on and the one traffic reaches the Services on, neither of which should hang on a single machine. The third is storage: a PersistentVolume that survives the death of the node that carried it. Everything else, the application workloads, repairs itself the moment those three hold, because that is precisely the job Kubernetes does well.

The control plane: odd and empty of workload

etcd is a quorum store, so the number of control-plane nodes is reasoned like a vote: you need a living majority to write. Three nodes tolerate the loss of one, five tolerate two and even numbers are a trap, because four nodes still tolerate a single failure while offering one more machine to fail and to corrupt. So you stand up three control planes for a normal cluster, five when the stakes justify surviving two simultaneous losses, never two or four. And you schedule no application workload on them: the control plane holds the keys to the cluster and a vulnerable app co-located with etcd is a direct path to the entire state.

This is where k0s earns its place. It is a single binary that bundles the whole control plane and its etcd, declares itself in a few lines and is driven across nodes by k0sctl from a plain inventory file. The neutral fallback exists if you prefer it, a vanilla kubeadm on Debian with stacked etcd and the whole architecture below maps onto it term for term; k0s only removes the housekeeping that kubeadm leaves on your shoulders.

The API VIP, with no external load balancer

A three-control-plane cluster has three API addresses and you need one stable address in front of them, otherwise the kubeconfig and the kubelets point at a machine that can die. The reflex is to stand up a HAProxy and a keepalived alongside, which is two more daemons to install, configure and watch for a function the cluster can carry itself. k0s bundles it: Control Plane Load Balancing (CPLB) runs a keepalived in VRRP across the control planes and holds a virtual IP that floats to a living node.

# k0s.yaml, on each control plane
spec:
  network:
    controlPlaneLoadBalancing:
      enabled: true
      type: Keepalived
      keepalived:
        vrrpInstances:
        - virtualIPs: ["192.0.2.10/24"]
          authPass: "a-cluster-secret"

This is the first KISS call of this post: the API VIP is a piece of the cluster, not infra on the side. The 192.0.2.10 becomes the API address in every kubeconfig and for every worker and it survives the loss of the control plane that carried it. CPLB pairs with Node-Local Load Balancing, which has each worker spread its own API calls across the living control planes; the two are built to coexist and together they remove the last excuse for an external load balancer.

The network: Cilium and one component fewer

For the CNI, the debate has been settled for a while: Cilium won the eBPF ground, with NetworkPolicy and L7, the kube-proxy replacement, BGP and Hubble observability in a single project. Calico remains a serious, more classic choice, especially if native BGP is already your world, but for a fresh cluster Cilium is the state of the art.

On k0s, you unplug the bundled CNI and kube-proxy to make room:

# k0s.yaml
spec:
  network:
    provider: custom
    podCIDR: "10.244.0.0/16"
    serviceCIDR: "10.96.0.0/12"
    kubeProxy:
      disabled: true

Then Cilium takes over from kube-proxy in eBPF, pointing explicitly at the CPLB VIP because a worker must reach the API before the pod network exists:

helm install cilium cilium/cilium -n kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=192.0.2.10 \
  --set k8sServicePort=6443 \
  --set ipam.mode=kubernetes

The Service LoadBalancer, without MetalLB

Off cloud, a type: LoadBalancer Service stays pending forever, because nobody assigns or announces the external IP. The reflex, again, is to add MetalLB. And this is the call where your question “what would KISS say” has a clean answer: since Cilium already runs the network, having it also do the LoadBalancer avoids a second project doing the same job.

Cilium splits the two halves of the problem cleanly. IP assignment is LB-IPAM, a pool declared as a CRD:

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata: { name: pool-lan }
spec:
  blocks:
  - start: "192.0.2.200"
    stop:  "192.0.2.250"

Announcement, in turn, depends on your network and it is the only real choice to make. On a flat LAN, L2 Announcements are enough: Cilium answers ARP for the LoadBalancer IPs, a node elected by lease carries each Service’s traffic and no router needs to know anything.

apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata: { name: l2-lan }
spec:
  loadBalancerIPs: true
  interfaces: ["^eth[0-9]+"]

On a routed network, you prefer BGP: the CiliumBGPClusterConfig peers the nodes with your routers and announces the Service IPs as routes, which spreads better and does not hang on a single elected node. Two honest caveats: L2 Announcements are still marked beta and require the kube-proxy replacement, which we enabled anyway and their leader election chatters with the API, so on a large cluster you raise the Cilium client rate limits. MetalLB becomes relevant again only if you stay on a CNI that does not do the LoadBalancer, typically pure Calico; the moment Cilium is there, it is a piece you spare yourself.

Storage: Rook-Ceph on the nodes’ disks

A PersistentVolume in HA means the data is replicated off the node that serves it and on bare metal the mature answer is Ceph, orchestrated inside the cluster by Rook . Ceph’s constraints dictate the same arithmetic as etcd: it wants three monitors for a quorum, a host-level failure domain and three-copy replication to survive the loss of a node without losing or blocking the data. Concretely, three storage nodes minimum, each with at least one raw disk, unpartitioned and with no filesystem, because Ceph wants the whole device to lay its OSDs on.

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata: { name: replicapool, namespace: rook-ceph }
spec:
  failureDomain: host
  replicated: { size: 3 }

The rest is the usual walk: the Rook operator, a CephCluster that declares the nodes and their disks, then a StorageClass backed by the pool so the PVCs serve themselves. On a cluster already built on Proxmox and Ceph, the sensible alternative is to consume the existing external Ceph through the CSI rather than stand up a second one inside Kubernetes, which cleanly separates storage from compute; but for a cluster that must stand alone, Rook-Ceph is the complete answer.

The numbers that matter

Three comes back everywhere in this post and it is no accident: three control planes for an etcd quorum that tolerates one loss, three Ceph monitors for the same reason, three copies of the data to survive a dead node. Five is the step above, when surviving two simultaneous failures justifies the extra machine and consensus latency. Even numbers buy nothing: four control planes still tolerate a single failure, for one more unit of attack surface and consensus cost. And the workers are free: they hold only what you schedule on them, not the state, so you run as many as the load demands.

Where to land

Three control planes empty of workload, five if the stakes deserve it, never an even number, because etcd votes and a majority must survive. The API VIP through k0s’s CPLB, so the API address is a piece of the cluster and not a HAProxy on the side. Cilium as CNI with the kube-proxy replacement and the Service LoadBalancer carried by that same Cilium, in L2 on a flat LAN or in BGP on a routed network, so you run one network component instead of two. Rook-Ceph on three nodes with raw disks, host failure domain and three copies, so the data survives the node. The throughline is that each single point of failure is covered by exactly one piece, chosen because it replaces two. The state of the art, on bare metal, is not the most components, it is the fewest that actually hold.

[edit: 2026] The Ansible implementation of all this: the k0s and microk8s roles.