Kubernetes, or not

Kubernetes solves a real problem, scheduling many workloads across many machines while healing itself and it solves it well. The trap is not the tool, it is reaching for it to solve a problem you do not have yet, paying up front a permanent operational tax for resilience you are not using. Because Kubernetes’ real expense is not conceptual, it is operational: the control plane to keep alive, the CNI to understand, the storage to stand up, the quarterly API updates to follow and an attack surface with its own hardening posts. Before signing up for that, it is worth walking down the floors and stopping at the first one that carries the load, because the right level of abstraction is the lowest that solves your problem, not the most prestigious.

One host, a few services: systemd nearly suffices

The vast majority of deployments are a handful of services on a single machine and there Kubernetes brings nothing but a layer to operate. The reflex is docker-compose and for development or a bit of tinkering it is perfect: one file, up, down, you grasp it instantly. For production on one host there is something better and quieter, which is to let systemd do its job, since it is already there, already the supervisor, already the machine’s dependency and restart manager.

Podman and quadlets make that clean: you describe a container in a small unit file and the systemd generator turns it into a real service, with dependencies, restarts, logs in the journal and all of it rootless.

# /etc/containers/systemd/app.container
[Container]
Image=registry.example.org/app:1.4.2
PublishPort=8080:8080
AutoUpdate=registry

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

A systemctl daemon-reload later, app.service exists, starts at boot and recovers on its own and podman-auto-update.timer pulls new images without a thought. It is declarative, versionable and adds no daemon to supervise, since the supervisor is the init the machine already runs. For one host, this is the floor where most projects should stop.

A few nodes, still simple: Docker Swarm

When you genuinely need several machines, for availability or for load, the next step is not Kubernetes, it is Docker Swarm and I have already made its case in a dedicated post . Swarm does real clustering with a disarming honesty: mTLS between nodes and rotating certificates out of the box, an encrypted Raft store, real secrets, an overlay network encryptable with one flag and a mental model that fits on a page. You already wrote a docker-compose.yml, you deploy it as a stack and you get replicas, rolling updates and failover without having learned anything new.

The charge against Swarm is social more than technical, it fell out of fashion; but for three to a handful of nodes running not-too-many services, it carries the load for a fraction of the operating cost. The day what you ask of it exceeds what it can do, you will know and that is precisely the day the Kubernetes question becomes legitimate again.

Bigger, but not ready for k8s: Nomad and Consul

There is an intermediate floor the hype always skips, between Swarm and Kubernetes: scheduling many workloads across many nodes without swallowing all of k8s’s complexity. That is the ground of Nomad , HashiCorp’s scheduler, a single binary that places tasks across an estate, with the same odd-server Raft model you find everywhere, three or five for quorum. Nomad does not only schedule containers, either: it also launches raw binaries, VM images, Java jobs, which is exactly what you want when the estate is not fully containerized.

Consul comes alongside for what Nomad does not do: service discovery, internal DNS, the key-value store and a service mesh with mTLS through Consul Connect when you want to encrypt and authorize the flows between services. Together, docker or podman to run, Nomad to place, Consul to connect, they cover a large part of what people think they need Kubernetes for, with two binaries a human can hold in their head. The trade-off is plain: you do not get the k8s ecosystem, nor its operators, nor its catalogue of Helm charts, nor the reservoir of people who already know it. If it is that ecosystem you are after, then that is the real argument for k8s and it should be named as such rather than dressed up as a scale problem you do not have.

What the glue costs

The lower floors do have a cost people forget to count, though, because it appears on no invoice, the cost of the glue. As soon as you move past the demo docker compose up, you have to write yourself what Kubernetes ships out of the box and I speak from experience, my own Swarm setups having all ended up dragging their Makefile along.

A tiny Makefile often does the job without growing into a monster: validate, deploy per stack, check service health and run smoke tests. For example:

STACK ?= edge
export $(shell test -f .env && grep -v '^#' .env | xargs)

.PHONY: check up down ps smoke
check: ; docker compose -f stacks/$(PROFILE)/$(STACK)/docker-compose.yml config >/dev/null
up:    ; docker stack deploy --detach=true -c stacks/$(PROFILE)/$(STACK)/docker-compose.yml $(STACK)
down:  ; docker stack rm $(STACK)
ps:    ; docker stack ps $(STACK) --no-trunc
smoke: ; curl -fsS https://$(STACK).example.internal/healthz || curl -fsS http://127.0.0.1:8080

Validating and merging compose files per profile and TLS variant, linting the lot, rendering configurations from templates with envsubst, generating secrets, initializing the cluster and its networks, deploying stack by stack, looping until replicas are healthy before running smoke tests and providing the backup and restore that go with it. Each of these targets is simple, but their sum is an in-house tool, with its own conventions, bugs and documentation, that someone will have to maintain and every newcomer will have to learn.

Kubernetes ships these primitives off the shelf, readiness probes instead of the wait loop, Secrets and ConfigMaps instead of templates, Kustomize overlays instead of YAML merging, the rollout and its status instead of the deploy script. And above all, it makes them a standard, since an engineer who joins already knows kubectl rollout status, whereas they will never know your Makefile before spending a week reading it. This is not a scale argument, it is a maintenance and hiring argument and it deserves to be weighed with the same honesty as the operational tax.

The threshold where Kubernetes finally earns it

There is a point where the honest answer is “yes, take Kubernetes” and it is better to recognize it than to avoid it on principle. It is when several of these real needs stack up: many teams sharing an estate with genuine multi-tenancy and fine-grained RBAC, a need for metric-driven horizontal autoscaling, an ecosystem of operators you want to consume, a platform you are building for others that must expose a standard API, in-house glue that has become costlier to maintain than the cluster would be to run, or simply a scale where the number of nodes and services exceeds what a human tracks by hand. At that threshold, Kubernetes’ operational tax stops being wasted spend and becomes the price of a service you actually use and that is where standing it up properly in HA on bare metal makes full sense.

The symptom to flee is CV-driven architecture, the kind that puts Kubernetes in because it is the line you want on your profile and not because the load calls for it. It is paid for in complexity every day, for the benefit of one.

Where to land

One host and a few services is podman plus quadlet, so systemd supervises what it already knows how to supervise, with docker-compose staying perfect for development. A few nodes that want availability is Docker Swarm, real clustering for a fraction of the operating cost and a model you can keep in your head. Many workloads across many nodes without k8s’s complexity is docker or podman with Nomad to schedule and Consul to connect. And Kubernetes when you truly have the multi-tenancy, the autoscaling, the operator ecosystem or the scale that call for it, in which case you stand it up seriously. The rule is the same at every floor: you pick the lowest level of abstraction that solves the problem you have, not the one that decorates best. Going down costs less than pretending to know how to drive what you have no need to drive. At every floor too, there is the option of not keeping running what nobody uses: a service called once a week has no reason to stay warm, which is the whole point of scale-to-zero .

[edit: 2026] mcorbin wrote at the end of 2025 what this post only touched on, the main benefit of Kubernetes is not scaling but offering off the shelf what you would otherwise spend years rebuilding yourself and I agree all the more readily since my latest Swarm monorepo, two deployment profiles and seventeen stacks, fits in a Makefile of more than six hundred lines. The floors in this post still hold, but I now count the glue from day one.

[edit: 2026] The Ansible implementation of all this: the k0s role .