blog.6admin.io
guest@6admin.io:~$
devops, docker, foreman, kubernetes, migration, swarm

Docker Swarm to Kubernetes: why I abandoned Swarm

Docker Swarm works well as a lightweight orchestrator for small-to-medium deployments. It is simple, mature and the entry barrier is low. But it hits structural walls on security and advanced container management. Here are the four.

No rootless mode

Docker rootless mode (dockerd-rootless-setuptool.sh with slirp4netns/RootlessKit) is incompatible with Docker Swarm. The official documentation explicitly lists:

  • Overlay networks: Swarm relies entirely on overlay (VXLAN) for inter-node communication. Without overlay, services on different nodes cannot reach each other. Swarm becomes a single-node-only tool.
  • AppArmor: no MAC (Mandatory Access Control) enforcement on containers.
  • Checkpoint/Restore: no CRIU snapshotting, no live migration, no stateful recovery.

This is a fundamental architectural constraint: Swarm requires kernel-level networking primitives (netns, veth, VXLAN) that rootless mode cannot provide. slirp4netns and RootlessKit operate in user namespaces incompatible with kernel-level overlay networking.

Reference: Docker Rootless - Known Limitations

The limited security model

  • No Pod Security Admission / Pod Security Standards (the K8s equivalent).
  • No native network policies (no L3/L4 ingress/egress filtering between services).
  • No native secret rotation.
  • No admission controllers to enforce policies at deploy time.
  • userns-remap is the best available mitigation, but it is not rootless.

No native autoscaling

Swarm has no horizontal pod autoscaler. Scaling is manual (docker service scale) or requires external tooling with no standard API.

MACVLAN pinned to a single node

In the project, DHCP and TFTP services require direct access to the physical network for PXE boot. Docker Swarm cannot combine MACVLAN with overlay networks: a MACVLAN service is bound to a specific host interface and cannot be scheduled across multiple nodes. This forces a single-node deployment for all network boot services, which is a single point of failure with no horizontal scaling possible.

The storage limitations

  • No CSI (Container Storage Interface).
  • No dynamic volume provisioning.
  • No storage classes or persistent volume claims.
  • Volume management is local or requires manual NFS/Ceph setup.

The Swarm / MicroK8s / K8s comparison

                    Swarm                  MicroK8s                  K8s
Complexity          Low                    Medium                    High
Installation        Built into Docker      Snap, easy                Complex
Cluster             ~50 nodes max          1-100 nodes               1000+ nodes
HA                  Basic                  Via clustering            Advanced
Networking          Built-in overlay       Calico, Flannel, etc.     Multi-CNI
Storage             Basic volumes          CSI drivers               Full CSI
Security            Basic TLS              K8s features              RBAC, PSS, NetworkPolicies
Ecosystem           Limited                Growing                   Very large

The argument that is not scaling

The argument you always hear against Kubernetes is “we are not Netflix”. But scaling is not Kubernetes’ first interest. Its first interest is replacement.

Before Kubernetes, what you manage by hand: load balancing (HAProxy + Keepalived + a floating IP), service discovery (an Ansible inventory or Consul), rollouts (a serial:1 script that drains, updates, reconfigures), storage (a manual mount per machine), certificates (static, wired into the lifecycle), secrets (Ansible Vault), cronjobs (crontab + lockfile), the firewall (iptables per machine, updated on every topology change), logs (logrotate per application), packaging (rpm, deb, tar.gz in /opt, systemd or a scruffy init script).

Kelsey Hightower summed it up: “Kubernetes is the sum of all the shell scripts, and practices, system administrators have cobbled together over time, presented as a single system behind a declarative set of APIs.”

Kubernetes replaces all of that with a few YAML files: Deployment, Service, Ingress, ConfigMap, Secret. kubectl apply and it is deployed. Service discovery, load balancing, scaling, secrets, cronjobs, firewalling, logging, fault tolerance: built-in. The format is declarative, standard and portable across clouds and enterprises.

The question is not “are you big enough for Kubernetes”. The question is: how much time have you spent rebuilding by hand what Kubernetes provides off the shelf?

The decision

k0s single-node to start, then multi-node multi-site as the fleet grows.

The migration effort

Four active stacks to migrate: edge (Traefik), net (PowerDNS, DHCP, TFTP, Squid), obs (VictoriaMetrics, Grafana , OTEL), foreman (Foreman, Smart Proxy, PostgreSQL). Each stack has its own complexity and the MACVLAN network (DHCP/TFTP) remains the critical point: Kubernetes does not handle MACVLAN natively, it requires Multus CNI.

The effort is estimated in weeks, not days and the path is progressive: k0s single-node first, multi-node next, multi-site last.

[edit: 2026] The Ansible implementation of all this: the k0s and microk8s roles.