blog.6admin.io
guest@6admin.io:~$
devops, git, gitops, nix, podman, rust, yggdrasil

Project 8: a cloud where the only authority is signed git history

Project 8 (Apache-2.0, v0.6.0) started from an uncomfortable question: why does every “self-hosted cloud” still need a controller, a quorum protocol and a token service to function? The answer in 8’s design: it does not need them. Each node is sovereign and equal. It joins a mesh, pulls its desired state from a signed git repository, verifies the signatures and converges locally. No master, no consensus token, no blockchain. Authority lives in Ed25519-signed git history, or it lives nowhere.

The stack, layer by layer

The README carries a five-layer mermaid; the load-bearing pieces:

  • L1, the mesh: Yggdrasil IPv6 overlay. Every node gets a stable, encrypted, self-healing address and node identity is a key, not an IP. The gateway’s routing table only accepts Yggdrasil-shaped addresses (0200::/7): malformed entries are skipped, never proxied to loopback or link-local. That is an SSRF guard hiding in a validation rule.
  • L2, the source of truth: Radicle (git P2P), repo rad:z4GctVJZH9WcgG23xVx2kSfieapPP. Nodes pull and clone it over the mesh; the daemon runs as a systemd user unit with NID peering. Nothing is ever pushed to a node by a privileged actor: the flow is pull-only by construction (ADR-0003).
  • L3, the agent: plan → diff → apply. A Python 3.12 prototype first (76% to 89% coverage), then a Rust port (ADR-0018) shipping as a static binary at 81% line coverage, with py+rs parity tests for every behavior: apply dry-run vs real report, podman /k0s runtime arg generation, signed-commit relay with attribution trailers and rollback on non-converging push.
  • L4, workloads: rootless Podman plus one k0s single-node cluster per machine (--single: kine+SQLite, no shared control plane, ADR-0007). Each node’s k8s depends on nothing else; the mesh is the only cluster.
  • L2 security, the fail-closed part: eight apply syncs the Radicle clone and verifies every new commit signature against state/trusted-keys/ before converging; a verification failure exits 2, the node keeps its last good state. The api also builds its allowed_signers file from a single snapshot of trusted keys (the original double-read was a TOCTOU waiting to matter).

Bootstrap day, or the twenty bugs of first convergence

The CHANGELOG’s v0.5.0 entry is one line and one lesson: “Mesh bootstrap green (2026-08-29)”, with ansible-playbook site.yml converging three libvirt VMs to failed=0 after ~20 first-run bugs across 9 roles and the inventory. The ones worth reading twice:

  • cloud-init binds network config to the first-boot MAC, so random MACs orphaned nodes: NIC MACs are now pinned (52:54:00:8a:00:0X) and the dnsmasq reservations make IPs and DNS stable across applies.
  • Two dnsmasq instances do not share a subnet: libvirt’s stock default network already owns 192.168.122.0/24, so the eight-net moved to 192.168.123.0/24.
  • Ubuntu 24.04’s AppArmor kills QEMU on type='volume' disks (virt-aa-helper in libvirt 10.0.0 does not resolve volume references, Permission denied): documented workaround in the quickstart, because “works on my machine” is a promise the next bootstrap day will collect.
  • yggdrasilctl without -json returns a table, which parsed as empty: the peers API reported nothing on a mesh that was up.
  • NixOS’ stdenv stripped the k0s binary overlay: a payload-less binary, “no payload attached”, on the declarative node.

None of these are exotic individually; the twentieth one is where a homelab either grows documentation discipline or dies of it.

The boring tools, deliberately

Provisioning is OpenTofu + Ansible against a dynamic libvirt inventory; the NixOS node is baked as qcow2 with overlays and deployed by colmena (ADR-0014); secrets ride fnox with age keys; supervision is a per-node VictoriaMetrics/VictoriaLogs/Grafana stack composed with kustomize helmCharts (ADR-0019) and Wazuh FIM agents on every node (ADR-0020, NixOS via a native agent wrapping the official .deb). 23 ADRs record every one of these choices, including the rejected ones. The license rule is written in an ADR too (0004): OSI-free licenses only, no exotic runtime to bet on.

The only authority is the log

  • Pull-only is the whole security model. A node that only pulls signed history needs no inbound authority, no token, no trust in the network it sits on.
  • Bootstrap bugs are config bugs wearing costumes. MAC pinning, subnet collisions, AppArmor, JSON flags: nothing failed in the new code; everything failed at the boundaries between tools.
  • Prototype in Python, port to Rust, keep the parity tests. The parity suite is what makes the port a refactor instead of a rewrite.
  • The AI gateway is a normal HTTP service. SSE streaming proxied from a detached thread (a slow consumer must not block the accept loop), 16 MiB bounded bodies, Yggdrasil-only routing. The AI part of the design fits in three paragraphs; the reliability part fills the CHANGELOG.