A persistent volume is not durable data

A few days after the log-driver DNS cascade , the same prototype node, the same power problem, but a failure of a different kind, sneakier, because it did not show up right away. After a hard shutdown of the hypervisor, PowerDNS’s MySQL database came back empty. The DNS records of the provisioned hosts, the A and PTR records created by Foreman, were gone. And nobody noticed, until an Ansible playbook failed hours later, unable to resolve a hostname.

A silent loss

This is the worst kind of failure, the one that raises no alert and surfaces at the worst possible moment. The base service records had reappeared on their own, because they are recreated idempotently on every deployment by the setup script; so I first believed the zone intact. But the hosts’ A and PTR records are created dynamically by Foreman and never re-seeded at deploy time: they no longer existed and nothing said so.

dig @127.0.0.1 ahost.devops.lab +short   # NXDOMAIN

The zone held only the service records, zero hosts. Foreman, on the other hand, had everything: the PostgreSQL database had survived, the hosts were still in the inventory. That asymmetry, Foreman’s PostgreSQL intact and PowerDNS’s MySQL wiped, is the heart of the story.

The mechanism and what I can prove

The hard stop rendered the MySQL datadir unusable and on reboot the container came back on an empty data directory, reinitialised, PowerDNS recreated its schema and the setup laid the base records back down. The datadir lived in a named, persistent Docker volume, so the loss does not come from a volume deletion and I verified that point before accusing anything: the project’s down only does a stack rm, which does not touch volumes and only an explicit, interactive cleanup command would have purged them.

The exact path of the corruption I cannot demonstrate with what I have: damage to the InnoDB tablespace, corruption of the guest filesystem on the virtual disk, a hypervisor write cache that does not guarantee fsync durability, all three are plausible. The established fact is sharp: the data did not survive the unclean stop and honesty means stopping there rather than embroidering a root cause the evidence does not support. The asymmetry between MySQL and PostgreSQL is itself a lead, a difference in flush guarantees between the two engines under a disk cache that lies about durability, but I leave it as a hypothesis to investigate, not a conclusion.

The real cause is not the power cut

The environmental trigger I had already covered in the previous incident. What makes this one serious is that the loss was total, silent and unrecoverable by construction and that comes down to three gaps that have nothing to do with the power.

The host records lived only in MySQL and were never regenerated at deploy time, so any loss of the volume meant the loss of the entire host DNS. There was no backup of that database: the volume was absent from the backup script, which only dumped two other services and the restore script actually referenced a volume name that did not exist, so a restore would have failed even if a backup had existed. And there was no detection: no alert, healthchecks commented out in the compose, a loss discovered by the accident of a failing playbook.

What saved it

Recovery hung on one thing, that Foreman is the source of truth and that it had survived. A Rebuild Config on the DNS part, as a bulk action over the managed hosts, reprojected the A records and the PTR once the reverse zone was recreated.

Along the way the incident laid bare a sleeping bug: the variable that defines the reverse zone was unset, so the reverse zone had never been created and the PTRs had never had anywhere to land. I had to set it, add it to the setup’s environment, redeploy and delete a stray zone created by the empty variable. That is the kind of detail you only see the day you have to rebuild everything.

The volume survived, the data didn’t

A persistent volume is not durable data: without fsync honoured end to end and without a backup, a hard stop can erase everything and the word “persistent” only protects against deletion, not against corruption. Data that is neither backed up nor regenerable from its source of truth is data at risk of permanent loss and here luck simply held in that the source of truth survived, which is not a strategy. A silent loss is worse than a visible outage, because a visible outage at least announces itself.

What I drew from it, concretely: a logical dump of the DNS database added to the backup and a tested restore, the healthchecks re-enabled, an alert on a canary record that stops resolving or on the zone’s record count collapsing and above all the formalisation of Foreman as source of truth and PowerDNS as a mere projection, with a one-gesture reprojection documented for any future loss. Durability on the hypervisor side, the disk cache that must honour fsync, remains the underlying point, coupled with the ordered shutdown already demanded last time.