The log driver that stopped the container from starting

The studio where this prototype ran had a power problem nothing on my side could fix: repeated micro-outages, a server room only partially on UPS and a host that fell over and rebooted with no clean shutdown of the guest. One morning of back-to-back outages, half the stack did not come back. The story is worth telling because the obvious culprit, the power, was only the trigger and the real fault hid in a single line of configuration that everyone would have waved through.

The symptom

On reboot, the base network stack comes back fine, the reverse proxy too, but the provisioning stack stays entirely at 0/1 and observability half down. A glance at the blast radius and the contrast jumps out: everything that depends on internal DNS is on the floor, everything that hosts the DNS is standing.

[OK]   net_pdns-auth: 1/1
[OK]   net_pdns-recursor: 1/1
[WARN] foreman_webapp: 0/1
[WARN] obs_grafana: 0/1
[OK]   obs_victorialogs: 1/1

The real, untruncated error points at the log driver, not the application:

starting container failed: failed to initialize logging driver:
dial udp: lookup victorialogs on 172.27.65.202:53: server misbehaving

The Docker daemon, initialising the container’s syslog driver, tries to resolve victorialogs.devops.lab, fails and because it cannot initialise the driver, it fails the task creation. The container does not start, not because of the application, but because its log has no resolvable destination.

The mechanism

The dirty shutdown had left the Swarm overlay network in an inconsistent state. The PowerDNS recursor could no longer reach its authority, pdns-auth, which was running and serving on its port: the link to it was broken and dig confirmed it with a SERVFAIL carrying an EDE 22 (No Reachable Authority). As a result *.devops.lab no longer resolved and the host’s fallback servers were the corporate DNS, which ignore that internal domain: resolution failed outright.

The detail that turns this transient glitch into an outage is the coupling. The network stack survived and for a good reason: its own log driver pointed at an IP, not a name, because it hosts the DNS and deploys first, so it cannot resolve a name to log, that would be a circular dependency. The other stacks, deployed afterwards, deliberately used the service name victorialogs, the portable, multi-node-minded spelling. That contrast, which I had not planned, was the natural experiment that confirmed the cause: the one driver that did no lookup was the one that survived.

Along the way I ruled out the false leads in order. No volume corruption, the down/up reuses the same volumes and comes back. No MySQL recovery at fault, XA crash recovery finished in a few milliseconds and the database was ready at once. The corruption was not global, only the network overlay was hit.

What does not fix it and what does

The most counter-intuitive lesson of the incident is in these three commands that change nothing:

docker service update --force net_pdns-recursor   # dig: still SERVFAIL
docker service update --force net_pdns-auth        # dig: still SERVFAIL
sudo systemctl restart docker                      # dig: still SERVFAIL

Recreating the tasks one by one, or restarting the daemon, is not enough: the faulty state is not carried by the runtime sandbox that the restart rebuilds, it is carried by the overlay’s service discovery, persisted in the Swarm raft and it survives all of that. What purges it is detaching every service of the overlay and redeploying it:

make down STACK=net && make up STACK=net

A few seconds of patience, the recursor does not listen immediately, then *.devops.lab resolves again.

The real fault

The unblocking fix was obvious once the cause was understood: align the other stacks with what the network stack already did, point their log driver at an IP rather than a name. With an IP and UDP, the driver init does no lookup and asks for no reply, the containers start even with the internal DNS down and the logs for the outage window are lost, which non-blocking mode already accepted. Validated in real conditions: the stacks came back to full health while *.devops.lab was still returning SERVFAIL.

But freezing an IP in place of a service name means giving up multi-node portability, so it is a prototype unblock, not an architecture target. And above all, it was not the real fault. The real fault, the one that turns an overlay that handles a dirty stop badly into a service outage, is the internal DNS architecture. Lacking write access to the group’s DNS, the project had invented a lab domain and self-hosted it inside the very Swarm it served, the host pointing at itself as resolver. A pragmatic and legitimate choice in a prototype, but one that leaves no fallback path: the internal names exist only in that PowerDNS, so the slightest disruption of it cuts all resolution and the log-driver coupling turns that into a start failure.

The fault wasn’t where we thought

The incident is not the power cut, which is only the trigger, recurrent and very real; the fragility that makes it an outage comes from the architecture. A clean reboot never reproduces the problem, only the dirty stop triggers it, so the UPS treats the trigger and leaves the debt intact. A real DNS, a zone or a delegation served by infrastructure independent of the Swarm, would have provided the missing fallback and the disruption of the internal instance would have cascaded into nothing. And a service name in a log driver is still the right spelling, portable and multi-node; freezing it to an IP unblocks the single node but couples the config to one host, which means an unblocking fix is never an architecture decision, just a patch you allow yourself knowing what it costs.

The durable follow-up, the one I put on the backlog, decouples log collection from the container lifecycle: a side collector as a global service that reads the logs locally on each node and pushes them over the overlay, with no host resolver and no frozen IP, which removes the DNS coupling and the single-node limit at once.