Chaos engineering: breaking things for real, but not just any way

In 2018, the same year I discover the twelve factors, I come across chaos engineering (I would be quite unable to find the exact blog posts that lead me there. That is a good reason to go back to the sources rather than my memory): the definition and the principles fit in a single document, which says better what I end up practising than my own rewording.

It is not breaking things

The most common misreading is to believe that restarting a machine in production is enough to do chaos engineering, which confuses a method and an incident: breaking things at random is an incident we failed to avoid, not an experiment. Chaos engineering is a discipline of experimentation, with a defined and measured steady state, a hypothesis about its persistence, a realistic perturbation injected and a rigorous comparison (without a measurable steady state there is no experiment, only an outage).

Where it comes from

The practice is born at Netflix, which publishes Chaos Monkey in 2011 as part of its Simian Army, driven by its cloud architect of the time, Adrian Cockcroft. The idea is simple and uncomfortable: since instances die anyway, we will kill our own, on an ongoing basis, so that nobody notices. The discipline gets its name and its formal principles in the mid-2010s, in the collective document Principles of Chaos Engineering . The vendor Gremlin finished installing the term in the vocabulary.

[edit: 2020] The O’Reilly book by Casey Rosenthal and Nora Jones has since come to complete the reference.

The four steps of an experiment

An experiment follows four beats: you define the steady state by a measurable output, throughput, error rate or latency percentiles; you form the hypothesis that this state persists in the control group and in the perturbed group; you inject variables that reflect real events, a server that goes down, a disk that fails, a network connection you cut; then you try to disprove the hypothesis, by comparing the steady state of both groups. The harder it is to perturb the steady state, the greater the confidence in the system.

The five advanced principles

Five principles frame the ideal: build the hypothesis around the behaviour of the steady state rather than the internal attributes of the system; vary realistic events, hardware or software, failures as much as load spikes; run the experiments in production, because it is the only place where real traffic flows; automate and repeat continuously, because a manual experiment is not sustainable; bound the blast radius at last, because experimenting in production forces you to contain the damage (that last principle is the one that separates the method from recklessness).

Chaos at home

All of this comes from afar: large-scale production experimentation, as Netflix practised it in 2011, is not the daily reality of a modest infrastructure. The adaptation takes some honesty. The first thing to give up is experimenting in production when you have no margin: on a Swarm running real services, you start in the recipe. You only move to production once the blast radius is bounded. The useful experiments are often the smallest: kill a replica to check that a service restarts and that the health probe sees it, inject a few hundred milliseconds of latency, fill a Proxmox disk to see what falls over first. Part of chaos, in fact, I already practise without naming it: a service that only runs on demand is permanently subjected to a cold-start experiment. Every incident documented in a post-mortem is a disproved hypothesis that could have been tested cold.

The tools

I stick to what I can read and understand, to separable bricks. Chaos Monkey remains the historical reference on the AWS side, LitmusChaos targets Kubernetes, Gremlin offers the same as a service, Pumba covers Docker and Compose, then toxiproxy simulates a failing proxy to test timeouts. My opinion fits in one line: as long as the blast radius is not bounded, the tool is just one more button.

[edit: 2019] Chaos Mesh has since come to occupy the same place in the Kubernetes ecosystem.

The link with R.A.C.H.E and post-mortems

There is an irony I enjoy, in the RACHE spirit: the satirical method claims that a system is working when it is unstable and close to failure. Chaos engineering says exactly the same thing, but refusing chance: you pick the moment, the nature and the scale of the shake instead of waiting for it to come on its own. A post-mortem and a chaos experiment are the two faces of the same gesture, one reconstructs after the fact what the other provokes on purpose. On the Twelve-Factor side, everything starts with disposability.

Verdict

I do not recommend chaos engineering to anyone without real observability and well-automated, industrialised infra: without those two prerequisites, the experiment degenerates into a surprise outage. To anyone who has both, on the other hand, I recommend it before production takes on the demonstration. You do not gain by breaking, you gain by knowing in advance how it breaks.