How Complex Systems Fail: a Synopsis

As Tony Hoare, winner of the 1980 Turing Award, once wrote, “There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.”

Complexity grows in conjunction with advancements in technology due to the nature of advancement itself. The software world is particularly cognizant of this fact due to the way programs are built upon each other to perform increasingly specialized roles. The layers of complexity created by even a simple program, such as a word processor, are astoundingly deep when you drill all the way down to the OS and machine layer.

The inevitability of complexity contributes to the importance of understanding how complex systems work, and how they fail. This latter tendency of complex systems is the focus of Richard Cook’s How Complex Systems Fail paper. Inside this fairly brief yet incredibly dense paper, Cook expresses and then expands upon 18 separate key observations on not only how complex systems fail but also why and what can be done to reduce the frequency and severity of failures.

The entire paper is worth taking the time to read and many systems design luminaries have suggested it should be mandatory reading for anyone in the industry. However, there are a few key points in the paper that are worth particular note and exploration:

#2 Complex systems are heavily and successfully defended against failure.

Cook starts off his “Short Treatise on the Nature of Failure” with the observation that systems become complex as methods are created to defend against the potential hazards of the system. In other words, layers of defense are established to fend off failures of the system and these layers of defense play a major role in increasing the complexity of the whole system.

The development of defensive layers involves multiple components such as backup systems, agent training, and policy creation. Therefore, the avoidance of failure inherently increases the complexity of any given system. The next logical inference given is that the more complex a system is, the more complex defensive measures must be.

#3 Catastrophe requires multiple failures – single point failures are not enough.

Continuing from point 2, Cook observes that layers of defense are effective at avoiding failure. As such, the only way that a system can fail catastrophically is through multiple, simultaneous points of failure. The defensive checks and balances in place ensure that an incident cannot occur as the result of a single point of failure.

The implication here is that complex system failures are complex and multi-faceted in their own right. Cook goes on to make observations relating to human psychology as a tangent of this core point where he discusses the propensity for finger-pointing. After all, it’s completely natural to want to identify the reason why a failure occurred.

The issue with finger-pointing (which is prone to happen as a result of post-mortems) is that there is almost never a single “root cause” of any given incident. It’s convenient to say that a plane crashed due to pilot error, but the pilot isn’t in charge of inspecting the landing gear or repairing faulty instruments.