Reliability Compounds

In 1998, I moved from Singapore to Massachusetts, culturally and geographically a pair of antipodes. One reason I was able to do this was the availability of air travel, the knowledge that I could hop on a plane and emerge, 24-30 hours later, at an airport on the far side of the world. A few decades earlier, when my grandparents left southern China for Singapore, a journey one-fifth the distance, passage on a steamship took five or six days, and planning the trip could have taken months. I think they returned to China no more than a handful of times over the course of their lives, whereas I’ve sometimes shuttled to Singapore two or three times a year. In a way modernity is composed of these: habits and affordances, abilities we rely on even though they would have been considered magic a generation ago. 

But the magical fact here isn’t that we have jet planes. It’s that modern airliners – insanely complex systems composed of millions of parts driven by millions more lines of code – are reliable to the tune of one accident per 1.13 million flights. It’s the reliability of air travel that’s the key enabling factor behind easy transcontinental travel. 

You can’t touch reliability, or polish it. You can’t point to a single factor that’s made air travel safer, because reliability is fundamentally an emergent phenomenon. We don’t even truly know how it comes about, but it’s also absolutely inseparable from modern life – in aviation and elsewhere. We rely on water to come from our taps, we rely on supermarket shelves to be full, we rely on power to flow at the flip of a switch. We rely on the collected information of the internet to be in our pockets at all times. All this stuff just works.

Nearly every journey I’ve taken between Singapore and the east coast has involved a Boeing 777, or one of its variants. It launched in 1995, three years before I moved to the United States, so its life in active service overlaps nearly perfectly with my need to fly around the world a couple of times a year. 

Boeing promised the 777 would be the most advanced airplane ever built, and also the most thoroughly tested. It was the first plane Boeing designed entirely in CAD, which at the time was still a maturing technology. Their earlier work with CAD systems had convinced them of its utility, but not having done a full airplane before, they didn’t trust the CAD software to catch every place where conduits might collide – so they planned to build a full-size fuselage and physically route every single cable before finalizing the design. After they did this with the front portion of the plane, they realized the CAD system hadn’t messed up once, so they never did the rest. They tested hundreds of life-size, working subassemblies: everything from the cup holders in the cockpits to the soft-closers for the toilet seats. To ensure the cabin doors would open if the plane iced over, they put a chunk of the plane in a freezer, and sprayed it with water for 8 hours, then tried to open the door. 

The 777 was also the first Boeing airliner to fly entirely by wire. A key reliability requirement for fly-by-wire systems is physical redundancy. Fly-by-wire aircraft need to have at least three copies of each system, all receiving, processing, and transmitting the same set of signals simultaneously. The systems monitor each other: If one starts acting out of sync, the combined system begins to ignore it – Minority Report on a plane. The Space Shuttle had a similar system, but was quintuply redundant: In addition to four primary flight computers, the Shuttle carried a fifth machine, running simpler software, in case the code on the other four had a bug. This was a physical admission of a truth software engineers have long understood: The more complex software is, the less reliable it is.

The development of the 777’s flight control software sounds like a nightmare for the ages (or possibly an epic). They were developing not just the software but a new microchip, and a new bus, now the industry standard, to transmit data around the plane. They chose to work in Ada, a language which had until then only been used on mainframes, and it had to be compiled for not one but two microprocessors. The original plan (later scrapped) called for the software for each of the primary flight computers to be written by a separate team, so no single bug would be introduced into all three codebases. 

Every new technology Boeing introduced was one more thing that had to be proven reliable enough to fly, so Boeing built a test rig called the Iron Bird. It was essentially the 777’s nervous system in a warehouse: the entire computer system, hooked up to a life-size, working cockpit and actuated flight surfaces. The idea was that by “flying” the Iron Bird thousands of times, they’d discover, before the first real plane was assembled, every possible behavior of this impossibly multifaceted system. Software’s supposed to be predictable, but computers are general-purpose machines, designed to accept and react to a vast range of inputs – and the 777’s software would be accepting inputs that were literally as variable as the weather. Maneuvers in three dimensions, made in all weather conditions, at every speed from 0 to Mach 0.85 – an impossibly diverse set of conditions to explore. 

But because the various components of the system, starting with the new data bus, were so delayed, Boeing wound up running integration tests in partially built aircraft on the factory floor. The Iron Bird had started “flying” just a few months earlier, completing only a fraction of 3,000 “test flights” originally planned. The “final” build of code was loaded onto the first production 777 only a few days before it took off for the first time. 

Yet somehow, the first 777 flew its first test flight roughly on schedule, in June 1994. It took off with pages and pages of known bugs and planned workarounds. The scene was captured in a PBS documentary called 21st Century Jet. In a made-for-TV moment, the chief test pilot, John Cashman, emerges from the plane as though it’s Air Force One. It had been, he would declare, the “Best first flight ever!” While multiple known bugs still existed, Cashman hadn’t discovered a single new bug in mid-air. 

I watched all 5 episodes of 21st Century Jet, and one scene in particular sticks in my memory. A crowd of Boeing employees stands in a hangar with a full size 777. The room is grey, the atmosphere expectant. The camera pulls back to capture the scale of the place. Even with an airliner in there, it looks roomy. The humans are all well back, behind clear barriers. Someone pulls out a pair of binoculars. The plane is held in a sort of marionette rig, which conjures a range of macabre associations. A PA system counts up, and the narrator explains that Boeing is trying to break the plane.

The FAA requires this test to establish the safety of an airplane design. Take a wing, bend it with a certain force, hold it there for 3 seconds. A wing that remains intact passes the test, and can be eased back into repose. 

But Boeing didn’t just want to pass the test; they wanted to know when the 777’s wings would actually break. When the camera first shows this airframe, the wingtips are already way above the fuselage, bent under 130% of the design load for the wing. The airframe is a sacrificial bird, the hangar is an altar. 

Back and forth the camera goes, from the faces in the crowd to the airframe in its cradle, with the PA counting up and up. The voice on the PA is as dry as a lab coat. Applause breaks out as the wing passes the official benchmark – 150% of design load, held for 3 seconds – and then the count resumes, 151, 152, 153, and then on 154, there’s a spinal crack as the cables twang and the manic smile of the bent wings shatters in a broken welter that looks like teeth. The test conveys conclusiveness, as a sacrifice this costly ought to. The cables go slack, the audience releases its collectively held breath, and then, they applaud. 

From a certain perspective that test was more a necessary ritual than anything else. A sample of one is statistically meaningless, even if it’s loaded with symbolic power. But one really cool thing about reliability is that it compounds: If Boeing’s production processes were sufficiently reliable, if they could trust their tooling and processes and instrumentation, then that one broken wing actually told them, and us, a great deal about the 777 fleet as a whole.

The agency we derive from a reliable world compounds too. I can spend my time writing this because I don’t need to worry about where my drinking water comes from, or my next meal. Actually conveying these words to you is almost effortless because Apple’s production processes are reliable, and the optic fiber backbones of the US internet. Seen this way, a world filled with reliable systems is a privilege, one which humanity has collectively earned through centuries of effort and experimentation, and one which it’s all too easy to take for granted. 

SCOPE CREEP.

• This article as a whole draws on the excellent reporting of Byron Acohido, especially...