Even a Perfect Procedure Can't Stop Human Error── Bringing aviation's habit of "making danger visible" into the daily work of a material review team

Polishing a procedure matters. But that alone isn't enough. The people who fly planes start from the assumption that humans make mistakes; they gather the errors and near-misses without hiding them, and turn them into learning instead of blame. How much of that real machinery can we move into the daily running of a material review team? This is a companion piece to Issue 34, "psychological safety."

01Every box was checked

One morning, a material came back to my desk. It was a briefing document I had reviewed a few weeks earlier and stamped as approved. On the list of doses meant for doctors, one figure was off from the approved content by one decimal place. It was caught only after the piece had been printed. To say the blood drained from my face would be too much, but my hands went still for a while.

I looked at the reviewer's record. The checklist was filled in from top to bottom. Dates, check marks, signatures—nothing missing. The person had followed the items in the procedure one by one. They hadn't slacked off. They hadn't cut corners. And still, the number had passed straight through.

The first thing that came to mind was the easy explanation: "someone just missed it." A convenient conclusion. Name one culprit and the matter closes. But I stopped myself from going down that road. Why would a person who checked every item, and even left a record, miss one single number? If I skipped past that and filed it under "carelessness," I felt the same thing would happen again.

The British psychologist James Reason (= a scholar who studied why people err, looking at the system rather than the individual) explained how accidents travel with a single picture: the Swiss cheese model (= the image of several slices of holey cheese stacked together). To prevent accidents, we stack up many barriers—the procedure, the checklist, the supervisor's second look, the final proofread before printing. Every barrier exists to stop an accident.

But every barrier has holes. Not one is perfect. The procedure has gaps, the checklist is hard to read, the person doing the second look is tired. Usually the hole in one barrier is covered by the next. That is why accidents don't surface. The trouble comes when the holes in each barrier happen to line up in a straight row. In that moment, the accident passes through every barrier in one breath. This time, the number came straight down that line.

A procedure is only one of those many barriers. Perfect it, and if the holes in the other barriers remain, the holes will someday line up again. My own eye, stamping the approval, is a barrier too. My barrier had a hole in it as well. Thinking of it that way makes it clearer what to do next.

02The one who let it through isn't at fault

Why did the number pass through? I dug a little deeper. On the surface there is, indeed, an operating mistake. The reviewer did not match the correct figure against the material's figure one by one. Or they meant to, and their eye slid over it. Reason calls this an active failure (= the visible operating mistake that happens right there on the spot). It is the error that occurs at the front line, just before an accident.

But I didn't stop there. Why did the eye slide? Looking at the actual checklist told me one reason. The box for writing the correct figure and the box for copying in the material's figure sat confusingly next to each other on the page. When two similar numbers stand side by side, the eye is quick to jump to "they're the same." The very design of the form invited the miss.

One more thing. The material came up for review during the closing week of the quarter. The volume being processed was nearly double the usual. The same person, in the same hours, working through twice the pages. The time per page is cut in half. Rush, and the eye slides.

These two—the confusing form and the quarter-end workload—were buried in the front line long before the accident. Reason calls them latent conditions (= weaknesses on the organization's side that have been lurking for some time). Usually they do no harm. But when the conditions align, they borrow one person's hand at the front line and surface as an accident. The hidden weakness woke up that day.

This is the fork in the road, I think. If we fix only the visible mistake, the story ends with "let's all be more careful from now on." Swap the reviewer for someone else, and the form stays confusing, the quarter-end comes around again. The lurking weakness remains untouched, so a different person leaks through the same hole. Change the people and the hole doesn't move.

So I try to shift the cause, little by little, from the person to the system. Move the form's boxes apart and rearrange them. Don't pile the quarter-end onto one person; split the review duties. Rather than scolding a visible mistake, it's better to close the holey barriers one at a time. That is how you stop the next straight line. That is what I, stamping the approval, can do.

03"The way we always do it" is the most dangerous

I don't really remember the day that shortcut began. Probably no one does. During a week when review requests had piled up at year-end, someone skipped part of the source check (= matching a material's numbers and quotations against the original paper or package insert, one by one). "I saw this chart last time. The content shouldn't have changed." That was the whole judgment. No accident happened. The deadline was met. The supervisor said nothing.

The next month, the same skip in the same situation. Again, nothing happened. By the time three months had passed, it was no longer even a skip. It had become "our way." New hires were taught that way from the start. Even the awareness that something was being skipped had vanished from the team.

The sociologist Diane Vaughan named this process—a dangerous shortcut dissolving into "the normal thing"—the normalization of deviance (= the way conduct that departs from the rules, repeated without any accident, comes to be accepted as the proper way of doing things). What she studied was the Space Shuttle Challenger, which broke apart in mid-air in 1986.

The part behind the accident—an O-ring, a rubber ring that seals the joint in the casing of the solid rocket booster and stops combustion gas from leaking—showed damage beyond the designers' expectations on launch after launch. By rights, that was a signal to stop and say "this is dangerous." But as launch after launch failed to explode despite the damage, the engineers began to think, "damage of this degree is within acceptable range." Each single judgment looked reasonable in the moment. The fact that they had flown safely pushed the next "it's fine" forward. The very standard for danger drifted, little by little.

What matters here is that they were not lazy, and not incompetent. If anything, they were earnest, attaching a plausible reason to each judgment. While things went well, no one stopped it. The reason to stop was drowned out by the record of success. A procedure can't write out every possible situation. "How much damage is allowed" was left to the front line's judgment in the moment. And that judgment drifted, little by little, toward the lenient side.

After Challenger was lost, many engineers said "it had been bothering me for a while." People had noticed. It's just that the repetition of no accidents robbed them of a reason to say the worry out loud. I can't read this story as someone else's problem. Our own skipping of the source check may have been partway down the same slope—just without an accident yet. While things go well, you can't see the slope's tilt. That's what makes it frightening.

04The checklist was born from a crash

One of our younger members once frowned at the checklist we hand to new hires. "Honestly, ticking every one of these is a pain. It feels like you don't trust me." I understand the feeling. Running your finger down the items one by one and marking each does feel like your ability is being doubted. The more capable someone is, the more easily they feel it.

So I told them a story about airplanes. In 1935, the U.S. Army held a flight test to choose its next main bomber. The favorite was the most advanced aircraft of its time, the Boeing Model 299—a large plane with four engines, bigger than anything before it. Flying it was a test pilot reputed to be the finest in the field. But shortly after takeoff, the aircraft stalled just as it climbed, slammed into the ground, and burned. The pilot died.

When they investigated, it wasn't a mechanical failure. The new plane had a lock on its control surfaces (= the rudder and such) that had to be released before takeoff. The skilled pilot had skipped that one release. There were simply too many operations to keep track of in his head alone. A newspaper wrote that it was "too much airplane for one man to fly."

But the story didn't end there. Those involved didn't give up on the aircraft; they found another answer. The pilot's checklist (= a short list, run through out loud one item at a time before takeoff and before landing, of the operations that must be done). Not raising skill, not making the machine simpler—just writing down the steps that are easy to miss and confirming each one. After they started using this plain sheet of paper, the Boeing Model 299 flew on without accident. It later became a famous aircraft that flew tens of thousands of missions.

The person who carefully unearthed this story is the surgeon Atul Gawande. In his book on the checklist (The Checklist Manifesto), he brought the same idea into the operating room. The more skilled the doctor, the more they think "surely I don't miss things." But everyone, amid the rush and the confidence, skips a simple step. When Gawande tried it in hospitals around the world, simply saying obvious items out loud—hand-washing, confirming the patient's name—visibly cut post-operative complications and deaths.

I told the younger member: the source-check list isn't a chain that binds you. It's a handrail passed down from those before us, set against a fact that one famous pilot taught with his life in 1935—that the more capable you are, the more you skip one thing. The trouble of marking the boxes isn't to doubt your ability; it's to carry your ability all the way through. When I said that, the edge left his face a little.

05Collect the near-misses as treasure

One day, in a review, I nearly let a material through. The vertical axis of a graph explaining efficacy changed the width of its scale partway up—a way of drawing that makes the difference look larger than it is. I noticed just before it went to print. A chill ran down my spine. And the next thing to cross my mind was, I'm ashamed to say, this: "Maybe I should keep quiet about this." I didn't want anyone to know I had missed something. That feeling has held me back for a long time.

This kind of event—"it didn't become an accident, but it was close"—is called a near-miss (= that split second when you flinch, when you catch your breath; a small danger that didn't turn into something big) in the professional world. It's a word used in factories and hospitals alike. The trouble is that the person who has a near-miss usually keeps their mouth shut. Because they're afraid of being blamed. And so the seed of danger vanishes, shared with no one—until, in the same place, someone else turns it into a real accident.

What's useful here is a system the U.S. aviation industry built. It's called the ASRS (= the Aviation Safety Reporting System; a channel for pilots and controllers to report events that were close calls). This system has two devices that read the human heart well: it is anonymous, and reporting brings no punishment. Who reported is kept hidden, and a person who confesses honestly is not disciplined on the grounds of that report.

The result was clear. The moment the design stopped assigning blame, the number of reports rose. People can speak of their failures when no punishment is waiting. The mass of gathered near-misses is analyzed and handed to everyone as dangerous patterns. One person's cold sweat becomes everyone's learning. A culture that hides failure, and a culture that collects it. How rarely accidents happen probably divides on this single point.

So what can we do in the review team? As a concrete idea, I'd like to propose a "close-call shelf" (= a place to gather the cases we very nearly let through). Names are withheld. Who missed it isn't written. What's written is only this: what kind of material, where, and why it was dangerous. Once a month, we all look at the shelf together. Not a blaming session—a learning session. That story of the graph's vertical axis, too, I want to put on the shelf myself. Shame, the moment it's shared, turns into a tool that protects the next person.

06The one who notices speaks up, regardless of rank

A member six months in said to me quietly after a review meeting: "That material earlier, something about it caught on me a bit… but the senior said it was fine." When I heard it out, it was a reasonable point. The qualifying note on the efficacy was too small, she felt. But she wasn't sure whether someone with so little experience should cut into a veteran's judgment. That material was later sent back. Her hesitation had been right.

This scene has a famous backdrop: the cockpit of an airplane. Once, in a number of crashes, the same thing happened. The co-pilot had noticed something wrong—the instruments are off, we're too low. But they couldn't clearly voice an objection to the higher-ranking captain. While they spoke in roundabout ways, the aircraft fell into the mountains, into the sea. It wasn't a problem of technology. The cause was that the one who noticed couldn't say they'd noticed.

Born from this reflection is CRM (= Crew Resource Management; crew coordination training so that whoever notices speaks up, regardless of rank or seniority). It isn't training to sharpen the captain's skill. It's training so the co-pilot can say, clearly and briefly, "Captain, we're too low." If you think something's wrong, don't go roundabout—put it into words. And the person above doesn't dismiss it as cheek; they listen first. Cockpit safety only began to turn once both of these were in place.

I see this as one step beyond the psychological safety (= here, the sense of ease in a workplace where saying something odd won't get you mocked) I wrote about in Issue 34. If psychological safety is "an air in which you can speak up," CRM is "how to speak up" itself. Even with the air, if you don't know how, the voice doesn't reach. A procedure inspects the content of the material. But it can't stop a judgment error by the human running that procedure. What stops that is another human, standing outside the procedure.

To that six-month member, I said afterward: "Next time, you can say it on the spot. In fact, I want you to." I won't forget her face as she nodded. A team where voices come out regardless of rank can probably catch the youngest person's small snag before it becomes an accident. Perfect the procedure, and people still err. Which is exactly why what stops the error is, again, a person.

07Keep caring about the small tear

That document was nearly perfect. The efficacy explanation, the way side effects were written, the placement of the notes—no problem anywhere I looked. Only on one page, the indentation of a chart's heading was off from the others. By one full-width space. A small shift you'd doubt any reader would even notice. I thought, "it probably just shifted when it was pasted," and started to tick the approval box. Then my hand stopped.

Putting into words, afterward, why it stopped is hard. If I had to, I'd say: the moment you file something under "just happened," you've stopped confirming—that feeling was what I disliked. The indentation might have shifted because someone copied that one chart from a different file. A different file might mean an older version. If it's an older version, the numbers might be from before the update. A small shift usually ends as just a small shift. But rarely, a large mix-up hides behind it.

Karl Weick and Kathleen Sutcliffe, two researchers who long studied organizational accidents, examined what "high reliability organizations" (= front lines where not a single failure is allowed, which still keep running with almost no accidents) have in common. The first thing they listed was the stance of staying preoccupied with the small sign of failure. The better a front line runs, the less it waves off a small, unnerving event as "no big deal," and the more it treats it as "this might be the start of something." It reads a small tear as a warning of a large hole. My hand stopping in front of that indentation was, probably, a very small instance of this.

Another thing in their work that stays with me is deferring the judgment to the person who knows most. An ordinary organization lifts hard judgments up to the higher-ranking person. But a high reliability organization, in dangerous moments, hands the judgment down to whoever knows that front line best. Not the title, but the person most in touch with the situation in that moment decides. In material review, that means the reviewer who knows a product's background and past issues in their bones can say, "this is worth confirming"—and that one remark isn't crushed by their inexperience or their standing.

So when a mistake is found, how should we behave? Here the aviation-accident researcher Sidney Dekker's "just culture" (= an organizational air that doesn't blame honest mistakes and keeps them apart from intent or cover-ups) comes into play. In the world of flight, a system where pilots and mechanics report their own close calls without hiding them has supported safety. String up the person who reported, and no one reports next time. So an honestly confessed mistake isn't blamed. But deliberate corner-cutting or deception is cut off from that and questioned properly. This one line of "separating" protects both honesty and accountability.

The review front line is the same, I think. When someone finds a miss and speaks up, do we hunt the culprit—"why didn't you notice?"—or can we say, "good catch, let's look together at why it happened"? Keep up the former, and people start hiding mistakes. A hidden mistake shows its face again, in a place where no one next can notice it.

Keeping to the procedure is indispensable, as a foundation. But a procedure can't always be kept the same way by a person who is tired, or grown too used to it. Which is exactly why, beyond the procedure, I want to lay down one more layer of air. If you think something's wrong, you can say it. Speak up, and you won't be blamed. The shift you found, you needn't carry alone—you can set it where everyone can see. From work that leans on one person's concentration, toward the practice of a team picking up small tears for one another. If I can move the air there, little by little.

That indentation shift, when I checked, was indeed a chart from one version back. The numbers, luckily, were the same—but one line of a note was still from before the update. I was glad I'd stopped my hand. And at the same time, I thought this: the next person to find such a shift needn't be me. Whoever finds it can pause the same way. Such a team, I quietly raise.