I do a fair amount of public speaking about safety topics, complacency, and flight discipline. The subject of checklists comes up frequently and when I offer my opinions, I can tell there is a sharp divide in the audience about what I have to say.
I think roughly half of these audiences are on my side: we need to use checklists as they were designed, line by line: "Challenge-Do-Verify." The other half nod politely, usually not saying anything. Every now and then somebody offers a counter point. But roughly half don't agree that checklists need to be done so formally and advocate what some call "the flow," but is more properly called the "Do-Verify" method.
So let's look at checklists with a fresh set of eyes, from the medical community. Then let's look at the early days of checklists in aviation, the official view (from the FAA), the most common objections (and counter-objections), and a real life example showing the pros and cons of using checklists using Challenge-Do-Verify.
Everything here is from the references shown below, with a few comments in an alternate color.
There are some parallels with the medical profession and aviation. We have many complex tasks that, taken as a whole, seem too varied and difficult to break down into checklists. But there are individual "sub-procedures" that do lend themselves to the idea we can check them to make sure they are done.
But if you suggest a checklist for these sub-procedures, you are likely to hear they are so simple that using a checklist would be a waste of time. Why do I need a checklist to start the APU? It is seven steps long, I have them memorized, and I never forget any of the steps. Except when I do.
Dr. Gawande studied a simple five step procedure that has been in use for years and years. Nobody gets them wrong. Except when they do. A checklist for this simple procedure has saved lives.
[Gawande, pp. 37-39]
In 2001, though, a critical care specialist at Johns Hopkins Hospital named Peter Pronovost decided to give a doctor checklist a try. He didn't attempt to make the checklist encompass everything ICU teams might need to do in a day. He designed it to tackle just one of their hundreds of potential tasks, the one that nearly killed Anthony Defilippo: central line infections.
On a sheet of plain paper, he plotted out the steps to take in order to avoid infections when putting in a central line. Doctors are supposed to
Check, check, check, check, check. These steps are no-brainers; they have been known and taught for years. So it seemed silly to make a checklist for something so obvious. Still, Pronovost asked the nurses in his ICU to observe the doctors for a month as they put lines into patients and record how often they carried out each step. In more than a third of patients, they skipped at least one.
The next month, he and his team persuaded the Johns Hopkins Hospital administration to authorize nurses to stop doctors if they saw them skipping a step on the checklist; nurses were also to ask the doctors each day whether any lines ought to be removed, so as not to leave them in longer than necessary. This was revolutionary. Nurses have always had their ways of nudging a doctor into doing the right thing, ranging from the gentle reminder ("Um, did you forget to put on your mask, doctor?") to more forceful methods (I've had a nurse bodycheck me when she thought I hadn't put enough drapes on a patient). But many nurses aren't sure whether this is their place or whether a given measure is worth a confrontation. (Does it really matter whether a patient's legs are draped for a line going into the chest?) The new rule made it clear: if doctors didn't follow every step, the nurses would have backup from the administration to intervene.
For a year afterward, Pronovost and his colleagues monitored what happened. The results were so dramatic that they weren't sure whether to believe them: the ten-day line-infection rate went from 11 percent to zero. So they followed patients for fifteen more months. Only two line infections occurred during the entire period. They calculated that, in this one hospital, the checklist had prevented forty-three infections and eight deaths and saved two million dollars in costs.
In 1934, the U.S. Army Air Corps asked industry to come up with a replacement for the twin-engine Martin B-10 bomber. Douglas and Martin came forward with bombers that were a marginal improvement, Boeing's Model 299 represented a revolutionary leap forward in bombers. But this crash of the prototype caused the program's cancellation, reasoning the new bomber was simply too complicated to fly. But the Army Air Corps wanted the airplane and ordered 13 of the bombers with several changes and a checklist for crews to follow. The aircraft became a work horse of the air war over Europe and over 12,000 were built. The aircraft, as it turned out, wasn't too complicated to fly. It just needed a checklist.
[Boyne] The creation of the checklist was delayed by an unrealistic reliance on the memory of pilots. This dated all the way back to 1903, with the Wright brothers’ intimate knowledge of airplanes.
By 1918, with vastly expanded pilot training in the United States, a special handbook was created by Curtis for the JN-4 “Jenny.” One section was titled “Hints on Flying” and provided a list of 18 items, each with considerable detail. Five items were devoted to actions to take before takeoff, nine covered in-flight procedures and safety precautions, two advised on landings, and two discussed ways to avoid stalls and spins.
As was the case with all of these checklist predecessors, pilots were expected to know the manual by rote. Sometimes this led to hubris, which led to accidents.
The nearest relative to a post-1935 checklist is found in “Hints on the Bristol Fighter,” dated March 30, 1918, and written by the officer commanding No. 39 Squadron. The section headings are similar to modern checklist, including specific sections designed to ensure pilots see that the pressure is holding, the ignition is fully advanced, the temperature is at least 65 degrees and not over 85 degrees, the oil pressure is OK, the blinds are open, and the tail lever is forward.
[Boyne] The Model 299 made its first flight on July 28, 1935, flown by Boeing’s chief test pilot Leslie R. Tower. Dubbed the “Flying Fortress” by Seattle reporter Richard L. Williams, the Model 299 was the prototype for 12,730 B-17s which followed. Then on Oct. 30, 1935, with the suddenness that characterizes experimental test flights, Boeing’s great gamble seemed to fail when the beautiful silver Model 299 crashed on takeoff from Wright Field. The tragic event seemed certain to lead to the cancellation of the program and an immediate change in Army Air Corps planning.
At about 9:30 a.m. on that October day, the Model 299 was manned by a very experienced crew, including Maj. Ployer P. Hill, Wright Field’s Flying Branch chief, and his copilot, 1st Lt. Donald L. Putt. Also on board were John B. Cutting, a flight-test observer; Mark H. Koogler, also from the Flying Branch; and Tower.
Observers described the initial run of the Model 299’s takeoff as normal, even though it broke ground at about 74 mph in a “tail low” attitude. As its speed increased, the bomber’s nose went up much higher than normal. Two men, 1st Lt. Robert K. Giovannoli and 1st Lt. Leonard F. Harman, sensed it was in trouble and ran forward as the airplane reached an altitude of about 300 feet.
The Model 299 stalled, turned 180 degrees, and fell back onto a field. It landed on its left wing, cushioning the impact, which probably saved the lives of several crew. Lying flat on the field, the bomber burst into flames. Amazingly, four crew members were able to crawl from the blazing wreckage.
Hill Air Force Base, Utah, was named after Major Hill, who died following this crash.
A board of officers convened at Wright Field to investigate the crash. The presiding officer was Lt. Col. Frank D. Lackland, for whom Lackland AFB, Tex., was later named. The board determined the accident was “not caused by”: structural failure; malfunction of flight controls, engines, or propellers; the automatic pilot; or any faulty structural or aerodynamic design. Instead, it ascribed the direct cause to the elevator control being locked.
The tail section of the aircraft was virtually all that survived the fire, but it contained the cause of the accident: an internal control lock that controlled both the elevator and rudder.
The board stated that—due to the size of the airplane and the inherent design of the control system—it was improbable that any pilot, taking off under the same conditions, would discover the locked controls until it was too late to prevent a crash. Ordinarily, pilots make checks of their movement as a precaution, but apparently this did not occur.
To avoid another accident, Air Corps personnel developed checklists the crew would follow for takeoff, flight, before landing, and after landing. The idea was so simple, and so effective, that the checklist was to become the future norm for aircraft operations. The basic concept had already been around for decades, and was in scattered use in aviation worldwide, but it took the Model 299 crash to institutionalize its use.
Yes, the following only applies to commercial operators, including 14 CFR 91K. But it should be considered a "best practice" among all non-commercial operators.
Photo: FAA Headquarters, Wilbur Wright Building, Washington, D.C., 21 Sep 2009, (Matthew Bisanz)
Click photo for a larger image
[FAA Order 8900.1, Volume 3, Chapter 32]
3-3401 A. Definition. A checklist is a formal list used to identify, schedule, compare, or verify a group of elements or actions. A checklist is used as a visual or oral aid that enables the user to overcome the limitations of short-term human memory.
3-3402 B. Criticality of Checklist Items. Checklist items can be ranked in criticality according to the potential effect of the crewmember failing to perform the action. Critical items are those items which, if not correctly performed, have a direct, adverse effect on safety. Noncritical items are “housekeeping” items or systems management items, which for operating practices must be routinely accomplished during a specific phase of flight, but if omitted would have a minimal effect on safety.
3-3403 METHODS OF CHECKLIST DESIGN. Operators may choose from at least two accepted methods of checklist design: the “challenge-do-verify” (CDV) method and the “do-verify” (DV) method. Available evidence suggests that safety is enhanced when the operator adopts and applies a consistent checklist design policy. POIs should use the following informative guidance when reviewing the design of an operator’s aircraft checklists.
A. “Challenge-Do-Verify.” The CDV method consists of a crewmember making a challenge before an action is initiated, taking the action, and then verifying that the action item has been accomplished. The CDV method is most effective when one crewmember issues the challenge and the second crewmember takes the action and responds to the first crewmember, verifying that the action was taken. This method requires that the checklist be accomplished methodically, one item at a time, in an unvarying sequence. The primary advantage of the CDV method is the deliberate and systematic manner in which each action item must be accomplished. The CDV method keeps all crewmembers involved (in the loop), provides for concurrence from a second crewmember before an action is taken, and provides positive confirmation that the action was accomplished. The disadvantages of the CDV method are that it is rigid and inflexible and that crewmembers cannot accomplish different tasks at the same time.
B. “Do Verify.” The DV method (or “clean-up” method) consists of the checklist being accomplished in a variable sequence without a preliminary challenge. After all of the action items on the checklist have been completed, the checklist is then read again while each item is verified. The DV method allows the flightcrew to use flow patterns from memory to accomplish a series of actions quickly and efficiently. Each individual crewmember can work independently, which helps balance the workload between crewmembers. The DV method has a higher inherent risk of an item on the checklist being missed than does the CDV method.
C. Selection of Design Method. Both the CDV and the DV methods of checklist design are currently being successfully used for normal checklists. Traditionally, operators have preferred the DV method for normal checklists and the CDV method for non-normal and emergency checklists. Operators have, however, successfully used the CDV method for all checklists. POIs may approve either method for normal checklists. In most circumstances, non-normal and emergency checklists are more effective when the CDV method is used. The correct accomplishment of the actions and procedures incorporated in the non-normal and emergency checklist categories is critical and warrants a methodical approach. Since these checklists are seldom used, however, crewmembers are usually not as familiar with the procedures incorporated into these checklists as they are with the procedures in normal checklists. In addition, many non-normal and emergency checklists do not lend themselves to developing flow patterns that crewmembers can readily recall. The CDV method also enforces crew coordination, cross-checking, and verification, all of which aid the crewmember in overcoming the adverse effects of stress. POIs should not approve or accept the DV method for non-normal or emergency procedures unless the operator can provide substantial evidence that the method is effective for this application.
3-3404 B. 7) In the taxi and pretakeoff phases, aircraft configuration (such as flaps, trim, and speedbrakes) and flight guidance items (such as heading, flight director, altitude select panel settings, and airspeed bugs) have proven to be critical. All flightcrew members should confirm these items, and at least two crewmembers should respond to applicable checklist items.
8) On approach, flight guidance checklist items have proven to be critical items. At least two crewmembers should confirm and respond to these items. A response should be required from each pilot when the same setting is required on two separate devices (such as computers, flight instruments, or altimeters).
9) All checklist items that are critical in the before-landing phase vary with the type of airplane involved. In the operation of small airplanes, the landing gear has proven to be a critical checklist item, and both pilots should confirm and respond to this item. Although the landing gear and flaps are critical items for large, transport category airplanes, the multiple warning devices and systems that are associated with these systems make the need for a response and confirmation by both pilots less critical.
10) All checklists, except the after-takeoff and after-landing checklists, should be accomplished by one crewmember reading the checklist items and a second crewmember confirming and responding to each item. POIs shall ensure that critical items on the before-takeoff and before-landing checklists are confirmed and responded to by at least two crewmembers.
It certainly makes sense that the after takeoff and after landing checklists are exceptions, you are very busy and your eyes need to be outside. But every other checklist needs to be accomplished using the challenge-do-verify method. That should end all arguments from the "I can flow any checklist" crowd. It certainly illustrates that the "silent checklist" cannot be used.
13) Checklists should not be depended on to initiate changes in aircraft configuration. Operators should key aircraft configuration changes to specific operational events. For example, the operator may direct the landing gear to be extended at glideslope intercept. For any adjustment of thrust or configuration, a command from the PF and an acknowledgment from the crewmember taking the action are required.
The most common objection to rigidly following a challenge-do-verify checklist is that it slows things down for procedures that are (and should be) ingrained because we do them so often. My answer to that is N121JM. But let's set that aside for now. There are other objections and counter points to those objections.
Photo: N121JM Wreckage, aerial photograph, from NTSB Accident Docket, figure 6.
Click photo for a larger image
I've timed several examples and asked others to do the same. With no exceptions, the results indicate it takes more time to complete a list of steps followed by a checklist, than it does to complete the checklist by reading each item (the challenge), accomplishing the item, and the responding to the challenge (the response). See A Real Life Example, below.
There are a number of accidents to prove this isn't so. Most notably: N121JM. I flew the GIV for almost a year, much of that time using a flow followed by the checklist. Our pilots were very good, highly experienced, and very conscientious. There are 22 items on the After Engine Start checklist, of which I noticed three were occasionally missed.
The problem with this argument is that once you perceive each item is completed via the flow, you subconsciously believe you are done and the checklist is a waste of time. You tend to rush through the checklist as a result and your eyes suffer from expectation bias, that is, your brain perceives things as they should be, not as your eyes report they actually are. Here is just one example from my time in the Gulfstream GIV, After Engine Start checklist:
The PIC of N121JM had a 11,250 hours of total flight time, 7 years in type, and 12 years with the company. The SIC had 18,530 hours of total flight time, 7 years in type, and 27 years with the company.
Of course the classic missed checklist item is the landing gear. The Aviation Safety Reporting System (ASRS) shows that "we" tend to average 100 gear up landings every year, for example. Here are a few:
Sometimes the results of a missed checklist item can be far worse. Here are a few examples:
I've run these tests in several aircraft and written before about the efficacy of Challenge-Do-Verify, most notably right here: Checklist Philosophy, in response to the crash of Gulfstream GIV N121JM. But in each of these cases I was qualified in the aircraft and perhaps my bias influenced the results.
So I reached out to someone flying an airplane I know nothing about, the Gulfstream G150. Now you might argue that a Gulfstream is a Gulfstream but that isn't true in this case. The G150 began its life as an Israel Aerospace Industries Astra, an improved version of the venerable Westwind. So here are those results. If you want to give this a try with your aircraft, I would be happy to publish your results too.
Photo: Gulfstream G150 N150GD, Geneva, 16 May 2015 (Markus Eigenheer)
Click photo for a larger image
Operators often customize their checklist to provide better standardization in their fleets, adapt to aircraft-specific customization, or to improve what the manufacturer offers. These checklists are quite often tailored to make the Do-Verify flow as efficient as possible. One operator, for example, uses the following G150 after engine start checklist:
The flow appears to be quite good, allowing the pilots to concentrate at first overhead and then below, with a minimum of skipping around. I would think this flow could be as fast as any after starting engines flow that I've ever seen.
There are really only two evaluation criteria here when comparing the Do-Verify Flow against the Challenge-Do-Verify method:
In a series of tests, the Do-Verify flow averaged 2:17 (minutes:seconds), while the Challenge-Do-Verify averaged 1:33. I've had similar results in the Gulfstream GIV, GV, and G450. I have been asking for several years now for someone to show me a contrary result. (I'm still waiting.)
Here is where there can be some controversy. Flow adherents insist Do-Verify is faster because everything gets checked twice. Reports of Do-Verify errors are, of course, anecdotal. I can tell you that over the course of two months when cataloging errors in the GIV, I found several examples of Do-Verify errors. In the next year flying that aircraft using the Challenge-Do-Verify method, I found no errors. But you can argue a test sample of one airplane and three pilots does not a iron-clad conclusion make. That's true.
Either method will have errors if you don't discipline yourself to look at the switch, light, gauge, lever, or whatever the item calls for. I would argue that the Challenge-Do-Verify method forces you to pace the checklist at the speed at which the pilot can perform each individual operation; while the Do-Verify method places pressure on the person reading the checklist to keep up with the responses. You are more likely to make mistakes when you rush.
Boyne, Walter J. “The Checklist” Air Force Magazine, August 2013, pp 52 – 56
FAA Order 8900.1 Vol 3 Ch 32, Manuals, Procedures, and Checklists for 14 CFR Parts 91K, 121, 125, and 135.
Gawande, Atul., The Checklist Manifesto: How to Get Things Right, 2009, Metropolitan Boos, Henry Holt and Company, LLC, New York.
NTSB Accident Docket, ERA14MA271
Copyright 2019. Code 7700 LLC. All Rights Reserved.