A Failure In Leadership

This article is based on a historically important article in the New York Times on the BP oil disaster. I will be referencing this article.

Point of Failure #1

The article, which is important and should be read in full, walks us through publicly known (though not all well known) data about the BP oil spill. Indeed, many of the details are known to me as an avid reader of news. The article does contribute greatly to our understanding of the human elements, and it is on this, and not technical issues, that I wish to focus.

But Mr. Holloway detected no concern in the drill shack. Whether the team was distracted by other tasks or rushing to get done or simply complacent may never be known.

“The question is why these experienced men out on that rig talked themselves into believing that this was a good test,” said Sean Grimsley, a lawyer for a presidential commission investigating the disaster.

“None of these men out on that rig want to die.”

Actually, it’s not hard to understand at all.

BP (formerly British Petroleum)’s executives were there on a mission: to get a troublesome well finished and capped before storm season. The New York Yankees of the drilling industry, with a perfect safety record for the preceding several years, had done the drilling fast – but fast creates risks.

At the critical point, BP’s people were talking with the Transocean people (who actually operated the rig).

Strictly speaking, the BP people did not veto ideas of proceeding more cautiously in the face of an unsatisfactory test as to whether there was any leakage of gas from the well. (You know, leaking gas can be, well, bad…) Rather, it’s that the BP people engaged in a filibuster, talking the idea to death until the Transocean people, under immense financial and business pressure (and the upcoming hurricane season) to get things done now, “talked themselves” into agreeing with an approach based on business issues and not safety.

Any objective observer would have screamed foul, but Transocean could not be objective. Not with BP people physically present and creating immense psychological pressure on Transocean’s people… but there was never a “veto,” and thus, never a specific decision that could be traced to a specific individual.

To the modern management mindset, an untraceable decision is an advantage. In a situation regarding industrial safety, it is a point of failure, a proverbial disaster waiting to happen, soon followed by a real disaster.

But we are not finished.

Point of Failure #2

Up on the bridge, the crew was busy showing off the Horizon’s impressive capabilities to the V.I.P.’s.

Andrea Fleytas, one of the bridge officers, felt a jolt. Curt Kuchta, the 34-year-old captain, also sensed something wrong. There was a high-pitched hissing. On a closed-circuit television, they could see mud flying into the sea.

Suddenly, gas alarms began lighting up Ms. Fleytas’s computer console. The lights showed gas spreading over the rig, from the drilling floor to the main deck. There were so many alarms it was hard to keep track of where gas was being detected. More frightening still, the lights were all magenta, signaling extremely high levels of combustible gas.

Don’t forget about the first line of this quoted text. It’s important.

After discussing how Transocean had changed the general alarm system to a manual one:

Ms. Fleytas, 23, had graduated from maritime school in 2008 and had only been on the Horizon for 18 months. This was her first well-control emergency. But she had been trained, she said, to immediately sound the general master alarm if two or more sensors detected gas. She knew it had to be activated manually. She also knew how important it was to get crew members out of spaces filled with gas.

Yet with as many as 20 sensors glowing magenta on her console, Ms. Fleytas hesitated. She did not sound the general master alarm. Instead she began pressing buttons that told the system that the bridge crew was aware of the alarms.

“It was a lot to take in,” she testified. “There was a lot going on.”

I’m keeping my quotations to the minimum possible, but this is the critical point.

First of all… have you ever had an alarm clock? Quite a lot of alarms come with a snooze button. Pressing this allows you to have the alarm temporarily stop ringing. After a fixed period of time, the alarm rings again. This allows a person to “snooze” for a time.

I don’t like snooze buttons. I could quite literally press a snooze button without realizing it, turning an alarm off and rendering the alarm impotent. For this reason, I keep my alarm clock physically out of reach. True story.

So, I don’t think the New York Times got this part quite right in terms of the implications.

Obviously, as in accordance with the article itself, the general master alarm was set so that it would not automatically activate immediately upon detection of dangerous gases. The key word, however, is immediately.

There’s no way that this woman would have had to press buttons to make the alarms go away unless it was similar to a snooze button: a manual override to avoid the general master alarm triggering after X period of time. This is a technological question, and if no one’s picking up on it, I’ll be very clear: this woman was muting the alarms.

Of course, we may ask ourselves… why?

Let’s go back to the top of the quoted text: because a gaggle of BP executives were being shown around the bridge.

The BP executives need not have ordered anyone to mute alarms. Indeed, it is extremely likely they did not. At this critical moment, they did damage merely by existing.

God Is Coming – Look Busy

My father used to work at IBM.

It was a very long time ago. His mindset truly is far better suited to being an inventor of a tool for the housing construction industry… which he now is. Patent pending. But, that is not the point of this story.

While at IBM, he was told by an executive that

God is coming in three weeks. – Nameless IBM executive

Treating this like a military inspection by a high-ranking officer, “real” work effectively stopped as the office building was subjected to an overhaul, including a complete repainting to make it look pristine for the higher ranking executive VIP, i.e. “God“. The pressure to look good is titanic, and is rolled downhill by the people in charge of a facility to even the lowliest worker. The word goes out, similar to what is usually thought of as a joke: “Jesus is coming. Look busy!”

The absolute last thing that people want to have happen during one of these inspections is for dirty laundry to be aired in front of VIP’s.

Remember, Transocean had a perfect safety record going back years. It was the cream of the crop for drilling rigs. This is why it had been picked to complete a behind schedule, technically difficult project that ought to have been treated with more care from the top down.

To sound that general master alarm would have meant heads would roll. It would have been an embarrassment. The woman could have lost her own job; or, she could have cost the job of one of her bosses and become persona non grata in the team. She would have been squealing, exposing internal business.

In other words, she made herself believe that the alarms couldn’t possibly mean what they did, because she felt the breath of the VIP’s on the back of her neck.

It isn’t that the VIP’s did anything in particular.

It isn’t that the VIP’s ordered her not only to not sound the general master alarm, but to hit the mother of all snooze buttons and silence the alarm system.

It isn’t that the VIP’s made her say they had a situation rather than say they had a well breach.

It is that the presence of VIP’s, in and of itself, made anything else too horrible to contemplate.

So, under immense stress, she cracked.

Not stress because an explosion was pending.

Not stress because she wasn’t trained to do what she should have done.

But stress because the VIP’s were right on the bridge at the critical moment.

For The Record

In the midst of trying to say that none of the general shutdown operations would have necessarily worked, well, it’s like what people say about Communism: they were never really tried.

In particular, this woman’s lawyer makes the point that:

Ms. Fleytas said it never occurred to her to use the emergency shutdown system. In any event, she explained, she had not been taught how to use it. “I don’t know of any procedures,” she said.

No, her job was not to use the emergency shutdown system. Her job was to sound the general master alarm, to alert to other people that they should use the emergency shutdown system.

That’s why a system that relies on people can break down when put under extreme stress: the stress of the presence of VIP’s.

Point of Failure #3

This is a broader point in some sense but… anyway, bear with me:

By 8 p.m., after redoing the test, they all agreed that the Macondo was stable. In a few hours, the drilling crew’s 21-day hitch would be done. They were working unusually fast. In seven years on the Horizon, Joseph Keith had never seen so much activity while sealing a well, and it made him uncomfortable. His job included monitoring gauges that detect blowouts. But all the jobs going on at once — transferring mud to a supply vessel, cleaning mud pits, repairing a pump — could throw off his instruments. Mr. Keith did not tell anyone that he was worried about his ability to monitor the well. “I guess I just didn’t think of it at the time,” he later testified.

You’ll see why I focused on this in a second:

Mr. Holloway and Mr. Barron were working on the main deck when Mr. Holloway happened to glance up at the drilling floor. He could not believe it. Drilling mud was gushing up from the well, just like a water fountain.

It would be nine minutes before the first explosion, well data shows.

Nine precious minutes.

The drilling crew had trained for blowouts. Floorhands like Mr. Holloway were the crucial first responders. A driller would call “Blowout!” and time their response. This usually involved quickly installing a special valve on the drill pipe to end the imagined blowout.

But confronted with the real thing for the first time, Mr. Holloway realized there were no floorhands on the drilling floor to respond.

So why weren’t there any floorhands on the drilling floor to respond? Because they were doing all sorts of other things!!!

This is critically important context. By splitting up the two parts, it may make for a more gripping tale, but the New York Times isn’t doing its most to tell us why those floorhands were absent. Fortunately, I was able to remember immediately and group the two parts together for this post.

No Clear Responsibility – That’s The Problem

So, there’s no one person who clearly deserves the blame. There’s no one person we can point to and say, “He was in charge.” There’s not even a group of easily defined people that we can say, “These people bear the direct responsibility upon their shoulders, and their shoulders only.”

That’s kind of the point. No one was responsible because there was no clear command structure, creating ripe conditions for manipulating safety protocols.

This manipulation did not require orders, either verbal or written. It was simply a product of management pressure through physical presence, talking a real-life safety problem away by making it seem not critically important, by taking advantage of areas of possible ignorance (i.e. Transocean’s people weren’t really trained in how to second guess an initial bad test; that’s why they ought not have tried, but they let themselves get talked into it), and by just in general having a one-track mind and being focused solely on the schedule and on the financial bottom line.

And to a man, these people will say it’s not their fault because they’re oil executives, not drillers, and it was Transocean’s responsibility to drill safely. In other words, Transocean’s people’s responsibility to tell them, to their faces, that they were ignorant hacks with no place on a rig performing a delicate and critical operation, who were rushing the workers and creating stress that encouraged sloppiness and distraction that could lead to a monumental industrial, corporate, and ecological disaster.

Right, no problem. Transocean will get right on it. Sure. Sure it will.

Let’s call this the New and Improved Golden Rule.

He who owns the black gold makes the rules. – Me

There’s no way in hell that Transocean’s people were going to tell the petroleum industry giant BP that it was making operations unsafe through its people acting like they owned the place. The percentage of ownership interest and BP being the operating partner of the well really isn’t the point here; it’s that VIP’s were interfering with the operational command of the oil rig at the exact moment disaster was most likely to strike, and at the exact moment the rig was the most vulnerable, in large part because listening to BP’s sweet whisperings had caused Transocean to distribute its people throughout the oil well on various duties to save time under what is, in hindsight, an infuriating and maddening presumption that things were safe and no one really needed to be minding the store.

A Hard Lesson For Leaders Everywhere

There is no strategy in bringing chaos to your subordinates’ efforts.

When a leader appears, it should be to make things better, and not worse.

It isn’t as if the party line on inspections by VIP’s is obscure and secret. This is how the real world works. An inspection making all sorts of waves is the least likely way to get an accurate view of the true situation, as junior “officers” spare no effort to deceive senior “officers” that things are better than they seem.

A group of high ranking people combines all of the disadvantages of leadership presence while providing none of the benefits.

Failure to establish clarity in who is in charge contributes to disaster.

Those with the knowledge and competence to run local affairs should not be interfered with without proper cause.

To hide behind blurred lines of responsibility while mouthing platitudes of “no one said to compromise safety” ignores that you do not have to say it to bring about that result. It is an abdication of responsibility.

A system that relies on human beings will break down under human created stress far more surely than stress created by inanimate objects.

When people feel they do not have the freedom to raise alarm, figuratively or (in this case) literally, history and training do not matter; people will freeze out of deference to human beings.

When a strategic failure is compounded by a leadership failure, disaster truly strikes.

Just because people go around saying “no one could have predicted this” does not change the above. I admit, “No one could have predicted this” is one of my least favorite lines. It is spoken in cases where it is completely untrue, and where someone should have predicted problems, but chose not to; and where someone should have overcome problems, but came up short.

Please, learn from these fools. Please. – J