rdn32

If you’ve ever worked for a company with an officious HR department, you will probably have been told of the importance of setting yourself SMART objectives.

SMART, in this context, is an acronym. It would have to be, really, wouldn’t it? I mean, if you found yourself in an organization where this sort of thing is taken with extreme seriousness then the genuinely smart thing would probably be to get out of there. As an acronym SMART can stand (if you go by Wikipedia and some simple arithmetic) for over three thousand combinations of words. However, we only need to concern ourselves with the most commonly given first two words: “specific” and “measurable”.

Before going any further I should probably admit that personally I’ve never got on well with things of this sort: objective setting and so forth. They may sometimes be helpful, perhaps as catalysts for useful conversations, and even when they’re not I can accept them as harmless nonsense. My hunch, however, is that they are most useful for very ambitious people, whereas I’m more motivated by curiosity than thoughts of promotion. This is hardly an unusual attitude in the software industry, so it generally hasn’t caused me any trouble. This is why I feel free to do things like discussing the Dunning-Kruger effect when asked to rate my own abilities or, as I will be doing here, telling the story of BP and the balanced scorecard.

The story I’m about to tell you comes from an interesting little book by Ray Fishman and Tim Sullivan called The Org: How the Office Really Works, which explains the logic of how companies operate in terms of “organization economics”.

As an aside, one of the things I got out of this book was that it clarified something that had puzzled me during my brief stint as a freelance programming consultant: the logic of what work an organization will do in-house, and what it will outsource. The authors say: “Jobs that stay inside the org are the hard ones: hard to measure, hard to define, and hard to do. If they were easy, we’d hire contractors to do them for us, and the market […] would work just fine at getting the job done.” So if the most important part of what you do in your job genuinely can be neatly specified and unambiguously measured then perhaps you should worry about being a prime candidate for being replaced by a contractor.

Anyway, on to the story of BP.

This starts with the transformation of BP from being a rigidly bureaucratic, state-owned company to being the model of a successful multi-national corporation. BP, and its CEO John Browne, became the toast of Wall Street and the Harvard Business Review. Rather than being subject to tight centralized control from BP’s London headquarters, local management were empowered: they were free to meet the objectives they were set any way they could, and were rewarded for their own performance. Having once been a loss-making dinosaur, BP became hugely profitable and, it seemed, everything a modern corporation should be.

That particular story got knocked off-course in March 23 2005 when there was an explosion at BP’s Texas City refinery which killed 15 people and injured more than 170. This was the worse industrial accident in U.S. history. In the wake of this catastrophe, a rather different story emerged.

BP had acquired the Texas City refinery in 2000 and, despite grave concerns about maintenance, had demanded a 25 percent reduction in costs. “BP paid Texas City management for cost cutting,” Fishman and Sullivan say, “so management cut costs.” If you cut corners you may be lucky for a while, but sooner or later your luck will run out. BP’s luck ran out in 2005 at Texas City, and again in 2006 at Prudhoe Bay, and most dramatically in 2010 with the explosion of the Deepwater Horizon Well in the Gulf of Mexico.

So far the story sounds like the familiar one: a soulless corporation and its greedy executives putting lives at risk in their relentless pursuit of profits. But digging a little deeper we see this isn’t the whole story. If BP saw Texas City in purely financial terms, you might expect the injury rate amongst employees to be bad, or at least typical; whereas in fact injury rates fell under BP’s ownership until in 2004 they were the lowest in the refinery’s history, a third of the industry average.

Why was this so? Well, BP didn’t focus solely on money. They employed a “balanced scorecard” in which a variety of metrics, not just financial, were used for assessing performance. Performance as a whole could be evaluated, and rewarded or punished accordingly. However “[BP’s] balanced scorecard leaned in favor of a short list of things that could easily be measured, such as injury rates.” This is why the injury rate at Texas City improved: the injury rate was one of the things being measured.

Measurements can go wrong in all sorts of ways. Gary Klein gives an example in his book “Sources of Power: How People Make Decisions”. A baby in a neonatal intensive care unit nearly died because the sac surrounding its heart filled with air, effectively paralysing the heart; but this was nearly missed due to an electronic monitor reporting that the baby’s heart was beating at around 80 times per minute. The problem was that the monitor was actually measuring electrical impulses in the baby’s heart, and not whether the heart was pumping blood around the baby’s body: with the baby’s heart paralysed the impulses had no effect.

However, this is a rather poor analogy for what I’ve been talking about, since in the case of a heart monitor the thing being measured (electrical impulses) and the thing of primary concern (heart beats) are precisely correlated in all normal circumstances and most abnormal ones too. Moreover the relationship is causal, and even in the rare, pathological cases where it breaks down the reasons for this are understood.

None of this is true in the case of using injury rates as a metric for general safety, as BP were doing. There isn’t any particular reason to suppose that minor accidents, once singled out for attention, must correlate with any other aspect of safety. The salient feature of injury rates is that they can be measured. Whereas the probability of a one-off catastrophe cannot be measured: the catastrophe either happens or (more normally) doesn’t happen. The risk of catastrophe is invisible.

People sometimes say of metrics that whilst they may not be perfect, they are at least better than nothing. This isn’t always true. Had BP lacked a metric for safety they would have been in a state of honest ignorance, and might have been more cautious in their quest for reduced costs. As it was, BP fooled themselves and their admirers that they had safety “covered” in their balanced scorecard, and so they let their metrics do their thinking for them. As Daniel Kahneman puts it, the human mind tends to go by the principle that “what you see is all there is.”

These buzzword bingo phrases — “specific and measurable objectives”, “balanced scorecard” — make their users feel so wise. But they’re not just harmless pieces of corporate nonsense, you know. They’re nonsense with a body count. BP, with their narrow focus on what was specific and measurable, couldn’t effectively deal with things which were neither. The miracle of BP’s system was supposed to be that they ensured maximum profits at minimum operating risk: their scorecards dazzled even as they failed to illuminate.