Megan McArdle shares some stories of metrics and their unintended consequences:
In December, doctors at a VA hospital in Oregon decided to admit an 81-year-old patient. He was dehydrated, malnourished, plagued by skin ulcers and broken ribs — in the medical professionals’ opinion, he was unable to care for himself at home. Administrators, however, overruled them.
Was there no bed for this poor man? No, the facility had plenty of beds; in fact, on an average day, more than half of the beds are empty, awaiting patients. Was there no money or medicine to care for him? No, and no. Reporting by the New York Times suggests that Walter Savage was, perversely, turned away because he was too sick. Very sick patients tend to worsen the performance measures by which VA hospitals are judged.
If this had happened in isolation, we could simply gape at the monstrosity that bureaucracies are occasionally capable of.
But such examples abound in health care. For example, in the 1990s, New York and Pennsylvania started publishing mortality data on hospitals and surgeons who did coronary bypasses. The idea was that more informed consumers would steer themselves toward the teams with the better statistics — theoretically good for patients, bad for slacking providers. The reality was less ideal: In those states, surgeons seem to have started doing more operations on healthier patients, while turning away the sickest ones who might otherwise have benefited.
From this we can take a few lessons. The first is one that has been well-known to other sorts of businesses: What you measure is what you get, not necessarily what you want. In fact, if your measurement is badly designed, you may get a great deal of something you don’t want.
To illustrate that, look at Wells Fargo, which recently paid a whopping fine because a badly designed compensation system encouraged low-level employees to muck around with customer bank accounts. These machinations generated effectively no revenue for the bank, and annoyed customers, but they did generate income for the employees — and eventually, a stinging, expensive rebuke from the Consumer Financial Protection Bureau.
[...]
I could reel off examples endlessly: purchasing managers who have cozy arrangements to buy a certain amount of product from their vendors in December, and ship it back in January, in order to help some sales director make quarterly targets … universities that compete to turn away as many students as possible, because doing so makes them rise in the U.S. News rankings … law schools that hired their own graduates for temporary make-work jobs in order to boost the schools’ employment statistics. All metrics will be gamed, and the games always have costs. And when the metrics involve our health, those costs can be very high indeed.
Health care and education are particularly ill-suited to management-by-measurement:
Most companies are dealing with reasonably standardized inputs, which can be turned into measurable outputs. But the less you deal with things, and the more you deal with human beings, the less useful productivity metrics are. Human bodies and human minds are both highly variable and immensely complicated. When you are working on them, it is hard to know how much of the final result is a result of your labor, and how much can be credited to the qualities of your initial starting material.
So when we measure outputs, we are getting at best a very distorted picture of the value of the services provided. Modern industrial management is simply not designed for this sort of situation. If you feed human inputs into a machine system, you are quite likely to grind up the humans in the process.
I have heard that one of the consequences of this phenomenon is the over-prescription of opiates. Hospitals are graded on the rate of readmission of discharged patients. Administrators found that one of the main causes of readmission was pain, so hospitals started giving more patients opiates to prevent them from coming back. This had the secondary benefit of allowing hospitals to discharge patients earlier than had been standard as well. So from the hospital’s business perspective, giving out opiates was a win-win.
I work in healthcare IT. We are constantly being asked to provide metrics to leadership and other groups. We are often astounded by how poorly thought out many of these requests are, as “validity” seems to be a completely foreign concept to most of the people asking for reports and whatnot.
re: “I have heard that one of the consequences of this phenomenon is the over-prescription of opiates. Hospitals are graded on the rate of readmission of discharged patients. Administrators found that one of the main causes of readmission was pain, so hospitals started giving more patients opiates to prevent them from coming back.” Beor etc.
As an individual that went through three accidents where pain was so bad that only opiates controlled the pain (spiral fx of humerus, herniated lumbar disc, hip fx) I can say opiates were a blessing. As I healed the need for pain relief diminished and the opiates were withdrawn but during that acute time of need my surgeon/physician gave me blessed relief. All of that was years ago. Recently my wife herniated a lumbar disk and NOW her physician did not start with an opiate to relieve her severe pain. Because of the “opiate crisis” he used much weaker drugs and “titrated” her up over time to an opiate. She had over a week of torture before she had true pain relief. Neither of us were anywhere near being addicted by the use of opiates for pain relief. Neither of us craved opiates after the pain subsided. Look for another factor than pain to explain the Opiate Crisis.
“Most companies are dealing with reasonably standardized inputs, which can be turned into measurable outputs”
It’s hard enough in business, when you are dealing with things like revenue, profit, and production quantities. The establishment of a new sales compensation system, for example, will often lead to behavior which was not predicted and is not desirable from a company point of view.
And I remember talking to one factory manager who said he didn’t like piecework systems because the workers, trying to make a little more, would do things not optimal from a safety perspective. 99.99% of the time it would be fine, but occasionally someone would lose a finger.
For a couple of interesting examples of incentive systems that misdirected, see my post Stupidity — Communist-Style and Capitalist-Style.
I’m with Dan Kurt on the use of opiates. I had an accident and without them — I know, I stopped taking them — I was in severe pain. Major pain. With them it was a dull pain, but without it was like torture. Not everyone gets hooked on opiates. The ones that do we should concentrate on helping them and not punishing everyone.
David, the version of that Soviet bathtub story that I heard involved cars — and people stealing each other’s windshield wipers.