The UK’s National Institute for Clinical Excellence has put acupuncture and chiropractic on the list of approved therapies for non-specific lower back pain, which has led, Daniel Davies says, to back-slapping and high-fiving from the crystals and “life force” crowd:
And so, NICE has decided, on the basis of “the evidence”, that acupuncture and chiropractic are a good way to spend the NHS’s money. Except when you look at it, “the evidence” isn’t really all that great. As Edzard Ernst points out, the Cochrane Institute (the other great temple of evidence-based medicine) actually found chiropractic to be more or less useless, while the evidence for acupuncture is that all of the ancient wisdom and theory of the meridians and qi doesn’t actually confer any great benefit over and above that which can be gained from simply lying on a table and being poked with sticks.Part of the problem is that in the specific case of lower back pain, it’s a notoriously difficult condition to understand or treat, and a lot of the art is simply to find a nice and professional-sounding way of saying “live with it, there’s nothing we can do” that doesn’t make the patient give up hope and suffer even more. But another part of the problem is that the overall assessment of what “the evidence” was, was made by a committee that had a bunch of spinal manipulation enthusiasts on it.
Which brings me to the problem; this is exactly what we should have expected, and it’s the reason why I’ve been putting the phrase “the evidence” in great big scare-quotes. Because the actual medical evidence on lower back pain isn’t something that can be nicely summarised in a slim paper guideline; it’s spread out across millions of individual lower backs, some fraction of the experiences of which are summarised into hundreds of research papers, which were then distilled down into the Cochrane review, which was itself processed through the NICE committee. Basically what evidence-based medicine is about, at this level, is somebody making a decision about what the facts are going to be.
And if that decision about “what the facts are” is one that is going to determine the handing out of large chunks of government cash, then you bet that the enthusiasts of every theory there is are going to move hell and high water to get themselves on that committee. Not out of any venial motive, but because they believe in their theory, and a contrary NICE guideline has the potential to kill it stone dead. So what happens is that the process of finding out the underlying truth, which is of necessity slow, unclear and often completely open-ended, gets accelerated and politicised. It’s what you might call “government science”.
It’s a phenomenon that’s very familiar to economists under the name “Goodhart’s Law”. Basically, Goodhart’s Law says that “any economic relationship which is used for policy purposes, ceases to be valid”. In other words, you can have an economic model which works tolerably well as an understanding of how, say, the relationship between money, prices and output works. But when you try to use that model to set interest rates, then suddenly the model itself is part of the recursion — part of the system that you’re trying to control — and this changes the nature of the relationship that you were trying to use.
Similarly, in the early days of the evidence-based medicine movement, when they were the Young Turks or punk rockers, shaking up a complacent medical establishment that had got out of touch with the cutting edge of medical research, they had the potential to do a lot of good. But now they are the establishment, and as a result of that, the very evidence that they rely on, is shaped by the fact that it needs to appeal to them. The fact that a movement which begun by trying to bring science back into medicine, has now ended up putting its imprimateur on some obvious pseudoscience, ought to worry us more than it does, because this is only the most obvious manifestation of the general problem.
(Hat tip to Megan McArdle.)