Why Most Published Research Findings are False:
Suppose there are 1000 possible hypotheses to be tested. There are an infinite number of false hypotheses about the world and only a finite number of true hypotheses so we should expect that most hypotheses are false. Let us assume that of every 1000 hypotheses 200 are true and 800 false.
It is inevitable in a statistical study that some false hypotheses are accepted as true. In fact, standard statistical practice guarantees that at least 5% of false hypotheses are accepted as true. Thus, out of the 800 false hypotheses 40 will be accepted as ‘true,’ i.e. statistically significant.
It is also inevitable in a statistical study that we will fail to accept some true hypotheses (Yes, I do know that a proper statistician would say ‘fail to reject the null when the null is in fact false,’ but that is ugly). It’s hard to say what the probability is of not finding evidence for a true hypothesis because it depends on a variety of factors such as the sample size but let’s say that of every 200 true hypotheses we will correctly identify 120 or 60%. Putting this together we find that of every 160 (120 + 40) hypotheses for which there is statistically significant evidence only 120 will in fact be true or a rate of 75% true.
[...]
Ioannidis says most published research findings are false. This is plausible in his field of medicine where it is easy to imagine that there are more than 800 false hypotheses out of 1000. In medicine, there is hardly any theory to exclude a hypothesis from being tested. Want to avoid colon cancer? Let’s see if an apple a day keeps the doctor away. No? What about a serving of bananas? Let’s try vitamin C and don’t forget red wine.
