In his review of How to Measure Anything, Luke Muehlhauser cites this passage about measurement inversion:
By 1999, I had completed the… Applied Information Economics analysis on about 20 major [IT] investments… Each of these business cases had 40 to 80 variables, such as initial development costs, adoption rate, productivity improvement, revenue growth, and so on. For each of these business cases, I ran a macro in Excel that computed the information value for each variable… [and] I began to see this pattern:
- The vast majority of variables had an information value of zero…
- The variables that had high information values were routinely those that the client had never measured…
- The variables that clients [spent] the most time measuring were usually those with a very low (even zero) information value…
Since then, I’ve applied this same test to another 40 projects, and… [I’ve] noticed the same phenomena arise in projects relating to research and development, military logistics, the environment, venture capital, and facilities expansion.
In Robin Hanson’s experience trying to sell prediction markets to firms, they usually express strong reluctance and even hostility to making markets on the specific topics that seem to be of the most info value:
They choose instead to estimate safer safer things, less likely to disrupt the organization.
For example, the most dramatic successes of prediction markets, i.e., where correct market forecasts most differ from official forecasts, are for project deadlines. Yet even hearing this few orgs are interested in starting such markets, and those that do and see dramatic success usually shut them down, and don’t do them again. One plausible explanation is that project managers want the option to say after a failed project “no one could have known about those problems.” Prediction markets instead create a clear record that people did in fact know.
If you’re not familiar with the book, Aretae recommends it highly — and summarizes what he learned:
- Measurement doesn’t quite mean what you think it means, and what it does mean is really important.
- Measures (and estimates) are always ranges + probabilities. They are not just numbers.
- If you ask someone to give you a 90% range, they will be wrong (on the too narrow side). You have to practice to get these ranges right.
- People mostly measure stuff they don’t need to measure at high cost, and don’t measure the important stuff at all.
- For anything you would like to measure, you can…it’s not too hard.
- Statistics + Excel can give you the rest of the information you need, both about what to measure and what the results mean.