My immediate takeaway is that you can’t measure things you cannot base decisions on. So unless there is a meaningful decision you’re trying to make it’s very difficult if not impossible to create meaningful measurements. This means that a lot of people saying “My Special Thing here can’t be measured” because ultimately they have not explained or figured out why they need to measure something or what they are planning to do with this measurement. This also shows that a lot of the KPI’s people see on their “dash boards” are probably kind of meaningless unless there are specific decisions associated with changes in the KPI outputs.
People are not very good at understanding statistics in general, even those who can do the math are often surprised what the math tells them. Yes gut feeling is sometimes right, but when it comes to statics it’s often way off. It is very interesting that he talks about calibrating people by taking various tests; this means that it is possible to train oneself or other people to be more statistically astute. Very cool.
Using various monte carlo calculation methods (mostly using Excel) he shows how to calculate the propabilities of various events. It’s interesting to see that it’s quite possible to caluclate propabilities if you assume that certain things are random and normally distributed. As long as you have a good idea of the 90% confidence interval of the participating variables you can actually calculate quite useful data.
By using EVPI (Expected value of perfect information), EVI (Expected value of information) and ECI (Expected cost of information) it’s possible to figure out at what point measurements is important and how much measurement will actually help. If it’s too expensive to do measurement to determine feasibility of something it might not be worth to do either the measurement or make the decision. For example if an advertising campain will cost $100,000 and it might cost $100,000 to determine how much it will help then it better create at least $200,000 additional income. There are so many “oh but, of course” moments while reading this book.
The books goes into detailed examples on how to do calculations using excel or other tools which can calculate propability distributions associates with various outcomes to figure out how measurements would affect the probability of something to happen. It’s very interesting how little data in some cases can already show so much information where previously no data seemed available.