Most project management forecasters are fools or liars, says new study from Oxford. Do you agree or disagree? Your experience?
The study referred to in the quote above was published by Flyvbjerg and is titled “Quality Control and Due Diligence in Project Management: Getting Decisions Right by Taking the Outside View” (International Journal of Project Management, Nov 2012).
The gist of the study is that:
- Decision making done under uncertainty is subject to inaccuracies caused by the prevalence of the “planning fallacy“, described, based on a study by Kahneman and Tversky, as a “systematic fallacy in decision making causing people to underestimate the costs, completion times and risks of planned actions, whereas people overestimate the benefits of the same actions“.
- Planning fallacy (according to Kahneman) is a result of decision makers’ inclination to take an “inside view“, whereby they focus ”on the constituents of the specific planned action rather than on the outcomes of similar actions already completed“
- The “inside view” tendency can be overcome (again, according to Kahneman) by adopting the “outside view” approach, “which consists in using experience from previous similar ventures already completed“.
- A combined approach, of quality control and due diligence based on the outside view, could be used as a method for evaluating and confirming the validity of forecasts before they get officially accepted.
- The above approach is tested on a real-life, multibilion-dollar project.
But back to the original question. The paper brings an interesting discussion, stimulated by Akerlof and Shiler’s concept of “Firing the forecaster”, raised in their book “Animal Spirits: How Human Psychology Drives the Economy, and Why It Matters for Global Capitalism” (note: affiliate link). Akerlof and Shiler attack the notion that current stock prices reflect current people’s interpretation of information pertaining to the future dividends and payoff generated from holding these stocks. They go further to suggest that
To pretend that stock prices reflect people’s use of information about those future payoffs is like hiring a weather forecaster who has gone berserk. He lives in a town where temperatures are fairly stable, but he predicts that one day they will be 150° and on another they will be –100°. Even if the forecaster has the mean of those temperatures right, and even if his exaggerated estimates are at least accurate in calling the relatively hot days and the relatively cold days, he should still be fired. He would make more accurate forecasts on average if he did not predict that there would be any variation in temperature at all. For the same reason, one should reject the notion that stock prices reflect predictions, based on economic fundamentals, about future earnings. Why? The prices are much too variable.
The conclusion suggested by Akerlof and Shiler is to simply fire the forecaster and this, in the context of their discussion, relates to the need to stop listening to the pundits and politicians “who have increasingly been singing the praises— and nothing else— of free markets“.
Now back to Flyvbjerg’s paper. Evaluating the effect that firing the forecaster will have (bearing in mind the scenario at hand – i.e a situation where a forecaster misrepresents his/her ability to predict reliable results) he argues that
…firing the forecaster may be letting off forecasters too easily. Some forecasts are so grossly misrepresented and have such dire consequences that we need to consider not only firing the forecasters but suing them, too, for the losses they incur. In a few cases, where forecasters foreseeably produce deceptive forecasts, criminal penalties may be in place, if the damage caused by forecasters is severe enough.
This is the point in the paper where Flyvbjerg brings a quote from Taleb’s The Black Swan (note: affiliate link):
“People … who forecast simply because ‘that’s my job,’ knowing pretty well that their forecast is ineffectual, are not what I would call ethical. What they do is no different from repeating lies simply because ‘it’s my job.’ Anyone who causes harm by forecasting should be treated as either a fool or a liar. Some forecasters cause more damage to society than criminals.” (Taleb 2010, 163).
Anticipating an obvious question regarding the validity of assuming that most project management forecasters are fools or liars, Flyvbjerg goes on to demonstrate how this is so:
Management research has recently developed strong theory to help identify the “fools” and “liars” Taleb talks about here. Being academic, the theories use more polite and opaque language, needless to say, describing Taleb’s fools as people who are subject to “optimism bias” and the “planning fallacy” and liars as those practising ”strategic misrepresentation,” “agency,” and the “conspiracy of optimism”…This research has demonstrated that fools and liars constitute the majority of forecasters and that forecasts are used as sinister tools of fraud more often than we would like to think.
I am somewhat hesitant to accept the last statement as a matter of fact or even as a valid interpretation. Both the “optimism bias” and the “planning fallacy” are known psychological conditions that seem to affect most of the observed population. To suggest that forecasters exhibiting such inclination are “fools” is to suggest that most people are fools. I have no moral issue in calling people who practice “strategic misrepresentation” liars but I am still in the dark as I have not seen any credible stats suggesting the ratio of “liars” in the overall estimators’ population.
All in all, fascinating and thought-provoking paper.
Think about it!