A recent discussion in LinkedIn made a bit of a stir when Bent Flyvbjerg (Professor and Chair, University of Oxford) asked the following question:

Most project management forecasters are fools or liars, says new study from Oxford. Do you agree or disagree? Your experience?

The study referred to in the quote above was published by Flyvbjerg and is titled “Quality Control and Due Diligence in Project Management: Getting Decisions Right by Taking the Outside View” (International Journal of Project Management, Nov 2012).

The gist of the study is that:

  1. Decision making done under uncertainty is subject to inaccuracies caused by the prevalence of the “planning fallacy“, described, based on a study by Kahneman and Tversky, as a “systematic fallacy in decision making causing people to underestimate the costs, completion times and risks of planned actions, whereas people overestimate the benefits of the same actions“.
  2. Planning fallacy (according to Kahneman) is a result of decision makers’ inclination to take an “inside view“, whereby they focus “on the constituents of the specific planned action rather than on the outcomes of similar actions already completed
  3. The “inside view” tendency can be overcome (again, according to Kahneman) by adopting the “outside view” approach, “which consists in using experience from previous similar ventures already completed“.
  4. A combined approach, of quality control and due diligence based on the outside view, could be used as a method for evaluating and confirming the validity of forecasts before they get officially accepted.
  5. The above approach is tested on a real-life, multibilion-dollar project.

But back to the original question. The paper brings an interesting discussion, stimulated by Akerlof and Shiler’s concept of “Firing the forecaster”, raised in their book “Animal Spirits: How Human Psychology Drives the Economy, and Why It Matters for Global Capitalism” (note: affiliate link). Akerlof and Shiler attack the notion that current stock prices reflect current people’s interpretation of information pertaining to the future dividends and payoff generated from holding these stocks. They go further to suggest that

To pretend that stock prices reflect people’s use of information about those future payoffs is like hiring a weather forecaster who has gone berserk. He lives in a town where temperatures are fairly stable, but he predicts that one day they will be 150° and on another they will be –100°. Even if the forecaster has the mean of those temperatures right, and even if his exaggerated estimates are at least accurate in calling the relatively hot days and the relatively cold days, he should still be fired. He would make more accurate forecasts on average if he did not predict that there would be any variation in temperature at all. For the same reason, one should reject the notion that stock prices reflect predictions, based on economic fundamentals, about future earnings. Why? The prices are much too variable.

The conclusion suggested by Akerlof and Shiler is to simply fire the forecaster and this, in the context of their discussion, relates to the need to stop listening to the pundits and politicians “who have increasingly been singing the praises— and nothing else— of free markets“.

Now back to Flyvbjerg’s paper. Evaluating the effect that firing the forecaster will have (bearing in mind the scenario at hand – i.e a situation where a forecaster misrepresents his/her ability to predict reliable results) he argues that

…firing the forecaster may be letting off forecasters too easily. Some forecasts are so grossly misrepresented and have such dire consequences that we need to consider not only firing the forecasters but suing them, too, for the losses they incur. In a few cases, where forecasters foreseeably produce deceptive forecasts, criminal penalties may be in place, if the damage caused by forecasters is severe enough.

This is the point in the paper where Flyvbjerg brings a quote from Taleb’s The Black Swan (note: affiliate link):

“People … who forecast simply because ‘that’s my job,’ knowing pretty well that their forecast is ineffectual, are not what I would call ethical. What they do is no different from repeating lies simply because ‘it’s my job.’ Anyone who causes harm by forecasting should be treated as either a fool or a liar. Some forecasters cause more damage to society than criminals.” (Taleb 2010, 163).

Anticipating an obvious question regarding the validity of assuming that most project management forecasters are fools or liars, Flyvbjerg goes on to demonstrate how this is so:

Management research has recently developed strong theory to help identify the “fools” and “liars” Taleb talks about here. Being academic, the theories use more polite and opaque language, needless to say, describing Taleb’s fools as people who are subject to “optimism bias” and the “planning fallacy” and liars as those practising “strategic misrepresentation,” “agency,” and the “conspiracy of optimism”…This research has demonstrated that fools and liars constitute the majority of forecasters and that forecasts are used as sinister tools of fraud more often than we would like to think.

I am somewhat hesitant to accept the last statement as a matter of fact or even as a valid interpretation. Both the “optimism bias” and the “planning fallacy” are known psychological conditions that seem to affect most of the observed population. To suggest that forecasters exhibiting such inclination are “fools” is to suggest that most people are fools. I have no moral issue in calling people who practice “strategic misrepresentation” liars but I am still in the dark as I have not seen any credible stats suggesting the ratio of “liars” in the overall estimators’ population.

All in all, fascinating and thought-provoking paper.

Think about it!

Print Friendly

Related Post

Letter to a Young Project Manager Dear L.J. We have barely met and had only the brief and passing opportunity to exchange a mere few words before a daunting and sombre thought enter...
The First Ever PM FlashBlog is Coming to a Blog Ne... Over the past couple of weeks I have been in touch with dozens of project management related bloggers to organize the first ever coordinated blogging ...
The Ten Commandments of Project Management Over the years I've seen many attempts to construct the "10 commandments of project management". I believe there is an element of cheekiness in this...
The Secret to Clearing the PMP Certification Exam ... The Project Management Body of Knowledge (PMBOK) The PMBOK, published by the PMI, is a compilation of the project management guidelines to be adopted...

3 Comments

  1. Pingback: New PM Articles for the Week of January 14 – 20 | The Practicing IT Project Manager

  2. Hi Shim,
    Overall, I think Prof Brent is definitely on the right track. Until or unless we get both legal and professional ACCOUNTABILITY, not only for project managers, but for cost estimators and schedulers, we will NEVER get any improvements.

    Look at medicine and civil engineering in the USA. While yes, there are incompetent, incapable doctors and engineers, overall the level of both medicine and engineering in the USA is generally considered to be very good. And why is that? Because the legal process has held BOTH of these professions accountable legally and financially for their “errors and omissions”.

    And not to drift too far afield, one of the “bright line tests” of any occupation being a profession is the ability to make INDEPENDENT decisions, with the flip side being legal accountability (“professional negligence”) if those decisions are somehow wrong, neither of which apply to project management or cost estimating or scheduling right now.

    See here for more details on what is needed to raise the professional standing of cost estimators, schedulers and project managers http://pmworldjournal.net/wp-content/uploads/2012/09/PMWJ2-Sep2012-GIAMMALVO-CompetencyofCostEstimators-SecondEdition.pdf

    One last item which you touched on that I think needs clarification. You were talking about ACCURACY of cost estimates. Unfortunately, that is an incomplete construct. Estimates must meet three criteria- they must be ACCURATE, RELIABLE and PRECISE. This graphic explains very nicely the nuances between these three terms http://www.gmat.unsw.edu.au/snap/gps/gps_survey/chap2/241relb.htm. So when we talk about any estimate, we need to talk about three dimensions of that estimate- Is it ACCURATE? Is it RELIABLE and how PRECISE is it. To do otherwise is missing some important dimensions.

    BR,
    Dr. PDG, Jakarta, Indonesia

    Reply

  3. Shim,
    Here is another comment on Prof. Bents paper. He cited that “The 2011 Oxford Handbook of Project Management identifies a “third wave” in project management research characterized by a positioning of the management of projects as a vital part of general management (Morris, Pinto, and Söderlund 2011:3). With projects taking center stage as delivery model for products and services in most organizations, project management can no longer be seen as a specialist subcategory of management brought to bear on special cases, but must instead be seen as a core business activity, vital to organizations as a whole.”

    But when you look at the “end to end” processes using projects as the means to create new assets, (products or services) as described in AACE’s Total Cost Management Framework (TCMF) http://www.aacei.org/aaceonly/tcm/ which integrates Asset Management + Portfolio Management + Program Management + Project Management into a single methodology, haven’t we in fact come full circle to General Management? (Just as Drucker described in his 1973 “Management: Tasks, Responsibilities and Practices”?)

    BR,
    Dr. PDG, Jakarta, Indonesia

    Reply

  4. Pingback: Some Initial Thoughts on the Queensland Government Ban on IBM | quantmleap

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

%d bloggers like this: