A quote of the day – re. the Standish Report

28. May 2010 · Write a comment · Categories: Project Management · Tags:

“Our research shows that the Standish definitions of successful and challenged projects have four major problems:

  • they’re misleading,
  • one-sided,
  • pervert the estimation practice,
  • and result in meaningless figures.”

Source: J. Laurenz Eveleens, Chris Verhoef, “The Rise and Fall of the Chaos Report Figures,” IEEE Software, vol. 27, no. 1, pp. 30-36, January/February, 2010. http://www.computer.org/portal/web/csdl/doi/10.1109/MS.2009.154

Also check out the following article:

  • Standish Report – Standish Report and Naive Statistics Finally, A Challenge to the Standish Report Project Failure Rate The current assessment of the Standish Report is: “The Rise and Fall of the Chaos Report Figures,” J.

The Scientific Method and the IT Projects Failure Rate debate

24. May 2010 · 4 comments · Categories: Project Management, Science Corner · Tags:

dc8949c7f8a84165a96d32b096cc4913 The Scientific Method and the IT Projects Failure Rate debateScientific method refers to a body of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge. To be termed scientific, a method of inquiry must be based on gathering observable, empirical and measurable evidence subject to specific principles of reasoning. A scientific method consists of the collection of data through observation and experimentation, and the formulation and testing of hypotheses (source: http://en.wikipedia.org/wiki/Scientific_method).

If you are a regular reader of this blog you will notice that I take great care in substantiating my arguments (for better or worse) with collaborative evidence based on a scientific approach. Applying a scientific approach means that arguments can be independently verified, and supporting sources can be checked and their authenticity confirmed.

Last week I read two articles which made me further concerned about the need to realize and advocate a more scientific approach in regards to project management blogs, some of which are evidently based on gut-feel at levels that are below a reasonable threshold.

Lawrence M. Krauss, in a Scientific American article (Dec 2009 – titled “War Is Peace: Can Science Fight Media Disinformation?”) makes the observation that “The increasingly blatant nature of the nonsense uttered with impunity in public discourse is chilling. Our democratic society is imperiled as much by this as any other single threat, regardless of whether the origins of the nonsense are religious fanaticism, simple ignorance or personal gain.”

A similar note is raised by Jim Giles, in “Living in Denial: Unleashing a lie” (NewScientist.com – 21/05/2010). I encourage you to read Jim’s article as it is a fascinating tale of information (or more precisely dis) information management in the modern era. The key point arising from his article is the ease in which false data can propagate and promulgate to the point where fiction and reality are no longer indistinguishable.

The second article I read last week was published by Geoff Crane of Papercut Edge. In his article (titled “Annual Cost of Project Failure“) he makes a reference to a paper published by Roger Sessions in Nov 2009, titled “The IT Complexity Crisis: Danger and Opportunity“. Again, I encourage you to read Roger Sessions document as it is crucial for understanding the logical flaws in his approach. In a nutshell though, Roger Sessions makes the following assertions:

  • A = 66% of all Federal IT dollars are invested in projects that are “at risk”;
  • B = Let’s assume 65% of the above projects will fail;
  • C = 2.75% = Proportion of GDP spent on IT
  • D = 7.5 = a multiplier representing the total $ impact of a failed project on the economy
  • E = $69,800 (USD Billion) – World Wide GDP
  • Cost of failure = A x B x C x D x E = $6,180 (USD Billion).

I’ve read Roger Sessions paper and realized it heavily relies on data provided in the Budget of the United States Government, Fiscal Year 2009, Analytical Perspective. Before you run quickly to read this document, make sure you go straight to chapter 9, titled “Integrating Services with Information Technology” as this is the place where intellectual challenges associated with the above can be found.

As I was researching the above topic I’ve come across an excellent analysis done by Bruce Webster (see his article titled “The Sessions paper: an analytical critique“). The article provides excellent analytical explanation  outlining the methodological issues arising from Roger Session’s paper. I’m not going to repeat this here because I’d like to encourage you to read Bruce’s article.

The bottom line is that there is on-going state of confusion and misinformation regarding the current rate of IT projects’ failure. I’ve addressed in a number of previous posts (see Related Posts below) my reservations regarding the Standish report and its prolific interpretation. Now both Roger Sessions and Geoff Crane make the point that it is not the numbers that are important as much as the magnitude. Both fail to see that if the numbers are questionable, the magnitude is of no value what-so-ever.

I am seriously concerned, professionally, with the fact that further analysis and interpretation is carried out on the basis of shaky foundations, to the point where claims are taken as facts and these facts are further used to prove unsubstantiated assumptions. I suspect that some of the claims are self propelled by consultants who want to advance their services. Others by those who believe they simply come across a good idea to write an article about. At the end of the day though, regardless of the motives, it is the readers who need to make up their mind. The only way to allow our audience to make a proper judgement call is by providing properly presented, fairly substantiated information. Without such due diligence the words we write are not worth the 0′s and 1′s they are written on.

Projects Failure Rate – the Threequel

28. March 2010 · 9 comments · Categories: Project Management · Tags:

A quick recap

Over the past week I’ve had an interesting discussion with Steven Romero about certain aspects associated with the use of the Standish Chaos Report generally, and how prevalent project failures are specifically.

I’ll skip the discussion we’ve had regarding the validity of the Chaos Report as we both agreed that it is flawed and misleading.

The discussion took on an interesting turn when we started discussing the very nature of what constitutes a failure and, more specifically, is it correct to say that many projects actually fail.

Current Conventional Wisdom

The current conventional wisdom seems to suggest that too many project fail. In an earlier exchange with Patrick Richard I’ve made the observation that in any human endeavor there will always be a line over which further improvements will require infinite levels of resources to attain. Patrick was suggesting in his post that projects’ success rate has flat lined over the years. My response to this was that the reason it looks like we don’t achieve a higher level of success rate is because we have probably reached that level beyond which far greater resources will be required in order to get that line any higher.

Patrick reminded me, quite correctly, that my assertion that a higher level of success rate is not possible is not quite correct as evident from our Space Exploration Projects, where a high degree of success is achieved. This is an absolutely valid point, which in my mind just strengthen my argument, because most projects are not run as space exploration projects, the reason being that in most traditional projects the cost of failure will not necessarily translate to massive explosions or costly loss of human lives. In that respect in most projects there is an acceptance of potential failure by the virtue of limiting budgets. So its not that we can’t do better, it is just that in the main we look at our opportunity costs and make a decision to take the risk of failing. Had our level of tolerance been adjusted to that applied to space exploration projects we would not have had nearly as many ‘failed projects’ as we seem to experience now.

Back to Steven’s Arguments

So now back to Steven’s arguments. I’ve referred him to my discussion with Patrick (as above) and he didn’t quite agree with my arguments (i.e where I suggest that not that many projects actually fail – Steven’s view is that there are far too many project failures). Steven’s rational to his conclusion is based on his definition of what constitutes a failed project. Let’s examine this closely:

Project failure occurs when:

  1. Organizations invest in the wrong projects (not those most essential to meeting Enterprise Strategy)
  2. The speed of the project does not meet the needs of the business (projects are approved predicated on when value is expected to be realized)
  3. The value of the project is not commensurate to the investment in the project (and I find most organizations are not even able to accurately determine investment value – and sometimes, the “actual” cost of the project, which includes indirect, business process change, and full-life-cycle costs)

My response to the above is as follows:

  1. Investment in the wrong project is an organizational issue that has got nothing to do with the projects themselves. This factor cannot be included in the definition of a failed project.
  2. Where the speed of delivery does not meet the business needs, provided that all other factors have been made (i.e. cost, quality, etc) – that should not be categorically classified as failed project. The best source I found for substantiating this point is another post by Steven, titled “Common IT Project Management Mistakes” where he makes the following correct observation: “There is a classic project management saying: Cost, Schedule or Performance – pick any two. Project Managers must know which of these Project Success Factors is most critical in regard to project success.” This says it all. Speed  (i.e. schedule) can in certain circumstanced be compromised provided that Cost and Performance haven’t been affected.
  3. Cost vs. Value – again this also is an organizational issue and have nothing to do with the projects themselves.

To Summarize

So, in summary, things are not as bad as they seem. As I quoted in my earlier post “The increasingly blatant nature of the nonsense uttered with impunity in public discourse is chilling. Our democratic society is imperiled as much by this as any other single threat, regardless of whether the origins of the nonsense are religious fanaticism, simple ignorance or personal gain.”

I couldn’t agree more.

Projects failure rate – the conventional wisdom is wrong!

22. November 2009 · 15 comments · Categories: Project Management · Tags: ,

72e240d8e38e4964a06d3a30170cf591 Projects failure rate – the conventional wisdom is wrong!

The Problem

Don’t be fooled, as despite what you might have heard, told or read, projects’ failure rate is not as high as some might want you to believe. I’ll say it again, now more explicitly: There is no reason to believe any of all these doom and gloom articles and expert papers, suggesting that a catastrophic ratio of projects have failed to deliver.

What I am referring to are expert reports, published in the last 10-15 years, all of which suggesting that the number of projects failed to meet some sort of criteria is nearing 70%! Got that? 7 out of every 10 projects are a failure. Let’s see what they say:

  1. Chaos Report (1994) – only 16.2% of projects were successful by all measures. Of the 70% of projects that were not successful, over 52 percent were partial failures and 31% were complete failures.
  2. OASIG survey (1995) – the IT project success rate quoted revolves around 20-30% based on its most optimistic interviews.
  3. Chaos Report (1995) – The Standish Group research predicts that 31.1% of projects will be cancelled before they ever get completed. Further results indicate 52.7% of projects will cost over 189% of their original estimates.
  4. KPMG Canada Survey (1997) – 61 % reported details on a failed IT project.
  5. Conference Board Survey (2001) – 40 % of the projects failed to achieve their business case within one year of going live.
  6. Robbins-Gioia Survey (2001) – 51 % viewed their ERP implementation as unsuccessful
  7. Dr. Dobb’s Journal (DDJ) Survey (2007) – 72% of all Agile projects were successful, compared to only 63% of traditional of Data Warehouse projects.
  8. Chaos Report (2009) – Only 32% of projects have been defined as being ‘successful’ compared with 35% in 2006.

The Conventional Wisdom

Ok, let’s leave these surveys for a moment and attempt to define what does ‘success’ actually mean. I believe it was John Kenneth Galbraith who, in his 1958 book “The Affluent Society”, coined the term Conventional Wisdom. Conventional Wisdom can be defined as “A belief or set of beliefs that is widely accepted, especially one which may be questionable on close examination”. JKG himself had the following to say about this term: “We associate truth with convenience, with what most closely accords with self-interest and personal well-being or promises best to avoid awkward effort or unwelcome dislocation of life. We also find highly acceptable what contributes most to self-esteem”. Conventional Wisdom, according to JKG represents a convenient view that may or may not be true. It does not mean that any Conventional Wisdom is false, it does suggest though that in some cases underlying assumptions and truths might collapse on close examination.

So what does it mean to have a ‘successful’ project? Furthermore, once defined, could this definition withstand the rigour of life, i.e. can it actually be achieved in real life situations?

The strictest definition for a project success would probably require that the project is completed on-time, on-budget while meeting all its in-scope requirements/specifications. This, by the way, seems to be the definition suggested by The Standish Group whose Chaos Report seems to be the driver for most future projections and success trends. If we were to adopt this definition how would be rate the following scenarios?

  1. The project was delivered on time but was 10% over budget
  2. The project was delivered on budget but was 10% over time
  3. The project was delivered on-time and on-budget but lacked a number of in-scope features.

The Conventional Wisdom is Wrong

According to the current Conventional Wisdom, projects exhibiting the above attributes will most likely be classified as failed projects. This however represents false reality as it is based on a false assumption, according to which project planning is a scientific process that can be executed with a high level of predictability and success rate. This is clearly not the case. Project planning and estimation is, despite all claims to the contrary, a process that is largely dependant on subjective human input and as such cannot be relied upon to guarantee 100% success rate. It is not that humans cannot produce a close to 100% successful processes, they most certainly can, and space missions are the best example to attest to this success. After all each space mission is a project that delivers (at least in most cases) a successful delivery that ends up with successfully returning the astronauts to earth. But the successful completion of the mission only proves that the scope was achieved. It says nothing about the timeliness and costliness of the project. If we were to adopt the Standish Group’s strict definition, there’s a good chance that some (if not all) of the ‘successful’ space missions will be deemed as failures as they failed to meet at least one of the ‘cost’, ‘time’ or ‘scope’ criteria.

As I was writing this article I came across a fascination article in “Scientific American” titled “War Is Peace: Can Science Fight Media Disinformation?” with the sub-title “In the 24/7 Internet world, people make lots of claims. Science provides a guide for testing them”. The author, Lawrence M. Krauss, states that “The increasingly blatant nature of the nonsense uttered with impunity in public discourse is chilling. Our democratic society is imperilled as much by this as any other single threat, regardless of whether the origins of the nonsense are religious fanaticism, simple ignorance or personal gain.”

I couldn’t agree more. The fast pace in which information is released and the large quantities of it do not allow us to apply due diligence and apply common sense and challenge the conventional wisdom thrown at us by experts – all claiming to provide us with their processed truth.

So I, for one, choose not to accept this Conventional Wisdom. I do not accept the definition that requires 100%, all round, success for a project to be deemed successful. If I were to accept it I would probably look for another profession as it would make me, in most cases, a failed professional – which I don’t believe I am.

What do YOU think?

I value your opinion, if you have any thoughts on the above please join in and share with others!