quantmleap

Dr Donald Ferguson, CTO of CA Technologies (cited in “The mantra of CA Technologies’ Donald Ferguson: Simplify“, 9 December 2010):

But the biggest problem is the complexity of IT. If you look at any IT in business, it is so complex it hinders the ability to be agile and efficient. Throughout the history of IT we all have said that we will make it simpler by adding more things to it, but that makes it more and more complex.

The key term used more often in technology projects is ‘integration’. This term encapsulates multitude of diverse activities, all evolved around the creation of a seamless flow of information across systems and technologies that have absolutely nothing in common. It could be building up a simple interface to read data stored in one database environment and ‘moving’ that data into another database or application. It can also mean the creation of complex data transports across diverse networks and incompatible data formats.

The principle behind any integration work is not complex. In fact, on the diagram where the solution is normally outlined, it seems like a simple set of arrows connecting various systems, technologies, protocols and environments together. And this is where the simplicity ends. Each of these harmless lines represents a collection of risks, starting with the level of technological competence required to implement it, to the availability of competent resources, vendor support, political pressures to use (or not to use) that technology, etc.

As a project manager you don’t always have the luxury of being involved at the stage of technology selection. Irrespective of that, however, the set of technologies you inherent require you to carefully identify the risks these decisions will impose on your project and manage the risks accordingly.

As a project manager you would expect that the technologies you use will provide an ‘enabling’ outcome, i.e. they will positively contribute to the objectives of the project. To determine whether or not this is the case you should ask your technology experts the following questions:

1. What do we need this technology for?
2. What tangible benefits would it deliver? Given the portfolio of technologies at our disposal, can any other technology provide similar (or better) functionality?
3. Have we used this technology before? Do we have access to competent resources that can drive this technology in the way we want them to?
4. How easy is it to integrate this technology with others proposed for the project? Has it been done before? Have we got the expertise and know-how of how to best make any collection of ‘joining technologies’ talk to each other? And if not, what do we need to do to make it work?
5. Have we designed a solution that represents more the ‘Conway’s Law’ than an objectively smarter solution? 

Use the above questions (and their detailed derivations) to clearly articulate the projects’ risks. Use it as an opportunity to inform decision makers of their responsibility to make it easy for you and others to get the job done.

Think about it!

In a previous post titled “Too much technology will be harmful to your project” I’ve made the observation that:

Having been in the Information Technology sector for almost 30 years I can confidently and comfortably conclude that innovation, flexibility and grand availability of sophisticated and feature rich technologies have resulted in substantial increase to projects’ complexity with an increased risk of project integration failures.

With that in mind, I’ve come across  a PC World article titled “Complexity of IT Systems Will Be Our Undoing” where John Dix interviews Roger Sessions, CTO of ObjectWatch, who makes the following observation:

But ultimately the only variable that appears to correlate closely with failure is complexity.So my basic proposal is that as systems get bigger and more expensive they get more complex and complex things are harder to deal with and therefore more likely to fail.

My two cents; the complexity of contemporary IT systems is directly related to increased complexity of organizational and IT processes. This increased complexity is further negatively contributing to projects’ successes.

Think about it!

Related articles

• Complexity & IT (ericbrown.com)
• Will the complexity of IT systems be our undoing? (infoworld.com)
• Complexity of IT Systems Will Be Our Undoing (pcworld.com)

Having referred in an earlier post to the issues arising  from needing to comply with too much process, I’ve come across a beautiful post by Eric D. Brown titled “Complexity & IT“. Eric has been diligent enough to read through a rather long memo published by Ray Ozzie who, five years ago, joined Microsoft as a Chief Software Architect. The memo, titled “Dawn of a New Day“, includes the following statement:

“Complexity kills. Complexity sucks the life out of users, developers and IT.  Complexity makes products difficult to plan, build, test and use.  Complexity introduces security challenges.  Complexity causes administrator frustration.”

In attempting to reduce the mounting road toll, the Australian Police has introduced the slogan “Speed Kills“.

In the context of IT Project Management it could easily be argued that “Process Kills“. It does not kill people but it certainly kills innovation, kills originality, kills inventiveness and kills imagination.

Think about it!

Conway’s Law teaches us that a complex organizational structure often results in complex systems. Assuming that organizational complexity is also a result of (or perhaps a catalyst to) the introduction of diverse technologies, it can be deduced that:

“An organization that designs a system is bound to produce designs which mirror the myriad of technologies the organization utilizes“.

Think about it!

I wrote in an earlier post (“how much process is too much process?“) about the tendency of some organizations to impose overly engineered processes in a (futile) attempt to increase quality.

Further to that and having been in the Information Technology sector for almost 30 years I can confidently and comfortably conclude that innovation, flexibility and grand availability of sophisticated and feature rich technologies have resulted in substantial increase to projects’ complexity with an increased risk of project integration failures.

Information Technology project tend to include, in one way or another, a number of database, networking, server, software packages and other perhaps more obscure technologies. Each of these technologies require the services of highly technical and trained personnel in order to best use the respective  tools to achieve the best result for the project. Now, the problem with today’s set of technologies is that they are all highly flexible, powerful, customizable and configurable; to the point that a considerable amount of effort need to be spent in order to ensure that the enormous capability built into the tool is harnessed appropriately.

Let’s use a common presentation tool as an analogy to explain the point above. Preparing a presentation is an easy task (provided you have the right tool and you know what you intend to convey to your audience). If you are an average user of Microsoft PowerPoint (or any other similar tool) you will surly know that in addition to the basic presentation capabilities you can introduce complex transitions, timers, sound and multimedia to enhance your presentation and turn it into an Oscar worthy event. Now, most people don’t require this extra elaborate capability and indeed, for many, this extra functionality is more of a distraction than an enabler as they get tempted by the available technology in an attempt to make use of all these gadgets because they are there.

A similar phenomenon which can be easily observed at the micro level can be seen also at the macro level. Organizations are lured into deploying and using large number of technologies, each of which come with a promise of amazing flexibility and functionality, while the consequences of deploying these technologies are not properly understood.

The questions I would like organizations to ask before deploying any new technology are:

1. Do we need this technology?
2. What tangible benefits, based on a clear selection criteria, does this technology have that we don’t already have today?
3. What percentage of the marketed feature would we actually use, compared with those features that we are really excited about but will never ever get used?
4. Assuming we conclude that we really, really, really need this technology, do we understand the impact on integrating this wonderful thing with our existing technologies?

The Conway’s Law which I got introduced to some time ago states that:

“…organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations“.

I would like to propose a slight variation on the above law (which I humbly call “Shim’s Law of the impact of Advanced technologies on projects’ risk“)

“organizations which design systems are bound to produce designs which mirror the myriad of technologies the organization utilizes“.

Think about it!

To ensure the successful completion of a project, it is of utmost importance for the project manager to find ways to handle uncertainties that can pose potential risks for a project. Risk management is an iterative process. Risks can relate to any aspect of the project – be it the cost, schedule, or quality. The key to managing risks is to identify them early on in the project and develop an appropriate risk response plan.

To develop a Risk Response Plan, you need to quantify the impact of risks on the project. This process is known as quantitative risk analysis wherein risks are categorized as high or low priority risks depending on the quantum of their impact on the project. The Project Management Body of Knowledge (PMBOK) advocates the use of Monte Carlo analysis for performing quantitative risk analysis.

What is Monte Carlo Analysis?

Monte Carlo analysis involves determining the impact of the identified risks by running simulations to identify the range of possible outcomes for a number of scenarios. A random sampling is performed by using uncertain risk variable inputs to generate the range of outcomes with a confidence measure for each outcome. This is typically done by establishing a mathematical model and then running simulations using this model to estimate the impact of project risks. This technique helps in forecasting the likely outcome of an event and thereby helps in making informed project decisions.

While managing a project, you would have faced numerous situations where you have a list of potential risks for the project, but you have no clue of their possible impact on the project. To solve this problem, you can consider the worst-case scenario by summing up the maximum expected values for all the variables. Similarly, you can calculate the best-case scenario. You can now use the Monte Carlo analysis and run simulations to generate the most likely outcome for the event. In most situations, you will come across a bell-shaped normal distribution pattern for the possible outcomes.

Let us try to understand this with the help of an example. Suppose you are managing a project involving creation of an eLearning module. The creation of the eLearning module comprises of three tasks: writing content, creating graphics, and integrating the multimedia elements. Based on prior experience or other expert knowledge, you determine the best case, most-likely, and worst-case estimates for each of these activities as given below:

The Monte Carlo simulation randomly selects the input values for the different tasks to generate the possible outcomes. Let us assume that the simulation is run 500 times. From the above table, we can see that the project can be completed anywhere between 11 to 23 days. When the Monte Carlo simulation runs are performed, we can analyse the percentage of times each duration outcome between 11 and 23 is obtained. The following table depicts the outcome of a possible Monte Carlo simulation:

This can be shown graphically in the following manner:

What the above table and chart suggest is, for example, that the likelihood of completing the project in 17 days or less is 33%. Similarly, the likelihood of completing the project in 19 days or less is 88%, etc. Note the importance of verifying the possibility of completing the project in 17 days, as this, according to the Most Likely estimates, was the time you would expect the project to take. Given the above analysis, it looks much more likely that the project will end up taking anywhere between 19 – 20 days.

Benefits of Using Monte Carlo Analysis

Whenever you face a complex estimation or forecasting situation that involves a high degree of complexity and uncertainty, it is best advised to use the Monte Carlo simulation to analyze the likelihood of meeting your objectives, given your project risk factors, as determined by your schedule risk profile. It is very effective as it is based on evaluation of data numerically and there is no guesswork involved. The key benefits of using the Monte Carlo analysis are listed below:

• It is an easy method for arriving at the likely outcome for an uncertain event and an associated confidence limit for the outcome. The only pre-requisites are that you should identify the range limits and the correlation with other variables.
  
• It is a useful technique for easing decision-making based on numerical data to back your decision.
  
• Monte Carlo simulations are typically useful while analyzing cost and schedule. With the help of the Monte Carlo analysis, you can add the cost and schedule risk event to your forecasting model with a greater level of confidence.
  
• You can also use the Monte Carlo analysis to find the likelihood of meeting your project milestones and intermediate goals.
  

Now that you are aware of the Monte Carlo analysis and its benefits, let us look at the steps that need to be performed while analysing data using the Monte Carlo simulation.

Monte Carlo Analysis: Steps

The series of steps followed in the Monte Carlo analysis are listed below:

1. Identify the key project risk variables.
   
2. Identify the range limits for these project variables.
   
3. Specify probability weights for this range of values.
   
4. Establish the relationships for the correlated variables.
   
5. Perform simulation runs based on the identified variables and the correlations.
   
6. Statistically analyze the results of the simulation run.
   

Each of the above listed steps of the Monte Carlo simulation is detailed below:

1. Identification of the key project risk variables: A risk variable is a parameter which is critical to the success of the project and a slight variation in its outcome might have a negative impact on the project. The project risk variables are typically isolated using the sensitivity and uncertainty analysis.
   

   Sensitivity analysis is used for determining the most critical variables in a project. To identify the most critical variables in the project, all the variables are subjected to a fixed deviation and the outcome is analysed. The variables that have the greatest impact on the outcome of the project are isolated as the key project risk variables. However, sensitivity analysis in itself might give some misleading results as it does not take into consideration the realistic nature of the projected change on a specific variable. Therefore it is important to perform uncertainty analysis in conjunction with the sensitivity analysis.
   

   Uncertainty analysis involves establishing the suitability of a result and it helps in verifying the fitness or validity of a particular variable. A project variable causing high impact on the overall project might be insignificant if the probability of its occurrence is extremely low. Therefore it is important to perform uncertainty analysis.
   

2. Identification of the range limits for the project variables: This process involves defining the maximum and minimum values for each identified project risk variable. If you have historical data available with you, this can be an easier task. You simply need to organize the available data in the form of a frequency distribution by grouping the number of occurrences at consecutive value intervals. In situations where you do not have exhaustive historical data, you need to rely on expert judgement to determine the most likely values.
   
3. Specification of probability weights for the established range of values: The next step involves allocating the probability of occurrence for the project risk variable. To do so, multi-value probability distributions are deployed. Some commonly used probability distributions for analyzing risks are normal distribution, uniform distribution, triangular distribution, and step distribution. The normal, uniform, and triangular distributions are even distributions and establish the probability symmetrically within the defined range with varying concentration towards the centre. Various types of commonly used probability distributions are depicted in the diagrams below:
   

   
   

   
   

4. Establishing the relationships for the correlated variables: The next step involves defining the correlation between the project risk variables. Correlation is the relationship between two or more variables wherein a change in one variable induces a simultaneous change in the other. In the Monte Carlo simulation, input values for the project risk variables are randomly selected to execute the simulation runs. Therefore, if certain risk variable inputs are generated that violate the correlation between the variables, the output is likely to be off the expected value. It is therefore very important to establish the correlation between variables and then accordingly apply constraints to the simulation runs to ensure that the random selection of the inputs does not violate the defined correlation. This is done by specifying a correlation coefficient that defines the relationship between two or more variables. When the simulation rounds are performed by the computer, the specification of a correlation coefficient ensures that the relationship specified is adhered to without any violations.
   
5. Performing Simulation Runs: The next step involves performing simulation runs. This is typically done using a simulation software and ideally 500 – 1000 simulation runs constitute a good sample size. While executing the simulation runs, random values of risk variables are selected with the specified probability distribution and correlations.
   
6. Statistical Analysis of the Simulation Results: Each simulation run represents the probability of occurrence of a risk event. A cumulative probability distribution of all the simulation runs is plotted and it can be used to interpret the probability for the result of the project being above or below a specific value. This cumulative probability distribution can be used to assess the overall project risk.
   

Summary

Monte Carlo simulation is a valuable technique for analyzing risks, specifically those related to cost and schedule. The fact that it is based on numeric data gathered by running multiple simulations adds even greater value to this technique. It also helps in removing any kind of project bias regarding the selection of alternatives while planning for risks. While running the Monte Carlo simulation, it is advisable to seek active participation of the key project decision-makers and stakeholders, specifically while agreeing on the range values of the project risk variables and the probability distribution patterns to be used. This will go a long way in building stakeholder confidence in your overall risk-handling capability for the project. Moreover, this serves as a good opportunity to make them aware of the entire risk management planning being done for the project.

Though there are numerous benefits of the Monte Carlo simulation, the reliability of the outputs depends on the accuracy of the range values and the correlation patterns, if any, that you have specified during the simulation. Therefore, you should practice extreme caution while identifying the correlations and specifying the range values. Else, the entire effort will go waste and you will not get accurate results.

Scientific method refers to a body of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge. To be termed scientific, a method of inquiry must be based on gathering observable, empirical and measurable evidence subject to specific principles of reasoning. A scientific method consists of the collection of data through observation and experimentation, and the formulation and testing of hypotheses (source: http://en.wikipedia.org/wiki/Scientific_method).

If you are a regular reader of this blog you will notice that I take great care in substantiating my arguments (for better or worse) with collaborative evidence based on a scientific approach. Applying a scientific approach means that arguments can be independently verified, and supporting sources can be checked and their authenticity confirmed.

Last week I read two articles which made me further concerned about the need to realize and advocate a more scientific approach in regards to project management blogs, some of which are evidently based on gut-feel at levels that are below a reasonable threshold.

Lawrence M. Krauss, in a Scientific American article (Dec 2009 – titled “War Is Peace: Can Science Fight Media Disinformation?”) makes the observation that “The increasingly blatant nature of the nonsense uttered with impunity in public discourse is chilling. Our democratic society is imperiled as much by this as any other single threat, regardless of whether the origins of the nonsense are religious fanaticism, simple ignorance or personal gain.”

A similar note is raised by Jim Giles, in “Living in Denial: Unleashing a lie” (NewScientist.com – 21/05/2010). I encourage you to read Jim’s article as it is a fascinating tale of information (or more precisely dis) information management in the modern era. The key point arising from his article is the ease in which false data can propagate and promulgate to the point where fiction and reality are no longer indistinguishable.

The second article I read last week was published by Geoff Crane of Papercut Edge. In his article (titled “Annual Cost of Project Failure“) he makes a reference to a paper published by Roger Sessions in Nov 2009, titled “The IT Complexity Crisis: Danger and Opportunity“. Again, I encourage you to read Roger Sessions document as it is crucial for understanding the logical flaws in his approach. In a nutshell though, Roger Sessions makes the following assertions:

• A = 66% of all Federal IT dollars are invested in projects that are “at risk”;
• B = Let’s assume 65% of the above projects will fail;
• C = 2.75% = Proportion of GDP spent on IT
• D = 7.5 = a multiplier representing the total $ impact of a failed project on the economy
• E = $69,800 (USD Billion) – World Wide GDP
• Cost of failure = A x B x C x D x E = $6,180 (USD Billion).

I’ve read Roger Sessions paper and realized it heavily relies on data provided in the Budget of the United States Government, Fiscal Year 2009, Analytical Perspective. Before you run quickly to read this document, make sure you go straight to chapter 9, titled “Integrating Services with Information Technology” as this is the place where intellectual challenges associated with the above can be found.

As I was researching the above topic I’ve come across an excellent analysis done by Bruce Webster (see his article titled “The Sessions paper: an analytical critique“). The article provides excellent analytical explanation  outlining the methodological issues arising from Roger Session’s paper. I’m not going to repeat this here because I’d like to encourage you to read Bruce’s article.

The bottom line is that there is on-going state of confusion and misinformation regarding the current rate of IT projects’ failure. I’ve addressed in a number of previous posts (see Related Posts below) my reservations regarding the Standish report and its prolific interpretation. Now both Roger Sessions and Geoff Crane make the point that it is not the numbers that are important as much as the magnitude. Both fail to see that if the numbers are questionable, the magnitude is of no value what-so-ever.

I am seriously concerned, professionally, with the fact that further analysis and interpretation is carried out on the basis of shaky foundations, to the point where claims are taken as facts and these facts are further used to prove unsubstantiated assumptions. I suspect that some of the claims are self propelled by consultants who want to advance their services. Others by those who believe they simply come across a good idea to write an article about. At the end of the day though, regardless of the motives, it is the readers who need to make up their mind. The only way to allow our audience to make a proper judgement call is by providing properly presented, fairly substantiated information. Without such due diligence the words we write are not worth the 0′s and 1′s they are written on.

I’ve recently come across an interesting discussion in HarvardBusiness.org titled “For Startups, How Much Process Is Too Much?“. The author of the article, Eric Ries, discusses the three “Build”, “Measure” and “Learn” stages applicable to start-up businesses. From my perspective, these three stages are very relevant to any new set of processes, irrespective if the business is well established or not. In the context of project management, establishing, or enforcing a set of management and control processes would need to be monitored along these three stages, at least in the early days of process adoption and, most likely, throughout their life-cycle when they get modified and monitored.

Over the years I’ve been involved in a number of governance discussions associated with the integration of a Project Management System (PMS) with a Software Development Life-cycle (SDLC) methodology, resulting in an all-arching Project Management Delivery Framework (PMDF). It never ceased to amaze me how ‘process rich’ (which in this context should be taken as a euphemism for ‘complex’) these frameworks tended to be.

PMP certified project managers who are familiar with the 9 knowledge areas and 5 process groups will be aware of the number of artefacts required to have all the underlying processes managed from a Project Management perspective.

When superimposing the artefacts mandated by the software development methodology (irrespective which one it is) we end up talking about a major overhead imposed on the project, an overhead whose material impact is cost and time for a fixed level of scope.

just to illustrate (very simplistically) the complexity associated with a standard software development project we need to remember that when talking about the PMBOK Execution Process Group, then within the context of a software development project, that will include a number of phases, including Planning, Analysis, Design, Construction, Testing, etc.

Considering the fact that each of these phases can affect or be affected by a large number of factors, including servers, networks, operating systems, security considerations, business areas, software packages, technology limitations, geographical locations etc. it is easy to see why the number of project deliverables (as opposed to product deliverables) could end up being humongous.

The inherent problem with the way most traditional projects are run is that some processes are enforced because they are seen as the only valid way to manage the built-in uncertainty that exists with any endeavour. So rather then following a process because it is ‘the right thing to do’, some (if not many) are followed while the only purpose they serve is providing a control and validation mechanism.

While I don’t have a simple solution to this predicament it is clear to me that in many cases projects are unnecessarily complex and carry a high degree of fat that with some effort could be disposed of. Let’s be honest about it, in many cases, we are so process-holic because we are concerned about the consequences (to us personally) should something go wrong at a later stage and the wolves will be out to look for the scapegoats. We would then need to explain how come we failed to obtain a 3rd revision sign-off to a scope/requirements/design/etc. document from the 9th reviewer/authorizer who happened to be overseas for an extended period of time and neglected to empower another business/technical/customer/etc. person to review and sign-off on his/her behalf.

Changing this situation requires a fundamental cultural change where it becomes ok to make mistakes. I’m not talking about allowing series offenders to rein free in failed projects, but I am suggesting that once we deploy the appropriate professionals into their respective roles we let them collaborate and make ad-hoc decisions to make sure that things are progressing in the most efficient way without being bogged down by prohibitive processes. Let’s empower our teams to agree what is required and what is not required on the way to achieving the project’s objectives. Let’s bring common sense and allow our team members to think about what really needs to be done and not about how to cover themselves should anything go wrong.

Let’s THINK.

In part 1 of this article I raised a number of risk related observations, particularly around the validity of Murphy’s Law as well as the reality behind the Law of Averages.

Another series of Scientific American articles (sorry but I’m a real Scientific American fan), titled “Why Our Brains Do Not Intuitively Grasp Probabilities” and “How Randomness Rules Our World and Why We Cannot See It“ describes the concept of “Folk Numeracy” which is “our natural tendency to misperceive and miscalculate probabilities, to think anecdotally, instead of statistically, and to focus on and remember short-term trends and small-number runs”. In a nutshell, we are evolutionarily evolved to clearly notice short term trends but are predisposed to forget or ignore long term trends. The author of these articles goes on to suggest that our intuition has evolved in a manner which enables us to utilize this capability in the context of social interactions and social relationships (which means that our intuition does play an important role in our ability to form alliances and identify social path that could be of some usefulness to us)  we are nevertheless ill equipped to use this capability when it comes to probabilistic problems.

In “Knowing Your Chances” (Scientific American Mind – April/May 2009), the authors make a reference to an early book published in 1938 by the English writer H. G. Wells, who predicted in his “World Brain” that statistical thinking would become an indispensable trait, similar to reading and writing. This prediction, however, has not materialized and the authors of the article make the observation that “At the beginning of the 21st century, nearly everyone living in an industrial society has been taught reading and writing but not statistical thinking – how to understand information about risks and uncertainties in our technological world.  That lack of understanding is shared by many physicians, journalists and politicians…and as a result, spread misconceptions to the public.”

So what does it all mean?

We are all naturally pre-disposed to a certain level of Risk Attitude. Risk Attitude (as defined by David Hillson & Ruth Murray-Webster) is a “chosen state of mind with regard to those uncertainties that could have a positive or negative effect on objectives, or more simply a chosen response to perception of  significant uncertainty”.

Josh Nankivel, based on a podcast by Cornelius Fichtner (which I thoroughly enjoyed while preparing for my PMP) gives a good summary of the commonly referenced Risk Attitudes (a complete copy of which is given below):

1. Risk Seeker – enjoys and seeks uncertainty in search of greater opportunities, can be overly optimistic and not take possible negative consequences seriously.
2. Risk Averse – uncomfortable with uncertainty, doesn’t like risk
3. Risk Tolerant – reasonably comfortable with uncertainty, but usually sticks head in the sand and ignores them
4. Risk Neutral – analyzes risks and weighs negative/positive possible outcomes and probabilities objectively.

Josh makes the observation, which I tend to agree with, that most project managers will tend to be Risk Tolerant. They will conduct basic Risk Identification process early in the piece but then rely on their gut-feel and ‘lets hope for the best’ approach when faced with reality. Josh goes on to suggest that the Risk Neutral is the goal and he is probably (excuse the pan) correct. The problem, as indicated above, is that for most of us this will require conscious effort and elaborate attention to details we are not naturally inclined to adopt.

Formal adherence to Risk Management processes can cut through the complexity and the PMBOK is certainly a good place to start as it refers to the basic tools and techniques required to ensure you manage your risks adequately.